Some services are hard to measure

Measuring the performance of relatively straightforward information technology operations such as network services can be hard enough. But how do you do the same for, say, outsourced application development, an activity sometimes labeled as more art than science?

Keane Inc. has been working with the Executive Office for U.S. Attorneys to help improve the reliability of mission-critical applications it delivers to the Justice Department and attorneys nationwide.

As a first step, Keane has applied industry best practices embedded in the Software Engineering Institute's Capability Maturity Model (CMM) to grade the improvement in development processes and project management disciplines. The project was recently rated at CMM Level 3, with Level 5 being the highest.

But coming up with the equivalent of a network service-level agreement (SLA) for application development services is difficult because there just is not much hard data available, said Glenn Giles, vice president of Keane Federal Systems.

What administrators can do is establish the equivalent of a quality-of-service measurement for application development by assessing what resources employees used to perform the work previously, and then comparing new costs and employee resources to illustrate before and after scenarios, he said.

"The next step will be measuring ourselves against actual SLAs, though it will probably be a general measure at first, rather than involve specific dollar payments and penalties," Giles said.

It may take another three years before this occurs, he added.