Integrators aim high on software methodology

The software development processes used by federal contractors to write the programs that run the on-board systems of the space shuttle, fighter jets and other complex equipment are graded, like a lot of things in government.

Not every company's development process goes through an evaluation, and not every company publishes the results, but some of the ones that do in recent months have announced their processes have attained the coveted Level 5 rating, the highest on the software Capability Maturity Model.

The CMM is a way of judging the maturity of the software development processes and identifying the key practices that are required to increase the maturity of the processes.

The CMM, which ranges from Level 1, the rating given to ad hoc, labor-intensive development, to Level 5, representing disciplined development that is well-managed and supported by technology, has evolved over the past 12 years. Its stewardship is the responsibility of the Software Engineering Institute, a federally funded research and development center at Carnegie Mellon University, Pittsburgh.

When the SEI was established in 1984, vendors and government officials acknowledged that quality software produced on schedule and within budget was a critical component of U.S. defense systems.

The software that runs the on-board systems of the space shuttle is an example of a product that was developed using processes that have achieved the first Level 5 rating, said Mark Paulk, a senior member of the technical staff at the SEI. The software was written by a group of engineers who originally were employed by IBM Corp. and now are part of the United Space Alliance, a joint venture team composed of Boeing Co. and Lockheed Martin Corp.

Errors that occurred in the group's software were so trivial that engineers had to do detailed post-shuttle flight analyses to find out what they were, Paulk said.

There are about 12 companies worldwide that use development processes worthy of the Level 5 rating, but only six have publicized the results of their evaluations.

The most recent Level 5 ratings have been announced by Computer Sciences Corp.'s Systems, Engineering and Analysis Support Center; Lockheed Martin Federal Systems; and Raytheon Systems Co.'s Command and Control Division. The other three are Boeing's Defense and Space Group, Motorola India Electronics Ltd. and the Ogden Air Logistics Center Software Engineering Division at Hill Air Force Base, Utah.

Other companies point to their certification under the ISO 9000 series of standards developed by the International Standards Organization.

There are a number of benefits to reaching the Level 5 rating. Among them are higher productivity and lower costs due to the reduction in the amount of reworking necessary, said Joe Castellino, director of program engineering at Raytheon.

In previous years government contracts formally asked vendors to include a software evaluation in their bid, but more recently the strong discriminator in contract awards has become past performance, Castellino said. However, the rating clearly is a factor.

"If you're at Level 5, you'll have better past performance. The higher SEI rating...strengthens our competitive position relative to being able to show past performance," Castellino said.

Among Raytheon's three groups that went through the evaluation and received the Level 5 rating were the software developers working on the Federal Aviation Administration's Wide-Area Augmentation System, which will allow aircraft to use satellite-based navigation. Castellino said, however, that the process used by the group is used throughout the company.

Paulk said there is a growing catalog of documentation on improvements in software development as companies move up the CMM scale.

"The first thing that happens is the prediction of the cost, and the schedule [of a development project] gets a lot better," Paulk said. "Typically, projects are over budget by double and [are] one year late. At Level 2 the target and actual performance are close to one another. At Level 3 you see the variation diminish dramatically."

Software developments completed by Level 5 organizations usually are within 3 percent of the estimated cost and estimated time, Paulk said, adding that it has been fascinating to see how the CMM has decreased the number of defects in software.

In every 1,000 lines of code produced by any typical software shop there are one to five defects. Level 5 development groups commonly report one to five defects in every million lines of code, Paulk said.

One study has shown that every maturity level correlates with a 13 to 17 percent decrease in effort compounded as you move up the levels, said Paulk, whose familiarity with the research and participation in numerous evaluations has made him a staunch backer of the CMM. His opinion is likely to be shared by companies vying to do more complex projects.