Does 'early and often' work for software?

“Vote early – and often,” was the byword of the wardhealers in the glory days of Chicago’s Democratic Party machine. Modern-day cynics might argue that’s still the case in Chicago and elsewhere around the world. But now the idea is gaining steam in other government quarters — namely, information technology departments. Instead of voting, though, people are now encouraged to deploy early and often when developing software for an important new application.

The catalyst is a methodology known as agile software development. It attempts to turn nimble, easy-to-change coding practices into a formal strategy that produces better software faster.

The strategy is a repudiation of traditional programming approaches, in which software developers receive detailed written requirements and then go off to toil in IT seclusion, sometimes for years, before unveiling the finished product to users. In the meantime, new business priorities and technologies might have surfaced that render some or all of the original requirements obsolete.

Instead, the agile approach forges a close working relationship between programmers and users. Together, they work on small chunks of the new application and conduct a series of reviews in the weeks it takes to write each iteration, called sprints. That feedback loop keeps the two groups in almost constant communication and allows for quick reactions to technology innovations and shifting requirements.

However, some observers say it is unrealistic to expect users to spend that kind of time when they have their regular responsibilities to attend to. They also question whether agile development can work in the conservative government environment, which is steeped in policies and processes that favor certainty and set schedules over shifting project scopes and changes on the fly.

Nevertheless, a growing number of federal IT managers have decided that agile development’s benefits are worth whatever effort it takes to clear the way for its use.

“It reduces a lot of risk, and you get capabilities into the users' hands much faster than before,” said Rob Vietmeyer, a Defense Information Systems Agency project director for Forge.mil, the Defense Department’s information sharing and software development Web site.

DISA used agile techniques to deliver initial capabilities for Forge.mil in just three months, a third of the time it would have taken using traditional methods, Vietmeyer said. Early testing also helped DISA spot security gaps months before they would have surfaced otherwise.

NASA has been an active agile development user for three years for agencywide applications, including financial management, human resources, procurement and identity management systems.

“We’re believers,” said Gene Sullivan, NASA's associate chief information officer for enterprise portfolio management.

The agency's enterprise operations moved exclusively to agile development after officials compared error-tracking logs of traditional projects to early efforts using agile development. The summaries showed a large reduction in the number of defects and change requests from the user community, Sullivan said. NASA has now used agile development for 19 projects.

Such successes are bringing a growing number of IT organizations to the agile party. Forty-five percent of the organizations surveyed this year by the programming magazine Dr. Dobb’s Journal said their development teams have adopted agile techniques. Of those, 69 percent said they were well beyond early implementation stages.

“In government, we see it particularly for mission-critical, safety-critical, DOD kinds of organizations” and for Web development, said Dave West, a senior analyst at Forrester Research.

Sullivan said the biggest incentive for using agile development for those types of applications is the ease of establishing users' requirements for new software.

With the traditional development model, “we would miss requirements and have a hard time getting our user community to identify what the correct requirements were,” Sullivan said. The approach would reveal problems so late in the process “we’d either have to delay rollouts or there wasn’t enough time to do adequate testing.”

Agile detractors

But not everyone is on board. Features that distinguish agile development from older methods — such as the SWAT team approach to delivering code quickly — could cause problems for government agencies, said Michael Daconta, chief technology officer at systems integrator Accelerated Information Management and a columnist for Federal Computer Week's sister publication Government Computer News.

Daconta wrote a column last summer that included agile development on a list of six IT trends that government managers should be wary of adopting. “In my more than 20 years of software development experience, I have never met a government program manager who is available on a daily or even weekly basis to help design an application on the fly,” he wrote.

West acknowledged that the pace can result in interdepartmental strain. “The business may not want to engage effectively with [developers] and frankly may be quite happy with the status quo because they’ve got other things to worry about,” he said.

Even thornier issues arise when agile development schedules bump into government budget and acquisition processes. For example, before systems development can begin, DOD requires acquisition programs to meet a host of checklist items, including the ability to demonstrate that the project’s technology works in the relevant environment. That creates a steep hurdle for developers who will be creating software capabilities on the fly over multiple years.

“Agile says we really can’t know what the world is going to look like two years from now so it doesn’t make sense for us to even attempt to fully plan, define and test prior to start,” Vietmeyer said.

Finally, rather than reducing costs, agile development could spawn new expenses.

A cousin of agile development called extreme programming calls for pair programming, in which two developers share a workstation and trade off writing code while the other watches for errors.

“Pair anything will produce higher quality, but are you willing to pay double the cost?" Daconta said. "That’s literally what you’re saying here.”

Workarounds

But advocates say the problems aren’t insurmountable. The payoff for getting stakeholders to sit down with developers every few weeks typically outweighs scheduling headaches.

“That’s the price to be paid to get something written quickly that matches your needs,” said John Gilligan, an IT consultant and former CIO at the Air Force and Energy Department.

In fact, once managers start seeing their feedback materializing in new code, they might be eager to participate.

“We’ve yet to meet customers that really didn’t want to be involved when projects are mission focused and [there’s a] need to get them out to the users who need them immediately,” said Amy Rall, division director for homeland security software solutions at QinetiQ North America, a systems integrator. The company has used agile development for a mainframe modernization project at the Homeland Security Department and for other projects.

Although program managers might struggle to make agile methods comply with DOD’s acquisition rules, Forge.mil and other projects have found a way to merge the two cultures. Vietmeyer has worked directly with the CIO and other policy staff members to show some agile development successes at DISA and get help when traditional practices are barriers to the new approach.

Flexibility might be another success factor. The Dr. Dobb’s survey found that only about 27 percent of agile development users closely adhere to a particular agile methodology, while the remainder mix agile variations and some non-agile methods.

Rall said QinetiQ defines a project's scope as a series of sprints and describes deliverables in a catalog of features, two agile components. But then it creates status reports using traditional tools, such as the Gantt charts project managers use to illustrate work elements and schedules.

“When you say to a business unit, ‘Here’s that backlog of features' and line that up against your traditional Gantt chart, they get very excited,” Rall said. “People still want to see those milestones [to] let them know what’s what.”