Agile at DOD -- going beyond sticky notes and standing meetings

“In DOD now, everybody says they’re doing agile. But when they say they are doing agile, they mean you stand instead of sitting down at meetings, and that you use sticky notes to track tasks. If you do those things, you’re an agile god. They are still doing lengthy requirements development, and they are still designing to exact requirements. But they do this in two-week sprints.”

The speaker is above is Col. Enrique Oti, who was the main presenter at the recent Code for America summit, which I blogged about recently in the context of civic tech involvement with the Defense Department. Oti is a 21-year Air Force veteran who spent many years working on cybersecurity and developed an interest in how Silicon Valley does software development while on a stint at the Hoover Institution at Stanford, where the Air Force sends a few officers each year. He was leaving just as Secretary Ash Carter was standing up the Defense Innovation Unit Experimental (DIUx), and he was one of the initial people assigned there.

The project about which Oti presented was to redesign a system at the Air Operations Center in Qatar, from which planes fly on missions against ISIS. One of the center’s many tasks was aircraft refueling planning.

With many different planes in the air at many different locations, the center needed to figure out each day when each one would need refueling, where it would be when refueling was needed, and what refueling aircraft should meet it, given the schedules and locations of those aircraft. The system the center used to record the wheres and whens of the daily planning schedule was a big whiteboard where people wrote information using markers.

This is a complex task. The input data was a series of math calculations that had been developed for Excel. Because developing a plan took so long, the planners did not have enough time to consider alternative plans that might be more efficient or effective. Also, if a change happened after they had already finished the plan, it was very difficult to update that plan. Instead, they would simply launch another tanker aircraft, which created extra fuel costs.

DOD had a big contract to redesign the entire Air Operations Center, with its tens of applications, using traditional software development methods involving detailed requirements development (very complex for a complex system), coding, and then, when everything was done, testing/retesting and rollout. After 10 years and $750 million spent, no new operating software had yet been delivered.

DIUx’s work in Qatar grew out of a visit from the Defense Innovation Board, which visits DOD installations to learn about how better to use technology (Raj Shah, the first head of DIUx, sits on the Board, as does Jen Pahlka, the founder of Code for America). Shah decided he wanted to take on the project as an example of how DOD could develop software faster and cheaper. Oti signed onto the project as an opportunity to use real agile at DOD. The first minimum viable product coming out of the effort was fielded after four months.

The effort avoided the endless requirements process for DOD software development. “Our requirements were that Raj told me we needed to make life better for users," Oti said. "We began by talking with users about their pain points, and our approach was then to build a minimal viable product around that pain point.” That minimum viable product involved integrating the planning with the calculations so that any decision made by the planner was immediately validated or invalidated, which rapidly sped up the process.

Oti described the big cultural change to allow agile to happen is allowing work to be done by a smaller number of lower-level people, rather than huge, ponderous staff committees that consider everything to death. He feels you need senior leadership who will buy into that approach to make it possible.

Listening to Oti describe what his team did, my reaction was that this was all pretty basic stuff, sort of agile 101. I hardly regard myself as an agile maven, but not much sounded new to me. However, Oti noted that these practices are still quite unusual at DOD, though there are small pockets of use. (He said the defense organization that has most strongly embraced agile is the National Geospatial Intelligence Agency, which Oti consulted for suggestions.)

For me the most interesting parts of Oti’s account were his discussions of product testing, cyber compliance, and personnel security clearances. The basic message was don’t wait until the end. Oti said there are COTS tools very widely used in industry to make these processes automated and continuous. Using these tools, new code is fed for testing as soon as it is ready, then quickly tested and put into use.

“It is a rigorous process, we are proving every day that it meets operating requirements,” he said. Continuous testing of increments of software is crucial to speeding up deployment, and prevents the do-loop of software not being tested till the end, problems discovered, and a cycle of fixes and re-tests. It is never perfect the first time around, but changes can be made based on feedback from initial use. The same approach is used for cyber compliance, using tools that permit continuous updating of cyber compliance throughout the process, rather than waiting to the end.

To deal with the molasses-slow security clearance process for new contractors, DIUx used a little-known provision of the existing regulations that allow for a "24-hour clearance," under which somebody can gain relatively rapid approval for short-term access to attend an activity that deals with classified material but doesn’t allow the person to have a full clearance or persistent access. – a good way to get commercial folks quickly involved in classified DOD areas on a one-off basis to ensure they have context on the problem.

Development work on the new process was done mostly by six in-house airmen, trained in agile by four people from a small agile vendor called Pivotal Labs, rather than the incumbent working on the big system redesign contract. The total cost of developing the new tool was $2 million. The new tool reduces the time to schedule refueling each day from 60 person-hours to four or five. Given that the operations center had been scheduling 1-3 additional sorties a day because of errors and last-minute changes, at $250,000 fuel cost a pop, the new tool paid for itself in a week.

DOD has a great success story here. After spending $750 million and 10 years on a traditional waterfall systems redesign big bang, the operations center had nothing to show. Agile allowed them actually to deliver an initial working capability in one area after a few months.

I will confess I was disappointed that, by Oti’s own account, DOD has not moved, after a fair number of years -- and two decades after it seriously began to spread in the private sector -- further towards agile. Will this be a turning point for agile in DOD?

The Air Force has announced they will now use agile for the big systems redesign for air operations centers. The tanker story got some considerable attention even before Code for America summit – Oti had been speaking about it so often that if anything when we talked he appeared tired of the topic. The Qatar whiteboard has even been memorialized by being sent off to the Air Force Museum. A visible success counts far more than a lot of theorizing. Stay tuned.