An important new guide to agile

Think about it for a bit, and it should become clear why a waterfall style of software development -- where one develops requirements and a work plan in advance, and only then sets the contractor about the job of producing the system -- became so entrenched in government for so long.

This approach appears to embody rationality -- thinking should precede and guide action. In everyday language, we admonish people to “look before you leap” and not to “go off half-cocked.” The ancient Chinese military strategist Sun Tzu wrote in The Art of War that, “the general who wins a battle makes many calculations in his temple ere the battle is fought. The general who loses a battle makes but few calculations beforehand.”

This idea of rationality also has a special place in bureaucracies, including government bureaucracies.

In procurement, the idea of establishing requirements in advance in an RFP is also a venerable part of the culture -- although the level of detail for requirements envisioned in procurement is less than that elicited in a waterfall development process. Here, though, the main justification involves fairness – if all bidders are bidding on the same thing, and what that thing is gets defined in advance, it becomes harder for the government to play favorites.

Agile software development has emerged because it turns out that common sense in reality isn’t so commonsensical. The reason is essentially the limits of what people can learn in advance during the thinking stage.

One limit is that the users of an IT app or system themselves are poor at knowing what they want from the product until they actually use it. So even if requirements development includes consulting the users -- which it often does not -- that consultation is likely to produce flawed information.

The second limit is that those developing requirements often cannot know how the world will change (new technologies, new uses for the product) between the time the requirement is generated and when the software is ready, so requirements will often come to be outdated by the time the development process is completed and the software handed in.

Agile is based on a different philosophy, which is to work quickly to get some version of an actual product, however incomplete, out there very quickly for users to experience. That hands-on feedback can then be used to further develop the product, quickly and through multiple iterations.

This approach is now dominant outside of government for developing software. Inside government, probably the best way to describe the status of agile is as a work in progress. In 2016 the U.S. Digital Service and the Office of Federal Procurement Policy published the Tech FAR, which dealt with many of the contracting issues around agile; I wrote a blog back then called Contracting for Agile about these efforts. Around the same time, USDS published a Digital Services Playbook, which was a great first pass at some of the management issues associated with doing agile.

Deloitte Consulting’s Center for Government Insights, headed by Bill Eggers, has now taken the Digital Services Playbook several steps further by producing a new guide, Agile in Government, to give specific, actionable advice to government managers embarking on an agile journey.

Perhaps the two most important parts of the guide are the related discussions of the role of the agile “product owner” (a new phrase from the agile world) and contract management for agile.

The guide notes that the product owner has three main jobs:

• to represent the business owners/users to the contractor development team (including getting reactions to newly fielded capabilities);
• to participate in the frequent demos (perhaps as often as every other week) of new capabilities;
• and to ensure that the government’s decisions are communicated to the developers.

The product owner needs to understand the business and the government users. As the Deloitte guide explains, this is a different job from that of the project manager in waterfall: “While the project manager focuses on resource management, keeping the project on track in terms of time and expense, the PO maintains a laser focus on ensuring that the software delivers the business benefits envisioned by the agency.”

The important, cautionary lesson here is that the frequent interactions between agency project leadership, developers and end-users in agile mean that contract management for agile is likely to be more resource-intensive than for waterfall.

The guide notes the “common pitfalls of under-resourcing and under-supporting” the jobs. In waterfall, the significant interaction between government and the contractor takes place first during the development of requirements, and then much later on when the contractor submits what is hoped to be completed work. In between, contract managers are basically overseeing reports from the contractor about work (said to be) completed; they are normally not seeing ongoing product demonstrations or learning user reactions to what has been produced.

The guide argues that the product owner job should be full-time, or almost full-time, during an agile effort. Lack of resources and/or skills for staffing this is one of the biggest constraints for the spread of agile, and it is fantastic that the guide is clear about this. (Some agencies may choose to hire a contractor as product owner, if there is not government staff, though this in some sense is regrettable.)

The guide also has a good section on what it calls “agile at scale” -- projects using agile that don’t include only one or a few apps, but come closer to the kinds of complex systems the government has often bought in the past using waterfall methods. A key insight here is that, in addition to the dimension of agile vs. waterfall development method, there is an independent dimension called simple vs. complex.

It is harder to succeed using waterfall than agile, but complex projects are harder and more prone to failure than simpler ones, even if agile is used. Agile for complex projects shares with waterfall the need to sequence and coordinate the production of different software development teams.

The guide even discusses the need for “multiyear roadmaps,” with many subtasks, to guide the effort forward -- almost a sort of advanced planning approach that might remind one of waterfall, an indication that simple vs. complex is a dimension independent of agile vs. waterfall.

The advantages of agile here relate to its culture -- just as agile for simpler projects encourages lots of interim, iterative demonstration and testing, rather than waiting to the end, so does agile at scale encourage the same for coordination of the production of different agile teams. In agile at scale, a governance mechanism for cross-team communication is important.

“Multiple Agile coaches and product owners should regularly convene a ‘scrum of scrums’ to facilitate cross-community communication," the guide states. "The Agile community of leaders…provides a more integrated and up-to-date message format that allows…interested stakeholders to receive real-time updates about progress during and after attending in-progress product demonstrations or working system capabilities.” Compared with waterfall, more is done earlier, in time for easier course corrections.

All of which is to say, if you’re getting started with agile in your agency, you need to check out Agile in Government.