What’s happening in agile

I recently had a chance to catch up with Mark Schwartz, the CIO at Citizenship and Immigration Services in the Department of Homeland Security. Schwartz is one of the leaders in the government's move toward agile software development, whom I first met when he was a participant in a Kennedy School executive education program. (I have blogged about him and agile several times in the past.)

My goal in following up was to find out from him whether there had been any important developments in agile software development in the government in the last year or two.He didn't disappoint.

Schwartz referred to a major development called "continuous delivery" (one aspect of a somewhat larger concept called DevOps). The basic idea is to shorten the cycle time for new software releases from the several weeks of an agile sprint (which, itself, is obviously a huge improvement on traditional government processes) to one day – or even, at some point, several times a day. This is done by making the requirement that developers are working on at a given time be truly bite sized -- a single discrete change in the system -- and by automating the delivery process.

Having the changes come in these tiny chunks has advantages even over traditional agile development, not to speak of older waterfall processes. Any time some new capability is fielded, there is a risk it won't work right. The larger and more complicated the release, the greater the risk of a problem, and thus of the need for costly rework (something that has always bedeviled waterfall-type development). If very tiny capabilities are added each time, very quick feedback from users is possible, and fixing problems is far easier because there is less sunk cost in what has been fielded. This approach also avoids the problem that arises with multi-week agile sprints that deliver multiple features. Even with agile, real bottlnecks can occur as some new features are ready earlier, but need to be held off to batch into what comes out of a sprint.

Continuous development got its big boost in 2009, when representatives from Flickr, the photo-sharing service now owned by Yahoo, talked at a conference about doing 10 new software releases a day. With the speed obsession of leading tech companies, there has developed a sort of "can you top that?" competition -- Amazon apparently can push out thousands of releases a day.

The government is unlikely ever to become Amazon (indeed, Amazon's tech competitors have trouble becoming Amazon), but certainly by government standards continuous delivery displays a lot of agility. At this point, in the government only Citizenship and Immigration Services (and some work supported by 18F) has projects that are delivering every day. Yet continuous delivery looks poised to spread soon elsewhere in government, especially given a recent hire by 18F of one of the leading experts on the topic in the tech world.

To make continuous delivery possible, many aspects of the process of actually getting the new code fielded must be automated, such as testing, deployment, configuration, and security. That's what USCIS has done. Oversight of software also needs to be done in a new way for continuous delivery to work. From the overseers' perspective, continuous delivery has advantages because the tiny releases are lower-risk and the ability to see immediately whether a new feature is working increases transparency.

USCIS has put in place a policy encouraging frequent deployments and laying out the appropriate control processes for them. For example, a team that uses continuous delivery must be able to roll back its deployments quickly if anything goes wrong, and must manage all of its code, test scripts, and deployment scripts in the agency's version control system.

Schwartz says that at this point all of his new projects are using continuous delivery.

And in an amazing coup for the government, 18F has just hired Jez Humble away from his position as Vice President of Development Velocity for Chef, a Seattle-based automation company for IT infrastructure and applications development. Humble is author of the book Continuous Delivery and a lecturer at the UC Berkeley School of Information.

Schwartz believes the pace of recruitment from the tech world into 18F, USDS and agency-based digital services is accelerating, something for which he gives former federal CTO Todd Park, who now lives in the Silicon Valley, much of the credit. Getting people from the tech industry into government can develop a snowball momentum of its own, as each new recruit makes it easier for other techies to make the leap.

These people are being mostly brought in as term-hired Schedule A employees, with few becoming civil service lifers. We may be on the cusp of an important transition in the federal career path hiring model, the development of a short-term (but non-political) career track to complement the traditional civil service "stay in government all your life" model.

I have advocated this idea for a long time, given the changes in values among young people (who expect to have several different careers during their working years). It is exciting to see this new model beginning to emerge; it is appropriate for lots of government jobs, but particularly so in the tech area as a way to compensate at least partly for the government's difficulty, for salary and other reasons, recruiting young tech talent.