The next big thing in IT procurement

At the beginning of the summer, I published a blog on how the Department of Homeland Security had tried an innovative approach to its FLASH competition for an agile contracting BPA, run into problems evaluating bidders, and finally withdrew the procurement -- with DHS Chief Procurement Officer Soraya Correa taking responsibility for the mistakes while defending the idea of trying new ways of doing things. The original blog focused on Correa’s work to overcome the culture of fear in government. But it also mentioned the nature of DHS’s innovative approach, where source selection “centered not around a traditional written proposal but instead around companies actually performing a real agile task that was video-recorded and examined as a central part of proposal evaluation.”

After many of the winning vendors wrote a letter to DHS praising the innovative procurement method, I wrote two more updates showcasing the small non-traditional IT contractors who were bidding for FLASH. While working on these blogs, for the first time I came across the phrase “tech demo” to describe DHS’ source selection approach. When all the non-traditional vendors I spoke with for the blogs stated they preferred source selections involving tech demos to those with traditional proposals, I decided I needed to learn more. This blog is the result of my investigations.

There are two big advantages of using tech demos for source selection. The most important is that it embodies a philosophy of “do, not just say.” This represents a break with the extravagant essay contests that characterize too much of traditional federal procurement, with lengthy, self-serving disquisitions on the company’s technical or management “approach” to the work at hand, slavishly keyed to every request or requirement in the agency’s RFP, in a ritual that sets the government far apart from how work is competed in the commercial world. By contrast, tech demos correspond to the common commercial approach to choosing a software developer using what are typically called “coding challenges” -- timed responses to an application development scenario where bidders submit what they have done and the customer evaluates it for completeness, ingenuity, error-freedom and security.

The second advantage of tech demos is related to the first. Traditional government source selection is strange and unfamiliar to most vendors outside the government contracting fraternity, and represents an important barrier to entry for new entrants into the government marketplace. Using tech demos is a way to bring in new competitors.

Michael Palmer, the DHS career IT program manager who ran FLASH, had been on detail for seven months at the U.S. Digital Service in 2016 and was asked by Correa to join a new DHS digital services team just after he returned. Talking more like a 20-something tech nerd than a long-time fed, Palmer told me that DHS “wanted to provide amazing digital service companies to our program offices so they could have fantastic outcomes.” While planning the FLASH acquisition, he got some advice from USDS.

There were 110 bidders with no screening downselect. Bidders were given a time and location to show up. They were asked to develop an app for giving kudos to an employee. They had only four hours to work. Everything was filmed, and afterwards each team had 30 minutes to make a presentation about what they had done and what choices they had made.

DHS’s evaluation looked not only at the technical quality of the code but also their skills at user-centered design. “We gave each offeror a user in the room to interact with, and that showed us a lot in a very short amount of time," Palmer said. "There are people out there who are highly skilled at design, and we saw some of them on the teams that were proposing. They were not requirements managers or graphic artists simply re-branded as designers, which is something that frequently happens nowadays as companies attempt to demonstrate that they have design capabilities.” (The importance of design skills, Palmer said, is something he learned at USDS.)

To evaluate proposals, DHS did an “all hands on deck” call for developer talent within the agency, as well as borrowing help from USDS. Since FLASH, DHS has competed a task order under EAGLE2 using a tech demo.

It turns out that the first federal organization to use a tech demo for source selection was the General Services Administration's 18F, for its agile software development BPA awarded in 2015, almost two years before FLASH. There were 180 bidders, again with no downselect. There was no proposal.

Vendors were originally given five days to develop a prototype (this was extended a few days because 18F got a lot of questions, and the agency was slow in answering). They submitted their prototype together with what Dave Zvenyach, who was managing the process, calls “documentation intrinsic to good software -- a version control system and an explanation for their design choices,” along with their labor rates. 18F thus gave bidders considerably more time to work than DHS later did. “We had lots of discussion about the length of time we should give," Zvenyach said. "I think this was a good choice because we didn’t want this to be Hackathon, but more like sprint cycle.”

Besides DHS and 18F, the other agencies that have used tech demos at this point are the Department of Veterans Affairs, the Center for Medicare and Medicaid Services, and the Department of Health and Human Services.

The VA used a tech demo to select a vendor for a task order for software development to support benefit appeals processing modernization. According to Mark Junda, the contracting officer for that requirement, VA’s digital services team “came to us and suggested this.” The digital services team conferred with 18F to get lessons learned from their agile BPA source selection. “We were all in, it was a neat opportunity to do new things,” he said.

Bidders had 72 hours to work on the coding challenge. The source selection process, unlike at DHS and 18F, also included a fairly detailed technical proposal. Members of the VA digital services team evaluated the coding part of the source selection. The VA also was the only agency I spoke with that required the team developing coding submissions to also be “proposed as key personnel to perform work under the resulting task order barring any unforeseen circumstances” -- a potentially important protection when awards are being made based on a small team's demonstrated talent.

Tech demos from HHS headquarters started in 2014, even before the 18F agile BPA. They were developed independently of other government efforts by Mark Naggar of the HHS Buyers Club, which had been established to promote innovative contracting before either 18F or USDS had gotten off the ground. (I actually blogged about what the Buyers Club did with their demo back in 2014.)

HHS used a downselect, with five finalists out of the 24 expressing interest, and then actually gave each of the five a payment of $5,000 to develop a prototype for digital modernization apps for the HHS planning and evaluation office. This did not have the timed-challenge rush of later tech demos, and there was also a technical proposal that was evaluated. “I think these are so popular,” Naggar told me, “because of they are like reality shows such as Top Chef. You are showing your capabilities, not your writing skills, not what they say they’ve done.”

Naggar also introduced the Center for Medicare and Medicaid Studies to the idea of using tech demos. According to someone I spoke with at CMS, that agency has now done a number of tech demos for task orders under a 2015 law that changed the way physicians were reimbursed. All have been task orders off a CMS BPA for agile software development. (There were only six vendors on the BPA, so no need for a downselect). “One of our contracting officers participated in the inaugural Digital Service Contracting Professional Training and Development Program in 2015," the CMS official said. "The use of demos was discussed amongst the cohort of contracting officers from various federal agencies who were involved in the training program. Working collaboratively with various program offices, incorporating tech demos or prototype exercises was not a particularly hard sell at CMS. We try to meet program offices where they are in their progression towards innovative procurements, and had candidates from various offices who were interested and engaged early on. In fact, many on the program/technical side were excited to incorporate a ‘show me’ rather than a ‘tell me’ element into the evaluation process.”

In addition to a tech demo, CMS generally has required that bidders submit a quality control plan and a roadmap/staffing plan, and often a statement of relevant experience. There was not, however, a traditional full-blown proposal. USDS advised CMS on these acquisitions, and “their skills allowed us to have engineers and data scientists on the evaluation panels," the official said. "Their participation greatly increased our ability to review vendor’s code and complex prototypes.”

There are a few issues involved in organizing these tech demos where practice varies among agencies. First, should agencies do early downselects before having a smaller group actually do a coding challenge? Neither DHS nor 18F has done this, while HHS has. At the VA debriefing on its tech demo, one loser requested that VA in the future should do an early downselect.

DHS’s Palmer favors using downselects in the future. "The use of a downselect could result in a more efficient process for a procurement and end up saving both the government and industry time and money," he said. A downselect economizes on vendor resources, especially important for very small businesses, while also helping the government on its biggest capacity constraint, skilled software developers who can evaluate submission.

Second, should the demos include only a coding challenge, or also incorporate a way to show abilities at user-centered design and/or working together as a team on an agile project? How much, if any, written proposal-like material (e.g. a staffing plan) should agencies require in addition to the tech demo itself. And how much time were vendors given for coding? Agencies differed on all these elements.

Third, only one agency required tech demo bidders to certify that their tech demo people would be key personnel on the work being awarded. 18F worries that having the demo work done by people who won’t continue afterwards compromises the usefulness of tech demos. And notably, one of the complaints in the bid protest against FLASH was that some winners were not represented by key personnel.

Finally (and partly because of the key personnel issue), 18F is now urging the evaluation of code repositories (publically available links with technical information on a firm’s previous coding engagements) rather than code challenges. (These are available only for open-source software, however.)

I asked all those I spoke with how difficult it had been to sell the idea of using a tech demo inside their organization. To my surprise, everyone I asked said it had not been hard.

Most of these agencies had dedicated procurement innovation units, which were on the prowl for interesting new ideas their agency could try. USDS and 18F also provided some version of top cover. This suggests that there is now a procurement innovation ecosystem in government that provides a supportive climate for trying new things, counteracting the popular stereotype of a super-cautious and risk-averse government. Naggar's independent early discovery of tech demos at HHS, separate from digital services teams later working on this, is further sign of the robustness of such an ecosystem. It is good news these people are there, and good news the government is willing to accept and even welcome them.

At this point, I think the main barrier to the spread of tech demos in government is the IT skills gap. Government is not exactly overflowing with developers having the modern software engineering skills needed to evaluate software or the other output tech demo bidders provide. Government needs more people with these skills -- not only for source selection, but for post-award IT contract management and even (dare one say) sometimes to do coding.

In the shorter term, however, I would urge that 18F and USDS be willing to offer their service (for 18F on a reimbursable basis) to help agencies evaluate “do, not just say” efforts. USDS officials told me they were at present not interested in getting into this line of work. 18F officials told me they had worked on several code evaluations for their clients, but always as part of larger engagements. Would 18F be willing to do an a la carte review if asked? The answer is that they don’t yet know. I guess we will have to wait and see.

If some version of “do, not just say” can spread, this will be very good news for government contracting -- not just because it is likely to produce better vendors but also because of a profound cultural shift it represents. I was really impressed when the contracting officer for the tech demo procurement at VA told me that during the agency’s post-source selection debriefing, one of the losing vendors asked whether the agency would be continuing to do this in the future. If so, the company said, it would need to get rid of some of its proposal writers and replace them by software developers. (Perish the thought!!).

If any agency that hasn’t done this before gives tech demos a try, I’d love to hear from you (steve_kelman@hks.harvard.edu). It would be nice to do a follow-up report on the experiences of people trying, or considering trying, this for the first time.