Why DOD is so bad at buying software

The Defense Department is one of the world's largest technology organizations, but it has trouble buying IT, particularly software. It can take years for DOD to make it through the process for buying technology — whether it's software to operate a fighter jet, tactical radios or the latest version of Microsoft Office — and by that time, the technologycan be out of date.

"Software advances every 12 to 18 months, so you're at least one iteration if not two iterations behind in the decisions that you made to get the program going in the first place," Dean Hullings, global defense solutions strategist at Forescout Technologies, told FCW. It also "opens up doors for other people to say, 'Well, wait a minute. We weren't part of this. We have capabilities, too.' And that prolongs the actual acquisition process."

Hullings said that the scuttled Joint Enterprise Defense Infrastructure (JEDI) program is emblematic of DOD's challenges with buying emerging technology. Pentagon officials wanted a cloud service that would enable civilian personnel and warfighters to securely access and share data and digital tools at the highest classification levels. It wasn't DOD's first cloud purchase, but it was a big one. However, the potentially $10 billion contract was subjected to intense political scrutiny and years of lawsuits and protests fueled by accusations of favoritism in how the contract was awarded.

DOD officials canceled JEDI and shifted to a multi-cloud, multi-vendor alternative called the Joint Warfighter Cloud Capability. In the meantime, the department still lacks the central cloud capability detailed in the JEDI solicitation in 2017.

The problem isn't a lack of money. DOD is poised to get nearly $740 billion in fiscal 2022, and investments in cybersecurity, artificial intelligence and general IT are expected to rise in future years. Furthermore, technology is not the issue. JEDI contenders Amazon, Microsoft, Google, Oracle and IBM have long offered robust cloud services, and myriad small and midsize companies are creating new products and improving them every day.

Nevertheless, DOD continues to struggle to buy the technology it needs. The problem has given rise to numerous congressional hearings to discuss reforming the budget and acquisition processes, policy papers on the Pentagon's valley of death (the gap between industry's development of an innovative technology and DOD's deployment of it) and watchdog reports that note where DOD is falling short in its attempts to quickly buy, fund, develop or field software.

The budget balancing act

Rep. Seth Moulton (D-Mass.), who was co-chairman of the House Armed Services Committee's Future of Defense Task Force, said the government will "agonize for months and years over exactly what technology we're going to use five to nine years from now." Instead, he argued that DOD should adopt a software-centric approach to technology acquisition.

"Hardware-centric worked for a long time; that's the way the world worked," Moulton said at the National Defense Industrial Association's Joint All-Domain Command and Control symposium in July. Now, however, "it's very much a software world…. We need to recognize that in a lot of cases, it's the software that is really the core technology, and we just need to be able to bolt on different hardware as it becomes available."

Reforming budget and acquisition processes is a well-worn topic on Capitol Hill and across DOD. But as weapons systems become more reliant on software than hardware, the request for tangible change is intensifying.

In a report published in February, the Hudson Institute recommends that DOD or Congress "sponsor a commission to study holistic changes to the planning, programming, budget and execution (PPBE) and appropriations process structured to ensure that the U.S. has a competitive advantage in long-term competition while maintaining Congress' constitutional role." The report also recommends "a limited-scope pilot project on an alternative resource allocation process, designed to foster adaptability in capability delivery and aligned around a high-priority national security operational challenge," among other topics.

Furthermore, the institute states that software performed 80% of system functions in the F-22 fighter jet in 2000, up from 45% for the F-16 in 1982. Today, software is so integral to the F-35 that the Government Accountability Office recently advised DOD to update the aircraft's modernization schedule, automate data collection on software development performance and set software quality performance targets, which DOD agreed to do.

During a hearing in May, Chairman Jack Reed (D-R.I.) said the Senate Armed Services Committeehas been focusing on reforms to the PPBE process, which has been in place since the 1960s. "It was a model that was appropriate for the Industrial Age, but we're in a post-Industrial Age," he added.

In its final report, the Cyberspace Solarium Commission recommended that the Federal Acquisition Regulation be amended to include cybersecurity requirements and the mitigation of software vulnerabilities, while the National Security Commission on AI (NSCAI) said DOD should accelerate its adoption of emerging technologiesby streamlining its acquisition process.

"Adherence to cost, schedule and performance baselines is rarely a proxy for value delivered but is particularly unsuited for measuring and incentivizing the iterative approaches inherent in AI and other software-based digital technologies," the NSCAI report states. "Unless the requirements, budgeting and acquisition processes are aligned to permit faster and more targeted execution, the U.S. will fail to stay ahead of potential adversaries."

So with evidence stacking up and consensus growing, why hasn't reform happened?

Terry Halvorsen, former DOD CIO and now general manager of client and solutions development for the federal and public sector at IBM, told FCW he doesn't think major acquisition or budget reform is the answer.

"Is the acquisition system bad? No," he said. "Is the acquisition system as responsive as it needs to be today to the changes in technology? No, and that's not just a government problem. That's also an industry problem.... I think the government is wrestling with how to streamline that process to be able to adapt to a world where Moore's law is no longer correct." Moore's law states that the number of transistors on a microchip doubles about every two years, but Halvorsen said, "We're changing faster than that."

Overhauling the budget planning and acquisition processes could do more harm than good, he added. Instead, DOD needs "a way to be able to better address the fact that in today's world, we're really not going to be able to predict the technology future."

In Congress, though, lawmakers continue to push the idea of making the budgeting process more flexible. The Senate's latest version of the National Defense Authorization Act for fiscal 2022 includes two provisions aimed at reforming the PPBE process: One would establish an independent commission to explore options, and another would direct DOD's CIO and chief data officer to develop a plan to consolidate the IT systems used to manage data and support the PPBE process.

Halvorsen argued that Congress could help by speeding its own budget approval process. "If you could get so the budget actually passed on time more frequently and was more adaptive, that would probably go a long way toward making things better," he said.

Enrico Serafini, CEO of pExchange, had a different suggestion. He said the best solution is for DOD to efficiently manage its budget to create more accountability. He told FCW that the latest discourse focuses on agile acquisition and how to buy technology faster to achieve objectives faster. "If you can provide the justifications and manage that in smaller cycles, I think that really ought to be the answer," said Serafini, whose company provides budget software, database management and advanced analytics solutions to DOD.

Changing minds, reforming habits

Nicolas Chaillan recently resigned as the Air Force's chief software officer because he said DOD's bureaucracy was at odds with its technological goals. He told FCW that the main problem is a lack of widespread, mandated workforce training in agile software development.

Officials think that "moving fast means you're going to fail drastically and create massive trouble when, effectively, if it's small incremental delivery and you see what sticks and you have the right baked-in security and DevSecOps...it's actually the only way to do business successfully," Chaillan said.

He argued that so-called colorless money — funding not explicitly tied to a specific appropriation category or military component — should be the way DOD does business. Additionally, he said, the department needs to foster true joint collaboration on key technologies, such as cloud architecture, AI and DevSecOps, and those teams should report directly to the secretary of defense.

"It's not just money, it's talent," Chaillan said. "We keep saying we have more money than China. No, we waste 90 cents on the dollar, so we don't really have more money than China."

Lt. Gen. Dennis Crall, director of command, control, communications and computers/cyber and CIO for the Joint Staff,echoed that sentiment and said DOD wastes money by not using modern software development practices.

"Technology is not my problem in this area," Crall said at the National Defense Industrial Association's symposium in July. "We waste a lot of money by bringing things in that have all the right protocols and development, but we probably have dozens of developers' toolkits.... We don't have good records management for some of these things so it's hard to trust them…. If this has been tested under these protocols and these standards, then the answer is you will have reciprocity and you won't spend the money to retest them."

The term "DevSecOps" has almost become pejorative because the methodology is often used incorrectly, Crall said, adding that officials need to clearly define and standardize the approach across DOD.

Chaillan said DOD could also improve security and accountability by being its own integrator of modular open systems instead of having companies do that work. "We need to understand enough to make architectural decisions," he said. "And when you start playing with Lego blocks instead of a big system, you want to be able to swap blocks…. It's not as big of a lift as some people think still today, particularly in software, and I think that's how you end up building the right fit and being able to try new things and be very agile and get rid of one Lego block replaced by another."

Forcing change from the inside

To try to improve the system from the inside, DOD has launched a series of pilot programs to test the effectiveness of buying software through a single budget activity rather than from multiple categories. The programs carve out research and development dollars for certain software-heavy programs, including the Algorithmic Warfare Cross-Functional Team, the Army's Defensive Cyber Operations, the Global Command and Control System-Joint, and the National Background Investigation Services.

In addition, DOD has adopted a software acquisition pathway. The Defense Innovation Board's Software Acquisition and Practices (SWAP) Study, released in 2019, recommended the approach along with a rewrite of the DOD Instruction 5000 series, the policy document that guides how the department buys and delivers the solutions it needs. The redesign was meant to allow for continuous software integration and delivery based on the technology industry's best practices.

"The pathway objective is to facilitate rapid and iterative delivery of software capability to the user," according to the Defense Acquisition University's website. "This pathway integrates modern software development practice such as agile software development, DevSecOps and lean practices."

Before DOD launched the software acquisition pathway in 2020, it was hard for DOD to prepare for future technological changes, said Hullings, who previously served as chief of cyber requirements at Air Force Space Command.He recalled making the argument for funding such preparations by using an iPhone analogy at a time when iPhone3 was the latest version. He told command leaders: "I need to develop requirements to counter threats coming from the iPhone 7...because we know the iPhone 7 will have threats and vulnerabilities that today we cannot counter. I need to start working on those solutions today, and I can't do it without money."

He said he believes the software acquisition pathway could help make those conversations less challenging.

The approach is still relatively new, so DOD doesn't yet have data to measure how the program is doing. Tory Cuff, senior adviser for agile acquisitions to the undersecretary of defense for acquisition and sustainment, previously told FCW that it would take time to "truly understand the impact of all these efforts that we are doing and updating within acquisition." Cuff said about 20 programs had adopted the pathway, and officials expect to see some initial data in early 2022.

Better together?

The fiscal 2022 budget request is the first time DOD's latest acquisition reform efforts are coming together. According to budget documents, DOD's pilot program to use colorless money for buying software covers eight programs. Officials proposed adding the Navy's Next-Generation Enterprise Network (NGEN), worth more than $950 million, but it was missing from the House Appropriations Committee's markup of the budget bill.

Stephanie Kostro, executive vice president for policy at the Professional Services Council, said the pilot programs mean "you can take the software development from cradle to grave or at least from cradle to fielding."

But she raised concerns about the pilot programs becoming a slush fund for non-IT-related items. "We've seen this in several other programs that are not software-specific, where something becomes a good idea and everything gets attached to it," said Kostro, who formerly served as policy director for the House Armed Services Committee. "Things like HVAC systems for heating and cooling are now part of the NGEN program. And so it makes me wonder: Why can't you procure HVAC units and cooling systems through the normal procurement process?"

She said the pilot effort could be tweaked to focus on software and relevant infrastructure. "When you have a good idea within the U.S. government, a lot of people want to hang stuff on it...like a Christmas tree," Kostro said. "But then they start hanging extra ornaments on it...and then it might topple over."

By contrast,Eric Lofgren, a research fellow at the Center for Government Contracting at George Mason University's School of Business, said the pilot programs don't give DOD enough flexibility to move money as needed.

"When you pull the funds out of [operations and maintenance] and then you put them into basically a research and development line item,…you've basically closed off that flexibility to reallocate funds between programs in the O&M phase," Lofgren said. "Within an appropriation, you still have program elements, [and it's] very difficult to move money between those program elements. Flexibility between programs is potentially more important to the department in terms of its ability to access technology and scale technology quickly and in a relevant time frame."

Still, Lofgren praised the pilot effort and said pairing it with the software acquisition pathway could result in more effective use of DevSecOps. He also noted that the Air Force's Kessel Run software factory could potentially join the pilot program. The recently released Kessel Run All-Domain Operations Suiteallows the software factory to continuously push out code to users for testing in an operational environment so developers can get quick feedback.

He added that the success of the software acquisition pathway is "a cultural question" because it comes with steep requirements, such as submitting a life cycle cost estimate. "Software is never done," Lofgren said. "But then there's this issue of...what is a life cycle cost estimate because it seems to be a continuing capability.... They just keep adding to it and fixing it and modifying it until something better comes around and displaces [it]."

Kostro said the effort is still in the initial exploration phase, "but if this ends up being as flexible and agile of an IT approach as intended, [DOD] can really leverage commercial best practices and use the contractor community…to address near-peer competitors."

The end results would be helping the U.S. "degrade the ability of a country like Russia to sort of rock an ally or even the United States from the inside by cyberattacks," which happened with the SolarWinds breach.

Jerry McGinn, executive director of George Mason University's Center for Government Contracting, said he expects to see more acquisition pilot programs focused on data-intensive capabilities, but that expansion could lead to a tug-of-war between DOD and Congress.

"You're going to see the tension between executive and legislative branches," he said. "The executive branch is going to want maximum flexibility, and Congress is going to not [want that] because they want oversight."

McGinn also predicted that a successful, expanded acquisition reform effort at DOD could fuel a burgeoning interest in undertaking major budget reform. "It's going to be hard to totally junk [the PPBE process]; there are too many vested interests," he added. "But maybe that is the right answer…. The way we do things makes it hard to [buy] software and these new technologies."

The true test of success will be whether these software acquisition approaches gain more widespread popularity, McGinn said. "What about aircraft carriers or Bradley [tanks]...would software be a separate line item in those in those budget requests [and] be colorless? Would that be part of the budget submissions for all the major platforms? If it starts going in that direction, then obviously this is sticking," he said.

Nate Ashton, managing director of public policy at Dcode, told FCW that putting software in its own spending category "doesn't necessarily align with how systems are built and operated. The next fighter jet, the next submarine, the next tank that's being developed is going to be a software system with some hardware components, not the other way around. Software doesn't exist in a silo anymore. It's fundamental and integral to every system that's being developed or operated across the board."

Ultimately, Ashton said DOD's ability to pique the interest of innovative technology companies will hinge less on new acquisition methods and more on DOD's successful adoption of emerging technologies, such as the Army's Integrated Visual Augmentation System.

"The more success stories that we end up seeing out of it, that's what's going to drive increased demand," Ashton said. "The venture community is paying a lot more attention to the government than they were a few years ago...[and] looking at the government like an essential customer."