Buy or build? For IT, it's custom vs. COTS

If agencies had all the money and time they needed, most would probably still prefer to build their own IT applications and house them on hardware located on agency premises. But that option is no longer viable.

Now, IT departments are governed by an ever-present squeeze on budgets and resources and a brave new world of 24/7 expectations — from the public to provide information and services and from their own employees for the up-to-date tools they need to do their jobs.

The appeal of building solutions in-house will likely never go away. The bespoke approach, after all, comes with the ability to exercise greater control over development and quality, integrate new technology more tightly with existing applications, better track documentation for ongoing support and react more quickly to requirement changes.

Some agency IT leaders have even pushed back against a headlong rush to the new. Last year, Terry Halvorsen, who was the Navy Department’s CIO at the time, cautioned that new is not always the answer.

“In today’s world, where technology changes in the blink of an eye and the race to be faster, more capable, bigger, etc., moves at lightning speed, we sometimes forget to ask ourselves that very simple question,” he wrote in a blog post. “As prudent stewards of [Navy] resources, we have to decide when it makes sense to invest in new technology” and when to stay with something that is still reliable and fully supports the Navy’s mission.

Nevertheless, new — even if that means embracing a modified version of old — is the goal for most agencies. The question then becomes whether to buy or find a way to efficiently and cost-effectively build.

From custom to COTS

Commercial off-the-shelf software has long been the answer for many government needs. For example, agencies have used Microsoft’s suite of word-processing, spreadsheet, collaboration and email applications for years.

However, the more complex the requirements that COTS software is expected to meet, the more that software must be customized. At some point, customization becomes so extensive and costly that it makes more sense for agencies to build the software themselves.

The Defense Information Systems Agency reached that point a few years ago when officials reviewed COTS applications and concluded that the available options were not appropriate, were too expensive or would have to be changed too much to meet the agency’s workflow requirements.

On the occasions when DISA officials turned to outside developers for help, they were told the applications could not be built at all.

So they decided to develop their own software, aimed initially at internal needs for human resources, security training and personnel management activities. The resulting Web-based application now comprises some 110 modules and is available to government and nongovernment entities as the Open Source Corporate Management Information System.

The open-source community’s credo of reusing software modules is becoming a standard way for agencies to spread the cost of software development among many users and thereby benefit from some of the volume-based advantages of COTS. The Department of Health and Human Services, for example, has made reuse part of its enterprise architecture principles, one of which states that HHS “evaluates investments against business requirements and service needs, with a philosophy of first reuse, then buy, then build.”

More recently, the General Services Administration latched onto reuse as a major driver for its IT projects. In 2014, then-CIO Sonny Hashmi made “platform reuse first” one of the nine key principles that would guide the agency’s IT modernization. In doing so, he essentially melded open-source with cloud-first policies.

Capitalizing on GSA’s existing investment in common application and infrastructure platforms “not only reduces IT costs and complexity, it also reduces the burden on GSA’s end users by reducing the number of disparate applications and tools they need to learn and access,” Hashmi said at the time.

He added that he hoped the approach would eventually lead GSA to build custom solutions for no more than 20 percent of projects.

Cloud can cut down on the one-offs

Deciding whether to borrow, buy or build requires a clear-eyed assessment of an agency’s particular needs and a broad understanding of what existing solutions can and cannot do, said Dave Zvenyach, director of acquisition and management at GSA’s 18F.

“In practice, much — if not most — of the things we need in government are very similar to needs experienced outside of government,” Zvenyach said. “In those cases, we borrow or buy. But where the government’s needs differ, we’ll look to borrow or build.”

Over time, he added, the hope is that the number of situations in which open-source solutions are borrowed will outnumber the situations in which they must be built.

However, some agencies will continue to need custom applications. For instance, the classified networks of intelligence agencies and some parts of the Defense Department cannot securely be connected to the Internet, which means that DOD and the intelligence community have missed out on some of the advantages of cloud services. That could be changing, though.

“The intelligence community has started to look at how to solve that problem by having public cloud providers come in and build a public cloud in a government facility,” said Stan Tyliszczak, staff vice president for technology integration and chief engineer at General Dynamics IT. “That gives them all the advantages of public cloud technology and the expertise associated with that, while still operating in a constrained environment where the physical and virtual access to resources is restricted for anyone who is not in the community.”

Last year, the CIA took the leap into the cloud after striking a $600 million deal with Amazon Web Services in 2013 to develop a private cloud for the 17 agencies that make up the intelligence community. Now users at those agencies can get a variety of computing and analytical services on demand from the CIA and the National Security Agency.

The deal was the result of a longtime effort by the CIA to boost the performance of its data centers and provide its analysts with better tools. It gives the agency “the flexibility to optimize resources across different workloads at different times,” agency CIO Doug Wolfe said in an interview last year with FCW’s sister publication GCN.

That should significantly improve the “time to mission,” he added, because it now takes just days to set up new data center environments and new analytics, rather than the months or years it took before.

NSA and DOD agencies with similar connectivity concerns are considering the same approach, Tyliszczak said.

Data vs. applications

Other agencies are discovering that they can get the advantages of the cloud while also having applications and services targeted to their specific needs, Tyliszczak said. For instance, they benefit from the economies of scale and operational efficiency that come with developing in the cloud rather than building an application themselves in a small-scale computing environment, and they shift the cost model from one that largely relies on upfront capital expenditures to one that has lower recurring costs.

“These days, you really should be able to put most of what an agency does into a cloud,” Tyliszczak said. “There will always be some legacy application that performs some unique function with unique interfaces that can’t be updated that way, but the reality is that much of the workload that agencies have today can be moved into come kind of cloud-based environment.”

There are other ways to look at the issues. Officials at Hitachi Data Systems Federal, for example, say the focus should be on the data that applications produce and where they need to go rather than on the applications themselves.

“In the past, agencies would buy a solution and then customize that to get the functionality they needed,” said Brian Houston, the company’s vice president of engineering. “What you would then find is multiple silos of infrastructure for the mainframe, for a [Network File System] type of environment, another for databases and so on.”

Instead, he said that if agencies have an infrastructure that allows them to move data to the appropriate performance tier, it typically doesn’t matter whether the application is built, bought or customized.

“If you build the infrastructure so that it is scalable and flexible enough, so that data can go wherever it’s needed no matter the protocol involved, you really take the risk out of the whole build/buy question,” Houston said. “The real issue is not about the application but what type of data needs to reside in the cloud, say, versus what needs to reside in an on-premises type of environment.”

He admitted that it isn’t always easy to convince agencies to follow that approach. But once they see the capabilities, it starts to make sense as far as budget and cost justifications are concerned. And it fulfills the mandate that agencies move to the cloud while allowing them to retain control over applications and data, which is still a fundamental concern.

Rephrasing the question

Officials are also rethinking how the often-turgid government procurement process could be revamped to meet the growing demand for new and better application functionality and the increasing speed at which applications and services must be produced. That’s why GSA has been developing special procurement processes for its upcoming blanket purchase agreement for agile services, which will enable the 18F program to get vendor help with user-centric design, agile software development and DevOps.

“Ultimately, we want access to the broadest pool of vendors possible in order to establish a marketplace,” Zvenyach said. “In this ‘alpha’ phase of the agile delivery BPA, we are using IT Schedule 70, which has thousands of vendors” and makes it relatively easy for other companies to qualify for an IT Schedule 70 award.

In the end, government technology development might no longer be a choice between buy or build.

“There are some things we’ll build, some we’ll buy, some we’ll commission to get certain parts,” said Aaron Snow, acting executive director of 18F. “Our mission is to help our agencies buy and build great digital experiences, however that may work out.”

Whether you use COTS, build custom solutions or do something in between is not really the question anymore, Tyliszczak said.

“It’s more a matter of how well the products you are buying off the shelf fit the agency’s need and how much customization is needed for that,” he said. “So it’s a hybrid of COTS, some customization and maybe some build, and the real challenge is in identifying how much of each is required.”