Achieving gains in government IT performance with DevSecOps

A software development team in the Boston office of Kessel Run, a program within the DOD's Defense Innovation Unit (U.S. Air Force photo by J.M. Eddins Jr.)

Eli Whitney, the inventor of the cotton gin, demonstrated the value of interchangeable parts in 1801 to the U.S. Congress, President John Adams and President-elect Thomas Jefferson. Whitney proved the viability and the military value of interchangeable parts by stripping down several muskets, then reassembling a functional musket from random parts from the disassembled muskets.

Today, we take for granted that parts are interchangeable -- from the bolt carrier of a rifle to the alternator on a transport vehicle, we assume that one is as good as another. But, as with information systems developed today, muskets of that era were bespoke artisanal creations. The parts for any given firearm were custom fitted to accommodate the variation in manufacturing for the other components comprising the whole. A gunsmith would be necessary to replace the hammer or pan of a musket and return it to working condition. The same can be said for information systems today that often require a specialist or team of specialists to configure, deploy, modify or repair in the instance of a failure.

To address the bespoke nature of information systems and to gain the same types of benefits for information systems that interchangeable parts brought to manufacturing, the Department of Defense is adopting DevSecOps. It's an approach that has seen accelerated growth in the public sector over the last two years, especially within DOD and warrants a closer look.

First, what is DevSecOps? DevSecOps is a combination of processes, tools and people, which combine with enterprise values across the disciplines of Development, Security and Operations. DevSecOps form a unique culture to enable more efficient delivery and management of secure software.

The integration of security augments the DevOps practices seen in industry. It’s important to integrate security throughout the process in government adoption to effectively reap the benefits of iterative improvements. Traditional development processes more often incorporate security as a checkpoint that needs to be passed, but does not integrate security concerns throughout the process. DevSecOps elevates security into a first-class citizen, instead of a bolt-on checkpoint. The adoption of DevSecOps is extensive across the federal government including the General Services Administration, Air Force, Army and Navy.

DevSecOps Processes

From a process standpoint, DevSecOps can be seen as an extension beyond the software development lifecycle practices found in agile development and Continuous Integration and Continuous Delivery (CI/CD) methodologies to improve the operational behaviors of the deployed system, including security. Taking the principles of continuous deployment, applying them to operations management and introducing configuration as code, operational tasks can be automated and through this automation, increasing resiliency.

Ultimately, the result can be push-button automation -- the ability to completely redeploy a component of infrastructure from bare metal to full operational capability by kicking off the appropriate automation playbook.

DevSecOps tools

The list of tools utilized in DevSecOps are myriad. It’s more important to have the right classes of tools than to have precisely the same tools as another DevSecOps-practicing organization might use. DevSecOps is a combination of all three pillars of processes, tools and people -- there is no single product that can be purchased. Unfortunately, there’s no such thing as a DevSecOps box we can install in a datacenter.

The collection of tools are focused around source code version control, build automation, test automation, security validation, performance testing, configuration management and extend into project management systems that permit prioritization, issue tracking and team collaboration.

DevSecOps people

The people component of DevSecOps is often the most challenging in the DOD. Conway’s Law says that organizations tend to build products with a design that reflects the communication structure of the organization. Organizations organized into silos trend toward applications built into silos. With the tight integration of roles required for effective DevSecOps adoption, many government agencies are seeing a need to flatten their organization structure and integrate IT professionals into cross-functional teams aligned across a product, rather than maintaining role based internal organizations that communicate through ticketing systems.

This restructuring and alignment of personnel helps drive results-driven outcomes by bringing everyone together, working toward the same goal: the successful release of their product.

DevSecOps advantages

DevSecOps brings several advantages to the table for DOD agencies, including shorter time to value, faster iteration to field new capabilities and moving risk left. The most valuable advantage is shortened time to value, whether that value be measured as innovation, reliability, or reduced rollout lead time.

The Waterfall process, the most common traditional development process, however, focuses on extensive requirements documentation and development up front. This can set goals for the development of features and capabilities for the first release of a product that do not all have significant impact or provide wide-reaching value across the user base. DevSecOps is able to bring more value to the enterprise, more quickly, through its iterative feedback process.

DevSecOps hurdles

One of the more significant hurdles in adoption of DevSecOps in government is the concept of the Minimum Viable Product (MVP). However, the MVP is a key component in the value achieved by moving risk left.

The Pareto Principle, or 80/20 rule, resonates even in product development. Often, 80% of users only use 20% of a product’s capabilities. If a product is developed through the traditional waterfall model, 80% of the users may be blocked from getting value out of the product while waiting for the development and testing of features they won’t use. The saying “perfect is the enemy of good” applies when trying to satisfy the urge to complete every desired feature in a waterfall release schedule. This not only complicates the integration and testing of all of the features, but delays the delivery of valuable features.

DevSecOps is positioned to significantly influence the traditional monolithic acquisition and development processes the federal government has canonized through the nearly 2,000 page long Federal Acquisition Regulation. The FAR defines the processes that must be followed for the acquisition of products and services by executive agencies. Because of the guidance of these regulations, the rhythm of the FAR processes strongly influenced the adoption of waterfall development practices. However, industry has demonstrated that the waterfall model of software development has significant shortcomings when it comes to the development of innovative, novel solutions.

Problem-solving with DevSecOps

When solving a new problem, there are two types of unknowns. Known unknowns are challenges we know upfront we will need to tackle. Unknown unknowns are those unforeseen challenges we will face as we break new ground in the production of a novel solution to an unsolved problem.

When dealing with the production of innovative solutions, the unknown unknowns are the true stumbling blocks that cannot be discovered or predicted when following a Waterfall model until after the schedule is set. This is another area where moving risk left is accomplished by maintaining small, strategic, incremental changes through the DevSecOps feedback loop. If a particular feature is intractable or sufficiently complex, it can be reprioritized, abandoned, dissected, or reimagined much earlier in the development process, before it can impact and possibly prevent the delivery of other valuable capabilities.

Many people, when thinking of DevSecOps, imagine cloud-native web applications using microservice or serverless architectures, or container-based workloads. But DevSecOps is not a process only for the development of these modern architectures. It can be used to develop traditional monolithic applications, n-tier architectures, service-oriented architectures and even embedded systems. DevSecOps, however, significantly improves the viability of producing these more sophisticated systems.

One of the key concepts being developed in the DOD is known as the “software factory.” Software factories are a realization of the concepts of DevSecOps and are focused on the remediation of the bespoke, artisanal nature of IT systems fielded by the DOD today. Software factories correlate with another development of the industrial revolution, the assembly line. A software factory is dependent on the adoption of DevSecOps principles, but the implementation of DevSecOps does not imply the creation of a software factory, just as the success of the assembly line is dependent on interchangeable parts. Through the adoption of DevSecOps principles, the DOD is moving quickly towards leaner, more responsive, more efficient armed forces.

Note: This is the first in a series of articles, co-published by FCW and the IBM Center for the Business of Government, looking at the impacts the adoption of DevSecOps is having on the federal government. Future articles in this series will explore these issues in greater detail.