Ways to Jump Cultural Hurdles to Realize Effective Government DevSecOps

Over the most recent years, cybersecurity incidents have become increasingly common. According to research by security firm Risk Based Security, 15.1 billion records were compromised last year, an increase by 284% compared to 2018.

While some of the breaches were the result of sophisticated hacks, many might not even have happened had vulnerability testing and other security measures been baked into the development process. Security breaches come with follow-on costs as well; they can often make organizations wary of innovating, fearing they might introduce vulnerabilities. But one approach promises to make secure innovation a reality: DevSecOps.

Billed as a way to build software faster while fostering collaboration and shared responsibility between developers, security and operations teams, DevSecOps is the combined result of processes, tools and culture in equal measure. The model brings in security earlier in the development lifecycle to spot and minimize vulnerabilities and bottlenecks. The end result is secure software, delivered rapidly to any environment with predictable quality.

Two decades ago, stability and security of software meant longer release cycles. However, modern DevSecOps demonstrates that stability and security are best achieved when automated testing and security are infused from the beginning. 

This method is sometimes called the “shift left” or a holistic “systems thinking” approach, in which the impact of a change in one area must be considered throughout the system and in terms of end-user experience. DevSecOps aims to make that type of shift left security thinking the norm, instead of it being bolted on at the end. And while it’s relatively simple to execute from a technical standpoint, as with any new technology, the status quo can often get in the way.

Cultural Challenges

Despite its increasing popularity in the commercial sector, DevSecOps has had limited implementation among government agencies, mainly because of cultural challenges, experts say. 

“One of largest misconceptions is you buy a tool and think you're done,” says Reuben Cleetus, senior cloud solution architect at Microsoft. “Or, say that you went to an agile class, so you’re good. That’s not quite how it works.”

Cleetus outlines the top three hurdles government IT leaders have adopting DevSecOps.

1. Leaders are often bogged down in heavy decision-making processes, which are the antithesis to building software quickly and securely.

2. Teams also might not be open to experimentation, a key tenet of DevSecOps, because they fear failure. 

3. Finally, leaders may think moving fast means compromised stability and security, when, in fact, the opposite is true.

“Organizations with a mature DevSecOps practice have security that is an order of magnitude better and a lower change-failure rate [in production] when compared to companies that don't practice DevSecOps,” Cleetus says. 

A New Way of Thinking — and Developing

To overcome these obstacles, IT leaders need to reframe their thinking. 

Traditionally, a developer wrote a piece of code and sent it to the operations team with a script to deploy it. Security was punted to much later in the process. That’s all changed with DevSecOps. 

Cleetus urges agency IT leaders to think of DevSecOps as democratizing the process. Everyone in the development lifecycle takes ownership of all aspects of the software, including security, not just one person or dispersed teams. This approach, however, requires significant cultural changes, including breaking down many of the silos that persist at government organizations as well as improving collaboration and lines of communication across agencies.

“We have to start emphasizing the performance of the entire system, and not just a single part or silo of a system or group,” Cleetus says. “It's the entire team taking responsibility for a piece of software through the entire lifecycle.” 

This is also where the psychological safety of teams to flag problems and raise concerns is imperative. 

“You’re no longer throwing a requirement or code over a proverbial fence, and priorities like finding security defects and meeting customer needs are goals shared by the entire team,” as Cleetus puts it.

In essence, DevSecOps drives developers to think about the risks and to work side by side with their security counterparts. Doing that ensures the team is gathering the right telemetry so everyone is aware of the risks and flaws and can quickly act on them. 

Fostering team ownership and having stakeholder involvement is hard to overestimate, Cleetus says. Be sure to create and reinforce feedback loops in each stage of the DevSecOps lifecycle to measure, prioritize, address and assess changes. Mature DevSecOps practitioners work to maximize the team’s empathy with customers by having frequent direct interactions between engineers and end users. 

Another common practice is using telemetry from systems to guide and validate incremental changes to assess the impact across the entire system. Something as basic as why something is being built at all, should be reinforced by feedback and telemetry. 

“This understanding comes from having end-user empathy so that you understand both the threats that customers are facing and their intents and purpose,” Cleetus says.

It’s also important to view the project manager or sponsor as the person who can provide context to team members — about the environment, where their code will function, the purpose of the code, and the role of the organization and the customer, rather than dictating requirements.

“That context is the most valuable piece of information that a team can get,” Cleetus says. 

Finally, create a culture that fosters experimentation and learning from failure.

The “fail-fast” lesson from the commercial industry is that experimentation can yield better ways of solving problems, and often avoids costly long-term problems by disproving a hypothesis. Like its predecessor, agile, DevSecOps is all about developing an iterative and collaborative approach that can drive organizational transformation, Cleetus says. But it can’t be done by simply checking a box.

“It's not buying a piece of software, and it's not going to a class and then getting a certificate,” he says. “This is a cultural change, and it's hard. It's a top-down commitment that involves ripping down silos.”

“These are quite ambitious goals,” he adds. “But these are achievable goals that the commercial industry has been embracing, and we’re seeing some federal agencies beginning to embrace them, too.”

Be sure to check out other topics covered in this series:

• Rapid Deployment: Why DoD Is Ready for the DevSecOps Era

• Shifting Left: How DevSecOps Strengthens Agency Security and Risk Management

• How Public Sector Developers Can Achieve DevSecOps Through Collaboration and Open Source Tools

• Breaking Down Silos: DevSecOps Makes Security Everyone's Business

• Government Innovation, Readiness Will Require More than Just DevOps