How DHS is Trying to Separate Quality Cyber Tools from Snake Oil

At its core, cybersecurity is a tight-lipped business. Organizations want to keep their digital weaknesses under wraps and bad actors want their activities to fly under the radar.

But a lack of communication can also keep groups in the dark about how well they’re protected online.

Today it’s difficult to assess how well different cyber tools defend against different types of attacks because that information is largely unstandardized and rarely made public, according to Erin Kenneally, who manages the cyber portfolio within the Homeland Security Department’s research division.

Companies each sell their proprietary solutions “as the greatest thing since sliced cheese,” Kenneally told Nextgov, and industry is left deciding what products to buy without much hard data to rely on. As such, organizations have a tough time accurately assessing their own cyber posture and predicting what the fallout would be from an attack, she said.

But through the Cyber Risk Economics program, or CYRIE, the agency’s Science and Technology Directorate hopes to give groups more transparency into the market for cyber solutions.

CYRIE hinges on the notion that rational actors will maximize their well being, a key tenant of economic theory. If organizations better understand the potential costs of cyber threats and benefits of security tools, the idea goes, then they would be smarter about investing their limited cybersecurity funds.

Through the program, the department is funding research in a dozen areas related to bolstering organizations’ cyber awareness, but Kenneally said most existing projects focus on two major themes: sharing threat data and quantifying risk.

Cybersecurity companies are generally reluctant to share proprietary information on their tools, which makes it difficult to single out the best products on the market, she said.

“Companies that are putting out [valuable] products ... are having a hard time disambiguating themselves from other companies,” said Kenneally, so the department is exploring ways to judge security tools’ quality without revealing too much proprietary data.

In January, Kenneally’s team awarded a Virginia-based cyber intelligence firm $350,000 to develop a platform where security experts can bet on which products would best defend against different types of simulated attacks. The system would offer descriptions of each tools’ attributes without revealing who developed them, allowing companies to anonymously crowdsource expert advice on potential threats and weaknesses.

“The end result is we end up sharing data across institutions,” Kenneally said. “I actually think this is the approach to addressing cybersecurity going forward.”

On the other side of the equation, her team is also investing in projects that standardize the system for judging organizations’ cyber risk.

Today many groups offer risk-scoring services to private businesses, but it’s often unclear how those scores are calculated. Companies might weight certain factors more than others, so it’s nearly impossible to get an “apples to apples” comparison of cybersecurity across the market, Kenneally said. Her team is currently finalizing a contract with a company to measure the relative quality of these different scoring services, she said.

And while it’s critical organizations understand how well they’re secured, Kenneally said it’s also important to know what the economic consequences will be if things go wrong. Armed with that information, companies could invest in security tools that make the biggest positive impact on their bottom line, and society at large could gain a better understanding of the toll of cyberattacks on consumers, something Kenneally said is largely ignored today.

To that end, her team is funding research at the University of Illinois, Chicago, with the intent of building a platform that crunches data and estimates the impact of cyber incidents in real time.

“We've got such widely disparate calculations,” Kenneally said, with estimates of the total economic cost of cyberattacks ranging from tens of millions to hundreds of billions of dollars. Without more accurate figures, industry and policymakers could find it nearly impossible to efficiently mitigate risk.

“We need to sing the same tune at some level,” she said. “There's such a wide variance, we need to come to more of a common ground.”