How the CIA is Making Sense of Big Data

For decades, the CIA’s spying strategy remained relatively unchanged, perhaps reflected best in the fact that last year’s creation of a Directorate for Digital Innovation was the first new directorate within the spy agency in more than 50 years.

Helmed by Deputy Director Andrew Hallman, the new entity is a result of big data – the technological explosion of connectivity and data among devices, sensors and people – and the CIA’s wish to make sense of it all.

In a rare public appearance at Tuesday’s Cloudera Federal Forum, which was hosted by the custom events unit of Nextgov's parent company, Government Executive Media Group, Hallman said the CIA’s old information collection strategies couldn’t “withstand the accelerating rate of information” produced globally or “keep pace with policymakers’ more rapid need to make decisions.” In short, the CIA wants to be more predictive and less reactive.

“We are developing policy approaches that affect outcomes instead of becoming reactionary,” Hallman said. “We’re not here to chase the news.”

With the CIA’s technology backbone – including a cloud computing environment built by Amazon Web Services and other unique capabilities – now fully in place, Hallman said the spy agency is “taking next steps” to solidify its strategy around harnessing this explosion of information.

Big data technologies allow analysts to piece together the “digital exhaust” of adversaries, Hallman said. And even “small fragments” represent major opportunities. Analysts can use disparate data and accompanying analytic capabilities to put together a puzzle, even if all the pieces are spread randomly over the kitchen table.

“The whole can be pieced together from fragments,” Hallman said.

Hallman said the directorate is focused on three ideas: “Optimizing the CIA’s underlying digital ecosystem; establishing sound data management principles; and making data useful to increasingly sophisticated consumers.”

These ideas, he said, represent not only a major technical change to the agency’s operations, but a cultural transformation as well. An increased emphasis on data requires a reduction in old bureaucratic and management policies – allowing the agency to get out of the way of itself and its analysts, who act as stewards of the intelligence community’s greatest asset.

Optimizing the digital ecosystem requires a “decentralized governance process with just enough high-level guidance to operationalize,” Hallman said. “Information is the currency of the realm in the intelligence world, and we want all our analysts to have (currency) in their pockets.”

If the CIA has rebuilt its technology infrastructure, the next step is “rewiring its organizational neural pathways” to ensure, for example, that the kinds of lapses in information sharing that preceded 9/11 don’t reoccur. He’s put an emphasis on rapid dynamic teaming, or digitally-networked teams, which collaborate in real-time “to storm vexing problems” when strange trends emerge or data suggests “subtle shifts or discontinuities.”

The emphasis is not only on sharing information with the CIA and other agencies within the IC, but “making the data useful to increasingly sophisticated consumers," Hallman added. 

“Given the complexity of national security challenges we face today, we have to optimize not only technology but our officers’ minds,” he said. “We are developing policy approaches that affect the outcomes instead of becoming reactionary.”