CIA’s New Big Data Hub Will be Hosted in the Cloud

The CIA is preparing to take the next step in its quest to shake up the status quo of siloed agencies within the intelligence community.

CIA Chief Information Officer Doug Wolfe confirmed Wednesday the intelligence agency will start using Cloudera’s Enterprise Data Hub platform by April, a move he expects “to extend the innovation and push the envelope on a whole range of different solutions” for all 17 IC agencies.

The enterprise data hub, also known as a “data lake,” would presumably provide standardized data sets compiled by intelligence analysts across various agencies to decision-makers among many other features found in the company’s widely used open source big data platform.

This hub, however, will be hosted on Amazon Web Services’ C2S cloud built for the IC, inaccessible to the public.

Cloudera’s data hub is one of many uses the CIA has been tasked with since it launched the cloud last summer, including a software marketplace that could revolutionize how IC agencies procure software.

Wolfe, speaking Feb. 25 at the Cloudera Federal Forum in Tysons Corner, Virginia, said the enterprise data hub should up and running on Amazon’s cloud in a month or so. The event was produced by Custom Strategies, a division of Government Executive Media Group.

“Building it on C2S allows you to get all their goods and products and serve those out," he said. "They do a pretty good job at packaging up open source solutions and providing quality control value add, but you’re still getting the benefit of open source development. I think that’s a good balance for us and will make adoption a lot easier for a lot of our customers.”

Customers – in this case, other IC components – running on virtual private networks within the AWS-built C2S cloud will have the ability to access Cloudera’s Apache Spark environment and security solutions.  

The CIA was able to leverage Cloudera through new license agreements that will last no longer than three years. The CIA already had a relationship with Cloudera, Wolfe said, although not in this kind of capacity. The company has received funding in recent years from In-Q-Tel, the IC's venture capital arm, in large part because of the company’s big data and analytic potential.

“My interest in doing this was to accelerate the opportunity to adopt goods and services and make it much more standard for people who use some of these products,” Wolfe said. “We’re very hopeful that it will be a very effective partnership.”

(Image via agsandrew / Shutterstock.com)