Spy Agency Seeks Data Miners

Director of National Intelligence James Clapper's office is launching a new research program to root out trends in public data on news websites, blogs and social media to predict events like revolutions, humanitarian crises, economic collapses and disease outbreaks.

The DNI's Intelligence Advanced Research Projects Activity office is soliciting research proposals for sophisticated software and other tools that can sort through "noisy data" to pull out meaningful patterns, according to an

>agency solicitation posted Wednesday.

The agency is interested in developing "methods that leverage population behavior change in anticipation of, and in response to, events of interest [and] processing of publicly available data that reflect those population behavior changes," the solicitation said.

The U.S. intelligence community has used publicly available or "open source" data since its inception as a supplement to satellite and signals intelligence and on-the-ground spying. It's less common, though, for the intelligence community to look at spikes and valleys in data rather than the data itself, essentially focusing on "volume rather than depth" as Wednesday's solicitation says.

Similar open source pattern assessments have been used in industry and in the NGO world. The Fund for Peace, for example, uses a similar process to develop its annual Failed States Index, which gauges how likely national governments and civil societies are to collapse from domestic turmoil and international pressure.