Pentagon Seeks Software to Analyze Data from Twitter

The Defense Department is looking for open-source intelligence software with access to at least 50 million websites and the ability to ingest data from sources like social media to create graphics—including location maps—for analysis in real time, according to a new solicitation. 

Open-source intelligence, or OSINT, refers to intelligence produced from publicly available information. This can include information like location data, which is often collected from smartphone apps and social media sites. The Defense and intelligence communities have used OSINT for decades, but the advent of the internet and social media has greatly increased the amount of public data available for collection and intelligence analysis.  

The DOD solicitation, which was published Thursday on beta.SAM.gov by the Pentagon’s Washington Headquarters Service, is focused largely on Twitter, the only social media platform the solicitation explicitly names. 

The solicitation is a small business set aside specifically asking for software fulfilling five requirement program elements and 12 characteristics. The vendor must be able to alert the government via its platform and through email messages, access at least 50 million unique websites, be a “vetted Official Twitter Partner,” provide the entire Twitter historical archive for analysis and be able to handle data in at least 150 languages. 

The complete list of required characteristics includes: 

• “The ability to ingest near-real-time social media feeds from Twitter and other platforms.
• The ability to simultaneously search this data for customized Boolean data sets.

• The ability to analyze this ingested data and immediately present the intelligence graphically in various formats, to include on geospatial maps and over time horizons.
• The ability to compute and highlight trend analysis based on the data sets.
• The ability to perform sentiment analysis of the data sets based on shifting online attitudes.
• The ability to distinguish between real authors and online bots which may be pushing
• disinformation.
• The ability to identify public reactions and significant events as they spike on social media.
• The ability to analyze all parts of online articles and search for keywords.
• The ability to export this analysis into comma-separated values for use in Excel.
• The ability to alert the Government of situations that warrant immediate attention.
• The ability to ingest and interpret data of most major languages.”

During AFCEA Inernational’s 2020 TechNet Cyber event last week, Col. David Violand, the deputy director of intelligence for Joint Force Headquarters Department of Defense Information Network, or JFHQ-DODIN, said in a conversation about cyber threats OSINT is an “invaluable tool” for intelligence efforts. 

Violand said DOD has developed an OSINT element of its own to help “bridge the gap between IC and DOD reporting, and frankly what the rest of the world is seeing.” 

“Through the use of publicly available information and subscription services, the OSINT team is able to tip network defenders, enrich our analysis and provide indicators that may not otherwise be available,” he added. 

Heather Williams, a senior policy researcher at the RAND Corporation, told Nextgov OSINT can be a valuable tool because it is often less expensive than other technologies, but commercial platforms may not be well-tailored to the particular needs of the defense and intelligence communities. Many commercial platforms that can be used for OSINT were developed for the marketing industry, she said.

“For example, a brand might be interested in … reputation, whereas the intelligence community is trying to understand the historical evolution of a problem,” Williams said, adding that solicitations of OSINT software likely aren’t going away any time soon because there’s a need to stay relevant. Williams co-authored a RAND report on OSINT in 2018. 

And while the same rules regarding the extent to which the intelligence community can collect and retain information about U.S. persons apply to OSINT, Williams said it may be more difficult to adhere to those rules when using OSINT.

“It’s pretty clear that that’s true because you’re just more likely to have a big vacuum that sucks up a lot of information and that information to have U.S persons information in it,” Williams said. This problem—how to maintain privacy—is one commercial vendors don’t necessarily have to deal with, she added.  

The offer deadline for the OSINT solicitation is Dec. 21.