Air Force Buys Streaming Data Warehouse for Pathfinder, JADC2 Concepts

The Air Force’s digital directorate awarded a contract worth up to $100 million over five years for a data warehousing capability that will pull together information from independent systems covering the air, land, sea, space and cyber domains to create models and visualizations that will improve situational awareness and allow operators to pinpoint threats to the homeland quickly. 

After a yearlong prototyping effort with the Defense Innovation Unit, data firm Kinetica was selected to support the North American Aerospace Defense Command and Northern Command Pathfinder program. The new contract expands the prototyping work by incorporating more domains and classification levels to the Kinetica capability.

Dan Nidess, a DIU contractor who managed the prototyping effort, told Nextgov Kinetica’s success during the prototyping effort stemmed from the company’s ability to address issues with the underlying data ecosystem on which end-users and operators rely. 

The sheer number of sensors, systems and tools operators must observe while monitoring for threats has only increased, but architectures have not been able to prevent this data from becoming siloed into separate systems that can’t work together.  

“They might find something of interest on one monitor that raises their attention and they need to turn to different systems to get amplifying information to try to figure out what it is—is it an aircraft? Is it a bird? Is it possibly a threat?” Nidess said. “I'm sure you can imagine that that eats up a tremendous amount of time for them just trying to manually correlate all these different feeds, time that could be better spent analyzing the situation and making a decision about it.”

The response time for turning raw sensor data into a visual representation on a map using the Kinetica capability can be as fast as a second, Amit Vij, co-founder and president of Kinetica, said. Vij said Kinetica can support datasets with trillions of records. 

“We started a database from the ground up leveraging, as opposed to a serial kind of compute model, a massive, parallel kind of compute model,” Vij said. “In addition, we can run analytics right within our database … The same video cards that render video games, we can render pictures and videos that are created right on the fly.”This means ingested data doesn’t have to be shuffled out to a separate visualization mechanism, which often creates bottlenecks in the software architecture stack, Vij added. Artificial intelligence and machine learning models will also help to pull out anomalies from the data, separating important information from the noise that can cloud it with false positives. 

“As the system, as the platform accumulates more information, it gets more and more intelligent, and produces those alerts based on past historical observed behavior,” Vij said about the future vision of the project. “And if it's within different tolerances that the government can change and adjust, it can produce more relevant information to the operator for them to execute against.”

Kinetica’s data streaming warehouse is deployed on Amazon Web Services’ GovCloud and is powered by graphics processing units from NVIDIA and Intel central processing units. 

The Kinetica project with the Pathfinder program is connected to the Advanced Battle Management Systems effort, which is also led by the Air Force. ABMS is the platform supporting the Defense Department’s Joint All-Domain Command and Control, or JADC2 concept, by creating a military internet-of-things framework connecting sensors through a united network.