Now Streaming: Government Data

The concept of data streaming is not new. But one of the most critical emerging uses for streaming data is in the public sector, where government agencies are eyeing its game-changing capability to advance everything from battlefield decision-making to constituent experience.

IDC predicts that the collective sum of the world’s data will grow 33%, to 175 zettabytes, by 2025. For context, at today’s average internet connection speeds, 175 zettabytes would take 1.8 billion years for one person to download. Streaming has only further accelerated the velocity of data growth. That includes in the government—as the Department of Commerce alone handles more than 20 terabytes of data per day, more data than the entire Library of Congress printed collection would total.

When leveraged effectively, streaming data offers tremendous potential for real-time improvements, allowing agencies to collect usable data from the start. Tapping the transformative power of public sector big data is key to modernization.

A Data-Driven Government

A recent study shows 82% of federal agencies are already using or considering real-time information and streaming data. Their needs are driven significantly by IoT’s evolution, where collecting, storing, and analyzing data becomes critical. Sensors deployed across government assets—whether in national parks, warfighting domains, highways, hospitals, satellites, farming, food-processing facilities or elsewhere—are continuously generating and collecting data. In addition to streaming data’s adoption at the Defense Department’s Joint Artificial Intelligence Center, it’s also being employed in data-powered smart cities and by host of other agencies bolstering their visibility, cybersecurity, constituent experiences, advancements in research and development and more.

To effectively leverage data for decision-making, agencies need a modern data platform capable of tapping into machine learning and artificial intelligence to analyze large amounts of log data in real-time. Most critical: cutting down the time to receive insight from collected data.

For example, in health care settings, you must process patient data like heartbeat and oxygen levels as close to real-time as possible. Paired with machine learning at the edge, informed medical decisions based on these real-time data streams are now possible. Paring down processing time will significantly improve agency mission effectiveness and, in turn, their outcomes.

Adopting machine learning capabilities allows agencies to reduce the time needed to gather useful information and make smart decisions. While many federal officials recognize these emerging technologies’ potential, implementation is too often hampered by limited experience, silos of data, and limited amounts of pre-labeled training data.

While these obstacles may seem difficult to overcome, working with a partner who has access to talent, executive leadership and multilevel buy-in can help accelerate adoption at any agency. In addition, agencies should evaluate use cases and plan their data journey accordingly, establishing a sound data strategy as the baseline for decisions.

A Data Strategy

At the highest level, a data strategy is a guide to highlight the significance of organizational data, recognize its unique attributes, and establish a set of principles to aid decision-making. A robust set of policies aimed at making data consistent, clean, and available across the organization’s hardware implementations is key to impacting everything from constituent services and infrastructure planning to resource allocation and national security.

Government agencies are tasked with migrating masses of data, often with an added level of security to safeguard critical data. Agencies also operate under regulatory constraints that determine who can use what data, in what ways, and for what purposes, adding additional complexity levels.

With these restrictions, agencies should look to unified and automated platforms for security, governance, and intelligence. Automated solutions can provide context to existing and newly generated data, increasing visibility, and understanding across the full spectrum of data and analytics activities within their organizations.

Without combining automation for real-time analysis with a robust data strategy, agencies cannot possibly keep up with the massive amounts of data generated by sensors in the field.

Whether it’s Census Bureau researchers and analysts using the agency’s troves of data to estimate survey response levels, the IRS and the Centers for Medicare and Medicaid Services utilizing ML and AI to uncover and mitigate billions of dollars in fraud, naval aviators using ML-based predictive analytics to track and manage aircraft maintenance, the value of leveraging automation to glean insight from data is being realized today—and underscores the promise for tomorrow.

Henry Sowell is the chief information officer for Cloudera Government Solutions.