Data overload threatens with rise of smart tech and real-time sensors

Marc Demarest is CEO and principal at Noumenal, a principals-only private intellectual capital consulting firm that specializes in IT, biotechnology, nanotechnology and clean technology.

How data is collected, analyzed and used is changing rapidly as real-time, sensor-based monitoring applications grow in popularity. In response, the federal business intelligence community must prepare for and embrace that change if the government is to remain fully effective in its regulatory, oversight, law enforcement and public safety roles.

Sensors and monitoring applications — whether placed on a street corner, on a car, inside the electric grid or inside a dialysis machine — produce streaming data that is live, continuously changing, structurally invariant and voluminous.

Marc Demarest will discuss the promises and perils of the streaming data revolution and its influence on performance management at the TDWI Government Summit April 5 in Arlington, Va.

The military, homeland security, financial, manufacturing and energy sectors increasingly rely on streaming data. Collection devices are appearing in the health care, transportation and public safety sectors at astounding rates.

“Smart” will soon be a staple of the daily lexicon. Smart buildings are being outfitted with sensors to gauge energy use and carbon emissions, and smart hospitals track medication and supply inventories.

For government, the challenge in adapting to this new reality will be to unlearn what it has been taught about business intelligence. Professionals in that field will still need to design and implement data warehouses and integrate the client-side environment. But if they apply long-established methods, they might experience spectacular failure because streaming data is fundamentally different from traditional business intelligence in three important ways.

1. The extraction, transformation and load process is no longer needed. Streaming data is loading constantly, so there are no updates, only inserts. Sources are widely distributed — there could be tens of thousands of them — and the target schema is usually simple.
2. Traditional business intelligence tools are not used with streaming data. Instead, analytics must be performed in or close to the database programmatically rather than relying on ad hoc query and reporting tools. The primary mission of streaming data analysis is to spot anomalies and outliers for further investigation, then perform a more iterative analysis to determine the root cause of the abnormality, what it means and what actions are needed.
3. Data visualization must be dynamic. The sliding time windows and multiple data dimensions required for effective sensor-based business intelligence aren’t well illustrated by static pie charts and visualization tools in spreadsheets. Heat-map and whisker diagrams that change with incoming data are needed.

The government business intelligence community still has time to adapt to this new reality while the technology industry works to create more powerful database management systems and a networking infrastructure with the processing capacity, flexibility and scalability required to effectively analyze massive amounts of data in real time.

Business intelligence professionals should begin now to educate themselves and visit colleagues who focus on applications for the military, energy management, and other agency programs that are ahead of the curve in using sensors and streaming data — or even consider switching jobs to one of those environments.

As is often the case, there will likely be cultural resistance to this change. However, if business intelligence professionals view streaming data as an exotic type that must be dealt with only now and then, they risk being drowned in what the National Science Foundation has characterized as a coming tsunami of sensor-based data.

Worse, failure to respond effectively to the new technical challenges posed by streaming data could render federal government agencies ineffective or even irrelevant in fulfilling their growing functions to regulate and oversee markets and ensure the well-being of their constituents.

Every area of our lives is preparing to go real time in its day-to-day functions. The federal government and its business intelligence community need to go real-time, too. And the time to prepare is now.

Editor’s note: Marc Demarest will discuss the promises and perils of the streaming data revolution and its influence on performance management at the TDWI Government Summit April 5 in Arlington, Va.