Why Dark Data is the New Shadow IT

It was just a few years ago that federal agencies were trying to figure out how to control shadow IT—those unordained IT applications and systems used by employees without explicit approval or knowledge of IT departments. Shadow IT represented a major concern for the federal government and industry alike for the increased security and data exfiltration risks it presented. Like shadow IT, the element of dark data lingers over federal agencies but represents a far greater security risk.

Unlike shadow IT, dark data also represents an opportunity for significant value.

Illuminating the Dark Data Problem

Dark data is all the unknown and untapped data across an organization—varying from data from IT infrastructure to sensor data from industrial HVAC systems—which is collected, but not used. This could mean the data is simply ignored by operations staff and leadership alike, or is collected by operations staff but never reaches the leaders who could use it for organization-level insights. Either way, what characterizes dark data is that it exists, but the people who need it don’t know where it is or don’t have access to it and therefore aren’t using it. 

Splunk’s latest State of Dark Data report revealed that public sector IT leaders are struggling with this very issue. Data is a far-reaching element that touches all aspects of an organization. This presents an opportunity cost too high to ignore: Control over data is both a risk-mitigating activity and a value-adding one. The more data you have, the more you understand potential threats as well as how your organization operates.  

So why aren’t agencies doing everything to capitalize on their data?

The Data’s Dark, but Full of Opportunities

Dark data is the result of years of turbulent data strategy, IT management and governance. No one is to blame for dark data’s prevalence: It’s hard balancing numerous responsibilities and the rapid proliferation of digital technology in government. As mission requirements have changed over the years, this often leads to complex and siloed digital ecosystems that are not easily managed. 

The result is many agencies lack the comprehensive data strategy and data visibility they now need. Although the government is working to address the problem, a solution remains unclear as the Federal Data Strategy’s Year-1 Action Plan is still working to implement comprehensive data governance and ethics strategies. Coupled with the growing skills gap and IT departments bogged down in ongoing maintenance and compliance work, there is a tendency to take a laissez-faire attitude toward data that is not causing any problems today. After all, what you don’t know can’t hurt you, right?  

So Where Do I Start?

Agencies need to start by getting away from manually managing data. Tools powered by machine learning can automate these types of routine requests with greater speed and accuracy, allowing human capital to be freed for other tasks. This represents a large shift in agency mindset, yes, but a shift that must be made sooner or later; manual IT operations is simply unsustainable. 

With the shift from manual to automated data tasks out of the way, IT teams can spend more time thinking about the way new technology can change the federal government’s mission. They can focus on how dark data can be translated to mission value and on creating pockets of innovation that can form the backbone of wide-spread transformation. 

Leadership must seek out primary access to data so they do not rely on summary reports for decision-making. Again: You don’t know what insights you might find if you don’t have access to the data. 

Creating a Data-Literate Workforce

Federal leaders need to institute an innovative culture and inspire their employees to take a forward-thinking approach to data. Along with that change, there will likely need to be some reskilling as manual processes die and employees get moved into more strategic roles. This is one of the biggest opportunities federal agencies have faced in a while. By providing avenues for employees to reskill, retrain and develop new skills, federal agencies have a chance to reinvigorate a federal workforce that is aging and struggling with retention. 

A more data-literate, data-minded workforce that shares and cultivates ideas enables a more holistic conversation around new thoughts brought forth to management. It puts the power of data in everyone’s hands, enabling federal employees to derive valuable insights that were previously inaccessible. 

The Road Ahead

Like shadow IT, dark data poses a major challenge to government agencies. But dark data also offers an opportunity that’s not being seized right now. And while the journey towards a state of data leverage is long and involves cultural changes that won’t happen overnight, the dividends of that journey are a more efficient, inspired and vibrant government ready to tackle new challenges. So go forth, seek out new data and new ideas; go boldly into a data-driven future.

Juliana Vida is chief technical advisor of public sector for Splunk.