NOAA Needs to Know If Its Big Data Cloud Partnerships Are Working

For the last three years, the National Oceanic and Atmospheric Administration has been giving its data to five of the largest cloud service providers in an effort to increase public access, lower costs for the agency and potentially create new markets. Now, as the initial run nears its end, NOAA officials want to know if it’s working.

NOAA established the Big Data Program in April 2015 with five major cloud providers signing on to a three-year cooperative research and development agreement: Amazon Web Services, Google Cloud Platform, IBM, Microsoft Azure and the Open Commons Consortium.

The goal was to unleash NOAA’s wealth of data in order to facilitate research and, possibly, create new markets in the same vein as making weather data publicly available. Under the Big Data Program, NOAA developed a pipeline to securely move troves of data to cloud environments owned and operated by the five partners. Those partners, in turn, could not charge for the data—which must be freely, publicly accessible—but could create value-added content around it at a premium.

When working as intended, the program would make it easier for the public to access specific data and offer additional tools to make it more digestible or useful in certain situations.

The program has published more than 40 data sets so far, including environmental data and satellite imagery.

The contracts have been extended through May 2019 as NOAA looks to modify the program based on lessons already learned and feedback from the partners and users. The agency issued a request for information Monday to garner input that will inform the next iteration of the program.

“In order to inform the future direction of the BDP, this RFI seeks direct input and feedback from all users of the data on their experience accessing NOAA’s open data through one or more of the five BDP Collaborators’ cloud platforms,” the document states.

Specifically, the CIO’s Office wants to know whether people are using the data, to what effect and whether the current distribution systems are working for them. The office is also interested in whether access to this data is creating new business processes, services or markets.

The program seems to be going well, according to a NOAA fact sheet listing some of the lessons learned. However, there is room for improvement, including building a framework that will be more sustainable over time.

“While the BDP activities demonstrate there is little doubt the cloud offers greater capacities and technical capabilities than NOAA is able to provide on-premises, the most effective business model to sustain a public-private partnership that would facilitate the BDP's goals long-term is still to be determined,” the fact sheet states.

So far, NOAA officials have been able to determine:

• There is “a demonstrable unmet demand” for the data.
• The cloud technology and transmission methods work.
• Cost issues can be worked out.
• NOAA must keep a central role in the process to ensure public trust in data integrity.
• Integrating third-party cloud-based tools leads to more engagement than just posting raw data sets online.

Lastly, officials have figured out that a loose agreement between parties won’t be enough to ensure success over the long term.

Responses are due by 5 p.m. on Oct. 15. Participants are encouraged to answer any or all of the questions, based on their level of interest and experience with the program.