For NOAA, flexible cloud enables mammoth computing

Sometimes less is more. Sometimes more is more.

Sometimes you just don’t know which will be which ahead of time – and flexibility is king.

Producing 20 terabytes of data every day, the National Oceanic and Atmospheric Administration is a prime example of the need for flexibility. And through its Big Data Project, NOAA’s bringing private-sector ingenuity to bear on the age-old problem: How can government get its work into the hands of the people who want it?

Making data public is embedded in NOAA’s history, but that doesn’t mean the organization is fully equipped to disseminate that data on its own, Alan Steremberg, NOAA’s big data-focused Presidential Innovation Fellow, said at AFCEA Bethesda’s June 30 Data Symposium.

Steremberg would know.

He helped start Weather Underground – the website, not the terrorist group (although both formed at the University of Michigan) -- in the 1990s and later went to work for the Weather Channel after it bought Weather Underground, giving him a firsthand look at two commercial ventures with business models reliant on packaging NOAA’s information.

Between real-time data and archived data, “these things are just really big and hard to move around,” Steremberg said. NOAA’s challenge became, “can we put compute alongside this data?”

In April, NOAA inked deals with Amazon Web Services, Google, IBM, Microsoft and the Open Cloud Consortium to do just that.

There has been “tremendous” industry uptake on NOAA’s data, Max Peterson, general manager at AWS Worldwide Public Sector, told FCW.

“Data’s interesting, there’s tremendous value in data, but the second half of that is having a partner ecosystem that turns around and makes that data relevant to something that somebody cares about,” said Peterson.

The AWS partner ecosystem teams NOAA’s data with companies that can use it – say, ESRI for geospatial mapping – and Peterson said AWS’s cloud infrastructure will deliver ever-more compute power to wrangle the data, without breaking the bank.

AWS’s Elastic File System, previewing this summer, enables users to pay only for the storage they use, Peterson touted. “It’s elastic, it’s on-demand, it’s scalable.”

AWS Lambda enables users to pay only when events come through and code runs, on the order of 20 cents per million events.

It has big implications for Internet of Things deployment such as NOAA’s, with myriad connected devices reporting constantly, and the overall flexibility of cloud means that vast data sets could be analyzed in their entirety, rather than broken off in sample set chunks.

“All of a sudden on AWS they can spin up 1,000 cores, or 5,000 cores, or however much compute they want to acquire to run their job,” Peterson said. “You can now run 100 percent of your data through whatever your analytic program is.”

From Steremberg’s perspective, private partnerships help ensure a “customer-first approach” at NOAA, with a focus on getting NOAA data on the “things that really drive our economy,” from weather to ocean currents for shipping data, packaged right.

“We’ll hopefully learn that there’s a model where everyone can make lots of money,” Steremberg added, saying startups and established companies alike could benefit from the cloud revolution.

In a landscape punctuated by costly mistakes and bumbling launches, NOAA Data Asset Portfolio Analyst David McClure offered some of the highest praise a federal technology program can get.

“What I’ve been struck with [is] things have been unfolding the way we expected they would,” McClure said of the Big Data Project at AWS’s annual Public Sector Symposium last week. “In a way, that’s surprising, that it’s working the way it’s supposed to work.”

GCN staff writer Amanda Ziadeh contributed to this report.