NOAA Shifts Some Key Environmental Data Processing to the Cloud

The National Oceanic and Atmospheric Administration’s National Environmental Satellite, Data and Information Service, or NESDIS, recently made the first major move in an ambitious undertaking to migrate much of its ground systems to cloud computing. 

In March, a portion of the Joint Polar Satellite System’s operational data processing ground system, which captures crucial data about Earth, made the shift. 

JPSS encompasses America’s most advanced series of polar-orbiting environmental satellites, according to NOAA, and this transition notably lessened the program’s hardware footprint. 

“Being first in the organization of NOAA to make such a major change of doing business—that’s not trivial,” NESDIS Chief Information Officer Irene Parker told Nextgov in a recent interview. “I think the team—both the government and contractor teams—worked really well together to rapidly learn the new technologies, especially with this pandemic, to be able to meet our timelines.”

Parker has been at NOAA since 2008, where she now focuses on an array of areas including cybersecurity innovation technologies and radio frequency and data management. She’s also spent time in industry, helped optimize U.S. Mint’s information technology manufacturing processes within the Treasury Department, and comes from a math-heavy educational background. In the chat, Parker shed light on this contemporary federal journey into the cloud and implications around the agency’s workforce and culture that emerged along the way.

The Zeros and Ones Can’t Change

Satellites in the JPSS constellation are ultimately associated with global measurements for atmospheric, terrestrial and oceanic conditions. The data they collect is instrumental to informing accurate forecasts ahead of severe weather events and assessing environmental hazards. 

“It looks at things like sea and land surface temperature, vegetation, clouds—you name it—rainfall, snow and ice cover,” Parker explained. “I mean, it does a wide variety of things.”

The program is a part of a collaboration between NOAA and NASA. This JPSS ground system essentially takes in data from the onboard sensors that are up in the air and produces first order scientific products, which NOAA then calibrates and distributes to downstream users across the government and private sectors. Those include the Defense Department, National Weather Service and Federal Aviation Administration—and these products also form the backbone of NOAA’s short- and mid-range weather forecast models, Parker noted. About 400 gigabytes of data flow into the satellites daily, 7 terabytes of which go out to customers. 

These JPSS data-driven services, according to the CIO, provide key observations for the nation, particularly for (but not limited to) predicting hurricanes, tornadoes and blizzards days in advance, and evaluating risks around droughts, forest fires, poor air quality and harmful coastal waters. 

“So, the integrity and the availability is very key for all of our missions, right? We can't have the data—the zeros and ones—change from when it comes from the sensors, in the sky down to the ground, and it can't change when it's processing,” Parker said. “And the system on the ground that's processing and creating all these important data products that I just went through—it has to stay available.”

This move into the cloud traces in some parts back to 2017, she explained, when NOAA’s existing prime contractor Raytheon presented a proposal to study moving the interface data processing segment, or IDPS, to the cloud—instead of refreshing its on-premise hardware as planned in 2019. After digging in, officials proposed the NESDIS migration to leadership in late 2018. They then set the goal to make the processing capability operational in the cloud by late 2020 or early 2021. Those embarking on the work did so speedily to make the shift, just as some of the involved existing hardware was nearing its end.

“And here we are,” Parker said.

Previously, raw data would be captured by satellite instruments and received through a system of antennas that was then consolidated at the NOAA Satellite Operations Facility in Maryland. There it used to be processed into those data products—but now, that information will be processed in the Amazon Web Services GovCloud. Parker said while there are presently multiple cloud service providers, when NESDIS started planning this in 2017, AWS “was the only one that was far ahead” particularly in meeting NOAA’s security requirements. At that time, she said it was also the only vendor that was federally certified at a high level for cloud cybersecurity.

“So, that’s why we went to Amazon,” she noted. 

This marks the service’s first operational system to transition into a FedRAMP-authorized cloud, but it’s just the beginning.

“How this fits into NOAA’s broader vision, basically, is that we would like to deliver directly to our customers in the cloud as a part of our modernization phase. Right now, although our processing takes place in the cloud, we still distribute via on-premise solutions,” Parker said. “So, that's our next step—really to move the distribution and sharing of our data into the cloud. We're working with the NESDIS cloud initiative to share lessons learned in how we can standardize tools and processes across the enterprise.”

This is Different

This was an early step in a complex process, but the agency is already encountering what could be considered perks of a switch to the cloud. For one, the new way of doing things reduces hardware—“racks and racks of computers,” as NOAA calls it in the release—in more than one location. 

“Previously, our ground system took up 53 racks at our Suitland, Maryland location, which is our NOAA satellite operations facility,” Parker explained, and “it took up 27 racks at our consolidated backup site in Fairmont, West Virginia.” Just looking at width, she said, a rack size is roughly over 20 feet. But in moving that IDPS, data processing segment specifically for product generation to the cloud, NESDIS can remove 19 racks from that Maryland location, and seven from the West Virginia-based spot.

“The key thing is the movement of the cloud also removed hardware racks at our vendor facilities. So it's not just at our operational locations, but we were able to decrease the footprint in both Maryland and in Colorado, where our vendors are located,” Parker said. “Overall, we almost reduced 40% of rackspace, and energy consumption at our NOAA and vendor facilities by going to the cloud.” 

Multiple steps were also taken in the migration to ensure that the agency’s cloud-based security posture remained consistent with the on-premise system, which must maintain high-security levels in integrity and availability, based on FISMA.

“We had to ensure that when we were going on to the cloud,” she noted. “So, what we did was we integrated our auditing and event correlation capabilities and we established a security enclave within the cloud environment to make sure we can manage accounts, access, inventory and vulnerability scanning.”

Over the next year, officials aim to host a series of optimization activities and also plan to take advantage of cloud-native security tools, to further reduce costs and simplify security monitoring operations, according to Parker. 

Looking back on the complex process, the CIO reiterated that a primary hurdle was ensuring those requirements were met even in the cloud and that it all remained seamless to NOAA’s end-users. “We basically took a three-phased approach, which was we wanted to operationalize it, we wanted to then optimize it—and then we needed to modernize it all without disrupting our user community because this is an operational capability,” Parker said.

Cultural challenges also surfaced, she noted, as personnel had to completely adjust to this new way of doing business.

“In the prior world, you could physically drive to Suitland, Maryland, or West Virginia and touch your racks. This is different. You have to trust the cloud vendor, right, that they're providing you this infrastructure for us to run our processing capability on,” Parker said. “And so that's a big mindset change for us—but we were able to do that.”