IG: NOAA, NASA Launched Next-Gen Satellite With Known Issues, Scrubbed Performance Metrics from Contract

Persistent problems with the premier sensors of the GOES-R series satellites—designed to provide the next generation of weather observation for North America—were identified before launch and not properly tested or resolved, according to a new inspector general report.

Further, the Commerce Department IG found evidence that program managers changed the evaluation criteria for the contractor after the issues were identified—metrics that would have led to a 40-75% reduction in payment had they remained.

The $11 billion GOES-R series of satellites includes GOES-16—launched November 2016—and GOES-17—launched March 2018—as well as the pending GOES-T and GOES-U still in production. The satellite constellation is equipped with a set of next-generation sensors to better predict weather patterns, including the Advanced Baseline Imager, or ABI, the “most essential instrument for mission success of the GOES-R satellites,” according to the IG.

However, shortly after GOES-17 entered orbit, the cooling system for the ABI instrument malfunctioned, “severely degrading” the amount of data the satellite could collect, NOAA officials said at the time.

“This is a serious problem,” Steve Volz, administrator of the National Oceanic and Atmospheric Administration’s National Environmental Satellite, Data and Information Service, or NESDIS, said during a May 2018 briefing with reporters. “This is the premier Earth-pointing instrument on the GOES platform and the 16 channels … are important elements of our observing requirements.”

Engineers with NOAA and NASA worked to get the instrument largely operational, though IG auditors found the program office should have seen these issues coming.

Testing on the ABI in March 2017 prior to launch showed issues with the cooling system. Engineers devised a workaround but only tested the changes at “ambient temperatures and pressures,” according to the IG, which did not simulate the conditions in which the satellite would be operating in space.

With the modifications tested at ground level, the risk management team labeled the issue as fixed and “did not categorize the thermal anomaly as a failure,” the IG wrote.

Using NASA’s “Test as You Fly—Fly as You Test” protocol, the program “accepted the risk of not re-performing the thermal vacuum testing of the satellite and proceeded to launch. We found this level of verification for ABI to have been insufficient given its importance to the [key performance parameter] capability for the mission,” the report states. “Re-testing the modified instrument and satellite in thermal vacuum could have provided a higher confidence level in the verification of the modification or potentially additional actions to preclude failure on orbit.”

The ABI cooling system subsequently failed in April 2018, shortly after the satellite launched in March.

Engineers have since been able to get the GOES-17 satellite to deliver 97% of the data it was designed to, though it will never be able to operate at the level intended. The 16 ABI channels will be available during the two “cold seasons”—May through June and November through January—but only seven will be available 24-hours during the remaining months. The other nine channels will be out of commission for two to six hours each day during that time, as direct sunlight will make the system too hot to function.

As these issues first came to light in 2017, an IG investigation found the program office altered the contract deliverables in favor of the vendor, Florida-based Harris, a tech firm with numerous government contracts. 

Evaluation metrics included in the original contract with Harris would have “classified instrument performance as either ‘degraded,’ meriting a reduction of up to 40% of award fees or ‘severely degraded,’ which merits up to 75% reduction,” the IG wrote.

Instead, the contract language was alerted “to align with the expected performance inputs from the funding agency,” i.e., NOAA, according to a NASA memo. The program office told the IG these changes were meant to streamline the evaluation process, as other metrics covered “the instruments’ radiometric performance.”

“However, our examination of the changes made to the ABI contract … found that criteria related to the thermal system and infrared calibration target were the only ones removed from the degraded and severely degraded classifications, soon after the program became aware of thermal and calibration issues with the ABIs,” the IG wrote. “We found that these issues would have likely reduced the contractor’s award fees for on-orbit performance without this revision.”

Auditors said they were not able to “fully explore this matter,” and left it to senior management to investigate further.

"On the issue of contractor award fees, before the GOES-17 launch, NOAA and NASA initiated changes to the contract evaluation terms to align the on-orbit performance criteria with the utility of the data from a user perspective. These changes were made for all instruments, not just ABI. These changes will allow the government to more effectively assess the earned fee for on-orbit performance based on user needs," a NOAA spokesperson told Nextgov. "It’s important to note the on-orbit evaluation for the GOES-17 ABI has not yet occurred and will take into account the impact of the LHP issue and the overall performance of the ABI."

And while the GOES-16 and -17 satellites are technically operational at this point, the systems are not working as intended and will “affect the reliability of the instruments, satellites and, ultimately, the constellation availability,” the IG wrote.

Those technical and contracting issues will need to be resolved as NOAA and NASA continue work on the second half of the GOES-R program—GOES-T and GOES-U, which will be given number designations once in orbit.

While engineers have been focused on the main sensor—the ABI—a secondary instrument, the magnetometer, which monitors the Earth’s magnetic field, is also not meeting expectations. The IG audit found the GOES-R magnetometers are less accurate than technology deployed on older satellites.

“GOES-16’s magnetometer has been less accurate than magnetometers on the previous generation satellites—GOES-13, -14, -15—and GOES-17,” auditors found. “GOES-17’s magnetometer has also not met its accuracy requirements,” they added.

As these tools fail to meet expected capabilities, the Space Weather Prediction Center told the IG it “continues to rely on the previous generation of satellites for these measurements.”

The audit also found issues in how the program planned to position the satellites within the greater constellation—though the team was able to adapt with minimal issues. The investigation also discovered the lifecycle costs will likely be higher than the original $11.7 billion projected, as ground system infrastructure—servers, software and security—will have to be updated over the next three years.

Editor's Note: This article was updated with a statement from NOAA.