Watchdog: IT glitch at NASA led to fire

A security patch that shut down monitoring equipment in a large NASA engineering oven resulted in a fire that destroyed spacecraft hardware inside it. Since the computer reboot to accommodate the software upgrade also crippled fire alarm activation, the fire in the oven wasn't discovered for three and a half hours.

This is just one example of how a lack of coordination between IT and industrial control systems can wreak havoc, according to a Feb. 8 report from the space agency's inspector general.

NASA has been automating many of its isolated, manually controlled technologies in favor of more sophisticated and interconnected IT  equipment. But the agency's approach to integrating cyber, IT and physical systems is still a work in progress, and gaps in standards, training and security best practices need to be remediated, according to the OIG.

In other words, NASA's approach to the security of interconnected control systems is "reflecting society at large," according to the report.

The details in the report show that NASA is suffering some of the same coordination problems and unintended side effects that have cropped up at companies and agencies mixing their manual operational technology  for infrastructure systems with IT systems.

According to the OIG, 65 percent of the agency's critical infrastructure -- including environmental monitoring and control systems that control heating, cooling, ventilation, power, rocket propulsion testing systems, spacecraft and aircraft command and control systems  -- are managed and supported by OT, or hybrid OT/IT systems.

Knowing which IT systems incorporate OT components is especially critical for NASA, it said, because applying traditional IT security practices to OT systems can cause underlying systems to malfunction.

NASA maintains a cybersecurity database to track traditional enterprise IT systems, but only a fraction of OT assets are identified in it. Of 397 systems listed in the database, 32 reported having OT components, the OIG said. 

According to the report, even though OT is critical to its operations and has a "significant presence across the agency," NASA hasn't defined OT adequately, developed a centralized inventory of OT systems or established a standard protocol to protect systems that contain OT components.

In addition to the oven fire, the IG described two other unfortunate incidents at the agency involving interconnected OT systems and the unintended consequences of those interconnections.

Vulnerability scanning used to identify software flaws caused physical  equipment to fail. That failure resulted in a communications blackout with an Earth science spacecraft during an orbital pass. As a result, the pass was rendered unusable and data could not be collected until the next pass.

Another incident involved a computer problem in a climate system in which a safety feature was disabled, causing the heat in a data center to jump 50 degrees in minutes, resulting in a shutdown of the center.

The report's list of NASA's OT shortcomings is probably familiar to IT managers at large organizations working to get a handle on the increasing number of internet-connected, or interconnected, devices and systems that were previously manually operated.

The list includes a lack of comprehensive best practices for OT, limited awareness of OT across the agency, policies that don't distinguish OT from IT and no training focused on protecting OT systems.

"As a result, NASA is not well-positioned to meet the security demands of an evolving OT environment and is assuming unnecessary risk for critical agency systems and facilities with OT components," it said.

Additionally, according to the OIG, the agency hasn't coordinated its physical security with the cybersecurity of its critical infrastructure with either agency stakeholders or with its Office of Strategic Infrastructure.

That "disjointed approach," it said, has led to duplicated effort and gaps in security plans.

The agency, according to the report, is taking steps to solve its issues, however. 

When the OIG performed its audit of the systems, it noted that NASA was working with the Department of Homeland Security's Continuous Diagnostics and Mitigation program contractor Booz Allen Hamilton to integrate information security tools for initial deployment.

Ultimately, said the report, CDM's success will depend increased collaboration among NASA mission directorates, the Office of the CIO, Office of Protective Service and Office of Strategic Infrastructure to accurately identify critical assets and improve the security.

NASA mostly agreed with the IG recommendations, but said it has its own plans to put them into practice.

In reply comments, NASA outlined plans to establish an Industrial Control System Working Group led by NASA's Enterprise Protection Program, with representation from relevant agency components, to inventory area of technological interdependency and to come up with a set of practices to guard against problems like those described in the IG report.

Additionally, NASA will define and segment operational technology and industrial control systems from agency IT.

NASA expects improvements to be in place by Oct. 1, 2018.