FAA finds Year 2000 tools not up to snuff

The Federal Aviation Administration faced with a time and date software problem that threatened this month to bring down one of its air traffic management systems was forced to manually sift through more than a million lines of code after software tools designed to find code containing times and dates failed to find everything that needed to be fixed.

Earlier this month the FAA had used Year 2000 software packages to try to find the lines of code containing times and dates in its Enhanced Traffic Management System (ETMS) which displays the locations of planes on a national and local scale and alerts air traffic controllers when traffic exceeds a specified limit. The FAA had to upgrade the ETMS software because for technical reasons the operating system was unable to process times and dates after 14:49 Greenwich Mean Time on Nov. 2.

The FAA's experience illustrates the daunting task federal agencies have in fixing computer systems so that they can properly process dates containing the Year 2000 and it shows how software packages designed to identify lines of code containing times and dates may not be a panacea for these problems.

The lack of a tool that provides a 100 percent solution "is a fairly serious issue" for agencies said George Haynes research director at IDC Government Falls Church Va. "It's generally a problem with tools out there. No tool is going to find it all. No one has the perfect tool yet."

Hewlett-Packard Co. the ETMS prime vendor was the first to notify the FAA that the system had a date-dependent problem in HP's Apollo Domain operating system that would render the system inoperable Nov. 2. HP delivered a software upgrade in April that fixed the system's clock and timing services. But when the FAA ran tests using the patch the system crashed. "If we tried to bring down the machines and then reboot them ETMS didn't work " said Bob Fietkiewicz test director at the FAA's Hughes Technical Center Atlantic City N.J.

The FAA discovered that the problem was with the software code that the agency had written during the past several years. "When we ran [commercial] Year 2000 code-checking software we discovered that these [references] sometimes didn't show up " said Bob Voss the integrated product team leader for air traffic management in air traffic systems development at the FAA. "Just running date- and time-checking software isn't enough you have to do hand analysis of the code."

As a result the FAA's Volpe Center Cambridge Mass. which developed the ETMS application software manually examined the system's 1.5 million lines of code and found 150 000 lines of code in ETMS that needed to be rewritten. The software baseline was then rebuilt tested and deployed. On Nov. 2 the FAA shut down and successfully restarted 450 ETMS workstations at more than 80 sites for a five-hour period.

"It all went like clockwork " said Delois Smith product leader for traffic flow management infrastructure at the FAA. "We had people at all the sites and coordinated it out of Boston. To pull it off we had over 100 people working on it."

While the ETMS fix was only a small subset of what will be required for the Year 2000 conversion effort already under way there were similarities.

They included inspecting the software code for time- and date-related services as well as conducting assessments of network protocols and connections to ensure that communications among multiple sites would process the correct time and date. Also both problems required upgrades of noncompliant software use of code-testing tools and the assignment of staff familiar with the older software.

"Tools are only as good as the people who define the date fields " said Kathleen Adams chairwoman of the Year 2000 Interagency Committee. "Everyone who maintains the software is responsible for software they maintain." Even Year 2000-compliant client/server software still needs to be tested to see if it will work with the mainframe-based software that has been fixed she said.

"Testing is going to be huge " Adams said. "You have to run enough test scenarios that can ensure all systems work together right. I can't talk enough about how critical testing will be especially downstream."