Watchdog criticizes Homeland Security's IT management of border initiative

The Homeland Security Department and a contractor rewrote most of the quality testing procedures for a project to deploy sensors and cameras along the U.S. southwest border, in part to make them more likely to pass inspection, an official with the Government Accountability Office testified on Thursday.

According to GAO's report, DHS has not effectively managed key aspects of SBInet testing. SBInet is the technology component of the department's Secure Border Initiative, which includes fencing that eventually will run along the U.S. southern and northwestern borders and a network of sensors, cameras and control towers to identify illegal crossings into the United States. Boeing Co. is the project's prime contractor.

The inadequate testing in turn has increased the risk that the system will not perform as expected and will take longer to deploy and cost more, said Randolph Hite, GAO's director of IT architecture and systems issues, said during a joint hearing before the House Homeland Security Subcommittees on Border, Maritime and Global Counterterrorism; and Management, Investigations and Oversight.

"Testing is not a one-time event; it's a series of incremental events that build upon one another," Hite said. "Each can be viewed as a link in a chain, and each must be well-planned."

The department's approach includes a series of progressively expansive test events designed to ensure that the program's systems are effective through user assessments, operational tests and evaluations. Some of these tests have not been completed yet, according to GAO, and a number of plans for recently executed test events were not well-defined. For example, none of the plans addressed testing risks and mitigation strategies.

Most alarming to lawmakers, however, was that 70 percent of SBInet test procedures were not carried out as initially written, but instead were changed during execution "because persons conducting the tests determined that the approved procedures were not sufficient or accurate," GAO stated. Managers did not revise procedures according to a documented quality assurance process, but instead based modifications on an undocumented understanding that they established with Boeing. The company so far has collected about $615 million under the contract, and upwards of $200 million has been spent on internal program expenses.

"Even though test procedures were largely defined, the procedures for 70 percent [of cases] were changed on the fly during execution, not subjected to written quality assurance checks and in some cases extensive," Hite said, adding some procedures simply were crossed out and replaced with handwritten steps.

He also noted a program office letter to Boeing stating changes made to system qualification test procedures appeared to be designed to pass the test rather than to qualify the system.

Roger Krone, president of Boeing's network and space systems division, stated in testimony the changes indicated "we didn't have the test procedures right to begin with," and the initial requirements were not well-defined. He also noted the modifications were made following an "informal agreement between the program office and Boeing," and the unwritten set of procedures was validated against standard quality assurance measures at the time.

"It doesn't pass the smell test; I don't care who signs off on it," said Homeland Security Management, Investigations and Oversight Subcommittee Chairman Chris Carney, D-Pa.

Even with changes to the testing procedures, GAO identified about 1,300 defects in SBInet systems between March 2008 and July 2009, with the number of new flaws on average increasing faster than the number being fixed -- a trend that Hite said, "is not indicative of a system that is maturing."

The defects included a circuit breaker that frequently tripped when a radar dish rotated beyond its intended limits, computer workstations that frequently crashed, towers that swayed beyond tolerable limits, false detections by radars, and blurry camera images. These defects delayed scheduled installation of sensors by about a month.

"It's a given that testing will result in the discovery of defects that will need to be resolved," Hite said. "But the bottom line message is that key aspects of most of the recent test events have not been effectively managed," which contributed to the issues identified.

"I can't promise [all issues will be addressed] overnight, but I can promise we're committed," said Mark Borkowski, executive director of the Secure Border Initiative. "It is important we do this right and at the end of the day, that we've tested [all the systems]. Ultimately, I am accountable."

DHS expects initial testing of systems in the Tucson, Ariz., area of SBInet to be completed and turned over to Border Patrol for operation testing by September, while the systems in the Aho, Ariz., region will be turned over in November. Homeland Security Secretary Janet Napolitano's Tuesday announcement that $50 million for the project would be reallocated to proven technologies means certain procedures for the initial deployment will be adjusted. Napolitano also plans to freeze SBInet funding beyond the initial deployment until an assessment that she ordered in January is completed.