App Failure Led to Long Airport Lines In Third CBP System Crash In Three Years

A relatively brief database outage led to long lines for international travelers Friday, as the Customs and Border Protection agency worked to restart its systems in the wake of another crash—the third in three years.

At around 3:15 p.m. Friday, a key application used to process international arrivals at U.S. airports—the Relational Data Base Management System, or RDBMS—failed to connect with the associated CBP database, according to an initial investigation. The failure resulted in a widespread crash affecting airports nationwide, forcing customs officers to use other means for screening passengers.

“During the outage, CBP officers continued to process impacted travelers through alternate procedures until systems were back online, however it did result in longer processing times for travelers in some locations,” a CBP spokesperson told Nextgov. “During this time officers still had access to security-related databases and all travelers were still screened according to security standards.”

CBP IT officials had to restart several app servers in sequence over the course of two and a half hours, restoring full functionality by 5:45 p.m., according to CBP’s timeline.

“The root cause was determined to be a software bug following a system update that prevented database connections and caused services termination,” the spokesperson said.

“The outage did not have any nexus to a cybersecurity threat or incident,” they added, and the issue was “resolved without incident.”

CBP has had similar outages in the past, including issues on New Year’s Day 2018 and almost one year earlier on Jan. 2, 2017, as well. CBP officials declined to name the system involved in the 2018 incident, but the earlier event involved the Treasury Enforcement Communications System, or TECS, built in the 1980s. The TECS screening app checks passengers against watchlists and, when working, is used on every traveler crossing the U.S. border.

The agency’s inspector general warned persistent issues would continue after the first TECS outage.

“Until such deficiencies are addressed, CBP lacks a means to minimize the possibility and impact of similar system outages in the future,” according to a report published November 2017.

The TECS system is now a system of systems, a CBP spokesperson told Nextgov, officially replaced by the Relational Data Base Management System. “A portion” of that system was affected Friday. 

“A software bug in the Relational Data Base Management System impacted the optimal resource load balancing and symmetric multi-processing at large scale,” the spokesperson said. “Fixes were quickly implemented with further optimization of configurations for added resiliency.”

Editor's Note: This article was updated with comments from Customs and Border Protection.