Hardware still hindering State visa process

After a June 9 hardware failure ground the State Department’s process for issuing visas overseas to a near halt, diplomatic posts handling two-thirds of the department’s visa capacity are back online and issuing visas, a spokesman said June 23.

“It’s going to take some time here for the backlog to clear as we continue to work the fix,” State spokesman John Kirby told reporters.

“It’s a hardware problem with … the physical database itself,” he added. “So we’re not talking about a software glitch, and that’s why we’re sure that this isn’t the result of some kind of cybersecurity breach.”

The basic problem was a halt in the flow of biometric data submitted from overseas posts to the Consular Consolidated Database, a system of about a dozen databases that handles visa and passport requests. When the hardware failure hit the biometrics database, technicians turned to a backup database, but the data there was corrupted and unusable, Bureau of Consular Affairs spokesman Niles Cole told FCW. And so a team of technicians had to rebuild and test that database before getting it back online to process visas.

“It took a while to recreate the backup system and then to test it,” Kirby said. Technicians “were testing it methodically using a single post to see if they could insert the new hardware and that it would work properly.”

The department issued more than 45,000 visas on June 22, Kirby said, adding that the department normally processes 50,000 visas a day. He did not have an estimate for how big the current backlog of visas is.

Kirby said last week that a team of more than 100 public and private computer experts were working to restore the CCD to full functionality. He said June 23 that that work continues around the clock. Oracle and the U.S. Digital Service, a federal IT team that grew out of the effort to fix HealthCare.Gov, are part of that group, Cole said. Oracle spokeswoman Katie Barron declined to comment on the company’s work on the recovery project.

The CCD has had software problems, too. It was forced offline for three days last July as Consular Affairs upgraded the system’s enterprise management platform. The outage caused a backlog in visa issuances that took about two weeks to clear. Cole has emphasized that the CCD’s latest glitch was unrelated to last year’s problems, saying, “The challenges we faced last year have been addressed.”