State's passport and visa system crashes

The colossal data warehouse that supports the State Department's worldwide visa and passport verification operations has crashed, potentially stranding thousands of people waiting for the documents around the world.

The Consular Consolidated Database (CCD) at the State Bureau of Consular Affairs "is currently experiencing technical problems with our passport/visa system," said Marie Harf, deputy spokesperson for the State Department in a July 24 statement to FCW. The Associated Press first reported the CCD problems on July 23.

Harf said the CCD, the agency's system of record, has had "significant performance issues, including outages, since Saturday, July 19." The issues, she said, have resulted in significant backlogs. Visas are approved, recorded and printed through the CCD. "Until the system comes back online, we are unable to print visas. We apologize to applicants and recognize this may cause hardship to applicants waiting on visas and passports. We are working to correct the issue as quickly as possible," she said.

Ongoing problems with the CCD have been no secret, an industry source familiar with the system told FCW. The database crashed earlier this year -- without the visible disruption of the July 19 incident, but seriously enough that Bureau of Consular Affairs CIO Greg Ambrose sent an email to State Department and industry stakeholders about the problem.

The CCD system is massive, containing millions of visa cases and photographs, with links to other federal agency security databases, including the FBI's Integrated Automated Fingerprint Identification System (IAFIS) and the Department of Homeland Security's Automated Biometric Identification System (IDENT). It is also the gateway to the Department of State Facial Recognition system and the NameCheck system.

According to a State Department privacy impact assessment completed in 2010, the CCD is one of the largest Oracle-based data warehouses in the world. As of December 2009, according to the PIA, the database contained more than 100 million visa cases and 75 million photographs, used billions of rows of data, and had a growth rate of about 35,000 visa cases every day.

The PIA said the database was created to provide Consular Affairs a near real-time aggregate of the consular transaction activity collected domestically and at consular post databases worldwide. "The CCD is the IT implementation that provides for a set of centralized visa and American citizen services supporting consular posts and back office functions," the assessment found.

Three of CCD's primary functions include supporting data delivery to approved applications via industry-standard Web Service queries, providing users with easy-to-use data entry interfaces to CCD, and allowing emergency recovery of consular post databases. Additionally, according to the PIA, authorized State Department and other government agency personnel can use the CCD Portal to view centralized data and access other applications.

Harf said the issue is worldwide and not specific to any particular country, citizenship document or visa category. "We apologize to applicants who are experiencing delays or are unable to obtain a passport, Consular Report of Birth Abroad, or visa at this time. We are working urgently to correct the problem and expect our system to be fully operational again soon."