Cloud failure temporarily crashes HealthCare.gov

HealthCare.gov is back online after a network failure at Verizon's Terremark cloud service took the site and its main data hub down Oct. 27.

"Verizon Terremark successfully resolved the issue with the networking component overnight, and as of 7 a.m. ET this morning the Data Services Hub was fully operational," HHS spokeswoman Joanne Peters said in a statement. "The HealthCare.gov technical team continued troubleshooting one issue with the online account creation process in the application and has now opened the online application and enrollment tools back up to consumers."

The outage was the latest in a series of technical glitches for the site, including long waits for service for users and problems in delivering information to insurance carriers on the back end.

However, this was the first problem caused by a cloud service provider.

The failure originated in a Verizon Terremark data center and likely affected other companies that purchase computing power from the tech giant, though Verizon has thus far not responded to media inquiries on the cause of the problem.

It is also unclear how many potential customers attempted to reach the site while it was down only to receive a message that said: "We are experiencing technical difficulties and hope to have them resolved soon. Please try again later."

The outage affected both the Federally Facilitated Exchange which serves insurance customers in 36 states, and the individual exchanges in 14 states and the District of Columbia. The state-based health insurance exchanges as well as the federal system, are dependent on the data hub to operate properly.

HeathCare.gov is the front door to a system for registering for insurance and shopping for policies under the 2010 health care overhaul. Since launching on Oct. 1, it has been plagued by errors and long waits for service on the user interface side, and problems in delivering information to insurance carriers on the back end. This is the first reported outage related to a failure at a cloud provider.

Separately, the HealthCare.gov website was redesigned over the weekend to remove a picture of a smiling woman, which was replaced with navigation icons that show users how to apply by phone, in person and by mail, in addition to online.

The administration is encouraging applicants to use these alternative enrollment methods while contractors focus on repairing software bugs and capacity issues. This effort is being led by Jeff Zients, a health care entrepreneur and former acting director of the Office of Management and Budget. The administration is promising that online enrollment will work as advertised for the vast majority of users by the end of November. 

CMS did announce that it had knocked a few items off the "punch list" being managed by Zients. The load times for comparing insurance plans have dropped considerably, from minutes to seconds. Also, eligibility notices which had been garbled now appear as designed to shoppers at the end of the application process.

Healthcare.gov is still delivering faulty enrollment information to insurance carriers. A fix to the delivery of enrollment forms, called 834s in the industry, remains a high priority for the team working on Healthcare.gov, but there's no work on when it will be completed. The issue "is on the top of the punch list," CMS spokeswoman Julie Bataille said, adding that CMS is having ongoing conversations with insurance issuers, "to identify , diagnose, and prioritize issues as we work through our punch list of fixes."

Bataille also acknowledged that CMS Administrator Marilyn Tavenner was in charge of implementing the IT pieces and other aspects of the 2010 health care overhaul. Tavenner is scheduled to testify before a hearing of the House Ways and Means Committee on Oct. 29 about the buggy launch of HealthCare.gov. However, CMS is still declining to name the technology specialists inside government and from the private sector who were tapped to lead a "tech surge" to identify problems with the software and the performance of Healthcare.gov. The group is thought to include several members of the Presidential Innovation Fellows team, as well as Federal CTO Todd Park.

It's not known which private sector firms outside of the contractors working on the project are pitching in, or whether they are getting paid. Rep. Darrell Issa (R-Calif), chairman of the House Oversight and Government Reform Committee, is threatening to issue subpoenas to find out the identities of the team behind the tech surge, and has also sent letters to leading technology firms inquiring whether they have detailed any staff to work on Healthcare.gov.

The highly public failure of Healthcare.gov is attracting a lot of attention in the federal IT community.

The American Council for Technology and the Industry Advisory Council (ACT-IAC) is holding its Executive Leadership Conference in Williamsburg, Va. While government CIOs and other IT executives aren't commenting for the record on HealthCare.gov, it is clear that there is an appetite to change some aspects of federal IT contracting and development.

"We do not have an 18-month or a two-year cycle to produce software based on requirements that we gather and develop in silos and that we pull together toward the end," said Lena Trudeau, associate commissioner of GSA's Office of Strategic Innovations in an Oct. 27 speech. While she was not directly addressing HealthCare.gov, she spoke in favor of close collaboration between agencies and IT contractors, and regular peer review of software code.