Zients: HealthCare.gov much improved, but still in progress

Jeff Zients, the management specialist and former top government official tapped to oversee repairs to HealthCare.gov, announced Dec. 1 that the "tech surge" had met its goal of making the site function smoothly for the vast majority of users.

According to Zients, the system is now operating according to its original specifications. Software fixes and hardware upgrades allow the site to accommodate 50,000 simultaneous users, which would allow about 800,000 users to access the site for half-hour sessions during peak periods of use, Zients told reporters on a press call announcing the improvements. However, there were early indications that the retuned HealthCare.gov wasn’t able to deliver optimal service at peak user volume.

"HealthCare.gov on Dec. 1 is night and day from where it was on Oct. 1," when the site launched, Zients said.

The performance of HealthCare.gov is a matter of increasing urgency for the administration because applicants only have until Dec. 23 to complete enrollment to be guaranteed coverage beginning Jan. 1, 2014. Many customers on the individual insurance market are required to enter the new health insurance exchanges because their old policies are being canceled. If these people can't sign up by the deadline, they face not only the prospect of being without health insurance, but also the possibility of having to pay a penalty to the IRS for not being covered.

Since being tasked with leading a tech surge to repair the troubled website, Zients instituted a new management structure that established vendor Quality Software Systems Inc. (QSSI) as general contractor and integrator – a role formerly played by senior IT executives at the Centers for Medicare and Medicaid Services.

President Barack Obama had set Nov. 30 as a deadline for having the system up and running for the vast majority of users. At the same time, the administration downplayed the deadline. CMS was careful not to characterize the event as a relaunch of HealthCare.gov. "It's not a magic moment, but a process of continual improvement over time" said CMS spokesperson Julie Bataille.

Even so, there was no shortage of activity in the days leading up to the deadline, with more than 50 fixes being implemented on Nov. 30, when the site was down for scheduled maintenance. These last-minute changes mostly focused on the back end, which transmits enrollment information to insurance carriers. The sending of enrollment forms, called 834s, has bedeviled carriers because of faulty or garbled data.

The surge so far has included more than 400 software bug fixes, a re-architected registration database that can accommodate more users, and the addition of 12 dedicated servers to the core database that allows consumers to compare, select and purchase plans. The overall system firewall has been reconfigured to increase throughput by a factor of five. Additionally, there is 24-hour system monitoring in place to detect glitches and performance problems.

System uptime has improved, with the site available more than 90 percent of the time, compared to just over 40 percent uptime when HealthCare.gov launched Oct. 1. The user error rates are down to below one percent, meaning that a user is far less likely to encounter crashes, error messages or other failures when creating accounts, updating information or moving from page to page on the site, according to the progress report delivered by Zients.

Developers have also created a new virtual waiting room for times when demand for the site exceeds its capacity. Users will have the option of being informed via email about optimal times to return to the site, and will receive a special link that takes them to the front of the queue. CMS deployed the feature on Dec. 2, when an estimated 35,000 users were on the site. It's not exactly clear why the queuing system was used with fewer than 50,000 simultaneous users. According to Bataille, the decision to queue users was made in real-time based on reports of declining response times and rising error rates. Bataille also said that HealthCare.gov attracted about 375,000 users between midnight and noon on Dec. 2, about twice the volume of a typical weekday morning.

The overall project management approach has been reconfigured using agile development methodology. There are twice-daily standup meetings to prioritize and assign fixes, and a permanent open-voice connection from a command center staffed by managers and IT developers making fixes.

Zients is set to take up the post of chairman of the Council of Economic Advisors in January. He offered no insight into how his role as point person on HealthCare.gov would be filled once he assumes his new duties, saying only that he had his "head down" on the job in front of him.