VA, DOD Electronic Health Record System Suffers Nationwide Outage

The electronic health record system used to manage patient data for the Defense Department, Coast Guard and a few Veterans Affairs Department medical centers went offline nationwide for almost three hours Wednesday, preventing clinicians from updating and, for a time, accessing medical records.

The EHR systems being deployed by VA—dubbed Millennium—and the DOD and Coast Guard—known as MHS Genesis—are both developed and maintained by Cerner, the latter in conjunction with the Leidos Partnership for Defense Health. The DOD started work on MHS Genesis in 2015, with the Coast Guard joining in 2018. VA joined the combined Electronic Health Record Modernization Program, or EHRM, in 2018, with the first instance going live in October 2020.

Three Oracle databases underpinning those systems went down shortly after 5 p.m. ET, preventing access to all electronic medical records at 66 DOD sites, 109 Coast Guard sites and 3 VA sites, a program official confirmed to Nextgov.

“Affected clinicians were unable to log into EHRM applications or retrieve EHRM data to legacy applications,” according to an IT ticket detailing the issue obtained by Nextgov.

All told, more than 95,000 users were affected by the outage. The EHRM program official said there was no evidence that any patients were harmed due to the outage.

The official also noted that while the main EHR was down, the system failed over to a backup “read-only” system, through which clinicians could review patient data but could not update that information.

The systems were fully restored shortly before 8 p.m. Wednesday, according to the IT ticket.

The outage was caused by a bug in the Oracle databases and not due to ongoing deployment efforts, Terry Adirim, VA EHRM Integration Office program executive director, said during a press call Thursday.

During the downtime, VA hospital staff were able to continue with “most clinical operations,” Adirim said, though patient information had to be recorded on pen and paper and later updated in the EHR system.

Adirim told reporters VA leadership was not aware of any specific disruptions to care during that time. During such downtimes, clinic leadership submit “trouble tickets,” in which they note incidents that led to undue patient harm and can suggest a likely cause—such as an EHR outage.

“From what we know now, there haven’t been any reported to us,” Adirim said. “The two outages … we are not aware, nor has it been reported to us that there has been any harm to patients.”

When prompted, Adirim clarified that reports might have been filed but those have yet to be fully investigated.

“There might have been reports but there were no patient safety incidents that we’re aware of,” she said.

The restoration work was a joint effort by all parties, the EHRM official said, including Cerner, Leidos, VA, DOD and Coast Guard technicians.

Spokespeople for Cerner and Leidos did not immediately respond to questions, including whether other government or non-governmental systems were affected.

Wednesday’s outage is not the first of the year. The EHR system at Mann-Grandstaff was taken offline on March 2 after an update led to potential data corruption, according to an email from the medical center director obtained by reporters at The Spokesman-Review.

During that downtime, staff were told not to admit new patients and to “provide only those healthcare services you are comfortable providing assuming all electronic sources of data are unreliable," the paper reported.

After that incident, members of the House Veterans’ Affairs Committee wrote to VA Deputy Secretary Donald Remy, the official in charge of the agency’s EHR rollout, and Cerner Government Services President Travis Dalton.

“We are also concerned by the disabling effects this data corruption incident had on multiple elements of Cerner’s electronic health record system,” wrote committee ranking member Mike Bost, R-Ill., and Technology Modernization Subcommittee ranking member Matthew Rosendale, R-Mont. “It appears to have caused a complete work stoppage in community care referrals and revenue cycle and partial work stoppages in pharmacy and scheduling for several days. Its impacts still have not been completely resolved.”

The lawmakers noted the system was not working correctly for several days and warned VA against “any attempt to soft-pedal the reality” of the situation.

Editors note: This article was updated to include details disclosed during a Thursday afternoon press call.