Software upgrade knocks out Defense health record system

Tom Grill/Getty Images

The Military Health System shut down the AHLTA clinical data repository -- which stores 9.7 million electronic records for active-duty and retired military personnel and their families -- after experiencing problems with a commercial software package that manages data storage, a top MHS official told Nextgov .

This shutdown, in turn, forced military clinicians to use the AHLTA -- or Armed Forces Health Longitudinal Technology Application -- electronic health record system in "local mode," without access to the main record database.

The AHLTA repository was taken offline at 8:26 a.m. EST on Tuesday and was restored at 6 p.m. the same day, according to Mary Ann Rockey, MHS program executive officer for Joint Medical Information Systems.

Rockey said the CDR was taken offline "in order to correct a problem with a version upgrade to storage services that was loaded over the weekend," which included Monday, the Martin Luther King federal holiday. She did not identify the commercial software package.

In her emailed statement, Rockey said the commercial storage software package was "part of a series of upgrades that will enhance AHLTA data storage speed and capacity of the CDR. Several software and hardware components have already been successfully deployed to production without any problems and have improved AHLTA operational availability, speed, usability and capacity."

She added, "our analysis determined the root cause [of the shutdown] was a new version of commercial software that was installed with the AHLTA upgrade. The commercial software causing the problem has been rolled back to a previous version while the vendor addresses the issue to prevent future disruptions."

AHLTA supports 77,000 clinical users in 63 military hospitals, 413 clinics, 15 deployed hospitals and 63 Navy ships. Rockey said MHS health care providers continued to see patients during the outage and to document care while AHLTA was in local mode, though certain provider-specific features and functions were not available.

She said all the service chief information officers were informed of the outage within 30 minutes. Roger Baker, CIO of the Veterans Affairs Department, told Nextgov that he must personally inform VA Secretary Eric Shinseki if the Veterans Health Information Systems and Technology Architecture (VistA) electronic health record system, which serves more than 6 million patients, experiences an outage lasting two hours.

A source familiar with AHLTA, who declined to be identified, said the daylong outage revealed that MHS has poor or nonexistent contingency operations plans in case of an AHLTA failure. This source said MHS should have a backup data center in case of a failure to the primary data center. Baker said VA has backup systems to ensure continued operations of VistA.