Symantec software upgrade caused Military Health Record System shutdown

The Military Health System identified Symantec's Veritas Storage Foundation storage software as the cause of a shutdown of the AHLTA clinical data repository, which stores 9.7 million electronic records for active-duty and retired military personnel and their families.

The Defense Information Systems Agency also acknowledged it played a key role when the AHLTA -- or Armed Forces Health Longitudinal Technology Application -- CDR shut down last week.

DISA manages the Military Health System's AHLTA Information System Enterprise, including the repository, which is hosted in a Montgomery, Ala.-based DISA computing center that agency spokeswoman Laura Williams did not identify in email communication with Nextgov.

MHS said it took the CDR offline for most of the business day Tuesday, Jan. 17 to correct a problem with an upgrade to storage service software that was loaded over the three-day Martin Luther King Jr. weekend.

An MHS spokesman subsequently identified the software as Veritas Storage Foundation sold by Symantec. Williams said DISA needed to upgrade the software "to address a vendor operating system limitation that prohibits the operating system from accessing large file systems. This upgrade is necessary to the CDR as it is being prepared for increased data input and future growth expansions."

Williams said the problem experienced with the upgrade was an issue fixed in the previous version but wasn't included in the current CDR operating system. She said Symantec is in the process of developing a patch, which is expected to be delivered within two weeks. During the interim, AHLTA will run on the older version of the Veritas Storage Foundation software.

DISA operates a backup repository for MHS in San Antonio, but after the failure of the primary system, the agency did not revert to that backup site at the direction of MHS, according to Williams. Instead, AHLTA users were switched to local mode using backup cache servers while the problem was being addressed, she said.

The agency then backed out of the vendor upgrade from the production CDR, rebooted the entire AHLTA system, and verified that all sites were fully functional, Williams said.

This fall-back gives clinicians access to data saved locally on current patients, a Navy doctor who declined to be identified told Nextgov. But, if a doctor was seeing a new patient at the time of the shutdown, then little information would have been available, including important prescription information needed to ensure the patient did not experienced a dangerous drug interaction, the Navy doctor said.

Symantec did not respond to an email query sent Monday on the software bug.