VA revisits data consolidation plan

The Veterans Affairs Department must answer new questions about its data center consolidation strategy after its electronic health records system was unusable for nine hours at 17 VA medical facilities Aug. 31, and backup procedures failed. VA officials have halted further consolidation moves until they finish evaluating the disruption.The disruption occurred as VA was moving its health records system, the Veterans Health Information Systems and Technology Architecture (VistA), from local hospital data centers to regional data centers. By creating regional centers, VA officials said they hope to provide better security and standardized management practices to safeguard critical health records and other electronic health information. VA has two regional data centers serving the West and Northeast regions. Officials planned to open two more regional centers by Dec. 31.The VistA disruption affected physicians and patients at VA hospitals, which rely on VistA’s electronic health records for patients’ clinical care. Without VistA, physicians had to revert to paper recordkeeping, which created opportunities for mistakes and slowed the delivery of health services. “Work to recover the integrity of the medical record will continue for many months since so much information was recorded on paper that day,” said Bryan Volpp, associate chief of staff for clinical informatics at VA’s Northern California Healthcare System. VA is investigating the incident internally and will hire a company to conduct an independent review to ensure that the department’s contingency plans work, Robert Howard, VA’s chief information officer, told lawmakers. Howard also is re-evaluating VA’s regional processing strategy.“We want to examine the whole program and build in more robust backup at the facility level,” Howard said. “We cannot allow hospitals to go down.”VistA stopped working in northern California because information technology employees did not follow policies and procedures, VA officials said in their initial assessment of the disruption. “The root cause of the outage was indeed…human error, more specifically, a failure to adhere to change management procedures,” said Jeff Shyshka, deputy assistant secretary of enterprise operations and infrastructure at VA’s CIO Office. The disruption happened during daytime hours, the busiest time for treating patients at the hospitals. Ben Davoren, director of clinical informatics at VA’s San Francisco Medical Center, related details of the incident during a recent hearing before the House Veterans’ Affairs Committee. He called it “the most significant technological threat to patient safety VA has ever had.”Shyshka said IT specialists from several VA health care facilities in the region were meeting at the data center. An employee did not follow procedures to vet and implement a system change, he said. VA resolved the regional data center problem by backing out of a network port configuration change that an employee made earlier that day. That port configuration is the likely cause of the problem, Shyshka said.“As with any collocation undertaking of this magnitude, there will always be the potential for human error,” Shyshka said. “Ensuring effective communications processes between the teams managing the collocated VistA systems and the IT staff at the local facilities is perhaps the greatest challenge.” Charles De Sanno, associate deputy assistant secretary of infrastructure engineering at VA’s CIO office, said he thinks what happened is an isolated incident. “I believe a mistake was made.” De Sanno is one of the architects of the regional data processing center program. He is also executive director of VA’s enterprise infrastructure engineering and northeast operations in New York. “What you have here is [like] someone crossing the street and not looking and [getting] hit by a car,” De Sanno said. Even if the change the employee made should have been made, it was not authorized, and it was implemented improperly, he said. “We have to ensure that we follow policy and procedure.”Under VA’s regional processing program, IT systems are physically secured in commercial data centers. Vendors own and maintain the data center physical plant and network infrastructure and provide support to VA and other customers using the data center. VA IT professionals operate, manage and maintain the department’s health care information systems remotely, Shyshka said.