How VA's $36 million move to the cloud evaporated

The Department of Veterans Affairs canceled a $36 million cloud computing deal with HP in May 2013 after a dispute between the CIO's office and the agency's inspector general over how long emails should be retained and concerns over system security, FCW has learned.

When the deal was announced in November 2012, it was one of the most ambitious cloud migrations of any federal agency, covering the VA's entire 600,000-strong workforce. HP Enterprise Services was selected as the integrator to deploy the system, which was set to go live with a March 2013 pilot for 15,000 users, involving calendar and email apps.

But the pilot didn't get off the ground because of "serious concerns" about the system's 90-day retention period for emails. Those concerns were raised in a Feb. 20, 2013, memo signed by acting Inspector General Richard J. Griffin that was obtained by FCW in a records request.

VA emails in the HP cloud, as explained in Griffin's memorandum, would be retained for 90 days before being automatically purged. The loss of emails, Griffin wrote, "raise[s] major concerns about our accountability and transparency in VA, VA's ability to defend itself in litigation, and, in particular, the OIG's ability to conduct our statutory independent and objective oversight of VA programs and operations."

The March pilot was going online just as then-CIO Roger Baker was preparing to depart, and Steph Warren, currently the head of the Office of Information and Technology (OIT) at VA, was preparing to take over on an interim basis. The 90-day retention order was put on hold, pending a study of the issue by a group of stakeholders, including OIG, the VA's general counsel and the National Archives and Records Administration, which sets government-wide policy for the storage of records.

By May 24, 2013, the deal was scuttled. The OIG wanted new contract language inserted into all VA cloud contracts designed to facilitate access and visibility into the system, preserve emails and increase the security rating under the Federal Information Security Management Act. There was pending guidance from NARA on records retention that would affect the disposition of email storage. It was determined that the necessary changes were out of scope with the HP contract, and it was terminated.

"The contract was awarded before the unique VA OIG requirements were fully elicited by the organization," Charles De Sanno, executive director of enterprise systems engineering at VA, wrote in a memorandum terminating the deal with HP.

Baker and Warren received an email – the senders' name was redacted in the FOIA request -- in August 2011 as the system was being drawn up, advising that they include language covering access for audit and investigation purposes in any contract for cloud services.

It's not clear from Warren's responses to Griffin's February 2013 memo whether the emails in the HP system were to be permanently expunged or automatically archived. The request for information put out by VA in 2011 suggests that all emails would be archived physically and retrievable via "rehydration," according to the contracting document. The 90-day limit, a VA spokesperson told FCW, "was the time for materials determined to be non-record" to live in the cloud.

Whatever the reason, the 90-day limit was not determined by capacity. The cloud-based system provided for 25 gigabytes for each user account. The average mailbox size under the local Exchange server system was about 150 megabytes, according to the OIG.

Personally identifiable information

The memo traffic on the cloud issue expanded to include other oversight issues regarding OIG's access to agency email.

The move to require personal identification badges to access VA computers had the effect of encrypting all email. In order to access email for oversight purposes, OIG investigators had to request decrypts from OIT, and requests quickly piled up, creating a weeks-long backlog. Eventually an interim solution was found, and since that time a vendor was identified to perform email decryption on behalf of internal VA customers with oversight responsibilities. That application is in development.

OIG was also frustrated by the inability or unwillingness of IT executives to comply with a request for any email aliases used by senior officials. While the OIT eventually supplied a computer-generated accounting of aliases in the VA systems, they did not provide "a list that was responsive to the request about senior leaders," said James O'Neill, deputy inspector general for investigations.

Finally, OIG was concerned that the security rating for the system was not high enough, considering that personally identifiable information might be moving across the VA cloud. The VA had contracted for a system rated "moderate" under the Federal Information Security Management Act regulations. The OIG wanted a rating of "high" because of the possibility that personally identifiable information could be at risk.

The FISMA ranking continues to be a sticking point. The OIG hasn't moved off of its determination that the VA's cloud should be "FISMA high". The VA's information security and IT operations experts recommended that "FISMA moderate was appropriate for this particular contract because VA's IT system is not officially a Privacy Act system of records, and because VA's email systems are not to be used to transmit sensitive information without encryption," a VA spokesperson told FCW.

According to documents, about $870,000 was obligated to the contract, but the dollar amount of sunk costs into the defunct contract are likely far higher, considering the staff time that went into the contracting process.

The VA still faces an email crunch. As Warren noted in a March 22, 2013, memorandum, the current VA email system dates back to 2006. The VA is maintaining the old emails and a "voluminous" number of attachments at "significant" cost," Warren wrote, adding: "If VA can migrate to technology such as cloud email and come to agreement on a reasonable retention period for email, the cost savings to VA will be considerable."

At this point, VA is retaining all emails indefinitely. "VA will revisit its email retention policy once [NARA] completes revisions to its guidance in this area," a spokesperson said. There are no current plans to put out a solicitation for a cloud migration.

HP had little to say on the matter.

"HP understands the Department of Veterans Affairs elected to terminate the Microsoft Exchange contract while it broadly re-evaluates its requirements to potentially move to a cloud-based solution with Microsoft Office 365. HP looks forward to continued work with the VA to address the agency's cloud security and privacy requirements," the company said in an e-mailed statement.