Survey: Agencies are failing to maintain critical electronic records

After investigating itself for almost a year, the Justice Department reported to the National Archives and Records Administration that it is unable to determine whether any e-mails related to its notorious 2002 "torture memos" were improperly destroyed.

The department's finding, which it delivered to NARA in February, appears to close a troubling case of lax records management by a federal agency. Officials at NARA accepted the explanation of the Justice Department's chief records keeper of why thousands of e-mails vanished when they were needed for an investigation and closed the agency's examination of the matter.

The torture memos case underscores the importance of preserving records, and it highlights shortcomings in the way agencies are handling the shift from paper records to electronic formats. Earlier this week, NARA released a report based on the self assessments of 270 federal entities. The survey indicates 95 percent of agencies "are at high to moderate risk of compromising the integrity, authenticity and reliability of their records."

The so-called torture memos are a case in point. Written by then-Assistant Attorney General John Yoo and several of his deputies, they provided the Bush administration with legal justification for using waterboarding and other "enhanced interrogation" techniques on prisoners in the war on terrorism.

The techniques were controversial because waterboarding and other forms of aggressive interrogation are banned under international laws and treaties that forbid torture. The memos provided an expansive new interpretation of presidential power in wartime and a definition of torture so narrow that almost no abuse was considered torture.

The memos remained secret until they were leaked in 2004.

Public uproar and a request from Rep. Frank Wolf, R-Va., prompted the Justice Department's Office of Professional Responsibility to investigate Yoo and others involved in writing the memos for possible ethics violations.

Five years later, the Office of Professional Responsibility found that Yoo and Assistant Attorney General Jay Bybee had exercised "poor judgment" in writing the memos. At the same time, OPR reported that it was "hampered" in its investigation by the apparent deletion of Yoo's e-mails, and the disappearance of most e-mails for Patrick Philbin, a deputy assistant attorney general who helped write the memos.

Failure by government agencies to properly maintain important records is widespread, NARA reported. In 2009, it established an annual requirement that all federal agencies subject to the Federal Records Act perform self assessments and report the results to the National Archives. NARA has legal authority to oversee and inspect agencies for compliance with laws and regulations that govern the disposition of official records.

The torture memos case was one of 28 records mismanagement incidents that remained unresolved at the end of 2010.

Others include improper handling of:

-- Defense Department "documents related to torture"

-- The Homeland Security secretary's briefing books

-- Consumer complaint letters to the Federal Trade Commission

-- Food stamp records

-- Security and Exchange Commission records

-- White House e-mails

-- Electronic records at the Labor Department

-- Defense Intelligence Agency employment records

-- FBI papers

One danger posed by lax records management is that agencies won't be able to efficiently carry out their missions, said Paul Wester, director of NARA's modern records programs.

Wester also worries "records that have enduring value in documenting the national experience" will be lost, that agencies won't be held accountable for their actions and that citizens' rights may be violated.

Records include "books, papers, maps, photographs, machine readable materials, or other documentary materials regardless of physical form or characteristics."

That's a vast assortment of stuff, considering all of the letters, brochures, messages, websites, reports, testimony and other communication government agencies generate.

Wester said ultimately, about "2 percent to 3 percent of all federal records have permanent value and need to be transferred to the National Archives." According to NARA's survey, much of the time that's not happening.

"Based on how agencies conduct their business today, we are particularly concerned about the continued preservation of and access to electronic records," Wester said in an e-mail.

Databases, geographic information and Web content all may be electronic records that are important enough to preserve, he added.

"Electronic records are particularly challenging because of the volume that the federal government generates, as well as the incredible number of different formats of electronic records that agencies have, and the complexity of the formats," he said.

To complicate matters further, many agencies don't have full-time records managers and most don't conduct annual evaluations of their records management.

When it comes to electronic records, many agencies resort to "inefficient and ineffective print and file practices" for preservation purposes, NARA said in its report. Others rely on IT system backups for record preservation even though backups are intended as an emergency retrieval system not as permanent storage.

In a federal workforce of 4.6 million military and civilian employees, there are only 3,174 records managers and "NARA believes that this number is actually lower," the report said.

NARA recommended it and U.S. agencies "should explore, test, and if effective, deploy automated solutions to manage records and information."

As for the torture memos, after a search, Justice Department investigators were "able to locate additional e-mails on backup tapes and a server that were not examined during the course of OPR's investigation," Justice records management chief Jeanette Plante wrote to NARA Feb. 4.

"That said, we have not and cannot determine whether an unauthorized destruction of records occurred," she said.

"In light of the fact that over seven years has passed since Yoo left the department and Philbin left OLC, it is impossible for us to determine whether technical issues caused the loss of the e-mails or whether they were deleted by an individual," Plante wrote.

Nonetheless, the server and 25 backup tapes yielded 15,151 e-mails and attachments that the Justice Department was unable to find during the torture memos investigation. At the time, the department said it was able to locate about 500 e-mails.

The new e-mails and their contents have not been disclosed, said Anne Weismann, who has followed the case as chief counsel for the watchdog organization Citizens for Responsibility and Ethics in Washington.

Even these new e-mails would have vanished "but for DOJ's failure to comply with its own policy on overwriting backup tapes," she said. The backup tapes were intended for emergency recovery, not for long-term storage, and Justice Department policy permits them to be overwritten after about a year, Weismann said.

"We have NARA to thank for pushing DOJ to account for the missing emails," she said. Since NARA has closed the torture memos case, it may too late to determine whether any e-mails in the matter were deliberately destroyed, she said.