NARA digs out of digital avalanche

The National Archives and Records Administration has decades of experience managing and preserving the nation’s most vital historical documents, from the Declaration of Independence and the U.S. Constitution to a vast gallery of photos from World War II.

The agency also catalogs and maintains vast warehouses of government documents, protecting the public record — a linchpin of democracy — for future generations.

But the tools of the trade that have served NARA for so long are no longer sufficient. With much of the business of government conducted online, NARA must transform itself by developing new tools, techniques and policies for ensuring that electronic records are searchable and accessible, not just now but for years to come.

The agency’s solution is the multimillion-dollar Electronic Records Archives (ERA), a project that NARA insists is progressing well despite audits to the contrary. Meanwhile, the electronic records continue to pile up.

In an interview Aug. 21, Paul Brachfeld, NARA’s inspector general, stressed the importance of making the government’s digital records readily searchable and retrievable for analysis and litigation, which ERA promises to do. But there’s a catch.

“To me, we’re kind of setting ourselves up as the only show in town for future access and analysis of records, and if it works, great,” he said. “If it doesn’t work, we got problems.”

Other issues loom. Open-government advocates are pushing for the next archivist to take a more active role in ensuring public access to government information, fueled by President Barack Obama’s commitment to increasing transparency. And the agency must address concerns about its ability to protect sensitive documents, which was called into question by a recent revelation that an external hard drive containing sensitive and potentially classified files from the Bill Clinton administration’s White House is missing from a NARA facility.

Those challenges face NARA’s next leader, experts say. If David Ferriero, Obama’s pick to be archivist of the United States, is confirmed by the Senate, he would find himself at the epicenter of the policy and technological challenges that the proliferation of electronic data poses for the government.

The swiftness with which digital formats, technologies and architectures become obsolete is of particular concern to those charged with preserving records in perpetuity. And the longer it takes to develop the solutions, the bigger the avalanche of digital data will grow.

The archives of the future

The Government Accountability Office expects NARA’s next-generation archive to cost more than $550 million, and NARA had already spent more than $237 million by the end of fiscal 2008. NARA officials refer to the project as the archives of the future and say it won’t depend on any particular software or hardware, something that is important to accommodate changes in technology.

But in its most recent semiannual report to Congress, NARA’s Office of Inspector General listed the ERA program at the top of the management challenges that the agency faces. GAO has also raised concerns about some aspects of the project.

GAO recently reported that NARA’s spending plan for ERA didn’t have enough detail and the agency didn’t have a contingency plan for the system or a fully functional backup and restoration process. Auditors also found methodological weaknesses during a review of NARA's fiscal 2009 spending plan for the program that could limit its ability to keep tabs on the project.

NARA’s acting archivist, Adrienne Thomas, said agency officials briefed Congress in April on specific outcomes ERA would achieve, adding that the agency would include more details in the next spending plan. She also said the agency is in the process of updating its project management tool and conducting a final review of its contingency plan for the system.

Meanwhile, even after revising its initial schedule for the system, NARA said it’s on track to achieve full operating capability when its contract with Lockheed Martin ends in March 2012. The contract is worth about $317 million.

NARA officials told reporters Sept. 2 that they had ingested 67G of data into an ERA system for federal records and 64.4T of searchable records from the George W. Bush administration. The Bush administration total of 77T is about 35 times the amount of data received from the Clinton administration, they said.

Martha Morphy, NARA’s chief information officer and assistant archivist for the Office of Information Services, said that at full operating capability, ERA will be able to ingest records, keep them safe and secure, and provide access to those records. She also said the system needs to be flexible to deal with different computer formats as technology evolves.

She said the goal is to have as much information online as possible by 2012 when ERA’s full operating capability is reached. But NARA hasn’t yet determined how much material will be online at that point because it will depend on budget constraints. However, it will be mandatory for agencies to use the system to submit electronic records starting in 2011, a step officials say will lead to more information being available online.

Meanwhile, Brachfeld said it wasn’t clear what the system will be able to do when it reaches full operating capability and what people using the system will see when they log on.

“I keep waiting to find bedrock, I keep waiting to feel comfortable,” he said. “I’ve not gotten an answer that I can embrace [or] feel comfortable in, and that’s a problem.”

David Kepley, ERA transition officer for federal records, told reporters Sept. 2 that when ERA reaches full operating capability, the user will have a unified research experience. “One-stop shopping, one portal that will direct you to all the materials that relate" to NARA, he said.

Sean Murphy, Lockheed Martin’s program manager for ERA, said the company has been working with NARA on a prototype for the public interface.

But Patrice McDermott, director of OpenTheGovernment.org, whom NARA recently briefed, said the project has been a tremendous disappointment considering the amount of money NARA has spent.

“I didn’t come away with a lot of confidence,” McDermott said of her recent meeting. “My level of confidence is not high…and the new archivist may want to bring in somebody from outside just to review what has gone on and to hold people accountable.”

She said NARA has oversold the system from the beginning. “It’s a series of problems, but I think at the end of the day, at least at this point, we don’t have what we would have expected to have.” However, she added that she was impressed by the system’s ability to let agencies manage their record schedules.

Meanwhile, Morphy told reporters the system’s core requirements haven’t changed, and the agency is now on the third of five increments. It is focusing on public access to records held in the agency’s pre-existing electronic records databases and developing a strategy to ensure that it can preserve electronic records as technology platforms change. NARA is also reviewing the architecture of the system to accommodate recent changes in technology.

Meredith Fuchs, general counsel at George Washington University’s National Security Archive, said she’s concerned that the public access component of the system remains unresolved. However, she said it’s good that ERA will let NARA monitor whether agencies transfer records.

NARA’s role up for debate

Beyond the obvious challenges of how to handle electronic records and bolster security, the new archivist will also have to set a course for the agency.

“My organization and most historians and openness advocates view NARA as an organization that can and should advance transparency in government and accountability in government,” Fuchs said. “NARA, I believe, sees itself more as the museum that holds old stuff and occasionally pulls out interesting things for the public to see.”

Other advocates agree with her sentiment and want NARA to take a more active role in efforts to improve access. Anne Weismann, chief counsel at Citizens for Responsibility and Ethics in Washington, said NARA should be a leader in that regard. CREW and Fuchs’s group sued NARA and Bush administration officials in 2007 because of that administration’s practices for archiving e-mail messages.

If the agency doesn’t have sufficient money or statutory authority to play that role, it should demand it, Weismann said.

“Governmentwide, I think that the new archivist needs to bring NARA into a leadership role on access to information and also in resuming their statutory responsibility for governmentwide records management,” McDermott said. “I think the new archivist needs to be a visionary and understand how technology can move government information forward.”

Focus on security

In an incident that has shined a spotlight on security issues, a 2T external hard drive that contains copies of records from the Executive Office of the President during the Clinton administration disappeared from a processing area at a NARA facility near Washington, D.C. NARA is offering a $50,000 reward for information leading to its recovery.

The drive’s disappearance has led to congressional scrutiny, thousands of breach notifications and increased security at NARA facilities. Brachfeld said the agency is also trying to identify the types of data that might have been on the drive. 

“There’s a tremendous, monumental, almost unfathomable amount of data — e-mails, correspondence, all quality and content of records — so we're just trying to better identify the entire scope of material that was on that drive,” said Brachfeld, who is leading the investigation.

A forensic analysis has reviewed only about 10 percent of the potential data, he said, adding that his office is working with subject-matter experts to use keyword search technology to identify records on the drive that might contain classified or otherwise sensitive information.

The security controls NARA had in place were limited and could be easily defeated. “It was not a proper work environment in which records of this quality and content should have been exposed,” Brachfeld said of his visit to the site where the hard drive went missing. “Security did not seem to be focus No. 1.”

“Quite simply, this was an accident waiting to happen, and now it has,” he told a House subcommittee July 30.

Michael Kurtz, NARA’s assistant archivist for records services, said in a recent interview that since the incident, the agency has beefed up enforcement of security procedures in processing areas, including sweeps of the processing room. He also said he believed his office’s overall record was strong.

NARA has taken all records that contain private information or presidential materials into secure areas where only a handful of people can access them, he added.

“I think the incident has really reinforced for us…that so much information can be stored in such a compact way that it makes the questions of privacy and security really paramount in ways in which was not the case with traditional paper records,” he said.

The investigation will likely carry over into the tenure of the next archivist. “We have hopes — obviously, I wouldn’t be doing this investigation if we didn’t have hope — that perhaps we could identify whether or not there was a specific individual that may have taken the drive, but we’re also…focusing on looking for classified material and the content of that material, if in fact we find it,” Brachfeld said.

In addition to physical security, NARA is also working to improve information security, which the agency declared a weakness in fiscal 2007.

Since then, Morphy said, officials have doubled the number of IT security employees and revamped the program. At this point, the agency is “in pretty good shape in terms of IT security,” she added.

However, the most recent semiannual IG report said IT security “continues to present major challenges for NARA.” And Brachfeld said NARA should consider its responsibilities linked to national security.

“We should better present the risk factors and the monetary requirements,” he said. “We’re not going to get funding unless we define the problem.”

He added that he looks forward to speaking with the next archivist and sharing his observations. “I hope that the next archivist will give me the opportunity to present my views, and whether they embrace them — I can’t control that,” he said.

Prudence Adler, associate executive director of the Association of Research Libraries, knows Ferriero from his work as director of the New York Public Library — one of the largest research libraries in the country. She described him as someone who understands the issues, which enables him to figure out the best solution.

“One of the really strong credentials he brings to bear … is his engagement with different constituencies,” she said.

If Ferriero is confirmed as the country’s next archivist, that ability is certain to be put to the test.