Into the backup fold

DOT's centralized backups improve protection of remote office data.

Much of what an organization does happens in the field, not at headquarters. In the corporate world, 90 percent of a company's employees, on average, work outside of headquarters, according to Nemertes Research, an information technology research company. That means a considerable amount of data exists outside the main office and its sphere of data protection.

Federal agencies may not be as radically distributed, but even those entities have significant remote operations. The Transportation Department's Federal Motor Carrier Safety Administration (FMCSA) is one example. FMCSA is based in Washington, D.C., but operates almost 90 offices nationwide. The agency seeks to reduce crashes involving large trucks and buses.

FMCSA officials didn't have to worry about backing up their data until the agency became a separate DOT administration Jan. 1, 2000. Before then, the organization was part of the Federal Highway Administration (FHWA). FHWA maintained the organization's data on Novell servers, which were backed up locally.

In 2000, FMCSA decided to make Microsoft Windows NT its operating system of choice. They sought to back up FMCSA's Windows servers, but decided they couldn't rely on administrative staff in remote offices.

"With almost 90 offices nationwide, we could not depend on local backups being consistent, timely and accurate," said Steve Erlitz, a senior systems engineer at FMCSA.

Cost of inaction

Officials at the fledgling administration were concerned that inadequate local backups could cause them to lose data.

"I heard stories that many backups were not being done correctly, and the data [was] being lost," Erlitz said.

He said he thought of the 1995 Oklahoma City bombing. "At that time, we did not have our own servers and all the data there" was lost, he said.

The fix

FMCSA decided to centralize backup operations. "We wanted a backup standard where all data was backed up to a central location and duplicated for disaster recovery, yet not tie up the data lines doing it," Erlitz said.

The administration's initial solution was to use a product called Live Vault from Live Vault Corp. "The product worked well, but the company discontinued it in 2003," Erlitz said.

Storage integrator IMS Systems proposed a branch office consolidation solution based on EMC's RepliStor and NetWorker (formerly Legato) products. FMCSA signed on.

Today, the administration's remote offices replicate data via RepliStor to two servers in FMCSA's headquarters.

The agency's two servers are backed up using NetWorker and two Advanced Digital Information Scalar 100 tape libraries. In addition, a copy of the data is sent to the administration's disaster recovery site.

Scott Carson, IMS Systems' chief technology officer, said the solution provides assurance that the administration's critical information is protected. "Originally, they really weren't able to certify that," he said, referring to the pre-centralization era. Now, backups are centrally administered and monitored.

The total solution, including software, hardware and the tape library, costs about $150,000.

Payoff

FMCSA's backup solution provides greater data protection without disrupting operations. Remote replication and backup have the potential to consume network resources, but RepliStor updates only those blocks that change rather than entire files.

"Unregulated backups can be a bandwidth hog," Erlitz said. FMCSA's solution "replicates changes — including new files — as they are saved rather than trying to run all changes overnight when other things are happening. The product controls bandwidth usage"

Another benefit is cost avoidance. Carson estimated the cost of assigning an IT worker to backup duty at $50,000 annually per remote site. For an organization with 90 offices, such staffing costs would reach millions of dollars.

Words to the wise: Smoothing remote backups
  • Choose the proper backup interval. Examine what kind of network connectivity you have between branch offices and central data center. Do you want continuous replication or one batch at the end of the day? The answer depends on how much bandwidth is available and how critical the data is.

  • Think about the big picture. In determining the importance of data, don't limit yourself to the data center. Scott Carson, chief technology officer at IMS Systems, said organizations should assess the value of data at remote sites and central facilities. "Consider the collective impact on the mission of your agency," he said.