CIOs preach the power of resilience

If federal IT managers have practiced their disaster recovery plans and their staffs know how to improvise, agencies may not be completely crippled by the potentially devastating cyber "Pearl Harbor" that former Secretary of Defense Leon Panetta ominously predicts could be lurking.

That was the message several current federal CIOs conveyed at a March 11 panel discussion on continuity of operations at Symantec's Government Symposium. In a keynote speech before the panel discussion, Panetta reiterated his October "Cyber Pearl Harbor" warning in light of growing tensions with Russia, competition with China and a restive world stage.

In his symposium speech, Panetta dubbed cyberwar "the most serious 21st century threat." Hurricane Sandy, he said, demonstrated some of the ruinous effects on infrastructure that a cyberattack might inflict.

In the face of such danger, federal agencies not only must have a disaster plan, but they should physically rehearse it -- repeatedly, said Rear Adm. Robert Day, the Coast Guard's Cyber Command commander and CIO.

Without practice, he said, plans rot on the shelf or fail in the face of real-life events. With increasing cybercrime, natural disasters and other potentially catastrophic events in the wings, not to mention local tech snags and outages, Day said the Internet and networks are more imperiled than ever. Even without a dramatic event, he said, federal agencies' network facilities eventually "will go down" because of some weakness, such as a router failure or other snafu.

In the end, natural disaster or cyberattack results are the same, Day said, with network facilities that don't work. It's then that an agency's plans had better be up to date and well-rehearsed. "We see so few organizations doing that."

Co-panelists Wolfe Tombe, chief technology officer at Customs and Border Protection, and John Dobriansky, executive advisor at the Federal Aviation Administration, agreed. All three noted substantial network outages at their agencies in the last few years.

Tombe said a local CBP router network outage at Los Angeles International Airport a few years ago that lasted an hour or so, causing a 17,000 passenger back-up at security lines, as well as air traffic delays because passenger manifest lists could not be checked. CBP officers, he said, improvised and used their cell phones to exchange information. Day noted Hurricane Sandy made the Coast Guard use its fallback servers in West Virginia linked to small aperture terminals set up in the disaster zone to continue emergency radio communications in the stricken mid-Atlantic area after the storm surge shorted out the telephone network. And Dobriansky said a 2012 fire at an FAA facility in New Jersey forced the agency to transfer some of its air traffic control to another facility in Melbourne, Fla. The outage lasted two weeks, he said.

In those instances, the disruption showed not only the weaknesses but also the strengths of their network disaster recovery plans. Day noted the importance of lower-tech workarounds that might be overlooked in such situations, especially by younger workers familiar only with the online world. "Young workers won't know how to do it," he said. "It makes me stamp my feet. Younger workers don't know what a fax machine is."

Day also noted that younger workers can also have ideas that older workers might overlook. After the earthquake that shook the middle-Atlantic states and Washington, D.C., in 2011, younger federal workers milling around on the mall in the aftershocks texted their colleagues rather than trying voice. "They knew that data was working and voice wasn’t," he said.

Day and Tombe said they perform regional and local drills of disaster recovery plans regularly. In Day's case, those exercises come just before hurricane season. "We do a data center fail over, forcing things onto our disaster recovery site. At the local end we do a once-a-year drill. Each sector physically re-establishes itself at its disaster recovery site." That means people physically pick up their gear and move hundreds, or even thousands, of miles to the alternate site.

Day recommended that IT managers file their most critical data and information on a disk or two and put those disks in their "go-bags."

"I can't tell you how many times I've heard 'I didn't think of that,'" he said.