Dial VOIP in case of emergency

To provide services for citizens affected by Hurricane Charley and two other storms in 2004, the host county for Disney World created a temporary emergency center in an empty warehouse that the county was planning to renovate. “It had nothing in it, including [no] communications infrastructure,” said John Amiot, enterprise operations manager for the county.

So officials trucked in PCs and network gear that supported VOIP, and within 24 hours, the facility opened for business with a line of storm-weary citizens already forming at the door.

“Our people could cut checks and provide all the things that people needed in order to survive,” Amiot said. “We were able to deploy in 24 hours because we had voice-over-IP technology already in place.”

Since then, the county has integrated IP telephony into its formal continuity-of-operations (COOP) plans by installing it in a regional computing center.

Expanding role
Business continuity plans are increasingly relying on new technologies to keep organizations running in the aftermath of natural disasters, terrorist attacks and more mundane incidents, such as local power outages.

“IP telephony and continuity of operations are an excellent marriage,” said Jim Biskaduros, a client delivery executive who specializes in security and intelligence at systems integrator EDS. In Orange County, the newly wired warehouse became an extension of its infrastructure. An IP network provided connectivity among employees and the county’s main offices.

“It was as if we had just opened a new building,” Amiot said. “This was in the center of one of the most devastated areas in the county, and it was strategically advantageous for everybody to come there versus getting them downtown while trees were down everywhere.”

By the time the last of the storms plowed through the area, the warehouse had logged more than 226,000 calls. The system automatically routed some calls normally intended for headquarters to the site.

Other agencies are also discovering IP telephony’s flexibility. The Department of Veterans Affairs turned to IP-enabled mobile communications trucks to keep Gulf Coast hospitals running after Hurricane Katrina hit last year. After finding success, the agency is extending contracts to formalize those backup communications tools and COOP plans.

Similarly, the Education Department didn’t initially have COOP in mind when it installed IP telephony in 2002. But the technology’s reliability spurred new ideas for how the department could use it.

“We said, ‘Hey, what if?’” said Peter Tseronis, Education’s director of network services. “We went back to the drawing board to seriously consider COOP.”

At those and other agencies, IP telephony provides resilience in emergencies because the networks that keep voice traffic flowing consist of widely distributed servers and connections that can pick up the slack if individual segments quit working. Traditional phone networks depend on point-to-point connections with fewer secondary options. Also, emergency workers can create ad hoc IP voice connections via IP phones or laptop computers with soft phone software that link to data network connections, including DSL and cable modems.

Upfront planning
Nevertheless, using IP telephony for COOP is still a relatively rare application at most agencies. “IP telephony is on their radar screen, but I wouldn’t say in general they’ve made lots of progress,” said Mike Corrigan, lead consultant at Suss Consulting. That’s partly because significant groundwork is necessary to make IP telephony reliable enough for COOP.

The first step for any basic IP telephony installation is for IT administrators to secure the data network that supports voice communications, said John Speicher, market development manager at Cisco Systems. That includes installing redundant IP exchanges — the computer servers that provide voice capabilities — network switches and uninterruptible power supplies that support the hardware.

COOP planners also need to understand the performance characteristics of their voice systems. “When you’re building your [request for proposals], account for survivability,” said Nora Freedman, research analyst for enterprise networking at market researcher IDC. Specifically, ask if solutions support automatic failover, and if so, how quickly, Freedman added. Failover means the systems smoothly and automatically reroute their functions if one or more component ceases to work properly.

Security is a greater concern for IP telephony than in traditional public switched telephone network connections. Traditional “voice is hard-wired from phone to communications closet to switch,” said Guy Clinch, solutions director for government and education at Avaya. “To compromise calls, you have to gain physical access. The data world is a lot more distributed with more access points.”

Experts say security needs to be an upfront consideration for COOP plans using IP telephony, not something that organizations cobble together in the middle of an emergency.

Agencies concerned about reliability and security, including those in the defense and intelligence communities, need private network links among the main headquarters and COOP sites. “I don’t think at this time the public Internet is reliable enough for a COOP-type service,” Biskaduros said. Another security strategy is to separate voice and data transmissions using Multiprotocol Label Switching and virtual local-area network technology. The former uses management software to string particular computers together into groups on the network rather than relying on hard-wired connections for the segmentation. The virtual approach simplifies network reconfigurations in an emergency.

Once administrators create individual voice streams, they can secure them with a virtual private network that encrypts communications. Separating data streams also delivers quality-of-service benefits. Administrators can give voice traffic priority over data packets to eliminate dropped or unintelligible calls.

COOP strategies that dispatch critical workers to home offices should provide them with VPN software to protect communications via the Internet, said Siafa Sherman, vice president of systems engineering at Nortel Government Solutions, an IP equipment vendor.

In addition to security considerations, the VA took an added step to prepare Gulf Coast facilities for the 2006 hurricane season. The department negotiated with telecom companies to ensure that all the services in the agency’s wide-area data network are part of the priority service restoration program that such companies make available to federal agencies.

“We want to make certain that we are queued up as higher priorities where life surety issues come into play,” said David Cheplick, deputy director of the VA’s Office of Telecommunications.

Service component
After its first IP telephony building came online in late 2002, Education used the technology to establish voice failover capabilities.

“The way we’ve structured and configured our IP communications network, we don’t have a single point of failure with our IP PBXs,” Tseronis said.

Education’s IP exchanges cover four sites in Washington, D.C. If an IP PBX fails in one of the buildings, other servers in the immediate cluster of computers automatically take over the traffic to keep communications running. If an event disabled the entire D.C. area communications infrastructure, calls would flow to Education’s disaster recovery site in Georgia, which would act as a communications hub to connect satellite offices.

The agency also installed routers that can send calls onto the public switched telephone network if the wide-area data network crashes. “That’s a definite no-brainer best practice,” Tseronis said. But he pointed out that such automatic failure capabilities take more than just buying hardware and expecting it all to automatically work. Like their VA counterparts, Education officials discussed their COOP plan with telecom service providers, which are responsible for rerouting calls if network segments go down.

“There is some manual intervention to ensure that if something fails, an alternative kicks in,” Tseronis said. “In some cases, it can be automated, but it takes working with the telcos.”

Discussions with service providers should occur early in the COOP planning process. “If you wait until the last minute, after you’ve developed your bill of materials to buy a new solution, it is not going to be well thought out,” Tseronis said. “That relationship needs to be born early on and continually cultivated.”

Joch is a business and technology writer based in New England. He can be reached atajoch@worldpath.com.