New VOIP monitoring tools would make Ma Bell proud

It’s no secret that voice-over-IP gear is ringing up sales. Some estimates peg public-sector adoption rates at more than 35 percent. Unfortunately, the darker secret for many of the organizations is the inconsistent voice quality many of those deployments experience once VOIP is in place.

Despite lengthy upfront capacity planning before the launch of an IP telephony system, as much as 20 percent of all VOIP communications suffer from dropped calls or poor-quality conversations, according to some IP-equipment vendor estimates.

Another frustration for voice systems administrators is the mercurial nature of network problems — untroubled callers today may have unacceptable connections tomorrow.

To cope, information technology managers responsible for VOIP systems are turning to performance-monitoring tools to identify the causes of disrupted calls and stop problems before they interrupt conversations. The return on investment for those tools comes if they successfully increase network availability and reduce the time it takes to resolve problems, said E. Glenn Rogers, deputy chief information officer at the Food and Drug Administration.

But no single tool provides a complete picture of the IP telephony network and related components for supporting voice communications. Instead, agencies must stitch together separate monitors for their overall IP data networks, the IP PBX — the computer that translates voice communications into IP packets — and the wide-area network that links geographically separated offices.

Integrating multiple monitoring tools can be a juggling act that creates coverage gaps or buries administrators in conflicting data. “All networks are different, and a negative reading on a particular stat may be bad for one network but normal for another,” Rogers said. “Ensure you baseline everything before implementing tools to properly set thresholds and expectations of your network.”

Problem areas

An organization may face poor-quality IP calls because of decisions it made before implementing the phone system. Inadequate upfront assessments can result in an insufficient network infrastructure that buckles when large amounts of voice traffic suddenly start flowing through the pipes. Voice applications are sensitive to even slight delays in the delivery of IP packets.

The Department of Veterans Affairs uses VOIP in outpatient clinics, but it has not deployed the technology in more critical mission areas. The VA will strengthen its enterprise backbone in anticipation of a more extensive deployment in the years ahead. Until then, the VA is using network modeling from Opnet Technologies to determine network performance needs and special handling requirements, said David Cheplick, deputy director of the VA’s Telecommunications Operations Management Service.

But even the most careful planning can’t eliminate problems once the system is in place. The main culprits are a trio of conversation-stoppers known as latency, jitter and packet loss. A variety of performance breakdowns cause them. Too much data can overload individual network switches or routers. Spikes in voice traffic can similarly overwhelm the IP PBX’s CPU, memory and hard drive resources. Or performance issues can originate in the high-bandwidth T1 or T3 lines that commercial service providers manage for agency WAN connections. Finally, the gateway that connects local-area networks with WANs can malfunction or become overloaded.

Agencies have three main performance-monitoring choices to help pinpoint problems.

IP PBX vendors such as Avaya, Cisco Systems and Nortel Networks include monitors that focus on PBX-to-handset communications and warn administrators when CPU and memory resources are nearing the breaking point.

Some IP PBX vendors also sell management tools that peer into the switches and routers of the larger IP network to scout for bottlenecks. Examples include Cisco’s CiscoWorks and Avaya’s VOIP Monitoring Manager.

“We can diagnose calls as they’re occurring,” said Kurt Kruger, a senior manager at Avaya’s System Management Group.

More detailed enterprise network monitoring comes with traditional network management consoles, such as CA’s Unicenter, Hewlett-Packard’s OpenView and IBM’s Tivoli. But voice savvy is limited with these products, even with optional VOIP modules available from the companies.

“You’ll wind up looking at your voice traffic without necessarily getting all of the details about voice conversations and real-time performance that you need,” said John Burke, principal research analyst at Nemertes Research.

Voice savvy

To fill that gap, several third-party companies sell monitoring tools tailored to VOIP. Burke said the additional voice capabilities are critical for large government IP telephony implementations that have 5,000 or more phones.

“At that point the management of the voice component of the network becomes pretty hairy, and specialized help is useful,” he said.

Monitoring tools look at voice-specific criteria such as mean opinion score (MOS), a measurement developed by the telecom industry to calculate voice quality. In addition, the tools can compare scores with bandwidth utilization specifications.

Some tools, including ones from Brix Networks and Qovia, install sensors on network switches, routers and other components to send performance details to a central monitor that collects and aggregates the network performance data.

Depending on the problem, network administrators may be able to use the information from the monitoring tools to give voice traffic a higher priority in the fight for network bandwidth or reroute traffic around a congested network link.

Some voice tools also analyze the individual channels that make up T1 or T3 lines for WANs. “They can correlate data from all the calls going over a channel to give you the information you need to go back to your service provider and say, ‘I’m sorry, but there’s a problem here,’” said Steve Mank, chief operating officer of Qovia.

Although useful, those tools are costly. Smaller implementations with about 200 users require an investment of $25,000 to $50,000, Burke said. Midsize operations with as many as 2,000 users will spend about $75,000 to $100,000, while larger agencies need to budget more than $100,000 for a VOIP deployment.

Multilayered approach

The FDA’s tool strategy relies on two layers of tools. The top layer consists of a manager of managers (MOM) that centralizes data from all the lower-layer monitoring sources. The FDA uses Cisco Info Center at the MOM layer. The lower layer includes CiscoWorks for router and switch management and configuration, and it uses software from Prognosis to track IP PBX call managers and servers. The lower-layer monitoring tools watch for resource utilization and capacity problems with memory, hard drives, Network Interface Cards and CPUs.

“Prognosis also gives us the added benefit of quality-of-service metrics to ensure proper voice quality for all calls,” Rogers said.

The FDA selected the voice-monitoring product because the agency could easily integrate it into the existing MOM. The agency also liked that it sent early warnings about performance problems to the staff before they became serious. The FDA also required the tool to perform capacity planning and provide a database of historical information that the agency uses to develop performance baselines and trend analyses.

Rogers said using the tools can be challenging. A significant amount of time and resources is necessary to evaluate and implement them and train employees to use them properly. “The configuration of these tools can be very intense,” Rogers added.

He also encountered incompatibilities among the various sensor agents used by performance monitors to troubleshoot networks.

“We may have been running four agents for different tools and had a server lock-up,” he said. To fix the breakdown, Rogers had to unload all the sensors and then reinstall them one at a time to find the cause of the conflicts.

“Although we had the items running in our lab together, it was a gradual problem that happened over time that was not measurable in the lab over a week’s time period,” he said.

Joch is a business and technology writer based in New England. He can be reached at ajoch@worldpath.com.