In need of a quick fix

Anyone who has ever tried to keep up with software patches knows the struggle can be akin to being trapped in a horror movie — something like "A Nightmare on Patch Street." Yet, with system security becoming more important in a networked world, managing all of those patches is increasingly a mission-critical function.

If agencies weren't already aggressively applying patches to fix critical security flaws, then the onslaught of computer worms that globally disrupted network operations last summer probably gave them a new sense of urgency.

Last August, the Blaster worm and its Welchia variant underscored the need for better procedures and tools for applying patches as soon as vulnerabilities are exposed.

There is little doubt that they are portents of things to come. Worms or malicious code can exploit a security flaw shortly after it has been exposed. These two worms exploited a remote procedure call vulnerability in several versions of Microsoft Corp. Windows software, overloading systems with self-generating bogus traffic.

Indeed, information technology managers in both the public and private sectors are finding it increasingly difficult to keep up with patches as the length of time continues to shrink between the awareness of vulnerabilities and the introduction of worms that exploit them.

A case in point: It took six months before the SQL Slammer worm, which adversely affected network performance worldwide in January 2003, exploited a known vulnerability in versions of Microsoft SQL Server database software. By comparison, the Blaster worm was released within weeks after the announcement of the vulnerability it exploited.

"It's becoming a challenge to keep up with current patches," said Mike Brown, director of the Federal Aviation Administration's Office of Information Systems Security. "How to maintain a current inventory of multiple systems given the profusion of vulnerabilities" is a challenge the FAA and other agencies are trying to address, he added.

According to the CERT Coordination Center, a federally funded Internet security research organization operated by Carnegie Mellon University, about 95 percent of network intrusions could be avoided by keeping systems up-to-date with appropriate patches.

The problem is that patches — pieces of software code inserted into a program to fix a defect — are often not applied in time, or at all. Manually testing systems for compatibility with patches and the process of deploying patches are time-consuming. Most organizations simply don't have the manpower to keep up, experts say.

The task is made even more difficult as critical security vulnerabilities continue to be exposed in software from Microsoft and other vendors. The CERT Coordination Center reports that more than 11,000 security vulnerabilities were discovered from 1995 to the first half of 2003. Add to this the fact that attack technology is becoming more sophisticated and you have a recipe for major network interruption.

Some security experts view patching as a reactive technology and, therefore, an ineffective tool to mitigate emerging cyberthreats.

"The average time to create a patch is two weeks," Gregor Freund, chief executive officer and co-founder of firewall vendor Zone Labs Inc., told an audience at a Networked Economy Summit earlier this fall in Reston, Va. What is needed is a "security ecosystem that is proactive," he said.

That may be true, but others note that patch management will have to be a component of vulnerability management for the foreseeable future. To be effective, however, patching must be done correctly, and it must be automated, experts say.

Eric Hemmendinger, research director for information security with the Aberdeen Group Inc., sees a clear indication that organizations are moving toward an automated approach to patch management. "What's driving it is the realization that you can't ignore patches," he said. "And if you can't ignore them, then the process has to be automated."

Agencies act

Ted Heazlit, information security program manager in the Agriculture Department's Agricultural Marketing Service (AMS), was an advocate of automated patch management well before the headaches caused by last summer's worm attacks.

"The presence of exploitable software code [has been more] of an issue in the last few years," Heazlit said. "Unpatched exploitable software became more of a factor in our cyberattacks we had here" at AMS. As a big user of Microsoft software, AMS has seen its fair share of security bulletins and patch alerts from the world's leading software maker.

Initially, Heazlit's division used Microsoft's Baseline Security Analyzer to scan for security problems in Microsoft systems. It also used HFNetChk software, developed for Microsoft by Shavlik Technologies LLC, which allows an administrator to check the status of patches on all machines in the network.

But AMS officials needed an automated tool that could be used to centrally administer patch management, yet at the same time give systems administrators in offices across the country responsibility for deploying patches on their own machines.

Heazlit chose PatchLink Corp.'s automated tool, PatchLink Update, after reading a review in a technology magazine. Plus, USDA officials endorsed the tool for use throughout the department.

However, an earlier version of the software did not enable AMS to delegate tasks to administrators in branch offices. The system allowed only one log-in ID and password, which in a decentralized environment gave everybody access to one another's machines, an unacceptable situation when critical servers are involved, he said.

A later version of the product provided the delegated administration Heazlit was looking for, and "that was our tripwire to deploy it as an enterprise tool," Heazlit said.

His team is now preparing to fully integrate PatchLink into the network. "We're creating regional PatchLink servers so administrators can log in and manage their own system," he said. "In addition, people here in [Washington,] D.C., can log in if they have responsibility [for operations] in Fresno, Calif."

The recent hands-on experience of one of AMS' local systems administrators provides a perspective on how an automated approach has made patching easier, he said.

This particular administrator manages more than 50 servers. From the weekend of Feb. 28 through March 3, he deployed eight patches per server, which required five reboots of each machine. All the patches were installed in a matter of hours without a hitch. Doing this same work manually, without PatchLink Update, would have taken a week, Heazlit said.

The Department of Veterans Affairs also was hit hard by the Welchia variant. The agency handled Blaster, but its network could not stem the flood of traffic generated by the Welchia worm, which was able to infect unpatched systems.

In the aftermath, Bruce Brody, associate deputy assistant secretary for cyber- and information security, said the agency discovered that its patch management procedures were horrendous.

Now, the "VA is currently deploying an enterprisewide software management solution that facilitates centralized oversight and management," Brody wrote in an e-mail message to Federal Computer Week, "along with dedicated patch management at the local, regional and organizational levels, and an enterprisewide vulnerability assessment and mitigation solution that includes the ability for centralized oversight and management."

The FAA is also moving toward an automated approach. Patch management tools from St. Bernard Software Inc. and vulnerability management tools from Citadel Security Software Inc. are among the seven applications agency officials plan to implement, the FAA's Brown said.

The agency is in the process of gathering requirements from related organizations. "We are working with the [Department of Transportation] to come up with a more automated system," Brown added.

Conceptually, the FAA's approach resembles a triangle. One side would be a scanner that detects vulnerable systems, and the second would be patch management. The third side of the triangle would be a trouble-ticketing system that ties into the other two and notifies administrators about vulnerabilities.

This highly automated process cuts down on the need for human intervention, Brown said, but by design, it does not automate everything. System administrators still want the ability to download patches so they can test them for accuracy and ensure that they do not disrupt existing system configurations.

A major requirement of the FAA's system is a centralized reporting capability, Brown said. The Federal Information Security Management Act of 2002 requires that agencies maintain up-to-date patches. Consequently, the Office of Management and Budget requires agencies to report on system vulnerabilities. A centralized reporting capability tied to trouble ticketing will help managers keep abreast of patching updates, he said.

The FAA is also testing the Patch Authentication Dissemination Capability (PADC) service provided by the Federal Computer Incident Response Center, now a part of the Homeland Security Department. That service gives users a way to get information on security patches that are relevant to their agencies and offers them access to patches that have already been tested in a laboratory. The service is available to civilian agencies for free. As of August, 41 agencies were using the service, according to center officials.

However, Brown said the "jury is still out" on PADC. In his case, he is waiting for the test results.

The VA is not using the patch dissemination service, according to Brody, but instead is evaluating the best way to subscribe to PADC while still being able to deploy internal, vendor-specific and commercial third-party products when appropriate.

The USDA's Heazlit said he didn't opt for the service because it was not robust enough for his requirements.

Critical elements

A sound patch management strategy involves more than just technology, experts say.

According to a General Accounting Office study submitted as testimony before a congressional subcommittee in September, other critical elements include management support, standardized policies, dedicated resources, risk assessment and testing.

Once the IT department has an up-to-date inventory of systems and knows which ones need to be patched, someone must be given the rights or permission to patch, said Alan Paller, research director for the SANS Institute, a security training and research organization.

The security experts often don't have the right to patch, while operations employees don't have the appropriate security knowledge. So agencies must assign a dedicated group of people from both sides to conduct patch management, Paller said.

Standardized policies are also important. FAA officials plan to release a policy for patch management soon, said Phil Louranger, program director for policy. The policy will enable the agency to "work from a common point of understanding and allow for as much automation as possible," he said.

However, Heazlit cautioned, "if you make this [patch management] process too much of a mountain, you lose your agility. You have to push [patches] quickly but with prudence. But if you wait around [and do nothing], it is conceivable you're going to be notified of a [vulnerability] when you are attacked."