Plugging security holes

Lacking automated patch management systems at the time, officials at the Department of Veterans Affairs were unable to stay ahead of a fast-spreading computer worm when it hit U.S. computer networks last year. A one-two punch from the Blaster worm and a variant named Welchia disrupted the VA's network for two days in August 2003.

"You take an organization of this size down for two days, that's pretty devastating," said Robert McFarland, the VA's assistant secretary for information and technology.

After the incident, officials promptly installed automated software patch management systems throughout the department to protect the agency's 300,000 PCs.

Consequently, damage from subsequent worm outbreaks has been minimal compared with the debilitating effects of the 2003 attacks, McFarland said. "The reason we don't have any problem here is that nobody wants to go through that again."

To protect against worm attacks, an automated patching system first scans a network's computers to discover which software patches are needed. Then it sends the necessary patches to the computers and installs them.

Compared with manual methods for applying security patches, automated patch management systems can apply patches more quickly and free employees' time for other tasks. For any but the smallest agencies, relying solely on manual patching methods is unworkable, said Peter Mell, a computer scientist in the National Institute of Standards and Technology's Computer Security Division.

Besides being faster than manual methods, automated patch management protects targets that are most vulnerable to malicious code attacks, said Shekar Ayyar, senior vice president of product marketing at BindView Corp. Ninety percent of Internet-based attacks target security vulnerabilities in computer operating systems and application software, he said.

Officials at Microsoft Corp., whose software is frequently the focus of such attacks, have begun releasing patches on a predictable schedule. That has made it easier to use automated patch management software, said Fred Duca, senior technology specialist in Microsoft's federal division.

A recent report on patch management published by the Government Accountability Office found that most administrators at federal agencies use a combination of automated and manual methods to patch computer and network systems. "Automated methods are most applicable to standard desktop systems," Mell said.

But for some server operating systems and customized applications, manual patching is appropriate and necessary, he said.

Industry officials say new regulations are partly responsible for an expanding market of automated patching systems. The systems provide accurate documentation that federal officials need to help them comply with the Federal Information Security Management Act, said Sam Curry, vice president of product management for eTrust security management at Computer Associates International Inc. "We help government agencies get real metrics," he said.

Although automated patching offers many advantages compared with manual methods, it has drawbacks. The systems can pose a security risk, although that risk has not been widely

exploited.

"If we try to automate things, the bad guys pretty quickly figure out how to exploit that automation," said Marcus Sachs, director of the SANS Institute's nonprofit Internet Storm Center, which monitors cybersecurity threats.

Mell agreed that centralizing control of any function, including patch management, creates a potential security risk. "It's something that needs to be considered and dealt with," he said. Within an agency, for example, someone could break into an automated patch management system and distribute malicious programs. "I've never known that to actually happen, but it is a possible risk," he said.

Another downside to automated patching systems is the large percentage of patches that require a reboot for installation. If the reboot happens at the wrong time, users can lose information, Mell said. "Some products are better than others [at] notifying users, but we've had difficulty with it here at NIST."

The alternative to automatic patch installation is to let users update their systems. But they often fail to act and certain patches never get installed, Mell said. The most effective approach that NIST officials have found is to send an e-mail message asking employees to log off their computers so systems can be patched.

NIST officials also register computers that should never be rebooted automatically.

An advantage of automated patching is that it is about the only way federal officials can apply patches fast enough to deter attacks on their systems. "It takes the average organization about 23 days to deploy mission-critical patches," said Paul Proctor, vice president for security and risk strategies at the Meta Group Inc., an information technology and business consulting company.

Another reason for the growing popularity of automated patching is the relatively low cost of automated patch management software. Its price ranges from $3 to $20 per desktop PC.

The effectiveness of automated patch management systems depends largely on their use. Federal officials are adjusting their policies and procedures as they gain more experience using the systems.

After their experience with Blaster, senior VA officials granted the Office of Cyber and Information Security the authority to set policies and manage cybersecurity throughout the VA.

Agency officials should set policies requiring that patches, especially security patches, be installed in a timely fashion, said Lynn McNulty, a member of the board of directors for the nonprofit International Information Systems Security Certification Consortium Inc.

"The policy would be that once a patch arrives on your desktop, it should be installed within a certain period of time." In addition, McNulty said, officials should set automatic procedures for ensuring that the patches are properly installed.

Testing patches before automatically installing them is another important procedure for which security experts said sufficient time must be set aside. Often, agency officials are faced with a trade-off between the time required to test a new patch and the urgency to install the new patch, Mell said. In extreme cases, the only available option is to stop using a vulnerable application until the patch for it can be fully tested, he said.

The number of companies that offer automated patch management tools has grown significantly. Companies that offer them include, among others, Bindview, BigFix Inc., BMC Software Inc., Computer Associates, Configuresoft Inc., Citadel Security Software Inc., Latis Networks Inc., Microsoft, PatchLink Corp., Secure Elements Inc., Shavlik Technologies LLC and St. Bernard Software Inc.

A future with bug-free software is one that most federal officials can only dream about. But meanwhile, they are doing their best with the tools available.

VA officials said they hope someday soon to have a single automated system with sufficient capacity to manage patching for all of the agency's desktop PCs. Until then, the combination of automated scanning and patching tools that security officials have installed are serving the agency well, McFarland said.

When the Sasser worm was released earlier this year, only 192 of the agency's 300,000 systems were infected. McFarland credits the VA's cybersecurity and information security officials with preventing more widespread trouble. "These guys are doing a very good job," he said.