Keeping systems properly stitched

Patch management is not new. Eight years ago, TuneUp Utilities (from TuneUp Software GmbH) and Oil Change (now owned by Network Associates Inc.) gave individuals the ability to scan their computers for a variety of software updates and apply the latest versions automatically. Unfortunately, the average user showed little interest in keeping up with the changes, and automated updating died out.

Today, patching is mission-critical as systems administrators race to apply security patches before hackers can take advantage of newly discovered vulnerabilities. Seemingly overnight, a slew of small but highly competitive companies has sprung up to meet the market demand for organized patch management.

Although no clear leaders have yet emerged, we looked at four of the top contenders. Two of these — BigFix Inc.'s BigFix Enterprise Suite and PatchLink Corp.'s PatchLink Update — are agent-based. Loading agent software on each supported node has several advantages, including the ability to manage automated patching on large networks. Shavlik Technologies LLC's HFNetChkPro, which does not use agents, works well in small to midsize offices.

These products have their advantages and disadvantages. Agents use less bandwidth and work well with roaming users. Nonagent products are simpler to use and make it easier to keep tabs on networks that change frequently, but as a network grows, the traffic-scanning process quickly eats up bandwidth.

St. Bernard Software's UpdateExpert is currently the only product that offers the choice of employing or not employing agents; other vendors are expected to add that feature.

Systems administrators face the dilemma of either applying patches as soon as they are received or waiting until new ones have been thoroughly tested. Patches applied too early may have serious bugs, but we all know what can happen if they are applied too late.

The decision about when to patch will be different for each organization. However, patch management programs can help if they give systems administrators detailed control over the patching process. For example, they may decide to distribute critical patches immediately but delay other patches until they have been completely tested. Administrators also may want to immediately apply patches from companies with a record of reliability — we have never observed problems with patches from Cisco Systems Inc., for example.

Another concern for systems administrators is patching roving users' computers. Installing agents on laptops should ensure that the computers receive the latest patches as soon as users connect to the network.

Administrators also want to group patch distributions to target different types of machines and departments within organizations. Ease of use is particularly important when deploying patches, and although administrators want patch information at their fingertips, they do not want extraneous data cluttering their screens.

Asset tracking is essential for a truly secure network, and some of the products we tested provide useful inventory reports. Unfortunately, that isn't always a high priority for systems administrators.

We spoke with administrators who use the products we reviewed and found that each had enthusiastic adherents — and critics.

Despite the large number of players, small patch management companies are making money, and no industry shakeout should be expected in the immediate future. However, many of these companies have reached a level of maturity that may attract the interest of corporations with deep pockets. After all, patch management is a form of software distribution, and in the long run — perhaps three years — automated software distribution will subsume this niche market.

For organizations that are small, have no budget for patch management tools or want to wait until the industry matures, Microsoft Corp. offers free alternatives such as Software Update Services and Microsoft Baseline Security Analyzer. For administrators who choose this approach, we recommend also considering VA Software Inc.'s free deployment tool, MbsaFU (sourceforge.net/ projects/mbsafu).

Although no product among those we tested had all the features we wanted, we found BigFix Enterprise Suite to be the strongest overall. For small offices that don't have to patch anything other than Microsoft products, HFNetChkPro would be our choice, mainly because of its ease of use.

BigFix Enterprise Suite 4.0

BigFix Enterprise Suite has a console/server architecture with the ability to deploy to multiple servers throughout the enterprise. According to company officials, each deployment server can support 75,000 clients. If a sufficient number of relay servers are used with no more than 7,500 clients per server and if each patch deployment is carefully planned, an enterprise could probably achieve that number. No other patch management product claims it can scale to this level.

We installed the deployment server on a Microsoft Windows 2000 Server with the required Internet Information Services (IIS) Web server installed. We were careful to meet BigFix's stringent requirements for a clean operating system. The BigFix installation program automatically loaded Microsoft SQL Desktop Engine 2000 on the server.

We would recommend a full-blown Microsoft SQL Server for administrators maintaining more than a few hundred machines. On the extreme end of scalability, it would be possible to utilize the server's advanced replication services to geographically distribute your databases to span wide-area network (WAN) links. Combined with BigFix's console/server architecture, this approach would allow many deployment servers to cover a vast area with all data being stored by a replicating ring of SQL Servers.

We were impressed by the program's health-check utility because it provides a "traffic light" display of all the system's important components. After flipping through several tabbed pages of green lights reporting on everything from Common Gateway Interface scripts to running processes, we figured we were good to go.

We used BigFix's Client Deploy Tool to load agents on PCs running Microsoft Windows 95, 2000, XP or 2000 Server. Then we were ready to begin actually managing patches. Not surprisingly, the first time we logged in to the console, we couldn't figure out how to get patch activities started. Had we read the manual, we would have known that we had to wait about 10 minutes while our server contacted BigFix's server to populate its patch database. Once this was completed, we had no navigation problems.

To our surprise, our system lacked many patches. We wanted to test BigFix's "effortless" automatic patch installation, which is supposed to avoid patch conflicts. So, we elected to install all the patches at once, which would have given us heart attacks had we done it in our real-life enterprises.

The deployment proceeded quickly with no problems. We were pleased that BigFix placed the patches in a queue system to await download by clients. Using this system, we were able to monitor the deployment's progress.

Overall, we give BigFix high marks. In addition to incorporating its own certificate infrastructure to handle security — instead of relying on the operating system or SQL Server — the product provides critical features such as bandwidth throttling and tiered/timed patch distribution.

The Windows client is also capable of policy enforcement. For example, if an organization's policy is to automatically uninstall Sharman Networks Ltd.'s Kazaa, then even when a laptop is separated from the network, BigFix will detect a new installation of Kazaa and automatically uninstall it.

The product's reporting tools are excellent and include the colorful graphics necessary for demonstrating progress to upper managers.

Support for Linux and Sun Microsystems Inc.'s Solaris — along with planned support for other operating systems — is the icing on the cake.

PatchLink Update 5.0

PatchLink Update features role-based patch administration. This allows patch management duties to be distributed among many people. It is perfect for administrators who don't want to do everything themselves.

The PatchLink software requires a Microsoft Windows 2000 Server. Like BigFix, it uses a

console/server architecture. Promotional material for the product boasts that it has all the right features to manage an enterprise full of misbehaving computers. All industry-standard features are built in, including the capabilities for multiple role-based administrators and to enforce a patch baseline that computers must comply with. Additionally, PatchLink offers client agents for Windows, Linux, Solaris and Novell Inc.'s NetWare.

It takes only one step to install the product. IIS is automatically configured, and the Microsoft Data Engine is installed. After 15 minutes, we were ready to use PatchLink.

The product is accessed using a Web browser and relies on Web-based authentication to identify users.

We liked the Web interface because it replaces the need for an installed console application and allows easy access. But this feature raises security concerns. We advise applying IP restrictions through IIS, allowing only address blocks from trusted hosts. Administrators at many large organizations install gateways to other networks for business reasons. But even though your network may be secure and well-maintained, your neighbors' or customers' networks may not.

PatchLink's interface is simple to navigate. We were immediately able to access all areas of the application and were never more than a few clicks away from information we needed.

We were pleased that the client was easy to install on workstations and did not require a reboot. Shortly after it is installed, it checks in with the server and shows up in the console listing. As computers are imported, they are automatically added to a default group that corresponds to their operating system type.

Before a patch can be deployed, the target workstations must be scanned to determine what software the machines already contain and what patches are needed. In the production environment, inventories will be conducted at scheduled intervals by each machine group.

PatchLink contains all the patches needed to maintain a diverse enterprise environment. The product offers support for more operating systems than any other product we tested, and there were even updates for some essential third-party applications, such as Adobe Systems Inc. Acrobat.

We did, however, find the patch interface to be a bit inefficient. When first entering PatchLink, all patches for all operating systems are displayed, with no method to search through them. We used our Web browser's search function, but it slowed us down.

Nonetheless, patch deployment is efficient and allows separate patch combinations to be applied to different machine groups.

As with BigFix, the PatchLink client supports policy enforcement, such as removing Kazaa from computers.

We would appreciate the ability to generate more status reporting while scanning or deployment is taking place. PatchLink does provide some information on the

deployment's status, but no debugging

option allows you to see what is taking place between the client and the deployment server in real time. We are also concerned about the product's lack of hierarchical group capabilities to mirror WAN boundaries.

After taking PatchLink for a spin, we would be comfortable using it with up

to 1,000 workstations. We have heard reports of beefy servers handling up to 5,000 systems.

PatchLink is an established company and, although it does not scale as high as BigFix, we give PatchLink Update high scores for a good graphical interface and an excellent selection of supported operating systems and applications. However, we do feel the graphical interface could be improved by cutting back on extraneous information, such as patches not related to the operating systems of the group being patched, and on the number of clicks needed to select the patches to be deployed.

UpdateExpert 6.1

The UpdateExpert system relies on three components: a management console used to direct patch activities, a master agent that handles the deployment and a leaf agent that is required only for machines that can't be managed via Microsoft Remote Management. Administrators are able to separate those computers in any combination. There can be only one master agent, but multiple instances of each component are allowed on the network.

The UpdateExpert folks were doing their homework when they built this client:

Command-line options for an easy silent install are provided from the start, and the client installs easily and requires no reboot. Administrators can install UpdateExpert on just one Windows 2000 or Windows XP Professional workstation. After launching the program, we had two options for management: through Windows Networking or an installable client agent.

We were pleased with what we saw while rummaging around the interface. Upon querying our Windows XP Professional test workstation, we were immediately presented with all installed software, intermingled with applied and available patches. This took only a few seconds on our local-area network.

Available patches that were not applied were marked with a gray light, and obsolete patches were shown with a red light. We appreciated this display but have a few suggestions. People automatically want to equate green and red light with traffic lights. We would recommend for the green light to signify installed, the red light to signify required patches that are not installed, a yellow light to represent optional patches that are not installed, and the gray light to represent obsolete patches.

UpdateExpert's graphical user interface is great. Our favorite function is the ability to have patch data from the provider's Web site (usually Microsoft) at our fingertips. However, some of the URLs that UpdateExpert associates with certain patches are outdated and return a "page not found" notice from the Web site.

As far as responsiveness and stability go, UpdateExpert does a great job of instilling confidence with its native Win32 interface. We are confident that if the product had multiple master agents, it would scale to meet the needs of today's large corporate or federal enterprises.

We see one serious limitation to this product's design: It lacks the ability to support multiple master agents. Although a single master agent may be capable of servicing thousands of machines, most organizations will not want to blast the same patches to machines over already strained WAN links.

Machines can be directly managed and accessed through UpdateExpert's intuitive interface. Otherwise, administrators are stuck importing machines from a text file. If you have installed leaf agents on machines, those machines will appear in UpdateExpert's network pane. You then have the option of adding those machines to user-defined groups.

UpdateExpert offers the ability to place groups within groups so you can organize machines according to WAN boundaries or existing network directory setups. This was a good thought on the part of the software designers.

Finally, UpdateExpert is unique in allowing customers to use or not use agents. This is ideal for mixed environments. For example, if you have a Microsoft domain and also use NetWare servers, you could employ Novell's ZENworks for installing patches. But it would be much easier to configure UpdateExpert on both areas of your network.

HFNetChkPro 4.1

HFNetChkPro is built on HFNetChk, used in Microsoft Baseline Security Analyzer and Systems Management Server. Shavlik Technologies developed both HFNetChk and Microsoft's analyzer.

We installed HFNetChkPro on a workstation running Windows XP. Our test clients consisted of Windows XP and Windows 2000 workstations, as well as a Windows 2000 Advanced Server.

After verifying and installing Microsoft's XML components and .NET Framework, the program required us to reboot before installation would begin. A few minutes later, the installation was complete and no further reboot was required.

HFNetChkPro is a breeze to navigate and use. You are immediately presented with a split-pane control panel. On the left, you have access to all computer groups,

recent scans, a folder of your

favorite items and the patches that are currently available for applying.

To get started, we added a Windows XP workstation to a machine group and started a scan. In about 60 seconds, HFNetChkPro reported that the scanning was done and presented us with a high-level summary of the patches the machine was missing. A few moments after initiating patch deployment, results were returned that outlined major points of failure for the machine. We really liked this feature because of the time it could save on large networks. HFNetChkPro reported that our machine was ready to deploy patches.

Once we initiated the process, the HFNetChkPro console contacted Shavlik's patch server. This went fairly quickly, with the entire download process taking fewer than 10 minutes for about 56 patches. Once patch download was complete, installation began immediately. We appreciated that we were able to monitor the progress of the patches being deployed because it's often hard to tell if a patch deployment has failed or is just running slow. HFNetChkPro lists the start time and the last time activity on the machine was reported.

The program provides opportunities to import machines from Microsoft Active Directory or text files or to browse the network. Although we found the program perfectly suitable for small organizations, we question if it would gracefully scale to 450 or more machines.

We would like to see the product adopt a console/server model so it could be deployed on multiple servers across a large enterprise and controlled from a single console. Such a model would also alleviate bandwidth and time constraints that inevitably surface when trying to make a centralized product like this scale to a large enterprise. Hierarchical groups that mirror WAN boundaries would also be helpful in managing the huge number of machines in an enterprise.

If you have a small number of machines at a few locations around town, HFNetChkPro may be just what you are looking for. During testing, it performed well and provided valuable information on deployments and patches.

Greer is a network analyst at a large Texas state agency. Bishop operates Peoples

Information.com, an Internet consulting firm. They can be reached at Earl.Greer@dhs.state.tx.us.