Virtual servers, real threats

Server virtualization ' a technique of running multiple combinations of applications and operating systems on a single computer ' in many ways looks like a cure for a variety of information technology ills. Agencies can use virtualization to reduce the amount of processing power that often sits idle on today's high-performance servers. Virtualization also enables more applications to run on fewer computer boxes, so agencies can buy less hardware and fewer power and cooling systems to keep the hardware humming. In government, where getting approvals for new hardware can be difficult, fewer machines mean fewer procurement battles. 'With virtualization, it's easy to provision systems in a timely manner instead of waiting around for 60 days before you get in a new system,' said Michael Voss, systems engineer at consultant Booz Allen Hamilton and technical lead for a virtualization project at the Food and Drug Administration.So what's not to like? Some public-sector IT managers and consultants say there's a flip side to the virtualization excitement. 'People have been too busy enjoying the benefits to become paranoid about the potential security risks,' said Simon Crosby, chief technology officer at XenSource, a provider of open-source virtualization software.In some cases, the potential security vulnerabilities of virtualization mirror those of physical server environments. However, virtualization also presents unique challenges, both technical and operational, that necessitate extra security precautions.Rick Truitt, network architect for Delaware, said he is still getting used to swapping traditional security-related hardware, such as network switches, for the software-based switches used in virtual environments and maintained by systems administrators. 'I was always very protective, and I now feel like I've lost a little bit of control,' Truitt said. In a virtual environment, 'you've got to have a little more faith in your admin team.' Systems administrators should also get additional security training, he said. Vendors, including market leader VMware, say virtualization can offer a more secure server environment than those composed of traditional servers. 'The isolation characteristics provide very strong properties that help stop penetrations from a security perspective,' said Aileen Black, vice president of VMware's federal and public-sector operations. Isolation refers to virtualization designs that don't allow separate operating system and application combinations to communicate with one another, even when they run on the same hardware.    Deployments at defense and intelligence agencies and Level 2 certification under the Common Criteria Evaluation and Validation Scheme are proof of virtualization's security strengths, Black said. In late August, VMware announced that it was part of a contract to develop a High-Assurance Platform workstation for the National Security Agency.In a similar move, the Marine Corps is virtualizing servers in its enterprise infrastructure as part of an ongoing project that will eventually include mobile devices used in combat applications.'In general, virtualization is going to give us a tighter and more granular level of security' than is possible with physical machines, said Maj. Carl Brodhun, the Marines' project officer for enterprise virtualization. The ability to isolate applications and protect the host platform by using secure local-area network segments and software-based communications switches makes it easier to block threats, Brodhun said. 'If you have strong multilayer boundary security and intrusion detection in place, the vulnerability of a physical host is fairly low to begin with,' Brodhun said. 'In a virtual environment, it's almost nonexistent.'Nevertheless, some IT managers and consultants say agencies must be alert to new types of vulnerabilities. First, IT departments must protect the central management module, known as the hypervisor. It is a key component in leading virtualization platforms, such as those from VMware and XenSource. The hypervisor is a control panel for directing interactions among the operating system, applications and hardware resources on a virtual server. If malicious code breaches the hypervisor ' an event known as hyperjacking ' malware could easily spread among all of the virtual servers. 'This [hypervisor] layer needs to be kept up-to-date with security patches and protected from tampering,' said Neil MacDonald, a vice president at technology research firm Gartner. 'This software is in the most privileged position on the entire machine. The role the software plays in the consolidated server makes it an attractive target.'Hypervisor breaches have been rare in real-world implementations, but security researchers have shown the danger under controlled conditions. For example, security consultant Intelguardians has been studying virtualization security for the past two years under a grant from the Homeland Security Department. Its work has demonstrated that malicious code can move from one virtual machine to another, said Tom Liston, senior security consultant at the company.Malware isn't hypervisors' only threat. Limiting unauthorized access is imperative because of hypervisors' managerial role in virtual machines. Virtual environments lack the extra security layers of physical environments, where access control is handled by data center, network and Web site administrators, all of whom apply separate technologies and practices to secure the infrastructure. 'When you combine everything in a single box, you lose some of the separation you used to have by default,' MacDonald said. 'How you manage who has access, when they have it, how they check in and check out, and what audits are created all become critical.'Another security vulnerability occurs when virtual server sprawl develops. The relative ease of creating virtual servers means people with even a moderate level of technical proficiency can quickly deploy servers and potentially bypass controls for enforcing an agency's security policies. 'Someone might open up a [File Transfer Protocol] server to the outside world that hasn't been authorized, or they might establish a new Web site on the public Internet that is not properly secured or patched,' said Andi Mann, research director at Enterprise Management Associates, a technology consulting company. Similarly, technicians can easily create snapshots of virtual-server configurations for loading onto new hardware to balance workloads. When organizations build libraries with scores of images, they create security vulnerabilities, MacDonald said. 'Do you really have all the images patched?' he asked. 'Are you able to assess them for correct configurations? Do you know their genealogy, where they came come from and who modified them?  Some auditing capabilities are required, and none of that exists.' MacDonald said few alternatives exist for patching images beyond loading every one on a hardware server and applying the patch. 'Imagine that you have a thousand of these images,' he said. 'There will have to evolve a category of tools for off-line patch configuration management. A couple of companies are working on this, but nothing exists yet.'Blue Lane Technologies offers one possible solution. That company sells a network appliance that intercepts network traffic flowing into virtual servers and applies a temporary patch to shield the servers from known vulnerabilities until administrators perform a permanent software upgrade.  Sometimes the source of a vulnerability is not the virtualization technology; it is how agencies deploy the technology that creates the vulnerability. Agencies can create pools of processing capacity that can be dynamically allocated among virtual servers according to prevailing demand. The communal nature of those pools means a virtual server infected with a virus can easily spread the threat to the other virtual servers housed on separate computers. 'When you deploy a large number of identical servers together, the challenge is how do you keep them protected from each other,' said Andreas Antonopoulos, senior vice president at Nemertes Research. 'A single, self-propagating threat could theoretically infect all of the servers in a very short period of time ' and by short I mean seconds.'He added that collections of physical servers can fight those outbreaks with network-based compartmentalization. In that scenario, a series of firewalls throughout the communications pipeline shield database servers from a direct attack. 'Within the hypervisor software, there's a virtual network switch that allows any machine to talk to any other machine on the same host without any interference,' Antonopoulos said. 'At the moment, we don't have security mechanisms to enforce access controls between the various machines. And you can't physically put a firewall between them without breaking the pool into smaller pools, which makes the whole thing less flexible.'Despite those concerns, agency IT managers have tools to address many vulnerabilities as security technologies evolve. Voss said IT managers should adopt standard practices used by Unix technicians to control access to the hypervisor. They include segmenting the control console into a separate network, or virtual LAN (V-LAN) ' a technique Delaware managers use.Because virtual servers in the same piece of hardware can't communicate directly with one another, Delaware sends the traffic via the V-LAN, which applies the same antivirus and intrusion-detection controls as if the messages were coming from the Internet or other public network.Antonopoulos warned that, left unchecked, the sprouting of virtual network segments could result in what he called V-LAN spaghetti. 'When you have enough servers, you end up with many, many dozens of V-LANs, and you have to set up the rules for how each one can talk to the other,' he said. One possible answer to security challenges caused by the sprawl of virtual servers comes in the form of configuration management software from vendors such as Configuresoft, Tripwire, IBM and CA, Mann said. To protect against the spread of viruses, many administrators take a one-for-one approach by loading antivirus programs on each virtual server, said Gary Sabala, principal product manager at security software vendor Symantec. 'Customers are protecting [virtual servers] like they would a normal server,' Sabala said. 'For each virtualized partition, they load an individual copy of the antivirus product in every one of these virtualized environments.' Sabala said that approach, although protective, can impede overall virtual-server performance as application files and network traffic going in and out of the virtual environment are scanned for problem code. 'We are looking at a different model for future releases to figure out how do we put in some more lightweight agents in these virtualized environments and still have a virtual partition,' he said. .