Are there perils in penetration testing?

The New York State Office of Cyber Security and Critical Infrastructure labored for several years to protect its networks and information technology assets with layers of security. It tightened existing IT policies, procedures and practices and instituted new ones, conducted gap analyses and established plans for mitigating security breaches.

Since then, the state has conducted regular scans to detect IT vulnerabilities — the equivalent of looking for frayed threads that can unravel and leave an agency’s networks and systems exposed.

The state will soon throw another protective blanket onto the pile. For the first time, it will try to hack into its own systems. For IT professionals, penetration testing is the ultimate security measure.

Penetration testing goes beyond tapping at the door, said William Pelgrin, chief cybersecurity officer of New York’s Cyber Security and Critical Infrastructure Coordination. “It’s breaking through the door.”

New York officials haven’t decided who they will let try to break through the door. Until recently, penetration testing was the exclusive purview of highly skilled technicians who employed extensive toolkits of specialized programs to probe and exploit system and network weaknesses. A deep-dive penetration test could take weeks to complete and cost hundreds of thousands of dollars.

But new automated tools promise to do the job more quickly and at less expense. Among those is Core Security Technologies’ Core Impact, which the company says can run a meaningful penetration test in a few hours. The company charges an annual fee of $25,000 for an unlimited-use license. State and local chief information officers and chief information security officers say they must evaluate the pros and cons of the new software options, including open-source applications such as Metasploit.

A growth industry
So when does it make sense to use an automated penetration test?

Advocates of the new tools say the applications give in-house security professionals more control, including the ability to perform penetration tests as often as they want. Critics of automated tools say they are a poor substitute for a thorough and nuanced manual test that a skilled practitioner performs. Most experts agree, however, that an automated penetration test in the hands of an untrained novice could do more harm than good.

“A fool with a tool is still a fool,” said Bill Harrod, a security management consultant at CA, formerly Computer Associates.

Penetration testing and other types of IT security assessments are a growing industry. In a world made increasingly unsafe by identity theft, online rip-offs and other cybercrimes, IT security pros are under pressure to fortify systems and networks and protect information assets. A growing number of federal regulations require public- and private-sector CIOs to harden their systems and networks against external and internal threats.

Vulnerabilities are increasing exponentially. The number reached 5,990 in 2005, according to Carnegie Mellon University’s Computer Emergency Response Team Coordination Center, an increase from 171 vulnerabilities reported a decade earlier.

First line of defense
Every vulnerability represents a potential security risk. Penetration testing determines whether the risk can be exploited. If a system’s owners can break in, an unauthorized hacker can, too. 

“Either you, the owner, can find them, or the hacker can find them, but they will be found,” said John Carpenter, a product manager at DevPartner SecurityChecker, an automated application security test Compuware developed.

A vulnerability scan is a first line of defense. It detects missing security patches and spots other vulnerabilities that could potentially compromise networks or systems. But scans often produce reams of data that show false positives and identify vulnerabilities that, for technical or practical reasons, no one could exploit anyway.

Penetration tests pick up where vulnerability scans leave off. Both manual and automated penetration tests try to exploit network vulnerabilities to determine whether they afford an opportunity for hackers to take over computer systems, gain access to private data or disrupt networks.

Armed with a Web browser and a proxy device, Brad MacKenzie, director of IBM’s X Force Penetration Test Team, recalled hacking into the network of a state prison system and gaining access to detailed records of current and former prisoners.

“It was as if we were sitting in a branch office” of the state’s prison system, MacKenzie said.

South Carolina’s strategy
Allowing a consultant to hack into your system and see sensitive data in the name of security makes no sense to James MacDougall, chief information security officer of South Carolina, which uses Core Impact for in-house penetration testing.

“We have outsourced some penetration testing, but I didn’t think it was wise to outsource the testing of critical infrastructure to a vendor,” MacDougall said. “We thought we should arm ourselves.”

Government procurement rules requiring agencies to select low-bid vendors can undermine the confidence of users, including law-enforcement agencies that “don’t want their data outside on the street,” MacDougall said.

But detractors of automated penetration testing say the applications can give agencies a false sense of security. In reality, the tests are only as effective as the people running them. The best tools are unable to discern the intent of hackers or reliably prioritize vulnerabilities.

Officials must remember, MacKenzie said, that an automated tool is always a step behind the elite attackers’ exploits.

But not far behind, said Max Caceres, director of product management at Core Security Technologies, which develops new commercial-grade exploits that mimic what the bad guys do. “We continually update the product with new attacks every week,” Caceres said.

Delaware’s approach
Delaware’s Department of Technology and Information has been using Core Impact for about a year as part of an overall IT security strategy that included a thorough assessment performed by an independent auditor.

The auditor completed technical scans and reviewed policies and physical controls, including firewalls and routers. The auditor also scrutinized application development practices.

The review also benchmarked the state’s security program, which complies with International Organization for Standardization 17799, a detailed international standard for managing information security.

Networks evolve constantly and automated penetration testing “is a way to do the maintenance in between the full-blown assessments,” said Elayne Starkey, Delaware Department of Technology and Information’s chief security officer.

However, Starkey said there are limitations to automated penetration tests.
“Don’t think of a tool like this as a silver bullet,” she said.

But as part of an overall security program, automated penetration tests can add value if handled properly, said Yong-Gon Chon, senior vice president of services at SecureInfo.

“My biggest fear is that you end up with an untrained new security engineer with a few grand in the security budget who buys a tool that they run, and it has a massive impact throughout the infrastructure,” Chon said.