Fast responders

No one likes to talk about it, but criminals are using the Internet to extort money from companies, particularly those whose survival depends on processing financial transactions online. First, a company notices that its servers are under attack and online transactions with the public are blocked. Then an e-mail arrives explaining that the attack will stop only if the company pays an extortion fee.

Such attacks are an example of the growing sophistication and targeted nature of computer security incidents that afflict some businesses and government agencies. Reporting and responding to such incidents demand significant attention and resources. Companies that are models for dealing with security vulnerabilities provide training to make their employees security-aware. But increasingly, they rely on the quick response of automated detection and remediation systems to protect information on their networks.

Security officials at some of the largest companies say incident reporting is still more of an art than a science. But security officials at three corporations -- AT&T, Booz Allen Hamilton and Northrop Grumman -- agreed to discuss a topic that others said they would rather not talk about. Several experts in the information security business also offered their advice on incident reporting. Those officials and other experts said their experience might be helpful to federal officials who must not only protect government information but also comply with the Federal Information Security Management Act.

FISMA requires federal agencies to report incident data to two agencies with different reporting needs: the Office of Management and Budget and the Homeland Security Department. That is a tall order for many agencies, said Kenneth Ammon, president of MCI NetSec Global Security Services, an MCI company.

"You have two different audiences that you're trying to please here, and you probably need two different approaches to satisfy the requirements," he said.

OMB, which monitors FISMA compliance, asks agencies to report the number and type of security incidents they had in the previous year. Critics say the requirement fails to recognize that some agencies detect thousands of security incidents because they have rigorous security monitoring programs, whereas other agencies do not.

"A department that isn't looking can say we have zero incidents to report, and a department that is looking has a lot," Ammon said.

Recognizing this problem, OMB worked with DHS this year to develop a more sophisticated security incident reporting template for agencies' 2005 FISMA reports, said Karen Evans, OMB's administrator of e-government and information technology. OMB will ask agencies to verify that they followed the new DHS guidelines when they report their latest security incident numbers.

Unlike OMB, DHS has always been interested in collecting real-time technical data that could provide an early warning of emerging threats to federal networks and information systems. Ammon said DHS could discover more threats by placing anomaly- detection devices on agencies' wide-area networks.

But because most agencies have resisted its efforts to put data-collection devices on their networks, DHS continues to have problems getting useful incident data.

Ammon said DHS should offer to subsidize agencies' purchase of such devices from a short list of approved vendors in exchange for access to the incident data from those devices. Most agencies would accept that approach, he said.

As for agencies' other concerns about nondisclosure, Ammon said, the Commerce Department has figured out how to protect and aggregate economic data from U.S. businesses, and DHS could do the same with agencies' security incident data.

Hit daily by hundreds of potential security incidents, federal agencies and large companies face a major challenge in identifying incidents that require further investigation, a process that security experts call a root-cause analysis.

"It's all about sifting through tons and tons of hay to find a few needles that might be in there," said Ed Amoroso, chief information security officer at AT&T.

AT&T collects security data from its corporate firewalls, intrusion-detection systems, servers, desktop computers and databases. "My security team monitors just about everything," Amoroso said. He added that the process has become much less arduous since AT&T added firewalls, intrusion-detection tools and antivirus protection at various Internet access points where AT&T's portion of the nationwide IP backbone connects with portions owned by MCI and other carriers.

By detecting and filtering problems on the public Internet before they reach AT&T's corporate network, Amoroso said, the daily workload of security incidents that his internal staff investigates is down to about 40.

Amoroso said government agencies should demand the same kind of service. Instead, most struggle to secure their networks against threats from the Internet, he said. No agency would avoid asking the electric company to help solve power spikes or other safety hazards, and yet that's the current situation in the telecommunications industry, he said, adding "We're just saying we can help."

AT&T has packaged that help in a service it calls AT&T Internet Protect, but so far few large agencies have signed up. Buying managed security services from AT&T and other carriers might take some time to catch on, if it ever does, said Timothy McKnight, chief information security officer at Northrop Grumman. "There's a lot of value there, and I agree they should bring it to the table," he said. The greatest value of such services would most likely be for small and midsize agencies or businesses, he added.

As threats increase and new regulations require compliance, companies and agencies are adopting more structured approaches to security incident reporting. "Many regulations specifically say you need to have a methodology to identify an incident and procedures to handle it," said Tracy Hulver, senior director of product management at netForensics.

The regulations, however, are often vague about what those standard procedures should be, he said.

Agencies and companies need "a chain of command, an escalation process and some sort of corporate governance as to when [they need] to call the authorities," said Ron Gula, president and chief technical officer at Tenable Network Security.

John Pescatore, vice president of Internet security research at Gartner, said he often refers the firm's corporate clients to a computer security incident guide, called "Special Publication 800-61," published by the National Institute of Standards and Technology in 2004. He also recommends a guide developed by the Australian Computer Emergency Response Team.

After initially struggling to create definitions, such as determining what a security incident is, Booz Allen officials set up standardized procedures for identifying and responding to incidents. They based those procedures on a process framework called the IT Infrastructure Library, which originated in the United Kingdom.

Daniel Gasparro, a senior director of operations at Booz Allen, said the company's implementation of that framework ensures an appropriate incident management response to whatever hits the corporate network.

Using that framework, Booz Allen's IT staff contained a coordinated denial-of-service attack that recently targeted the company. "Those are the ones you always get concerned about," Gasparro said. But with a detailed response plan that included having certain scripts ready to run, the company prevented a major corporate network outage, he said.

At Northrop Grumman, information security officials made several recent procedural changes to improve security incident reporting. They set up a toll-free number and single-purpose e-mail address for reporting incidents. They created the Computer Security Incident Response Team of business managers, corporate communications officials and security experts, who follow a documented plan when they respond to security incidents.

"We can't have every executive calling the line and telling them what they want them to do," McKnight said. The team members train and practice their response as if they were conducting a fire drill, he said.

The focus on security incidents and the creation of security operations centers are fairly recent corporate activities. Unlike network operations centers, which have been around for a while and operate according to well-defined procedures, security operations centers are still maturing, Hulver said. Security center "operators are more apt to say, 'Gee, something weird is happening. Let me go dissect what's going on,'" he said. But because businesses and agencies often have both types of centers, he added, it is important that they have standard operating procedures for communicating and passing tasks to one another.

Effective sharing of security incident information has largely been an elusive goal for many companies, just as it has been for DHS. "Aggregation is a powerful thing," especially when aggregated data reveals patterns of activity, said Mike Caudill, incident manager of Cisco Systems' Product Security Incident Response Team. Sharing incident information "can help minimize the impact of an incident or put a stop to an incident."

But most companies have been reluctant to share incident information with other companies or the U.S. government. A group of private-sector information sharing and analysis centers set up to share security incident information within different industry sectors and with DHS have been failures, with one exception, Pescatore said. Because DHS does not share incident information with the centers, he said, "the benefit back to them does not exceed the risk they perceive in making that information available."

The one exception is the financial center, which is working, Pescatore said. But managed security providers have some of the most valuable collections of security incident data. Companies such as Counterpane Internet Security, VeriSign and Symantec manage thousands of firewalls for hundreds of corporate customers, he said.

With that managed security data, a company can learn, for example, whether it is the target of an attack or simply a random casualty of a mass attack. "That's a mechanism where we've seen information sharing work pretty well," Pescatore said. The danger of targeted or customized attacks is that the hackers will create a Trojan horse to harm a specific company, he said.

Security companies usually don't respond by writing an antivirus signature if a virus attacks a single company, Pescatore said, adding that "the rise of targeted attacks has poked big holes in a lot of companies' intrusion-detection strategies."

Corporate security officials say companies and agencies should spend what they can afford to automate their security incident reporting. Security reporting works best "if data is being collected from everywhere in real time," Amoroso said. "What industry and AT&T are trying to do is automate as much as possible because the social process and the human interaction around [incident reporting] will always be very imperfect."

Having real-time incident-detection capabilities also produces a desired social response, Amoroso said.

"If you have powerful tools that are collecting data and you're very successful at detecting even minor changes in the infrastructure, people are going to be very careful," he said, adding that they will think twice about attempting to sabotage a corporate network.