Agencies seek security metrics

As if creating information security programs is not hard enough, most government

agencies are now realizing that they have no way to measure the effectiveness

of those programs.

The key, they say, is defining metrics. Across government, agencies are

trying to develop yardsticks against which to measure the success of their

programs. But while many agencies are trying to get a handle on these issues,

none are working together.

"There are lots of players out there, but there is no real rule book, and

there seems to be very little sharing," said Fran Nielsen, a computer scientist

at the National Institute of Standards and Technology's Computer Security

Division.

The NIST Computer System Security and Privacy Advisory Board (CSSPAB)

sponsored a workshop last week about security metrics, trying to determine

what solutions are available to federal agencies and what work needs to

be done.

The issue is multifaceted. Agencies need to figure out how to measure the

level of risk to a system — to know what security to put in place — the

security capability and awareness of employees, and the improvement from

one measurement to the next.

The biggest problem is determining what needs to be measured, workshop participants

agreed. "Measurement is fine, but measurement that does not link to action

does no good," said James Craft, information system security officer at

the U.S. Agency for International Development.

And an agency should not just perform measurements and find vulnerabilities

without measuring whether they are fixing those vulnerabilities and improving

their security, said Bill Hadesty, associate chief information officer for

cybersecurity at the Agriculture Department. "You've got to understand whether

you're solving the problem," he said.

It appears that agencies have a lot of tools with which to work. For

example, plenty of metrics exist for individual security products, including

the National Information Assurance Partnership's Common Criteria Evaluation,

since it is fairly easy to measure whether a product does what a vendor

claims.

But agencies have no clear way to measure the effectiveness of those products

when they are put together into a network. And it is even harder to measure

the effectiveness of security awareness and training programs, which aim

to reduce the number of vulnerabilities created by human error.

Meanwhile, a joint public/private- sector organization has developed

the Systems Security Engineering Capability Maturity Model (SSE-CMM), based

on the Carnegie Mellon University CMM system to measure the maturity of

an organization's processes.

Also, the CIO Council's Security Committee is developing an Information

Technology Security Assessment Framework based on the CMM system, the Office

of Management and Budget's Circular A-130 Appendix III and other federal

guidelines.

The General Accounting Office provides its Federal Information System

Controls Audit Manual to agencies and inspectors general to use as security

audit metrics. Internally, GAO auditors use a five-level system that measures

the effectiveness of agencies' security. The Defense Department is also

developing its own metrics system, the Information Assurance Readiness Assessment.

Still, the plethora of possible solutions leaves most agencies trying

to figure out which way to go.

A governmentwide standard could help everyone get on the same page,

said Franklin Reeder, chairman of the CSSPAB. Because of the relative immaturity

of this area, the board hopes to be able to foster the continued development

of these models and systems. "We hope to come out of this with the basis

for a conversation about what we do next," he said.