'Hits' don't measure up

"Performance measures" is a popular phrase these days, but agencies are slow to apply performance measures in a systematic way.

Take agency Web sites, for example. The primary performance measures that many agencies use for their Web sites seem to be more hits, more hits and more hits.

But what does "more hits" measure? Simply that a lot of people out there clicked on an Internet site. That kind of measure tells the agency exactly nothing about whether the site is achieving its performance goals. For all the agency knows, visitors could be clicking because they are furious at the agency.

Perhaps a better question is: What are the performance goals for agency Web sites? And are the measures that agencies use telling them whether the sites accomplish their intended jobs?

Some agencies have given a lot of thought to those questions. Charles McClure of Florida State University, Kristin Eschenfelder of the University of Wisconsin and I recently completed a study for three agencies in which we developed a set of performance measures for assessing agency Web sites. The three sponsoring agencies were the Defense Technical Information Center, the Energy Information Administration and the Government Printing Office.

We found that the topic of Web site performance measures has at least three dimensions when applied to government agencies. The first is that of law and policy. Feds must be careful that they are complying with legal and policy mandates regarding privacy (including children's privacy), freedom of information, intellectual property, accessibility for people with disabilities and security. They must be sure that their Web sites conform to relevant parts of the Paperwork Reduction Act, the Government Paperwork Elimination Act and the Federal Records Act.

Management factors are the second dimension for Web site performance. Does a site have adequate infrastructure, including equipment and staffing, and is the agency controlling who is posting content to the site and ensuring the quality of the content?

Third, quantitative performance measures fall into a fivefold conceptual grouping: extensiveness, efficiency, effectiveness, service quality and usefulness. Although each concept has specific measures associated with it, they overlap in practice.

One tip all study participants agreed on was that counting Web site "hits" is a lousy measure of anything. Much better basic measures are units of content retrieved — such as number of documents downloaded — or number of user sessions per time period.

Check out the study report at fedbbs.access.gpo.gov/libs/measures.htm. The sponsoring agencies and the authors will be happy if the study just gets more agencies thinking about what they are trying to achieve with their Web sites and talking about what are the best ways of measuring their goals.

Sprehe is president of Sprehe Information Management Associates in Washington, D.C. He can be reached at jtsprehe@jtsprehe.com.