Measuring the payback

No one can say exactly what a more effective search engine is worth to an agency, but everyone agrees that being able to find information beats not being able to find it. The problem is that it is impossible to quantify the cost of not finding something you may not even know exists.

Even the benefit of finding a desired document more quickly is hard to quantify because you can't guarantee the user will make productive use of the time saved.

Improved searches do yield clear results when they are applied to e-commerce sites, said Prabhakar Raghavan, chief scientist and vice president of emerging technologies for Verity Inc. For example, it's easy to determine how many Web site visitors actually make a purchase. If a search tool makes it easier for shoppers to find what they seek, that will translate into more sales. For federal agencies with sites visited by citizens, such improvements can boost user satisfaction, if not revenue.

The U.S. Postal Service uses Semio Corp. technology and tried to analyze its return on investment, but the slippery nature of the numbers produced an unsatisfactory result, said John Gregory, a marketing specialist with the Postal Service. "It is extremely difficult to analyze, and I'm not thrilled with the accuracy of the number," he said. Perhaps the most intangible benefit of effective search tools is alleviating the frustration of a fruitless search for something you know exists. Mike Frame, deputy director at the U.S. Geological Survey's Center for Biological Informatics, knows the feeling. "We had a budget document," he said. "I knew the author, but didn't know the title, and I couldn't find it."

Now the agency uses Hiawatha Island Software Co.'s Hi-Search metadata interface for its searches, and Frame is happier.

"I bet I spent, that day, 30 minutes looking for that document," he said. "If we'd had that interface, I'd have done it in 30 seconds."