Measuring agency performance: Out of reach

It is clear to Dean Mesterharm, the Social Security Administration's deputy

commissioner for systems, that investing in information technology is essential

for SSA.

To improve delivery of $416 billion in benefits to 51 million

Americans each year, SSA spent more than $1 billion to put a PC on every

worker's desk. Now employees have at their fingertips reference manuals,

electronic claim forms, decision support software, a case control system,

quality assurance and other information. Automation enables SSA to serve

an ever-increasing number of Americans with 25 percent fewer employees than

it had in 1985.

"That couldn't happen without automation," Mesterharm said.

But trying to prove it is another matter.

"There's not a simple mathematical equation that compares before and

after," Mesterharm said. "It's complex because a lot more things are changing

than just the technology part." Processes, laws and workloads, for example,

are in a state of flux, he said.

Like other agencies, SSA has sent Congress its first performance report

required by the Government Performance and Results Act. And like other agencies

that struggled to meet the March 31 deadline, SSA has had a hard time providing

a solid cost-benefit analysis for its spending on information technology.

GPRA reports are supposed to provide taxpayers a businesslike accounting

of how agencies spend money on programs, including IT, and what the spending

yields.

But senior government officials, including agency chief information

officers, concede that the first report cards generally fall short of providing

the clear and useful picture of spending and results envisioned in 1993

when Congress passed GPRA.

"No one should pretend that we don't have a very long way to go," said

Joshua Gotbaum, controller and associate director of the Office of Management

and Budget.

IT has proven to be one of the hardest functions of agency operations

to fit into a corporate accounting archetype. Together, government agencies

spend more than $30 billion a year buying and maintaining computer systems

and software. But senior agency executives say they are often unable to

pinpoint — unable sometimes to even estimate — the value derived from that

spending.

"It's a fair question" when Congress asks agencies to quantify IT spending

results, said Daryl White, chief information officer at the Interior Department

and co-chairman of the CIO Council's committee on capital planning and IT

management. In exchange for handing agencies billions of dollars, lawmakers

are right to want to know what the money is buying. "We can't just tell

them they're getting policy," he said.

But enumerating the returns for IT investments "has been troublesome

for CIOs," White said. "And it will continue to be so until we can decide

on a common metric." The problem is that neither Congress nor anyone else

has devised a workable way to perform reliable cost-benefit analysis for

government spending on IT.

The Problem of What to Measure

With GPRA, Congress intended to make agencies operate more like businesses,

OMB's Gotbaum said. "It forces us to focus on the bottom line." That's easy

in business, where the bottom line is "either you do or don't make a profit,"

Gotbaum said. "Government is much messier."

No agencies exist primarily to make money, he said. They exist for national

defense, to help the elderly and the poor, to protect the environment and

for dozens of other purposes. But the lack of a single, simple measure such

as profitability makes it more difficult to gauge performance.

For some agencies, performance can be measured by the number of people

served, the number of crimes solved, improvements in health or the decreases

in complaints. But assessing the value of the various contributors to those

achievements is problematic.

No one would question that the Environmental Protection Agency had made

a valuable contribution if its policies reduced the number of deaths due

to pollution. But it would be difficult to attach a precise dollar amount

to those accomplishments.

It would be even more difficult to calculate the contribution IT spending

made to those outcomes, said Paul Wohlleben, former deputy CIO at the EPA.

Yet information systems are woven throughout EPA activities. Without them,

the agency would be crippled.

Even when technology is tied directly to improved agency performance,

it can be difficult to weigh costs against benefits, said Patrick Plunkett,

a program analyst for the General Services Administration's Office of Governmentwide Policy. The National Weather Service, for example, has won praise

for providing faster warnings of tornadoes, thunderstorms and hurricanes.

Better radar, supercomputers and other advanced technology made better forecasting

possible. But the agency has not made a clear assessment of the value of

IT's contribution to that outcome, he said.

Often, the best measure may not be dollars, Plunkett said. "Congress wants

to see that the money they spend on information technology has some kind

of return." Showing that service has improved, taxpayer satisfaction has

increased or costs have been avoided may be sufficient, he said.

Trying to Solve Computer Chaos

If the accounting Congress wants is difficult, the reason for it is

simple.

GPRA was passed when polls showed that the public believed the government

wasted 48 cents of every tax dollar it collected. Information technology

was one of the notorious fiscal black holes.

Major federal agencies, from the Internal Revenue Service to the Federal

Aviation Administration, and programs such as Social Security and Medicare

were haunted by computer horror stories.

In 1994, then-Sen. William Cohen (R-Maine) released a report, "Computer

Chaos: Billions Wasted Buying Federal Computer Systems." It attacked agencies

for failing to perform better cost-benefit analyses on their computer programs

and led to passage of the Clinger-Cohen Act, which requires such analyses.

But the act lacked deadlines and penalties and provided no money or

strategy for agencies to do a better job of developing and maintaining IT,

said Joan Steyaert, deputy associate administrator of GSA's Office of Information

Technology.

By 1996, cost overruns on government computer projects had exceeded

$200 billion, Steyaert said. Deadlines regularly slipped, programs were

unstable and the worth of many projects was questionable, she said.

One of the biggest problems — then and now — is a lack of data on true costs,

true benefits and program risks, said David McClure, an IT analyst at the

General Accounting Office. "There's no baseline," and it will take years

to collect such data, he said.

The dearth of data makes it harder for agencies to develop sound IT

performance plans and reliable performance reports. And the deficiency has

been noted on Capitol Hill.

A review last summer of GPRA performance plans submitted by the 24 largest

federal agencies showed that many did not include performance goals for

IT, said an aide to Sen. Fred Thompson (R-Tenn.).

The Customs Service, despite planning a $1.2 billion computer modernization

project, did not write IT performance goals, the aide said. Neither did

the Agriculture Department's Bureau of Land Management for its automated

land and mineral records system project, which was under development for

15 years and had a budget that nearly doubled from its original $240 million

price tag before BLM killed it last year.

GAO in March lodged similar criticism against the Department of Housing

and Urban Development, the Department of Veterans Affairs, the National

Weather Service and SSA.

"From our perspective, information technology is a major challenge for

every federal agency," Thompson's aide said. Thompson, chairman of the Senate

Governmental Affairs Committee, is a bulldog on government computer reform.

Success Stories

But there have been some GPRA successes as well.

Gotbaum cites the Health Care Financing Administration as having had

some success in trying to improve IT development. The agency is recovering

from a computer disaster by following GPRA guidelines.

In 1994, HCFA began to work on a massive $151 million Medicare Transaction

System that was supposed to take over the work of nine older systems and

process $280 billion worth of Medicare bills each year. But by 1997, the

agency had spent $43 million and was farther than ever from owning a workable

Medicare billing system. Far behind schedule, the estimated cost had ballooned

in price to nearly $1 billion. All the agency could do was pull the plug.

"HCFA's computers were a mess," Gotbaum told the House Rules Committee March

22. The agency regrouped and submitted a new modernization plan to OMB,

but it, too, was unworkable. HCFA had failed to follow the basic guidelines

of GPRA, he said.

"We went back to first principles," Gotbaum said. HCFA set IT performance

goals and then developed an IT architecture around them. Last year, OMB

sent HCFA's plan to Congress, which approved funding this year and is expected

to approve funding for 2001, Gotbaum said.

GAO also has been trying to impose GPRA-style discipline on agency IT

projects. Its most significant action so far has been to correct "serious

management and technical weaknesses with the IRS' multibillion-dollar effort

to modernize its tax systems," GAO reported in March.

In 1996, Congress cut off most of the funding for the IRS' Tax System

Modernization program, charging that the agency had little to show for the

$3.3 billion it already had spent on system upgrades. The IRS then developed

a new multibillion-dollar modernization project called the Prime Systems

Integration Services Contract, in which it plans to follow more businesslike

IT development processes.

At GAO's direction, the IRS established a facility to test new computer

systems before installing them. The tax agency also agreed to hire only

software contractors with "mature software development capabilities."

GAO last year persuaded the IRS to adopt an incremental approach to

the multiyear IT modernization program, trimming the IRS' budget request

from $500 million to $323 million.

'Slower Than a Turtle's Crawl'

If figuring out how to comply with GPRA has been frustrating for agency

managers, waiting for more businesslike performance from agencies has been

equally frustrating for Congress.

"The government has made progress toward this goal, but at a pace slower

than a turtle's crawl," said Rep. Stephen Horn (R-Calif.). "So far, only

a handful of agencies have had limited success" at linking their program

plans to their budget requests, as GPRA requires, Horn said. That linkage

"is supposed to form the basis of their budget requests," he said.

Unresponsive agencies aren't the only problem, said Horn, who is chairman

of the House Government Management, Information and Technology Subcommittee.

"Widespread, enthusiastic leadership is missing at all levels of government,

from Congress and the OMB to each federal department and agency."

"It irks me not to see a lot of people show up at these meetings," he

groused during a Rules Committee hearing on GPRA March 22. Only four of

13 committee members attended, and only Rep. John Linder (R-Ga.) stayed

for the whole hearing.

Linder said he may propose a change in House rules that would force

committees to pay closer attention to agency compliance with GPRA during

budget deliberations.

Closer oversight by Congress would help, agreed GAO comptroller David

Walker. And tying GPRA compliance to budgets would be a good way to make

agencies try harder to develop useful performance goals and measurements,

Walker said.

Better oversight could begin this spring. The incoming GPRA performance

reports should be adequate enough to give Congress some insight into agency

goals and performance in time for budget hearings.

Among the items lawmakers should look for in the reports is "the status

of the agency's efforts to use information technology to achieve results,"

Walker said.

Going Beyond 'Gotcha'

Agency officials familiar with the politics of the annual budget exercise

remain wary of the direction GPRA-based budget hearings might take.

"Performance measurements are supposed to tell how well programs were

doing," said GSA's Plunkett, but "the Washington environment is about "gotcha.'"

There is widespread concern among agency officials that when performance

reports reveal deficiencies, they will be used to hammer agency heads and

be turned into ammunition in budget battles and wider political brawls.

"Agencies are afraid to be too honest," he said. "This is a political

environment, and decisions are made on a political basis."

Gotbaum has the same concern. "When an agency makes a good faith effort

but doesn't meet their goal, or if the measures need to be revised, we need

to be careful," he told the Rules Committee. If reporting poor performance

leads to public criticism, agencies will conclude that the reward for honest

analysis will be punishment. "If agencies come to that conclusion, GPRA

will never work," he said.

Gotbaum urged Congress to use "a combination of public encouragement

and private criticism" to elicit honest assessments of program costs and

payoffs from agencies.

When GPRA was passed, former OMB Director Franklin Raines foresaw the

problem in slightly different terms. If agencies' budgets are based on their

performance reports, he said, the only performance reported will be good

performance.

If agencies are to actually perform better, GPRA has to be more than

a report, said Interior's White. Agencies have to change the way they work.

"You need a cultural change," he said.

Agencies have traditionally measured success by the amount of money

they have spent on programs, the number of staff members they employ and

the number of tasks they have completed, GAO's Walker said. Such measures

miss the point. Agencies should be measuring whether their programs contribute

to the quality and security of citizens, he said.

GPRA was intended to force a fundamental shift in the way agencies think

about carrying out their missions. Instead of focusing on "activities,"

agencies are supposed to focus on "outcomes."

For IT managers, that means no longer calling it a success when new

computer systems are installed and IT budgets are fully spent. Success must

be measured by what IT contributes to the agency's mission.

But the cultural change needs to go further still, White said. "It is

laudable that someone wants to hold the government accountable for the funding

given to us by the taxpayers, but if they're going to hold me accountable,

they need to give me the authority and resources to succeed."

Although Interior's budget includes about $500 million for IT, White

said much of it is outside his control. Instead, much of his responsibility

is for "support functions" — such as ensuring that the agency payroll system

works — and not for functions that directly affect the department's primary

mission of managing national resources, he said.

"Chief information officers and chief financial officers are not getting

sufficient support from the people they report to," said Jerome Climer,

president of the Congressional Institute. "You're not seeing the political

appointees asking a lot of performance results questions yet."

The task agencies face in developing meaningful IT analyses remains

a monumental one. But one, OMB's Gotbaum believes, that is doable as agencies

learn what is expected. "These changes are neither quick nor easy," Gotbaum

said. "They take time.

"The vision created by GPRA is a challenging one, but it is important

and, with effort, achievable," he said. "Government, like everyone else,

learns by doing."