IT dashboards: Picture perfect?

For many years, agencies have been tasked with measuring the performance and management of their technology programs, usually under the watchful eye of Congress and the White House. With agencies spending about $82 billion a year on IT, transparency is seen as a powerful motivator for keeping IT programs on track.

Over the years, report cards, watch lists and high-risk lists have been used to draw attention to systems that were over budget and behind schedule. In this era of open government, many of those tools have been superseded by online dashboards that display data in a visually compelling way to show how an organization or program is meeting specified performance goals.

Dashboards are proving to be useful tools in the campaign for better IT management. The most visible example is the IT Dashboard, launched by the Office of Management and Budget in 2009 to improve the oversight, transparency and accountability of federal IT programs.

The IT Dashboard offers a view into how agencies are managing 700 major IT programs that are worth a total of $40 billion. It tracks the progress of those programs by synthesizing and graphing data from Exhibits 53 and 300 (reports to OMB that detail agency IT investments and performance plans) along with other agency updates and investment information. If a project is running behind schedule or over budget, the dashboard will display that information, along with a color-coded risk rating on a spectrum from green for low risk to red for high risk.

OMB has added other data streams since launching the site to provide more agency-specific data or deeper views into trends or context, said an OMB official who spoke on background. The dashboard also offers insight into how well agencies are meeting Digital Government Strategy milestones and complying with PortfolioStat reviews, which detail agencies' processes for managing their IT portfolios and highlight duplicative programs that are ripe for consolidation.

However, the OMB official said the dashboard is more than a collection of data. It also helps improve IT management by increasing the quality of agency performance data, improving transparency and visibility regarding the health of IT investments, and increasing agencies' focus on project management and IT governance, according to the official.

The PortfolioStat process is getting an upgrade as well. U.S. CIO Steven VanRoekel announced PortfolioStat 2.0 in a memo and accompanying blog post in March. The upgraded process streamlines agencies' data-collection requirements, reducing 30 reports down to three. VanRoekel wrote that it "improves analytics, consolidates the agency's strategic IT direction and management improvements into one central plan, and holds agencies accountable for the goals set through last year's process." The first reports — agencies' Information Resource Management Strategic Plans — were due to OMB on May 15.

Transparency helps

The IT Dashboard offers more insight into why a program is struggling than the watch lists of the past, which were incomplete and offered little transparency, said David Powner, director of IT management issues at the Government Accountability Office.

"The dashboard has transparency, and transparency doesn't hurt," he said.

In addition to making OMB's and Congress' IT oversight duties easier, the detailed data has bolstered the authority of agency CIOs, who are evaluated on their IT management success.

"There were some CIOs who, when the dashboard was rolled out, didn't have access to the data they needed" to send to OMB because they weren't involved in the IT investment process, Powner said. That had to change, which meant giving CIOs the appropriate authority and streamlining internal processes.

Nevertheless, Powner said there is room for improving the IT Dashboard. Indeed, many experts have criticized the accuracy and completeness of the data on which the dashboard is built. "The chatter is that the data is not very good, but it's not an excuse not to make it good," he said. "We think agencies need to keep working on it."

For example, Powner testified at a House hearing in January that the Defense Department had reported that it had no high-risk IT programs when in fact it had many. The GAO report on which Powner's testimony was based warned that "DOD is masking significant investment risks...and has not delivered the transparency intended by the dashboard."

"We want to continue to push for transparency and better mechanisms," he said. "The dashboard, although far from perfect, is one of the better models because it allows for comparisons. It's not only what is spent, but are we getting the right return on investment and who is accountable."

Larry Allen, president of Allen Federal Business Partners, said accuracy is always an issue when data is pulled from multiple sources, something the Federal Procurement Data System amply demonstrated. "You end up with an unclear picture," he said. Agencies across government collect and report their procurement data to FPDS, but it has been criticized in the past for containing inaccurate or incomplete information.

To improve the IT Dashboard's accuracy and value, OMB has added submission validations and dynamic data quality reports. Those tools help agencies correct their submissions or address issues by flagging potentially inaccurate information, such as unrealistically high or low values for certain data fields, according to OMB.

What dashboards can't do

For a dashboard to have real value, the garbage in/garbage out rule applies to more than the actual data provided, said Roger Baker, former CIO at the Department of Veterans Affairs. It determines what data you collect in the first place.

He said agency leaders need to ask themselves what they are measuring, "and is it something that is really essential to your business? The dashboard is of use when it is targeting information that you can use to manage your business."

VA, which hosts multiple IT dashboards, has a program management and accountability dashboard that tracks whether each IT project is meeting its milestones and aggregates the relevant data into a single view. "At VA, it's an important piece of how we manage because it tells us the status of our IT portfolio," said Baker, who became chief strategy officer at Agilex in April.

That data also feeds into the federal IT Dashboard, which Baker said has done a decent job of shining a light on problem programs. "The IT Dashboard has not by itself fixed the entire problem, but it is motivating agencies to make changes, which means that they are managing better, more tightly," he added. "If an agency receives a red grade, then they know someone will come and ask questions."

Stan Soloway, president and CEO of the Professional Services Council, said that although dashboards can be useful, they can't do all the work for agencies.

A dashboard won't help leaders make better management decisions if the information is not timely or the business rules are weak. "And knowing that the program is not going according to plan isn't enough," he said. "You need to know why."

In other words, a dashboard "doesn't eliminate the need for real analysis," Soloway said. "It's a good tool to help you find it, but you have to dig deeper."

Allen said a good IT dashboard requires a certain amount of central management, which can be tough for agencies to do. But overall, "I think it's more effective to have dashboards than not," he said. "It's better than flying totally in the dark. But there is a difference between a dashboard and a GPS."

Health IT Dashboard: Insight into industry improvements

The Department of Health and Human Services officially launched the first version of its Health IT Dashboard about a year ago to offer a one-stop shop for people interested in health IT and the impact it is having on the health care industry.

It lets the interested public know the impact and benefits of the American Recovery and Reinvestment Act of 2009 and delivers greater insight into the activities of grant recipients, state and federal agencies, and other stakeholders, said Victor Lazzaro, project director for the Health IT Dashboard at HHS' Office of the National Coordinator for Health IT.

Among its offerings, the dashboard allows users to generate maps of health IT adoption statistics, such as how many providers are using electronic health records and what states are offering courses in health IT through their community colleges. It also provides statistics on how grant money is being spent nationwide and how well states are meeting meaningful-use goals for EHRs.

More than just a place to find data, the dashboard is speeding the adoption and use of health IT, Lazzaro said. "We're not just summarizing data," he added. "We think people can look at the data and use it to amplify and accelerate their own efforts."

Most of the data is updated monthly, and the dashboard is built on open-source software so it is cheaper to maintain, Lazzaro said. In the next six to 12 months, he plans to add more datasets and enable better visual comparisons.

GWAC Dashboards: Business intelligence for agencies

The General Services Administration's GWAC Dashboards allow agencies and vendors to view information on governmentwide acquisition contracts, including task-order obligations, number of awards and the vendors that have been issued task orders by each agency. That information was hard to get before the dashboard was launched and in some cases required a Freedom of Information Act request.

The GWAC Dashboards, which are updated nightly, help vendors "better manage their contracts, leverage data to form business intelligence about their target accounts, and develop marketing plans," according to a GSA spokeswoman.

They can also help agencies conduct market research, evaluate task-order activity and vendor capability, and review small-business activity.

In the near future, GSA plans to improve download times and enhance the dashboards with more datasets and side-by-side chart displays.