Can Metrics Capture the Value of 'Innovation'?

Government chief information officers struggle to quantify nontechnical factors such as "organizational health" and "innovation," a new report argues.  

After interviewing 27 government CIOs, Arizona State University Associate Dean Kevin Desouza concluded that most "have a very good inventory of metrics" for technical performance -- such as server downtime or application usage -- but are less sure how to measure performance characteristics of the organization itself.  

"When I asked them the question, 'how do you measure innovation,' many of them drew blanks," Desouza told Nextgov. "But my follow-up question was, 'can you give me a case when you really thought the IT department was innovative?' When you ask that question, they give you concrete examples . . . [about] how it saved time and money."

CIOs rarely attempted to quantify the results demonstrated in those anecdotes, Desouza found.

"I want to push the CIOs and say, 'you're doing very innovative things, but if you don't capture the metrics . . . the only thing that's going to be in the news is every time you fail.'"

The report was issued by IBM's Center for the Business of Government, one of the federal technology contractor's research arms.

In the report, Desouza recommended CIOs consider metrics such as:

• Number of ideas submitted by employees (over 30, 60, 90 days)
• Amount of budget spent on prototyping and experimenting with emerging technologies
• Percent of CIOs and key functional managers’ time spent on charting the future (strategic innovation) rather than on day-to-day operations

But in interviews with CIOs, Desouza discovered many felt organizational barriers could prevent them from collecting this information. 

“We work in a classified environment with built-in processes that slow down access to information," one CIO told Desouza's team.

Another told researchers that building such measurement systems is "a long game," adding, "I do not know if it is worth it, given that I have other major priorities to deliver on and have less resources than ever before.”

In many cases, budget cuts are placing even more pressure on CIOs to quantify the value of their cutting-edge, experimental efforts, Desouza said. 

Without nontechnical metrics, he said, "every time there’s a cut, they lose the ability to innovate. They have to choose between maintaining things or trying new things. And obviously, they choose to maintain."