GAO Handbook Helps Answer Whether That New Tech Is Right For Your Agency

The Government Accountability Office’s new technology assessment team published a handbook explaining its processes and providing a resource to other federal agencies—or any organization—trying to figure out whether a technology is right for its mission needs.

The Technology Assessment Design Handbook was published by GAO’s new Science, Technology Assessment, and Analytics Office, a team established to look at emerging technologies and provide unbiased information to Congress and other government leaders. As part of its work, the STAA Office looks at a range of issues around budding technologies, including how their uses can and will affect society; the risks and benefits of implementing those technologies; the status, viability and maturity; and any planned or ongoing federal investments.

“GAO has defined [technology assessment] as the thorough and balanced analysis of significant primary, secondary, indirect and delayed interactions of a technological innovation with society, the environment and the economy and the present and foreseen consequences and impacts of those interactions,” the handbook reads. “The effects of those interactions can have implications.”

For GAO—and anyone else trying to determine whether a technology would help or hamper their efforts—the first step is developing a hard design plan and getting that down on paper.

“Data collection and quality assurance of data can be costly and time-consuming,” the handbook states. “A thorough consideration of design options can ensure that collection and analysis of the data are relevant, sufficient, and appropriate to answer the researchable question(s), and helps to mitigate the risk of collecting unnecessary evidence and incurring additional costs.”

The handbook breaks the process down into three phases.

Phase 1: Determine the Scope

Perform the initial “situational analysis” to level-set, then determine the scope; goals, purpose and objectives; “the problem;” and any existing policies that apply. During this phase, assessors should consider what is known and not known about the technology.

The GAO team also suggests this is the time to reach out to potential stakeholders, both inside and outside the organization.

Phase 2: Develop Initial Design

During this phase, assessors should put the design framework down on paper, review it with stakeholders and solidify the initial design for the review.

By this point in the process, the review teams should have an appropriate design, methodology and analytical approach; identify available data sources or need to gather data; determine possible policy goals and how they might be analyzed.

Phase 3: Implementation of Design

The final phase includes implementing the review as designed while being willing to modify the plan as needed.

The handbook offers a detailed breakdown for each phase, including example questions and objectives to help assessors hone the design and approach to glean the most useful insights.

The team also looked at the biggest challenges to producing a useful assessment, such as explaining complicated topics in easy-to-understand language and avoiding bias, either from the team itself or from stakeholders with specific agendas.

GAO stressed that this handbook is a loose guide to developing an assessment.

“While [the handbook] presents TA design as a series of phases, actual execution is highly iterative and nonlinear,” they wrote. “Teams may need to be prepared to revisit design decisions as information is gathered or circumstances change.”

No matter the scope or design, GAO offered seven questions to help determine whether the assessment will be useful:

• Does the design address the needs of the requester?
• Will the design yield a quality, independent, balanced, thorough and objective product?
• Will the design likely yield information that will be useful to stakeholders?
• Will the design likely yield valid conclusions on the basis of sufficient and credible evidence?
• Will the design yield results in the desired time frame?
• Will the design likely yield results within the constraints of the resources available?
• How will policy options be identified and assessed, if applicable?

GAO also pushed the handbook as a living document and is actively looking for comments and feedback.

“Given that GAO is likely to learn from its current expansion of TA work, GAO will review and update this draft handbook as needed, based on experience gained through ongoing TA activities and external feedback,” the handbook reads. The team is accepting comments for one year through the email address TAhandbook@gao.gov.