'Mystery shopper' technique could be effective counterterrorism tool

The business of connecting the dots of information received from diverse sources and then developing conclusions about terrorist threats is more complicated than often suggested in media accounts. Nonetheless, we have a right to demand that counterterrorism organizations measure their performance and learn how to refine their analysis methods so they do the best possible job of protecting us.

One approach that might help is for the National Counterterrorism Center, an organization established in 2004 specifically to work on connecting dots, to adopt a version of the “mystery shopper” technique that many businesses use to measure and improve their performance.

In traditional settings, retail companies hire a mystery shopping service to send “undercover” shoppers into their stores to collect information on the customer experience, identifying both strengths and weaknesses. Management then uses that data to devise training programs to improve customer service.

Here’s how it would work in the National Counterterrorism Center: A dedicated staff would be responsible for sending information about fictional individuals into different parts of the intelligence system, whether National Security Agency telephone intercepts or reports from agents on the ground. The unit would be responsible for tracking whether and how analysts evaluated the information — that is, did they successfully connect the dots?

The center’s performance would be measured by the percentage of cases the analysts correctly identified as exhibiting a high risk of terrorist behavior. The metric could be tracked by organization, by team and eventually — when enough cases have been assembled — by analyst.

Before starting, the mystery shopper unit would need to develop a consensus about which information patterns suggest a high risk of terrorist activities. And it would need to avoid planting too many high-risk rumors, which could overwhelm the center’s resources. So the unit would also need to send noise or low-risk information as well and track the percentage of good hits versus low-risk hits. A mystery shopper unit would also want to vary the types of information it sends to see if certain sources or types are less likely to be successfully identified.

Like performance measures in general, if such metrics are used as a management tool, they should improve the performance of the counterterrorism system. Establishing challenging performance targets will motivate employees to work harder and be more vigilant. Varying the kinds of information fed into the system will allow officials to diagnose its weaknesses so agencies can better target training or changes in techniques.

The center would need to create a learning environment in which honest errors are discussed but not punished — what professor Amy Edmondson, who has done research on how hospitals can best reduce medical errors, calls psychological safety.

The country needs the best possible performance from its National Counterterrorism Center. There are management tools that have proven effective for improving performance in many contexts outside counterterrorism. The government can no longer afford to ignore them.