Census dinged by IG for canvassing error rate

A recent Department of Commerce inspector general report raises questions about the quality control procedures and data quality related to how the Census Bureau uses satellite imagery to count its blocks.

As part of its larger turn to more modern tech solutions to cut costs, Census opted to reduce field operations by leaning on satellite imagery to survey America's approximately 11 million blocks and identify any changes made from the office, rather than traversing each block on foot.

In 2010, Census spent $444 million to hire almost 149,000 temporary workers to walk all 11 million blocks and identify every household where someone could live. By comparison, the new process, known interactive review, involved hiring 136 clerks at an estimated cost of $12.9 million.

The OIG report noted that the bureau did not complete a formal cost estimate for the interactive review operations and that its estimate "is not transparent and discrepancies between estimated and actual costs, if found, cannot be researched and identified."

The bureau confirmed to the OIG that, as of June 8, 2017, all 11 million or so blocks had been reviewed at least once.

Auditors, however, found that Census's quality assurance plan for the interactive review operation, which was drafted but never finalized, does not document the methodology used to calculate error rates and their potential impact on data quality.

To make sure the process is conducted correctly, interactive review entails a double-blind review -- meaning that after the initial review, a second clerk reviews the same block. If the two agree, no additional reviews are performed. If they don't, a third reviewer, called the adjudicator, breaks the tie and can charge errors to either clerk.

According to Census policy, if a clerk has an error rate below 5 percent after completing 500 blocks, only a sample of his or her future reviews will go through a double-blind check. But if a clerk's error rate exceeds 5 percent during or after the baseline period, he or she is supposed to have all future reviews checked.

However, auditors found, in practice, "every single clerk qualified for sampling after his or her first 500 blocks," calling into question the accuracy of the collected data.

Additionally, auditors found that the reason Census performed fewer quality assurance reviews was because bureau policy allows adjudicators to forgive errors when imagery data is "highly ambiguous."

Because these forgiven errors are not factored into error rates, auditors reported that this practice "interferes with business processes established to ensure" low error rates.

OIG recommends Census finalize the quality assurance plan, complete with clear explanations of how errors rates are calculated and descriptions of internal controls. The auditors also recommend ensuring current practices for error forgiveness are addressed when developing business rules to meet the 5 percent undetected error rate.

Census said it had no fundamental disagreements with the findings or recommendations and will prepare a formal action plan, clearly define all internal controls and revise its business rules.