Watchdog: IRS Declined to Patch Known Weakness Ahead of Tax Day Crash

Officials at the IRS might have avoided a Tax Day hardware failure this year but opted to stick with what they had rather than take a chance on an “unstable” patch, according to an audit by the Treasury Inspector General for Tax Administration released Monday.

On April 17—Tax Day 2018—a then-18-month-old piece of hardware supporting a lynchpin of the agency’s data storage systems experienced a caching issue, causing a cascade of failures that affected 59 different digital systems. IRS employees were able to get all systems back online within 11 hours, but the agency decided to extend the filing period another day to account for the confusion and loss of availability.

The TIGTA audit confirms many details of the incident and shines some new light on the firmware bug that caused the outage and the fact that IRS officials were aware of it and chose not to update their systems based on a contractor’s internal policy.

“In June 2017, International Business Machines (IBM) initially discovered the firmware bug associated with the IRS Tax Day outage and developed a fix, made publicly available in a November 2017 microcode bundle,” auditors wrote.

After identifying the bug, IBM representatives disclosed the issue to Unisys, IRS’ vendor on the Enterprise Storage Services contract that helps IRS integrate new technologies with legacy systems. Unisys recommended the agency wait to apply the patch, citing an informal internal company policy that “requires microcode bundles to have, at a minimum, 450 machine weeks in a production environment prior to installation on IRS equipment,” auditors said.

Based on that recommendation during a December 2017 meeting, “the IRS agreed with the decision to continue with what was considered a more stable version of the microcode … for the remainder of the filing season,” the report states. “However, there was no detailed discussion regarding individual fixes and whether these defects and associated updates could affect the IRS environment.”

Auditors noted IBM engineers worked on custom code in January 2018 for another customer that experienced issues with the same bug but IRS officials were never made aware of that fix.

After the Tax Day issues, IRS officials ultimately decided to go with the original microcode bundle issued in November 2017, and have updated all storage systems as of June 9.

Inspector general auditors also cited Unisys for not meeting its obligations under the Enterprise Storage Services contract and urged IRS to push for cash remuneration.

“During the Tax Day outage, the contractor failed to meet the [service level objective] requirements for initial acknowledgement of the problem, creating a plan for resolution and actual resolution of the problem,” the report states. “As part of the ESS contract, if the contractor fails to perform the services within the time specified, the contractor shall pay to the government liquidated damages.”

Spokespeople for IBM and Unisys did not immediately respond to requests for comment.

The inspector general offered four recommendations:

• Document lessons learned from the 2018 crash and implement a corrective action plan.
• Formalize the monthly microcode bundle meetings.
• Ensure that decisions not to install microcode bundle updates are documented and approved.
• Seek liquidated damages from Unisys and make modifications to the Enterprise Storage Services contract.

IRS officials agreed with all four and have already “formalized the monthly microcode bundle meetings and sought damages from the Enterprise Storage Services contractor.”