Crowdstrike IT outage linked to update using new threat detection system
Digital billboards in New York City's Times Square rebooting after an error in a Crowdstrike software knocked Windows-based systems offline on July 19, 2024. The company has traced the outage to the source in a preliminary incident report. Selcuk Acar/Anadolu via Getty Images
A routine update intended to enhance clients’ security capabilities clashed with a new cyberthreat classification framework rolled out in February, causing affected systems to crash.
A CrowdStrike-enabled global IT outage that crippled millions of Windows computers last Friday was linked to a February overhaul of an internal system used by the cybersecurity company to classify suspicious cyber activity passing through customers’ devices.
The February change was designed to better spot certain cyber threats through Named Pipes, a core Windows feature that allows a computer’s operating system to manage data and facilitate communications between other devices on a network.
Specifically, a revamped “Template Type” label system used by CrowdStrike was meant to better detect cyberattacks occurring within Named Pipes, but a validation bug allowed a template update that contained incorrect data to pass through standard checks without being flagged, according to a preliminary incident review released by CrowdStrike late Tuesday.
In essence, the new detection rules — built on the system introduced in February — were included in the faulty update. While the cybersecurity firm says the February system had been thoroughly checked, one of the new rules contained a flaw that wasn’t caught before being released last week.
The company’s flagship Falcon platform is designed to deter hackers from accessing a client’s systems at all levels of a device or across devices connected to their network. To do this, the product needs to tether onto computers at a root level, where their operating systems sit. Once installed, the platform has full access to the crown jewels of the computers, where it can stop threats moving about at all points.
CrowdStrike helped pioneer endpoint detection and response technologies that stop digital threats from infiltrating systems by shielding “endpoint” devices like laptop computers or phones that often provide hackers an entryway into networks. As part of this, the Falcon offering includes a template system that helps define how it should respond to cyberattacks once they are detected on a network.
But the Friday update, designated as “Rapid Response Content” that can be deployed to customers without changing the foundational aspects of the software, slipped through the cracks of a testing structure used by the company to check product code before being deployed.
A new process is in development to prevent similar errors in the future, CrowdStrike said in the incident review. The company will revamp the testing processes for Rapid Response Content that would include adding “additional validation checks” to a scanning system used to check for bugs in product code, it says.
“I want to sincerely apologize directly to all of you for the outage. All of CrowdStrike understands the gravity and impact of the situation. We quickly identified the issue and deployed a fix, allowing us to focus diligently on restoring customer systems as our highest priority,” CEO George Kurtz said in a statement.
The recent outage has already created secondary hacking opportunities being leveraged by cybercriminals, Nextgov/FCW reported Monday.
Crowdstrike is in wide use across federal agencies and it is a key vendor on the governmentwide Continuous Diagnostics and Mitigation cybersecurity support services contract. The company has also secured contracts with the Justice Department, State Department and DHS. Last Friday, multiple federal agencies were affected and some had to shutter business operations for the day.