Linux backdoor was a long con, possibly with nation-state support, experts say

Last year, JC Herz and her team at cybersecurity firm Exiger found a vulnerability in a federal system’s open-source software that interacts with troves of sensitive government data. They immediately alerted the system owner and defense stakeholders connected to the intelligence community and the Pentagon.

The vulnerability was not lodged in the system’s code, but was, quite literally, the maintainer that sent commitments to the system: a single Russian government employee.

“If this was compromised, it would have been front-page news everywhere,” said Herz, the SVP of Exiger’s Cyber Supply Chain group. Her experience underscored the dangers of accidentally allowing the wrong people inside sensitive open-source systems. Why deploy code to circumvent a network’s security protocols and risk being caught when one could instead play the long game and, eventually, have access to everything inside?

That same story has a new chapter, this time for a tool used far beyond one government agency.

The positioning of a deeply-embedded Linux vulnerability that set off alarms in the open-source community this past week was being covertly planned for years, and the entity involved in the maneuver has strong ties to nation-state hackers, Herz and other analysts say.

A malicious actor planted the flaw into XZ Utils, a widely used Linux file compression and transfer capability, sometime around mid to late February. It contained a self-installation script that would have enabled the malign code to plant itself into production versions of Ubuntu, a Linux distribution used by major companies like Instacart, Slack and Robinhood.

Open source code is everywhere in commercial systems. The 2024 Open Source Security and Risk Analysis Report from Synopsys found open source components in more than 96% of over 1,000 commercial codebases, with 84% containing at least one known vulnerability.

Because the tool is open-source, it relies on contributions from community members who keep it up to date with patches and contributions. The updates are often discussed on forums with voluntary software maintainers, who chat with one another about proposed changes.

A user known as “Jia Tan” — who had been contributing to that open source community for years — reported a bug March 28 requesting that the version of the software be updated with the malign code tucked inside, justifying it would fix issues in Debian, another Linux distribution whose community provides a free-to-use operating system. It was caught by Microsoft engineer Andres Freund last week, and other Linux communities soon sounded the alarm.

“It takes the type of investment that you typically only see from nation-state actors,” said Silas Cutler, an espionage malware analyst and senior cyber threat research director at the Institute for Security and Technology. “They had an incredibly good technical grasp of the [XL] library.”

For the long haul

If allowed to propagate, the backdoor could have rendered the open-source Linux ecosystem ripe for exploitation. The mechanism targeted was a Secure Shell — or SSH — tool, which compresses and scrambles data sent over a connection. The planted weakness could have let bad actors gain access to entire systems by letting them bypass authentication protocols used in the SSH process.

The entity would have held a “skeleton key to the world” and been able to “cross huge amounts of the internet without any barriers in front of them,” Cutler said.

Jia Tan, who is also affiliated with username “JiaT75” has been contributing to the XZ developer community since at least 2022, according to analysis from Bitdefender. The account was created in 2021, and spent several years building trust with other contributors. 

Around March 9, the user added a piece of code with a hidden backdoor that, once triggered, sabotages the tool and grants access to systems used by XZ Util without having to be authenticated. The move appears to have been a “meticulously planned, multi-year attack” possibly supported by a hacker linked to a nation-state group, Bitdefender technical solutions director Martin Zugec said.

It’s very possible that Jia Tan was not a single entity acting alone, Herz said.

“This was an identity created for the purpose of taking the action, and based on our data, there seems to have also been decoy actors that were created around the same time to corroborate or support this attack,” she said.

Eyes on the target

Law enforcement is very likely probing the incident, said Chris Stangl, a former FBI Cyber Division agent who helped investigate the Log4J vulnerabilities that emerged in late 2021. 

“I guarantee you CISA and the FBI are looking at this in a way that ensures this doesn’t happen again, and are asking what kind of guidance could be disseminated and what the motivation of the actors were,” said Stangl, now a managing director at consulting firm BRG.

Investigators might be analyzing the update code that Jia Tan and affiliate users deployed to see if it’s been tied to other nation-state hacking groups, said Ami Luttwak, CTO of cloud security company Wiz.

Jia Tan carefully uploaded code updates during its tenure as a fake contributor, some seeming to occur during Chinese business hours and other times indicating European. Ultimately, it might be impossible to determine their exact origins in the near term, Luttwak added.

“The only thing we know is that there is an email that was used,” he said. “And that’s part of the challenge in open source — you don’t really know who’s behind it.”

“We are deeply focused on the open source security challenge generally, and we’re working with partners to get a better understanding of the XZ Utils issue,” CISA executive assistant director for cybersecurity Eric Goldstein told Nextgov/FCW on the sidelines of an International Association for Privacy Professionals conference on Thursday. An agency spokesperson referred Nextgov/FCW to its earlier alert on the incident.

The NSA and Office of the Director of National Intelligence declined to comment. An FBI spokesperson also declined to comment, saying they could “neither confirm nor deny the existence” of an investigation.

The open-source rift

The incident is likely galvanizing conversations on Capitol Hill and in the intelligence community about the risks and trade-offs of free-to-access software.

The Linux event, in particular, presents a double-edged sword debate in open-source security: A fraudulent user tried to deploy a malign version of the tool for widespread usage, but real contributors were able to stomp it out before it became more severe.

It was a win for the open source community, said Stangl, but lawmakers may consider ways to further engage companies and developers to better manage product development lifecycles and code used in sensitive systems.

“This should be a wake up call for software developers and open source about how they are vetting contributors,” he said. “What are they doing upfront to ensure that it’s secure? What does their code review look like?”

But some fear the incident may put open source in a bad light. “I worry that it’s an opportunity people are going to take to impose regulations that are not necessarily in open source’s best interest,” Cutler said.