An alert developer appears to have prevented a backdoor — likely introduced into a compression utility by state-backed actors — from being distributed to production Linux systems. The malicious code appears to allow the bypassing of checks during SSH authentication.
Andres Freund, a Microsoft software engineer who spotted the backdoor in xz Utils, said the malicious code had been introduced in versions 5.6.0 and 5.6.1. The suspicion that it may be a state-sponsored act has arisen because the code has been spirited in over a long period.
Freund wrote on Friday: “After observing a few odd symptoms around liblzma (part of the xz package) on Debian sid installations over the last weeks (logins with ssh taking a lot of CPU, valgrind errors) I figured out the answer: The upstream xz repository and the xz tarballs have been backdoored.
“At first I thought this was a compromise of Debian’s package, but it turns out to be upstream.”
One of the developers concerned, who had the handle JiaT75, had been a maintainer or the package for more than two years. Freund added: “Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system.
“Unfortunately, the latter looks like the less likely explanation, given they communicated on various lists about the ‘fixes’ mentioned above.” His reference was to various additions suggested by the suspicious maintainer for fixing issues with the code: here, here, here, and here. [Thanks to Dan Goodin for these four links.}
SSH or secure shell is an utility used to log in securely to systems, with the majority of Linux systems using a port known as OpenSSH that is maintained by the OpenBSD project, an Unix clone.
The only production Linux system in which the doctored code was distributed appears to have been the Tumbleweed stream put out by the OpenSUSE project. The developers at that project wrote on Friday: “For our openSUSE Tumbleweed users where SSH is exposed to the Internet, we recommend installing fresh, as it’s unknown if the backdoor has been exploited.
“Due to the sophisticated nature of the backdoor an on-system detection of a breach is likely not possible. Also rotation of any credentials that could have been fetched from the system is highly recommended.
“It has been established that the malicious file introduced into Tumbleweed is not present in SUSE Linux Enterprise and/or Leap. Current research indicates that the backdoor is active in the SSH Daemon, allowing malicious actors to access systems where SSH is exposed to the Internet.”
Debian issued patched versions of xz Utils for its testing, experimental and unstable streams of development. Red Hat said on Friday, “Fedora Linux 40 users may have received version 5.6.0, depending on the timing of system updates. Fedora Rawhide users may have received version 5.6.0 or 5.6.1.”
“At this time the Fedora Linux 40 builds have not been shown to be compromised. We believe the malicious code injection did not take effect in these builds. However, Fedora Linux 40 users should still downgrade to a 5.4 build to be safe.”
Later, Red Hat added: “We have determined that Fedora Linux 40 beta does contain two affected versions of xz libraries – xz-libs-5.6.0-1.fc40.x86_64.rpm and xz-libs-5.6.0-2.fc40.x86_64.rpm. At this time, Fedora 40 Linux does not appear to be affected by the actual malware exploit, but we encourage all Fedora 40 Linux beta users to revert to 5.4.x versions.”
It is likely to be quite some time before calm returns; former senior Debian developer Joey Hess provided one reason, noting that the accounts used by the suspected malicious actors had made more than 700 commits [code contributions] over the past two years.
Wrote Hess: “I count a minimum of 750 commits or contributions to xz by Jia Tan, who backdoored it. This includes all 700 commits made after they merged a pull request in 7 Jan 2023, at which point they appear to have already had direct push access, which would have also let them push commits with forged authors. Probably a number of other commits before that point as well.
“Reverting the backdoored version to a previous version is not sufficient to know that Jia Tan has not hidden other backdoors in it. Version 5.4.5 still contains the majority of those commits.”
And he added: “The package should be reverted to a version before their [the bad actors’] involvement, which started with commit 6468f7e41a8e9c611e4ba8d34e2175c5dacdbeb4. Or their early commits [should be] vetted and revert to a later point, but any arbitrary commit by a known bad and malicious actor almost certainly has less value than the risk that a subtle change go unnoticed.
“I’d suggest reverting to 5.3.1 – bearing in mind that there were security fixes after that point… that would need to be re-applied.”
The individual who has come under suspicion appears to have been offering help off-list as well, in what could be an effort to ingratiate himself/herself with other developers and thus allay any suspicion behind these offers should it arise.
As is to be expected, there have been lengthy discussions about the issue; Linux Weekly News has a long thread here, while Hacker News had more than 2000 posts [when I last looked] here.
