Backdoor in upstream xz/liblzma leading to SSH server compromise

Yikes! Do you have any info on the individual's background or possible motivations?

Given the recent ( not so recent ) attacks/"bugs" I feel there is a need to do more than the already hard task of investigating and detecting attacks but also to bring IRL consequences to these people.

My understanding is that right now it's pretty much a name and shame of people who most of the time aren't even real "people" but hostile agents either working for governments or criminal groups ( or both )

Getting punched in the face is actually a necessary human condition for a healthy civilization.

I think this has been in the making for almost a year. The whole ifunc infrastructure was added in June 2023 by Hans Jansen and Jia Tan. The initial patch is "authored by" Lasse Collin in the git metadata, but the code actually came from Hans Jansen: https://github.com/tukaani-project/xz/commit/ee44863ae88e377...

Thanks to Hans Jansen for the original patch.

https://github.com/tukaani-project/xz/pull/53

There were a ton of patches by these two subsequently because the ifunc code was breaking with all sorts of build options and obviously caused many problems with various sanitizers. Subsequently the configure script was modified multiple times to detect the use of sanitizers and abort the build unless either the sanitizer was disabled or the use of ifuncs was disabled. That would've masked the payload in many testing and debugging environments.

The hansjans162 Github account was created in 2023 and the only thing it did was add this code to liblzma. The same name later applied to do a NMU at Debian for the vulnerable version. Another "<name><number>" account (which only appears here, once) then pops up and asks for the vulnerable version to be imported: https://www.mail-archive.com/search?l=debian-bugs-dist@lists...

I wonder who the target was!

Unfortunately, this is how good bad actors work: with a very long-term point of view. There is no “harmless” project any more.

Fascinating. Just yesterday the author added a `SECURITY.md` file to the `xz-java` project.

If you discover a security vulnerability in this project please report it privately. *Do not disclose it as a public issue.* This gives us time to work with you to fix the issue before public exposure, reducing the chance that the exploit will be used before a patch is released.

Reading that in a different light, it says give me time to adjust my exploits and capitalize on any targets. Makes me wonder what other vulns might exist in the author's other projects.

Yesterday sure was fun wasn't it :p Thanks for all your help/working with me on getting this cleaned up in Fedora.

Sleeper.

Also thanks to Debian for modifying openssh.

One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity.

Yes, there are things delivered with that complexity. However, as an example, sysvinit is maybe, oh, 20k lines of code including binaries, heck including all core init scripts.

What's systemd? 2M lines? It was >1M lines 4+ years ago.

For an init system, a thing that is to be the core of stability, security, and most importantly glacial, stable change -- that is absurdly complex. It's exceedingly over engineered.

And so you get cases like this. And cases like that, and that over there, and that case over there too. All which could not exist, if systemd didn't try to overengineer, over complicate everything.

Ah well. I'm still waiting for someone to basically fork systemd, remove all the fluff (udev, ntp, dns, timers, restart code, specialized logging, on and on and on), and just end up with systemd compatible service files.

But not yet. So... well, oh well.

The systemd notification protocol could have been as simple as just writing a newline to a pipe

It basically is. libsystemd links to liblzma for other features not related to notifications.

(The protocol is that systemd passes the path to a unix socket in the `NOTIFY_SOCKET` env variable, and the daemon writes "READY=1" into it.)

FWIW, I did a quick check on a Devuan system. The sshd in Devuan does link to a libsystemd stub - this is to cut down on their maintenance of upstream packages. However that stub does not link to lzma.

so now security-critical daemons like openssh have additional dependencies like liblzma

Systemd itself seems security-critical to me. Would removing other dependencies on libsystemd really make a secure system where systemd was compromised through its library?

That is all the protocol is. From https://www.freedesktop.org/software/systemd/man/latest/sd_n...:

These functions send a single datagram with the state string as payload to the socket referenced in the $NOTIFY_SOCKET environment variable.

The simplest implementation (pseudocode, no error handling, not guaranteed to compile), is something like:

    const char *addrstr = getenv("NOTIFY_SOCKET");
    if (addrstr) {
        int fd = socket(AF_UNIX, SOCK_DGRAM, 0);
        struct sockaddr_un addr = { .sun_family = AF_UNIX };
        strncpy(addr.sun_path, sizeof(addr.sun_path), addrstr);
        connect(fd, (struct sockaddr*) &addr);
        write(fd, "READY=1");
        close(fd);
    }

Uh. systemd documents the protocol at various places and the protocol is trivial: a single text datagram sent to am AF_UNIX socket whose path you get via the NOTIFY_SOCKET. That's trivial to implement for any one with some basic unix programming knowledge. And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise. In particular non-C environments really should do their own native impl and not botjer wrapping libsystemd just for this.

But let me stress two other things:

Libselinux pulls in liblzma too and gets linked into tons more programs than libsystemd. And will end up in sshd too (at the very least via libpam/pam_selinux). And most of the really big distros tend do support selinux at least to some level. Hence systemd or not, sshd remains vulnerable by this specific attack.

With that in mind libsystemd git dropped the dep on liblzma actually, all compressors are now dlopen deps and thus only pulled in when needed.

5.6.1-2 is not an attempted fix, it's just some tweaks to Arch's own build script to improve reproducibility. Arch's build script ultimately delegates to the compromised build script unfortunately, but it also appears the payload itself is specifically targeting deb/RPM based distros, so a narrow miss for Arch here.

(EDIT: as others have pointed out, part of the exploit is in the artifact from libxz, which Arch is now avoiding by switching to building from a git checkout)

On arch, `ldd $(which sshd)` doesn't list lzma or xz, so I think it's unaffected? Obviously still not great to be shipping malicious code that just happens to not trigger.

I upgraded Arch Linux on my server a few hours ago. Arch Linux does not fetch one of the compromised tarballs but builds from source and sshd does not link against liblzma on Arch.

  [root@archlinux ~]# pacman -Qi xz | head -n2  
  Name            : xz  
  Version         : 5.6.1-2  
  [root@archlinux ~]# pacman -Qi openssh | head -n2
  Name            : openssh
  Version         : 9.7p1-1
  [root@archlinux ~]# ldd $(which sshd) | grep liblzma
  [root@archlinux ~]#

The writeup indicates that the backdoor only gets applied when building for rpm or deb, so Arch probably would have been okay either way? Same with Nix, Homebrew, etc.

Thanks to autoconf, we're now used to build scripts looking like gibberish. A perfect place to hide a backdoor.

A big part of the problem is all the tooling around git (like the default github UI) which hides diffs for binary files like these pseudo-"test" files. Makes them an ideal place to hide exploit data since comparatively few people would bother opening a hex editor manually.

Yeah, now imagine they succeeded and it didn't cause any performance issues...

Can we even be sure no such successful attempt has already been made?

The use of "eval" stands out, or at least it should stand out – but there are two more instances of it in the same script, which presumably are not used maliciously.

A while back there was a discussion[0] of an arbitrary code execution vulnerability in exiftool which was also the result of "eval".

Avoiding casual use of this overpowered footgun might make it easier to spot malicious backdoors. Usually there is a better way to do it in almost all cases where people feel the need to reach for "eval", unless the feature you're implementing really is "take a piece of arbitrary code from the user and execute it".

[0] https://news.ycombinator.com/item?id=39154825

It looks like an earlier commit with a binary blob "test data" contained the bulk of the backdoor, then the configure script enabled it, and then later commits patched up valgrind errors caused by the backdoor. See the commit links in the "Compromised Repository" section.

Also, seems like the same user who made these changes are still submitting changes to various repositories as of a few days ago. Maybe these projects need to temporarily stop accepting commits until further review is done?

"Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system. Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the "fixes" mentioned above."

Crazy indeed.

I'm curious now. What is ifunc? (Had difficulty finding it through a search)

Interesting, I used https://ossinsight.io/analyze/JiaT75 to identify contributions from the account used by author of the backdoor. It looks like the account made other potentially problematic contributions to other projects.

The disabling of ifunc in this PR against Google's oss-fuzz project maybe one way they tried to prevent this particular backdoor being flagged by that tool? https://github.com/google/oss-fuzz/pull/10667

https://github.com/google/oss-fuzz/pull/10667

Is there a way to easily/reliably disable ifunc globally on a system (e.g. ubuntu/debian) without breaking a bunch of things?

FYI this looks for pkgs with liblzma:

dpkg -l |grep liblzma

Versions >= 5.6 are compromised

Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.

I'm surprised .deb doesn't have a better approach. RPM has epoch for this purpose http://novosial.org/rpm/epoch/index.html

The article gives a link to a simple shell script that detects the signature of the compromised function.

Running OpenSSH sshd from systemd

I think this is irrelevant.

From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.

Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.

I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related?

I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?

My normal internet connection has such an appalling upload that I don't think anything relevant could be uploaded. But I will change my ssh keys asap.

Ubuntu still ships 5.4.5 on 24.03 (atm).

I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github tag except for Changelog and some translation files. It does match the tarball content, though.

So for 5.4.5 the tagged release and download on github differ.

It does change format strings, e.g. ``` +#: src/xz/args.c:735 +#, fuzzy +#| msgid "%s: With --format=raw, --suffix=.SUF is required unless writing to stdout" +msgid "With --format=raw, --suffix=.SUF is required unless writing to stdout" +msgstr "%s: amb --format=raw, --suffix=.SUF és necessari si no s'escriu a la sortida estàndard" ```

There is no second argument to that printf for example. I think there is at least a format string injection in the older tarballs.

$ dpkg-query -W liblzma5

liblzma5:amd64 5.4.1-0.2

One thing to note is that the person that added the commits only started contributing around late 2022 and appears to have a Chinese name. Might be required by law to plant the backdoor.

That would be quite scary considering they have contributed to a wide variety of projects including C++ https://learn.microsoft.com/en-us/cpp/overview/whats-new-cpp...

LANG only needs to have some value, the concrete value does not seem to matter.

I mean, a backdoor at this scale (particularly if it wasn't noticed for a while and got into stable distros) could be worth millions. Maybe hundreds of millions (think of the insider trading possibilities alone, not to mention espionage). 2 years doesn't seem like that much work relative to the potential pay off.

This is the sort of case where america's over the top hacking laws make sense.

I would be curious if their commits could be analyzed for patterns that could then be used to detect commits from their other account

Every single commit this person ever did should immediately be rolled back in all projects.

Couldn't the autoconf soup be generated from simpler inputs by the CI/CD system to avoid this kind of problem? Incomprehensible soup as a build artifact (e.g. executables) is perfectly normal, but it seems to me that such things don't belong in the source code.

(This means you too, gradle-wrapper! And your generated wrapper for your generated wrapper. That junk is not source code and doesn't belong in the repo.)

I've analysed the backdoor myself and it's very sophisticated, not poorly made at all. The performance problem is surprising in this context, but I think next time they won't make that mistake.

So many malicious actors have been caught because they accidentally created a mild annoyance for someone that went on to bird-dog the problem.

s/can do/have done/

I don't want to read too much into it, but the person (supposedly) submitting the PR seems to work at 1Password since December last year, as per his Linkedin. (And his Linkedin page has a link to the Github profile that made the PR).

it's not the same GitHub account as upstream

This is valuable information, and a sign that this may be the tip of an iceberg.

As a bit of an aside, I would never accept a PR like this, and would always update $large_vendored_dependency myself. This is unreviewable, and trivial to insert any backdoor (unless you go through the motions of updating it yourself and diffing, at which point the PR becomes superfluous).

This is how I once inserted a joke in one of our (private) repos that would randomly send cryptic messages to our chat channel. This was pretty harmless and just a joke (there's some context that made it funny), but it took them years to find it – and that was only because I told them after I quit.

That said, looking at the GitHub account I'd be surprised if there's anything nefarious going on here. Probably just someone using your repo, seeing it's outdated, and updating it.

The backdoor (test binary blob and autoconf) is not part of the pull request.

If you installed xz on macOS using brew, then you have

  xz (XZ Utils) 5.6.1
  liblzma 5.6.1

  brew upgrade xz

cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

That's the entire point. You did everything you could by getting someone else look at it and saying it's fine.

the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

That is actually a major point of a lot of corporate security measures (shifting risk)

Thats basically the whole point actually... A company pays for insurance for the business. The insurance company says sure we will insure you, but you need to go through audits A B and C, and you need certifications X and Y to be insured by us. Those audits are often industry dependent, mostly for topics like HIPAA, PCI, SOC, etc.

Insurance company hears about supply chain attacks. Declares that insured must have supply chain validation. Company goes and gets a shiny cert.

Now when things go wrong, the company can point to the cert and go "it wasnt us, see we have the cert you told us to get and its up to date". And the company gets to wash their hands of liability (most of the time).

The issue is caused by patches to add integration with systemd, so no, this won't affect SSH on a Mac.

Homebrew reverted to 5.4.6 once the maintainers became aware. The current understanding is that macOS is not affected, but that's not certain.

[1]: https://github.com/Homebrew/homebrew-core/pull/167512

For anyone else feeling some deja vu about ifunc / Valgrind errors, this RedHat issue [1] was previously linked from HN a week ago [2].

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=2267598

[2]: https://news.ycombinator.com/item?id=39733185

That name jumped out at me, Hans Jansen is the name Dominic Monaghan used when posing as a German interviewer with Elijah Woods. Not that it can't be a real person

https://youtu.be/IfhMILe8C84

I get the feeling that a number of the comments are all the same person / group.

Wow, that's a lot of anonymous accounts adding comments there urging for a fast merge!

And this "Hans Jansen" guy is apparently running around salsa.debian.org pushing for more updates in other projects as well: https://salsa.debian.org/users/hjansen/activity

The lesson here seems to not depend on tools written in languages that have complex, obscure build systems and no one is either able or interested to read. Using tools rewritten in Rust, Go or any other languege which resolves dependencies within project seems the only way to do hardening here.

It may not have really mattered much for NixOS:

b) argv[0] needs to be /usr/sbin/sshd

For once, the lack of FHS interoperability is a benefit, if only on accident.

Potentially the purpose is that if someone goes to the effort to get those details together, they are more likely to send the same report to other trusted individuals. Maybe it was originally there to add legitimacy, then they got a report sent in, and removed it to slow the spread of awareness

FWIW, it felt intimidating as hell. And I'm fairly established professionally. Not sure what I'd have done earlier in my career (although I'd probably not have found it in the first place).

RHEL won't get this bug for 2 years =)

RHEL no, Fedora 41 and Rawhide yes.

https://www.redhat.com/en/blog/urgent-security-alert-fedora-...

https://lists.debian.org/debian-security-announce/2024/msg00...

Note that the malicious binary is fairly long and complex.

This attack can be stopped by disallowing any binary testdata or other non-source code to be on the build machines during a build.

You could imagine a simple process which checks out the code, then runs some kind of entropy checker over the code to check it is all unminified and uncompressed source code, before finally kicking off the build process.

autogenerated files would also not be allowed to be in the source repo - they're too long and could easily hide bad stuff. Instead the build process should generate the file during the build.

deb packages are xz compressed...

FreeBSD is not affected as the payloads in question were stripped out, however we are looking into improvements to our workflow to further improve the import process.

Has anyone been prosecuted for adding a backdoor

Google up Randal Schwartz. Caution: clickhole.

It takes a minute to create a new email address. And you can change or fake an email address on a git commit trivially. You, too, can writing code as anyone you want by just doing "git commit --author='Joe Biden <icecream@whitehouse.gov>'". On the internet nobody knows you're Joe Biden.