Xz/liblzma: Bash-stage Obfuscation Explained

the bad actor was a co-maintainer of the repo (and even more active than the original maintainer for quite some time) with full commit rights. This was strait committed to master, no PR and no review required.

edit: also this was heavily obfuscated in some binary files that were marked as test files ("good" and "bad" xz compressed test file). No way to spot this if you don't know what you're looking for.

The commit messages for the test files claim they used an RNG to generate them. The guy making the release tarball then put the final line in the right place without checking it in.

There are no code review on packets with 1 active maintainer.

They already exist in the source. They're split in the compression test files themselves. (Unless you meant some other binaries?)

The macro defined in build_to_host.m4 is probably called on the tests subdirectory, so it gets these files as a parameter.

EDIT: It is called here and will do the extraction of the backdoor when run in the 'tests' subdirectory:

https://salsa.debian.org/debian/xz-utils/-/blob/debian/unsta...

EDIT2: So it will get the directory as a parameter, the actual file is encoded indirectly here:

https://salsa.debian.org/debian/xz-utils/-/blob/debian/unsta...

This grep will only match bad-3-corrupt_lzma2.xz

OMG that's evil. The diff just shows:

  +
  +.
  +

This is very likely just a mistake and not deliberate.

a) absolutely nobody uses cmake to build this packet

b) if you try to build the packet with cmake and -DENABLE_SANDBOX=landlock, the build just fails: https://i.imgur.com/7xbeWFx.png

The "." does not disable sandboxing, it just makes it impossible to build with cmake. If anyone had ever actually tried building it with cmake, they would get the error and realize that something is wrong. It makes absolutely no sense that this would be malicious attempt to reduce security.

I really hate writing these compile/build time conditional things. It's hard to have tests that it's enabled when it should be and disabled when it isn't, especially if it's in the build system where there's no unit test framework.

And that's with just accidentally booking it so the test always fails or always succeeds when it shouldn't. You can see why it's a juicy target for malicious actions.

You do realize that "that shit" was part of the obfuscated and xz-compressed backdoor hidden as binary test file, right? It was never committed in plain sight. You can go to https://git.tukaani.org/xz.git and look at the commits yourself – while the commits of the attacker are not prime examples of "good commits", they don't have glaringly obvious red flags either. This backdoor was very sophisticated and well-hidden, so your comment misses the point completely.

i don’t have a better answer, but this convoluted mess of bash is a smell isn’t it?

It's a very old smell, basically.

The central problem is that back in the 80s and 90s, there were scads of different Unix-like systems, each with their own warts and missing features. And software authors wanted to minimize their build dependencies.

So many communities standardized on automating builds using shell scripts, which worked everywhere. But shell scripts were a pain to write, so people would generate shell scripts using tools like the M4 macro preprocessor.

And this is why many projects have a giant mass of opaque shell scripts, just in case someone wants to run the code on AIX or some broken ancient Unix.

If you wanted to get rid of these impenetrable thickets of shell, you could:

1. Sharply limit the number of platforms you support.

2. You could standardize on much cleaner build tools.

3. You could build more key infrastructure in languages which don't require shell to build portably.

But this would be a massive undertaking, and a ton of key C libraries are maintained by one or two unpaid volunteers. And dropping support for "Obscurnix-1997" tends to be a fairly controversial decision.

So much of our key infrastructure remains surrounded by a morass of mysterious machine-generated shell scripts.

  > inadvertently

(NOTE: I may misunderstand the risk of the XV backdoor - "it exec'd"..., and so my premise may be irrelevant to this convo.)

Is there a way to run BASH such that it does not allow EXEC'ing things? Like, have a "secure mode" for bash?

EDIT: For xv's configure script, I cannot imagine how one could run BASH in any hypothetical "secure mode". So, Nvm.

The shell is generated, not written. There are mountains of generated shell configuration code out there due to the prevalence of autoconf, which relies on these M4 (a macro preprocessor) scripts to generate thousands of lines of unreadable shell script (which has to be portable).

This is how a non negligible number of your tools are built.

this convoluted mess of bash is a smell isn’t it

At a glance, I don't think so. At least, not the fact that the bash looks like a convoluted mess. Sometimes that's how a tight bash script looks (whether it SHOULD be tight or not is a different argument).

For me, the thing that looks suspicious is the repeated tr calls. If I see that, I assume someone is trying to be clever, where 'clever' here is a pejorative. If I were a maintainer and someone checked that in, I'd ask them to walk me through what it was doing, because there's almost always a better solution to avoid chaining things like that.

The real problem here is that there wasn't another maintainer to look at this code being brought in. A critical piece of the stack relied on a single person who, in this case, was malicious.

It’s clever but not entirely novel, this is kind of the intended usecase for these

Opportunity to write a paper

I've generally seen this with Unix installers from commercial software vendors.

You get a giant .sh file that displays a license, asks you to accept, then upon acceptance, cats itself, pipes through head/tail, into cpio to extract the actual assets.

mplayer/mpv has a lot of binary files in their test suite. They are slimmed-down copies of movie formats created by other tools, specifically to ensure compatibility with substandard encoders. If you were to generate those files at test time, you'd have to have access (and a distribution license!) to all those different encoders.

I don't think treating those binary files in the repo as red flags is in any way useful.

Any library that works with file formats needs binary files.

A lot of them malformed (or output is slightly different than standard output), because they need to ensure they can work even with files generated by other programs. Bugs like ' I tried to load this file and it failed, but works in XYZ' are extremly common.

These formats are often very complex and trying things like'zeroing out a high bit' doesn't cut it. Youvwould end up with binary code encoded in source.

Edit: one of simple improvements github/other forges could do is show content of archives in a diff. The payload was hidden in a archive test file and it would be displayed in a diff instead of "binary file change, no idea what is in it"

Absolutely yes. As a rule of thumb, for sure. However, in reality the problem isn’t binary per se, but anything that’s too obfuscated or boilerplatey to audit manually. It could be magical strange strings or more commonly massive generated code (autoconf?). Those things should ideally not be checked in, imo, but at the very least, there needs to be an idempotent script that reproduces those same files and checks that they are derived correctly from other parts of the code. Ideally also in a separate dir that can be deleted cheaply and regenerated.

For instance, in Go it’s quite common to generate files and check them in, for eg ORMs. If I run `rm -r ./gen` and then `go generate`, git will report a clean working dir if all is dandy. It’s trivial to add a CI hook for this to detect tampering. Similarly, you could check in the code that generates garbage with a seed, and thus have it be reproducible. You still need to audit the generators, but that’s acceptable.

verify the identity of open source maintainer/s and question their motive for joining the open source project.

This kind of goes against the whole "free" thing.

"(B)urn (A)ll (S)atanic (H)elper-scripts" ? get the kindling?

Maybe.

But maybe there is not many (critical) ones out there. Otherwise, I believe we would encounter more often those kind of situations.

Running the tests does not modify the binary. The build script was modified to create a malicious .o file which was linked into the final binary by the linker as normal. Tests were only involved in that the malicious .o was hidden inside binary test files.

Furthermore, the attacker covered their tracks on the initial payload with an innocuous paragraph in the README. ("Nothing to see here!")

    bad-3-corrupt_lzma2.xz has three Streams in it. The first and third
    streams are valid xz Streams. The middle Stream has a correct Stream
    Header, Block Header, Index and Stream Footer. Only the LZMA2 data
    is corrupt. This file should decompress if --single-stream is used.

    $ cat tests/files/bad-3-corrupt_lzma2.xz | xz -d --single-stream
    ####Hello####

And lastly, they act as a marker that can be used by a later regex to locate the file _without_ referencing it by name directly nor using the actual Hello and World strings.

    $ gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/ 2>/dev/null`
    $ echo $gl_am_configmake
    ./tests/files/bad-3-corrupt_lzma2.xz

That's quite funny - yes, not only is this a horrible wilful backdoor, it is also a GPL violation since the backdoor is a derived work without included source / distributed not in "the preferred form for modification".