The xz attack shell script

I think the common usage of bash and other languages with a dense and complicated syntax is the root problem here. If build scripts were written in python, you would have a hard time obfuscating backdoors, because everyone would see that it is weird code. Bash code is just assumed to be normal when you can’t read it.

The issue was that built artifacts weren’t immutable during the test phase and/or the test phase wasn’t sandboxed from the built artifacts.

The last build system I worked on separated build and test as separate stages. That meant you got a lot of useless artifacts pushed to a development namespace on the distribution server, but it also meant later stages only needed read access to that server.

I do not agree with your generalisation, the Meson build system is well thought, it has a clear declarative syntax that let you just express what you want in a direct way.

The designer of Meson explicitly avoided making the language turing complete so for example you cannot define functions. In my experience this was an excellent decision to limit people tendency to write complex stuff and put the pressure on the Meson developer to implement themselves all the useful functionalities.

In my experience the Meson configuration are as simple as they can be and accommodate only a modicum of complexity to describe OS specific options or advanced compiler option one may need.

Please note that some projects' Meson file have been made complex because of the goal to match whatever the configure script was doing. I had in mind the crazy habits of autotools to check if the system has any possibly used function because some system may not have it.

I see what everyone is saying about autotools, and I never envied those that maintained the config scripts, but as an end-user I miss the days when installing almost any software was as simple as ./configure && make && make install.

Agreed, and I think this leads to the question: how much risk do we face because we want to support such a wide range of platforms that a complex system is required?

And how did we get to the point that a complex system is required to build a compression library -- something that doesn't really have to do much more than math and memory allocation?

Every time someone explains to me autotools, the individual pieces sort of make sense, yet the result is always this inscrutable unreadable mess.

I don't know why.

Let me reword that for you:

No one has any business saying they know what something does until they've actually read it.

Beneath the placid surface of abstraction is the den of the devil.

Yeah.

I haven’t used it for some time but autoconf always seemed like a horrible hack that was impossible to debug if it didn’t work properly.

That was bad enough back in the days where one was mostly concerned with accidents, but in more modern times things that are impossible to debug are such tempting targets for mischief.

I'm glad to see I'm not a minority of one here. Looking at autoconf and friends I'm reminded of the dreck that used to fill OpenSSL's code simply because they were trying to account for every eventuality. Autotools feels like the same thing. You end up with a ton of hard to read code (autogenerated bash, not exactly poetry) and that feels very inimical to safety.

The devil is in the detail and nothing obscures details like complexity.

Same reason why I don't like TypeScript in its current form. It's not worth the extra complexity it brings.

Autotools need to go away, but most of the underlying use cases that cause build system complexity comes down to dependency management and detection. The utter lack of best practices for those workflows is the root cause of complexity. There is no way to have a mybuild.toml build config on the face of those challenges.

Just the fact that you have multiple platforms suggests few people will fully understand the entire complex.

Right there with you. It's really tempting to blame this entire thing existing on m4 but that's the trauma talking.

I don't buy it. Most private companies would require a face-to-face meeting when you start. Even if you're fully remote, the expectation would be that at some point you would meet your coworkers at meetspace, most likely before you get to commit anything substantial. The ones that are worth penetrating will almost certainly require a background check.

And then, once you're in, you cannot just commit to your target project willy-nilly, as your manager and your manager's manager will have other priorities. A for-profit company's frequently dysfunctional management would actually work as deterrent here: you don't just need to justify your code, you will have to justify why you were working on it in the first place.

I’m gonna cry survivorship bias here. How do we know how many similar attempts succeeded? How many of the ones discovered have been written off as honest mistakes? How can we know for sure that e.g. Heartbleed wasn’t put there by someone on purpose (and that that someone isn’t filthy rich now)?

When you get hired to a private company, they know who you are. That’s an immediate deterrent against trying anything funny. On Github, no one knows who you are. It might be harder to backdoor a project without getting noticed, but there is no risk to getting noticed. You can try as many times as you like. Jia Tan is still at large, and didn’t even have to plan their whole life around living in a non-extraditing country (if they aren’t in one already).

off topic, but how many actual non-extradition countries are there these days (for the US)?

Even countries we have strained relationships with will extradite as part of a negotiation when it's convenient for them politically.

Russia probably wouldn't have even kept Snowden if it wasn't state secrets he revealed. If it was just some random data breach they would have prisoner-swapped him for an oligarch caught money laundering elsewhere.

Could be interesting as a class project to investigate other small relatively innocuous but near ubiquitous projects that could be treated the same way and investigate whether something similar could be done or has been done already. Just making a list of them would be useful if nothing else.

a complex APT supply-chain attack

What do you mean by APT? If you mean Debian's package manager, that's not what this attack was. This was done upstream and affected non-apt distros just as much.

It's true that upstream is part of apt's supply chain but focussing on apt is misleading.

edit: why the downvotes? I get from the responses that I was wrong but given how the exploit was initially found in a Debian system and a lot of people very quickly jumped on the “Debian patched a thing and broke security” bandwagon, I don’t think it was much of a leap to wonder if that’s what was meant.

Acronyms and initialisms are not the best way to convey specific information.

The changes to build-to-host.m4 weren't in the source repo, so there was no commit. > > The attacker had permissions to create GitHub releases, so they simply added it to the GitHub release tarball.

What are some simple tweaks the "Debians" of the world can do to mitigate this kind of stuff?

Not trust "hand-curated-by-possibly-malicious-maintainers" GitHub release tarballs, only trust git commits?

Whitelist functions that are allowed to do IFUNC/ELF hooking to core OpenSSH functions?

Add to that list suppression of warnings from valgrind and address sanitizer without any justification. And no tracking issue to follow up on fixing the problem so the suppression can be removed.

Committing binary files to source control rather than including build commands to generate the files is a bit of a red flag.

hello lib/firmware

How could it play any role here? It doesn't bring any dependencies, does it?

Jia Tan asked distros to update quickly just before it became public. How possible is it, there is another account / person who learned earlyer from people around Andreas Freund, the backdoor would become public. How possible is it, there is still another insider around?

That seems to be largely grasped at straws and connecting dots without reason.

For example oss-fuzz was building xz by cloning the github repo directly. There was never a chance for oss-fuzz to discover the backdoor, because only the tarball had it, not the repo itself. So that oss-fuzz PR might genuinely just be a genuine thing unrelated to the backdoor.

Thx. Timeline shows attack begun by adding ignore entry in .gitignore file. That is hard to detect nowadays.

This just gives a partial high-level look at how the exploit gets planted in liblzma, it doesn't cover how the exploit works or its contents at all.

You're not wrong, but IMO that isn't the real problem. The real problem is the combination of highly popular core open-source utilities with near-zero maintenance resources.

This was all possible because XZ, despite being effectively everywhere, has one actual part-time maintainer who had other things going on in his life. That also means that there's nowhere near enough resources to redo the build system to some more secure alternative. If they had, say, 10 enthusiastic and skilled volunteer contributors with plenty of free time, they could do that, but then a new person appearing out of nowhere with a few helpful commits would never have a chance at being made an official maintainer or sneaking sketchy tools and code past the other maintainers.

Not that I'm blaming XZ or the real maintainer. Clearly whoever was behind this was looking for the weakest link, and if it wasn't XZ, it would have been something else. The real problem is the culture.

So I guess what this really means is someone at a big corporation making Linux distros should audit their full dependency tree. Any tool in that tree that isn't actively maintained, say at least 3 long-time active contributors, they should take over one way or another - whether that's hiring the current maintainer as a full-time remote employee, offering to buy out the rights, or forking and running under their own team.

I'm not necessarily super thrilled with that, but I guess it's the world we live in now.

If I were starting something from scratch I'd do Bazel for internal deps, Nix for external ones. If that's a tall order, so be it: we train people. Investments in determinism usually end up paying off.

modern build systems > Autotools, even Makefiles > C/C++'s header system with conditional inclusion

Wouldn't it be more accurate to say something like "older build systems"? I don't think any of the things you listed are "modern". Which isn't a criticism of their legacy! They have been very useful for a long time, and that's to be applauded. But they have huge problems, which is a big part of why newer systems have been created.

FWIW, I have been using pants[0] (v2) for a little under a year. We chose it after also evaluating it and bazel (but not nix, for better or worse). I think it's really really great! Also painful in some ways (as is inevitably the case with any software). And of course it's nearly impossible to entirely stomp out "genrules" use cases. But it's much easier to get much closer to true hermeticity, and I'm a big fan of that.

0: https://www.pantsbuild.org/

The internal build tool is Blaze (Bazel is the open-source cousin).

I was under the impression that Blaze is Google's Bazel deployment, i.e. that they're the same code. Is that not correct?

Google gets to decide what platform they want to support for their backend. I wonder what their build code would look like if they tried to support the top-10 common OS from the last 30 years.

Ugly and broken build systems are most of the reason why Gentoo's 'sandbox' feature and equivalent for other distributions exists.[1] These sandboxing features have mostly been used in the past to prevent an ugly shell script in the build process for libuselesscruft from doing something similar to "rm -rf" on the build system. More recently these sandboxing features are helpful in encouraging reproducible builds by alerting maintainers to build processes which try and obtain non-deterministic information from the operating system environment such as the operating system name and version, host name and current time stamp.

There are a few gaps I think xz-utils highlights:

- Repositories containing a mixture of source code, build scripts, test frameworks, static resources and documentation generation scripts are all considered to be a single security domain with no isolation between them. If you look to Gentoo's src_prepare function as an example, we perhaps should instead split this into build_src_prepare, doc_src_prepare, test_src_prepare and install_src_prepare instead. If source code is being built and binaries generated, the sandboxed build directory should perhaps not contain test files and documentation generation scripts. If the package is being installed with "make install" (or equivalent) then static resources (such as a timezone database) should be available to copy to /usr/, but build scripts used to generate the binaries or documentation do not need to be available to "make install" (or equivalent).

- Sandboxing used for package building hasn't traditionally been implemented for security reasons in the past. Sandboxing should perhaps be hardened further with modern and more complex approaches such as seccomp to further protect build systems from the likes of libbackdoored that are targeted towards package maintainers/Linux distribution build systems. As a further example to seccomp, Gentoo's 'sandbox' has Linux namespace isolation built in, but not yet enabled whilst it is tested.

- There is a lack of automated package management tools (including dashboards / automatic bug creation) for comparing source trees in Git to released tarballs and making discrepancies more visible and easier for maintainers to review.

- There is a lack of automatic package management tools (including dashboards / automatic bug creation) for detecting binary and high entropy files in source trees and confirming they are validly formatted (e.g. invalid tag in a TLV file format) and confirming that test and example files contain nothing-up-my-sleeve content.

There has already been an accelerated move in recent years towards modern and safer build systems (such as meson and cargo) as 80's/90's C libraries get ripped out and replaced with modern Rust libraries, or other safer options. This is a lot of work that will take many years though, and many old 80's/90's C libraries and build systems will be needed for many more years to come. And for this reason, sandboxing/isolation/safety of old build systems seemingly needs to be improved as a priority, noting that old build systems will take years or decades to replace.

[1] https://devmanual.gentoo.org/general-concepts/sandbox/index....

[2] https://devmanual.gentoo.org/ebuild-writing/functions/src_pr...

Makefiles are not a Linux thing, but Unix.

And if you confuse C with C++ and Make with CMake/Autotools/Autoconf, you have a lot to learn. Look: simple and portable makefiles work:

git://bitreich.org/english_knight

Is it though? The attacker probably has Linux x86 target(s) in mind and IFUNC support isn't guaranteed to work with other platforms.

I also thought it was odd. There's also different random bytes (not random text, actual random bytes) prefixed to the start the scripts, but the bytes are prefixed by a hash symbol, which comments them out, so they don't affect the script. It seems intentional, but I can't think of why they would be there. I thought maybe xz would skip compression if the input was short/not complex enough or something, so they were added to pad the size, but removing them and re-compressing with xz seems to properly compress it, none of the original plaintext is in the compressed archive bytes.

One thing I noticed while trying to reproduce the exact bytes included in the .xz file committed to git is that the script's xz stream doesn't seem to be compressed by any of the default xz presets, I was only able to reproduce it by using `xz --lzma2=dict=65536 -c stream_2`. All the default numbered presents chose a different dictionary size. Another odd seemingly intentional choice, but again I don't understand the reasoning.

It can be to make space for script changes: you may overwrite the first bytes of the script.

Or just add some laziness.

Maybe if run on non Linux it could be found out either by crashing or leaving some trace because of OS differences

Can it be to enlarge/or obfuscate parts of the compressed test file? Perhaps without the repetitions the compressed file has some strange binary triggering some security or antivirus software?

No thanks.

That's not going to help, and will be fairly easy to circumvent for nation state actors or similar advanced persistent threats who will not have a problem adding an extra step of identity theft to their attack chain, or simply use an agent who can be protected if the backdoor is ever discovered.

On the other hand, the technical hoops required for something like that will likely cause a lot of damage to the whole open source community.

The solution here is learn from this attack and change practices to make a similar one more difficult to pull off:

1. Never allow files in release tar-balls which are not present in the repo.

2. As a consequence, all generated code should be checked in. Build scripts should re-generate all derived code and fail if the checked in code deviates from the generated.

3. No inscrutable data should be accessible by the release build process. This means that tests relying on binary data should be built completely separately from the release binaries.

It might prevent attacks under different aliases, but a determined organization will be able to create a verified account, if only because nobody, certainly noy github, has the will and means to verify each account themselves.

Two problems with this:

1. Many important contributors, especially in security, prefer to be pseudonymous for good reasons. Insisting on identity drives them away.

2. If a spy agency was behind this, as many people have speculated, those can all manufacture "real" identities anyway.

So you'd be excluding helpful people and not excluding the attackers.

The lesson I take away from this incident is that we probably shouldn't be allowing anonymity for core contributers in critical open source projects. This attack worked and the attacker will likely get away with it free of consequence, because they were anonymous.

This would be impossible to enforce, and might not be a good idea because it enables other ranges of attacks: if you know the identities of the maintainers of critical open source projects, it’s easier to put pressure on them.

The attack almost worked because of too few eyes

If this was a state-actor (which it definitely looks like it) then what validation are you going to do? They can probably manufacture legitimate papers for anything.

Driver’s license, SSN, national ID, passport, etc. If the government is in on it then there’s no limits.

The only way would be to require physical presence in a trusted location. (Hopefully in a jurisdiction that doesn’t belong to the attacker…)

Who designates it as critical?

If someone makes a library and other people start using it, are they forced to reveal their identity?

Do the maintainers get paid?

And I am 100% certain that most of their dev teams do not spend time scanning these for obscure injection vectors.

This rationale baffles me, it feels that the dependency-hell circlejerk crowd is working on making OSS maintainers look even more bad with this scenario.

Any given commercial operation that claims any credibility for itself does supply chain analysis before adopting a dependency. This is, among other things why ordinarily you'd pay RedHat to maintain a stable Linux Release for you and why projects such as FreeBSD severely limit the software they ship in the default install.

If you are affected by this mess, I'm sorry to say, but it's your fault. If you are worried about developers of software you use for free, as in free beer, going rogue, either put in incentives for them to not do that (i.e. pay them) or fork the project and implement your own security measures on top of what's already there.

If you're worried that you could encounter exploits from dependencies in commercial software you use, you should negotiate a contract that includes compensation from damages from supply chain attacks.

If you're unwilling to do that, sorry mate, you're just unprofessional.

Inb4: Yes, I am really trying to say that you should check the supply chain of even your most basic dependencies such as SSH.

That's why I don't see e.g. TP-Link basing their router firmware on OpenWRT as a win, and why I want the "vanilla" upstream project (or something that tracks upstream by design) running on my devices.

Applies to all of my devices btw. I don't like Android having to use an old kernel, I didn't like MacOS running some ancient Darwin/BSD thing, etc. The required effort for backporting worries me.

Don't get me wrong, I'm not saying OSS has no vulns.

there will be hundreds of cases like it uncovered in the next few months, I am certain of it.

"Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system. Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the "fixes" mentioned above." (https://www.openwall.com/lists/oss-security/2024/03/29/4)

So, it's like story of those security researchers injecting bugs into the kernel https://thehackernews.com/2021/04/minnesota-university-apolo...

I'm saying this isn't that easy to pull off, and it's unlikely we'll see hundreds of similar cases.

It's pretty common to run tests on CI after building to verify your particular setup doesn't break stuff.

Last time we were doing that we were preferring git upstream, though, and generated autocrap as needed - I never liked the idea of release tarballs containing stuff not in git.

Because, despite containing some amount of generated autoconf code, they are still source tarballs. You want to be able to run the tests after compiling the code on the destination machine.

Your best bet may be in the chat (from https://www.openwall.com/lists/oss-security/2024/03/30/26 ):

Matrix: #xz-backdoor-reversing:nil.im

IRC: #xz-backdoor-reversing on irc.oftc.net

Discord: https://discord.gg/XqTshWbR5F

I agree, I haven't seen anything about decompiling the object file.

If I had a project to develop a backdoor to keep persistent access to whatever machine I wanted, it would make sense that I would have a plug-in executable that I would use for multiple backdoors. That's just decent engineering.

It is all broken. The quick hack to make things kind of work today, compounded across fifty years and a million developers.

There are occasional shining lights of stuff done right but I think it's fair to say they're comprehensively outcompeted.

This is probably the worst take ever.

It's pretty amazing how the tools keep up with the increasing complexity of the products we make.

And to be honest, in most cases just make it simpler, I think people just don't like to learn new stuff.

Second this.

And in the event someone is pedantic enough to actually care: that person will be considered a pariah that all development stifles due to.

Tensions with the team for nitpicking etc;

FD: I have a situation like this now, I am not the one being picky- one of the developers I hired is. I had to move him out of the team because unfortunately his nitpicky behaviour was not well regarded. (he also comes from eastern europe and has a very matter-of-fact way of giving feedback too which does not aid things).