return to table of content

XZ backdoor story – Initial analysis

dvsfish
86 replies
15h48m

Great read - the lengths the attacker went to to not only hide it but also build the trust to be able to pull this off is incredible. So calculated. One can only imagine they felt a great deal of panic and despair as the security community lit up the issue, regardless of a near zero chance of repercussions. Very clever - those involved in catching this are, quietly, heroes

jerjerjer
38 replies
15h23m

The use of trie structure to compare strings without hardcoding raw, searchable string values is very inspired.

But yes, some black hat had a really really bad day: being so close to owning any ssh-enabled recently-updated box and when they can finally see a finish line it is just poof... gone in a day. It was two years of effort from at least one guy and more likely a team.

lenerdenator
8 replies
14h5m

I think it goes beyond just this particular attempt. I bet they were looking for other targets, targets that are widely-used but maintained by resource-deficient projects. They wanted to do maximum damage with the fewest possible supply chain attacks.

The fact that many projects are now searching deep and wide through their commit histories, looking at just who they're taking code from, beginning to develop frameworks for attack mitigation and remediation... an entire type of previously very-promising attack is completely burned. This has been a massive defeat. And it happened all by chance.

fauigerzigerk
4 replies
11h17m

>an entire type of previously very-promising attack is completely burned.

I fear it's not just the attack that is burned. If new contributors have to be distrusted and/or go through some sort of vetting that isn't based on the merit of their contributions, that is a terrible blow to the entire open source movement.

The threshold for young coders without a great deal of history and without a network of contacts to become contributors to important open source projects has just gone up massively.

Brian_K_White
3 replies
8h55m

Nah. Hardly going to change anything for the worse, only for the better.

The bar is still trivial. The worst part is that it will be difficult to be anonymous. There are a lot of valid, even life & death reasons for anonymity, and that will now be much harder.

So the loss is all the contributors who don't dare let their government or employer see them helping the wrong projects, or in some cases don't want to disclose that they are any good at coding or particularly interested in security etc.

bradleyjg
1 replies
7h9m

I bet a bunch of employers don’t want any unauthorized contributions to open source. For governments it seems much more niche—-only a few specific projects would raise red flags.

tw04
0 replies
6h21m

The average business still isn’t going to have any clue who GitHub user coderdude57 is in real life. But coderdude57 may be forced to hop on a video meeting to prove who he is to a project lead before his code is accepted.

cqqxo4zV46cp
0 replies
7h55m

“GitHub Verified”.

mgaunard
2 replies
6h12m

The main vector of attack was patching the code as part of running tests.

I would expect to see more projects try to protect against this in general by attempting to separate the building and testing procedures.

cesarb
1 replies
5h51m

The main vector of attack was patching the code as part of running tests.

No, it was not.

The test files were used as carriers for the bulk of the malicious code, but running the tests did nothing malicious; the supposedly corrupted test file did act as a corrupted file in the test. What extracted and executed the code hidden in these test files was a bit of code injected in the configuration steps, which in turn modified the compilation steps to extract and execute more code hidden in these test files, which in turn extracted and injected the backdoor code. This all happened even if the tests were never run.

I would expect to see more projects try to protect against this in general by attempting to separate the building and testing procedures.

The suggestions I've seen on this line of thinking were to not only separate the building and testing steps (so that the testing steps cannot affect the output of the build step), but also to remove all test files before doing the build, and adding them back for the testing step. The important part being to remove all binary test files before doing the build, and having two separate steps only as a side effect of not being able to test without putting the test files back.

I don't think this approach is going to be adopted, as it's too much work for little gain; people are focusing more on making sure all files on the tarball used for the build either come from the corresponding tag in the repository, or can be proven to have been deterministically generated from files found in the corresponding tag in the repository. That would have either caught or ignored the key file added to the tarball which started the extraction and injection steps.

mgaunard
0 replies
1h57m

It's injected as part of configuring the build of the tests.

Just make the two separate projects, where the test project depends on installed binaries.

issafram
8 replies
13h23m

It was most likely done by a government. Not just one hacker.

gizmo686
7 replies
12h2m

Governments are made up of people. There is likely at least 1 real person (if not more) for whom this attack has been the entirety of their professional life for years.

Brian_K_White
6 replies
8h48m

If so, then they were paid for those hours and are perfectly whole right now. Maybe their career takes a dent but I do not weep for whoever this was no matter which government they worked for, unless it was so bad they were actually a slave and forced to do this against their will.

The government who paid them just lost the investment, but who cares about them? It's only a good thing if this does not get a reputation for being a good invesment.

If it was a criminal, then the same as the government they lost the investment but again that's only a good thing.

cesarb
2 replies
7h45m

> and when they can finally see a finish line it is just poof... gone in a day. It was two years of effort from at least one guy and more likely a team.

If so, then they were paid for those hours and are perfectly whole right now.

People are not ideal money-fed automatons. Even if you're fully paid for your software development, it feels bad to see all your work thrown away.

Brian_K_White
1 replies
7h38m

If only I had also said something like "maybe their career takes a dent but I do not weep for anyone who worked on this" to indicate that I understood that obviosity.

cesarb
0 replies
6h56m

> People are not ideal money-fed automatons. Even if you're fully paid for your software development, it feels bad to see all your work thrown away.

maybe their career takes a dent but I do not weep for anyone who worked on this

Even if they were fully paid, and even if their career is not affected at all (or even improves, "I wrote the xz backdoor" would be an impressive line in a curriculum if you disregard the moral aspects), it can still feel bad to see your work thrown away so close to the finish line. People are not career-driven automatons.

But I agree with you, I do not feel bad for the author of this backdoor; whoever did that does deserve to see this work unceremoniously dumped into the trash. But I can understand why the ones who worked on that code would feel bad about it.

cqqxo4zV46cp
1 replies
7h50m

Are you being argumentative for the sake of it?

The statement was that someone had a really bad day.

The implication that people are driven by money and nothing else, or even that they have no right to feel like they had a bad day if they were paid, is absurd.

Nobody is saying that you should be sympathetic. It’s just an interesting comment: An interesting thing to think about. A worthwhile contribution to the conversation. This was someone’s very bad ending to a very long project.

Brian_K_White
0 replies
5h45m

Ok, true. It was a bad day.

thaumasiotes
0 replies
3h28m

Maybe their career takes a dent

I don't see why; news coverage has pretty uniformly taken the view that "this guy was an evil genius, and we escaped his incredibly well-executed plot by the skin of our teeth".

jlmorton
6 replies
8h35m

It was two years of effort from at least one guy and more likely a team.

You think these guys have standups?

jaza
4 replies
8h30m

Gonna be one hell of a retro...

kevindamm
2 replies
7h57m

I wonder if they do blameless post-mortems..

rnts08
0 replies
6h27m

With the likely culprit they won't get blamed just a bullet in the back of the head and a bill for it sent to the family.

ankitg12
0 replies
7h4m

Blame it on outside party.

lovasoa
0 replies
2h49m

The project manager who said performance issues would be fixed in v2 is going to have a hard time.

saagarjha
0 replies
7h28m

Yes.

jumelles
5 replies
15h14m

Honestly? Good. Anyone doing this sort of work willingly deserves worse.

optimalsolver
4 replies
11h27m

That they were harassing and manipulating a lone, unthanked maintainer who had already told them he was dealing with mental issues makes them evil, IMO.

The honorable thing for "Jia" to do after this epic failure is seppuku, or whatever his or her local equivalent is.

cqqxo4zV46cp
3 replies
7h45m

Nobody sees themselves as the bad guy, and that’s not the same as “some people are just fundamentally selfish”. There are definitely loads of people that’d feel like the end justifies the means. There are plenty of people for whom a workday involves doing far worse things for the world than cyberbullying one person, and will look you in the eye and justify it. Plenty of that stuff is socially acceptable in many many mainstream circles. “Being mean to a maintainer” is just one that this community is especially sensitive to, because it involves a highly personified victim that they can relate to.

optimalsolver
2 replies
6h45m

These maintainers add vast amounts of value to the modern world, though most of the people that benefit indirectly from their work can't really conceive of what it is they do.

People like "Jia" are pure parasites. It's one of the best cases of "why we can't have nice things" I've ever seen.

qweqwe14
1 replies
2h55m

Yeah, they add vast amounts of "value", including (accidentally) reinforcing the status-quo.

It would definitely be interesting to see what would happen if the attack wasn't noticed, but instead people focus their interest on attacking Jia Tan because "wow, that guy is one hell of an asshole, it sure is a great thing that he failed!".

Whether or not this attack was the rare one that failed out of many similar ones is largely irrelevant to people. Quick, discuss this one particular case where we noticed it, news flash and all.

People like "Jia" are pure parasites

They are "parasites" because they don't do what they are "supposed to"? That's pretty crazy. I guess every person that's doing what matches their interests is somehow a bad person/parasite/etc. Or is that only if what they do is forbidden by some rule book? Do you see what I'm getting at here?

"pls sir dont create linux backdoar"

optimalsolver
0 replies
1h2m

I guess every person that's doing what matches their interests is somehow a bad person/parasite/etc

If they inflict harm on their fellow humans in order to do so, then literally yes.

There's a reason why a society will collapse if you have too many of this type of person.

Terr_
3 replies
11h47m

The use of trie structure to compare strings without hardcoding raw, searchable string values is very inspired.

Aside, this is one of my grumpy-old-dude opinions when it comes to regular non-malicious code: Source "greppability" is a virtue, one that must be balanced against "don't repeat yourself" and "compile time checking", etc.

Some examples off the top of my head:

1. One import/alias per line, no wildcards.

2. If you have to concatenate a literal string across multiple lines, try to ensure the break isn't inside an interesting substring someone would search for.

3. If you write a compile-time constant like foo=4*6*9, the comments around it should also contain the result. Sure, someone might screw up and forget to keep the comment in-sync, but it can also be invaluable when someone else is tearing their hair out trying to find out why the production logs are saying nothing but "Illegal frump: 216".

Don't get me wrong, I love a good IDE, and many will offer "Find Usages" or "Find Literal"... but it doesn't always work all the time, for all languages in the project at once, for everyone on a team who might use other tools, or during a PR review through a web-browser, etc.

ncruces
0 replies
7h50m

If you have to concatenate a literal string across multiple lines, try to ensure the break isn't inside an interesting substring someone would search for.

This is called semantic line breaks and is also important to get (admittedly more naive) diffs sensible.

https://sembr.org/

kubanczyk
0 replies
4h31m

4. Avoid complex regexes. E.g. prefer '>>>>>' over '>{5}'

btown
0 replies
5h47m

Along the lines of source greppability, at some point someone will try to replace successive a.foo_x, a.foo_y, and a.foo_z accesses with a loop over foo_{suffix}, thus breaking any attempt to grep for foo_y. That someone may be me! This should almost always be struck down in code review wherever possible. Being a bit more DRY isn’t worth it!

KeplerBoy
1 replies
12h54m

How much effort was it really? Yes, they worked on it over 2 years, but I guess it wasn't more effort than a few hours every other week (apart from engineering the actual exploit). After all putting in a full time effort as an unpaid contributor would be suspicious in itself.

Assuming they work 40 hours a week and are doing this in a team (presumably every major player has such a team or is scrambling to get one now), one must expect many potentially infiltrated projects out there.

NamTaf
0 replies
6h57m

“Apart from engineering” is doing some seriously heavy lifting here. Writing the actual code is likely just an afterthought by comparison. The engineering - identification of a target weakness, the design of the exploit chain, etc. - is overwhelmingly going to have been the lion’s share of the effort.

wjholden
0 replies
12h8m

I agree, I had never heard of using a trie as a means of hiding strings.

I'm not familiar with any programming language that provides tries in a standardized way. They're not so hard to code, so I wonder if this will become a trend in future malware.

inkyoto
19 replies
13h47m

The fundamental question remains: considering the in-depth knowledge of low-level (a small scale disassembler), system programming and the practical application of data structures (i.e. using a trie for all string operations to avoid the use of string constants), was it a state sponsored attack, or it was a one man show who did it for personal gain? Attempting to backdoor nearly every single Linux instance of sshd goes beyond the definition of just being brazen.

codezero
8 replies
13h34m

That nobody is talking about the person's IP addresses (the xz project was hosted on a personal server of the maintainer), or any details about their actions indicates to me it was a state actor and the original xz maintainer is cooperating with law enforcement to uncover their actions.

fl7305
1 replies
6h58m

Or someone further west who did this after work during the evenings.

Or someone further east with a classical hacker "let's get up late in the afternoon, and start the serious hacking late at night."

cesarb
0 replies
6h32m

From that article, "To further investigate, we can try to see if he worked on weekends or weekdays: was this a hobbyist or was he paid to do this? The most common working days for Jia were Tue (86), Wed (85), Thu (89), and Fri (79)." That makes it more likely this work was done during working hours; someone doing things outside of work hours would be more likely to produce the same amount (or more) on weekends and holidays.

tonypace
0 replies
6h37m

Moscow is the same time zone as Israel.

cdelsolar
1 replies
12h45m

I thought it was on GitHub pages?

codezero
0 replies
12h14m

Before pages the project was hosted on the server of the original maintainer.

willvarfar
0 replies
12h4m

Whilst the ip addresses and email headers etc should be examined meticulously, in the distant hope that they lead somewhere, the chances are that they won't. Very basic opsec.

rwmj
0 replies
11h40m

CISA is investigating it.

dataflow
4 replies
13h6m

The fundamental question remains: considering the in-depth knowledge of low-level (a small scale disassembler), system programming and the practical application of data structures (i.e. using a trie for all string operations to avoid the use of string constants), was it a state sponsored attack

I think you're missing where the difficulty is. I'd argue the technical knowledge here isn't the hard part here. You can find more than enough folks who would know how to do everything you listed outside of any state-sponsored context. The hard part is the methodical execution, including the opsec, backup plans, plausible deniability, false identities, knowing how to play the long game, being able to precisely target what you want, understanding human psychological tendencies, identifying unforeseen security vulnerabilities, etc... these are the things that I imagine are likely to distinguish talented individuals from state-sponsored actors, not something like "knows tries and x86 assembly".

And taking all those into account, this absolutely smells state-sponsored. I just see no way that a private entity had enough of an incentive, foresight, experience, and patience to play such a long game with such careful maneuvers.

bradleyjg
2 replies
6h59m

A lot of the filter is just limited time and energy. A college kid has plenty of that, but not the right skills. There are more than a small number of people with the right skills but they have day jobs. That’s what makes think whoever was working on this was being paid upfront to do so. That said an obsessive with an easy day job and not a ton of family responsibilities is always a possibility.

yodsanklai
1 replies
6h29m

There are more than a small number of people with the right skills but they have day jobs. That’s what makes think whoever was working on this was being paid upfront to do so.

Doesn't this apply to many OSS contributors? people with skills and free time?

bradleyjg
0 replies
6h27m

I’d guess that the vast majority of new open source code last year (weighted by loc and usage) was written by people being paid to do so.

inkyoto
0 replies
10h9m

I do not think I am missing out: the supply chain attack which included playing the long game, subverting the trust of the development community is the real issue that the open source community has no defences against – the thwarted attack has surpassed the scale of all previous supply chain attacks on NodeJs, Python and similar ecosystems and went deep down into the low level, technical layers as well.

The assault was comprehensive, holistic and systematic in its approach – this article does not mention it, but other reports have indicated that the person behind it also managed to compromise the PKI layer at the edge between OpenSSL and sshd which brings an extra level of complexity to the backdoor.

AtNightWeCode
4 replies
9h45m

I think it is a professional attempt. It does not have to be a goverment. There are other cyber criminals.

lmm
3 replies
9h32m

It does not have to be a goverment. There are other cyber criminals.

Criminals with the resources to maintain such a complex operation for two years?

trallnag
0 replies
8h7m

Hobbyist around the world maintain projects for years and decades

rvnx
0 replies
8h4m

Yes, crypto-boys for example. If the reward is 1B USD in crypto stolen from wallets why not do spend a year adding a backdoor ?

AtNightWeCode
0 replies
8h51m

Difficult but not impossible. Some of the malware and ransomeware stuff is also created by experts.

codezero
8 replies
13h35m

or they are totally nonplussed because they are not even a single individual, and a state actor who is doing this on 10-20 other projects for the past 5-10 years which haven't got the same attention. This is their job, being compromised is always a risk.

redox99
4 replies
13h17m

It doesn't matter if it's a team working for a state actor. You'd still be pissed your 2 year project failed so close to the finish line.

codezero
2 replies
13h7m

Not if this is one of a few dozen or few hundred similar ongoing operations. The risk is always there, they have to expect some amount of failure. Open source software is constantly being probed for vulnerabilities in every way possible, from random commits to high level members of committees for standards. Every operation is one in a grab bag of disposable and deniable efforts.

I also am a little biased and assume being burned in state-sponsored acts is similar to the no-blame culture of breaking infrastructure in tech :) because by all accounts this compromise was extremely well done, until it wasn't.

Also, we can't be sure the compromise wasn't intentionally telegraphed to cause some other action (using a different library) on purpose.

mimimi31
0 replies
7h20m

Not if this is one of a few dozen or few hundred similar ongoing operations. The risk is always there, they have to expect some amount of failure.

That actually makes me think it's not happening at a larger scale, since we'd likely have heard of at least a few similarly elaborate cases being uncovered by now. If not during the attempt itself, then at least at some later point in time.

Either almost all of these operations remain undetected, because they are even more sophisticated and much of the world's software ecosystem has been secretly compromised for years or there aren't actually that many such operations.

HPsquared
0 replies
8h13m

They might even get a kick out of people discussing how clever their hack was. A rare bit of public acknowledgement, in a way.

Brian_K_White
0 replies
8h32m

So what? The nature of the project matters. Seriously F all of the people who worked on it however many or few it was.

It's one thing to attack a target, it's quite another to try to give yourself a master key to every rpm or deb linux box in the world.

burntsushi
2 replies
6h14m

Unrelated note: I had always thought "nonplussed" was basically a synonym for something like "bewildering confusion." But the way you used it in this context suggested the exact opposite. It turns out that "nonplussed" has also come to mean "unperturbed": https://en.wiktionary.org/wiki/nonplussed

Quite confusing, because the two different meanings are nearly opposite to one another.

See also: https://www.merriam-webster.com/grammar/nonplussed

thaumasiotes
0 replies
3h18m

Quite confusing, because the two different meanings are nearly opposite to one another.

It's pretty easy to see where the innovative sense came from: "plussed" doesn't mean anything, but "non" is clearly negative. So when you encounter the word, you can tell that it describes (a) a reaction in which (b) something doesn't happen. So everyone independently guesses that it means failing to have much of a reaction, and when everyone thinks a word means something, then it does mean that thing.

You see the same thing happen with "inflammable", where everyone is aware that "in" means "not" and "flame" means "fire". (Except that in the original sense, in is an intensifying prefix rather than a negative prefix. This doesn't occur in many other English words, although "inflammation" and "inflamed" aren't rare. Maybe "infatuate".)

codezero
0 replies
3h9m

Wow, now I need to go figure what source material I misinterpreted to thoughtlessly use this word incorrectly. Thanks!

macrolime
4 replies
12h1m

I wonder then, whether the attacker has already started over with a new open source project, or found something else to do.

Or was xz only one of multiple targets where the overall plan didn't actually fail and an updated backdoor, without the performance issues, will soon be injected again through some other project.

peddling-brink
2 replies
11h58m

Started over? They’ve certainly been cultivating other unrelated sock puppet accounts that they’ve switched over to.

Brian_K_White
1 replies
8h21m

I thougt I read several years ago, like 10 or 15 even, that they could fingerprint people by communication patterns. Like how you can sometimes recognize an author just by the way they write, just by the text itself.

Surely that would be something the current ais could do 100x better?

So I wonder if there isn't a business there in scanning anonymous posts to associate them with others to deanonymize them all?

You could obviously use an ai to rewrite your text, but I bet there is something there that can still be correlated. Simply avoiding using favorite words and phrases doesn't change the essence of whatever you're saying.

I think the concept of sockpuppet is about to get complicated. Really it probably already is. If I can imagine something, someone else has probably already been doing it and countering it, and countering that for a decade.

rvnx
0 replies
8h6m

You can see what projects they are following on GitHub (notably rust compiler, some embedded platforms, etc)

grumpyprole
4 replies
10h40m

Those who caught it were indeed very clever. But the attacker did make mistakes, particularly the Valgrind issues and large performance regression. That ultimately is what raised suspicions.

glandium
1 replies
9h46m

The large performance regression was a feature of the string obfuscation.

grumpyprole
0 replies
7h18m

Interesting! So perhaps not really effective obfuscation after all!

hiisukun
0 replies
6h23m

"Large" is doing quite a bit of work here: the engineer who found it had disabled turbo boost and was, by at least a little serendipity, quite performance focused at the time rather than being busy on other elements of their release target.

INTPenis
0 replies
6h18m

This is why I think it all came falling down due to the teknoraver patch to libsystemd to replace lzma with libarchive.

Because anyone has the resources to test this locally and find the issues, but they just didn't have the time anymore.

cnvogel
3 replies
10h24m

Here’s what I don’t get: why the many layers of obfuscation in the build phase? (I understand why in the binary linked into ssh.)

Once the first stage, extracting a shell script from one of the „test“ data blobs, has been found it was clear to everybody that something fishy is going on.

It’s inconceivable that I’ve would have found the first stage and just given up, but then it was „only“ a matter of tedious shell reversing…

They could easily done without the „striping“ or „awk RC4“, but that must have complicated their internal testing and development quite a bit.

makomk
0 replies
9h6m

Reduces the attack area through which this could be found, I expect. Without all the obfuscation someone might spot suspicious data in the test data or at some other stage, but this basically forces them to find the single line of suspicious shell script and follow the trail to find the rest of the stuff added to the build process.

cesarb
0 replies
6h37m

It’s inconceivable that I’ve would have found the first stage and just given up

But what you were looking at might not be the first stage.

You might be looking at the modified Makefile. You might be looking at the object files generated during the build. You might be looking at the build logs. You might be investigating a linking failure. The reason for so many layers of obfuscation, is that the attacker had no idea at which layer the good guys would start looking; at each point, they tried to hide in the noise of the corresponding build system step.

In the end, this was caught not at the build steps, but at the runtime injection steps; in a bit of poetic justice, all this obfuscation work caused so much slowdown that the obfuscation itself made it more visible. As tvtropes would say, this was a "Revealing Cover-Up" (https://tvtropes.org/pmwiki/pmwiki.php/Main/RevealingCoverup) (warning: tvtropes can be addictive)

almostnormal
0 replies
10h0m

Here’s what I don’t get: why the many layers of obfuscation in the build phase?

For a one of its kind deployment it would probably not matter. However, deploying to multiple targets using the same basic approach would allow all of them to be found once one was discovered. With some mildly confsing but different scripting for each target systematic detection of others becomes more difficult.

Dalewyn
3 replies
14h14m

One can only imagine they felt a great deal of panic and despair as the security community lit up the issue,

Jia Tan mook: "Sir, they found us out."

Jia Tan officer: "What's the sitrep, are they coming for us? Are they getting our operatives in the field?"

Jia Tan mook: "No sir, they're just reverting and blacklisting our accounts and releases. Muttering something about 'uptime'."

Jia Tan officer: "Shit, we're dealing with sysadmins."

flomo
2 replies
13h48m

--

arrowsmith
0 replies
13h39m

I imagine that government security agencies are mainly thinking "shit, why didn't we think to do that?"

Dalewyn
0 replies
13h40m

If a government agency tasked with national security started caring about a critical home project by Nebraskaman only and long after Nebraskaman collapsed, that country needs a new government agency.

vlovich123
26 replies
16h39m

So if I'm understanding correctly, a useful hardening step would be to ensure that each dynamically linked library had it's own GOT & that the tables are marked as read-only once dynamic linking completes (i.e. you can't reach across dynamic boundaries to patch someone else's ifunc entries).

That would improve the supply chain security of code getting linked in somewhere but never executed.

EDIT: Or even better, perhaps ifunc should be implemented in a declarative fashion so that you can't just cause arbitrary code execution from each library you link against. That might be harder to implement at this point considering back compat, but probably is something that could be layered in over a longer time period (i.e. if you build any library with "declaratively linked ifunc" feature bit set, then the dynamic linker will force all linked libraries to have that feature flag or fail to launch).

jerjerjer
13 replies
15h30m

From another angle: build systems.

Currently building most libraries usually involves executing a series of highly complex byzantine scripts, requiring turing-complete environment. This gives attacker an endless attack surface, and when the build process is hijacked - well, the opportunities are there.

Moving to a declarative build process with only a limited state machine as an executor would help. Requiring all source blobs being reproducible might also be something to think about.

kimixa
9 replies
13h25m

Having a "simple" declarative build definition won't help if the thing that interprets and executes that definition is shipped with the package, as said interpreter is likely orders of magnitude more complex and harder to personally review. As is what happened with the xz example - the "attack" was hidden in code that is autogenerated by autotools, not the build or configuration definitions.

People put trust in distros and packagers to having something of a review chain - there's 0% chance you personally be an expert in everything executing on your workstation right now (outside of maybe toy systems). I'm not an expert in m4 or bash, but I hope enough experts are in the chain to get to my distro's package library are that such things are less likely. But that is all bypassed here.

I think this particular approach is a one-off, as I know of no other build environment where it's expected to have the generated executable of the build system "helpfully" packaged in the tarball as a packaging step.

If it is in some I'm not aware of, I hope that decision is being re-examined now.

bonzini
5 replies
5h18m

No, the attack was not hidden in code that is autogenerated by autotools. If that was the case, rerunning autoconf (which most distros do) would have disabled the attack.

Instead it was hidden in plain sight in the source that is processed by autoconf.

it's expected to have the generated executable of the build system "helpfully" packaged in the tarball as a packaging step.

While this is true, most distros as I mentioned above rerun autoconf to ensure consistency in the tests and possibly to include autoconf bug fixes.

rcxdude
3 replies
4h21m

The jumping off point was modified autotools output. re-running autogen did effectively disable the attack. The payload was stashed in some test files, but the build system needed to jump through quite some hoops to actually get that into the compiled library. Apparently the target distributions didn't re-run autogen for this package.

markhahn
1 replies
3h36m

merely keeping build and test environments isolated would have entirely prevented this attack.

better hygeine that way would also simplify both environments, since the tools for each are fairly different.

bonzini
0 replies
2h7m

How are they different? Both use make and the C compiler.

xz is not different in this respect from any other C, C++, Go or Rust program.

masfuerte
0 replies
3h59m

This is what the early reporting said but the article has additional info.

The code to include the backdoor in the build was in an m4 script.

The initial reporting said that this code was not present in the github source, but the post-autogen code (including the attack) was included in the github releases.

The article says that this modified script was present in the source on tukaani.org, which was controlled by the attacker and used by the distros as their upstream.

If you downloaded from github and reran autogen you were OK. If you downloaded from tukaani and reran autogen, like the distros did, you lost.

kimixa
0 replies
5m

the "build-to-host.m4" file seems to originally be from gnulib, and if that is installed on the system is not required by the build. So I see that as "part of the build system" myself.

I mean the github repository with exactly the same .ac and .am files works fine with local automake/autoconf generation without that file existing. And thus no backdoor (the test files are still there, but "harmless" without the initial kicking off point to actually de-obfuscate and include their contents)

MichaelZuo
1 replies
6h59m

This doesn't work for anything actually complex.

For example if you've ever taken a look at the bluetooth specs you would not trust a single person in the world to implement it correctly and you probably wouldn't even trust an arbitrarily large team to implement it correctly.

Unless they had a long demonstrated and credible track record of shipping perfectly functional products and an effectively unlimited budget, i. e. Apple and maybe 1 or 2 other groups, at most.

jancsika
0 replies
4h58m

For example if you've ever taken a look at the bluetooth specs you would not trust a single person in the world to implement it correctly and you probably wouldn't even trust an arbitrarily large team to implement it correctly.

I messed around a tiny bit with Bluetooth on Linux recently. Going to rankly speculate that Bluetooth is such a special case of hell such that it makes a distracting example here.

I mean, as a joke suppose we wanted to design a 3.5 mm patch cord that pitch shifts down a 1/4 step for randomly chosen stretches. It turns out to be easy-- just remove the wire from the casing and replace it with cheap bluetooth chips at either end. You'll get that behavior for free! :)

Compare to, say, USB, where your point above would apply just as well. I wouldn't be distracted by that example because even cheapo, decades-old USB drives to this day let me read/write without interleaving zeros in my data.

Shit, now I'm distracted again. Does Bluetooth audio even have the notion of a buffer size that I can set from the sender? And I guess technically the receiver isn't interleaving the signal with zeros-- it's adjusting the rate at which it sends blocks of the received data to the audio subsystem.

Was Bluetooth audio basically designed just for the human voice under the assumption we're constantly pitch shifting?

Oops, I almost forgot-- the sample-rate shift is preceded by a dropout, so I do get interleaved zeros in the audio data! I actually get a smoother ramp in my Prius changing from battery to ICE than I do in my wireless audio system!!!

Anyhow, what were we talking about again?

AtlasBarfed
0 replies
10h32m

It needs money and people.

Government funding for defense of the economy and computing in the "free world".

Certainly the defense department has a billion to spare, and so does the EU

nonrandomstring
1 replies
5h26m

Yes. Other than the vulnerability of developers/maintainers the other big takeaway I get from this incident is that build systems have become dangerously unwieldy. There's just too many moving parts and too many places to hide bad stuff.

tsimionescu
0 replies
4h28m

Build systems have always been like this. It is in fact more recent build systems that are limiting the amount of crazyness you can do compared to the older ones.

duped
0 replies
4h7m

This is how we got autoconf in the first place - an under specified and under powered build configuration language (make) which led to people that did need arbitrary logic to generate the build files at configuration time.

Don't limit the build system language. It makes builds more complex, not less.

saagarjha
3 replies
15h29m

Yes and no, but mostly no. This would prevent simple use of ifuncs in this way, but it's important to understand that the author of this could inject arbitrary code into the library that ends up in the address space of a sensitive process. At that point, all bets are off: it could remap the GOT as writable if it so chose, or (this is mostly here for the EDR people will certainly bring it up after reading this) if trying to do that is flagged as "suspicious" or the OS gains the ability to block such a transition, the injected code can subvert control flow in hundreds of other ways. It has arbitrary read/write, code execution, everything: there is no security mitigation that can stop it. If it so wishes it can leak private keys and send them to the attacker directly. It can spawn a shell. Trying to design protections at this stage of compromise is a fool's errand.

tux3
2 replies
9h33m

it's important to understand that the author of this could inject arbitrary code into the library that ends up in the address space of a sensitive process. At that point, all bets are off: it could remap the GOT as writable if it so chose, or (this is mostly here for the EDR people will certainly bring it up after reading this) if trying to do that is flagged as "suspicious" or the OS gains the ability to block such a transition, the injected code can subvert control flow in hundreds of other ways.

I think it's important to push back on a very specific point here. It's true in general that if the attacker has added a backdoor in your library, and you're going to call that library's code, you've pretty much lost. Game over. Go home.

But this was a very different attack, in that the attackers couldn't directly target sshd. They couldn't manage to target any library that sshd calls directly either.

The library with the backdoor is code that was never actually called at runtime. This is important because it means it *doesn't* have a hundred ways to reach code execution. It only had a few select ways, mainly constructors and indirect function resolvers.

The amount of weirdness in glibc's runtime loader is not unbounded. There aren't actually a hundred places where it allows random libs to run code before main. And we should take a good look at those couple place that are clearly juicy high-value gadgets to attacker.

When glibc's runtime loader first loads a binary, first reaches a relocation for a STT_GNU_IFUNC, everything is still in a pristine state and no arbitrary code can run without being explicitly called. Attackers don't have magical powers that allow them to run code before you hand them the control flow. At this point in the runtime loader, an ifunc resolver cannot do anything without being caught. It cannot "just open /proc/self/mem" or "just call mprotect". It cannot "just disassemble the caller" or "just overwrite the GOT".

I really want to hammer home how different that is from letting the attacker run code after main. There's nothing you can do if you directly call a backdoored library. But we shouldn't let attackers get away with spending 400ms parsing and disassembling ELFs in memory in an ifunc resolver of all things. Overwriting the GOT like nobody's watching.

The backdoor isn't magic. For all the beautiful sophistication that went into it, it made many mistakes along the way that could have led to a detection. From valgrind errors to unacceptable amount of noise (400ms!) before main.

saagarjha
1 replies
7h17m

Well, they picked a target that doesn't get called directly, and found a way to sneak code into it without a static constructor. If that didn't work (and I don't fundamentally think it wouldn't–people aren't checking these very closely; the ifunc stuff is just obfuscatory bonus) they would target something that was directly used.

tux3
0 replies
6h29m

I would be happy with that result. Targeting something that's directly used by sshd means a much smaller attack surface. It's much harder for the attackers.

The danger with supply-chain attacks is that it could come from practically anywhere. Attackers can choose to target an overworked maintainer in a third-party library, and it's much easier for them than going after OpenSSH itself.

About the OpenSSH maintainers, they're known for being the paranoid amongst the paranoid. No one's infallible, but if attackers are forced to go directly after them instead of bullying smaller libraries, I'll have a reason to feel safer about the reduced attack surface :)

ajross
2 replies
15h23m

the tables are marked as read-only once dynamic linking completes

Alas this won't work. Dynamic linking is lazy, there's no moment when it's "complete". The correct function pointers get loaded and inserted into the table (in place of stubs) when called the first time, which can be arbitrarily far into the future. In fact in most large library ecosystems (gtk apps, etc...) most of the linked functions are never called at all.

lyu07282
0 replies
9h41m

in this case it takes advantage of the fact that sshd is compiled with early binding not lazy. The ifunc resolver function is called early, right after the dynamic libraries are loaded. It's eager binding (that's what LD_BIND_NOW=1 / -W,-z,now are doing) is a security feature, GOT table will be readonly early. Didn't help with security in this case lol

candiddevmike
1 replies
4h38m

Are there any programming languages that can sandbox library imports?

pcwalton
0 replies
1h42m

Not really, except for a few researchy object capability based systems. (This is kind of the promise of ocap systems: functions you call can only use what they're given and nothing more.)

If you don't trust a library, you can run it in a separate sandboxed process (or app domain, or wasm container, or whatever your language provides).

afh1
1 replies
6h39m

If you build for the host architecture you can completely disable ifunc without losing anything. In Gentoo it is common to build with -march=native, and disabling ifunc is as simple as setting -multiarch in glibc's USE flags. I've seen no negative impact from it.

markhahn
0 replies
3h32m

goodie! more, and less-controlled build environments to compromise, what could possibly go wrong?

BobbyTables2
0 replies
14h6m

And yet such is all trivially defeated by writing read-only memory using /proc/self/mem.

Even a Rust program with no unsafe code can do this.

Patching readonly libc code from Python without external modules is also trivial, resulting in arbitrary execution!

asveikau
21 replies
15h33m

I saw the xz repo is back on GitHub, Lasse and a new contributor were cleaning it up. They removed ifunc support and they committed code to generate test files into the repo, so that they can do test files without blobs. It looks like they're doing good stuff.

saagarjha
20 replies
15h28m

a new contributor

Jia Tan, is that you?

willsmith72
18 replies
14h13m

I know part of the fun of oss is anyone can contribute behind an anonymous github username, but is this a wake up call for the package owners?

Do we need at least some form of basic identity verification (even in the form of an employer link, LinkedIn, or similar)?

CiPHPerCoder
6 replies
13h51m

I've argued in a blog post [1] that we need to delineate between "open source developer" and "supplier". If we don't do that, calling thankless unpaid volunteers and hobbyists a "supply chain" is kind of insulting [2].

I don't believe that "identity verification" for F/OSS developers is a good idea. Suppliers? Sure. That can be a contract negotiation when you decide how much you pay for it.

Also, I don't think identity verification helps when your adversary is a nation state, which can just falsify government identification if it suits them.

[1] https://scottarc.blog/2024/04/04/open-source-supply-chains-a...

[2] https://crankysec.com/blog/supply/

xcrunner529
5 replies
13h43m

Just because it can by beaten doesn’t mean making it harder isn’t useful. This person/team used a VPN. Masking your location is a big red flag for just dev work like this. These things could be exposed in UI.

CiPHPerCoder
3 replies
13h39m

Just because it can by beaten doesn’t mean making it harder isn’t useful.

Fair.

This person/team used a VPN. Masking your location is a big red flag for just dev work like this. These things could be exposed in UI.

I disagree strongly, and am surprised to hear this argument on Hacker News of all places.

jamespo
1 replies
9h13m

The VPN is just part of the picture (sock puppet accounts complaining about speed of dev, no meaningful history of other contributions from the dev, no trusted "personal network" for the dev, etc) that in hindsight should have raised red flags.

markhahn
0 replies
3h21m

new project idea: OpenBackgroundCheck

volunteer osint researchers attempt to dox any identity you submit, so you can know whether they're the Right Kind of OSS contributor or not.

/s

ogurechny
0 replies
12h46m

People are so used to see artificial bureaucratic structures as more real than their real counterparts that they constantly invent such naive solutions. “Just make the gub'ment provide an official paper (with a stamp) that Joe Random Dude is a real developer, a father of two, not a fan of satanic metal music, and the project will be safe”.

asveikau
0 replies
1h55m

Most people you interact with electronically, you don't even bother trying to see if they're using a VPN or make any attempt to geolocate them.

redox99
3 replies
13h14m

State actors (probably the case here) can handle all that stuff.

danieldk
2 replies
10h50m

If an identity (known or pseudonymous) was signed by a government, there would at least be some accountability. Like if Jia Tan's identity is signed by government X, then there is a good hint that they are a state actor for X or otherwise there is a burden for X to come up with convincing evidence that this is not the case.

Obviously, every system is going to have weaknesses, but it would at least introduce some accountability.

The main issues are all related to privacy. What if your government is not nice and you don't want them to track all your contributions?

wasmitnetzen
1 replies
10h15m

Let's say we would have definite proof that this was a Chinese op. Then what? No other government is gonna complain too hard because they know that their own agencies are doing the very same thing too.

Additionally, creating fake identities in other states is base level spying.

danieldk
0 replies
8h6m

No other government is gonna complain

Maybe not. But to the open source community it does provide more information. Now it's completely unclear who compromised xz, if the account had a government-level signature of some sort, more is known. Also, the open source community could have different levels of trust in authors with different signing authorities [1].

Additionally, creating fake identities in other states is base level spying.

We are not talking about a fake identity here (as in a fake passport or whatever), but getting a signature from a device that is presumably in a facility that is disconnected from the network. The user's identities would also live on a hardware device (e.g. chip embedded in a passport/id).

You could steal the hardware device to steal someone's developer identity. But the identity can be revoked once the device is reported/stolen. So, it is hard to play the long game with a stolen identity, outside blackmailing or otherwise compelling a developer to make their identity available (but even in that case it'd give some leads in understanding who the actor is).

Apple Developer signing keys are similar. You can verify where an app comes from. If a developer goes rogue or their key is compromised, Apple can revoke it. Apple could do evil stuff by creating sock puppet accounts, but that would devalue the trust in their signing key.

---

[1] I should note that I am not in favor of this. It's a privacy nightmare and it would give governments a lot of control over developers. I just think the argument upthread that a state actor could fake anything is not true. If e.g. the US had such an identity signing system, a foreign power could not forge the identities as long as the US private key is not compromised.

usefulcat
0 replies
13h34m

Do we need at least some form of basic identity verification

For dedicated attackers (as was the case here) that will move the goal posts ever so slightly farther away, but not much else. I see how it's tempting to focus on identity, but I don't see how it's productive.

some_furry
0 replies
13h58m

No. I love contributing as my fursona sometimes. I don't want to dox myself to do so.

We shouldn't need to know who you are to be verify that your contribution is trustworthy. Code should stand on its own merits. Ideas like this tend to dovetail into "real name policies", which suck https://www.eff.org/deeplinks/2014/09/facebooks-real-name-po...

maxbond
0 replies
6h36m

Identity is a red herring. Backdoors are intentional vulnerabilities and are discoverable through vulnerability analysis.

You can't solve identity, and real, verified people can still betray you. Even trustworthy people can betray you, either unwillingly or unwittingly.

You can't solve vulnerability analysis either (it's the halting problem), but you also can't shirk it. You might as well go all in on it rather than undermining the basis of open source collaboration and many eyes making bugs shallow.

But it certainly is a wake up call of some kind.

logro
0 replies
7h47m

For a state level attack, that wouldn't even help. They'd just use state level forged documents.

cesarb
0 replies
7h19m

Do we need at least some form of basic identity verification (even in the form of an employer link, LinkedIn, or similar)?

Not all free software contributors are employed. They might be too young (it's not unusual to start contributing to free software while still in school), too old and already retired, be part of a family where another family member is the working one, do freelance or informal work, or be searching for work at the moment.

And even those who are employed do not necessarily have it visible on their employer's website; for instance, I don't think my current employer has any publicly available page listing all employees, and I never had (and don't have any desire for) a LinkedIn account.

ahoka
0 replies
3h52m

some projects are asking for physical id and signing a contract since decades

johnwheeler
20 replies
9h55m

Here’s a twist - with the amount of effort the attacker went through to evade detection in their scripts and code, this whole project could function as a diversion or at least fallback to one while multiple efforts go on simultaneously.

How do you get one step ahead of that? Does the community focusing on SSHD impact some other part of the system as a whole? Some other technological aspect? Social?

Tin foil hats are fun!

bandrami
8 replies
7h22m

I'm not optimistic. Nobody ever even audits the binary contents of flatpaks on flathub (were they actually built from the source? the author attests so!). You don't really even need a fancy delivery mechanism like this backdoor had.

smallerfish
6 replies
7h13m

It seems like 75% of people are comfortable installing latest & greatest developer tools with `curl > bash`, so it's no wonder that flatpak and similar have momentum. Really, we desperately need a better sandboxing UX in linux.

christophilus
2 replies
6h48m

That is better to me than installing a random binary. You can trivially audit the curl. Also, the curl > bash thing usually comes from the project maintainer (as in the case of Bun) vs some rando flatpacker.

smallerfish
0 replies
6h27m

Not if the bash part then turns around and installs a random binary, which is often the case. How do you know how thoroughly it's been audited by third parties? Does it have a security patch mechanism built into it?

dTal
1 replies
5h25m

"Sandboxing" is an understandable kneejerk reaction to the threat of untrustworthy software, but I think it's the wrong one - the classic "technical solution to a political problem". Software must compose. After all, how would sandboxing have helped here? You cannot sandbox sshd!

In fact we already have a very good solution, and it's simply the infrastructure of a typical Linux distro (replicated by package managers on other systems, such as F-Droid). It's quite an impressive feat really - a diaspora of independently trusted organizations, each with their own governance structures and PGP-enforced webs-of-trust, crafting public repositories of known-good software built from publicly published source code using publicly published build scripts, delivered to the user over strong encryption. On the whole this does an incredible job of keeping malicious or subversive software out, and we've all got so comfortable with just being able to "apt-get install <foo>" without worrying about malware that we've forgotten what a dangerous business running software from the internet used to be.

Which is why this xz attack is so unusual and surprising - it's like the police stopping an attempted murder in the nick of time in a sleepy crime-free town. As shocking as it is, we must ignore the cries of "something must be done" and resist the temptation to put up barbed wire fences all over our nice town.

smallerfish
0 replies
5h8m

I very much like distros, but unfortunately they are by nature slow to adopt new versions.

For example, Ubuntu 24.04 will ship without Deno. It will also ship with Cargo 1.75, which is too old to build Deno (there's some weird packaging issue). So, anybody who wants Deno on Ubuntu has to install it using either `curl | bash`, nix (which of course seems promising), or asdf (which I know nothing about). Most devs would choose curl|bash.

Why is Deno not packaged for Ubuntu/Debian? Because it's "hard", and it's hard in different ways for different distros (the debian ticket for this has sat without action for 2-3 years, and I think the Deno ticket a similar amount of time). Cross-distro solutions like flatpak (or nix) are 100% necessary if we are to get rid of curl|bash. But, the sandboxing story with flatpak/snap is currently pretty bad; apps just fail in mysterious ways if they try to access resources they're not allowed access to. Nobody wants Windows Vista, but it has been done reasonably well in various other OSes.

moooo99
0 replies
6h10m

I think Flatpak being a open source effort for a distro independent distribution channel is already pretty dope UX. Having this verification would sure be nice, but Verified publishers now being a thing is also an advantage over some other formats being used

Intralexical
0 replies
6h16m

Nobody ever even audits the binary contents of flatpaks on flathub (were they actually built from the source? the author attests so!).

IME/IIRC There aren't (or shouldn't be) any binary contents on Flathub that are submitted by the author, at least for projects with source available? You're supposed to submit a short, plain-text recipe instead, which then gets automatically built from source outside the control of the author.

The Flathub service then uses the manifest from your repository to continuously build and distribute your application on every commit.

https://docs.flathub.org/docs/for-app-authors/submission/#ho...

Usually the recipes should just list the appropriate Git-based URLs to get the source code, or, for proprietary applications, the official .DEBs. Kinda like AUR, but JSON/YAML. Easy to audit if you want:

https://github.com/orgs/flathub/repositories

Plus, if properly configured, Flatpak is supposed to securely sandbox every application anyway, although other systems it needs to be compatible with (E.G. X11) currently prevent that.

mkup
3 replies
7h36m

My idea of getting one step ahead of similar backdoors is to wrap sshd traffic into a spiped tunnel. Spiped is compiled from source and linked statically (last stable version from 2021).

markhahn
1 replies
3h28m

jeez, why not just compile ssh static?

mkup
0 replies
9m

I still want sshd updated by my OS package manager. Statically linked spiped, which is out of scope of my OS package manager, is just a second line of defense.

cpach
0 replies
5h23m

spiped is solid. Wireguard is another good option that is included in the Linux kernel.

darthrupert
2 replies
6h47m

1. Have multiple LLMs scrutinize every pull request that is created in the public.

2. Stop accepting shit tools like M4 and C for creating any software.

You can never make the work of black hats impossible but you can trivially make it a whole lot harder.

uecker
0 replies
2h4m

I am quite happy with the quality of C. M4 not so much. In terms of supply chain attacks, cargo is the worst I have ever seen.

Intralexical
0 replies
5h51m

1. Have multiple LLMs scrutinize every pull request that is created in the public.

---

You:

Below is a hexdump of a test file added to our code repository. Is it any cause for concern?

    ```
    0000000 375   7   z   X   Z  \0  \0 004 346 326 264   F 002  \0   ! 001
    0000010  \b  \0  \0  \0 330 017   # 023 001  \0  \f   #   #   #   #   H
    0000020   e   l   l   o   #   #   #   #  \0  \0  \0  \0 022 210 337 004
    0000030   Y   r 201   B  \0 001   %  \r   q 031 304 266 037 266 363   }
    0000040 001  \0  \0  \0  \0 004   Y   Z 375   7   z   X   Z  \0  \0 004
    0000050 346 326 264   F 002  \0   ! 001  \b  \0  \0  \0 330 017   # 023
    0000060 340 005 026 001   _   ]  \0 005     245   -   U 273 230 004   |
    0000070 306 231 274 240   f   M 315   1 255  \v   )     270   (  \f   .
    0000080 274   3 263   ~   * 323 312   n 324   C 034   Q   Y 314 217 341
    0000090 350 343 357   @ 227 271 247   w 026 255 237   U   *   H 367 240
    00000a0   K 260   p   9   5 370   j 017 203   ? 307   ] 005 326 220 240
    00000b0 352   I 236   `   ? 304 306 365 246 375 226   [ 257   a 177   -
    00000c0 261 033 204 001 203 251 346 252 243 244   < 016   h 256 273 356
    00000d0 320 206   ! 355 367   P   ) 247 035 226 357 026 224 324   u   K
    00000e0 255   8 277 326 350 347 205  \n   : 220 334 342   u 216   M 357
    00000f0  \f   =   c 316 244   %  \0 233   E 354   x   " 376 250 266 247
    0000100   ^ 303   S 273 220 271   0   | 232 222 252 033 251  \r 372   g
    0000110   n 221 177   ) 270   l  \v   X 315   w   ,   (  \t 254   *   $
    0000120 023     031   !   | 003   J   0 262 002 030 230 321           4
    0000130 340 233 216 337 360 261   \   ] 216 016 020 206 001 357 346 332
    0000140 337   z 306 363 374   ] 354 323 335   [ 337 214   z 330 226 027
    0000150   j 255 037 260   ' 027 356 333   F   W   V 200 030   H   ; 337
    0000160  \n 245   q   7 372 360   I 263   ; 326 316 321 306 030   q   K
    0000170 337 264 030   1 345   z 032   @ 357 301   ] 302   U   < 037 262
    0000180 363 272 241 231 346   s  \0 337 216 235 375 236   [ 037 215 037
    0000190 274 341 033 200  \0  \0  \0  \0   u 216 336   U 336   r   1   u
    00001a0  \0 001 311 002 227  \n  \0  \0  \n 224   r 242 261 304   g 373
    00001b0 002  \0  \0  \0  \0 004   Y   Z 375   7   z   X   Z  \0  \0 004
    00001c0 346 326 264   F 002  \0   ! 001  \b  \0  \0  \0 330 017   # 023
    00001d0 001  \0  \r   #   #   #   #   W   o   r   l   d   #   #   #   #
    00001e0  \n  \0  \0  \0   x 360  \v   ) 314   g 337 330  \0 001   & 016
    00001f0  \b 033 340 004 037 266 363   } 001  \0  \0  \0  \0 004   Y   Z
    0000200
    ```
> Copilot:

The hexdump you've provided appears to be a binary file. It's difficult to determine if it's a cause for concern just by looking at the hexdump. Here are a few things you could consider:

1. *File Origin*: If this file was added by a trusted team member and it's expected to be part of your code repository, it's probably safe. If the file's origin is unknown or it was added unexpectedly, you might want to investigate further.

2. *File Purpose*: Understanding what this file is used for can help determine if it's a cause for concern. If it's a binary used for testing or a necessary part of your application, it might be fine. If its purpose is unclear, it's worth checking.

3. *File Analysis*: You could use file analysis tools to examine the binary file. This could provide information about what the file does, which can help you decide if it's a cause for concern.

4. *Security Scans*: Consider running security scans on the file. There are many tools available that can scan binary files for known security vulnerabilities.

5. *Code Review*: If this file was added as part of a commit, make sure the commit was reviewed thoroughly. Unexpected binary files can sometimes be a sign of an insecure direct object reference (IDOR) vulnerability.

Remember, when in doubt, it's always better to err on the side of caution. If you're unsure, consider reaching out to a security professional. They can help you analyze the file and determine if it poses any risk.

Lol.

---

2. Stop accepting shit tools like M4 and C for creating any software.

You can never make the work of black hats impossible but you can trivially make it a whole lot harder.

What could be more trivially done than simply replacing C for all code?

mathverse
1 replies
9h36m

We dont need any tinfoil hats. FOSS projects were being backdoored even before they became a thing the difference is that this time it's state sponsored actors doing it.

The old groups GOBBLES,ADM,ac1db1tch3z,~el8 were doing it, private "security researchers" like isec.pl were doing it.

This time it's a problem because state actors are abusing the corporate capitalism that created this era of underpaid people working on foundational projects. The bad actors have unlimited resources for their objectives.

That's basically what created the demand and ineption of groups like NSO,Zerodium etc..

Basically before that exploits and backdoors were worthless and hackers hoped to be sponsored or hired by companies like Qualys.

xhkkffbf
0 replies
2h43m

How did "corporate capitalism" create this situation? Some guy decided to volunteer his time to write some code. If anything, it might be said that corporate capitalism paid him well enough so he had the spare time?

And I don't see what money has to do with any of this. There could be some well-paid programmer slipping backdoors into proprietary code just as easily. The salary or openness didn't affect the outcome. It's just as easy for salaried programmers to betray their bosses and their users.

hmottestad
0 replies
9h32m

You could write all code yourself for absolutely everything with your own dedicated crew of vetted developers.

Or buy closed source from someone like Microsoft, hoping that they have the resources and actually use the resources to scrutinise code more vigorously.

And there is always the approach of having a great secops team to detect strange network activity and attempts to escalate.

hackeraccount
0 replies
2h15m

My tin foil hat idea is for people to be suspicious about open source contributors. Not from countries or even one's who have seemingly never been met.

But from contributors who have no history and no footprint other then the project they're working on. That should be a red flag in the future.

buster
12 replies
10h54m

What I am really curious about is what caused the initial delay in ssh which was what triggered the research into xz in the first place. Has someone figured that out?

pja
10 replies
10h26m

It's doing an extra ECC operation on every connection, but that shouldn't take 500ms on a modern CPU.

The people who have reverse engineered the code say it also requires the command message to be bound to the ssh host key as well, so if the host key is an RSA key then it might be doing an extra RSA decryption operation on every connection as well?

That would probably do it.

tux3
4 replies
9h53m

I believe it's all the ELF parsing and disassembly in memory that happens on startup.

The really went crazy with the home-made x86 disassembler and everything they do with it. Must have been super fun to write though! Someone clearly had a great time coming up with clever ideas for a backdoor.

pja
3 replies
9h45m

That only happens once at startup though?

IIRC the original detection was because sshd was using a lot more CPU coping with the torrent of spam connections any sshd on the internet gets than it usually would.

tux3
2 replies
9h27m

sshd forks a new clean process for each connection, so this whole machinery happens each time you connect. Each connection is a full fork/exec, and the backdoor has to set itself up from scratch again, including all the parsing and hooking.

rsc
1 replies
3h35m

I have looked in the sshd code at https://github.com/openssh/openssh-portable and I cannot find it forking and re-execing _itself_. It forks and execs other commands, of course, and it forks to handle new connections but does not re-exec in those paths that I can see.

If some inetd-like program were listening on port 22 and fork+exec'ing sshd to handle each incoming connection, that would explain it. But on my systemd-based Linux system I see a long-running sshd that appears to be taking care of port 22.

I do agree that it seems like the best explanation of the delay is that somehow sshd was being exec'ed per connection, but I haven't seen all the dots connected yet.

tux3
0 replies
3h18m

This is behavior introduced in OpenSSH 3.9 (grep for "re-exec" in the release notes https://www.openssh.com/txt/release-3.9)

You should also be able to see the error message "sshd re-exec requires execution with an absolute path" in the source. If you follow the `rexec_flag` tested above this message, you can see where it calls execv, later in the code.

honzaik
3 replies
8h8m

Another interesting thing regarding the ECC is they use Ed448, compared to something conventional like ECDSA with P-256 or Ed25519, which is way slower (30x-ish slower verification):

(OpenSSL benchmark numbers)

                              sign/s verify/s

   256 bits ecdsa (nistp256)   34642.6 11700.3

                              sign/s verify/s

   456 bits EdDSA (Ed448)   3209.5 409.5
There is basically no incentive to use Ed448 unless you think ECDSA with 256-bit curves is insecure or will become in the near future.

Aachen
2 replies
7h6m

At >400 operations per second, that doesn't explain most of the 2/second (500ms) operations the discoverer apparently observed. Were they running on an old Pi and you on a currently high-end CPU or so? Basically, what hardware is this benchmark on?

honzaik
1 replies
7h0m

I mean it highly depends on the CPU so I only posted it to show the relative slowdown compared to ECDSA. I ran this on my free tier Google Cloud server so it is not some super CPU.

However yes, even on this, not so powerful CPU, it doesnt take 500ms so I dont think it explains it.

Aachen
0 replies
59m

Thanks! That indeed sounds like it rules this out as the reason why it was found.

Curious that asymmetric crypto isn't even the slow part here. Feels like they just messed up somewhere (but I don't have the low-level asm/C skills to check that without considerable time investment)

markhahn
0 replies
3h25m

not ecc - this article mentioned all the effort put into hiding suspicious strings. all those string-obfuscating lookups took lots of extra cycles.

Beijinger
0 replies
4h5m

Somebody suggested: On purpose.

Easy way to find out from outside if server is compromised without trying to inject code first.

Animats
5 replies
11h38m

Why is "hooking" allowed at all in non-debug code? The legit applications all involve tracing or profiling.

rwmj
3 replies
11h33m

What does "non-debug" code mean in this context? Most distros compile everything with -g and then separate out the debug symbols into a different package, because you almost always want to be able to attach gdb to any process when something goes wrong (and no, recompiling at that point is a much worse idea, because it means installing compilers on your production system as well as not being able to debug what was actually running).

It doesn't matter anyway, because once you're running inside a process you can do most anything you like. This was a convenient mechanism, but there are scores of ways to scan over the symbols and modify code inside the same process.

Animats
2 replies
11h5m

"You almost always want to be able to attach gdb to any process when something goes wrong".

Now, this is why we can't have secure systems.

Somehow, I've never needed to attach GDB to a running process. Start up programs under GDB, yes. Read crash dumps, yes. If you're using a debugger in a production system, something has gone horribly wrong.

troad
0 replies
10h39m

If you're using a debugger in a production system, something has gone horribly wrong.

That's right, hence the debugger.

rwmj
0 replies
9h50m

Or you want to capture accurate core dumps for a tool like ABRT. We (Red Hat) work very closely with customers to find out what is and isn't acceptable to be running on their production systems, including customers who store very sensitive data.

saagarjha
0 replies
7h20m

Hooking of what?

usr1106
4 replies
8h18m

As far as the 3 first stages are concerned that article adds little to what has been known for 2 weeks. Just a nice write-up with flow charts.

But the analysis of the binary in that detail seems new to me.

So how has the source code shown there been produced? Running a disassember, understanding what the code does and renaming everything to descriptive names? That looks quite like quite an achievement in 2 weeks.

r721
2 replies
8h8m

Authors GReAT

Global Research & Analysis Team, Kaspersky Lab

https://securelist.com/author/great/

The article is apparently authored by malware analysis team from Kaspersky Lab, so they are probably quite good at reverse engineering binaries.

usr1106
1 replies
7h58m

Sure. But that does not answer any questions about their tooling or methods. Considered a business secret? They present the readable C code of the binary payload like it has appeared just out of the blue.

saagarjha
0 replies
7h21m

They're using IDA Pro, with names that they presumably came up with themselves by analyzing what the code does.

giantpotato
0 replies
7h32m

The tool used in those white screenshots is called IDA pro, a decompiler.

https://hex-rays.com/ida-pro/

nonane
4 replies
10h38m

Has anyone done an analysis of the bug in the backdoor that caused valgrind errors and slowness in SSH which eventually exposed it?

lyu07282
3 replies
9h28m

Invalid memory write apparently: https://www.mail-archive.com/valgrind-users@lists.sourceforg... The valgrind "fix" was to disable ifunc which disabled the backdoor so the errors went away.

The slowness derived from all the symbol and instruction lookups it did afaik

itomato
0 replies
6h8m

Handier for sure, but not on my clipboard

levleontiev
4 replies
11h32m

Sorry if I trigger anyone's paranoia, but 1. who told you that it's not a state-backed attack? 2. and if 1 is true, who told it, it's not Russia staying behind it?

and if 2 is true, the KGB university alumni Kaspersky definitely is related to it.

thrdbndndn
3 replies
11h15m

And it's relevant to the article because?

fodmap
2 replies
9h34m

I suppose they say that because the article has been written by Kaspersky, and it's hosted on a server under their control?

thrdbndndn
1 replies
9h12m

Even let's go all the way down and say the backdoor was directly written by Kaspersky themselves, does it change the value of the content (which is purely technical analysis) anyhow?

ogurechny
3 replies
12h15m

As an armchair specialist, I don't support the “owning each ssh server on the planet” reactions.

Public networks have too many eyes potentially (not just the admins responsible for specific systems). Traffic to ssh server that corresponds to suspicious activity but is also absent from logs would raise the alarms quite soon.

Hooking the build process also means that the main targets are places that are important enough to have their own siloed software repositories. Getting into two LTS systems was a bonus. I suppose that simpler exploits are available for popular systems (if they run regular complex applications instead of being a minimal empty box that securely does nothing).

The number of modification/decryption layers is probably not arbitrary. Most likely, each was added to evade detection by one or another protection system you and I don't know about, but with which attacker is familiar.

raggi
1 replies
10h37m

Can you point to an URL where a system has a documented or implemented configuration that correlates traffic to log entries in a manner that can be alarmed on without just going off all the time?

Junk packets to port 22 arrive all the time.

ogurechny
0 replies
7h45m

I'm not talking about random packets. There is global and local traffic collection (after all, each pimple on the face of humanity wants to have own little NSA after a wonderful advertisement campaign, and lots of “engineers” are OK with getting paid for that). So when someone capable enough matches obvious abnormal ssh traffic to some intermediate victim with later unwanted behavior of that system, and ties it to the general activities of some known long term actor, that system is going to be studied most thoroughly. Using this backdoor over the internet would mean risking getting busted after each single use. On the contrary, inside high profile “secure” networks such covert management channel would worth the price even if it's single use (as last measure shutdown switch, for example).

I suppose “smart” readers can be as easily manipulated to look in any wrong direction as “dumb” ones. All stories mention what each decryption and indirection step does, but they don't explain why exactly those steps were chosen, that this one hides from intrusion detection system A, and that one prevents collection of runtime activity artifacts by tracing system B, and so on. On one hand, it would be impolite to announce that you study, and know well, how to break your colleague's product, on the other hand, some of those systems might be secret if we talk about state level.

rcxdude
0 replies
4h5m

It's obviously not something you just blast into a botnet or something, that's too high-profile. But it would give you one hell of a tool for targeted attacks, and it would be very hard to detect: I don't think there's any automated system that would identify it. Probably the best chance would be on a system which was later identified as comprimised where the operator is sufficiently paranoid to be logging all ssh traffic and doing some inspection of that afterwards (which would be very tedious). Otherwise, the main form of detection would be inspection of the binary itself, which is likely where the various layers come from. It seemed designed to be stealthy from inspection of the code to inspection of the resulting binary, but the fatal flaw there was it didn't have debug info covering it and it lit up on a performance analysis.

All this adds up to a group which wanted to have an easy way into a relatively small but diverse set of high-value targets, it's likely they would have waited for the targeted distro released to reach fairly widespread adoption before exploiting it, and they would have tried to keep it low-profile enough that it would not be discovered for some time.

TheSeeker11
2 replies
11h54m

I was rather amused to see Chris Titus announce that "XZ Backdoor is NOT that bad!" He seems to have done next to no research into this https://www.youtube.com/watch?v=s_WkMtEPsuE

rwmj
0 replies
11h36m

Never heard of him before but after watching 10 seconds he seems like an idiot.

Delk
0 replies
4h32m

Skimming through it, it sounds like the video is about 8 minutes to say that the vulnerability only affected test versions of Debian and Fedora.

Which is true, and the number of actual running systems that the backdoor already made its way into is likely not massive -- especially in terms of production or otherwise critical systems.

But that doesn't mean it wasn't bad. It's rather a disaster that was only avoided pretty much by accident.

andy_ppp
1 replies
7h57m

Imagine all the back doors people haven’t discovered…

ta1243
0 replies
7h56m

Especially in closed source software where nobody has an incentive to look, any performance hits will be ignored, code analysis tools can't be run (why bother - can't exactly sell it as a new feature)

HankB99
1 replies
2h48m

I'm curious about the technical skills needed to pull this off. I'm not a maintainer and have never built a Debian package. It appears to me that it required a fairly deep knowledge of the package build pipeline to accomplish this. Is that well enough documented somewhere that someone with a technical background and desire to learn could pick it up? What's the possibility that this person (or team) has been an active participant in the Debian community? Likewise with the RH technology and team, if that differs.

chuckadams
0 replies
8m

The exploit itself is the interesting piece, which obviously took someone with lots of black hat experience to write. The trie-based string comparison trick is particularly clever, though even that is fairly simple compared to the stuff that comes out of the likes of the NSO group (check out the Wikipedia article for "Weird Machines", that's the sort of stuff they write).

The build process of the exploit is also somewhat interesting, but what kicks it off is just an extra m4 file that autogen picks up and dutifully runs with its out of the box settings. That extra file was just added by hand to the dist tarballs, since the server was also owned by the attacker. It doesn't appear in the repo anywhere. Five minutes of reading autoconf docs and you too can create an autoconf trojan with ease.

The various Jia Tan sockpuppets were actively pushing both Debian and Redhat to include the backdoored version of xz, so the answer to your last question is yes, they are in fact active participants in the community.

xg15
0 replies
19m

Unlike other supply chain attacks we have seen in Node.js, PyPI, FDroid, and the Linux Kernel that mostly consisted of atomic malicious patches, fake packages and typosquatted package names, this incident was a multi-stage operation that almost succeeded in compromising SSH servers on a global scale.

I think this sentence actually understates what happened.

What I find more frightening than the technical aspects of the backdoor is the amount and degree of "social engineering" involved: The backdoor was the final product, but inserting it was only possible because at that point the entire xz project had long been taken over by malicious actors ("Jia Tan" and friends) - after a yearlong psychological operation against the maintainer. All that without the maintainer or anyone else noticing anything.

(You can see the same mentality reflected in the backdoor code itself - I find it remarkable how much of the code is devoted to active misdirection. It's not just trying to look harmless, the code actually tries to build a "narrative" through commit messages, comments, variable names, choice of commands, etc, of what it's ostensibly doing while really doing something completely different - leading anyone who'd inspect the code first to doubt themselves and their understanding of the code, then suspect a bug and then, maybe, suspect malicious intent)

That's spy novel stuff - and if something like this is possible, it makes me wonder what else might be going on in other projects right now.

schuyler2d
0 replies
3h43m

Quite rightly most attention has been on the operational success of the backdoor and how it achieves it's goals.

But I would love some deeper analysis of the mistakes and over-engineering parts. In the Bryan Cantril interview [1] Andrés suggests there are a lot of dumb things done because it was a kind of off-the-shelf backdooring piece that didn't necessarily know how it would be deployed. Eg the symbol table lookup that led him to investigate.

Similarly, why were they chopping off 48 bytes using RC4 [2]

So I'd love folks to talk about how (given more time/better team) this could have been better or where they messed up more

[1] https://youtu.be/jg5F9UupL6I?si=gvXsYFXgagkGOMd4 [2] https://twitter.com/matthew_d_green/status/17744729080201014...

robryk
0 replies
7h37m

The only variable inside its own table is yolAbejyiejuvnup=Evjtgvsh5okmkAvj, which in this context serves as a kill switch.

This seems false? The trie contained also strings like `WAYLAND_DISPLAY=`, so IIUC the backdoor would remain dormant if someone tried to run sshd from something that looks like an interactive session.

phatfish
0 replies
5h11m

It's a helpful collection of information on this topic. But Kaspersky... really? No one with any sense has that voluntary backdoor in their network.

noname120
0 replies
11h15m

What proves us that tukaani and jiaT75 aren't the same person? Creating a new nickname and handing write access to it sounds like the perfect way to shift blame and claim that his trust was betrayed once the backdoor is found out.

matt3210
0 replies
12h36m

I hope there aren’t any like this which went unnoticed :-(

m3kw9
0 replies
15h10m

I’ve seen some google zero day vulnerability hack analysis and those looked out of this world but this hack must be one of the all time

lyu07282
0 replies
9h57m

The authors show a great deal of knowledge of glibc internals, the sort of things you really have to be neck deep in the source code for. Lots of novel techniques in there.

The custom elf parser and disassembler are so complex that I can't imagine they didn't use that code / or will never use that code for something else in the future.

I wonder if this gets the serious investigation it deserves, but I doubt it.