Xzbot: Notes, honeypot, and exploit demo for the xz backdoor

I like the theory that actually, it wasn’t luck but was picked up on by detection tools of a large entity (Google / Microsoft / NSA / whatever), and they’re just presenting the story like this to keep their detection methods a secret. It’s what I would do.

I'm not sure why everyone is 100% sure this was a state-sponsored security breach. I agree that it's more likely than not state-sponsored, but I can imagine all sorts of other groups who would have an interest in something like this, organized crime in particular. Imagine how many banks or crypto wallets they could break into with a RCE this pervasive.

Was the performance issue pure luck? Or was it a subtle bit of sabotage by someone inside the attacking group worried about the implications of the capability?

If it had been successfully and secretly deployed, this is the sort of thing that could make your leaders much more comfortable with starting a "limited war".

There are shades of "Setec Astronomy" here.

Do we have a detailed technical analysis of the code? I read a few analysis but they all seem preliminary. It is very useful to learn from the code.

I don't think it was executed incredibly well. There were definitely very clever aspects but they made multiple mistakes - triggering Valgrind, the performance issue, using a `.` to break the Landlock test, not giving the author a proper background identity.

I guess you could also include the fact that they made it a very obvious back door rather than an exploitable bug, but that has the advantage of only letting you exploit it so it was probably an intentional trade-off.

Just think how many back doors / intentional bugs there are that we don't know about because they didn't make any of these mistakes.

I read someone speculating that the performance issue was intentional, so infected machines could be easily identified by an internet wide scan without arousing further suspicicion.

If this is or becomes a widespread method, then anti-malware groups should perhaps conduct these scans themselves.

one question I still have is what exactly the performance issue was? I heard it might be related to enumeration of shared libraries, decoding of the scrambled strings[1], etc. anyone know for sure yet?

one other point for investigation is if the code is similar to any other known implants? like the way it obfuscates strings, the way it detects debuggers, the way its setting up a vtable, there might be code fragments shared across projects. Which might give clues about its origin.

[1] https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01

Yeah, we should probably expect that there are roughly 1/p(found) more of these lurking out there. Not a pleasant thought.

It'd make total sense if it was. This way you get to have the backdoor without your enemies being able to use it against your own companies.

I agree that this is probably about persistence. Initially I thought the developer was playing the long-con to dump some crypto exchange and make off with literally a billion dollars or more.

But if that was the case they wouldn't bother with the key. It'd be a one-and-done situation. It would be a stop-the-world event.

Now it looks more like nation-state spycraft.

This is just a safety measure that it does not blow up in your own face (country).

More to the point it prevents your enemies from using the exploit against friendly targets.

The tradeoff is that, once you find it, it's very clearly a backdoor. No way you can pretend this was an innocent bug.

Also people can come and immediately undo whatever you did if it's not authenticated.

*host key fingerprint, but I assume what you've meant.

It's practically a good backdoor then, crypto graphically protected and safe against "re-play" attacks.

The libarchive diff didn't create any vulnerability. The fprintf calls were consistent with others in the same repository.

This is not that concept. That concept is no one but us can technically complete the exploit. Technical feasibility in that you need a supercomputer to do it, not protecting a backdoor with the normal cia triad

this is probably the right answer. hacking group getting a 0day to sell to nation states on a per-use agreement.

The thing about port knocking is that if you're on a host where you don't have the ability to port-knock, then you're not able to connect.

If that's important, one should be able to set up port knocking such that you're able to do the knocks even by changing the port in a sequence by hand on e.g. a web browser address bar.

The thing about port knocking is that if you're on a host where you don't have the ability to port-knock, then you're not able to connect.

You can type http://hostname:porttoknock in a browser.

As long as you're not behind a super restrictive gateway that doesn't let you connect to arbitrary ports, you're golden.

Port knocking with ssh over https using client certs.

And port knocking is one of the few effective methods against the standard distributed brute forcing slow attacks.

Note if you blanket-ban IN/RU/UK/CN/HK/TW/IR/MX/BZ/BA + tor/proxies lists, than 99.998% of your nuisance traffic issues disappear overnight. =)

https://github.com/CERN-CERT/pam_2fa

I wouldn't really consider port knocking to be that effective of a way to block connections. It is really only obscure, but port knocking software is very accessible. So if you know a port needs to be knocked, it's not hard for an attacker to get to it.

The main benefit of port knocking is that it allows you to present your ports as normally closed. If you have a system and you are worried about it getting port scanned for whatever reason, it makes sense to have a scheme where the ports are closed unless it receives a knock. So if someone gets access they shouldn't for whatever reason and pulls a portscan off, the information they get about your system is somewhat limited.

In this scheme, it would be much better to have an access point you authenticate with and then that handles the port knocking for the other devices. So it is a kind of obscure method that is really only useful in a specific use case.

As far as SSH keys go, I would argue that SSH support is so ubiquitous, and SSH access is so powerful that it is a reasonable security tradeoff. I also don't think that SSH ports should be exposed to the internet, unless it is a specific use case where that is the whole point, like GitHub. I'm very down on connecting directly through SSH without network access. The xz situation has validated this opinion.

I personally don't know any application where you really need supreme admin access to every device, from any device, from anywhere in the world, while at the same time it has extreme security requirements. That's a pretty big task. At that point, constructing a dedicated, hardened access point that faces the internet and grants access to other devices is probably the way to go.

You just described the problem with using keys for any login, like the latest fad is?

And generally with depending on a device (phone, xxxkey or whatever) for access.

My solution to this has been creating a public bastion server and use Wireguard. Wireguard listens on a random UDP port (port knocking is more difficult here.) This client is set up to have a dynamic endpoint so I don't need to worry about whitelisting. The key and port information are stored in a password manager like Vaultwarden with the appropriate documentation to connect. Firewall rules are set to reject on all other ports and it doesn't respond to ICMP packets either. A lot of that is security through obscurity but I found this to be a good balance of security and practicality.

The thing about port knocking is that if you're on a host where you don't have the ability to port-knock, then you're not able to connect.

Then you attach a device that can have port knocking to that unsupported host. Also, I remember it was called port punching not knocking.

Absolutely, everything in security is a tradeoff. I guess the real point is that there should be layers, and even though you should never rely on security through obscurity, you should still probably have a bit of obscurity in the mix.

You can as well find yourself on a host that doesn't have SSH or network that filters SSH traffic for security reasons.

Port knocking is definitely dumb, but "increase your password lengths, or cryptographic key sizes" does nothing if your ssh binary is compromised and anyone can send a magic packet to let themselves in.

Strict firewalls, VPNs, and defense-in-depth is really the only answer here.

Of course, those things all go out the window too if your TCP stack itself is also compromised. Better to just air-gap.

Security-by-obscurity is dumb, yes. But in the context of supply-chain exploits in the theme of this xz backdoor, this statement is also myopic:

If you want more secret bits which users need to know in order to access your system, increase your password lengths, or cryptographic key sizes.

If your sshd (or any other exposed service) is backdoored, then the "effective bits" of any cryptographic key size is reduced to nil. You personally cannot know whether or not your exposed service is backdoored.

Bottom-line is: adding a defense-in-depth like port knocking is unlikely to cause harm unless you use it as justification for not following best-practices in the rest of your security posture.

I don't think I agree. This backdoor means that it doesn't matter how long your key lengths or cryptographic key sizes are; that's kinda the point of a backdoor. But an automated attempt to find servers to exploit is not going to attempt something like port knocking, or likely even looking for a sshd on a non-standard port.

Kerckhoff's law goes (from your link):

The principle holds that a cryptosystem should be secure, even if everything about the system, except the key, is public knowledge.

With this backdoor, that principle does not hold; it's utterly irrelevant. Obscuring the fact that there even is a ssh server to exploit increases safety.

(I dislike port knocking because of your third point, that it complicates access. But I don't think your assertions about its security principles hold water, at least not in this case.)

(Ultimately, though, instead of something like port knocking or even running sshd on a non-standard port, if I wanted to protect against attacks like these, I would just keep sshd on a private network only accessible via a VPN.)

Security through obscurity is only a problem if obscurity is the main defense mechanism. It's perfectly fine as a defense-in-depth. This would only be an issue if someone did something stupid like set up a passwordless rlogin or database server expecting port knocking alone to handle security.

Also as pointed out elsewhere, modern port knocking uses Single Packet Authorization which allows for more bits. It's also simpler and uses a different mechanism than ssh (which due to its age, has historically supported a bunch of different login and cryptography techniques), which reduces the chance that an attacker would be able to break both.

It seems that we now have a clear proof that it's actually helping versus certain type of attacks (including source-code supply chain attacks).

So would have a vpn or using a bastion host with a custom non standard ssh implementation...

At some point you have to make the choice to not implement a security measure and I would argue that should stop at vpn+standard software for secure access.

If you are a bigger company, probably add SSO and network segmentation with bastion hosts and good logging.

Port Knocking doesn't add any security benefit in the sense that there are known non avoidable security risk aka your transmit your password(knocking) in clear text over the network.

You also add another program with potential vulnerabilities, and as port knocking is not as popular as e.g. sshd, wireguard, maybe it gets less scrutiny and it leads to a supply chain attack?

Security measures are also not free in the sense that somebody has to distribute them and keep the configuration up to date, even if that person is you, that means syncing that connect-to-server-script and keeping it in a secure location on your devices.

say that port knocking is pointless and security theatre

Who was saying that?

The advantage of port knocking to me is just reducing the amount of garbage script-kiddie scans. IMHO the design of `sshd` needs to just assume it will be slammed by garbage attempts and minimize the logging. I've heard of `fail2ban`, but banning does nothing as the bots have an unlimited number of IPs.

The reality is that security through obscurity works really well as a layer in an otherwise already good security model. You make sure to test it with some red teaming, and if they fail to get through, you give them all the details about your tricks so they can also test the actual security.

The "obscurity" part mostly serves to limit the noise of drive by and bot'd attacks, such that each attack attempt that you end up stopping and catching is a more serious signal, and more likely to be directed. It's about short circuiting much of the "chaff" in the signal such that you are less warning fatigued and more likely to seriously respond to incidents.

The obscurity is not meant to prevent targeted attacks.

While 'security through obscurity' shouldn't be your only defense, it still plays a crucial role within 'defense in depth' strategies.

In the past, sensitive networks relied heavily on obscurity alone. Just dialing the right phone number could grant access to highly sensitive networks. However, there's a cautionary tale of a security researcher who took the phrase 'don't do security through obscurity' to heart and dared hackers, leading to disastrous consequences.

Obscurity, when used appropriately, complements broader security measures. Take SSH, for example. Most bots target its default port. Simply changing that port removes the easy targets, forcing attackers to use more sophisticated methods, which in turn, leaves your logs with more concerning activity.

nor can you audit every line of code all the time

You can if you distribute this job among volunteers or hire people to do that. There are millions of developers around the world capable to do this. But reality is that nobody wants to contribute time or pay for free software.

It's old and there are probably friendlier options out there now, but

https://github.com/moxie0/knockknock/blob/master/INSTALL

Sounds like (another) good reason for IPv6 - your box can have many, very obscure addresses :)

NSO Group

They are an Israel based company, that sell zero-click RCEs for phones and more. Such malicious software was involved in the murder of journalist Jamal Khashoggi.

Their exploits, developed in-house as well as presumably partially bought on the black market, are some of the most sophisticated exploits found in the wild, e.g. https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-i...

JBIG2 doesn't have scripting capabilities, but when combined with a vulnerability, it does have the ability to emulate circuits of arbitrary logic gates operating on arbitrary memory. So why not just use that to build your own computer architecture and script that!? That's exactly what this exploit does. Using over 70,000 segment commands defining logical bit operations, they define a small computer architecture with features such as registers and a full 64-bit adder and comparator which they use to search memory and perform arithmetic operations. It's not as fast as Javascript, but it's fundamentally computationally equivalent.

Probably Stuxnet.

Stuxnet?

Bob Maxwell (Ghislaine's father) sold backdoored software to corporations and governments all around the world, including US targets, on behalf of Israel's Mossad.

"The Maxwell-Mossad team steals spy software PROMIS from the United States, Mossad puts an undetectable trap door in it so Mossad can track the activities of anyone using it, then Maxwell sells it around the world (including back to the U.S. -- with the trap door)."

https://kclibrary.bibliocommons.com/v2/record/S120C257904

Thanks, i was wondering about this since day 0 and was too lazy to look it up. Yes it can be spoofed, but I imagine a good chunk of day-to-day is work is semi-interactive, which would make it preferable to have the attacker be in the same tz as the victims. Anyone know what tz Lasse was at? If not (eg he’s in the US), then I’d say Occam’s razor that the attacker is working those UTC 10-18 office hours without extra steps. Tz proves nothing and for a 3y low-intensity operation I’d just assume the attacker won’t introduce that much friction only to mislead. I’m sure there are much stronger signals in the investigation work that’s going on now. Unfortunately, given the hush-hush-by-default nature of our beloved intel agencies, we’ll probably never know.

It's rather trivial to fake git commits.

Basing this stuff on email times (especially replies to emails sent that same day) would be more relevant. However. if this operation was pulled off with the precision and opsec that it seems to have been, I wouldn't be too surprised if whatever group is behind this attack would've sent over + funded a developer somewhere in a time zone of their choice.

No doubt any nation state is able to station someone in Russia/Israel/one of the n-eyes countries.

I doubt OSINT will figure out who is really behind this, but I'm sure governments will, soon enough. Whether or not they report the truth of their findings, and if one should trust the official reporting, is yet another tough question.

I like to think that the performance issue in the exploit was actually an exploit of an exploit, a counterintelligence act, by some "good samaritan" :D

https://en.wikipedia.org/wiki/Tailored_Access_Operations

The Clipper chip? Not sure if that's what they had in mind. Nowadays it's maybe just how the NSA has rooms attached to backbone providers like https://theintercept.com/2018/06/25/att-internet-nsa-spy-hub...

Israel is not in Europe

Russia is all over the place in terms of targets/sophistication/etc ... but this feels a bit beyond anything I recall them accomplishing at a state level.

https://en.wikipedia.org/wiki/Triton_(malware) (https://www.justice.gov/opa/pr/four-russian-government-emplo...)

Even with the MIT disclaimer and the author not being the distributor or have any relationship with the distributor. Publishing vulnerable open source software to GitHub with a disclaimer that says it isn’t fit for any purpose seems like a bit of an oversight of using MIT license in distros to me.

CIA didn't put anyone new into gitmo for years.

The 30 remaining gitmo prisoners are all W Bush holdovers that all subsequent administrations forgot.

Hacking statues cover unauthorized access, and there's no evidence that unauthorized access has occurred, unless we see someone actually making use of the backdoor in the wild. Accessing a system without authorization is a crime, but distributing code that contains a backdoor is not a crime. The attacker was authorized to publish changes to xz.

Laws can be made retroactively

Not in the United States. https://constitution.congress.gov/browse/article-1/section-9...

Indeed. I think a lawyer would have written "tortious", there being no pain involved.

That said, if you’re doing this for your jurisdiction’s security agency, you’ll certainly be protected.

Likely a team of people at a three letter agency.

Or...falling back on less noticed contingency plans...

Or, they must be strapped into a chair having teeth being pulled out by whoever directed them.

I think it would help to try and secure signing infrastructure as much as possible. First of all, try to have 3 devs for any given project (this is the most difficult step). Make sure logging into or changing any of the signing stuff requires at least 2 of the devs, one to initiate the JIT and one to monitor.

Additionally, take supply chain steps like requiring independent approval for PRs and requiring the signing process to only work with automated builds. Don't allow for signing bits that were built on a dev machine.

Finally, I think it would help to start implementing signing executables and scripts which get checked at runtime. Windows obviously does executable signing, but it largely doesn't sign stuff like Python scripts. JS in the browser is kind of signed given that the whole site is signed via https. It's not perfect, but it would help in preventing one program from modifying another for very sensitive contexts like web servers.

Like everything else in this world, its utterly and completely unfixably broken: There is no way you can have complex huge dependencies properly.

The best most paranoid thing is to have multiple networking layers hoping they have to exploit multiple things at once, and whitelist only networking for exactly what you are expecting to happen. (which is completely incompatible with the idea of ssl, we would need a new type of firewall that sits between ALL applications before encryption, like a firewall between applications and the crypto library itself, which breaks a bunch of other things people want to do)

No. They obviously didn't do that so you're just being sarcastic but not actually making any point of your own in addition to that.

It is an obvious place for sure, but it also would have been picked up if the builds where a bit more transperant. That batch build script should have been questioned before approval.

It’s got me suspicious of build-time dependency we have in an open source tool, where the dependency goes out of its way to prefer xz and we even discovered that it installs xz on the host machine if it isn’t already installed — as a convenience. Kinda weird because it didn’t do that for any other dependencies.

Have you considered reaching out to the maintainers of that project and (politely) asking them to explain? In lieu of recent events I don't think anyone would blame you, in fact you might even suggest they explain such an oddly specific side effect in a README or such.

It's a POC nevertheless, it's a complete implementation of the RCE minus obviously the private key.

Could the provided honeypot print out keys used in successful and unsuccessful attempts?

Should be able to do it by having the scanner take multiple samples. As long as you don’t need a valid login and the performance issue is still observable, you should be about to scan for it with minimal cost

When loading liblzma, it patches the ELF GOT (global offset table) with the address of the malicious code.

How was this part obfuscated/undetected?

Given the sophistication of this attack it would indeed be downright negligent to presume that it's the attackers' legal name and that they have zero OPSEC.

It's also worth pointing out, given the almost two years of seemingly valuable contribution, that this could be a real person who was compromised or coerced into pushing the exploit.

Interesting to look through the "starred" repos of that account ... seems like a hit list: https://github.com/JiaT75?tab=stars

It's alleged to be https://news.ycombinator.com/user?id=dang

Interesting... Though you can edit whatever log file you want

That would look the same as a random failed ssh login, which happens all the time. The connection isn't maintained past that point (unless the payload chooses to do so).

That's insane. How exactly does this happen? Are there no EDR/IDS who can detect an RCE at the connection stage?

I don't think this is responsive to my comment.

Seems like single-byte XOR would be almost as good for obfuscation purposes, but I guess using chacha20 from the public key as a pseudo-OTP doesn't hurt.

The only real improvement that I can see being made would be adding perfect forward secrecy so that a logged session couldn't be decrypted after the fact. But that would likely add a lot of complexity (I think you need bidirectional communication?)

Yeah.

Speaking only hypothetically, but two points:

1) No-one has been is proven to "be" anyone in this case. Reputation is OSS is built upon behaviour only, not identity. "Jia Tan" managed to tip the scales by also being helpful. That identity is 99% likely to be a confection.

2) People can do terrible things when strongly encouraged or worse coerced. Including dissolving identity boundaries.

The first problem can be 'solved' by using real identities and web of trust but that will NEVER fly in OSS for a multitude of technical and social reasons. The second problem will simply never be solved in any context, OSS or otherwise. Bad actors be bad, yo.

Passive aggressive accusation.

This style of fake doubt is really not appropriate anywhere.

No, he likely is not. But the patch series includes commits co-developed by Jia Tan, and lists Jia Tan as a maintainer of the kernel module.

Thanks! That’s a little disappointing since I would have thought that the way it hooked those functions could’ve been caught by a generic heuristic but perhaps that’s more common than I thought.

That relies on knowing what to look for. I.e. "the malicious library". The question is whether any of these solutions could catch it without knowing about it beforehand and having a detection rule specifically made for it.

Guard Duty does have some ptocees level monitoring with some recent additions: https://aws.amazon.com/blogs/aws/amazon-guardduty-ec2-runtim...

The main thing I was thinking is that the audit hooking and especially runtime patching across modules (liblzma5 patching functions in the main sshd code block) seems like the kind of thing a generic behavioral profile could get but especially one driven by the fact that sshd does not do any of that normally.

And, yes, performance and reliability issues are a big problem here. When CarbonBlack takes down production again, you probably end up with a bunch of exclusions which mean an actual attacker might be missed.

The core source of the vulnerability (symbol lookup order allowing a dependency to preempt a function) might theoretically be fixed at the Linux+OpenSSH level.

Fedora too.

Yeah a quick Google brings up lots of real Jia Tans who clearly aren't behind this and hopefully aren't being wrongly accused here. But it is clearly a real and common name.

Nope, it wouldn't have been in RHEL 10 or any of the rebuilds. CentOS Stream 10 already branched from Fedora / ELN. The closest it would have gotten is a Fedora ELN compose, and it's doubtful it would have remained undiscovered long enough to end up in CentOS Stream 11.

Some tools scan for crypto algorithms, typically by searching for the magic numbers, Ed448 is so new many tools probably don't recognize it.

Malware authors frequently change the crypto magic numbers to prevent detection.

The issue is that fraud is anti-inductive: when it's discovered, it changes.

Statistical AI systems are just naive induction, and so largely useless.