return to table of content

How I tripped over the Debian weak keys vulnerability

ArtTimeInvestor
29 replies
1d12h

Reading this makes me wonder how probable/improbable it is that this kind of thing has happened / will happen with the seed generation function in one of the popular Bitcoin hardware wallets.

And what the fallout would be.

withinboredom
18 replies
1d9h

Considering that each key can be generated deterministically from a seed key and the seed keys are not infinite… it’s only a matter of time (millions of years compute time with current technology, and not out of reach for state actors who can spend infinite money to run that much compute in a matter of weeks) until every wallet is accessible by anyone who knows the address.

Bitcoin isn’t that safe to hold value in, if you might be targeted by someone with brains and money.

posnet
10 replies
1d8h

Just for fun, because it's hard to appreciate how strong a 256bit ECDSA key is.

Base Numbers:

- 2*128 guesses on average

- public state of the art for ECDSA on an FPGA is 1315tps [0]

- retail price of said fpga $10,000 [1]

- total net income for xilinx from advanced FPGAs FY2022 (936M * 0.74 + 879M * 0.72) = 1325M [2]

Ballpark numbers, we'll assume that attacker can buy 10x every FPGA xilinx made in 2022 for a 50% discount and can run them non stop for zero cost.

We'll also assume they have a bunch of secret math geniuses and have a faster ECDSA implementation that can do 1,000,000,000 tps ( or 1,000,000x SOTA)

- (1325,000,000 / 5,000) * 10 = 2,650,000 FPGAs

- 2,650,000 * 100,000,000,000 = 2,650,000,000,000,000tps

- 2*128 / 2,650,000,000,000,000 = 1.28e+23s

- 1.28e+24 / 60 / 60 / 24 / 365 = 4,080,000,000,000,000 years

- ~4 quadrillion years

- or 4x the time until every planet has been ejected from every star system and the sun has cooled to 5K [3]

But why stop there, lets assume that the attacker can use the entire planets GDP to buy chips and has a 1,000,000,000,000,000x faster ECDSA implementation.

- World GDP (2022): 101.3 trillion

- (101,300,000,000,000 / 5000) = 20,260,000,000

- 100,000,000,000,000,000 * 20,260,000,000 = 2.66419e+28tps

- 2*128 / 2.66419e+28 = 12,772,451,173s

- 12,772,451,173 / 60 / 60 / 24 / 365 = 405 years

So even then, we wouldn't see a single BTC key broken within our lifetime.

(Unless you believe that 3 letter agencies have successfully built a quantum computer that practically implement shore's algorithm, in which case you should probably be more worried about the fact that they can break public key encryption globally)

[0]: https://arxiv.org/pdf/2112.02229.pdf

[1]: https://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=...

[2]: https://web.archive.org/web/20211203065624/https://investor....

[3]: https://en.wikipedia.org/wiki/Timeline_of_the_far_future

withinboredom
9 replies
1d7h

I don't think you understand what I mean. I'm not talking about cracking a key. I'm talking about creating a key from seed 0, then 1, then 2, etc. etc., until you have an index of every possible key.

Then, since you know every possible public key ... it's just a matter of looking up its private key in the index.

HOWEVER, I was off by a few magnitudes on how big the seed numbers get. If you are curious, this is basically the biggest seed number:

904,625,697,166,532,776,746,648,320,380,374,280,100,293,470,930,272,690,489,102,837,043,110,636,675 ...

or 904 trevigintillion (thanks wolfram alpha!)

So, yeah, it's not happening anytime soon, though it doesn't invalidate the premise, just changes the timeline to unachievable with current technology. Technology could improve to the point where it is feasible in a human lifetime.

yau8edq12i
2 replies
1d7h

Where and how do you store your "index"? The number you wrote is multiple orders of magnitude higher than the number of atoms on Earth. It's not simply "not happening any time soon". It's physically impossible.

withinboredom
1 replies
1d6h

Good point :)

Edit to add: just found https://keys.lol that does exactly this, but generates a few dozen at a time and doesn't store the results.

Retr0id
0 replies
1d5h

This is like saying the Library of Babel[1] contains every book ever written, past present and future. Yes, it's technically true and a fun thought exercise, but it's not a threat to the publishing industry.

[1] https://libraryofbabel.info/

tzs
1 replies
1d3h

That kind of reminds me of the idea I've heard several times of compression by indexing π. The idea is that every possible finite byte sequence occurs somewhere in π [1] so if you've got some file of N bytes that you want to store you just need to store N and an index to a place in π where that sequence of bytes occurs.

The problem is the the index to the first occurrence of any particular N byte sequence will on average take around N bytes to store so your "compressed" file is about the same size as your input file.

[1] This is not known to be true. A number that has the property that every possible finite sequence of digits in base b occurs in the number's base b expansion is called a "normal number in base b". A number that is normal in base b for all integer bases b ≥ 2 is simply called a "normal number". Almost all real numbers are normal, but it is not known if π is among them.

Dylan16807
0 replies
23h19m

Almost all real numbers are normal

Well if we're being picky then I'm going to point out that almost all real numbers cannot be written down.

Of the numbers we can write down in a reasonable way, the computable numbers, we've only proven a relative handful of them to be normal in any base, and barely any normal in all bases.

So if you have an arbitrary irrational number where you can ask if it's normal, then the answer is probably "we don't know, we haven't figured out a proof", rather than "almost certainly". Those overwhelming odds about "almost all real numbers" don't apply to your number, because your number is computable.

01HNNWZ0MV43FF
1 replies
1d7h

It would require a break through in quantum stuff, or new laws of physics to produce a 256 bit rainbow table, right?

"Computers made of something other than matter and occupying something other than space"

https://security.stackexchange.com/a/25392

withinboredom
0 replies
1d6h

Or ... just generate a few here and there, and eventually, you'll find something. Looks like https://keys.lol is an example of just that.

pdonis
0 replies
1d3h

> I'm not talking about cracking a key. I'm talking about creating a key from seed 0, then 1, then 2, etc. etc., until you have an index of every possible key.

Computationally the two are the same: cracking a key generated by a secure algorithm is equivalent to generating all possible keys and checking to see which one matches.

Dylan16807
0 replies
23h45m

I'm talking about creating a key from seed 0, then 1, then 2, etc. etc., until you have an index of every possible key.

The cracking algorithm talked about earlier is that you create a key from seed 0, 1, 2, etc. until you find a match, and then you can keep going to look for more matches if you so desire. Or you can go in a different order, because the outcome is the same.

The only change in your version is that you don't look for matches until you've finished going through every seed.

That makes your version strictly slower.

It's not invalid, it's just worse, for the use case of cracking existing keys.

And for the use case of cracking keys made after you do most of your computing, you still wouldn't use the "one giant index" method for precomputing things. It's not efficient.

Retr0id
6 replies
1d7h

This is simply not true.

lyu07282
4 replies
1d4h

It reminds me a bit of when people worry about collisions in random uuids

Zigurd
1 replies
1d2h

I once had to convince someone much smarter than me that UUID collisions (for a messaging system we were developing) would not happen. It was a surreal experience.

a1369209993
0 replies
1d

Are you sure they were talking about UUID collisions in a idealized/theoretical system? If you generate UUIDs using, for example, a random number generator with only a 16-bit seed, you absolutely will see collisions, for much the same reasons as in TFA.

Retr0id
1 replies
1d3h

Those are more subtle, and often well-founded, due to the birthday paradox.

Retr0id
0 replies
21h38m

(and sometimes weak entropy sources)

postalrat
0 replies
1d4h

It is but probably though new math and not faster machines.

ArtTimeInvestor
3 replies
1d12h

In 2013, Bitcoin's market cap was less than a Billion, I think. And those Android wallets probably only held small amounts.

But now that pension funds get into Bitcoin, one has to wonder what would happen if a giant like Coinbase turns out to be holding their funds in cold storage but with guessable keys.

xiphias2
1 replies
1d8h

Even if those ,,small amounts'' were 5% of net worth of some people (which was not uncommon), the same small amount would consist of more than 99% of the net worth of the same people currently.

As Android wallets were the easiest to use wallets and exchanges routinely robbed people, it mattered a lot even back then.

While Coinbase has a quite good protocol for securing keys, what's scary to me is that so many ETFs are trusting it to be the only key holder instead of using multi-signature wallets between multiple entities (like Fidelity and Unchained capital as the other 2).

ArtTimeInvestor
0 replies
1d7h

Is it documented, how the individual ETFs are holding their assets?

I know they state that they do it via a custodian and name it like "Coinbase", but is that really all? All they got is an IOU from Coinbase? Maybe they have more specific setups?

3abiton
0 replies
1d11h

It's crazy to to from a hobbyist niche commodity to a nearly gold reserve alternative.

pcthrowaway
0 replies
1d11h

Interesting to see Vitalik pop up in that thread

trashburger
0 replies
1d12h

Feels like that would be noticed rather quickly given that you would be able to access someone else's funds on the network automatically, whereas in the SSH vulnerability case you have to proactively check if servers you want to access have one of the bad fingerprints.

pants2
0 replies
1d12h

The $160M Wintermute hack was a result of insecure key generation in a public library, though it's unlikely that one was introduced intentionally.

macrolime
0 replies
1d12h

https://www.unciphered.com/blog/randstorm-you-cant-patch-a-h...

Over the last 22 months, Unciphered has been working on a vulnerability which affected BitcoinJS, a popular package for the browser based generation of cryptocurrency wallets, as well as products and projects built from this software. Over a period of years, this vulnerability caused the generation of a significant number of vulnerable cryptocurrency wallets.

nonrandomstring
19 replies
1d8h

I feel like we, as an industry, got a bit lucky that someone with the skills, time, and energy was on hand at the right time to make a huge difference.

This is where statistics of "many eyes" and "sunlight as disinfectant" hits home. However bizarrely improbable that anyone would just wander along and find a a bug, people do, because they can. With proprietary/closed code, the probability is zero.

candiddevmike
11 replies
1d6h

With proprietary/closed code, the probability is zero.

People discover bugs all the time in closed source software...

nonrandomstring
8 replies
1d5h

Which people?

thfuran
4 replies
1d5h

Customers, though they usually aren't finding security vulnerabilities.

nonrandomstring
2 replies
1d5h

Customers

Ah I see what you mean.

Problem of language then.

"Finding bugs", to me as a coder, means something different from observing the effects of bugs.

Sure, I can see incorrect behaviours in lots of proprietary software. And I can guess what might cause it. But without the source code that's not the same as "finding bugs".

xyzzy_plugh
1 replies
1d4h

"finding a bug" in source code implies there's a mistake there. A bug can be that the implementation doesn't match the spec, or that the spec changed, with respect to the user (customer).

Reporting bugs vs finding them is perhaps what you're looking for, but I'd argue finding is overloaded. It's perfectly valid to kick the tires on some software and find some bugs.

I've found thousands of bugs in closed source software before, and subsequently reported those bugs.

nonrandomstring
0 replies
1d3h

"finding a bug" in source code implies there's a mistake there.

This is a good point of course. Implementation, protocol and runtime bugs are indeed invisible from the source POV. As are lower level Ken Thompson "trusting trust" [0,1] bugs in your tool chain.

Most of all though, many "bugs" are just malicious functions the coders meant to put in there.

[0] https://cybershow.uk/episodes.php?id=2

[1] www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf

prmoustache
0 replies
1d4h

In this very blogpost, we are very much in a situation where users (github + the author) found the bug.

They however didn't find the cause of the bug.

And in many software, open source or proprietary, there are a lot of bugs sitting out in issues tracker for days, weeks, months or even years.

myself248
1 replies
1d3h

Three letter agencies, and blackhats, who have the code even though it's not "open". They then use those vulnerabilities without disclosing them or fixing them, for as long as they're able.

nonrandomstring
0 replies
1d3h

Amen to that. Damn right proprietary code is a double danger in that regard. Officially illegal to inspect, except to a few criminals. Isn't the entirety of Microsoft's Windows code now "out there"? It has been for years, right?

smsm42
0 replies
1d1h

True. What is missing is the next step - looking at the code and figuring out why the weird thing is going on. In closed source, what'd happen is a call to support, who may or may not properly file a bug, which may or may not get to the eyes of an engineer that knows what's going on, who may or may not be interested in actually fixing it... But more likely it'd end up in the huge pile of "you're just holding it wrong" tickets and nobody ever would pay attention to it, because it's impossible for one company to pay attention to millions of weird complaints, and nobody else can. It's a numbers game, and with closed source, the numbers are much less favorable, because it's economically impossible for it to be otherwise.

noodlesUK
0 replies
1d1h

I'm sure that lots of HNers including myself have encountered security vulns in different proprietary pieces of software that we encounter.

I can think of at least two or three occasions where I discovered an exploitable vuln, didn't have a contact to report it to, decided I didn't want to have the back-and-forth of some non tech company threatening me with a lawsuit, and took no further action (typically for lower severity vulns in less important services).

With open source software you can hop on GitHub and dig deeper into a vuln or other bug and actually report it somewhere.

Twirrim
2 replies
1d2h

I was reflecting on xz being an absolute triumph of open source software:

1) Someone noticed something that was odd. They were able to go look at what was happening, along with source code and see that suspicious things had happened.

2) They were able to reach out to security experts across almost all the main distributions and get additional eyes on it who also confirmed there was a security thing and were immediately able to take actions to deal with the insanity.

3) Disclosure went public, and lots of eyes with expertise across all kinds of aspects of software and security have been able to tease apart what was done, and how, figuring out risks etc.

4) Other suspicious commits across other bits of software, from the same developer, have been tracked down and identified and the work continues in figuring out consequences.

5) Every distribution has become alert to the minutia of the ways that things were compromised around the build archives etc and have been able to start figuring out how to catch other cases and stop this in future. Stuff will start to spread to various open source projects as distributions work on their packaging tooling/processes, helping ensure other projects don't become vulnerable to compromise through the same mechanisms.

If you contrast this with closed source, where unless there is an exploit, it's reports of e.g. "Your software is running slightly slow" like in the xz/openssh case, is unlikely to get much attention, if any. Then once the closed source company finally finds out about it, at best you'll get a very carefully phrased explanation of what happened, revealing just the bare minimum amount of data they think they can possibly get away with. That badly harms the whole industry's ability to avoid repeats.

nonrandomstring
1 replies
1d2h

Absolutely agree that xz is a paradigmatic model of how well things can go with free open source. However, that also led me and others to a lot of dark reflecting on how vulnerable and exposed open source devs and maintainers are [0] and about complex social engineering attacks on the supply chain. It also raises troubling questions about how proprietary vendors can take advantage of that and spin it to their own ends.

[0] https://cybershow.uk/blog/posts/poison-code

gunapologist99
0 replies
1d1h

And SolarWinds is a prime example of the opposite approach.

rmetzler
1 replies
1d4h

I also think that open source is better than closed source. Nothing to argue about.

What I was wondering when I read the same sentence you quoted: how many really serious security bugs like Heartbleed, CVE-2008-0166, or the zx drama are happening without people finding out about it and publishing their findings?

rstuart4133
0 replies
17h16m

In open source there are only two likely outcomes when someone notices a security issue: either they plan to hoard it for their on gain, or the tell the world about it and earn the kudo's. The ability to earn kudo's is a right proper pain in the arse because a lot of things that have little to no impact on security are loudly touted as security bugs and so a lot of time is wasted on triage of non-security issues in open source.

The problem with closed source is there is a third possibility: ignore the problem and save on the cost of fixing it. The responsible disclosure regime we have now is because companies almost always chose this option, ie denied it was a problem and refused to invest to fix it. When the discoverer then released the bug anyway they we so enamoured this this approach tried solving the disclosure problem by suing the researcher.

If you think companies still don't ignore security issues when they are given a choice, you are kidding yourself. The problem compounds because when you do find a bug open source makes it easy to see if you can use it to create a security issue. In proprietary code that's much harder, so I'm 100% certain a fair number of potential security issues don't get patched because it isn't obvious how to exploit them. Nonetheless they are chinks in the armour, so they give the bad guys a excellent set of starting places to start looking.

pixl97
1 replies
1d5h

Well, in closed they find 'bugs' all the time, a large portion of the time you don't need the code at all. That said, you're not solving the issue and probably not doing anything about it.

On top of that, the vast majority of people won't know what to do if they find a bug. In the distant past I was one of those people, and only realized some of the bugs I saw in hindsight. It's been nearly 30 years so the details are mostly gone now, but I remember messing around in Windows and I could make the Microsoft Netmeeting application crash with a buffer overrun error.

Of course I was really new to computers then and understanding that buffer overflows in networked applications were really bad things (and seemingly beyond a lot of people that had been in the industry). Even then attempting to report security issues back in those days would have been far more difficult (hell and even risky in many cases).

So really it is many things that are required. Running into the issue. Having a deep enough understanding of computing to realize the issue is bad. Having a means of reporting the bug to a place where people will look at it. And a security culture that knows when and how to act on the bug report.

nonrandomstring
0 replies
1d4h

I think this is a good taxonomy/sequence to note

1) Observing

2) Understanding as bad

3) Reporting (or fixing)

4) Closing the loop on remedy

Both proprietary and open code runs into problems at (3) because people don't want to hear it, for commercial or ego reasons.

With FOSS at least you get the direct intervention route of simply fixing and publishing the patch, which done responsibly may or not force a maintainer's hand. With proprietary you can piss into the wind and be ghosted, or sued, hence more irresponsible/anonymous disclosure.

Anything that maximises the likelihood of getting from (1) to (4) safely must be a good thing, so I think FOSS yields the better security model.

cozzyd
18 replies
1d12h

I wonder if GitHub is still running a patched openssh...

ptman
8 replies
1d5h

OpenSSH 6.2 from 2013 added AuthorizedKeysCommand, so no patching needed.

jwilk
7 replies
1d4h

The command would still need to print millions of SSH keys on stdout, so this doesn't really help.

cozzyd
6 replies
1d4h

I think it takes the user as an argument, so it could relatively nicely solve the problem (though maybe would be too susceptible to DOS attacks since it requires a new process per connection attempt, although openssh should rate limit anyway?)

jwilk
5 replies
1d3h

The SSH username is always "git".

cozzyd
4 replies
1d3h

ah, right. I do wonder why they do it that way...

progval
3 replies
1d3h

HTTPS URLs are inconvenient for private repositories, and username has to be part of SSH URLs that people share when setting up a submodule or linking to a repository.

jwilk
2 replies
1d3h

username has to be part of SSH URLs

No? Usernames are optional in SSH URLs.

ongy
1 replies
1d2h

But then it defaults to the users username on device

Which isn't always in sync with their GitHub username. Which might even be "root".

cozzyd
0 replies
1d

surely anybody using GitHub via ssh knows how to edit their .ssh/config

TheDong
4 replies
1d12h

It's pretty easy to check something kinda close to GitHub's source code to see, if you want.

Just buy a copy of GitHub Enterprise, deobfuscate the files (it's a fun challenge to deobfuscate them, and not all that hard), and look around to see.

It's unfortunately not open source, so we can't share and talk about the code, link to it on GitHub, or anything like that, but if GitHub Enterprise is still using such a patch, it's like the prod github is too.

rudasn
1 replies
1d12h

How do they distribute github enterprise? Like a VM or image?

iforgotpassword
1 replies
1d12h

Assuming they wrote an entirely new sshd in ruby or something. Otherwise why should they ship the source code to a patched-up opensshd...

TheDong
0 replies
1d11h

You can see them writing the ssh public keys that are entered into the settings page _somewhere_ in ruby, and you can then kinda guess :)

mkj
1 replies
1d7h

They were using libssh in 2015. (Edit, I did say golang, but after checking it was bitbucket that used golang)

cozzyd
0 replies
1d4h

Indeed, the libssh web page lists GitHub as an adopter.

jarofgreen
0 replies
1d12h

Well, I'll bet they haven't gone back to "put all the keys in ~/.ssh/authorized_keys" ...

hanwenn
0 replies
1d12h

github uses something called babeld (just telnet to github.com port 22; the version string is printed immediately.)

immibis
6 replies
1d11h

As I understand it, the OpenSSL RNG was seeded by uninitialised stack memory and the PID. Debian seeded it with just the PID. But wasn't that already quite dangerous even without the Debian patch?

tialaramex
4 replies
1d7h

Yes, this misunderstanding does seem to be very popular. No, that's what was going on here, the OpenSSL code had two places where it copies of a bunch of bytes, and one of those copies can end up with uninitialized junk, so that's wrong.

Somebody wrote a patch which fixes this, and then - even without assistance from an LLM just using normal human incompetence, somebody said "There's another similar copy nearby, we should get rid of that one too" and Debian landed the patch to apply both changes.

The result is now OpenSSL doesn't copy any bytes. It doesn't copy the uninitialized data, which is good, and it also doesn't copy the truly random entropy into the pool. Oops.

ptx
1 replies
22h19m

From Russ Cox's post[0] about this, it sounds like the place in the code that initializes the buffer with actual random data also uses the parts of the buffer that haven't been initialized. So there weren't two separate pieces of code, one copying junk and one copying actual random entropy, but rather "at least three different places" that did both, if I've understood this correctly.

[0] https://research.swtch.com/openssl

tialaramex
0 replies
18h40m

Cox had the advantage (I presume) when writing this of looking at the original source, for which all the links I found are long dead (because presumably this code moved to git years later) and he's a smart guy so we should definitely believe what he describes over my recollections.

But I think you're slightly wrong here, I think Cox is describing three cases where OpenSSL takes a bunch of bytes and says here you go entropy pool, here's some potential entropy to add, if it wasn't really entropy that's fine, stir it in anyway.

A. Buffers you've provided and are asking to pour random data into. OpenSSL thinks this is clever - best case the data is unpredictable and it helps, worst case it's just zeroes and the entropy pool will handle that anyway. Valgrind and similar tools will be very angry when you do this to actual uninitialized buffers, which in C is really easy to write and so likely to be common. This is the case that already had a comment about such problems and is the second of two places Debian removed.

B. Buffers OpenSSL tried to put entropy from the Linux random device into. OpenSSL doesn't bother checking that the randomness went into the buffer, after all if it didn't the buffer is uninitialized... This is the first of the two places Debian removed (in terms of line number). This is where "Debian Weak Keys" come from because now the "random" numbers are not random. In almost any real world scenario this code was unproblematic although very stupid, until it was patched.

C. A Buffer read from a file the user provided with entropy in it. This only ends up with uninitialized data if the file shrinks while OpenSSL is trying to read it, which can happen but would probably be some sort of attack. Again though OpenSSL doesn't care, Debian did not patch this code AFAIK but very few people care about it.

Edited to add: One thing I really like in the discussion below the actual post you linked is the idea that the OpenSSL people are thinking the Debian dev wants to debug their software and are saying it's fine to comment out the randomness in order to help them do debugging not realising the developer actually intends to remove the randomness because they believe it's a bug. When you re-read the mailing list posts with this understanding and forgetting the Debian Weak Keys bugs, it snaps into place - we can't know if that's really what happened of course, but it makes sense.

orra
1 replies
1d3h

and one of those copies can end up with uninitialized junk, so that's wrong.

To be fair to the Debian patch, reading from uninitialized junk doesn't sprinkle randomness into the desired buffer: it invokes undefined behaviour. The Debian patch was lawful!

tedunangst
0 replies
23h46m

The part where they also zeroed the buffer with intentionally random data may have been lawful, but it wasn't good.

jwilk
0 replies
1d10h

This is incorrect. OpenSSL seeded its RNG also with data read from /dev/urandom.

matsemann
4 replies
1d11h

Ezra Zygmuntowitz pointed GitHub in my direction, and let me take the time to really get into the problem with the GitHub team

This sentence is funny to me. Maybe because I'm not a native speaker, but I can't avoid reading it as a big problem with the GH team itself, heh. I was expecting the next sentences to "get into" that.

I wonder if Luciano hadn’t found it, how long it might have been before it was found

I think perhaps only GH or one of the big cloud providers would have stumbled upon it, few places I can think of that would have thousands and thousands of keys stored from users.

arnsholt
1 replies
1d9h

In syntactic problem, this is called the PP attachment problem. Basically, should the sentence read (get into the problem) (with the GH team) or (get into (the problem with the GH team)). Turns out to be a very hard thing to do correctly.

gosub100
0 replies
1d5h

Is that the same thing as being a context sensitive grammar?

alias_neo
1 replies
1d10h

This sentence is funny to me

It does / can read both ways, but a comma after "problem" would have removed that doubt.

thedanbob
0 replies
1d4h

But that wouldn't be correct syntax. That would imply the middle section was an addition to the complete thought of the outer two sections, e.g.

Ezra Zygmuntowitz pointed GitHub in my direction (and let me take the time to really get into the problem) with the GitHub team

Ezra Zygmuntowitz pointed GitHub in my direction (...) with the GitHub team
ckastner
4 replies
1d10h

An important fact that I only recently learned about this vulnerability is that the change that introduced it was not some rushed act: the maintainer raised the problem they were seeing on the OpenSSL mailing list, proposed a change to fix the problem with a request for feedback, and got some (including from upstream).

The result was a terrible vulnerability, but it seems more of a case of spectacularly bad luck of everyone not spotting the issue.

yashap
2 replies
1d4h

Probably more lacking automated test coverage than bad luck? For security critical RNG code, seems to me you’d really want to have a test where it generates a tonne of random numbers and asserts that they’re all unique.

gunapologist99
0 replies
1d1h

It's not possible to assert that a number is not random (or is). You can look at it and you just don't know. This was the issue with the PRNG blackbox mixing.

Obligatory Dilbert https://imgur.com/uR4WuQ0 and XKCD: https://xkcd.com/221/

bdonlan
0 replies
1d4h

In this case, the problem was the seed was low entropy, so if you generated a bunch of random numbers with a single initialization of the RNG seed you would get unique values. It's a tricky scenario to test for if you don't know of the failure mode...

orra
0 replies
1d3h

At the time, it felt like Debian got a lot of flack for the bug, but there's the collaboration attempt you mention above.

Plus, the upstream OpenSSL code was invoking undefined behaviour. Hence the compiler could have validly made the exact same transformation as the Debian maintainer. At the time this felt academic: surely compilers can't be that mean! Since then I think undefined behaviour is better understood as a thing to avoid entirely.

Then, eight years later, Heartbleed was discovered. And we suddenly all realised how badly maintained OpenSSL was. In their defence, it was pretty much a volunteer job. Thankfully, subsequent funding has improved the situation.

mkesper
3 replies
1d11h

I remember looking at SSH public key fingerprints in Savannah at the time and wondering why SO MANY started with AAAA (thought it was some padding first...)

denton-scratch
1 replies
1d

That's the header on the public key itself; the fingerprint GP referred to doesn't have a "header".

tedunangst
0 replies
23h44m

The broken keys didn't all have fingerprints of AAAA either.

Sesse__
2 replies
1d10h

“While I’ve not found a description of exactly when and how Luciano Bello discovered the vulnerability that became CVE-2008-0166 […]”

From my IRC logs at the time:

  17:23 < luciano> has really an accident. I was needing many primes numbers... 0:-)
  17:23 < Sesse> and you got the same numbers every time?
  17:25 < luciano> Sesse, not every time :P

rmetzler
0 replies
1d4h

Luciano in these lines sounds like he just generated lots of keys and then started to wonder why he got more collisions than expected.

Thanks for sharing.

cryptonector
0 replies
1d1h

I miss bash.org.

tzs
1 replies
1d1h

After examining a variety of different possible solutions, we came to the conclusion that the least-worst option was to patch OpenSSH to lookup keys in a MySQL database, indexed on the key fingerprint.

Why MySQL rather than sqlite?

They were trying to speed up access to ~/.ssh/authorized_keys. That's the kind of situation mysql is designed to shine in. I would have expected that patching OpenSSH to check for ~/.ssh/authorized_keys.db would be less work than patching it to use MySQL.

klysm
0 replies
1d

I’d imagine there are many machines involved and maintaining one database is typically easier than maintaining many. I’d also imagine MySQL was already in operation, so there wasn’t any startup cost to storing data there. They already had a database of users _somewhere_

tomlong
0 replies
1d8h

I realise it is separate to the weak key prime discovery, but it is interesting how much slow SSH login times are a thread that is worth pulling for one reason or another.