This seems like a weird place to be touting memory safety.
It's ntpd, it doesn't seem like a place for any sort of attack vector and it's been running on many VMs without exploding memory for a while now.
I'd think there are far more critical components to rewrite in a memory safe language than the clock synchronizer.
I'm the person driving this.
NTP is worth moving to a memory safe language but of course it's not the single most critical thing in our entire stack to make memory safe. I don't think anyone is claiming that. It's simply the first component that got to production status, a good place to start.
NTP is a component worth moving to a memory safe language because it's a widely used critical service on a network boundary. A quick Google for NTP vulnerabilities will show you that there are plenty of memory safety vulnerabilities lurking in C NTP implementations:
https://www.cvedetails.com/vulnerability-list/vendor_id-2153...
Some of these are severe, some aren't. It's only a matter of time though until another severe one pops up.
I don't think any critical service on a network boundary should be written in C/C++, we know too much at this point to think that's a good idea. It will take a while to change that across the board though.
If I had to pick the most important thing in the context of Let's Encrypt to move to a memory safe language it would be DNS. We have been investing heavily in Hickory DNS but it's not ready for production at Let's Encrypt yet (our usage of DNS is a bit more complex than the average use case).
https://github.com/hickory-dns/hickory-dns
Work is proceeding at a rapid pace and I expect Hickory DNS to be deployed at Let's Encrypt in 2025.
Why are C and C++ all of a sudden unsafe? Did I miss something?
What is safe now? JavaScript? PyTorch?
One of the major drivers (if not the driver) for the creation of Rust the fact that C is not a memory-safe language.
This has been known for decades, but it wasn't until 2010 that a serious attempt at writing a new system-language that was memory safe was attempted and got traction - Rust.
https://kruschecompany.com/rust-language-concise-overview/#:....
How is C not memory safe? If I access memory I didn't allocate the OS shuts the program down. Is that not memory safety?
(Unless you're running it on bare metal ...)
The real problem is when you access memory that did allocate.
So we need a new flag for gcc that writes zeros to any block of allocated memory before malloc returns, not a new language.
We have that already. There are still other problems that exist.
That wouldn't make it safe. It would just make it crash in a different way, and still be vulnerable to exploitation by an attacker.
You'd probably want an alternative libc implementation rather than a compiler flag.
However, calloc everywhere won't save you from smashing the stack or pointer type confusion (a common source of JavaScript exploits). Very rarely is leftover data from freed memory the source of an exploit.
If only the very competent people that decided to create Rust had thought of asking you for the solution instead...
Have a little humility.
This is not a question for HN at this point. It’s like asking why SQL query forming via string concatenation of user inputs is unsafe.
Google it, C memory boundary issues have been a problem for security forever.
To be honest, that's a surprisingly deep question, and the answer is something I'm yet to see any developer I worked with understand. For example, did you know that SQL injection and XSS are really the same problem? And so is using template systems[0] like Mustache?
In my experience, very few people appreciate that the issue with "SQL query forming via string concatenation" isn't in the "SQL" part, but in the "string concatenation" part.
--
[0] - https://en.wikipedia.org/wiki/Template_processor
Really, the problem is in combining tainted input strings with string concatenation. If you have certain guarantees on the input strings, concatenation can be safe. That said, I still wouldn’t use it since there are few guarantees that future code wouldn’t introduce tainted strings.
The problem that causes it is in photo cases that a flat string representation kind of mixes code and data. And therefore escaping (such as not allowing quotes in the user input to terminate quotes in your query, or not allowing html tags to be opened or closed by user input, nor allowing html attributes to be opened or closed by the user input, etc) is needed.
Really? To me it's pretty obvious that not escaping properly is the issue, and therefore the same issue applies wherever you need escaping. I don't think I've ever heard anyone say that SQL itself was the problem with SQL injection. (Although you certainly could argue that - SQL could be designed in such a way that prepared statements are mandatory and there's simply no syntax for inline values.)
Memory errors can be exploited by a clever adversary to control your process in a variety of unpleasant ways, see: https://en.wikipedia.org/wiki/Memory_safety#Types_of_memory_...
I think you have a missunderstanding on what memory safety is.
Memory safety means that your program is free of memory corruption bugs (such as buff overflow or under flow bugs) that could be used to retrieve data the user isn't supposed to have, or can be used to inject code/commands that then get run in the programs process. These don't get shut down by the OS because the program is interacting with its own memory.
No, that's a security vulnerability waiting for someone to exploit.
That's not what memory safety refers to.
Highly recommend Alex Gaynor's intro to memory unsafety https://alexgaynor.net/2019/aug/12/introduction-to-memory-un...
I would consider that serious attempts have been made since 1961, however all of them failed in the presence of a free beer OS, with free beer unsafe compilers.
The getting traction part is the relevance part.
Yeah, all previous attempts at making such a language lacked two things:
1. They didn't have a big, well-known, company name with good enough reputation to attract contributors.
2. They didn't have brackets.
Success was because traction, traction was because appeal, and appeal was mostly because those two things. Nothing else was new AFAIK.
No shared mutable state in an imperative language is common, as are memory safe languages with performance close to C's? Didn't see the latter in the language shootout benchmarks, in fact Rust is much closer to C than the next closest thing
I take a slightly different view of this, though I do not deny that those things are important. #1 in particular was important to my early involvement in Rust, though I had also tinkered with several other "C replacement" languages in the past.
A lot of them simply assumed that some amount of "overhead," vaguely described, was acceptable to the target audience. Either tracing GC or reference counting. "but we're kinda close to C performance" was a thing that was said, but wasn't really actually true. Or rather, it was true in a different sense: as computers got faster, more application domains could deal with some overhead, and so the true domain of "low level programming languages" shrunk. And so finally, Rust came along, being able to tackle the true "last mile" for memory unsafety.
Rust even almost missed the boat on this up until as late as one year before Rust 1.0! We would have gotten closer than most if RFC 230 hadn't landed, but in retrospect this decision was absolutely pivotal for Rust to rise to the degree that it has.
Tcl
They have been unsafe from their very early days, Multics got a higher security score than UNIX thanks to PL/I, and C.A.R Hoare has addressed C's existence on his Turing Award in 1980, while Fran Allen has also made similar remarks back in the day.
All of a sudden? They've been unsafe for decades, it's just that you had less of a choice then.
Because you can cast a pointer to a number and back again. Then you can stuff that value into index [-4] of your array. More or less.
It continues to astonish me how little people care (i.e., it triggers the $%&@ out of me). I really appreciate the professionalism and cool rationale when faced with absolute ignorance of how shaky a foundation our "modern" software stack is built upon. This is a huge service to the community, kudos to you and many others slowly grinding out progress!
Lol, shaky indeed. A business person once said, "can you imagine if machine engineer (like auto makers) behave like software engineering?".
Seems no digital system is truly secure. Moving foundational code to memory safe seems like a good first step.
That's because there is no such thing as "truly secure", there can only be "secure under an assumed threat model, where the attacker has these specific capabilities: ...". I agree that software engineering is getting away with chaos and insanity compared to civil or other engineering practices, which have to obey the laws of physics.
Most of the security bugs we hear about don't cause random crashes on otherwise healthy machines, because that tends to get them noticed and fixed. It's the ones that require complicated steps to trigger that are really scary. When I look at NTP, I see a service that:
- runs as root
- talks to the network
- doesn't usually authenticate its traffic
- uses a bespoke binary packet format
- almost all network security depends on (for checking cert expiration)
That looks to me like an excellent candidate for a memory-safe reimplementation.
ntpd can (and should) run as a user
Makes outbound requests to the network. For it to be compromised, the network itself or a downstream server needs to be compromised. That's very different from something like hosting an http server.
Yes it does. ntp uses TLS to communicate with it's well known locations.
Not great but also see above where it's talking to well known locations authenticated and running as a user.
It's a service that to be compromised requires state level interference.
I don’t think NTP uses TLS by default. The closest equivalent I can find online is NTS, which was only standardized in 2020 (and for which I can’t find clear adoption numbers).
(But regardless: “the transport layer is secure” is not a good reason to leave memory unsafe code on the network boundary. Conventional wisdom in these settings is to do the “defense in depth” thing and assume that transport security can be circumvented.)
ntpd-rs supports NTS and we (Let’s Encrypt) are using it a little bit. But it is far from having any wide adoption yet.
There aren’t many public NTS servers: https://netfuture.ch/public-nts-server-list/ and https://github.com/jauderho/nts-servers have some listed.
The network is just a bunch of strangers in a meet me room.
Ntp does not use TLS widely. It’s also a UDP protocol which makes it subject to spoofing attacks without a compromised network (no protection from kernel network stack verifying TCP sequence numbers).
For servers it's definitely harder, although definitely not only state-level, but another big issue could be client-side. NTP servers can be set by DHCP, so the admin of any network you connect to could exploit such a bug against you. And once you have code execution on a desktop OS, all bets are off, even if you're not under the primary UID.
It's not the most important threat vector, but it also doesn't seem as difficult as some of the other system services to rewrite, so I'd say it was a good first step for the memory-safe-everything project.
My knee-jerk reaction is that TLS is not authentication. After skimming through the relevant spec [0], it is interesting how NTP uses TLS.
[NTS-KE] uses TLS to establish keys, to provide the client with an initial supply of cookies, and to negotiate some additional protocol options. After this, the TLS channel is closed with no per-client state remaining on the server side. [0]
0. https://datatracker.ietf.org/doc/html/rfc8915
It's present on loads of systems, it's a very common service to offer, it's a reasonably well-constrained use case, and the fact that nobody thinks about it might be a good reason to think about it. They can't boil the ocean but one service at a time is a reasonable approach.
I'll flip the question around, why not start at ntpd?
Easy, because there are loads of critical infrastructure written in C++ that is commonly executed on pretty much every VM and exposed in such a way that vulnerabilities are disasterous.
For example, JEMalloc is used by nearly every app compiled in *nix.
Perhaps systemd which is just about everywhere running everything.
Maybe sshd, heaven knows it's been the root of many attacks.
There’s a nice pure-rust ssh client/server already.
Systemd should just be scrapped. This week’s wtf “systemd-tmpfile —-purge” intentionally changed its behavior to “rm -rf /home”. Confirmed not-a-bug. There are dozens of other comparable screwups in that stack, even ignoring the long list of CVEs (including dns and ntp, I think). Rust can’t fix that.
I haven’t heard of any issues with jemalloc, though that seems reasonable (assuming calling rust from C doesn’t break compiler inlining, etc).
Prod ready, audited, non-buggy in actual use? And if so, do you have a link so I can start test-deploying it?
Initially closed not a bug, but then Poettering overruled that decision and implemented a fix which is already released.
https://github.com/systemd/systemd/commit/e76015738942246db7...
JEmalloc is used by very very very few apps compiled for *nix. That's a conscious decision that an app developer needs to make (and few would bother without specialized needs) or a distro packager needs to shoehorn into their package builds (which most/all would not do).
I do think that memory safety is important for any network service. The probability of something going horribly wrong when a network packet is parsed in a wrong way is just too high. NTP typically does have more access to the host OS than other daemons, with it needing to adjust the system clock.
Of course, there are many other services that could be made memory safe, and maybe there is some sort of right or smart order in which we should make our core network infrastructure memory safe. But everyone has their own priorities here, and I feel like this could end up being an endless debate of whatabout-ism. There is no right place to start, other than to just start.
Aside from memory safety though, I feel like our implementation has a strong focus on security in general. We try and make choices that make our implementation more robust than what was out there previously. Aside from that, I think the NTP space has had an under supply of implementations, with there only being a few major open source implementations (like ntpd, ntpsec and chrony). Meanwhile, NTP is one of those pieces of technology at the core of many of the things we do on the modern internet. Knowing the current time is one of these things you just need in order to trust many of the things we take for granted (without knowledge of the current time, your TLS connection could never be trusted). I think NTP definitely deserves this attention and could use a bunch more attention.
NTP is a ubiquitous network service that runs directly exposed to the Internet, and that seems to me like a good thing to harden. Making NTP more secure does not stop anyone else from working on any other project.
Agreed. It is a "good old" binary protocol, so the many gotchas of text protocols are not there.