The server chose violence

This sounds good when the system is small and tight and applications are written mostly by people who designed the whole system.

I think that's intentional because that's what Hubris is aimed at.

This sounds good when the system is small and tight and applications are written mostly by people who designed the whole system.

Swift death to deviance is a way to keep the system tight. The designed scope probably keeps it small anyway. Scopes have a way of creeping, but I don't think people will want to force tasks into Hubris that would be better on the host rather than in its embedded controllers.

To be fair, this is how abort works in any library you call into that’s in your process.

not handling “theoretical” errors that can’t occur in practice

The Dennis Nedry approach to counting dinosaurs in Jurassic Park.

Indeed, this happened with Symbian. An IPC server could panic the client. As an application developer without access to the OS source code this was pretty terrible. Not all preconditions were easily understood and could vary between devices and OS versions.

But as an application developer, I’d be somewhat scared to interface with third-party code over an IPC model where the other service can at any time send back an instant death pill to my process.

For service, think "OS interface". If you make a bogus kernel call on a monolithic kernel, it would be reasonable for the OS to kill you. Also note that when you say "process" it might be different than you think because threads all share the same address space on hubris.

It seems like in an embedded environment, it's good to resolve these misunderstandings immediately when they occur, regardless of whose fault it is.

The server says "that client is bad!" so the kernel kills it. The problem is really that the two didn't understand each other.

I am not familiar with what you're talking about (but Cliff may already know), I'll have to look into it. But Hubris is a synchronous system, and also, these faults aren't catchable, so I'm not sure how directly relevant it is. What's the specific issue you're worried about?

Your Erlang vibes are there for good reason, it's certainly an influence.

Recommended languages are Java (ultimately a failure despite vast effort)

Why is Java a failure? Recent JVMs have come a long way, and GraalVM makes it somewhat comparable to Go-like languages.

I understand the historical hate and how Oracle bought it, but it really isn’t that bad of a language if you’re using modern Java.

huh? Do not see anything asynchronous in author's work. It's all synchronous, because IPCs in hubris are synchronous too.

But by what mechanism would that strategy be able to understand the fault that occurred, in order to try again better?

I think the general idea is to apply this to problems which are clearly the result of an invalid program state, and therefore not reasonably recoverable. They are either caused by bugs, an attack, or corrupted hardware. In all cases you shouldn't continue, because there's something seriously wrong with the caller. If the caller continues, it could only cause more damage.

It sounds a bit like Erlang/OTP's "let it crash" philosophy. Erlang is used in quite a bunch of mission-critical hardware and is famous for its reliability, so it might not be such a huge dealbreaker in practice.

that can tolerate no real-world chaos, and I'm not aware of any commercially viable realms which would either.

Watchdog timers will happily kill/restart your processes that don't poke them often enough. Even in my hobby exercises I've seen I2C busses hang up often enough(and bring the whole system down!) when some protocol bit goes wrong that I think the design is actually quite inspired. As I understand it this isn't talking about known error cases(that are handled) but protocol mismatches and other things that shouldn't ever happen.

Many other comments touched on it but it's a purpose built OS, much in the same way I'm not going to build a UI in Erlang, Hubris seems well positioned for the space that it occupies.

It’s a 2000 line rust embedded systems kernel that doesn’t support adding new tasks at runtime. It is written to go deep in the guts of the 0xide server racks.

Hubris is not an academic exercise: it runs at the heart of every element of the Oxide rack (compute sled, switch, power shelf controller) -- and its design is informed by delivered utility above all else. Indeed -- and as Cliff elaborated in the blog -- REPLY_FAULT was something that he thought initially perhaps too aggressive, but it was our own experience in building, deploying, and (it must be said!) debugging the system that gave him the confidence that it would make our systems more robust, not capriciously faulty.

For more details on the thinking here and what it looks like in practice, see (e.g.) [0] and [1].

[0] https://www.mattkeeter.com/blog/2024-03-25-packing/

[1] https://cliffle.com/blog/who-killed-the-network-switch/

Hubris isn't a general-purpose OS, it runs on a low-level processor inside the Oxide server rack. I believe Hubris doesn't even allow new kinds of processes at runtime; all possible executables must be determined at compile time.

I find this attitude bizarre. Just earlier today I used python debugging to quickly figure out why an error was occurring. Being able to see the state of the variables without having to print each helped solve it instantly.

I find more bugs with a debugger. There’s typically the bug I was looking for, and then smaller bugs that didn’t technically cause the problem but contributed, and may be involved in the next issue. I want to fix those too, and sometimes first.

It's partly a religious thing, partly what you're used to and partly using the right tool for the job. Some programmers use debuggers as a crutch and some complex systems (e.g. that involve multiple distributed components or are timing dependent) can't be easily debugged using traditional debuggers.

EDIT: yet another factor is sometimes you may not even have access to the system you need to troubleshoot. Being able to reason about code execution without observing it is a useful skill (and still a debugger is a useful tool).

It seems that Hubris is not designed as a general-purpose operating system. Processes are defined at build time.

The reason why servers can shoot back at their clients is reliability, not security. Errors are thought to originate from bugs, not from deliberate attacks. The extreme reaction of the kernel ensures that developers find them as soon as possible.

Of course, there is an overlap with security, and this can be a useful fallback measure in the event that a process tries to do something that it isn't supposed to do.

I think when B gets faulted A would get an error about a dead server and would have the opportunity resend the same message to a newly reset server not a cascading crash.

In some sense, yes, this is kind of like the kernel sending SIGKILL to a process.

I don't think this will make reuse much worse even in a general programs, as long as there is a good division between expected errors (file not found) and unexpected (invalid operation code). In fact, there are a lot of ignorable errors in Unix which IMHO should have been raising a fatal signal instead, as this would substantially improve general software quality.

As an example: trying to close() invalid FD is a a non-fatal error which is very often ignored. But it is actually super dangerous, especially in multi-threaded apps: closing wrong fd will harmlessly fail most of the time, but 1% of time you'll close a logging socket or a database lock file or some unrelated IPC connection.. That's how you get unreliable software everyone hates.

In a sense, it does: waiting is one of the main jobs of an OS kernel.

Yes, we have an in situ dump facility, which Cliff mentioned at the end of [0]; it's been essential for debugging these issues when we hit them.

[0] https://cliffle.com/blog/who-killed-the-network-switch/

In a sense, though most would think of "remote" as being "over the network," and that's not the case here.

The designer of Hubris (and several folks who work on it) are familiar with QNX, for sure.