Bytecode Breakdown: Unraveling Factorio's Lua Security Flaws

Factorio disabled bytecode loading in response to this. Bytecode did allow for some cool stuff like writing mods in a preprocessor language that spits out Lua bytecode, but ultimately the security issues were more important to address.

Almost all of the debug library was made unavailable to mods as well, for similar security reasons.

Factorio has stuff like this:

https://mods.factorio.com/mod/Moon_Logic

Besides, it's quite limiting to create software that can't just execute in a Turing complete environment.

Anyway, we really need interpreters that include a strong capability system.

not just because of rice's theorem

I don't think Rice is relevant at all. I guess Rice is a useful first screen. Do you believe you can "just" Decide this correctly? If so, Henry Rice got his PhD half a century ago for proving you can't do that, stop.

But assuming you've made your peace with accepting only a subset of the inputs that would actually meet your requirements, Rice is done. And you're right - now instead of an impossible task you've just got an extremely hard task. This means when you fail (which you will) at least nobody will tell you it was impossible, if that helps?

Factorio should not have done this.

In general, verifying programs is extremely hard, not just because of rice's theorem but because it's so easy to miss a spot, especially for non-trivial bytecode languages like lua's.

Perhaps a dumb question then. Java has famously had a bytecode verifier for decades. Is it the case that:

a) bytecode verification is fundamentally easier in statically typed languages

or

b) it's just as hard for Java, but Java has had decades to work on it and it's still taken a long time to fix all the bugs/security issues.

Eventually every game developer learns the hard way that they must remove the bytecode ability from lua's loadstring() function.

E.g. here's a 12 year old blogpost on the topic from the ROBLOX developers: https://archive.is/oXPyM

To be honest, it would probably be better off disabled by default. Its legitimate uses are pretty niche.

rice's theorem

That doesn't apply here. By the broad definition of "syntax" that Rice's theorem takes, the things you want to verify on the bytecode are syntax.

Java, Wasm and BPF demonstrate that it is possible to have statically-verifiable bytecode for JIT-compiled languages. Lua’s problem is that the bytecode doesn’t provide the information necessary to fully verify its safety.

Lua isn't really sandbox friendly, although that's a common misconception.

Lua (by design) doesn't provide termination guarantees or a good way to force an untrusted program to terminate. If you accept untrusted lua input, be prepared for your program to halt indefinitely.

Lua is great for semi-trusted input, AKA things you download from the internet, where you do a minimal amount of due diligence, but in case the code is actually malicious, you want to severely limit (but not completely eliminate) the harm it can do.

If you actually need Javascript-style completely untrusted input,, what you want is the Roblox fork called Luau[1].

[1] https://luau-lang.org/sandbox

I wonder if this affects Roblox variant of Lua?

I wonder if they'd take patches to stop the interpreter trusting the input

The stance of the Lua developers AIUI is that processes that execute arbitrary Lua code should only accept source, and disable direct loading of byte code.

This seems reasonable to me, as it preserves the option of directly loading trusted byte code, while avoiding putting dynamic checks into the interpreter that would affect all users.

Sandbox friendly?

It's not exactly a simple task to make safe interpreter for bytecode as some other languages have shown. It's a trade-off which also simplifies the reference implementation.

I wouldn't trust most of these interpreters when it comes to third party code execution, I barely trust my browser even with all the R&D money and attention web browsers receive.

No, expected. Only execute bytecode that was actually generated by the correct compiler. Otherwise you get memory safety violations or sandbox escapes. (Or sandbox escapes via memory safety violations.)

Just as you don't execute arbitrary machine code.

Luau has the same thing, and you don't see Roblox suffering from sandbox escapes all the time, do you?

How is JIT relevant here? Unless I missed something, the attack uses straight-up malicious byte code, it doesn't exploit the JIT compiler.

The issue here isn’t things being “verifiably secure”. languages like js and lua run in a sandboxes environment where the only functions and operations that are permitted are those explicitly added by the host environment. Those sources languages are easily validated as correct code.

[edit: I realize I should clarify something “correct” and “verifiable” here do not mean “bug free”, it means ‘cannot interfere with or violate language or environment state, memory, or other invariants’]

The issue here is that the hosting environment is allowing the user/attacker to provide the bytecode that is generated from the provably correct code. That byte code is not itself verifiable statically, and is not verified at runtime (and it might not even be possible to).

This is not to say that bytecode is not verifiable - Java, .NET, or even WASM (which is intentionally very low level) are verifiable bytecode environments. The issue is that a byte code must be _designed_ to be verifiable (and early Java bytecode was not due to JSR or similar iirc). Lua’s bytecode is designed for execution, and so allowing arbitrary bytecode execution is not too dissimilar from a JS engine allowing a website to provide direct access to their bytecode interpreter which would be similarly catastrophic.

JIT has not been secure since Spectre and Meltdown.

Not sure why this comment got downvoted or flagged all the way to ghost text. Seems reasonable to me even if some people disagree.

(I actually disagree also but I can still respect this take, which was clearly also well intentioned and spoken.)

I would take a look at BPF.

A multiplayer game where a server can be extended with custom code via plugins

A game called Mordhau (based on Unreal engine) had a built-in "message of the day" feature where server owners can put in a URL that loads an in-game browser when the player connects. No client side option existed to disable the browser and I believe the devs eventually disabled it completely but I'm not sure the status of it now.

Just shows how complex games / game engines are getting where you have an embedded web browser for seemingly no good reason.

You should never assume any method of executing any attacker controlled code is safe, unless something explicitly calls that out and also has put Google-level amounts of effort into supporting that.

The first thing to look for is if the solution states clearly that it is a speculation-safe sandbox. I do think that not many will do that, but there are some. And go from there.

Most likely, it’s not very good. Why do you think every console manufacturer, from Xbox to Sony to Nintendo, does not allow connecting to arbitrary server IP addresses or modding support?

It’s not merely a business decision (like some believe) to force people to use official Online services. Think about it: Restricting connecting to third-party server IPs means that any bugs in the network code, or in the rest of the game, even atrocious ones, will never be exploited. Restricting mods (even “safe” mods like Lua) further prevents exploits. This makes sense - buggy network code has tanked multiple consoles’s DRM historically.

And not just exploits - these consoles pride themselves on doing their review process before the code becomes available (despite oversights). Allowing executing of Lua, from a remote system, basically means a game could be reconfigured remotely after approval potentially from the developer themselves - not something any console manufacturer wants to permit without very close inspection.

Code practices? Factorio is one of the most well programmed, stable, and consistent piece of software I've ever seen. It's almost a shame to see skilled people work in games because of how desperately other fields need people who are good at programming.

Yup. Intend to assume that any game with remote interaction is completely insecure. It is best to run Steam and all games in some sort of sandbox.

Flatpak is probably a helpful start. While containers aren't a strong security boundary at least simple exploits won't work.

Yeah, best to keep a separate computer for gaming for this reason. Definitely don’t put important documents or work stuff on it. It would be ideal to isolate it in a VM, but setting up a gaming VM can be a massive pain in the ass and exclude you from some games that use anticheat.

The mechanism Factorio uses is to sync user inputs, not game state changes (the reason isn’t explained, but I strongly suspect it’s because user inputs are less data; small inputs can cause big game state changes, but not vice versa).

If the user types a command, in order to preserve synchronization, the game must:

- Run the command on all other clients.

- OR it could sync changes made to the game for just commands; in other words, the other clients apply the changes caused by the command instead of running the command directly. But that would be an unreasonable amount of extra work just for a small feature and to make exploits harder.

- OR the server simply disallows clients from running Lua commands, which is the case for some servers.

I don’t get the second part though: why a map can store arbitrary Lua code that runs when the map loads.

If clients don't run the same code they will desync the moment their state diverges. I haven't played multiplayer factorio but I expect you can't even join a server unless you're running the same factorio version and mods as other players.

why can’t we just run it on their machine and propagate any game state changes (if the script adds an inserter, for example,)

Because that's an unbounded amount of traffic. You can reliably write data into RAM at many gigabits per second, whereas network connections are variable and many of them won't carry more than a few kilobits at the 99th percentile (note that you roll that 100-sided die like 30 times per second, so "1% situation" lag spikes are something you'd run into constantly)

I sometimes use Lua commands in single player to clear biters from a certain region for example, which removes many entities. Propagating those sorts of changes on multiplayer (or take a more plausible example: wave defense that eventually spawns loads of entities at once) would cause a big lag spike if you have a few players that all need to receive this data, whereas simulating it locally on each machine is no problem

Factorio multiplayer bandwidth is like a dozen kilobytes per second from what I remember, and this post agrees https://forums.factorio.com/viewtopic.php?p=125328#p125328 (couldn't quickly find an exact number though it must surely be out there). If you make it O(n) for every lua-touched entity in the game, it would quickly balloon into the megabits constantly and many mods would just not be viable for multiplayer for most people

If a client runs a Lua script, why can’t we just run it on their machine and propagate any game state changes (if the script adds an inserted, for example,) as if the player made those changes themselves?

The game already has to run Lua scripts as part of the simulation, potentially as part of in-game events which aren't directly triggered by players. A player running a script from the console is handled by that same interpreter -- making it run in a completely different operating mode where any changes to game state are replicated would be much more complicated and prone to error.

Or, from the other direction: the game's multiplayer model is all based around a replicated simulation where player inputs are fed into the simulation. Treating a player running a script as a special kind of event involving the text of that script is the simplest and most obviously correct way to implement that.

Running the scripts outside the simulation and syncing their commands alongside user input would definitely work on a technical level.

But I think you're massively underestimating how much these scripts can do. Many mods would flood the network connections. And there would also be an awkward delay for all script actions.

Yes, but only because you might lose your job from playing too much factorio. :) the exploit was not a risk for vanilla unmodded single players, and has been patched in any event.

No.

The interesting takeaway I got was how badly the Lua developers failed on their bytecode veryfier. Not some complex issues, but simple ones like of by one errors when modelling basic instructions like jmp or the issue that the Lua interpreter would try to interpret anything it got its hands on as instructions, even data sections the veryfier would not touch.

That advertised features can still cause harm to end users, particularly those who don’t know what Lua or bytecode are?

It’s possible that the bytecode interpreter has a bug that lets one run arbitrary bytecode, even in environments where `loadstring` is disabled.

It’s been several years since I’ve tried but most JS engines are out of date and barely maintained and the ones used for browsers are made for browsers first and they are not built for you to integrate at all.

Lua is specifically built for you to integrate so there are many resources and a huge community backing you.

1. Most JS engines are vastly more complex to embed than Lua, which is one of the easiest pieces of software to compile I can think of 2. You're conflating common browser APIs with JS. A JS engine does not provide a canvas, or a DOM. V8 does not, for example: those are things you would need to add yourself.

Yup. I'm the one who did the Roblox bytecode exploit that lead to it being disabled: the specific attack there was that getmetatable internally leaves the metatable value on the Lua value stack even if it ends up returning the __metatable locked message, which you could retrieve with a crafted bytecode chunk. I leveraged that into getting the metatable for the global environment, which was the entire Lua standard library, and crucially the same table across Roblox's script permission levels. Roblox used "context levels" for seperating priviledge Lua scripts, which interacted with the server API endpoints, and normal game logic. By poisoning the priviledge context's metatatable with my own functions I was able to capture the server endpoint URLs and accesskey, along with things like send arbitrary HTTP requests to their CDN from the server to steal any place file. This easily could have been arbitrary RCE on their servers instead: the crafted bytecode chunk attack for getting a r/w primitive from for loops was published a little bit before this attack happened, IIRC, but there was a lot of less CTF style writeups about Lua internals and I was, like, 15 and an idiot.

In practice it's really hard to sandbox arbitrary user controlled scripts. Even after this I found a half dozen other bugs in their Lua<->C++ bindings that you could leverage into server code execution. V8 and other browser engines still having JIT bugs and DOM manipulation exploits every other week should terrify any developer who thinks "oh I'll let my users do a bit of scripting".

Also, be very careful with sandboxing. It can be deceptively easy to break out of.

Back in 2018 when Surviving Mars came out with mod support (no sandbox of any kind), someone asked about os.execute on Reddit I think?

I whipped up a mod showing a couple other fun things you could do. I must've panicked someone because they pushed a sandbox update out pretty quickly, then sent me a polite pm asking me to let them quietly know about issues before hand.

About a week later I sent the dev an email with 5-6 different ways to get access to _G

That's what they do. They add additional layers on top of the sandbox like jailing. You can jail your sandboxes, but it's not so easy to make that a multi-platform solution for gaming. I think for games I would just stick to a fast interpreter and apply some generally appropriate measures to discourage timing attacks.

I guess it goes to show how many people are innately good or benign (not sure what word to use in English). The newsmedia would have you believe otherwise, and the average comments on such news perpetuate the belief, but if this were all true, how can we have all the luxuries we do? The healthcare and social aid programmes? It's not like there are no problems in the world, but clearly far more people are constructive than destructive

Perhaps this is too tangential but I guess it's front of mind since I just came from the HN thread about Panama papers where people were acting all unsurprised and like every wealthy person anywhere was evil and now got fully acquitted from any prosecution when neither is actually the case, as then a handful of comments pointed out (imo successfully, but you do have to read down the thread and not get caught up in cynicism)

People are doing it. The results have yet to be promising.

The remediation was:

1.1.104: https://github.com/Rseding91/Factorio-Lua/commit/4d924b69808...

and

1.1.107: https://github.com/Rseding91/Factorio-Lua/commit/ce12474c7fc...

The most relevant part was the change to luaB_load in 1.1.104 which simply disabled bytecode loading.