Nix is a better Docker image builder than Docker's image builder

I've gone down the same path. I love deterministic builds, and I think Docker's biggest fault is that to the average developer a Dockerfile _looks_ deterministic - and it even is for a while (build a container twice in a row on the same machine => same output), but then packages get updated in the package manager, base images get updated w/ the same tag, and when you rebuild a month later you get something completely different. Do that times 40 (the number of containers my team manages) and now fixing containers is a significant part of your job.

So in theory Nix would be perfect. But it's not, because it's so different. Get a tool from a vendor => won't work on Nix. Get an error => impossible to quickly find a solution on the web.

Anyway, out of that frustration I've funded https://www.stablebuild.com. Deterministic builds w/ Docker, but with containers built on Ubuntu, Debian or Alpine. Currently consists of an immutable Docker Hub pull-through cache, full daily copies of the Ubuntu/Debian/Alpine package registries, full daily copies of most popular PPAs, daily copies of the PyPI index (we do a lot of ML), and arbitrary immutable file/URL cache.

So far it's been the best of both worlds in my day job: easy to write, easy to debug, wide software compatibility, and we have seen 0 issues due to non-determinism in containers that we moved over to StableBuild in my day job.

>Documentation is just plain AWFUL (as in: complete and technically accurate, but maddeningly obtuse)

Documentation is often just plain erroneous, especially for the new CLI and flakes, not even edge cases. I remember spending some time trying to understand why nix develop doesn't work like described and how to make it work like it should. I feel like nobody ever actually used it for its intended purpose. Turns out that by default it doesn't just drop you into the build-time environment like the docs claim (hermetically sealed with stdenv scripts available), it's not sealed by default and the commandline options have confusing naming, you need to fish out the knowledge from the sources to make it work. Plenty of little things like this.

>In theory, Nix is idempotent and deterministic

I surely wish they talked more about edge cases that break reproducibility. Things like floating point code being sensitive to the order of operations with state potentially leaking from OS preemption, and all that. Which might be obvious, but not saying obvious things explicitly is how you get people shoot themselves in the foot.

It’s really not that bad. However, with a standard NixOS setup, you still have a tremendous amount of non-reproducible state, both inside user accounts and in the system. I’m running a “Erase your darlings” setup, it mostly gets rid of non-reproducible state outside my user account. It’s a bit of a pain, but then what isn’t on NixOS.

https://grahamc.com/blog/erase-your-darlings/

Inside my user account, I don’t bother. I don’t like Home Manager.

The strength of Docker is the chaos itself.

That depends whether you are okay with chaos.

It appears that you are, so it is suitable tool for you. Choose the right tool for the right job.

---

Docker is a poor choice for people who are interested in deterministic/reproducible builds.

I recently faced a similar hurdle with Nix, particularly when trying to run a .NET 8 AOT application. What initially seemed like it would be a simple setup spiraled into a plethora of issues, ultimately forcing me to back down. I found myself having to abandon the AOT method in favor of a more straightforward solution. To give credit where it's due, .NET AOT is relatively new and, as far as I know, still has several kinks that need ironing out. Nonetheless, I agree that, at least based on my experience, you need a solid understanding of the ins and outs before you can be reasonably productive using Nix.

Give a try to Fedora Atomic (immutable). At this point I have pretty much played around and used every distro package maneger there is and I have broken all of them in one way or another even without doing something exotic (pacman I am looking at you). My Fedora Kinoite is still going strong even with adding/removing different layers, daily updates, and a rebase from Silverblue. Imho rpm-ostree will obsolete Nix.

I use both Docker and NixOs at work. I've never had any of the problems you seemed to have above. Docker is fine, performance wise it's not great on Macs. I love nix because it's trivial to get something to install and behave the same across different machines.

Nix Doc are horrible but I've found that ChatGPT4 is awesome at troubleshooting Nix issues.

I feel like 90% of the time I run into Nix issues, it's because I decided to do something "Not the Nix way."

I'm just here to give you points for the Discworld reference.

It has a bit of a learning curve that is worth it - it's an incredible tool.

Documentation is just plain AWFUL (as in: complete and technically accurate, but maddeningly obtuse)

That has been the case for as long as I can remember. I gave up on Nix about 5 years ago because of it, and apparently not much has changed on that front since then..

Maybe it won't be your cup of tea given your reference to Emacs, but there's guix if you want to try a saner alternative to nix.

Yes at this point I hope someone builds a friendlier version on top of Nix, so we can cleanly migrate completely away from it.

Just out of curiosity. What were you trying to do that didn't work?

You complain about the documentation, and the first thing I wonder is if you’ve tried using one of the prominent chatbots like chatgpt or claude to help fill in the gaps of said documentation? Maybe an obvious thing to do around here, but I’ve found they help fill in documentation gaps really well. At the same time Nix is so niche there might not have been enough information out there to feed into even chatgpt’s model…

That's kind of an unfair comparison. The flake you linked:

1. Defines a Go binary (i.e., how to build it)

2. Defines a Docker image that uses said Go binary as an entry point

3. Defines a NixOS module that creates a systemd service that runs the Go binary (only relevant on NixOS)

4. Defines a NixOS test for the module that ensures that the NixOS module actually creates a systemd service that runs the Go binary as expected. The NixOS test framework is actually quite impressive - tests run in a QEMU VM that also runs NixOS :)

Note that only (1) and (2) are relevant to the linked article (+ some of the surrounding boilerplate).

Those 120 lines are not exactly representative. For example if I was writing this for a single service, I wouldn't bother making a new module and would inline it instead. Then, you've got many lines which would map 1:1 to an inlined systemd service description, so you're not getting rid of those whatever the system you choose. There's also a fancy way to declare multiple systems with an override.

This example is a "let's do a trivial thing the way you'd do a big serious thing". If you wanted to treat it as a one-off, I'm sure you could cut it down to 40 lines or so.

its 120 lines of code to deal with a binary and its systemd setup.

That binary + config file are effectively as close as we're going to get to a "flat pack" on linux.

Im not sure what is the forest and where are the trees but this example shows exactly what we have lost sight of.

These 120 lines do quite a lot more, don't they?

Aren’t Nix builds actually deterministic in that they’ll build the same each time? Docker doesn’t have that, you’re just using prebuilt images everywhere. Determinism has a computer science definition, it’s not “build once run anywhere,” it’s more like “builds the exact same binary each time.”

Docker builds are not deterministic, I don't get where you get that idea. I can't count the hours lost because the last guy who left one year ago built the image using duck tape and sed commands everywhere. The image is set in stone, but so is a zip file, there's nothing special here.

Building an image using nix solves many problems regarding not only reproducible environments that can be tested outside a container but also fully horizontal dependency management where each dependency gets a layer that's not stacked on one another like a typical apt/npm/cargo/pip command. And I don't have to reverse engineer the world just to see what files changed in the filesystem since everything has its place and has a systematic BOM.

I still don't understand what it does better than Docker

It doesn't break as you scale. If you don't need that, then keep using Docker. (Personally, for me "scale" starts at "3 PC's in the home", so I eventually switched all of them to NixOS. I don't have time to babysit these computers.)

Docker build are deterministic and easily reproductible

No, they definitely aren't. You don't really want to go down this rabbit hole, because at the end you realize Nix is still the simplest and most mature solution.

That's the interesting bit about Dockerfiles. They look _looks_ deterministic, and they even are for a while while you're looking at it as a developer. I've done a detailed writeup of how it's not deterministic in https://docs.stablebuild.com/why-stablebuild

Most Docker builds are not remotely deterministic or reproducible, as most of them pull in floating versions of their dependencies. This means that the same Dockerfile is likely to produce different results today than it did yesterday.

I haven't used java in over a decade so won't be able to help much with that, but for example I was able to get my application to fit in just 70MB container including python and all dependencies + busybox and tini

It looked something like this: https://gist.github.com/takeda/17b6b645ad4758d5aaf472b84447b...

So what I did was:

- link everything with musl

- compile python and disable all packages that I didn't use in my application

- trim boto3/botocore, to remove all stuff I did not use, that sucker on it's own is over 100MB

The thing is what you need to understand is that the packages are primarily targeting the NixOS operating system, where in normal situation you have plenty of disk space, and you rather want all features to be available (because why not?). So you end up with bunch of dependencies, that you don't need. Alpine image for example was designed to be for docker, so the goal with all packages is to disable extra bells and whistles.

This is why your result is bigger.

To build a small image you will need to use override and disable all that unnecessary shit. Look at zulu for example:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen...

you add alsa, fontconfig (probably comes with entire X11), freetype, xorg (oh, nvm fontconfig, it's added explicitly), cups, gtk, cairo and ffmpeg)

Notice how your friend carefully extracts and places only needed files in the container, while you just bundle the entire zulu package with all of its dependencies in your project.

Edit: tadfisher seems to be more familiar with it than me, so I would start with that advice and modify code so it only includes a single jdk. Then things that I mentioned could cut the size of jdk further.

Edit2: noticed another comment from tadfisher about openjdk_headless, so things might be even simpler than I thought.

Oh, and openjdk_headless skips the GTK and X dependencies that you won't need for Spring.

That's because you're including two JDKs, zulu and the one that gradle includes via its jdk argument. Look for gradleGen in nixpkgs to see what I mean.

And sorry for gradle2nix, I'm working on an improvement that's less of a hack.

While I was using Nix, the resulting image was twice the size of the image built without Nix.

I would be very interested to know where the difference is; is nix including things it doesn't need to? Is the non-nix build not including things it should?

I decided to participate in your challenge and cleaned up your Nix code a little bit. It seems like the main task of the challenge is building a really small JRE.

I've switched to using a headless OpenJDK build from Nixpkgs as a baseline instead of Zulu, to remove all the unnecessary dependencies on GUI libraries. Then I've used pkgs.jre_minimal to produce a custom minimal JRE with jlink.

The image size now comes out to 161MB, which is slightly larger than the demo_jlink image. This is because it actually includes all the modules required to run the application, resulting in a ~90MB JRE. The jdeps invocation in Dockerfile_jlink fails to detect all the modules, so that JRE is only built with java.base. Building my minimal JRE with only java.base brings the JRE size down to about 50MB, the resulting (broken) container image is 117MB according to Podman.

I've also removed the erroneous copyToRoot from your call to dockerTools.buildImage, which resulted in copying the app into the image a second time while the use of string context in config.Cmd would have already sufficed.

I've also switched to dockerTools.buildLayeredImage, which puts each individual store path into its own image layer, which is great for space scalability due to dependency sharing between multiple container images, but won't have an impact for this single-image experiment.

This is mostly a JRE size optimization challenge. The full list of dependencies and their respective size is as follows:

  /nix/store/v27dxnsw0cb7f4l1i3s44knc7y9sw688-zlib-1.3                            125.6K
  /nix/store/j6n6ky7pidajcc3aaisd5qpni1w1rmya-xgcc-12.3.0-libgcc                  139.1K
  /nix/store/l0ydz31lwa97zickpsxj2vmprcigh1m4-gcc-12.3.0-libgcc                   139.1K
  /nix/store/a3n1vq6fxkpk5jv4wmqa1kpd3jzqhml9-libidn2-2.3.4                       350.4K
  /nix/store/s5ka5vdlp4izan3nfny194yzqw3y4d1z-lcms2-2.15                          445.3K
  /nix/store/a5l3w6hiprvsz7c46jv938iij41v57k6-libjpeg-turbo-2.1.5.1                 1.6M
  /nix/store/r9h133c9m8f6jnlsqzwf89zg9w0w78s8-bash-5.2-p15                          1.6M
  /nix/store/3dfyf6lyg6rvlslvik5116pnjbv57sn0-libunistring-1.1                      1.8M
  /nix/store/a3zlvnswi1p8cg7i9w4lpnvaankc7dxx-gcc-12.3.0-lib                        7.5M
  /nix/store/657b81mfpbdz09m4sk4r9i1c86pm0i8f-app-1.0.0                            19.0M
  /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44                        28.8M
  /nix/store/b1fhkmscb0vff63xl8ypp4nsc7sd96np-openjdk-headless-minimal-jre-21+35   91.4M

https://github.com/max-privatevoid/hackernews-docker-challen...

Don't you just stick the JAR in?

Hard to beat jib (https://github.com/GoogleContainerTools/jib/tree/master/jib-...) for minimal Java OCI containers.

I use Orbstack and it works flawlessly to do this. Really good tool. I use Docker to cross-compile for {aarch64,amd64} x {linux,darwin} since not all the cross-compiling is super robust across our stacks (I'm using a specific glibc for one Linux part, etc.). Just a bunch of docker on my Darwin aarch64 and it compiles everything. Good experience.

The big problem is how docker was designed. It is essentially a jail that is supposed to contain a Linux binary.

Things are straight forward on Linux. You build your binary, place in a docker container and you are done. The nix code will also be straight forward. If you can build your code, then creating a container is just one more operation away.

Unfortunately docker requires Linux binary and you are on Mac. So the docker desktop actually runs a Linux VM and performs all operations on it, abstracting this away from you.

Nix doesn't do that and you have two options:

1. Do cross compilation, the problem is that for this to work you need to be able to cross compile down to glibc, the problem is that while this will work for most community used dependencies you might get some package where the author didn't put effort making sure it cross compile. To make things worse the Hydra that populates standard caches that nix uses, doesn't do cross compile builds, so you will run into lengthy processes that might potentially end with a failure.

2. You can have a Linux builder, that you add to your Mac and configure to send build jobs for x86_64-linux to that builder. Now you could have a physical box, create a VM, or even have a NixOS docker container (after all docker ion Mac runs inside of the VM).

The #1 seems like the proper way, while #2 is more of a practical way.

I think you are running into issues, because you're likely trying #1, and that requires a lot of experience not only with Nix, but also with cross compiling. I wish Nix's Hydra would also build Darwin to Linux cross compilation as that would not only provide caches, but also help making sure the cross compilation doesn't break, but that would also increase costs for them.

I think you should try the #2 solution.

Edit: looks like there might have been an official solution to this problem: https://ryantm.github.io/nixpkgs/builders/special/darwin-bui... I haven't used it yet.

In Docker, the dark corners have dust. In Nix, the dark corners have a grue.

The way I've set this up for our macos devs at work was a script that runs nix builds inside docker-for-desktop using the official nixos upstream docker image (and some tricks to get ssh forwarding, filesystem mounts, ...) working. Works quite alright. Benefit is you don't need some weird Linux remote builder vm with ssh running.

https://github.com/gytis-ivaskevicius/high-quality-nix-conte...

This sort of twenty-minute adventure?

macOS is a definitely rougher. I use colima there, and it does alright. There are one or two bugs with it, but I think those are primarily around volumes. But it does alright with building Docker images.

The rougher part is the speed of it; it's a one-two punch between the hardware & the fact that Docker has to emulate a Linux VM.

I would rephrase this as:

The Dockerfile format imposes a hierarchical relationship between layers. This quickly becomes very annoying, since dependencies usually form dependency graphs, not dependency trees.

Alternative tools, like nix (probably bazel too), are not bound in the same way. They can achieve fine grained caching by mapping their dependency graph to docker layers, which is something that can not be expressed with a Dockerfile.

In Docker, if the layers are cached, the layer with apt-get won't be automatically invalidated (unless --no-cache or any changes to the upper layers).

It is very easy to overload package versions locally for your needs, and it's quite easy to push that upstream to Guix so that others may benefit as well :)

This is my one gripe with Guix. Alas, we (apparently) can't have it all!

What a weird thing to complain about.

Just override the font if you dislike it that much. That's the great thing about the interwebs, it's all just text, you can format it however you want.

Assuming you're using something based off chrome: https://chromewebstore.google.com/detail/font-changer/obgkji...

Or Firefox: https://addons.mozilla.org/en-US/firefox/addon/refont/

The font is a custom build of Iosevka, which is almost certainly inspired by the commercial font Pragmata Pro (https://fsd.it/shop/fonts/pragmatapro/). When Pragmata Pro was first released a little over 10 years ago, it sold for around $400 (I know this because I and many, many others bought a copy back then).

As another commenter points out, you may have some rendering issue. Alternatively, you may just not like the font. Can't please everyone.

Where on Earth are you getting that result from? [1] gives an 11 MB image.

[1] https://nix.dev/tutorials/nixos/building-and-running-docker-...

There are ways to properly build a nix container image so this kind of things doesn't happen. You'll find plenty of projects on GitHub dedicated to only that.

That's pretty much how building on nix works, except you don't need base image or your application file, you specify what command to run from which package and it will be placed in the container with all runtime dependencies automatically.

Of course you can customize the container further if needed.

As many bazel rules, rules_oci's predecessor (rules_docker) was an unmaintained spagetti of hell, now we are pushed to rules_oci, and it's rpmtree recommended way of installing rpms which then in turn don't support post install script...

All this bazel-is-our-savior complex burns down when we want to build a tiny bit complicated thing with it. And we unfortunately do try to do that (our devbox image), which with full caching takes long minutes to an hour, and it's a freaking thousands pine mess instead of using a lined dockerfioe with pinned versions.

I absolutely hate bazel and its broken unmaintained Google-abandoned rulesets and I wish we either used either Docker or Buck2 for everything.

Author of nix-snapshotter here.

Yes, one of the main downsides of Docker images using Nix is the 128 layer limit. It means we have to use a heuristic to combine packages into the same layer and losing Nix’s package granularity. When building containers with Nix packages already on a Nix binary cache you also have to transform the Nix packages into layer tarballs effectively doubling the storage requirements.

Nix-snapshotter brings native understanding of Nix packages to the container ecosystem so the runtime prepares the container root filesystem directly from the Nix store. This means docker pull == Nix substitution and also at Nix package granularity. It goes a bit further with Kubernetes integration that you can read about in the repo.

Let me know if you have any other questions!

I haven't tried it yet as I need to produce containers that can work on public cloud k8s, but it definitely looks like the way to go. All the existing methods for grouping store paths into layers are finnicky, brittle, and non-optimal.

to build multi-arch images, it actually wants to execute stuff as the other architecture

You should be able to cross compile binaries for other architectures without actually running them. As long as the package's build files support it of course.

and only supports using qemu with hardware virtualization for that

That doesn't sound right. You can use qemu for architectures that be only software emulated too.

The minimal example is discussed here:

https://discourse.nixos.org/t/how-do-i-get-a-shell-nix-with-...

I don't want to say it should be as simple as using pkgCross (https://nix.dev/tutorials/cross-compilation.html), but... are some specific issues with the usual process that you're running into?

What did your final Nix and Docker file look like, and did you have to use `buildFHSEnv` at all to support the odd 3rd party binaries?

I think Nix really needs some articles outlining how to play well and smoothly transition from an existing system piece by piece.

  >  I spent a half a day or so relatively recently trying to build our CI base image with Nix (at the recommendation of our infra team), but it was huge, and some stuff didn't work because of linking issues.

[1] https://hub.docker.com/r/nixos/nix/tags

Hmm? Cross compiling to docker images is exactly what I used nix for. I even had musl being used, it was the smallest image I could build with any tool and built the images quickly and consistently in ci with caching working well.

I never saw went being used so im a bit confused where that came into play for you

The text is written to be spoken, it works better when I present it. I'll have the video edited next week, that may flow better for you.

I don't like Ubuntu or Debian as a base image for docker, but it's typically my go-to if I need glibc stuff or browser emulation.

Is there a better alternative like Alpine but with glibc that isn't Debian?

So one of the pillars of the article is that docker builds aren't reproducible, but Nix is.

But... is a lot of that irreproducibiity (apologies for that word) because there's no guarantee one of the docker layers will be available?

And... does Nix have some guarantee to the end of the universe that package versions will stay in the repository?

[1] also uses this approach.

[1] https://mitchellh.com/writing/nix-with-dockerfiles

I believe for a developer tool to success, for the most common thing to do there has to be at least three ways an engineer may misuse your tool and still get it "done" (by leaving non-obvious tech debts behind).

This is true for git, but not so true yet for Nix, so I'm not sure a GitHub-like moment helps.

Ron from flox.dev here, the note brought a lot of smiles across the team. We've been working on this for a while now and would love to hear if there is anything we can prioritize or do to keep making it better.

Nixpkgs' streamDockerImage does something similar, instead of storing a multi-layer tarball it produces a script that cats them all together on demand, ready to feed into `docker load` or whatever.

What is automatic about docker? Do you mean other people have already put in the work? Or do you mean that it's more trivial to pip/npm/cargo install stuff?

what is the base image for the generated image?

Default is none (i.e. like "FROM scratch" in a Dockerfile); you can specify a baseImage if needed, but I haven't had to yet. It works by copying parts of the nix store into the image as needed, but see also below.

wouldn’t the image size be large if the glibc is copied over again

The original Nix docker-tools buildImage did suffer from poor reuse of common dependencies. Docker already has a way to reuse parts of images (e.g. if you build 7 images where the first N lines of a Dockerfile are the same, the 7 images will use a shared store for the results of running the first N lines). There are several backends for Docker storage that accomplish this in various ways (e.g. FS overlays, tricks with ZFS/btrfs snapshots).

Nix docker-tools now has a "buildLayeredImage" that uses this ability of Docker to share much of the storage for the dependencies, so if you build several images that all rely on glibc, you only pay the cost of storing glibc in docker once.

Since there are a plethora of dagger projects, lazyweb: https://github.com/dagger/dagger#readme

They also recently released their "github actions" replacement <https://news.ycombinator.com/item?id=39550431> but holy hell their documentation is just aggressively bad

That's very interesting, because as someone familiar with nix my take was that this was information I considered to be aimed at people who weren't familiar.