Exploring Podman: A More Secure Docker Alternative

Podman was good when it supported systemd unit files, so I could auto start and auto update containers, even entire pods with systemd.

Then they removed that in favor of Quadlet. Now in order to do a single container I can do a unit file, but for a pod, I need to use a Kubernetes cluster definition.

Plus, unlike Docker their containers bow to SELinux definitions, so I have repeatedly struggled with containers unable to access mapped directories.

So what is it, Podman? Should I just use Kubernetes? Should I just make dedicated directories for everything instead of mapping logical places for things?

I already was defining my infrastructure with docker-compose.yml files, and found out that podman-compose has a poorly documented feature that generates systemd units. It doesn't use the now-deprecated podman feature, it writes the unit files itself, and I find the process much smoother than the podman feature anyway.

To enable the feature:

    $ podman-compose systemd -a create-unit

To register a systemd unit:

    $ podman-compose systemd -a register
    $ systemctl --user enable --now "podman-compose@$PROJECT_NAME"

Updating looks like this:

    $ podman-compose pull
    $ systemctl --user restart "podman-compose@$PROJECT_NAME"

($PROJECT_NAME is usually the directory name.)

Source code for the feature is here if you care to inspect it:

https://github.com/containers/podman-compose/blob/f6dbce3618...

Admittedly, I'm still on podman 4.3.1, but I don't see any reason why this would stop working in later versions of podman.

I thought docker compose was for local dev only and not meant to be used for production workloads?

I'm using podman-compose for my homelab, which is obviously fine.

But even for small-scale single-node production use cases, I suspect that podman-compose with systemd doesn't have the same concerns as docker-compose does. Since you're registering the workload with systemd, it'll restart with the node as easily as any other service, and rootless containers are a big win for security.

Where you can't keep using (podman|docker)-compose is when you have to scale up a service beyond a single node.

You can keep using x-compose on several nodes, you just need e.g. ansible or salt on top of it. For many things this is still a local maximum compared to a K8s cluster or "just ssh in'.

Why not docker swarm then?

For a lot of the services I'm thinking of for this case the scheduling may be across multiple nodes but it may not be flexible - e.g. this is my preferred way to run things which need guaranteed local iops. So Swarm maybe helps with service discovery etc. but its main purpose is lost.

I have a lot of complaints about Docker Swarm but they're either about its ownership issues or relatively minor (but - lots of minor) issues, if you want to use it it's fine. But you do still need an orchestration layer above it anyway.

It's great in prod. Doesn't make $ for companies marketing k8s iaac devops, so no one to advertise.

Why?

Its documented. If you just type `podman-compose` in the command line you get...

    usage: podman-compose [-h] [-v] [--in-pod in_pod] [--pod-args pod_args]
                          [--env-file env_file] [-f file] [-p PROJECT_NAME]
                          [--podman-path PODMAN_PATH] [--podman-args args]
                          [--podman-pull-args args] [--podman-push-args args]
                          [--podman-build-args args] [--podman-inspect-args args]
                          [--podman-run-args args] [--podman-start-args args]
                          [--podman-stop-args args] [--podman-rm-args args]
                          [--podman-volume-args args] [--no-ansi] [--no-cleanup]
                          [--dry-run]
                          {help,version,wait,systemd,pull,push,build,up,down,ps,run,exec,start,stop,restart,logs,config,port,pause,unpause,kill}
                          ...
    
    command:
      {help,version,wait,systemd,pull,push,build,up,down,ps,run,exec,start,stop,restart,logs,config,port,pause,unpause,kill}
     ...
        systemd             create systemd unit file and register its compose stacks
                            
                                When first installed type `sudo podman-compose systemd -a create-unit`
                                later you can add a compose stack by running `podman-compose systemd -a register`
                                then you can start/stop your stack with `systemctl --user start podman-compose@<PROJ>`

That’s… not documentation. That’s a CLI helpfile. It’s better than nothing but also what is completely broken with the “move fast and break things” mindset.

Sure looks like documentation to me, but I don't need pretty webpages. 'man' is more than sufficient for tools like these.

How would I ever read this documentation without installing postman? Am I suppose to download and install every tool just to read about how it works?

it's documentation enough to not call it an undocumented feature.

the concern with truly undocumented functionality is that it's not included intentionally: either it's a bug or an experimental feature that could be removed or changed with no notice. a poorly-documented feature, on the other hand, will probably at least get a deprecation notice before it disappears

it's documentation enough to not call it an undocumented feature.

Which is why I didn't call it that, I called it poorly documented.

Yeah, I eventually found that, but only after I finally stumbled upon someone referencing the feature in a GitHub issue.

The --help is fine documentation for the people who have already installed the tool, but it doesn't help people like OP who just want a simple way to run multiple containers as a systemd unit and don't yet know that podman-compose has a solution.

That's why I said "poorly documented" instead of "undocumented". It's there once you know where to look.

That's much more elegant than my hacky hand-written systemd unit (one-shot, remain after exit) to start podman containers.

unlike Docker their containers bow to SELinux definitions,

That's a bug in docker. If your system isn't configured for SELinux, disable it.

Also the systemd files generated by podman-generate-systemd are just executing "podman start containername", you can write them on your own easily but (unlike e.g. docker-composr) the containers are black boxes pretty much.

The advantage of quadlet is that the definition of the container is declared in the .container file; before I used to write the podman run command line manually in a handwritten systemd unit, and quadlet is a big improvement in that respect and can be an alternative to docker-compose (with advantages and disadvantages).

That's a bug in docker. If your system isn't configured for SELinux, disable it.

I feel obligated to say that you should set it to permissive mode, never disable SElinux.

    sudo setenforce 0

I know it's a hard sell, but the 30 minutes to an hour to understand filesystem contexts and how to copy them from a good place isn't a lot

It's likely to little gain. I know why people don't, but it is very accessible to those open to it

The more one adheres to the FHS, the easier SELinux is

I have been doing Linux sysadmin for 20 years and I just stopped trying to understand SELinux. It looks and feels like an abomination borne out of some IBM or other antediluvian corporate UNIX system for programmers wearing suit and tie.

Terrible documentation, terrible mental model, terrible CLI UX, terrible error messages.

I run Fedora and SELinux is working well enough, but it's a piece of machinery I can't wait to see replaced, however useful people swear it is.

It was originaly developed by the NSA, so it's even worse than corpos wearing suit and tie, it came from spooks.

Probably intentionally complex for just that reason. Easy to leave a misconfiguration.

I totally get it. I exert next to zero effort to keep it around for my systems!

For systems that shouldn't do much more than exactly what is prescribed, it's acceptable, is what I'm after... I guess.

I can't do justice to the source, but there's a concept about our programs/creations reflecting us.

Like a peer hints - SELinux reflects an agency like the NSA, draconian. Good and bad

understand filesystem contexts and how to copy them from a good place

I’m not really sure what this means (I have 0 knowledge on selinux)

edit: Apologies for the wall. I think I finally landed on a decent mix after many edits. I'm finished now, lol.

SELinux has a bit of a well deserved reputation... but I, a fairly silly person, have managed to work with it

This video likely explains things far better than I can in this post:

https://www.youtube.com/watch?v=_WOKRaM-HI4

I'll probably fail with specifics, where they certainly do a better job.

So. First it's important to know SELinux runs in one of two modes:

    * A targeted mode where well-known/accounted-for things are protected. For example, nginx
    * A more draconian mode where *everything* is protected

People often present the first [default] mode as if it were the second.

The protection is based on policies that say 'things with this label/at this path are allowed to do XYZ'.

It's very focused on filesystem paths and what relevant applications try to do.

It's entirely manageable, but admittedly, complicated. Without practicing the words I can't express them.

Most people having trouble with SELinux are defying some convention. For example: placing application scratch data in '/etc'.

Policy management is a complicated topic.

The policy can be amended in cases where the standard doesn't apply; I won't cast judgement - sometimes it's a good idea, sometimes not.

Another way to handle this is to copy the label from one path and apply it to the one your application requires/customizes. This is less durable than leaning on the policy.

It acts as a sort of central DB... the goal is to make things such that the policy stores all of the contexts so the files/dirs can have "labels" applied for SELinux

Very enlightening, I really appreciate the time you took to outline all of this, thank you!

Some secondary reading, I've referred to FHS at times. It's the 'Filesystem Hierarchy Standard':

https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html

They provide guidance on how a given filesystem path should be used.

This has informed the default SELinux policies greatly; familiarity turns hassle into informed assumptions/ease.

Only if you want it back in enforcing mode after the next reboot. If you want to make the change permanent, you need to set the following in /etc/selinux/config as well:

SELINUX=permissive

I meant disable labeling in podman. :)

You should set it to permissive mode if you think you will ever need to re-enable that trainwreck.

Plus, unlike Docker their containers bow to SELinux definitions, so I have repeatedly struggled with containers unable to access mapped directories.

Add the following to containers.conf if you don't want to deal with it:

   [containers]
   label=false

If you don't like podman's default security level, there is usually a way to turn things off.

The only times I had issues with SELinux and podman, it was because I forgot to add the z flag to the volume:

    podman run -v .:/app:z image

This only happens locally as files in your home have strict security rules, never had any issue on a CentOS server.

Note there's also uppercase Z, for when the volume shouldn't be shared with other containers: https://unix.stackexchange.com/questions/651198/podman-volum...

I recently migrated over to NixOS which treats systemd as the source of truth for everything, including containers. I found this model extremely intuitive, but it was difficult to apply this to Docker Compose without a lot of manual migration. So I ended up writing a tool that handles this for you — it converts your Compose files into a NixOS config that can be interpreted and managed natively.

https://github.com/aksiksi/compose2nix

This is a cool idea. Two questions:

Arion can wrap docker-compose and run as a project or part of a nixos config. Did you come across Arion before creating this, and have you compared them?

I had a brief look through your examples and it doesn’t look like compose2nix implements docker-compose’s network per compose file. Is this something you want to add?

So, from what I understand, arion provides a Nix frontend for Docker Compose. This allows you to write Nix that runs via Docker Compose. It doesn’t solve the migration problem: if you have an existing Docker Compose project, you still need to manually convert it into Nix for arion to consume.

My tool does the opposite: it takes a Compose file and converts it into OCI containers in Nix. The idea is that your Compose file is the source of truth, and you simply generate Nix to run on NixOS. One benefit here is that you can easily migrate an existing Compose project into native Docker/Podman containers running on NixOS. This removes Docker Compose from the equation entirely - essentially a “reimplementation” of Compose.

It should support the default network per Compose project. See: https://github.com/aksiksi/compose2nix/blob/6dc451fd960f7a9b.... If you’re talking about something else, please feel free to open an issue and I can take a look.

So what is it, Podman? Should I just use Kubernetes?

If you're talking about a production system for any business larger than a 10 person tech startup; yeah, probably. Alternatively there's Docker Swarm and Hashicorp Nomad. Though Swarm is not nearly as flexible, it's just easy to use. And Nomad... well, let's just say I've been paying closer attention to Hashicorp's build processes in their open source repos like Packer and Vault as of late and they do some stuff that seems shady to me so use at your own risk.

AFAIK podman either already supports pods in quadlet container files, or will in the near future. https://github.com/containers/podman/pull/20762

This. Quadlet wasn't really needed and just complicated matters for me, so I went back to docker-compose.

You can generate Kubernetes yaml files with podman kube generate. Then you can use systemd to run the generated files: https://www.redhat.com/sysadmin/kubernetes-workloads-podman-...

I’ve been complaining about our OPs people pushing Colima over podman and now you’re making me think my podman info is outdated. Ouch.

That was a cool feature (I didn't realize it was gone, that's unfortunate), although I felt the generated code wasn't super great, and if the container is stateless (excepting what's stored in volumes of course) then it's so simple to write your own systemd unit file that I just do that now. I wrote it once and pretty much just copy/paste it when needed, changing the podman run command for image names, port numbers, volumes, etc. For example, here's what I use for Jellyfin. Just drop at `/etc/systemd/system/jellyfin.service`:

    [Unit]
    Description=Jellfyin
    After=NetworkManager-wait-online.service
    
    [Service]
    Restart=always
    ExecStart=/usr/bin/podman run --rm --volume '/srv/jellyfin/config:/config:Z' --volume '/srv/jellyfin/cache:/cache:Z' --volume '/media/media:/media:Z' --publish '8096:8096' --publish '8920:8920' --publish '1901:1900/udp' --publish '7359:7359/udp' --name 'jellyfin' 'docker.io/jellyfin/jellyfin:latest'    
    ExecStop=/usr/bin/podman stop -t 5 jellyfin ; /usr/bin/podman rm -f jellyfin
    
    [Install]
    WantedBy=local.target

Note: You can also just `s/podman/docker/g` and reuse the same service file with docker, which is really convenient for systems where you have no choice

Docker does use selinux policies.

Mapping directories from the host requires that you change selinux labels on those files so that the container process can access the files. That's just how selinux works.

Long time fan, yet - agreed.

I keep forgetting this transition happened until I try to 'podman generate systemd [...]'

This is rare because I wrote an Ansible role to do this in a way that feels nice.

Anyway, it really feels like podman lost the mark. I've already subscribed to the unit file maintenance/relationship planning thing. Just let me use the generator. I don't care about Quadlets or how they might be better.

`podman generate systemd` is still there, and I see no reason you couldn't use it. it's just a bunch of podman commands wrapped in a unit file, no magic.

feels like a lot more cruft than quadlets to me though.

What are you talking about? Quadlet generates ordinary systemd service units with podman commands. Isn't that what you were using?

It's just a simplified layer ontop of those old systemd units.

Plus, unlike Docker their containers bow to SELinux definitions

Isn't that great? You want SELinux if you deploy containers on prod.

I have repeatedly struggled with containers unable to access mapped directories.

Use the `z` or `Z` flag, e.g. `podman run -v mydir:podman_dir:z nginx` to get SELinux labels updated automatically (and temporarily) by podman.

I still don't really understand why Red Hat invests into creating a Docker alternative, but I really like it. Podman does pretty much everything Docker does, but it has more features (e.g. pods) or the way Podman does it tends to be better (e.g. daemonless container spawning process).

The main issue to a common developer would be Docker compose I suppose, which if you use simple compose files, there's actually a podman-compose script that attempts to be compatible with the Docker compose spec.

There's also using Podman as a backend for docker-compose [1]. Overall, in 2024, I see no reason using Docker at least on Linux boxes. Not sure how Podman fares on macOS or Windows.

[1] https://www.redhat.com/sysadmin/podman-docker-compose

They invest in it because Docker is waaay out of spec with how things are done on Linux. Doing things that break the system, struggling with rootless for years, and who can miss the vendor lock-in? Podman is open and compliant and compliments k8s as well. It’s just nonsense the amount of effort developers have invested in Docker because it was first to market for easy containers.

Docker is every bit as open and not locked in as podman. Perhaps even more so as it's so widely used and doesn't require redhat specific projects around it. Are you confusing docker with docker hub?

No, I’m not confusing requiring a DockerHub account to even install Docker. Though that is an excellent example of it’s non-openness, thank you.

What? That's just completely wrong. Even for docker desktop (which is completely different from docker engine) you don't need an account

It appears they have since reverted the decision in 2020 but it used to require logging in for Docker Engine:

https://github.com/docker/docs/issues/6910

I did not know this as I stopped using Docker long ago.

Oh yeah to be clear, I absolutely agree that docker desktop as a whole is a mess especially since they keep introducing more ways to tie it up to docker hub etc. I wouldn't use it unless I'm on windows. So yes, avoid docker desktop but docker (the engine) itself is thankfully completely separate from docker desktop.

It’s really not if you’re talking about Docker Desktop or any of the commercial products: https://docs.docker.com/subscription/desktop-license/

Podman Desktop is completely open.

Well sure, then let's talk about podman desktop. I'm pretty sure the discussion was around docker itself, but maybe I got confused... Because otherwise you can just use other tools rather that docker desktop to manage docker containers (eg rancher desktop, which is also open source).

doesn't require redhat specific projects around it.

that is like saying oh no docker requires docker (company) specific projects around it

and as far as I can tell docker in recent years mainly cares about docker desktop and swarm which are less open then podman given their business model

Docker still sees a ton of development. I wouldn't be surprised if it sees more dev than podman.

And my point was more so that podman is obviously designed around the rhel ecosystem. I'm not saying it's closed! Just that even if we were to (wrongly) argue that one of the two is more "locked in", it's clearly podman. Docker is so much more widely used, ported, is basically as completely "non locked in" as it could be.

The only possible "lock in" is maybe the docker images namespace defaulting to docker hub but imo that's trivial and basically more of an early design choice that can't be reverted.

By all means, we can argue about technical differences but the often repeated argument about docker being less open than podman or whatever is just not true

docker isn't really struggeling with rootles, it works with rootless since a long time

they just never bothered making it the default or officially supporting it

which in context with how they acted about other security problems int the past tells a lot about how serious the docker company takes security on linux

I see no reason using Docker at least on Linux boxes. Not sure how Podman fares on macOS or Windows.

I hope the majority doesn't end up with your point of view. Docker is not RedHat/IBM. If Docker goes away RedHat gets to continue to push their corporate agenda with a heavier hand. There are some advantages to Podman, but there are also some things that have not been executed well in RedHat's mission to replace all things Docker. If you look at the history of how RedHat approached the container space they could have contributed, and improved, a lot of projects they ultimately wasted time and effort on. RedHat isn't a company that has an agenda of doing right by the customer and that's why you should consider what you hope for from a bigger picture perspective.

Red Hat had ample reason to develop Podman when they started the project and no reason now to abandon it.

Red Hat tried to work with Docker, but it didn’t go well. Docker also shot itself in the foot trying to push Swarm over Kubernetes and a bunch of other silliness until they had some serious management/leadership changes.

Podman, IIRC, is 100% FOSS. Not sure you can say the same for Docker. If Red Hat gets stupid with Podman, the rest the community can pick it up and carry on under a new name. Not true of Docker. If Docker gets bought by, say Broadcom tomorrow, the community can only fork the bits that are open.

they could have contributed, and improved, a lot of projects

they tried to work with docker, it didn't work out, that is why we have podman

and given the headache "securely" using docker is I'm really happy that they did

Honestly, I trust Red Hat a lot more than Docker.

After the first two or so sentences, you don't mention a single specific thing in your comment, so it's difficult to understand what you mean.

I use Podman on Mac OS. I've found the experience better than Docker for the most part, especially when it comes to supporting older versions of Mac OS. The only major downside of Podman on Mac OS is that you cant use the host network in a container, but people will rarely want to use the host network in a container anyway. This is something to keep in mind if you want to experiment with network things in a container and want to retain the same hostname and IP address of your host, however. I manage to work around this limitation anyway.

I'm a podman beginner, trying to install ollama-webui(1) using Podman on M2 MBA.

I started up Podman Desktop, and did a terminal command "docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v ollama-webui:/app/backend/data --name ollama-webui --restart always ghcr.io/ollama-webui/ollama-webui:main" based on Github's instructions, but it gave a error message something about "host".

the exact error message was ""Error": "failed to create new hosts file: unable to replace \"host-gateway\" of host entry \"host.docker.internal:host-gateway\": host containers internal IP address is empty""

Do you know what is the problem and how do I overcome this?

If I run the above command using Docker Desktop, it runs and installs Ollama-WebUI just fine.

Thank you.

(1) https://github.com/ollama-webui/ollama-webui ("Installing with Docker")

The only major downside of Podman on Mac OS is that you cant use the host network in a container

And this isn't even a Podman-specific issue, it's true of Docker Desktop as well [0]:

The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.

[0] https://docs.docker.com/network/drivers/host/

I still don't really understand why Red Hat invests into creating a Docker alternative, but I really like it.

They originally tried to work together with docker to fix various issues (e.g. systemd compatibility for certain use case) which had popped up in the past but this turned out not very fruit full.

If you combine this that docker had and still has a uncanny amount of security issues(1) and docker like containers/images being very widely used by developers they had very little choice then to create their own implementation which is more in line with the values and approaches of RHEL.

(1): Like docker still not defaulting to rootless even through it can. For a hardened context both the the docker user group and the ways to run it without the group are security wise a no-go in many use-cases. Like docker initially doing way to little to make sure non privileged containers are at least somewhat sandboxed and taking forever to fix it even when it became known it's an issue. Like the way it interacts with firewalls and networks rules. Like the way it interacts with SELinux. Etc. In companies with dedicated Linux system administrators which care about security docker being banned is not that rare.

I use Podman on Windows. Docker on Windows has been an endless stream of annoyances and frustration for me, starting from the fact everything points you to install an aggressively user-unfriendly GUI that starts up heavyweight processes on boot, nags you to sign up for a cloud account for some reason, and then nags you again that you shouldn't be using this so-called open source product for work anyway. Installing "just the command line" docker as an unsupported alternative is unnecessarily complicated, and last time I tried most WSL VMs didn't even support it out of the box anyway.

Enter Podman, you just winget install podman and it gets out of your way. When you need to run a container, you start up the Podman VM. When you don't, your system behaves exactly as it always did before. If you need to run something in a compose file, podman-compose is there. Maybe it can't handle esoteric configs, but it's worked for my use cases. The only thing Podman doesn't do is integrate with VS Code properly, but honestly the struggle of getting Docker to behave itself on Windows is far more annoying to me than missing some keyboard shortcuts in VS Code, so for me it's not a problem.

I would (and do) recommend Podman as the default solution for containers on Windows. The only reason I can see to use Docker is if your company is paying for it, or if you have a complicated compose setup, in which case it might be worth moving to Kubernetes anyway.

I almost never see what is IMHO the killer feature of Podman touted as a reason to prefer it over Docker: Docker mangles your network config. It is a nightmare trying to run Docker and KVM virtual machines with bridges at the same time. Podman on the other hand plays very nice OOTB.

I've also had a lot of VPNs break and/or be broken by Docker. I don't know much about the way podman does networking, but whatever it is they did a good job thinking it through and it has yet to interfere with anything else I do. I definitely can't say the same for Docker

It is a nightmare trying to run Docker and KVM virtual machines with bridges at the same time.

I'm doing it right now, no nightmares, just works, odd.

Are you bridging to the main eth interface on the host? (not just a KVM private bridge). Can you share your KVM configs?

Podman is free. Docker is also free, but a pain to install docker without docker desktop.

No pain at all.

Buildah (which may well work with Docker, but is a Podman peer) is the killer feature in my opinion. Dockerfiles are unadulterated shite. One of my pet peeves is "bored developers" writing DSLs/programming languages (especially in YAML, but that isn't the case here) when an off-the-shelf language would have done. Dockerfiles are a genuinely fantastic example of why this nonsense needs to end.

To see why, look at Buildah without `bud`. Instead of a silly DSL that becomes annoying the second your use-case veers an inch off the happy path, you can use Bash, or Fish, or whatever it is that you want.

These types of bad decisions carry the into the rest of the Docker ecosystem. DCS and it's (perpetually incomplete) replacement are yet more examples: instead of using established signing protocols (like Cosign does) they had the desire to build an obnoxiously complex and hard to automate (especially key rotation) system.

I had some issues with podman working on my m1 mac about 1-2 yrs ago.

I’ll give it a shot again. Looks like it has matured very fast.

I've had issues as recently as this week with podman on an m1 laptop - the container would start but after a short time would freeze completely and couldn't even be killed - only solution i found was to restart my laptop. Eventually, I gave up and went back to using docker.

Same, I tried it on Thursday, I would recommend people on M1 Macs to stay well clear of podman for now.

Yep, there's a thread with hundreds of people on M1,2,3 macs having issues, it bit some people at work on newer macs. If you installed it a while ago, it's fine, but if you installed recently, it's buggered. The solution was to install podman-desktop which was somehow packaged such that it's not getting the bug (podman machine start hangs forever).

I’ve been using it for years on M1 Macs without issue. I wrote up some notes:

https://gist.github.com/acdha/9be1c3521af4f18d9f86264a889581...

Glad that podman is getting more traction tbh, too many tools are built with the assumption that people add the sudo docker group which break if you have any kind of security conscious docker setup (such as not blindly giving it root access).

I always find it amusing that the cutting edge future of serverless/containers/etc was built on the back of running a bunch of stuff as root.

Very cool.

I'm trying to get up to speed with docker atm, is it the applications inside the container that are typically running as root (which people are careless about from a "whats the worst that can happen" attitude?) or is the typical pattern that people are running the docker host process as root (maybe in order to allow the host to expose ports at 80/443 without some reverse proxy/firewall in front of it?)

With the default Docker installation, root in a container is root on the host. uid 1234 is uid 1234 on the host.

With Podman (and Docker rootless), subuids/subgids[1] are _usually_ used. Root in the container is root only within the Linux user namespace. You can map users to real users, including yourself and root - which is how distrobox works, but it's secure by default: your subordinate users can't even access your resources.

The primary limitation of rootless is networking, the kernel doesn't have a built-in way to do rootless networking in the way that you want for containers.

[1]: https://www.man7.org/linux/man-pages/man5/subuid.5.html

Yes the default is insane. Orgs that took away root from devs two decades ago let some of this stuff slip through..

deleted

It's supposed to be a drop-in replacement for docker - IE, you can literally alias docker to podman. I did that for a while, but ran into some command line options that weren't supported.

I see they've added some k8s integration, but not sure how you see it being a k8s alternative? Maybe a minikube alternative...

it's sadly only mostly a drop-in replacement

it misses some features docker has and docker misses some features podman has and some subtle behaviour differences can lead unexpected outcomes (mostly in favor of podman doing things better)

many of the differences are irrelevant but can be an issue in scripts calling docker/podman cli

one of the more surprising discrepancies we ran into was that due to the rootless podman nature you don't need to "slowly copy the context to the daemon". In our case this lead to some slightly sub-optimal setups running just fine with podman but then taking way to long to run with docker as it copied a ton of data.

generally in my experience if you only use podman or only use docker and the "drop-in" aspect only matters for switching from one to the other permanently it works grate, but if you idk. want to use docker on some systems and podman on others it might not be the best idea

Maybe you’re thinking of something else, because Podman is mostly a replacement for Docker and shares most of the same command line, not k8s

umm what gave you that idea?

I have such a huge arsenal of custom tools to manage Docker, that I envy Podman users because I can't move to it because of this technical debt.

I just keep hoping that Docker isn't that bad and is a good alternative to Podman, because I've read mostly good things about the it, while Docker usually gets dragged through the dirt.

Podman itself is good but on the tooling side it falls down.

Docker tooling tends to work with podman but you have to do a bunch of stuff like run a server to imitate docker. At that point why bother?

A bunch of stuff being enabling a single systemd service, and pointing the DOCKER_HOST env to said service.

From what I remember it wasnt just that.

Big picture, it feels like Podman is essential the same way Linux used to be.* It doesn't matter if very few people use it -- it's presence prevents its much bigger privately-owned brother(s) from doing terrible things.

*(I say "used to be" because Linux is now even more essential and central, not less.)

It doesn't matter if very few people use it -- it's presence prevents its much bigger privately-owned brother(s) from doing terrible things.

I'm going to assume that "bigger privately-owned brother(s)" you're referring to Docker? If so, ironic given all of the evil things RedHat and IBM do in the OSS realm.

What evil things? (honest question)

I was burned by the CentOS shenanigans but I'm not aware of anything else.

And my beef with CentOS is the way they handled it, if they had left me enough time to migrate my servers we'd still be cool.

Oh I meant Microsoft and Apple as the evil bigger brothers in the case of Linux, and Docker in the case of Podman?

https://www.techrepublic.com/article/how-to-fix-the-docker-a...

I almost got burned by this.

Docker even just messes with iptables in its default configuration/installation. It's always been a pain point especially if one wants to use the newer nftables.

I've got a cryptominer in one of my personal selfhosted container of a github project meant to be used with a vpn only, due to this insecure by default choice of docker. Still salty about it...

Speaking of networking issues, one big problem I ran into was running out of memory for network devices when using a nested setup (requiring a system restart to fix). Would have been a great lxc alternative otherwise.

As a certified RHEL engineer I have been using Podman for years already.

To be perfectly honest I do enjoy it for all my personal container use. But at work I still use docker for our developers. There is so far nothing I can offer our developers that can match docker compose in simplicity.

We even use buildah in CI pipelines when we make container images, but specifically for developer end users docker compose is still dominant.

You should be able to use podman with docker compose though (https://www.redhat.com/sysadmin/podman-docker-compose)

Should, but there are still edge cases where people use weird edge features of Docker's implementation that don't always work out of the box with Podman (it's a lot better now, though, and most of my compose files are fine either way).

I've tried several times over the last few years and it always messes up somewhere, usually with networking. I stopped trying.

Somebody should tell them Docker can run in rootless mode.

It can but:

- They by default don't.

- It's only semi officially supported. It requires non official tooling, through they link to that tooling officially in their doc. But issues specific to rootless docker seem to not be much of a priority.

- roots less docker sometimes has some slight issues, but mostly minor stuff

The fact that they can make it save to use, but do not, is a really huge red flag. And it's not the first time they didn't take security on linux not serious at all.

To clarify what I mean with "make it save", the docker user group allows easy gaining root light access, which is a huge security no-go. And the alternative is using sudo or similar all the time which also is a security no-go (if you e.g. have a dev system, it's okay for starting docker images as services, but then limitations with systemd integration make podman often a better choice here anyway). And while there are ways to make it work without a security no-go and the daemon the last time I checked they weren't out-of the box and in my opinion too brittle.

Now if you have a single user system you maintain yourself and have a single user+admin+sudo right user or similar maybe then you don't care about docker group or using sudo too often. But if it's a company managed system with reasonable security requirements it's an absolute no go.

It requires non official tooling

Rootless Docker has been merged into the official since Docker 19.03, and graduated from experimental since Docker 20.10.

The "tooling" is available in the official apt/dnf repo too: - https://download.docker.com/linux/ubuntu/dists/jammy/pool/st... - https://download.docker.com/linux/centos/9/x86_64/stable/Pac...

One of the nice things about podman is that it's super easy to configure image caches. So instead of rewriting all my image references to use my local cache tool, I can just set a cache directive in podman and everything works transparently.

Can’t you run your own docker registry and enable a pull-through cache on it with just a couple commands?

Can you? Pull through cache is awesome but I have been leaning on harbor registry to implement it. Harbor is great but is a full featured registry application. Would be sweet if pull through can be quickly spun up ad hoc.

Can anyone tell me why neither Docker nor Podman allow you to dynamically modify forwarded ports? It would allow zero-downtime updates of containers (starting new container, wait for it to be healthy, update port forwards, stop old container).

And no, reverse proxies do not solve this problem; lots of protocols (e.g. SSH) have no equivalent to X-Forwarded-For for identifying the remote host.

Likely due to technical limitations. At least docker does its port-forwarding with iptables (or these days, nftables), and the forwarding rules themselves span multiple custom tables/chains. If you ever do 'iptables-save' to inspect what rules have been created when a container is running, it may look a bit funky.

So updating a purportedly single rule might actually require to update several underlying traffic mangling rules, with logic that is not readily apparent. Or even easy to reason about. When you add the ability to route traffic directly from container to container without passing through the outermost interface, things can get quite hairy.

Don't take me wrong. Podman is great and I use it instead of docker nowadays but when I started using it thinking it was just a docker replacement I got burned by UID and GID mappings, SELINUX policies, missing DNS configuration and more.

More than once I wrecked my whole setup running system migrate as a way to fix problems. It has a whole thing about security ACLs, ID mapping and labels. A chmod -R under your home folder will probably kill all your containers.

While I'm happy with the results it was far from an "it just works" solution like Docker. I imagine things probably have been improved since I started using it.

I just started using it this year (as a way to isolate various dev environments, and as a way to prevent npm from having trivial access to my entire dev machine). It was easier to use than Docker (in my opinion). It seems to me and things have improved from what you experienced.

One other thing podman (unlike Docker) is missing is ability to run x86 images of Apple silicon under Rosetta. QEMU turns out to be too slow for real use.

Could anyone please advise a paper on Docker architecture, discussing its design choices (e.g. client-server model) in detail?

Sounds good, I like their security first approach and some of the decisions they've made, going for secure defaults out of the box, and that it works with docker compose. I wonder if podman gains enough traction, at some point, they decide to go their own way with regards to the commands and the yml, because right now it seems to be a tool that 'hangs on to' docker and docker's compose file format.

It would be good to have a swarm alternative in podman, it seems like k8s is a crutch for lack of orchestration. With their good security hat on they could probably come up with a sane, simple way of running containers at small scales without having to dive into a PhD in k8s which doesn't have secure defaults out of the box, while maintaining compatibility with docker compose format.

Anyway that's a good intro thanks for sharing, I'll be trying it later.

Also worth mentioning that while docker has been catching up by offering a rootless mode for years now, it insists on running a daemon process.

That daemon is a subtle but incomensurate burden when adopting the least privilege principle.

In environments running multiple hosts which themselves run multiple containers, typically: k8s, it forces your hand in either giving in and grant docker (the daemon) root privilege if any one of your container needs root, or to exclude that container from running in that environment altogether (since it would fail to execute if the docker daemon is in rootless mode). Of course the most secure and wise option would be to refractor that container and whatever it's doing to run rootlesssly, but sometimes this is simply not a reasonable or even possible option in order migrate massive complex platform of hundreds of micro services with its own history and justified security exceptions.

K8s (and Openshift, which adopts a stronger security by default configuration set) provides control over which service accounts is granted such exception.

tl-dr: use podman/buildah rather than docker, use openshift rather than vanilla k8s.

I tend to just make and run shell scripts that configure and run bubblewrap[1].

Everything is nicely explicit and allows for a good mental model of what's going to happen when you run it.

    source "/path/bwrap_helper.sh"
    FLAGS=(
      ${FLAGS_ROOTFS_DISTROX_MIN[@]}
      ${FLAGS_ENV_XDG_GUI[@]}
      ${FLAGS_PULSE[@]}
      ${FLAGS_GPU_ACCEL[@]}
      --new-session
      --bind /path/jail123 /home/user
    )
    exec bwrap "${FLAGS[@]}" --seccomp 10 10< /path/a_filter.bpf -- /usr/bin/gui_app "$@"

[1] <https://github.com/containers/bubblewrap>

The comparison is unfair in that it compares docker-as-root vs rootless podman.

A more sensible comparison would be between docker rootless and podman rootless.

I agree that rootless containers and isolated namespaces are critical security features. But with docker rootless, this is also possible and not complicated. You just have to do it. I have written a blog post to set up Mastodon in docker rootless with all the best practices currently available [1].

The benefit with sticking with docker is that accessibility is better: More communities, more blogs, broad availability of docker compose configs, more peers knowing how to use it etc. In the end, both podman and docker run processes in isolated namespaces on the host.

[1]: https://du.nkel.dev/blog/2023-12-12_mastodon-docker-rootless...

Podman is designed to help with this by providing stronger default security settings compared to Docker. Features like rootless containers, user namespaces, and seccomp profiles, while available in Docker, aren't enabled by default and often require extra setup.

Seccomp has been enabled by default since 2015: https://github.com/moby/moby/pull/18780

It is true that Rootless isn't enabled by default but its "extra setup" can be done with a single command (`dockerd-rootless-setuptool.sh install`)