In more recent years, Linux distributions have become quite interesting. The hypothesis of immutable Linux have gone from pure thought, to full throttled theory. There exists a plethora of options out in the wild. All from different companies, distributions, and communities.

Now while many options exists, for me, I been debating on three. Dive so deep I hit bedrock with Nix and NixOS. Accept Kubernetes as the one true OS through Talos. Or drink the orange glowing Kool-Aid of snaps in Ubuntu Core. Bare with me. There's logic in these.

The choices

I been debating the choice on what to use as my de-facto immutable OS for a while now.

The three I settled (for now) are ones I think make the most sense. To set and forget. Distributions where I can realistically automate the entire thing. Without much fear of the ground falling apart. Because unlike normal Linux distributions, immutable OS versions are the wild west of Linux distros. All choose radically different ways of achieving same goals. Choice is great for ecosystem growth. Terrible for a foundation.

Ubuntu Core

The one I'm sure gets most debate is Ubuntu Core. Even though it's the most systematically grounded version of an immutable OS. The design is fairly simple. Use snaps. But the genius of this is how Canonical made every part of the OS into an isolated snap. Snaps you can update and change without impacting the 'core' of the OS.

You can update the entire OS with changing the snap version of the core you are based from. So jumping from Ubuntu core 20 to Ubuntu core 22 was doing something like this. And as Canonical has matured with how Ubuntu core operates, they have also made different layers to this design. Using the atomic principal.

So now there are snaps for the linux kernel. And configuration based ones called gadget snaps. Responsible for handling bootstrapping on a specific hardware. All following the same everything is a snap.

You can still run any application you want. The difference here is you have to package them differently. Or at least differently from the norm.

The con with Ubuntu core is that you have to accept their way of packaging software with snaps. There is no alternative. You must have Canonical as your overseer here. The pro is everything stated before benefits your ability to have a stable immutable system. It should not be understated. The ease that comes with having maintainability so low, with ubuntu core, is pretty remarkable.

Talos

Talos differs in a very simple way. The ENTIRE operating system. It is Linux technically because Talos uses the Linux kernel. It immediately deviates from there. You don't use an OS with Talos. What you use is Kubernetes.

Kubernetes is infamous at being complex. I even wrote about my annoyance years ago. After years, that article is still true for self hosted Kubernetes cluster. So is Kubernetes worth self hosting? I can wholeheartedly say NO. But with an exception. The exception here is if you are using Talos.

You see, Talos strips away practically everything of the OS. But it does it in a way that makes a lot of sense. You don't use ssh, because there is no shell to ssh to! Instead you use their gRPC API component; apid. Interfacing with the operating system. Talos has no systemd. Instead, replaced with their own PID 1. An init system called machined. With the whole purpose running what kubernetes needs and the gRPC interface to define the OS.

In practice, Talos is actually dead simple to use and administer. It makes using and maintaining kubernetes strikingly easy. New version release? Run talosctl to upgrade:

talosctl upgrade-k8s --to 1.28.0

Same is true for updating Talos itself. Because the OS is atomic, there is very little thought process required to handling failures. You rollback like nothing happened. Here, you simply use kubernetes.

The con with talos isn't actually the use of talos. It's the principal of only using kubernetes for everything. If you not comfortable with a job orchestration like it, DO NOT USE. If you somehow like ssh or want to install other things directly on the OS, DO NOT USE. If you don't want to do literally everything as code, definitely DO NOT USE.

But if you do value what Talos offers, it's immensely difficult not to choose it. So many problems are simply non-existent on Talos. The OS makes you question why even bother with the old ways.

NixOS

The prophecy is written. NixOS is the answer to our Linux administration ways. It will solve all our problems with packaging software. We will know of history as before and after NixOS. And quite frankly, it really does feel this way.

NixOS shines in the same ways the others in this list shine. It rethinks what a Linux is and could be. If you make absolutely everything atomic, to the core of how you package and run software, then do you even need to care if your OS breaks? NixOS potency is the nix programming language. Not to be confused with the nix as a package manager. Or the OS who is also called nix.

Unlike the other options on this list, the con with NixOS is quite immediately apparent. It's the difficulty you first have learning the ways of then learning the ways of using nix. Documentation is quite difficult to come by for nix. Much of your time will be left questioning how anything even functions. There's also then the full upgrade to nix design called nix flakes. All enough to really put incredible friction. Friction on nix usage and nixos adoption.

However, the moment you passed the hurdles of learning nix, nothing comes even close to its versatility. I personally have migrated most of my software to run with nix. Or at the very least, build with nix. For work, I have development environments strictly nix based. And the list goes on.

With NixOS, it's the same principal. You write your closure and you can be assured it will work. No matter what mess you do to the machine. You can roll back like nothing ever occurred. There's nothing really like Nix and NixOS. The principal is that you handle the hurdle of defining how your software is built. From then on, it will just run.

No more conflicts with versions of python. No issues with running two independently different versions of the same software. Because of the closure design, there's no need to containerize your applications. They 'just' work. With zero conflict running on the same host. Optimally, NixOS gets you the closest to the promise of Gentoo, but entirely atomic and immutable.

So what to choose?

I don't know yet. The reality is, all three options serve very similar ideas of running an immutable OS. They simply attack the problems differently. Talos packages software in kubernetes manifest essentially. Ubuntu Core is snaps. And NixOS is nix closures. All with different tradeoffs that are far too long to add to this never ending post.

For security reasons, I would probably go for Talos. Because of the stripped purpose of the OS, there's a smaller footprint to a security issue. Yes. Even with Kubernetes.

For maintainability, I think Ubuntu Core is prime. Canonical has been doing Linux distributions for decades now. They know what it means to make something function. Every Ubuntu core release has a maintenance window of up to ten years. Meaning, if I want to just run my thing, with no fuss, this will be it.

For customization, nothing gets even close to what NixOS promises and delivers. I would be able to take all the Nix flakes I been writing for myself and run straight on NixOS. True “it works on my machine” on all machines.

#linux #immutableos #ubuntucore #talos #nixos #atomic #ubuntu #kubernetes