↓Skip to main content

Nix the Hard Way

·3211 words·16 mins
Unrefined Kepler Nix
Author
Jonas A. Hultén
Opinionated engineer who rambles semi-coherently about DevOps, type systems, math, music, keyboard layouts, security, and misc.

The time has come to kind of, sort of actually learn Nix. I’ve been faffing around in NixOS for the better part of a year now, so I have seen plenty of room for improvement in how I manage my systems. For instance, my “main” machine’s configuration.nix file is pushing 500 lines, which is… a bit of a quagmire.

This post exists in order to detail my learning experience in figuring out the Nix language and how to actually configure systems in a way that won’t just turn into spaghetti.

Nix
#

Nix is a functional programming language. So far, so good — I know Haskell to a passing degree.

What this means, to the unfamiliar, is that it’s a programming style that’s way more akin to mathematics than conventional imperative programming. Imperative programming is a series of commands, executed in order. Functional programming is stringing functions together in order to work the input data into a suitable output.

My great problem with Nix so far is that I don’t know what “suitable output” is, nor really what the inputs are. In designing the “Kepler” system — reminder: that’s my be-all, end-all configuration solution for all my machines — I want things to be modular, self-contained, and relatively easy to use. I’ve tried faffing about with declaring my own modules for configuration, but I keep getting type errors.

The official Nix documentation is, in my experience, incredibly hard to read. Moreover, there seems to be a schism in the “official” Nix community on either side of the “nix flakes” debate — the documentation either doesn’t mention flakes at all, or mentions nothing but flakes. This makes it a little difficult to get anywhere.

Thankfully, there are a myriad blogs available which go into the topics. Some useful examples I’ve found are:

The starting point
#

Up until recently, I had one (1) NixOS machine — my janky laptop. I then got a second — a discount server in a datacenter in Finland. Managing these two by smacking the configuration.nix file until it worked was well enough, but I have plenty of old derelict computers I want to get back into gear, and then making sure that each machine has some sort of unified configuration starts to become an issue.

I recently retired my old desktop computer and shoved it into a closet I’ve dubbed a “Server Room” to live out its days as a server. Step one of this process — after the obligatory data rescue of stray files I had on the soon-to-be-wiped disk — was to shunt NixOS onto the machine.

Step zero was actually solving The Bootstrapping Problem — I discovered that, of course, making a custom NixOS boot disk was done via Nix, and there was a helpful tutorial on the Wiki. Using that, I could easily build a NixOS boot image that had my SSH public key pre-loaded. That doesn’t help me in the cloud, where I can’t supply a custom ISO, but it does help for all local machines.

Anyhow, once nixos-generate-config had done it’s thing and bootstrapped a configuration.nix and hardware-configuration.nix file, I could step in and inject my own config.

Here’s an example, taken from the above mentioned old desktop:

# configuration.nix

{ config, lib, pkgs, ... }:
{
  imports =
    [
      ./hardware-configuration.nix
    ];

  boot.loader.grub.enable = true;
  boot.loader.grub.device = "/dev/sdb";
  boot.tmp.cleanOnBoot = true;

  networking.hostName = "mercury";
  networking.domain = "kepler";
  networking.networkmanager.enable = true;

  services.openssh.enable = true;
  services.openssh.settings.PermitRootLogin = "no";

  services.resolved.enable = true;

  users.mutableUsers = false;
  users.users.sldr = {
    isNormalUser = true;
    description = "Jonas A. Hultén";
    extraGroups = [
      "networkmanger"
      "wheel"
    ];
    hashedPassword = "[redacted]";
    openssh.authorizedKeys.keys = [
      "[redacted]"
    ];
  };

  nix.settings.experimental-features = [ "nix-command" "flakes" ];

  time.timeZone = "Europe/Amsterdam";
  environment.systemPackages = with pkgs; [ vim curl neofetch nixfmt bat ripgrep htop ];

  system.stateVersion = "24.05";
}

This is then simply used as a module in the core system flake.

Don’t actually ask why I’m using flakes. They looked neat, and it’s mostly just an exercise so I can learn. 
# flake.nix

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
  };
  outputs = inputs@{ self, nixpkgs, ... }: {
    nixosConfigurations.mercury = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [ ./configuration.nix ];
    };
  };
}

So that’s where we’re starting off from.

The next step(s) are to split out segments from the “legacy” configuration.nix file into flakes, which I can then reuse across machines.

However, the very first thing I tried to do was to restructure the files so that I could maintain all my machines in the same directory structure. This means moving the “legacy” files into legacy/mercury and putting a flake.nix in legacy to handle importing them.

What I mean by “legacy” in this instance are the configuration.nix and hardware-configuration.nix files which were created during system installation. It seems like the path of least resistance to handle them using a flake before I start gutting them and re-implementing whatever config is in them using more flakes.

However, since I vehemently believe in not repeating myself, I’d like this flake to take an argument — the hostname — which points to the “legacy” files to include, rather than needing to update the flake any time I add a new machine.

This is where I got stuck.

I also got sidetracked for a few days messing around with this blog itself, refining my CSS-fu and taking a hard detour through gradients and color space mapping. More on that later (see the meta tag).

A recursive mess
#

As mentioned above, I initially thought I could implement the legacy flake with a function that took the system’s hostname as an argument. Then I thought to myself “Hey, I can be more clever than that.” NixOS should already know the system hostname, so shouldn’t I just be able to reference it via ${networking.hostName}?

Long story short: no.

Or, in all honesty, probably, but that would involve me figuring out how to feed the system configuration into a function from inside the flake which is defining the system configuration. NixOS (and Nix in general, I guess) is supposedly good at doing that sort of iterative evaluation stuff, but I couldn’t figure out how to do it. While I am exhausted from banging my head against this, I can’t seem to find any examples of anyone doing this either.

So, we’re back to creating a flake that has a function that handles the imports. After far too many syntax errors, this is what fell out:

{
  description = "Hardware and unported interface flake";

  inputs = { };
  outputs = { self, ... }: {
    nixosModules.default = ({ name, ... }: {
      imports = [
        ./${name}/configuration.nix
        ./${name}/hardware-configuration.nix
      ];
    });
  };
}

I could then — successfully — roll that into the root flake by calling the module like a function:

  inputs = {
    ...
    legacy.url = "./legacy";
  };
...
      modules = [
        (legacy.nixosModules.default { name = "mercury"; })
      ];
...

A word of warning: if you decide to rename a flake, make sure to rename all references to it. I spent far too long trying to work out why Nix was insisting the legacy flake was still called fundamentals (the old name) until I found that it still said fundamentals in the root flake outputs function arguments, i.e.

...
  outputs = inputs@{ self, nixpkgs, fundamentals, ... }: {
...

The Nix error messages were less than helpful, just saying that it couldn’t find the fundamentals flake in any registry, while I banged my head against the desk, thinking I’d scrubbed all references to it, up to and including destroying the flake.lock file and digging through the garbage collector roots to see if Nix had cached my old sins anywhere.

Nope. Just a SNAFU on my end.

Refactoring Kepler
#

Unrelated to the above, I already had a little Kepler git repo set up which mostly contained some loose ansible and tofu code, as well as a flake to create a dev shell. Now, it was time for the two to meet, making a be-all, end-all flake.

Now, it turns out I’ve learned a thing or two about Nix since I messed with this flake last time, so step zero was cleaning up the implementation of the dev shell.

For some ungodly reason, I’d implemented flakes inside Kepler to use flake-utils, but not the root flake. Ostensibly, this doesn’t matter, since I currently don’t run Nix on anything that isn’t x86_64 Linux, but there’s a good chance I’ll try out an ARM server at some point (or put NixOS on one of my myriad Raspberry Pis, now that I think about it), so might as well make the dev shell multi-system.

Skipping over what it used to look like, here’s the current state of affairs:

{
...
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
    flake-utils.url = "github:numtide/flake-utils";
    ...
  };

  outputs = inputs@{ self, nixpkgs, flake-utils, sldr-meta, ... }:
    # Create the dev shells
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
        sldr = sldr-meta.packages.${system};
      in {
        devShell = with pkgs;
          mkShell {
            name = "keplersh";
            buildInputs = [ ansible ansible-lint dhall opentofu sldr.gitgum ];
            shellHook = ''
              source .secrets
            '';
          };
      });
}
The astute observer will note the sldr-meta flake which is being used. That’s just a helper flake to access some of my own applications and helper scripts. In the above example, it’s being used to include my gitgum utility.

It took me a while to work out how I could use the eachDefaultSystem function while also adding other flake outputs, until I learned/realized that I could use the // operator to glue on another set, which will then contain the nixosConfigurations.

Glueing it together
#

Copying over some code from the experimental directory, and organizing it a bit better, we get the following:

...
    legacy.url = "./nix/system/legacy";
  };

  outputs = inputs@{ self, nixpkgs, flake-utils, sldr-meta, legacy, ... }:
    {
      # Create the system configurations
      nixosConfigurations = {
        mercury = nixpkgs.lib.nixosSystem {
          system = "x86_64-linux";
          modules = [ (legacy.nixosModules.default { name = "mercury"; }) ];
        };
      };
    } //

    # Create the dev shells
    flake-utils.lib.eachDefaultSystem (system:
...

And, according to nix flake info, that works as intended.

Now we’re cooking with Nix.

Breaking it apart again
#

The end goal of this whole flake journey is to modularize my configuration, so that I can slot in well-known pieces of configuration across my machines — and update all machines at once, for that matter. So that’s what we’re going to get to next.

The first, and maybe most obvious chunk of code to break out is the definition of my user. But first, we have to take a detour down Pragmatism Lane.

When flakes aren’t the right answer
#

Going into this, I envisioned my entire configuration broken up into a slew of flakes, each defining it’s little piece of system config. Now, that makes sense — to a point. Each flake can have its own inputs, like a reference to nixpkgs or some other fun stuff, but consider the hypothetical flake that contains my user definition.

It won’t have any inputs — everything that it’s defining is a NixOS builtin. It’s not going to be installing packages. It’s just defining my user and some related bits and bobs. Consequently, there’s no need for putting it in a flake, since there’s no need for it to have a lock file to track and lock its inputs.

Indeed, I’ve read that the general best practice when using flakes for defining packages is to still have a sidecar default.nix which does all the heavy lifting, while the flake just imports it. That’s what my gitgum flake does.

So, it’s now time to start breaking apart some chunks into good ol’ fashioned Nix modules.

The users modules
#

This part turned out easier than I thought it would be, honestly. I cut out the user config, stuck it in a new file, and listed it in the modules list. Bing bang boom, it works.

# nix/system/users/sldr.nix
{ ... }:
{
  users.users.sldr = {
    isNormalUser = true;
    description = "Jonas A. Hultén";
    extraGroups = [
      "networkmanger"
      "wheel"
    ];
    hashedPassword =
      "[redacted]";
    openssh.authorizedKeys.keys = [
      "[redacted]"
    ];
  };
}

# flake.nix
...
      nixosConfigurations = {
        mercury = nixpkgs.lib.nixosSystem {
          system = "x86_64-linux";
          modules = [
            (legacy.nixosModules.default { name = "mercury"; })
            ./nix/system/users/general.nix
            ./nix/system/users/sldr.nix
          ];
...

The users/general.nix file just contains the single line users.mutableUsers = false;, but establishes the general structure I intend to have for my configuration — there are specifics, which have some sort of name, and there is general config, which should apply to everything.

After some more hacking and slashing, here’s the structure that’s starting to emerge:

nix
├── sldr-meta
│   └── flake.nix
└── system
    ├── general.nix
    ├── legacy
    │   ├── flake.nix
    │   └── mercury
    │       ├── configuration.nix
    │       └── hardware-configuration.nix
    └── users
        ├── general.nix
        └── sldr.nix

The system/general module contains misc. settings which should apply to all systems, regardless of architecture or type (such as running an SSH server). However, it currently also installs some system packages, and I should probably migrate that elsewhere.

Code formatting is important, so here’s a tip.

Nix flakes support providing a default formatter, which you can then invoke by running nix fmt, which will auto-format all .nix files in the current directory as well as any subdirectories.

In my case, I set it inside the flake-utils.lib.eachDefaultSystem function — since formatters are applications, you need to specify the architecture — and then it was as easy as:

# Set a formatter
formatter = pkgs.nixfmt-rfc-style;

Package management
#

Package management is where it actually makes sense to use subflakes. For instance, I have LaTeX installed on my old laptop. Since it isn’t flake-configured, any time I update my system, there’s a good chance I also update LaTeX — which, as some of you may know, is a huge amount of packages. The same goes for my printer drivers, which for some reason have to be built from source. It’d be nicer to stick those packages behind flakes and then only update them as needed.

At the same time, there are a fair few packages I want on the bleeding edge — ssh is one, just for the security reasons. The kernel is another.

A good place to start is the general packages module, which currently looks like this:

# nix/pkgs/general.nix

{ lib, pkgs, ... }:
{
  # Install basic system packages
  environment.systemPackages = with pkgs; [
    bat
    curl
    git
    git-crypt
    gnupg
    htop
    neofetch
    nixfmt-rfc-style
    openssl
    pinentry-all
    ripgrep
    tree
    vim
  ];

  # Enable direnv globally - might as well
  programs.direnv.enable = true;
}

These are the packages which I expect to have on all machines, just because… because.

Now, a question of semantics: do I treat services like packages and split them off into their own little category too? Or does that belong under system? Technically, this also belongs under system, since these are packages which are installed system-wide, so where do I draw the distinction?

I was going to go back and edit this part to reflect what I actually settled on, but my thinking turned out to be that putting packages under nix/pkgs was a mistake — the top-level concern is whether something is system-wide or user-specific. Consequently, the newly created pkgs/general.nix file has been moved to nix/system/pkgs and I will create a services module there as well.

I’d like to think that it’s more honest — if more verbose — to show my work, rather than edit my work down to what turned out as the “best” idea.

With some additional jiggery-pokery, I have abstracted out the ssh service, and I’m getting closer to where I want to be.

Hardware support
#

Here, it once again turns out that the nerds over in the Nix community have done the hard work for me. Rather than needing to set up my own modules and import/exclude kernel modules and what not, the nixos-hardware repo exists. Better yet, it has a flake, meaning I don’t even need to write anything to handle it — just slap that into the flake inputs and off we go.

inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
    nixos-hardware.url = "github:NixOS/nixos-hardware";
...
  };

  outputs =
    inputs@{
...
    }:
    let
      defaultModules = [
        ./nix/system/general.nix
        ./nix/system/pkgs/general.nix
        ./nix/system/services/ssh.nix
        ./nix/system/users/general.nix
        ./nix/system/users/sldr.nix
      ];
    in
    {
      # Create the system configurations
      nixosConfigurations = {
        mercury = nixpkgs.lib.nixosSystem {
          system = "x86_64-linux";
          modules =
            defaultModules
            ++
              # Hardware config
              [
                nixos-hardware.nixosModules.lenovo-ideapad-slim-5
                nixos-hardware.nixosModules.common-cpu-amd-pstate
              ]
            ++
              # Custom flakes
              [ (legacy.nixosModules.default { name = "mercury"; }) ];
        };
...

You’ll note that I’ve also started defining my own list of defaultModules which I can slot into any future machine I want to set up. To me, this seems like a better approach than having a super-module which, in turn, includes those other modules — it means the actual system definition lives here in the root flake, rather than elsewhere.

Revisiting the legacy flake
#

You may have realized it before me, but the legacy flake is completely superfluous in light of my earlier discussion about flakes vs. modules. By all rights, it should be a module — it has no independent inputs. So why isn’t it?

Put simply: I forgor 💀

However, the problem DID rear its ugly head when I got tired of just configuration, jammed all the remaining config into the legacy files, and triggered an actual rebuild.

Now, keep in mind: I’d been running nixos-rebuild dry-build routinely throughout this work. I was relatively sure this would behave as expected.

So I built a new boot generation (since I’d switched desktop manager) and rebooted. Then, after reboot, I’m greeted by… a black screen with a white login prompt.

I double check the settings. services.xserver.enable is true. So… why no X?

Cutting to the chase, the legacy flake wasn’t actually working. It referenced the files like it should, but for one reason or another, they weren’t actually imported. Consequently, all my legacy config was unapplied.

The solution was to use a regular ol’ fashioned module. Weirdly. To the best of my understanding, the legacy flake was correctly making a module, but… I’m not going to dig into it deeper. Using a module solves the same problem with fewer lines of code.

Here it is, in its entirety:

# nix/system/legacy/default.nix

{ name }:
{
  imports = [
    ./${name}/configuration.nix
    ./${name}/hardware-configuration.nix
  ];
}

This does present an interesting conundrum, because this file technically isn’t a module — it’s a Nix function which returns a module. Consequently, I have to handle it a bit differently.

  outputs =
    inputs@{
...
    }:
    let
      legacy = import ./nix/system/legacy;
...
    in
    {
      # Create the system configurations
      nixosConfigurations = {
        mercury = nixpkgs.lib.nixosSystem {
          system = "x86_64-linux";
          modules =
            defaultModules
            ++
              # Legacy config
              [ (legacy { name = "mercury"; }) ]
            ++
...

And that is where I’ll call it a day.

Been working on this post for well over two weeks now, and I want to play with my computer for a bit, rather than just configure it. Also, I need coffee, so let’s get to that.

Tot ziens