Slowly filling up /boot

So guess what, /boot went full.

Actually it’s worse than that: I didn’t care, because I thought that I would just need to invoke the garbage collector to fix the issue when it will show up.

Garbage collecting

So, let’s run sudo nix-collect-garbage --delete-older-than 7d as planned… and it doesn’t clean up /boot

Searching online for the correct nix tool to do this and the stuff popping up look like:

• Don’t let this happen, fix your config (fair enough, that’s a valid answer, but not to this problem)
• sudo nix-collect-garbage --delete-older-than 7d yes, if /boot is not already full
• Complain about the fact NixOS does not prevent this to happen (to be fair I’m not aware of a distro preventing this, and this is an issue I quite respect as it is often noticed the hard way on the next boot)

What bothered me a bit (and that’s why actually this post exits) is that there does not seem to be a “clean” nix way to do this.

So, maybe I didn’t find the clean solution I looked for but what at least worked for me (and should work on any system): delete manually a few entries in /boot/EFI/nixos/

I only deleted the last two oldest kernels and initrd sets, don’t need to nuke everything, as I wanted to check that the garbage collector worked as intended, when it can.

sudo nix-collect-garbage --delete-older-than 7d

Yes… this cleans /nix/store, but I couldn’t find a message about removing stuff in /boot on success (if /boot is full, it does not report anything either…).

Now, building is possible again nixos-rebuild switch --flake /etc/nixos and df reports a sane /boot again.

I’d like to investigate the internals of NixOS to understand why nix-collect-garbage cannot delete stuff in this very situation or who exactly deletes stuff.

Notes

NixOS gives a great feeling of resilience. If you want to sneakily nuke NixOS, maybe start with dd if=/dev/urandom of=/boot/let_us_fill_this_boot_partition and break the current system.

Apparently you also can run in a very similar issue if /tmp fills up (iirc a full /var on Debian also leads to interesting issues).

There is this fairly old NixOS open issue about this

Use something like boot.loader.systemd-boot.configurationLimit = 5; in your config.