Audit the box before you wipe it

Part 2 · Budget homelab

Halfway through the migration (see The box and the move to the new server), I realized that I had never moved the message-archive pipeline. Its only copy (about 108 GB) was still on the old server, and the pipeline was dead. The service-by-service moves had told me what had moved, not what was left behind. I decided not to wipe anything until I had audited the entire box from top to bottom.

I labeled each artifact as either migrated, re-derivable, drop, or ACTION. The latter meant it existed nowhere else and I needed to make a call on it. I decided not to wipe anything until all ACTION items were cleared. Most of the results were boring: datasets that had already been copied, data that could be regenerated. Boring was what I wanted: it meant I could safely lose it.

The first ACTION item was a 106 GB dataset containing an old personal archive (documents and keys) that I didn’t have another copy of. It had the same name as a message-archive dataset I already had on the new server, so I’d have assumed it was safe based on the name, but the contents were different. I moved it into a child dataset under the existing archive parent on the new server, so it automatically inherits the recursive snapshots and nightly backup without any extra setup. I used a ZFS snapshot and zfs send into a receive, and verified that both sides had 903 files that were byte-identical.

The scan of my git repositories found the other three ACTIONs: one for a dashboard repo that had no remote (the old disk was the only copy), another for a repo with a single unpushed commit, and a third where I’d stashed some work in progress. All are now on my self-hosted git, and I used git stash branch to put the stash on its own branch (rather than merging it into the WIP commit I’d pushed, which might not match).

Not everything that was flagged got migrated. The old server had a copy of the bird-detection app from BirdNET listening in the backyard. I didn’t want to drop that unless I was sure the new server had everything the old one did.. had the old box picked up anything new that I’d missed? It turns out that both have the same earliest detection, but the new one has 22,304 detections compared to the old one’s 18,035. So the old copy went on the drop list.

Halfway through, the monitoring agent reported healthcheck zombies in 30 containers and suggested I set init: true on them all. It had never looked at the parent process of those zombies. Turns out all 16 recurring zombies were [git] <defunct> children of one container: forgejo-runner. The daemon is PID 1, so it has no init to reap the git commands it spawns for each CI job. Thirty containers sounded like a host-wide pattern, but it was really one container leaking in bursts.

I rejected the 30-container change and set init: true on the runner only, as Forgejo issue #7854 and Gitea issue #19077 recommend for runner-as-PID-1. Docker’s --init runs tini as PID 1 to reap. Zombies dropped from 16 to 0. The leak was slow, so this was hygiene, not rescue. I noted the global Docker daemon init setting as a next step if leaks ever show up across several containers. I also made the agent confirm script changes with me before applying them.

    container_name: forgejo-runner
    network_mode: host
    restart: always
    # PID 1 is `forgejo-runner daemon` (a Go process, no init). It shells out to
    # `git` per CI task and doesn't reap orphaned children → `[git] <defunct>`
    # zombies accumulate in bursts per job (slow PID-slot leak; cgroup pids.max).
    # tini as PID 1 reaps them. Exactly the fix Forgejo #7854 / Gitea #19077
    # recommend for the runner-as-PID-1 case. Single-process daemon → safe.
    init: true

With all ACTIONs cleared and nothing irreplaceable left, I exported the ZFS pool and powered the old server off. Then I discovered that the router’s watchdog from OpenWrt to the rescue was still watching it, pinging it every five minutes and trying to Wake-on-LAN it if it didn’t answer. I could tell because the “down” state file was stale. I removed it from the watch list so it stays down while the drives come out.

With the old server now dark and the pool exported, all that was left was for it to give up its drives. I’ll pull the two I want to keep and put them in the new server.