A used HP EliteDesk with a few mods runs my house

Part 1 · Budget homelab

After 38 days of uptime, my old server had 87 pending package updates, including a kernel bump. The old hardware was a Dell OptiPlex with an i7-6700, which had been running 45 containers and storing all data on a single 2 TB Samsung SSD in a one-disk ZFS pool. I replaced it with a used HP EliteDesk 800 G6 SFF with an i7-10700, which cost me under $300. I decided to do a fresh install of Ubuntu Server instead of moving the old NVMe, because it would give Ansible a chance to replay its configuration and get rid of the unused GNOME desktop (which uses about 1 GB of RAM and consists of 41 packages) and the 12 to 15 GB of cruft in my home directory. More time up front for sure, but worth it.

ZFS mirror and the parallel run

For storage, I went with two 8 TB drives in a ZFS mirror instead of mdadm, because ZFS does block checksums (against bit rot), snapshots, and I can move the pool to another machine with zpool export and zpool import. I’m not using L2ARC or SLOG, because RAM ARC is sufficient and nothing is bandwidth-bound. Similarly, for memory, I planned 4x16 GB at 2133 (64 GB dual channel) instead of 32 GB at 3200, because capacity is more important than bandwidth for a ZFS box running 45 containers. For now it’s running 32 GB.

I decided to run both machines in parallel on the LAN and migrate services one by one. The old box was my fallback, so nothing got deleted, and the data directories were my rollback net. Throughout this, I had an open hardware question: I’d saved a build of the exact same machine on r/homelab, the same EliteDesk 800 G6 SFF, the same i7-10700, and it made me wonder: does the G6 SFF actually have two M.2 slots? I didn’t know for sure by the time I was done.

Benchmarks and the pool

The first benchmarks against the old box showed the CPU to be around 80% faster, and the NVMe drives were 50 to 75% faster. One Seagate Exos 8 TB was second-hand, with 54,000 power-on hours, zero errors, but a high G-sense count of 14,573. It ran at 55°C during the extended SMART test (about 13 hours). The second Exos was factory-fresh with zero hours. Both went into the mirror.

I created the pool with ashift=12 and lz4 compression. I had 14 datasets planned, but reduced that to 8. I set recordsize=8K for Postgres and 1M for media and Time Machine. I capped ARC at 8 GB of the 32 GB RAM. I didn’t create a backups dataset because a backup on the same disks as the protected data isn’t a backup.

I set a few defaults on the Docker daemon before starting any containers. Using my own address pool means Docker networks won’t collide with my VPN subnets. Enabling live-restore means containers will keep running through a daemon restart. I learned on the old box that having Ansible own everything leads to quick fixes that bypass it and accumulate drift. So on the new box, Ansible only owns the base system. Containers are deployed from git, see How a deploy works with Dockhand. The first service I deployed was a Grafana/Loki/Alloy/Tempo observability stack.

// ansible/templates/docker/daemon.json.j2 (excerpt)
{
  "log-opts": { "max-size": "10m", "max-file": "3", "compress": "true" },
  "exec-opts": ["native.cgroupdriver=systemd"],
  "live-restore": true,
  "userland-proxy": false,
  "shutdown-timeout": 30,
  "default-address-pools": [
    { "base": "<private /16>", "size": 24 }
  ],
  "bip": "<private /24>"
}

Reboot in place first

I rebooted it in place before moving it anywhere. That way I could separate a clean-boot recovery from any cable damage during transit, while still having remote access as a safety. A little script baselined the pre-reboot state and snapshotted the post-reboot state, listing running containers with a no restart policy and running-but-not-enabled system services. The diff showed what wouldn’t come back on its own. The rehearsal caught Uptime Kuma crash-looping because its empty uninitialized database was throwing no such table: setting. I stopped it until I could copy the real database from the old box.

The move and the mains pull

The real move was a ZFS snapshot of the pool, a graceful power-off, and a walk to the new spot. I put it on a 1000VA UPS, hooked up power and USB, plugged in the USB microphone for the bird detector, and set the BIOS to power on when AC returns. After it booted, 39 of 40 containers came back byte-identical, the mirror was intact with zero errors, and the Exos drives cooled from 55°C to 42 and 41°C.

I tested the UPS by pulling the mains power. Network UPS Tools reported it was on battery and discharging, and sent me an ONBATT push notification. Power was restored in less than a minute, so it went back online and started charging without shutting down. That’s by design: NUT only shuts the server down when the UPS reports low battery.

# ansible/files/nut/upsmon.conf.j2 (excerpt)
# Conservative: graceful shutdown only when the UPS asserts LOW BATTERY (LB).
RUN_AS_USER nut
MONITOR {{ nut_ups_name }}@localhost 1 upsmon <redacted> master
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
POWERDOWNFLAG /run/nut/killpower
NOTIFYCMD /etc/nut/notify-pushover.sh
NOTIFYFLAG ONLINE   SYSLOG+EXEC
NOTIFYFLAG ONBATT   SYSLOG+EXEC
NOTIFYFLAG LOWBATT  SYSLOG+EXEC

Cutovers and persisted config

The biggest cutover was my smart home stack: Home Assistant, Mosquitto, Z-Wave, Zigbee, Matter, ring-mqtt, go2rtc. I migrated the Z-Wave and Zigbee USB sticks myself, and my verdict on Home Assistant was “works, all devices there”. The API showed 833 entities. But then ring-mqtt kept using the old box’s MQTT broker after the cutover. Turns out its persisted config.json mqtt_url trumps the env variable, so changing the env var did nothing. Same story with Home Assistant .storage URLs and zwave-js-ui settings.. the saved file always beats the container startup config.

BirdNET-Go had a similar problem, but with hardware: the config pointed to sound card 1 (the mic on the old box), but the new box’s USB mic was card 0. I patched the config and pinned the card order with a modprobe option (because the UPS might power-cycle the box). I migrated 18,035 detections with clips (7.0 GB on the old box, 4.3 GB on ZFS thanks to lz4 compression, not data loss).

The things I dropped instead of migrating were MagicMirror, Termix, guacd, and the old box’s AdGuard Home replica. I retired the AdGuard replica by repointing the resolver first, so nothing depends on a DNS server about to vanish.

Nineteen healthy and a git split

All 19 migrated services are now healthy on the new box. Their old copies are set to restart=no, and I caught one straggler that was still unless-stopped. Every stack’s secrets are checked against 1Password, and none are left only on the old host. The last mistake was in git: a new-box clone pushed only to Forgejo and an old-box clone only to GitHub split the history. I merged them behind safety tags and a full git bundle, then gave both clones both push URLs as in Self-hosted git with Forgejo and why some repos never touch GitHub. The iperf3 server is moved and does 933 Mbit/s on the LAN. The old server is still running, but its copies are stopped, and two unknown LAN clients still have NFS mounts on it.

A clone is never sufficient

The lesson I wrote down at the time: a git clone (or zfs send, or a git-sync) is necessary but never sufficient. What I missed were host-local files under ~/.config/ and persisted app config pointing at the old box. Both failed quietly hours after cutover. Running both in parallel made mistakes cheap: the old box was intact with its data directories untouched, so fixes were config changes, not restores.

The old box is still up and still holds data, but I need to audit what’s on it before I wipe it.