Autoheal opt-in, and healthchecks that actually test something

The first sign of trouble was when autoheal kept restarting a container even though the service in the container was fine. It turned out that the image didn’t include curl, so the curl-based healthcheck never succeeded. I fixed that by rewriting the healthcheck to use Python’s urllib instead. A few days later, the Z-Wave controller was restarted twice in the middle of its own soft reset because the healthcheck failed while the controller was busy. I fixed that by loosening the healthcheck to about three minutes.

Autoheal is a small container that watches Docker healthchecks and restarts containers that aren’t healthy, paging me via the Apprise relay to Pushover. I had it running label=all, which put every container with any healthcheck in scope, including hurried healthchecks I’d written myself and healthchecks baked into images I hadn’t read. The scope included the agents’ credential proxy, which holds OAuth grants and shouldn’t be restarted blindly. When I did a best-practices review of the server in June, label=all was the biggest operational red flag: the wrong healthcheck could trigger a restart loop, bouncing an identity-holding service unattended. The review advised me to do both: make autoheal opt-in and audit the depth of healthchecks. Autoheal is now opt-in, and the diagram below shows the bar a check must clear before it gets the label, with the old loop underneath.

The label

The switch was one env var: AUTOHEAL_CONTAINER_LABEL from all to autoheal, so only containers with autoheal=true are watched. I kept the same coverage: the label went on exactly the containers already watched (36 services, 29 compose files, including image-provided checks). Changing the label recreates the container, so I did all 36 labels in one commit and held them until a push window. The credential proxy stayed deliberately unlabeled, so its restart is manual.

  autoheal:
    image: willfarrell/autoheal
    # Watches every container with a HEALTHCHECK; restarts any that go unhealthy.
    environment:
      AUTOHEAL_CONTAINER_LABEL: all
      AUTOHEAL_INTERVAL: 300
      AUTOHEAL_START_PERIOD: 60
      WEBHOOK_URL: http://apprise:<port>/notify/<key>

  autoheal:
    image: willfarrell/autoheal
    # OPT-IN (2026-06-11, was label=all): only restarts unhealthy containers
    # carrying the label autoheal=true. A wrong/shallow HEALTHCHECK on an
    # unlabeled container can no longer cause a restart loop, and
    # identity-holding services are deliberately excluded.
    environment:
      AUTOHEAL_CONTAINER_LABEL: autoheal
      AUTOHEAL_INTERVAL: 300
      AUTOHEAL_START_PERIOD: 60

Once autoheal stopped covering for me, the hidden problem showed up: my two DNS query-log shippers from AdGuard Home and the query-log shipper went unhealthy once per boot. Their old autoheal restarts had coincided with every long boot, always six minutes or so in, and autoheal had fixed it every time, so I’d never seen it. The cause was Dockhand re-owning the entire stacks folder at startup, which flipped the owner of the shippers’ cursor file I’d kept in there. I moved the shippers’ runtime data to the storage pool and made a runbook rule: no runtime data ever inside the stacks tree. How a deploy works with Dockhand explains the folder and why Dockhand considers it its own.

deep checks that reach the real service

The depth audit changed what “healthy” means for some compose file or image checks. For Forgejo, it’s / (proves only the web listener) to /api/healthz (also checks the database and cache), which matters for the git server in Self-hosted git with Forgejo and why some repos never touch GitHub: the listener can answer even if the database is gone. For Home Assistant, it’s / to /api/, accepting either 200 or 401. Containers that lacked checks got new ones: ten total, across the observability stack from Metrics and logs from the network gear and the self-hosted archive app. The archive app’s Postgres now runs pg_isready, and its Valkey cache pings (both PONG and NOAUTH mean alive, both prove a live responding server). The cache’s password stays out of the healthcheck.

  forgejo:
    healthcheck:
      # /api/healthz checks DB + cache backends, not just the listener.
      test: ["CMD-SHELL", "wget -q --spider http://localhost:<port>/api/healthz || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
    labels:
      - "autoheal=true"

    # PONG|NOAUTH both prove a live, responding server; avoids putting the
    # password in the healthcheck (it's only on the command line, not in env).
    healthcheck:
      test: ["CMD-SHELL", "valkey-cli ping 2>&1 | grep -qE 'PONG|NOAUTH' || exit 1"]

Every command was run in a live container with docker exec before committing, which paid off immediately: the archive app’s localhost resolved to ::1 but the app listened on IPv4 only, so the checks now probe 127.0.0.1. Two compose files claimed in comments that they lacked curl or wget, but busybox wget was there.. docker exec beats a comment. Some images have no shell (Tempo, AdGuard’s sync container), so no check, but that’s covered by the target-down alert and Uptime Kuma.

Ship unlabeled, soak, then label

I’ve started deploying new checks without autoheal=true.. they earn the label after a quiet stretch. A wrong check with autoheal=true means an instant restart loop; without it, it just shows unhealthy. This paid off a month later when a Loki 3.4 → 3.7.3 image update broke the exec check (no /bin/sh, no wget), so Loki stayed permanently unhealthy but didn’t loop because it lacked the label. Very satisfying to see a rule I’d written specifically for this case work perfectly the first time it mattered. I removed the check (Loki is covered by the target-down alert and Kuma like other shell-less images), but the discipline held.

A passing check can still be broken

In a late July fleet health audit (a 48-hour sweep of all 78 containers’ logs, fanned out to read-only subagents in a Claude Code session), I found node-exporter logging ~1,230 errors/minute (~1.7 million lines/day into Loki) after an image bump. Container was healthy, Kuma was green. Turned out it was its own healthcheck: wget --spider closes the connection after headers, and version 1.12.1 logs a broken-pipe error for each metric family (~615/check). I’d never thought of a check being green and noisy at the same time. New repo rule: an exit-0 healthcheck can still trash the app’s log.. watch what the probe makes the app write, not just its exit code. A full-body fetch fixed it (errors → zero).

     healthcheck:
-      test: ["CMD-SHELL", "wget -q --spider http://127.0.0.1:<port>/metrics || exit 1"]
+      # Full-body fetch, not --spider: v1.12.1 logs one broken-pipe ERROR per
+      # metric family (~615/check) when the probe closes after headers.
+      test: ["CMD-SHELL", "wget -qO /dev/null http://127.0.0.1:<port>/metrics || exit 1"]

Two days later, the same rule caught Home Assistant: a check that counted a 401 from /api/ as healthy, but the HTTP integration logs every unauthenticated request as a http.ban warning (2,879 of them in 24 hours, plus a standing “Login attempt failed” notification) while the container read healthy the whole time. Both candidate probes were tried live in the container first.. /manifest.json gives a clean 200 without auth. After the recreate, the warning count went to zero.

     healthcheck:
-      test: ["CMD-SHELL", "code=$$(curl -s -o /dev/null -w '%{http_code}' http://localhost:<port>/api/); [ \"$$code\" = \"200\" ] || [ \"$$code\" = \"401\" ]"]
+      test: ["CMD-SHELL", "code=$$(curl -s -o /dev/null -w '%{http_code}' http://localhost:<port>/manifest.json); [ \"$$code\" = \"200\" ]"]
       interval: 30s

The placebo, replaced at the source

The fix I’m proudest of is the Reddit archiver from Stash 3 running my Reddit archive like a real service. The image’s healthcheck was python -c "sys.exit(0)" (always passes), and the run loop kept going after a failed run, so failures were invisible no matter how long they lasted. It’s my own app, so I fixed it there: the app now touches a heartbeat file at boot and after each successful run, and the check fails if the file is older than HEARTBEAT_MAX_AGE. One image, two cadences (via env var): the regular archiver now fails after two missed two-hour cycles, the refresh container after two missed daily cycles plus an hour of slack. Both lost their autoheal labels while the new check soaks.

       - SCHEDULE_INTERVAL=86400
+      # 24h cadence: 2 missed cycles + 1h slack
+      - HEARTBEAT_MAX_AGE=176400
     restart: always
-    labels:
-      - "autoheal=true"
+    # autoheal label removed while the success-gated heartbeat healthcheck
+    # soaks (two-step rule); re-add after ~a week healthy

The smoke test exposed another issue: a run without credentials reported “completed successfully”. The script returned an exit code of 0 on config validation failure. I thought the exit code was honest, but I was wrong. A success-gated check is only as honest as the wrapped exit code. This would have refreshed the heartbeat on failure, turning the check into another placebo. Now the script exits with 1 on config validation failure.

Still soaking

The bar now is a check that tests the real service, is smoke-tested in the live container, soaks unlabeled, and stays quiet in the app’s log. The label never goes on a credential holder. As of this writing, the June batch of soaks, which includes both archiver containers, is still unlabeled, well beyond the week I allotted for them. Removing the label was a single commit, but restoring it requires my return and judgment that the check earned it. Newer services like the Home Assistant MCP server and Matter server follow the same rule: labels deferred.