From a cron job to Sentinel: a resident agent that checks my infra

Sentinel started out as a nightly cron job. Back in April, it was a systemd timer that ran at 07:00 every day, executing claude --print on the infra-health skill, which in turn posted the health report. The first night, it failed after zero seconds because the model I had set for the job was its own fallback model, so the CLI refused to start. Fair enough, RTFM moment. I switched the fallback to haiku, and the second night it ran.

The nightly job was just a one way report. If something needed action taken it was a burden for me to handle. I wanted a full devsecops agent I could interact with, and not only health checks, but actions and approvals for fixes or improvements sitting in my queue to review if needed, with severity escalations built in. It made more sense as its own agent in NanoClaw, and the existing scripts ran on it unchanged.

In May I moved it to its own agent group in NanoClaw, a small agent runtime I run at home. Upstream NanoClaw and my fork are on GitHub. It’s been renamed twice now, from Ranger to Warden to Sentinel, and its mandate has expanded from health checks to remediation and deploy validation.

Switching from a nightly report to an always-on agent raised the question of cost. I realized that if I woke the model every 15 minutes just to confirm everything was fine, most tokens would be spent on green runs requiring no judgment.

As a token savings play and for speed and efficiency, I have automated non-LLM checks in bash every 15 minutes, and they only escalate to the LLM when something actually fails. NanoClaw exposes a wakeAgent property as part of scheduled task exit conditions, so that if any automated check fails, the agent can wake (wakeAgent = true) and examine the script output. The model is there for judgment, escalation and action, not for pinging services.

The script is called Infra Pulse. It has 82 checks across 9 categories (containers, DNS, disks, backups etc.). Several of them read existing container healthchecks. NanoClaw runs it before the agent, and the last line of output is JSON. If all pass, that’s all you get.. the task closes without calling the model.

{"wakeAgent": false}

If any check fails, that same line includes wakeAgent: true plus the findings, injected into Sentinel’s prompt. I decided to wake on any single failed check (instead of confirming, then waking) because, as I noted at the time: “Simpler, LLM decides if it’s noise.” Each finding is suppressed from re-alerting for an hour, so a flapping service won’t keep paging me. The pulse writes a last-run metric to VictoriaMetrics, and Grafana pages me if it goes stale.. a dead pulse otherwise wouldn’t make a sound.

Sentinel is read-only.. it observes and proposes. Any finding of medium+ severity that includes a root cause and rollback path triggers the request_approval tool, spawning an Approve/Reject card in the Slack channel. The fix only runs after approval. A short list of critical-only fixes can run unasked, but promotion to that list requires two past incidents and a seven-day trial where the rule only simulates action.

The TV tuner Pi was an exception I made by hand. My ruling was “tell it to reboot the pi if any issues”, so Sentinel got a restart-then-reboot rule for that device only, operator-approved as an exception to the usual two-incident bar.

The first time I ran pulse, Sentinel woke up with 10 findings, all false positives. Probes were looking for services where there weren’t any, or an auth-protected service returning 401 was being counted down. The house was fine, the checks were wrong. Same-day tuning got that down to 1 finding, and now 401/403 are considered alive. A slower variant: a weekly deep scan had been yellow for months, traced to a stale hand-kept list treating sidecars and exporters as user-facing. Correcting the list (not faking healthchecks) fixed it.

Alerts needed the same treatment. All house alerts go through one relay (built on Apprise), and a second Sentinel task reads new alerts through the same script gate, waking only for ones it hadn’t seen. A noise audit a week later showed 134 alerts in 30 days, of which 68 were from three alerts that got stuck firing and re-notifying every six hours. Approved fixes stopped re-paging for already-acknowledged conditions. Sentinel keeps a tuning log in the runbook, snoozes accepted findings, and the snoozes expire on their own.

Sure there have been false positives and stuck alerts paging me, but the agent has a self learning loop that tunes these out based on observations and feedback.

Since I first rolled this out, I have been extremely impressed with what is possible: automated actionable events surfacing when I fat fingered a deployment, a credential changed for a service, a port left open accidentally, etc. The value has been tremendous. It’s like having a 24/7 team on call to fix things, taking a lot of the server management overhead out.

I’m working on better alert triage and onboarding more into pulse scripts as time permits. And a better “allow list” of actions so I don’t have to manually approve everything.

In June, my AI sysadmin and security agents assessed the whole homelab.