Michael Zazon: Notes

Is built with AI an insult?

Sun, 01 Nov 2026 00:00:00 GMT

humans shipped bugs for 50 years. the difference now is we get to catch them faster

But “vibe coded” has come to mean a label for anything AI touched, instead of what it originally meant: not looking at the code at all. Simon Willison draws that line clearly, saying that if you reviewed and understood the code written by the LLM, it doesn’t count.

AI is a tool; what you produce with it is your responsibility, whether it is good or bad.

Every company I have worked with over the last few years from startups to the largest banks in the world are building and shipping products that delight their users at record speeds using AI assisted software development and agentic coding.

In my career I have written code, reviewed code, and reviewed and approved and rejected PRs all by hand in the past.

Reviewing and rejecting has always been my job. The agents didn’t take that away. It’s just that now the author of the change isn’t me, but the owner is still the same person it always was: me. (see Who owns the code Beyond Vibe Coding held up against 2026)

And yet most people are doing that.. prompting and praying. That is exactly why there’s such a negative stereotype on ai coding, and the ai hate isn’t irrational because most people don’t know how to use ai correctly.

I only have issue with AI-generated garbage posts that clearly have not been even read or edited by the OP. If they can’t take the time to read and edit what AI spits out, why should we?

It’s difficult to argue against this. A lot of AI-generated content is being published without the author even reading it, and it shows. Unfortunately, this label tends to stick because it’s often well-deserved.

This keeps the agent code mine, because I can put up review gates and keep the interfaces small. Automated reviewers can stare at diffs forever, but they can’t know what the purpose of a change is (see Every PR has a crowd of reviewers now and none of them know why). That’s still my job.

The issue is not the AI usage, its what follows. Lack of understanding, lack of knowledge, lack of competence in the end due not using those skills anymore..

This is where I’d draw the line. I’d judge whether or not someone understood the code and answers for it. It’s the same question as “who gets paged when it breaks?” which is at the core of The AI argument at work is really about who’s on call.

Every PR has a crowd of reviewers now and none of them know why

Sat, 31 Oct 2026 00:00:00 GMT

In my career I have written code, reviewed code, and reviewed and approved and rejected PRs all by hand in the past. Now, when you make a PR on open source repositories you have all kinds of automated feedback hitting from CodeRabbit, GitHub Copilot, Claude, many others with system diagrams, analysis, breaking change analysis, security analysis all on top of the deterministic checks that we’ve all become accustomed to.

You can have many reviewers on a single pull request. CodeRabbit and GitHub Copilot code review can both comment before you even look at the diff as a human. Most of the feedback from the bots is good.

The issue with agents is that they do not know the weight of things. They can’t contextualize how severe an issue is.

This is where the human comes in. Severity is the human reviewer’s contribution.. the bot will flag a missing null check and a broken billing path in identical tone. A human still needs to decide which ones actually block the merge.

Thats just reviewing the code. I agree that this works well, but it also doesnt really scale, its a very long process. Everything that you will question can open a whole discussion, and before you realize you end up spending hours to review code that already works but is just not perfect.

Frankly, if your agent is doing 100k lines you are working on too large of an increments.

Before you even open the PR.. keep your changes small.

It doesn’t have to be like that if you had a ticketing system (GitHub issue?) that acts as spec or PRD

A review bot without a ticket can only review the code that’s given, it can’t tell if the change is the one that was requested. My agents operate under review gates for the same reason (see Running an agent fleet like a small team). They propose, I rule.

What I know for certain is that LLMs are infinite text spew machines. If you give them an excuse to spew more text they will.

Having an extra review bot on every PR just gives you more text to read and there’s no signal to tell you which comment is important. The job I used to do manually was read the lines. Now I get to provide the intent and decide what’s important. It still helps to write down your review rules, as I argued in Looks Good to Me and the case for writing down review rules.

Most agent harnesses rot because nobody deletes anything

Fri, 30 Oct 2026 00:00:00 GMT

A 6x gap from the harness alone explains why everyone’s “same model” gets such different results..

People tend to focus on the model when making comparisons, and rarely talk about the harness they’re using.

The “mine past corrections into rules” bit is right but the hard part is pruning, not writing. Rules accumulate forever if nobody checks whether they still fire.

If you can’t point to a specific save, cut it. Most harnesses rot because people only ever add.

My carry blocks already do this for open items. Each one ends in one of three ways: it’s done, it’s dropped with a reason, or it’s promoted to the backlog. Rules need the same exit. A rule without an exit keeps occupying your context window.

how do you define best? What metrics are you using: instruction adherence, task completion, context usage, or something else? A smaller kernel may win on efficiency, but does it do what I care about.

If a shorter file drops the single rule that prevents a bad deploy, it scores better on context usage but loses what I wanted. I want fewer rules, each one justified. Vercel has a course on building a harness.

It really shows how much everything around the model matters. IMO it also quietly debunks the “AI will replace devs” narrative, because that stat shows just how much the person piloting the agent makes a difference.

Figuring out what to discard is part of the piloting. Next up, I want to put the rules and guardrails I’ve gathered from community threads and best practices guides to the same test. For now, I’m adding Marmelab’s write-up to my reading list.

When a bad certificate chain meets agents that work this fast

Thu, 29 Oct 2026 00:00:00 GMT

Bold to assume a super budget esp32 IoT device would check for revoked certificates. These companies crank these things out very cheaply and move on to the next.. software updates and security patches are far and few between.

I’ve been stuck on this reaction ever since the October eMudhra advisory, which revealed that they’d issued 8 intermediate certificates with weak keys that anyone can use to forge a certificate for any domain. There’s a test page that shows whether your client still trusts them.

Most IoT devices wouldn’t really be able to handle this case anyway.. virtually none check CRLs or OCSP, and they don’t bundle intermediate CAs to mark them as invalid

The CAs are baked into the firmware pretty much all the time..

The trust store on the device is what it shipped with. It can only be changed by a firmware update from the vendor. Actions the device owner takes on their network can’t change what certificates the device believes in.

A lot of embedded devices (e.g. ESP32) only include the certificate chains for the servers they contact, so this won’t be useful in those cases. Anything running Linux will probably be using ca-certificates though, which will have it.

This was true for the host, which had already been patched by unattended upgrades, but I’d assumed it covered more than it did: most containers ship with their own CA store from the day the image was built. How many images were running around with a frozen copy of that list? I switched to a single shared read-only CA bundle, refreshed from the host’s, mounted into containers. The agent did the inventory and edits, and I approved each batch before it rolled out.. the same ask-before-fixing habit I’d gotten used to with Sentinel. Node’s system-CA flag was a no-go because it only adds trust, it can’t remove a bad root.

No need to chuck tokens at it, just use mitmproxy

Most of the discussion seems to be about people scanning their own devices for fun, which is cheap these days whether you use agents or not. But that speed is available to anyone, and Anthropic reported an AI-orchestrated espionage campaign. In the past, such flaws would have waited until someone bothered to exploit them. Now, with near-zero cost to scan and try, the gap is human-speed patching. That’s why I used agents defensively, and the shared bundle was rolled out within days of the advisory.

So I made two decisions. First, I made my containers’ trust stores host-owned instead of frozen to each image at build time. Second, I left the agent on the inventory so that next time there’s an advisory, it’ll be a batch job for me to approve.

Fold the subagents back in, keep the second opinion

Wed, 28 Oct 2026 00:00:00 GMT

They’ve been singing that song for a while. Cut down your CLAUDE.md. You don’t need skills. Use fewer subagents. I’m not saying I don’t believe them, but cherry picking examples to “let our big model handle everything” is also a self-interested thing to say.

It’s fair to be skeptical when a vendor of a large model encourages you to rely on their large model. But it’s worth reading Anthropic’s guide on folding subagents into the main loop.

I dont understand how a single main agent and ever increasing context window can be better than doing granular implementation tasks in a subagent with fresh context.

The real benefit of a subagent is fresh context. The guide reserves subagents for measured context bloat and disposable parallel jobs. In all other cases, a clean slate is paid for with a handoff.

The moment it rewords intent or leaves out an important detail, the subagent has absolutely no idea about this (how would it?) and does the (ever so slightly) wrong thing.

That cost is exactly why my own division is between sessions. I have one lead session that maintains the ledger and checks the work. Every project has its own peer session, so all intent lives in a single place.

And review has to come from a different model in a different harness, no matter how lean the rest gets.

Having different models from different providers review and collaborate with your main agent gives massively better outcomes. It is slower but the trade off is better and more researched results

I’d fold the helpers into skills and keep the review as a separate model with its own context. It’s in the same spirit as the review gates I have in my agent fleet. Next up I want to try Anthropic’s single-agent loop migration kit, starting with the evals as the guide suggests.

My agents can message my coding sessions now, and back

Tue, 27 Oct 2026 00:00:00 GMT

There was a point where work repeatedly stalled because of.. me. The only way for the host session to get a message to my agent was for me to post a message in Slack from my account, which the agent couldn’t tell from me actually typing it. And there was no way at all for the agent to get a message to the session. So I was copying messages by hand between them, and anything that needed both sides to work was waiting on me. I realized I was the integration layer between two pieces of software that I run.

I wanted to do this in a more Claude Code native way. Claude Code already has a list of agents that sessions can see and message. So how could I get my agents on that list? Also, if the peer is on that list, I don’t run into the Slack problem where all the messages appear to be from me.

Agents on the peer list

This way, the agents show up as regular peers in Claude Code’s own list of agents. The session messages the agent the same way it would message another session, and the agent responds. The session doesn’t need to know much about it at all.. no containers, no Slack, just a name on a list (which took me a moment to appreciate). Under the hood, the NanoClaw host starts one small stand-in process for each allowed agent and registers it as a peer using the agent’s name. The agents themselves reply through the host-messaging tool. (You can read more about NanoClaw and why I run it in the background in Why I dropped OpenClaw for a smaller agent runtime.)

Who sent this?

The Slack experience taught me that the key question on arrival is: who actually sent this? I realized the claimed name is worthless.. a message can assert any name it wants. So I grab the sender from the session registry. Containers can’t touch the registry, so an agent can’t pretend to be another one. Now every message includes its true sender, something my old Slack posts never managed.

Note · Don’t trust the claimed name

A message can assert any name it wants. I take the sender from the session registry. Containers can’t touch the registry.

I wanted proof, so I made my work pass some acceptance gates, one of which was a spoof test. It passed: the sender stayed the real sender. I also capped it at 30 messages an hour per agent, so no one agent can spam my sessions with traffic.

It works while the agent is asleep

My favorite thing is what happens when I stop the agent. Since these agents run in containers, they aren’t always up. So if an agent is stopped, is the peer any good? I have a stand-in that lives on the host, outside the container, so even if the agent is stopped, it still gets the message. I tested it, and it answered in about 16 seconds. The session doesn’t know the difference between an agent that’s sleeping and any other peer on the list.

Sessions and agents, no middleman

This is a real game changer because now sessions and agents can coordinate without me in the middle. A session can ask an agent directly, and the agent can come back to the session on its own, so I’m not holding up work waiting for me to carry it. I’ve already rolled this out to five agents, including Otto and Sentinel.

This fits right into how I already work.. I’m used to running several Claude Code sessions at a time, and I run my agents like a small team, where they propose and I rule. The sessions and agents were already on either side of me, so taking the hand-copying out of the loop keeps all the decisions with me but finally gets rid of my job as the human text courier. And I get it done with a sender I can actually trust every time.

Next up.. more agents on the list.

A Philosophy of Software Design and its sharpest critics

Mon, 26 Oct 2026 00:00:00 GMT

Deep modules is an idea I keep thinking about.. a small interface with the hard stuff hidden behind it. It’s the one I see most in my own personal work and in my professional line of work. Strategic vs tactical programming as well.

The book is John Ousterhout’s A Philosophy of Software Design, which has been out in its 2nd edition since 2021. The sharpest counterpoints I’ve seen to it are in his written debate with Robert C. Martin, the Clean Code author.. that debate got 542 comments on Hacker News. Deep modules are already all over the place in the tools I rely on, as this thread on r/ExperiencedDevs points out. The file system, for example, and mutexes.. just a few simple calls, lots hidden behind. The top comment there says the book’s popularity is a swing back away from Clean Code.

The deep module idea

In chapter 4, Ousterhout argues that the best modules provide lots of functionality through a simple interface, and that creating many small classes (“classitis”) adds complexity. Eric Normand objects to this idea, saying “Module depth is an attempt to rationalize an intuition. However, it is a dangerous idea.” The most common objection in this r/softwarearchitecture thread is that the examples in the book are small, contained projects, while real systems sprawl across many. That same thread also tries to reconcile the two books by saying Ousterhout is talking about modules and Martin is talking about classes, and a deep module can be made of small single-purpose pieces. In practice, I’ve been designing skills and services with a simple front and lots of complexity behind because agents write a lot of the code, so keeping the interface small makes it easier to review. This idea is packaged up for agents by Matt Pocock in this skills repo.

Tactical tornadoes

In chapter 3, Ousterhout says that it’s not enough to just write working code.. you should spend some time thinking about the design. He warns about the “tactical tornado”, where you ship features quickly and other people have to clean up your mess. Yevgeniy Brikman counters with “the book doesn’t seem to acknowledge there are contexts when tactics are a better option”, like when you’re doing quick experiments and the code will be thrown away in a few weeks or months. Coding agents make this question even more interesting.. olano.dev argues that tactical tornado is the new default when more work is delegated to agents.

The comments debate

Chapters 12 to 16 discuss comments and how they can capture the mind of the designer and things that code can’t say. Section 16.3 says comments belong in code, because a developer “is unlikely to think of scanning the repository log”. Martin’s response in the debate was “Comments are always failures.” But since then, coding agents have introduced many comments themselves. An issue asking Claude Code to stop over-commenting has 252 reactions and notes that a CLAUDE.md rule “does not reliably suppress it”. And a study of LLM-written comments found that experts rated most of them as good as or better than the originals.

So for comments: in my repos the commit history, ADRs and a CLAUDE.md carry the why more than inline comments do. Agentic coding models tend to lean hard on comments, so we are seeing an uprising in them now. For better or worse.

Designing my home app by living with it

Sun, 25 Oct 2026 00:00:00 GMT

I decided on the look of the app using mockups at phone width and real numbers. I compared the more subdued style I already had in the first version of this app from A home super app for the house and the homelab against a brighter “Living colour”. I decided to go with Living colour for the whole app, light mode first (as that’s what I’ll use most) and dark mode following the phone.

After the first pass, I had content spilling to the edges, so I asked Codex to generate some reference mockups to polish the design. When they came back, I knew that was the look I wanted. The rules I came up with after reviewing them were to have a 44 px touch target for every chip, in line with Apple’s layout guidelines.

I decided on the controls in a similar way: code the options, try them at phone width, no spec. For the kids’ rooms dimmer, I came up with three coded mockups. I decided to keep mockup A, which has a single slider (0 is off) with minus/plus buttons either side in 5% increments.

The real test was that Saturday when I had the app on my phone all day and sent feedback in rounds and shipped fixes between lunch and early evening. Early complaints were about the Sonos page having a “weird look” and the setpoint for one room’s mini-splits “never works”. Why would tapping the plus do nothing? Home Assistant reports a half-degree setpoint rounded to the even number, so 75.5 shows as 76. Tapping to increase in half-degree steps from the shown value was landing on another half degree that looked unchanged. I fixed that by making the climate tiles step whole degrees.

Scrolling was fidgety too: the folding header resized the page mid-gesture and on short pages kept folding/unfolding at the bottom. Now the big weather block scrolls off with the page (iOS large title style) and sheets close by pulling down. A separate bug in the deploy step that filled new tabs (see One WebSocket and a tile registry behind my home app) resurrected a deleted tile. It’s gone from the seed now and shortcuts never go in it.

Next up is the Life tab in the same app. It starts with birds, what’s on Plex, the photo frame. Also need to teach the seed merge to remember deletions.

One WebSocket and a tile registry behind my home app

Sat, 24 Oct 2026 00:00:00 GMT

Halfway through the design of the app from A home super app for the house and the homelab, I came up with a new requirement. I wanted to be able to rearrange and reconfigure the tiles on my phone without having to change any code or start a coding session just to move a tile. I already had the “everything possible, whittle later” rule for scope, but I kept coming back to the question of how I’d get the 40th tile in there without anyone touching any code. The shape had to make it cheap to add a tile.

I compared a thin proxy and a server pre-rendered page against a third shape.. server adapters feeding a tile registry, the layout living as a document on the server. I went with the registry because it was the only one where adding a tile and rearranging the grid both stayed cheap. The backend holds all the tokens, so the phone never holds a credential. I didn’t want the page to call Home Assistant directly because then the token would sit in the browser. The non-Home Assistant tiles needed a server anyway. It’s a small Fastify backend serving a Preact app.

The backend keeps a single Home Assistant WebSocket open and subscribes only to the entities the current layout needs (out of 1,555 in the house at the time). The homelab sources are polled with the same adapter contract. The phone only sends the tile id and action, and live tile updates come back over server-sent events. House rules are baked into the code: locks can be locked from the app, but never unlocked. The water main is display-only (it already closes itself on a leak), and the smoke/CO alarms and boiler are untouchable.

Edit mode on the phone does what I wanted: a long press lets me drag or resize a tile, and I can pull new ones from a catalog. The registry-backed build grew from a handful of live tiles to dozens within a day. Every new kind of tile is just one extra entry in the catalog. The Infra tiles read from VictoriaMetrics as the first metrics database on my home server, the Sites tiles from Ten years of site analytics in Loki, and they use the same tile contract as the Home Assistant-sourced tiles.

Part 3 is about the design side of the app.. the coded mockups I decided to keep, and a full day of releases that came from using the finished app on the phone.

First on wins: a coordinator for our Mitsubishi heads

Fri, 23 Oct 2026 00:00:00 GMT

Every spring and fall somebody’s room would stop heating because another head on that compressor was cooling, and you’d only find out by walking around. I have nine Mitsubishi heads on two outdoor units (five on East, four on West) and they’re always fighting. This fall I wanted Home Assistant to be able to resolve that, prioritizing bedrooms at night.

This is because of the “first on wins” behavior I talked about in part 1 (along with the heads’ CN105 firmware). The first head to be powered on determines the heat/cool mode for all heads on that outdoor unit. The MXZ multi-zone manual says “the unit selected last goes into standby mode.” This is especially bad in the fall when you might want heat in one room and cool in another on the same day.

The only HACS project I could find was ha-mxz-coordinator, which I studied closely before building my own. It requires exclusive write access to the heads, has no time-of-day awareness (so no bedroom night priority), and parks idle heads in fan mode. Mitsubishi considers fan to be cooling, so a parked head can prevent a room from getting heat. I decided to borrow some ideas but not install it, as Mitsubishi’s own Kumo Cloud changeover groups were a better spec to copy.

Version one ranked rooms, and the protected sleeping room kept its unit’s mode even if it was already comfortable. Its design review called that the first-on-wins pattern we fight every day, now on purpose. Shadow mode (log what I’d do, don’t touch the heads) caught a real conflict within two days: one morning the Master Bedroom asked to cool and idled at 75 to 78 °F for half an hour while a kid’s room held the East unit in heat. It answered the oldest question.. “Will a blocked head ever signal that it’s waiting?”.. with “No, it just idles, nothing in the data says it’s waiting.”

I did the redesign with AI agents on this build, but I approved the new spec before any run. Each room/zone has one thermostat, holds to ±2 °F of its setpoint, and the coordinator picks heat/cool per head so outdoor units never fight. A room in range claims nothing, so a comfortable room no longer ties up the compressor. The band is ±2 because a head will overshoot its setpoint by about 1.5 °F on its own.

On the first shadow night of the new version, at about 3 a.m., the living room was at 64 °F and wanting heat, while the dining room was at 74 and wanting cool, and they were on different outdoor units. I made First Floor a single zone, so its average temperature picks heat/cool. The Master Suite spans both units: the bedroom head is on East, the closet head on West. The closet head is a backup on the other compressor.. it engages if the bedroom head is parked or the room lags far behind, and kicks in sooner at night.

Manual override is always possible: if you select Heat or Cool on the remote or app, that choice wins for 2 hours; same if you make the change on a head itself. After that, the room reverts to Auto. The W100s from part 3 control the room thermostats, not the heads.

I brought both units online on October 10, before the W100 buttons started working again after they went deaf. That was intentional: the room thermostats and the app are the real controls, the buttons are a convenience and could catch up. Since then, we haven’t had to override anything on a head, and nobody needed to bypass the room thermostats.

Nobody walks around turning heads off anymore. We set a number per room and leave it at that. The Climate tab in our home app writes the same room thermostats. Next up is adding the boiler and radiators as backup heat for the cold months.

A page on my site for your agent

Thu, 22 Oct 2026 00:00:00 GMT

More and more people discover things through an AI agent, less and less a search box. Even Google applies AI results on top of standard web search now. If my notes are easy for an agent to read and cite, they get read and cited. This is largely an experiment at this stage, but knowing for a fact that my site is crawled by agents, this seemed like a no-brainer to wire up and build.

So I merged all my writing into one stream: Notes (a simple human word for navigation). The subtitle is now “What I’m building, reading and noticing, for you and your agent.” The original reasons I built this site are still in Why I built this site.

An agent visiting will find /llms.txt explaining who I am, when it’s worth using this site, and then the 20 newest notes. The /agents page has a context block to copy and paste into an agent, and there’s a “For your agent” link in the footer. Every note has a Markdown copy at its own URL plus .md. All generated when the site is built (see How this site is built). My favorite line in llms.txt: “If a topic isn’t covered, say so rather than extrapolating.”

In robots.txt, I welcome the big AI crawlers by name, then every other crawler. I end with a Content-Signal line (contentsignals.org) that permits every use, including ai-train=yes (model training). Plenty of sites let answer engines in and block the training-only crawlers. I let both in.

I am ok with learning crawlers on this site. I leverage this technology heavily and I hope that putting my knowledge out there is helpful to others in any way possible.

Next up is link drops with my comments from my saves (the collector is in Stash automating Reddit saved content into my personal knowledge base) so an agent can find more than just long posts. I think search and a query API can come later when I have enough content to justify building it.

Our Aqara W100s went deaf exactly 24 hours after pairing

Wed, 21 Oct 2026 00:00:00 GMT

The +/- on all four doorway W100s only blinked the setpoint, nothing made it to Home Assistant. This started the night A doorway thermostat for our Mitsubishi heads went live, when I paired two more units and rewired them all to drive the Climate Coordinator’s room thermostats. The units are on ZHA via W100 device support. HA could still write to the screens, but the displays were deaf to their own buttons for ten days or so.

It was a great example of patient debugging and using agentic probes and monitoring for this purpose: verbose logging and captures, analysis and trial and error. Since we had multiple devices, we were able to set a few canaries to see what the root cause was.

We had a few theories early on, each of which seemed plausible for a day: the two days of Auto pushes; the HA restarts; the W100s live for about 24 hours after the last press. We tested them by varying one thing per unit, from re-sending the thermostat-enable frame every six hours (re-enable unit) to doing nothing (control unit). All 3 died 24h after joining +/- 2 seconds, regardless of what we did to them. So the clock starts at pairing, and has nothing to do with the buttons.

The captures showed exactly a 24-hour dropoff by the units. We saw, after enabling detailed captures on the ZHA side, that the devices were asking for a descriptor at that exact second. Then an analysis of the library’s response of Lumi showed what we were looking for all along.

bellows (the radio library) reports the coordinator’s manufacturer as Lumi to the Aqara device for 180 seconds after joining, then switches back to Silicon Labs. The W100 re-queries a day later, sees Silicon Labs, assumes no Aqara hub, and drops thermostat features. Zigbee2MQTT does the same switch during Aqara joins, because Lumi devices otherwise lose features. This explains why resets/restarts never helped.. the coordinator’s claimed identity never changed.

I wrote a little custom HA component (in my config repo) to permanently keep the coordinator pretending to be Lumi, since the W100s do this check every day. The ones that were already wedged didn’t self-recover, so I did a 10-press reset on each. A few days later all four had passed the daily check and were responding again. I posted the cause and fix in my threads on r/homeassistant and r/Aqara, where I’d asked if anyone had ever seen the W100 go quiet like this when paired to an Aqara hub.

Next is reporting the 180s revert upstream. Now that we have figured this out, we will likely add a couple more W100 units to the house since they are awesome devices. They do exactly what we were looking for, which was replacing the OEM remotes and having bidirectional communication with HA.

A home super app for the house and the homelab

Mon, 19 Oct 2026 00:00:00 GMT

Speed and endless design possibilities were the main driver for this. HA allowing communication on the WebSocket opened up a slew of possibilities, coupled with design capabilities of Claude models and GPT-6. We also wanted to integrate some data and controls that don’t live in HA.

The result is Maison, a web app launched from the iPhone Home Screen that gives me an overview of both the house and the homelab at the same time. It doesn’t use Lovelace, but is instead built directly on the Home Assistant WebSocket API. The original working name was “mote”, which is my son’s word for remote. I renamed it after a family iPhone dashboard called Maison on r/homeassistant (its code), whose Lights tab is “just a remote”.

So we did some analysis on the button presses in Lovelace to see what we would need to bring into this build, and as expected it was largely climate related. So shortcuts and one-tap points were key to prioritize. Lights came in second, along with some quick-fix buttons for some TV shortcuts that we use.

Before designing any control, I pulled 26 days of Home Assistant logbook entries through its REST API and filtered them for entries containing a person’s name. Climate control accounts for about three quarters of the 637 manual actions we took in Home Assistant. (Notably, light changes made from the Hue app or a wall switch make it into the logbook without a person attached, so this count misses most of our light usage.)

This insight led to a one-tap setpoint + mode (Auto, Heat, Cool, Off) on each room tile. No sheet to open, just an Undo toast if you tapped the wrong one. I shipped it the same night, as v0.3.0. The tiles write to the room, never to a single head, so the room logic from Nine Mitsubishi mini-splits in Home Assistant for 15 per head still decides what each head does.

Other data sources like Cloudflare usage, aviary and airspace became nice-to-haves. I was really looking for a combination dashboard and home control that could be a home super app of sorts.

Three days in, I have six tabs (Maison, Lights, Sonos, Climate, Infra, Sites). Sites uses Ten years of site analytics in Loki and Infra uses VictoriaMetrics as the first metrics database on my home server. Hiding a tab is per-phone, so Infra stays on mine. Sites exists because once I had the house tiles working, I wanted to know “where are my site visitor statistics? graphs, charts, visuals?”

In Part 2, I’ll show you how I built it. It’s a single Home Assistant WebSocket feeding a little backend, a tile registry, and a layout I can rejig from the phone. Part 3 is about designing through iteration: the mockups I coded and the one that survived, plus a day of using it on the phone and shipping fixes as I went.

Keeping my site analytics in Loki for the long haul

Sun, 18 Oct 2026 00:00:00 GMT

I wanted to see visitors to all my sites in one place: hits, IPs, locations, pages, RUM/user behaviour. I started by pulling from Cloudflare’s edge. The edge showed me things my own server never logged: a home page request from China answered by the edge, a handful of scanners probing for /.env. Those requests never made it to the origin, so they never showed up in the access log. I’d been using the origin log as if it were the whole picture. That gap is why this project exists.

The second thing I learned is that Cloudflare forgets. I measured it in one-hour probe windows: edge data goes back 31 days, Web Analytics about six months. So I decided to keep history: a copy in Loki persisted for ten years via Loki’s per-stream retention. At about 1 MB a day, a decade costs almost nothing.

The result is a single small Python worker that every hour queries the Cloudflare GraphQL Analytics API for edge requests to each domain and Web Analytics page views + vitals for each site, then writes one JSON line per result row to Loki. A Grafana dashboard named “Sites” reads those lines. The sites include michael.zazon.us (discussed in How this site is built) and the bird site. If the worker ever gets more than three hours behind, it’ll warn me (analytics never pages me).

I chose to have Loki store the data because paths and referrers have a vastly larger number of distinct values than what would make sense in a metrics database like VictoriaMetrics as the first metrics database on my home server. I also chose to give the worker its own read-only Cloudflare token instead of using the bird site’s token. I didn’t want to have no worker at all and instead make live Grafana queries to Cloudflare because that would only show me what Cloudflare still remembers, and the token would be inside Grafana.

Building this was just an evening’s work, from first idea to working release. The edge polling used to happen inside the old poller in the bird site, so the new worker needed to take over that job losslessly. The handover went smoothly: for each domain, the old poller’s last hour was immediately followed by the new worker’s next hour, with no gap or duplicates.

The first version of the dashboard didn’t work because the tables collapsed to a single row and the top-pages panel hit Loki’s series limit. To my curiosity, this was mostly because of unwanted requesters: one day of edge data contained about 2,700 distinct host/path pairs, mostly scanners looking for files. A dashboard change and raising one Loki limit fixed that.

  # site-analytics' Sites dashboard ranks edge paths with topk(); Loki needs every series
  # first, and a day of Cloudflare edge data has ~2,700 distinct host/path pairs (mostly
  # scanner probes). Default 500. 2026-09-25.
  max_query_series: 5000

The most instructive problems were the quiet ones: these weren’t returning errors, they were just returning.. nothing. First, I found one by manually testing queries: I’d create a Web Analytics query window longer than seven days and get.. nothing. An eight-day window returned zero rows, but a seven-day window over the same data returned rows. If I’d trusted that empty result, I would have wrongly concluded that I’d had no visitors.

Tip

Don’t judge “no Web Analytics data” from a query window wider than seven days.. Cloudflare returns zero rows, no error.

The second quiet failure was more subtle: my third site wasn’t recording anything. It turned out that the redirect rule I’d set up for that domain was also matching /cdn-cgi/, where Cloudflare’s auto-injected beacon sends its reports, so they were all being redirected away without error. Once I excluded /cdn-cgi/ from that rule, recording started. I also discovered that one of my zones was injecting a second beacon per page, so I switched to only the hand-installed snippet.

Tip

If a redirect rule covers a whole domain, keep /cdn-cgi/ out of it.. the injected Web Analytics beacon sends its reports there and you’ll lose every one.

The last quiet failure is happening at home. My household isn’t represented in Web Analytics because my home DNS blocker resolves the beacon’s reporting domain to nothing, and my router forces all devices through it (Forcing every DNS query through AdGuard and the LG TV). I hadn’t realised that it was blocking the beacon I’d added myself, so now I’m testing from my phone on cellular.

Tip · Your DNS blocker hides you

My home DNS blocker resolves the reporting domain of the beacon to nothing, so I tested this from a phone on cellular.

I still need to add the city and network lookup for each request that I left out of the first version, but in the meantime more history is piling up in Loki every hour.

The always-on Mac

Sat, 17 Oct 2026 00:00:00 GMT

I have always liked old laptops as homelab servers: the battery, screen and keyboard are built in. And you can usually find them pretty cheap on the second hand market. I like to be cost conscious with any home lab purchases, try to minimize ewaste, use what I have regardless of age, etc.

This time it’s a 2021 MacBook Pro with the M1 Pro and 32 GB memory. It replaces my 2017 Intel MacBook, which was stuck on a version of macOS that Apple stopped supporting. I manage my dotfiles with chezmoi, just like on my other machines: chezmoi one source of truth for every dotfile. An always-on old M1 Pro with 32 GB runs (small/quantized) local LLMs, embedding models, including models I fine-tuned, backs up iCloud, and is the standby in my DR plan if the main server goes down. It’s a great little machine.

The iCloud “pump”

The iCloud part only works because it’s a Mac, so that was the job that justified it. The only way to get at the originals in Photos is through the local Photos library on a Mac.. there’s no server-side API. So the Mac is the pump, not the store: it downloads and exports the originals, then the main server keeps a copy and controls the schedule. For the whole pipeline story, see Photos iCloud is a sync service not a backup.

The first export run processed 60k+ photos, came to 550 GB, took 5 hours, and encountered 0 errors. I assumed all of my photos would already be on a machine I owned, but my daily laptop was using Optimize Storage and only held 13 GB of originals. So before the first run completed, it was true that no machine in the house had every original. This strategy helped close the gap.

Warning

Optimize Storage keeps only small versions of most photos on a Mac. A laptop with it turned on is not a backup of your library.

28 tokens a second from a 26B model - decent

The biggest surprise was how well an old laptop can run a 26B model at close to 28 tokens a second fully on the GPU. I did a little bake off on my Mac using Ollama and three models: Gemma 4 26B-A4B at 27.6 tokens/s decode, Qwen3.8-27B at 7.3, and Muse Glimmer 30B at 6.6.

The difference is in the model design. Gemma is a mixture-of-experts model, so only ~3.8 billion parameters are active for each token, while the two dense models use all their parameters for every token on the M1 Pro, making them compute-bound. I noted previously that this was because Qwen’s new architecture didn’t have a Metal path, but Muse uses conventional attention and is equally slow, so I went back and questioned other assumptions. A 32k prompt takes about 70 seconds on Gemma and about 13 minutes on either of the dense models.

Only macOS or Metal

As a rule, I only run jobs on the Mac that require macOS or Metal. Docker on macOS runs in a VM and doesn’t support USB passthrough, so Home Assistant will continue to run on the main server with the Z-Wave and Zigbee radios. BirdNET will continue to run there, too. Writing down this rule is one of A cloud architect’s habits applied at home and helps keep the two machines from drifting into each other’s jobs.

The same thinking informed my choice of storage: my Photos library lives on a 1 TB SD card in the Mac’s built-in card reader, not on a USB-C SSD (which would be about ten times faster) because this is an always-on machine and nothing should stick out to get knocked loose. If the card fails, I can always re-download from iCloud, so no data will be lost. The copy on the main server is the canonical one.

Lid closed, always on

The Mac silicon is so efficient and sips power too. So it’s easy to leave on. Keeping it awake with the lid closed takes one pmset command, and I applied it before anything else because I was away from home during setup and keeping the Mac reachable mattered more than any other step.

pmset -a sleep 0 disablesleep 1 displaysleep 0 hibernatemode 0 standby 0 autopoweroff 0 autorestart 1 womp 1 powernap 0 tcpkeepalive 1

Tip

On Apple silicon, disablesleep is all you need to keep a closed laptop awake.. no nvram tweak.

If I lose the main server

If I lose my main server to fire, theft, or a dead board.. the plan is critical services back on the Mac within hours. It’d run under OrbStack with plain docker compose and an override file for each stack. Secrets come from 1Password, which the Mac can already read. I’ve only drilled this once (cloning the infra repo from the GitHub mirror when the self-hosted git server was unreachable), and it took 5 seconds. I still need to schedule more drills.

The role rule was written for the Mac’s first jobs, and where I’m heading now stretches it. Focusing on what’s possible next: proving that an always on Mac may be the ideal solution long term for my Linux containers, services, and home server needs. Local LLM inference is a huge benefit to learn data strategy, LLM fine tuning and LLM ops. I may start to migrate services to the machine from my main server to spread the load around a bit.

Your org has a strategy. Crafting Engineering Strategy says write it

Fri, 16 Oct 2026 00:00:00 GMT

Every large engineering organization already has a strategy implicit in the projects they fund and exceptions they approve. It’s just usually held by a few senior people and leaves with them. So I’m not saying orgs don’t have strategy, I’m saying they don’t have a written strategy. In 2026, the absence of that document is where a lot of AI and cloud spend goes unmanaged.

Will Larson’s book Crafting Engineering Strategy (O’Reilly, October 2025) defines strategy as “the art of reproducibly making good decisions.” If you don’t take anything else away from this book, you should at least start documenting the decisions you’re already making in terms of diagnosis, policy, and enforcing operations. I’m looking at this from the perspective of working with large financial institutions and Fortune 500 companies over the last ten years, now as a principal cloud architect.

Larson breaks this down into five steps: explore the problem and options, diagnose the real situation honestly, refine the approach using tools like systems models and Wardley maps, set policy, and execute operations that implement that policy. He notes that strategies fail more often from skipping a step than from flawed thinking.

I think big enterprises typically skip the last two steps (policy and operations). If you have a strategy that doesn’t change any decision and doesn’t have a mechanism (review, exception process) associated with it, it becomes a slide in a deck that isn’t consulted when the next real decision needs to be made. As Larson wrote in a previous essay, good strategy is boring. It is built bottom-up from existing design decisions, not dreamed up in an offsite. But for diagnosis to happen, you need the raw material, and the maps from The Staff Engineer’s Path provide that (see If you keep one thing from The Staff Engineers Path keep the maps).

This applies to 2026 AI investment. A study from MIT last year, reported by Virtualization Review, found that about 95% of organizations saw no measurable return from generative AI investments (although the methodology is questioned). Gartner expects over 40% of agentic AI projects to be canceled by the end of 2027.

I read this as AI spending without written policy, so a portfolio of pilots with no rule about which to stop. I keep asking what a stop rule would look like in most programs. The book has a useful answer: the LLM-adoption case study narrows the policy to a single concrete use case rather than a broad mandate. That’s the first move I’d copy.

Platform consolidation and cloud cost are in the same category. They’re policy questions, not migration programs. Policy sets the default path, operations decides who grants exceptions and who reviews them. In a regulated enterprise, strategy has to be legible to finance and risk, not just engineers, or it won’t survive the first time someone asks questions about the budget.

The strongest objection is that strategy documents get written to be written, teams ignore them, and new leaders use them to build empires. I think this describes a strategy that lacks both diagnosis and operations. The cure for both is a boring written strategy that’s rooted in decisions people recognize so you can point to a line and say “yeah, I disagree with that.”

As an engineering leader, pick one live decision (like a list of permitted AI tools, or a platform default), write down on one page the diagnosis and the policy, and then name the mechanism that enforces it (a design review, or a standing exception process with an owner).

Vibe Coding's head chef ages better than its tools

Thu, 15 Oct 2026 00:00:00 GMT

Imagine you’re the head chef in a busy professional kitchen. You have a brigade of sous chefs working for you, and they’re faster than any human cook. A customer sends back a plate of overdone fish. That plate is your plate. You can’t blame the sous chef for what comes out of the kitchen.

This is the central image of Gene Kim and Steve Yegge’s Vibe Coding. It’s an idea that’s really stuck with me.

In this kitchen, you’re doing a lot less personal chopping. Instead, you’re running the line: managing multiple cooks at once, tasting every plate before it goes out, making sure they all meet the same standard so the dishes work together. Kim and Yegge describe this as the work that used to fall to a team lead, but now rests on one person and a few agents. (Mike Hadlow points out it works best if you know how a real brigade is organized.) The difference from a real kitchen is that these cooks never clock out and never stop asking for the next ticket. So it’s your attention that runs out first.

This is already how I work. In Running an agent fleet like a small team, I described how I give each agent a role and run every change through a review gate before landing it. The agents propose, I decide.. that’s the head chef’s job at small scale. Everything that comes out of my kitchen is still mine, and I talked about ownership in Who owns the code Beyond Vibe Coding held up against 2026.

The tooling for agents changes with every model release, but directing cooks and tasting plates doesn’t. If the part of a vibe coding book that sticks around is how you manage a kitchen, was it a management book all along?

Engineering leaders going hands-on and who keeps the team

Wed, 14 Oct 2026 00:00:00 GMT

Accept that you will probably do less people management for now. This is not really the time. ICs don’t expect it as much. We don’t even know what that career looks like. We will figure it out. We will get back to it.

I agree with probably 50% of this advice and would endorse it today. Leaders who disengaged from hands-on work had a bad year. The Great Engineering Leader Career Break reports many CTOs and VPs of engineering quit their roles, largely because of founders’ unrealistic expectations around AI.

Some founders are pushing you to fire half the team. The other are pushing you to show revenue going 2x when AI is going to go 2x and so on. It’s unrealistic expectations.

That all lands on the people whose job is to protect the team from it.

With smaller teams, you need fewer leaders. A lot of smaller teams come with middle management being cut as well.

As the layer that does the people stuff thins out, the advice to the leaders left behind seems to be to temporarily stop doing as much of the people stuff too, until things settle down and someone figures out what that career even looks like.

People go on vacation. They create energy with each other. They inspire each other. They hold a high bar. They keep each other accountable. This is not going anywhere.

This is management work, or at least what good management quietly protects week after week. No one’s agent is holding your team’s bar.

Two people and a bunch of agents, that’s not going to be a team. That will be a one off project maybe.

I agree. And that’s what makes it so hard for me to accept “for now”.. you’re telling me that teams are a thing that last, but you’re also telling the people who help glue those teams together to step back.

Hands-on: I support it. You can’t lead work you don’t understand, and work shifts too quickly to lead from a distance. But “for now” can become permanent without anyone noticing if no one’s responsible for going back. This all came out of a talk at LDX3 New York.. imagine a team of 5 people, with agents, and a lead who’s shipping code all day. Who’s going to notice if one of those five people is burning out?

Our heat pump heads as Bluetooth proxies

Tue, 13 Oct 2026 00:00:00 GMT

HA now knows which room my phone is in, because the nine ESP32 boards I have plugged into the CN105 ports of my Mitsubishi heads (see the first part) double as passive Bluetooth proxies, and the Bermuda integration translates what they hear into a room.

This enables more precise occupancy tracking of who. Also another unlock: when our toddler runs off with the Apple TV remote and hides it, we can easily pinpoint it. It’s the little things :)

I came across a few Reddit posts on folks running Bermuda in Home Assistant and was really curious about the use cases. Tracking AirTags in the house, phones, Apple Watches, and experimenting with lower cost BLE devices was intriguing. I’m finding that there is a plugin for everything in HA and each one more tempting than the next, so no surprise that I wanted to peel back this onion. I looked into the hardware required for this and to no surprise the brilliant ESP32 was at the heart of everything. Turns out I have 10 of these around my house now (9 Mitsubishi heads and one Navien tankless water heater) and the locations are spread around naturally such that it’s ideal for BLE triangulation.

Previously, I didn’t have Bluetooth enabled in Home Assistant (my host adapter entry was disabled, no Bermuda or Private BLE Device integration) and in each head I have an M5Stack AtomS3 Lite running ESPHome using the esp-idf framework with the MitsubishiCN105ESPHome component. esp-idf is what ESPHome recommends for their Bluetooth proxy so I was already on the right framework. The tenth board I have (the Navien) is left out of this, it has an RS485 going to the water heater and isn’t a proxy.

One of the only concerns I had was the shared hardware on the ESP32 for Wi-Fi and Bluetooth, given I rely on these to be the backbone of my climate controls of the house (more details on the climate coordinator later). We needed to thoroughly test these out before rolling any driver changes out to everything to activate the BLE listening. We can’t tolerate any Wi-Fi drops or flakiness, in other words.

The ESP32-S3 has a single 2.4 GHz radio which Wi-Fi and Bluetooth take turns using. The Bermuda wiki warns that setting the scan interval and scan window too close together can cause Wi-Fi dropouts and Home Assistant timeouts. A sweep of Reddit mostly reassured me: one user runs five Mitsubishi heads from an ESP32 doing CN105 plus BLE proxy on the same board, proxies fine alongside serial and IR. The bad reports were all load-related: busy ESP32s, unstable active GATT connections, Bermuda plus active proxies crashing a Home Assistant Green, silently hung scanners “fixed” by watchdog reboots.

Those reports shaped the package: the heads only listen, so passive scanning, passive proxy, no GATT connections (I don’t need them for anything in the house). The scan window is 160 ms every 320 ms (half duty), Bermuda suggests 300 ms. The free heap sensor exposes memory leaks. Every head includes the same common/ble-proxy.yaml:

# Bluetooth proxy on every head (Kitchen pilot 2026-10-06, fleet 2026-10-09).
# Passive only: presence (IRK/iBeacon) and BLE sensors need no connections,
# and every bad report from other users involves active GATT connections.
# The S3 shares one radio between Wi-Fi and BLE, so the scan window starts at
# half duty; raise it only if Wi-Fi and the CN105 link stay clean.
esp32_ble_tracker:
  scan_parameters:
    interval: 320ms
    window: 160ms
    active: false

bluetooth_proxy:
  active: false

debug:

sensor:
  - platform: debug
    free:
      name: Free Heap
      entity_category: diagnostic

Each head picks it up with one line under packages: in its own file, for example kitchen-ac.yaml:

  ble_proxy: !include common/ble-proxy.yaml

I decided to roll this out to our kitchen split head first, monitor the impact and closely monitor the telemetry from the device. It ran for almost 3 days with no issues… green light!

The Kitchen head has the strongest Wi-Fi signal in the house at around −36 dBm, so I figured it would be the best place to start. I flashed the firmware on October 6 at 14:41. The boot log showed passive scan at 160 of 320 ms, scanner running, and HA automatically added a Bluetooth entry. In the first five minutes, it heard about 60 devices, mostly Apple and Eve, had 174 KB of free heap, and the Wi-Fi signal was −37 dBm. If the silent scan hangs, that’s pilot failure.. a watchdog reboot is unacceptable on a heat pump controller.

The soak checks continued to be boring. After about 40 hours, the head had not rebooted, the CN105 link was never broken, Wi-Fi was at −37 to −40 dBm, and the free heap was flat at around 174 KB. At about 67 hours, on the morning of October 9, the situation was the same, with about 55 devices in view and the last advert 0.2 seconds before the check. All unavailable in the history was fleet-wide and due to a Home Assistant restart, not the head.

The same afternoon, I gave the other eight heads the same package in sequence from 13:12 to 13:22, verified their MAC address before the OTA, and all came back with the CN105 link and their previous climate state. At 15:23, Bermuda 0.8.7 was live on all nine proxies.

I seem to be picking up all sorts of passers-by, UPS deliveries, DoorDasher phones coming up to the front door, and all kinds of interesting things.

As of the night of October 11 (about 2.4 days), Bermuda has seen 739 devices total, 78 of which were visible at the time via all nine proxies. I can resolve my iPhone to a room, but not its exact position within the room, since I’m estimating distance based on signal strength on each proxy.

On the climate side, same check: no unplanned reboots, and the CN105 link is still up on all nine heads. Each unavailable was either a HA restart or a planned OTA, so there were zero single-head dropouts. Mean Wi-Fi signal per head is unchanged from pre-proxy. Mean free heap is 173 to 174 KB, with brief dips recovered.

I think this opens up a lot of future use cases on cheap hardware, or even repurposing existing gear (I’m very much against e-waste, and for being thrifty in one’s hobby.)

I have only scratched the surface with Bermuda, and some fine tuning is underway to make the most of it. I can see more interesting automations taking shape soon. I’m not sure if this replaces the need for mmWave presence sensors, but maybe combined with some Wi-Fi sensing techniques and some of the things Hue is doing with their Pro hub and presence detection, a combination of these could lead to some fun discoveries and use cases!

The heads have one more job in this house.. the climate coordinator that ties them all together is up next.

Share a photo from my iPhone and it's on the e-ink frame next

Mon, 12 Oct 2026 00:00:00 GMT

Previously, I had to open the Pigment web app on my phone and manually crop each photo to fit the panel’s shape before sending it to the frame (as described in A family e-ink photo frame in a day Pigment). Now, I can simply use Share in Photos and select Send to Pigment. The photo is next up on the frame, already cropped. I tried it with my first 3 shares and they all made it through, and I got a notification on my phone saying “3 sent to Pigment”.

iOS doesn’t give home-screen web apps a share target, so it was either a native app or an Apple Shortcut. The Shortcut won. It stays trivial because Pigment (the server) picks the album and the crop. An iPhone portrait is already the panel’s 3:4, so Pigment uploads it whole. Any other shape gets the largest 3:4 crop around what the photo is about, automatically done by sharp’s attention crop. Gotcha: the already-open Pigment app didn’t show new shares until I made it refresh when brought back to the front.

The frame grabs the photo next time it wakes up and checks in with Byonk. So I can get the photo to the frame without having to go through an intermediate step that requires the Pigment app to be open.

My wife already edits photos with her Lightroom presets and fashion 3 looked good on the frame, so her edits were the target. Pigment recreates her three Lightroom presets at render time from the slider values stored in each preset, leaving the originals alone. I didn’t want to use a lookup image for each preset because she would have had to export them all by hand.

Measuring showed her edits were basically “fashion 3” plus about half a stop of exposure, so that’s what I set as the default. I tested it on one of her real before/after photos and the color difference in Oklab went from 0.088 (untouched) to 0.031 (with fashion 3 + half a stop). Photos that she already edited in Lightroom don’t get the look (their metadata says so), but her reds still come out richer than Pigment’s.

Tune the looks on real photos and check them on the frame, then get the smaller frames going. A signed Shortcut would make it a one-tap install for the family.

Running the real Obsidian client headless in a container

Sun, 11 Oct 2026 00:00:00 GMT

I found that the official Obsidian headless sync client (Obsidian Headless) does most of what I need but to interact with some of obsidian’s plugins APIs from my AI agents, I needed a long-lived obsidian as a service and running inside the container on my Linux server was a fleeting idea that turned into something I lean on daily.

The sync client keeps the files in sync, but the plugin APIs only exist inside a running app, so if you want an agent to be able to query a plugin, it needs a live, responding, real Obsidian. The vault is the system described in My Obsidian setup and how I use it and this piece gives agents access to the vault.

One container instead of two services

I started with this setup: two host services, one for the desktop app running on a virtual display, and one for the headless sync client running alongside it. I set up a watchdog to run every 5 minutes and a reconcile to run hourly. But then I wondered: do I really need both? Could I just run one real desktop client and have it do everything, including sync?

I can. I now run a single container based on the linuxserver Obsidian image as a real desktop client (headless). It runs first-party Sync, community plugins, and exposes the Local REST API and TaskNotes API from the same container. I got rid of the watchdog and hourly reconcile, and my Ansible playbook for deployment shrank to a third of its previous size, making everything easier to maintain.

The cutover that deleted nothing

The riskiest moment was the first sync against my live vault (a misconfigured client can delete real notes!). I was concerned about how to know if the new client had seen everything before trusting it. I did a pure download of the entire remote vault into a scratch folder using my production config, then did an rsync checksum diff against the live vault. Only then did I point the container at the real vault. As a result, the first real sync didn’t delete anything.

That staging pull exposed a mistake I’d made: 7,495 files were silently skipped because I’d taken a shortcut and added some file types directly to the sync filter instead of using the setter, stranding existing remote files of those types. Toggling those types off and back on through the proper route brought them down. It was a good lesson in how the filter can look right but leave thousands of files behind. (No harm done here, of course.. it was in the scratch folder.)

Tip · Stage a sync cutover

Pull the remote vault into an empty folder with your production config and compare checksums against the live copy before the new client touches it.

I added a sidecar that forwards the plugin APIs, so agents and scripts keep using the same address as before.

The daily workbench

I use the obsidian CLI and REST API for TaskNotes the most, but I have also introduced some other custom plugins in development, and I can test these on this container image easily and take screenshots and iterate quickly (using the Chrome DevTools Protocol).

The container also doubles as a workbench for me: I can open the full Obsidian GUI in the browser, and with the Lean Terminal plugin, I get a terminal inside the GUI, where I have Claude Code installed in the container. So the very same place where I serve my agents is also where I work alongside them. (The vault search is a separate project, see niko hybrid search over my Obsidian vault)

Note · Terminal binaries stay put

Terminal binaries don’t sync between devices.. what’s installed in the container stays in the container.

A bug only an always-on Obsidian would find

And when I used the GUI, I discovered a mystery: the Tab key was landing in the wrong pane. It was a global key handler in the Tasks plugin. I wrote my own 25-line plugin to fix it locally, then filed an issue upstream with my tested-and-working fix branch, so the fix can make its way to others too.

Next up: Auto-restart, which I want to enable once the container has a long healthy record. After that, I’ll build more agent tools on top of the REST and TaskNotes APIs, since I always have a real app available to call.

My cloud architect habits at home

Sat, 10 Oct 2026 00:00:00 GMT

So I run my homelab like I would a production customer environment, based on my years of experience as a cloud architect, software developer, and a sysadmin professionally. The habits I forged in these roles over years and years have stuck, even though I’m working with a much smaller budget, of course (both in terms of money and time).

Always be building. That’s my motto. I’m a lifelong learner, and the way I learn is by doing: reading, writing, experimenting, and trying new techniques in software development, agentic development, DevOps, system administration and architecture. Writing about it is part of the process, because researching a post pushes me deeper and makes the lessons stick. And when I tinker at home, I hold myself to the same standards I use in my day to day line of work. That consistency is the point.. the habits only become second nature if I practice them everywhere.

I’m not always able to replicate the best practices of enterprises in terms of scale, but I strive to do a lot with a little, and I have a blast doing it.

DevOps at home

The code side is where I spend most of my “work” time. All of my repositories use Conventional Commits, and semantic-release looks at my commit messages to determine the next version number and create the release. I don’t pick version numbers manually. Every repo has the same four-job CI pipeline: lint, check PR title, run tests, and a release job. I use pre-commit, but with a single global hook dispatcher instead of copying it into every repo. The pre-commit runs gitleaks, trufflehog, and other checks, but again, nothing is copied into my repos. Hosting is handled by a self-hosted Forgejo instance, which is deployed with Ansible and then hardened. Forgejo is the only remote for all my repos except home-infra, which also pushes to GitHub as a second remote. Every repo has a rule that it must build from a clean clone, because one repo once failed to do so.

My infrastructure and data follow the same pattern. All host and service configuration is in Ansible. Applications deploy via Dockhand webhooks, so a git commit updates the running service. Secrets come from 1Password via its CLI and 1Password Connect.. never in environment files. For data, ZFS snapshots are managed by sanoid, and an offsite copy lands on Backblaze B2. The key insight: that copy is append-only.. even a compromised machine can’t delete backups. But I don’t just trust the green checkmark. A weekly job restores from the offsite copy to prove the data actually comes back.

Operations: The lab looks the most like a small production system. I use Loki for logs, VictoriaMetrics for metrics, and Tempo for traces. All alerts are routed through a single relay, apprise, so that I can change my notification settings in just one place. Healthchecks monitor whether a service is actually doing its job, not just whether the process is running. Deep health checks wherever possible. The UPS is honestly tested by unplugging it on purpose. Big decisions are documented with architecture decision records.. it’s a daily habit at work, and I wrote about it in Architecture decision records deserve a release cycle.

Scaled down on purpose

I run the homelab on a strict monetary and time budget (having two kids, they get priority and my time!) but that being said I maintain best practices in as much as I can do.

With the money limit, I can’t have a cluster of servers. I have one main server, so I need to implement redundancy only where it counts. ZFS snapshots replace having a staging copy of each service, so I can roll back instead of testing on a duplicate. I use consumer hardware wherever it’s adequate. With the time limit, I can’t have a change board (it would be me, anyway). Being the only one on-call means I only get alerts for things that matter; I tune out the noise. Automation and agents take care of repetitive work (a write up on sentinel agent coming soon), so I spend my hours with my kids instead of toiling.

The dividing line for me is cost. There are practices that are cheap to keep up (conventional commits, pre-commit hooks, infrastructure as code, append-only backups) that I don’t negotiate on at home. But practices that cost money or evenings (a cluster, a formal change process) I scale down intentionally, and I know exactly which ones I’m giving up.

Agents and the guardrails

These habits matter more now that AI agents are committing to my repos all day. A CLAUDE.md (or AGENTS.md) in every repo defines conventions like house rules.. the agent reads them before touching anything. Global pre-commit hooks catch secrets before the agent pushes. Conventional commits mean its changes are easy to review and semantic-release can cut a versioned release without rewriting history. ADRs give agents context about past decisions so they don’t accidentally reverse choices they don’t understand. And irreversible actions still need my click of approval.

The guardrails are what make the budget work. They let agents do the toil without going rogue, which is how I can maintain a lab this size on the hours I have available. Without the hooks and commit conventions, I’d have to review every line of every agent change individually, and that review time would eat into the same evenings the budget protects.

It’s a fun, lighthearted hobby, and I get to build really cool stuff (home automation via Home Assistant, and Aviary, for example) but I take it seriously.

Accelerate and the four key metrics when an agent ships the change

Fri, 09 Oct 2026 00:00:00 GMT

If you have an agent writing tested code for you and shipping it within minutes, your four key metrics from Accelerate will look great almost by default. Lead time drops to minutes, deploy frequency increases per merged change, and your dashboard lights up green no matter the quality of the work. It makes me wonder what those numbers even measure when the author of the code is a program.

Accelerate by Nicole Forsgren, Jez Humble, and Gene Kim was published in 2018 and drew on 4 years of research from the State of DevOps survey. The metrics that came out of it have become a common vocabulary for engineering leaders to talk about delivery. Readers of the book often feel that the main thing it brings to the table is evidence to help justify practices that should be obvious in the first place.

Chapter 2 introduces the four metrics (split into two pairs: throughput (delivery lead time and deployment frequency) and stability (time to restore service and change fail rate)) and describes the research that shows that high performers excel at both (debunking the myth that you have to choose between speed and stability). The chapter also debunks common proxies (lines of code, velocity, utilization) as measures of productivity.

While metrics are outcomes, the book mostly discusses drivers: 24 capabilities that are measured alongside the four metrics. There are also separate chapters discussing continuous delivery practices and architecture. In particular, loosely coupled design (where a team can test and deploy their part without coordinating with other teams) is mentioned as one of the biggest contributors to continuous delivery.

One criticism of the research is that the data comes from surveys using an open questionnaire distributed via social media, which is snowball sampling. DORA’s own lead, Nathen Harvey, has warned against using the metrics as targets or for comparing teams, because any metric that becomes a goal will be gamed.

The metrics have changed since the book. DORA renamed time to restore to failed deployment recovery time and added a fifth metric later: rework rate. The 2024 report linked a 25 percent increase in AI adoption to an estimated 7.2 percent decrease in delivery stability. The 2025 report says AI now improves throughput but still harms stability, describing it as an amplifier of whatever system it enters.

My own fleet of agents is just one small data point, but my monitoring agent, Sentinel, filed a correct fix for a real outage, and the proposal went unapproved for some 13 hours. Approving it as-is would have edited deployed files, which would have been overwritten by the next push. The outcome is described in Running an agent fleet like a small team: infra fixes now arrive as repo changes.

The lead time of the fix was the time I took to say yes to the agent’s proposal. The agent had finished its part in minutes, so speed isn’t in that number. It makes me wonder: when the work itself is nearly instant, whose clock does the lead time measure?

If an agent ships the change, do lead time and deployment frequency measure the review gate more than the work? If so, maybe change fail rate and rework rate will continue to be the numbers that tell us something about the software itself.

A doorway thermostat for our Mitsubishi heads

Thu, 08 Oct 2026 00:00:00 GMT

With all 9 of our Mitsubishi heads talking to Home Assistant via CN105, I wanted a better thermostat for each room.. I went with the Aqara W100, a small battery powered Zigbee display with a temperature sensor inside. These replace the stock Mitsubishi remotes we used daily until this point.

We chose to mount the W100 at the wall by the door in every room. Partially because that’s a much better thermostat location than the ceiling where the split is, but also because we inherited the house with the Mitsubishi remotes attached to the same spot (above the light switch in each room entry) so it made sense from an experience perspective (and the screw holes were already there!) A great part about the upgrade aside from the thermostat enhancement is the two way communication with Home Assistant via Zigbee. When automations trigger modes, the remote reflects it (set point, fan speed, etc.) compared to the Mitsubishi remotes which are one way (they only know their own state, so if HA changes something the Mitsubishi remote doesn’t know about it). Furthermore, the controls are very intuitive (plus minus and a mode button) and this leads to a really nice and snappy user experience.

Teaching the head to listen

Out of the box, the head regulates on its own return-air sensor up at the ceiling, so I needed to figure out how to get it to use the W100’s readings. The MitsubishiCN105ESPHome component exposes an obvious option for this: remote_temperature_source, which takes a Home Assistant sensor. Perfect! But Home Assistant’s ESPHome integration only forwards state changes, so if the room is steady, the head would get no updates and silently fall back to its own sensor. Darn it… But! The firmware exposes a set_remote_temperature action, so I have Home Assistant push the W100’s reading to the head every 5 minutes, but only while the W100 has reported in the last 12. If 15 minutes go by without a push, the head reverts to its own sensor, so a dead W100 or Home Assistant going down can’t strand a room.

Some GL06 owners report the unit overwriting the remote value with its own value, so I wanted to check that. I did a 15 minute cooling capture and observed the head following all pushes across 269 reads, never falling back on its own sensor. So in my case, the overwrite didn’t happen. I also observed a kid’s room turn off when the W100 reached setpoint, not the head’s own sensor, later that evening. I’m also listening for these possible events in my logs and alerting on it if it happens. I’ll update this post with any findings.

The W100 can be run in button mode or thermostat mode in Home Assistant. Button mode gives clean presses and 1 degree F steps but the screen can’t show heat/cool. We opted to go with thermostat mode.. since having the snowflake icon or flame icon on the wall matters a lot in our shared compressor situations where multiple heads on the same outdoor unit can’t run different modes (as I alluded to in the past). It pairs right up to ZHA with no Aqara hub required and no cloud required after setup via a community quirk (W100 support is not yet upstream).

Better rooms, fewer short cycles

We are seeing consistently better room temperature and less short cycling of the units with this change, and it lets us achieve better room comfort more reliably. The user experience is also top notch. We are also able to have one thermostat act as a zone climate controller (something we’re working on) rather than controlling an individual head, we can control a group of them and have a single set point, making the system a little more user friendly.

The fine print

Soo, the drawbacks: these are battery powered units and we made them report temperature changes more often. As a result, we expect a battery life in months not years as advertised, but we’re ok with that given our needs. There is also the quirk that this device even though configured to display in Fahrenheit sends commands in Celsius which Home Assistant has to interpret. I believe we have found an acceptable solution for now.

The setpoint traveling between the W100 and Home Assistant is whole degrees Celsius in both directions, but the screen steps in half degrees. So every second press is a change to the AC, and a single half-step press doesn’t send anything. I had a case where the screen said 71 but HA said 69.8. I added a resync that pushes the real setpoint from the AC back to the screen a minute after any activity, and every 15 minutes. One other quirk: 79 °F doesn’t exist. HA offers it, but the head maxes out at 26 °C (78.8 °F).

The reporting change is what costs the battery. By default ZHA had the W100 reporting on a 0.5 °C change, at most every 15 minutes. A kid’s room was overshooting setpoint by 4 to 5 °F on each cooling run. On Oct 4 we changed it to 0.1 °C changes, between every 30 seconds and every 5 minutes, and the overshoot stopped.. the battery pays for that.

A W100 in every doorway

We aim to have these for all second floor rooms and one for our main floor which includes the kitchen, living room, and dining room units (somewhat of an open floor plan so treating them as a zone makes sense).

That main floor zone is the job for the coordinator, which is up next in this series. That is, one set point for a group of heads, and the system figures out which heads run and in what mode.

The Mitsubishi heat pump telemetry hunt

Wed, 07 Oct 2026 00:00:00 GMT

I was hoping to aggregate as much data as possible from the compressors to be proactive, estimate energy, and to help build smarter automations around. The native Mitsubishi ecosystem seems to be able to estimate energy and generate statistics so I was hoping to be able to get these from CN105 with my new setup. At minimum I was hoping to be able to get outdoor temperature from the outdoor compressor units to help build automations and logic around (ie don’t heat if it’s warmer than X degrees right now, or in the last 12 hrs, plus using a forecast integration, would be very helpful for what I’m trying to accomplish).

It turns out most of the data was there, it was just not being emitted from the heads that we tested initially.

A whole lot of zeros

The MitsubishiCN105ESPHome component I built the firmware on in part 1 has optional sensors for compressor frequency and energy, and I turned them on. On the first two heads I tried (a kid’s room and the guest bedroom) they all read 0 even while the head was blowing cold air. The component already logs every frame it receives at debug level, so I could look at the raw bytes without the UART debug option (that one crash-loops the firmware before Wi-Fi comes up, so it’s banned in this project). The status reply with the unit running had operating set to 1 and every measurement byte at 0:

06 00 00 00 01 00 00 00 00

256 codes, five answers

I believed they send data after talking to a tech, so I didn’t want to take the zeros at face value. Mitsubishi’s own apps weren’t much help either: Kumo has no energy fields at all, and MELCloud labels its numbers “calculated values”. CN105 is a request/reply serial protocol the community worked out in SwiCago/HeatPump, where the controller sends a request with an info code and the head answers with a frame for that code (the status reply’s first byte above, 0x06, is its code). The component only asks for codes people already know about, so the question was whether the data lived behind a code nobody was asking for.

On a throwaway branch I had the guest bedroom head send one read-only request every 4 seconds for all 256 info codes. Only the five known codes ever answered. The unit stayed healthy the whole time, and the few dozen collisions with the normal polling cycle were harmless. I also did a load test (in case the numbers only show up when the compressor is working hard): the head in a kid’s room at 61°F with the fan on high for 10+ minutes. The room temp dropped about 4°F and all 186 status replies were identical zeros. So the conclusion was “MSZ-GL doesn’t report any of it” and I removed the six sensors from the firmware.

Confidently wrong

When we tried to analyze the serial data streams and didn’t have any luck, I went looking for others who’d seen the same and also looked into research we did prior to the project. I found that some users thought the model we were running didn’t emit this telemetry, some users said maybe only certain production batches emitted it, and other guesses.

Looking back, every byte we captured was real. The mistake I made was jumping to a conclusion after seeing two heads - both on the same outdoor unit. These are multi-split systems (East feeds five heads, West feeds four), and the research said only a “lead” head on each outdoor unit might carry outdoor data. Lesson learned: sweep every head before deleting a sensor.

The master head hunch

My hunch about the master head (“Which one is the master head.. this might be the one that sends extra values!”) turned out to be a good instinct to chase down, and I’m glad we went down that rabbit hole.

When I fitted the last head, its room-temperature reply had bytes nobody had seen before. We swept all nine and found that 4 of 9 report a lifetime energy counter and runtime minutes: the kitchen, master bedroom, master closet and playroom. The living room reports outside air temperature only. The heads we tested first were exactly the zero kind. It’s not a lead head thing, since there are two reporting heads on each outdoor unit, and it’s not size either: all three GL09s report but only one of the three GL06s does. My best guess is a board revision or production batch (the GL06 serials split between 23E and 28E) but that’s still unconfirmed.

Counting what, exactly?

Runtime is easy - ticks once per minute of run (11 ticks in an 11 min run) and lifetime is in the 8800-11000 hour range across the heads. Energy is weird - the 4 heads read between 67 and 106 “kWh” lifetime, ticks in 0.1kWh steps, but 106kWh / ~10279 run hours is like 10W average which is obviously wrong. It also didn’t tick once in 11 mins at full load. So either the scale is off, or it’s measuring something other than what the name says (or it wraps?) - I’m treating it as relative only until I can get it on a meter. Still no live power or compressor frequency on any of the 9 heads.

Don’t cool when it’s 61 outside

The outside temperature is the piece I’m putting to work right now. My climate coordinator in Home Assistant has a weather gate: in Auto, a room may cool only while it’s at least 58 °F outside now and for the next 3 hours (the coldest of the current temperature and the next three hourly forecasts from the weather integration), and heat only on days whose forecast high is 75 °F or less. Both thresholds are settings. The cool gate originally used the day’s forecast high, but lows here sit in the 40s on 70-degree days, so Auto cooled all night: on Oct 5 the head in a kid’s room cooled at 61 °F outside and the room fell to 67.6 °F. Since Oct 6 the cool gate reads the outdoor units’ own temperature from CN105 when it’s live, and the same readings will feed the boiler’s balance point (5 °F by default, since both outdoor units are Hyper-Heat).

Like I mentioned, we’re building automations around the outdoor temp currently, and I’d like to get into energy estimations in the future but that will probably require some AC clamps on each unit to do what I really want to do. I’ll be doing a thermostat in the doorway next in this series, and then the coordinator itself (set a zone temp and have the system figure it out).

Input-bound teams and where AI savings actually show up

Tue, 06 Oct 2026 00:00:00 GMT

I think there are two types of jobs. If you want to simplify an organization, there are supply constrained and there are demand constrained jobs inside a business. So input bound and output bound. What is the constraint on a job?

This is the theory of constraints applied to a job description. Goldratt says that the output of a system is limited by its constraint. In other words, before you can evaluate how AI will impact your team, you need to understand what your team’s work is waiting on.

Legal and customer service are input bound. If I deploy a lot of AI, I don’t suddenly get a lot more contracts, right?

This applies within IT too.. access-request queues, change-approval queues, ticket-desk queues, all depend on others submitting work. AI reduces the handling cost per queue item. And since the volume of incoming work doesn’t change as your team gets faster, you can realize savings there too.

But engineering is a different kind of team. If you make your team faster, they won’t finish the backlog sooner.. they’ll just ship more stuff and the backlog will refill anyway.

And so it may be that you have less input bound functions and areas as a proportion of your total employee base and you have more creative output bound functions.

A rule of thumb for this decision.. before you cut an engineer justified by faster tools, ask what’s constraining the team. If they’re output-bound, then the question of headcount becomes a question of roadmap. If a platform team can double their speed, does the business actually want twice as much stuff built? Usually the answer is yes.

I think it’s very easy for people to have very binary takes on these things, right? I find it amusing that if you put an engineer, a designer, and a product manager in a room, they’d all sit there and they’d all say, “Oh, man, it sucks to be you. Your job’s going to go away.” And I’m like, “Ah, that probably is your answer right there.”

If everyone in the room is confident that the vanishing job is someone else’s, I think that means nobody has really looked at what their own work depends on.

I think that’s going to take much longer than people think. We’re big in technology. Next year everything will be like this and it takes 10 years to happen, but it will happen.

This framing came up in a Decoder conversation about enterprise software, and I think the slow timeline gives us time to properly classify things. Which teams are actually waiting on input? Did we check before we let a spreadsheet do that?

My Obsidian setup and how I use it

Mon, 05 Oct 2026 00:00:00 GMT

I have used Obsidian (and other text-based note-taking apps) for years. Early on in my professional journey I chose to implement note taking in plaintext and markdown. An Obsidian “vault” is just plain markdown in a folder, it syncs across my machines, and agents can read and write it like any other files. So it became a natural place where agents and I can collaborate, where I can oversee and orchestrate my fleet and have complete visibility into output.

Obsidian is an app on top of a folder. The folder is what matters. My notes are just files, so there’s no special agent integration. The agent opens a note the same way I do in my editor. This site publishes from the same vault (more on that in How this site is built).

One file per task

Tasks live as notes (via the brilliant TaskNotes plugin), so agents and I work from the same list. I use a few Bases views of my TaskNotes as my daily triage.

TaskNotes replaced both TODO.md and Linear for all my task tracking. Each task is a file, so agents updating different tasks never conflict. The Assignee property lets me mimic Linear’s states when handing work to an agent. I migrated about 70 tasks (most from the old TODO.md, the rest from Linear) and numbered priorities from 0-critical to 3-low so Bases views sort properly, making my triage view instantly readable.

Nothing gets past the inbox

I use the Inbox methodology religiously: nothing makes it into the vault past the inbox until it’s been reviewed by me and is properly formatted and linted.

All new items end up in _Inbox/ first, either captured by me or written by an agent. The shared agent inbox skill makes two passes over the folder, with a gate before deletion, so nothing leaves until I review it. I was able to triage my inbox from 17 files to 0 in a single session, with each note routed to its home folder and given its vault frontmatter. That gate is why agents can have write access to the vault.

Frontmatter is the contract

I have strict rules on frontmatter in my markdown within my vault. A shared schema / contract is critical. For example, anything that is purely agent created in my vault gets a property in frontmatter so I can trace back the provenance of where that note came from.

Some core fields on every note are type, context, created, modified, tags. I have a provenance property called ai which is set to true for notes created by agents (session handoffs, transcripts, etc) and false for my own notes. Transcripts are machine generated and may have errors that the native captions wouldn’t so I mark them as ai: true along with the name of the model that produced them. This way I can filter them for review and know which ones to re-run after a model upgrade. Here’s an example of the frontmatter from a project note, trimmed for brevity:

---
type: project
context: personal
created: 2026-07-14
modified: 2026-10-20
ai: true
status: active
summary: Containerized Obsidian ...
tags: [self-hosting, devops]
aliases: [Obsidian Container]
---

Tip · Settle your frontmatter early

If you’re building a vault that agents will write to, lock in the shared fields before the first agent note lands. Every agent writes to the same contract from day one, and every filter or view you build later covers all of it.

Search first, write back after

The agents search the vault before they answer and write their handoffs back into it, so the next session picks up where the last one stopped.

I’m using niko for search and Memento for agent memory, and the same search covers my saved Reddit archive from Stash (now in its own store outside the vault). When a session ends, the agent writes a handoff note to _Agents/claude/sessions/ with type: session and handoff: true in the frontmatter. 810 of them so far, and they mean the next session always starts with context from the last one without me having to re-explain the project.

The plugin stack

TaskNotes so every task is its own note
Bases for my triage views
Templater for note templates
Linter to keep formatting consistent
Local REST API so agents and scripts can talk to a running Obsidian instance
Lean Terminal for a terminal pane right next to my notes
Minimal as the theme

The terminal pane is the biggest change to my workflow. I can now run Claude Code in a Lean Terminal pane next to my current note, so writing, tasks, and agent direction all happen in the same app. One surprise: the Tasks plugin’s autosuggest captured Tab app-wide, so tabbing in the terminal indented the note. A small local plugin (suggest-focus-guard) fixed that, and I filed the issue upstream.

My vault now has an always-on copy: the real Obsidian desktop client, headless in a container. With Obsidian Sync and the app’s APIs enabled, my agents now always have a running instance of Obsidian to work with. I’ll explain how I set that up in my next post.

Our Mitsubishi mini-splits in Home Assistant for $15 per head

Sun, 04 Oct 2026 00:00:00 GMT

We have a boiler system with radiators in our home but also have 9 Mitsubishi mini-splits connected to two Hyper-Heat outdoor units (one feeds 5 heads and the other feeds 4). All of these heads operated on their own. As you may or may not know, a multi-zone outdoor unit will only operate in one mode at a time, whichever head calls for heat or cool first “wins” and sets the mode for the entire unit. So if one head is cooling on one outdoor unit, another head connected to that same unit can’t heat, and vice versa. This meant that if we wanted to quickly cool a room, we had to walk around the house and figure out which head was running so that we could turn it off to get the conditioning that we needed at the time. The scheduling functionality on the stock remotes was all time based and not very intuitive.

We wanted to be able to automate the units, control them from our phones, set up zones, rules, automations, and eventually use the boiler as second stage heat when needed in the colder months. As of this week, we have all 9 heads reporting their real state to Home Assistant locally for about $15 per head, with no cloud involved. Being able to set a zone temperature and have the system figure out the coordination for us is what we’re going for.

Expensive, flaky, or nothing at all

I only had Airzone Cloud on one head (the 3rd floor guest bedroom) the previous owners installed. It was ugly (an external control box needs to be installed on every head), really finicky (random Wi-Fi dropouts) and the unit was about $200 which meant if I wanted to run them on all the units it would have been $2,000+ total.

I ended up running some Broadlink RM4 mini IR devices (about $25 each) in a couple rooms that I needed remote control of (a kid’s room and our master bedroom) as a stopgap until I could find the time to work out a longer term solution. These little devices are great for what they are (the ability to learn IR and a database of codes built in) and made remote control of the Mitsubishi heads quick to onboard. They were of course stateless.. just like the standard Mitsu remotes (they don’t sync with the units, they only have the state of the last commands they sent). The other six heads I had no remote control of at all.

The CN105 port, hiding in plain sight

I ended up doing some research and found out about the internal serial port (CN105) on the Mitsubishi boards. This is the port that Airzone and other hardware integrations plug into. I also discovered some open source projects based around the ESP32 that talk to this port. This was starting to feel like the Navien RS-485 work I did recently.

There’s a lot of open source work built around this port. These are the options I looked at:

SwiCago/HeatPump, the Arduino library many others build on
gysmo38/mitsubishi2MQTT, ESP firmware that talks MQTT
geoffdavis/esphome-mitsubishiheatpump, ESPHome component on SwiCago’s library
echavet/MitsubishiCN105ESPHome, ESPHome component and the one I cloned and modified
Serin Labs, a ready-made controller if you’d rather not flash anything

$15 per head, no soldering iron

I went with the trusty M5Stack AtomS3 Lite I used in the Navien project coupled with a custom cable made by a company called Serin Labs out of New Jersey which made the entire project solderless and plug and play. The Atom draws its power from the CN105 port itself so there’s nothing else to wire up. Total price ended up being $7.50 for the Atom and $7.50 for the cable!

Each head gets its own Atom on the CN105 port running ESPHome and all nine talk to Home Assistant over Wi-Fi.

Two days, nine heads

The install procedure was pretty smooth, I worked in Claude Code on the firmware (cloned from echavet’s project with some of my own modifications) and then it was just a matter of going to each head (9 total, all MSZ-GL units) over the course of Oct 1 and 2, taking the cover off (dusting and cleaning while I was up there), bench flashing each Atom on my laptop over USB, reserving its address on the router, installing it in the head, adopting it in Home Assistant, and making sure Wi-Fi was strong and data was flowing.

A couple spots needed extra attention. The GL24’s CN105 port hides behind the right corner box (the service manual shows where). The closet head grabbed a weak access point at -82 dBm so I pinned it to a stronger one and moved the Atom inside the housing, which brought frame loss from 15% down to 3.6%. Once all nine were reporting I retired the old Airzone Cloud and SmartIR integrations in Home Assistant.

Three sessions, one job

The work split across multiple Claude Code sessions. Claude Code has a feature where open sessions can coordinate and communicate with each other over an internal socket, so the Home Assistant session could ask the home-infra session for an address reservation without me copying anything between them. I had a cn105-mitsubishi session which was dedicated to flashing and firmware for the Atom devices, I had one session in my home-infra project which manages my Flint 2 router with Ansible (for the address reservations), and another session which was dedicated to my Home Assistant configuration. This separation of duties keeps the context clean in each session and allows for very dedicated sessions to specific projects, with all of the project’s rules and conventions loaded and in place where needed. I wrote about how I keep the rules and context around in Memento and running a bunch of sessions at once is a whole ’nother post.

Next up in this series: the quest for better telemetry out of the heads, a thermostat by a doorway, the coordinator (where you set a zone temp and the system figures it out), and the boiler as second stage heat. And since there’s now an ESP32 in every head, I’m also looking at using the Atoms as Bluetooth proxies for room-level presence in Home Assistant with Bermuda.

Hot water at the tap with Home Assistant

Sat, 03 Oct 2026 00:00:00 GMT

Hot and warm water instantly at the fixture, when we want it, without paying for recirculation all night. That’s what local control of our Navien actually unlocked for us. The first post covered the hardware: an ESP32 on the NaviLink port, the ESPHome integration, and about $30-40 in parts. ESPHome adoption gave us 37 entities to work with.

Scheduled recirculation

The community repo ships a Home Assistant blueprint for scheduling when recirculation runs. We set a schedule helper for 06:00 to 22:00 and left it alone. No recirculation overnight, no burning propane while everyone is asleep. We use presence sensors too, so if we’re not home, it’s not spinning cycles.

The history shows the heater cycles for about 2 minutes, roughly every 30 minutes inside that window. Not a constant burn - the NaviCirc valve under the kitchen sink opens, water loops, and the heater fires just long enough to keep the lines warm.

The NaviCirc is a temperature-activated valve that uses the existing hot and cold lines - no dedicated return pipe. The loop is simple: heater pushes hot water out, it runs through the hot line past every fixture, hits the NaviCirc, crosses over to the cold line, and returns to the heater.

The all-night override

We have a tile on the dashboard that overrides the schedule and holds recirculation on all night. The override can be for anything, but for us instant hot water for baby bottle warming has been great. Walk to the kitchen at 2-3 AM for a baby bottle warm-up, turn on the tap, hot water is there.

Hot-water prime for the washer

This one is the nerdiest and maybe the most useful.

Our LG washer and dryer are connected to Home Assistant, so we know when they come on. When a wash starts, the automation presses the Hot Button once to prime the hot line so the first fill is actually hot, then holds recirculation off so rinses stay cold. On the first real wash the outlet climbed from 114.8 to 125.6 F before the first fill.

Propane

Propane usage has been very minimal for us to be honest, maybe $20 extra since the recirculation went in. Insulated pipes have helped too. The heater is about a third of what leaves the tank, so tracking it precisely takes a little math. I did hit one bad 0 reading from the heater’s gas counter that double-counted the lifetime total until an availability guard on the sensor fixed it, but that was a one-time fix.

What’s next

We plan to install a second NaviCirc at the 2nd-floor guest bathroom.

How I verify backups: a weekly restore, not a green checkmark

Fri, 02 Oct 2026 00:00:00 GMT

I migrated everything over to the new home server, migrated the nightly backups, and migrated the failure alerts (I have Backrest page me via its own error hook if a run fails).

But I didn’t migrate the weekly restore test. That was still running on the old server that I was retiring. So every night I got a nice green checkmark on the new server, but I had no idea if I could restore any file from it. The question had gone from “did it run?” to “would it come back?”

I decided against the most thorough option: restoring the entire full snapshot every week and comparing it to the live disk (too slow, too much space for a weekly). Instead, I restore one sample file and all my database dumps. I can restore individual files cheaply with restic restore --include (restic restore docs), and I want to test restoring the databases because you can have a backup that seems fine but is actually useless.

I needed to fix the databases before this test would be meaningful. The problem is that I was backing up the live DB files, so restoring a copy of a file that was in the middle of being written to might not be consistent, rendering any checks worthless. So I set up dumps to run a few minutes before the nightly backup (SQLite’s online backup, pg_dump for Postgres). A job on Sunday morning restores the dumps and checks them. It’s silent on success, and only pages me if something goes wrong.

Warning · Don’t back up live DBs

If you restore from a copy of a live database file that got caught in the middle of a write, the restored copy might not be consistent. So it’s good to do dumps a few minutes before running the nightly backup (SQLite has an online backup mode, and you can use pg_dump for Postgres).

The first time I ran the full suite, 28 of 28 tests passed, but only after fixing many things on launch day. The biggest fix was alerts: my notification relay was returning success when I tested sending but silently dropping every message (One alert relay for everything apprise). Other launch-day fixes were timers that never fired and a failure alert that was “heavily truncated on fail”.

The checker needs checking

A few weeks later, I got my first false fail. It turned out that Plex’s library DBs use a custom tokenizer, so stock sqlite3 can’t run PRAGMA integrity_check on them, it errors with unknown tokenizer: collating. The run showed 3 failures, but nothing was wrong. I didn’t want to exclude Plex (which would silence the noise but stop checking Plex), so I made the checker distinguish between a tooling limit and actual corruption. The fallback is that if it can’t run the integrity check, it reads the schema, which still tells me the restored file is a valid DB and not truncated.

res=$(sqlite3 "${HOST_TMP}/${f}" "PRAGMA integrity_check" 2>&1)
if [[ "$res" == "ok" ]]; then
    report PASS "sqlite integrity: ${f}"
elif [[ "$res" == *"unknown tokenizer"* || "$res" == *"no such module"* || "$res" == *"in prepare"* ]]; then
    # That's a tooling limit, not corruption. Fall back to a tokenizer-free
    # structural probe: a readable schema B-tree proves the restored file is a
    # valid, non-truncated sqlite DB.
    objs=$(sqlite3 "${HOST_TMP}/${f}" "SELECT count(*) FROM sqlite_master" 2>&1)
    if [[ "$objs" =~ ^[0-9]+$ ]]; then
        report PASS "sqlite integrity: ${f} (integrity_check N/A: custom tokenizer; schema readable, ${objs} objects)"
    else
        report FAIL "sqlite integrity: ${f}: schema unreadable: ${objs}"
    fi
fi

While chasing that, I found a false pass, which is a worse thing to worry about. It turns out that if you run sqlite3 on a nonexistent path, it’ll create an empty database and report ok. So a mangled path would pass, but not because it was testing anything. I learned that the checker itself needs to be checked, and now I’m suspicious of an easy ok.

Warning · sqlite3 false pass

sqlite3 will create an empty database if the path doesn’t exist and report ok. A mangled path will pass PRAGMA integrity_check without actually checking anything.

One file from every copy

At first I only had the local repository, but over the summer I added the ability to restore individual files from every other copy: the local USB disk, the SSD tier, the offsite bucket, and the cold bucket (see My backup strategy for the layout and the offsite). Each file is md5 compared to the live file, so I know I’m getting the file’s content, not just something. (the comment on the cold bucket leg says “a copy nobody restores from isn’t a copy.”)

# Runs once per offsite repo: main (covered set) and cold
# - a copy nobody restores from is not a copy.
age_h=$(( ($(date +%s) - epoch) / 3600 ))
[[ "$age_h" -le "$MAX_AGE_H" ]] \
    && report PASS "${label} freshness: ${snap} is ${age_h}h old (${count} snapshots)" \
    || report FAIL "${label} freshness: ${snap} is ${age_h}h old (> ${MAX_AGE_H}h)"
for sample in "$@"; do
    rrestic restore "$snap" --target "$restore_dir" --include "$sample" >/dev/null 2>&1 || true
    ...
        # Live copy on hand: prove content, not just presence.
        h_live=$(md5sum "$sample" | cut -d' ' -f1)
        h_rest=$(dx md5sum "${restore_dir}${sample}" 2>/dev/null | cut -d' ' -f1)
        [[ -n "$h_live" && "$h_live" == "$h_rest" ]] \
            && report PASS "${label} restore: $(basename "$sample") (${lines} bytes, md5 matches)" \
            || report FAIL "${label} restore: $(basename "$sample") md5 MISMATCH"
done

I added two more legs. Every week, I grab a photo from the offsite bucket and md5 compare it to the copy on the server. The photos are the most irreplaceable, and the first one matched byte for byte (Photos iCloud is a sync service not a backup). The test only pages on failure, so a verify that silently stopped would be invisible. So I publish a last-success metric and Grafana alerts if it gets older than eight days.

Silent gaps

The health check in late September revealed two silent gaps. Both were cases where everything kept passing. The first was a Plex dump that had stopped refreshing. The old copy kept passing the integrity check for about two months. The database was stale, but still valid. I changed it so that any dump much older than its snapshot fails. The orphaned old dump failed the new check (it was 1488 hours old) and I deleted it.

# Dump age vs the snapshot: a dump the nightly job stopped refreshing (source DB
# deleted, or its dump failing and being rm'd while an old copy survives) still
# passes integrity forever. `restic restore` and `docker cp` both keep mtime.
d_mtime=$(stat -c %Y "${HOST_TMP}/${f}")
d_age_h=$(( (SNAP_EPOCH - d_mtime) / 3600 ))
if [[ "$d_age_h" -gt "$MAX_AGE_H" ]]; then
    report FAIL "db dump: ${f} is ${d_age_h}h older than snapshot ${SNAP_ID} (source gone or dump failing?)"
    continue
fi

The second gap was the dump script itself. It was exiting success every night even though two of the dumps were failing. Nothing upstream noticed. Now it errors on any failed dump and publishes its own metrics.

Never a full restore

The same health check gives me the honest bottom line. Backups happen on time, verifications pass, but I’ve never restored and started a service from a backup. Git repositories, documents, my photo library’s database, my notes vault.. never restored beyond a single file. Postgres dumps are listed but never restored. The latest run shows 74 passed, 0 failed, 1 warning (an unpushed commit).

This proves I can get my files back, but not yet that I can get a service back. I plan to add a deeper monthly run that restores a database from backups and queries it, going beyond a single file.

Wiring our Navien water heater into Home Assistant

Thu, 01 Oct 2026 00:00:00 GMT

We moved into a house in Westchester NY after relocating the family here in 2024 for my job in NYC. The previous owners put in a Navien tankless unit before they moved out to replace the old boiler system. It has worked great, endless hot water, great temperature control, but we noticed the wait at certain fixtures in the house was becoming annoying.

Reason for the project was to enable recirculation on a schedule so we have better control of how and when our recirc happens, since we already ran Home Assistant. Navien has a cloud option but ours was never on it, and I always prefer local for anything IoT and home automation. Like $30-40 in parts and some time.

Why recirc matters for us

For washing laundry and washing dishes, it’s a big win for enzyme performance and cleaning power to have hot and warm water instantly at the LG washing machine, steam dryer, and dishwasher in the kitchen. That’s a huge huge thing for us with lots of messy clothes with two little boys.

I installed a Navien NaviCirc recirculation valve at the furthest fixture past the 1st floor washer/dryer (kitchen sink) which is a temperature valve that uses the existing hot and cold lines (no dedicated return line). I plan to put another one of these in at the 2nd floor guest bathroom.

The hardware

The heater is a Navien NPE-240A2 and it has a 5-pin NaviLink connector on the front board which has 12V, B, A, and GND. This connector talks RS485 and is how Navien’s proprietary Wi-Fi module talks to the unit.

The controller is an M5Stack AtomS3 Lite on top of the Atomic RS485 base which handles the RS485 level conversion, and the whole thing is powered from the heater’s 12V pin (no separate power supply needed).

Firmware

I used htumanyan’s navien ESPHome component with one local edit: the default UART pins are set up for a Tail485 board, but the Atomic RS485 base uses tx on GPIO6 and rx on GPIO5. Swap those in the YAML and it works. I did the wiring and flashing myself, with Claude Code helping on a few bits.

The community did a lot of the hard work already. The HA community thread (385 posts and counting) and jhenkens’ hardware guide are the main references. There’s also a Reddit thread that walks through the hardware side in plain English if you’re just getting started.

Lessons

A few things worth knowing going in:

The NaviLink setting has to be enabled in the heater’s installer menu, or else the bus will be completely silent. If you’re not getting anything, check that before you start re-wiring things.
The working A/B polarity reads about -2V idle. That feels counterintuitive but trust it.
Don’t add the UART debug byte-dump logger. This crash-loops the ESP before Wi-Fi comes up. It requires a USB reflash at the heater to fix.
No termination resistor needed on this unit, it works fine without one.

What’s next

The hardware side is the easy part.. the automations are where it really pays off. Part 2 covers the Home Assistant side - scheduled recirc, the override button, and what we can actually do now that the heater talks to HA.

Running an agent fleet like a small team

Wed, 30 Sep 2026 00:00:00 GMT

I want to show how I run an agent fleet like a small team: each one has a role, there are review gates, and they propose while I rule. One lead session keeps the ledger and hands each item to the session that owns that project. Sessions dedicated to projects, at my scale, keeps things clean and aligns with my cognitive ability (an app build might need local server work, some cloudflare terraform, or a cloud dependency, and I have clean separation of these responsibilities across the agents).

The most instructive gate was the one I wasn’t watching: I reported “Infra Pulse seems to fire but do nothing” (the pulse was fine), but the Sentinel had caught a real outage in reddit-stash and filed a correct proposal to fix it. The proposal went unapproved for some 13 hours despite hourly reminders in the channel. On closer look, approving it as-is would have edited deployed files directly, so the next push would revert them. The result: infra fixes now arrive as repo changes. High-severity cards also ping my phone via Pushover. The card itself is built with Slack Block Kit on NanoClaw.. I walked through that in From a cron job to Sentinel.

The second gate came from the agents themselves. At least three times in one week, a session refused to act on an approval relayed from me by another session, and instead asked me directly in its own window. Why? A relayed yes is a report about my decision. Now an outward-facing change waits for my answer in the session that owns it.

Agents rejecting relayed approvals from other sessions is exactly the type of guardrails that instills confidence, trust, and makes this system work for me. They will want a specific approval from me directly on high impact changes. Especially with Claude Opus 5.5 this works well. In previous Claude models, I had observed behavior of agents invoking tools to insert their own approvals through the tmux interface to other sessions to move work along (!!!) which definitely is interesting and amazing but does not instill trust of acting within the guardrails I set up for my agentic work.

Ruling also means deciding what to hand back to the agents. The lead decided one block of home-infra backlog triage on my behalf, and I can overrule any part of it. The things that still need me are at the bottom of the status note, the shortest list. Fusion got ~7x cheaper but I admitted I wasn’t reading its nightly docs. I changed Fusion’s job to be Slack-first briefings instead of switching models. The day-to-day of how the lead and peers run is Running six to nine Claude Code sessions at once.

I’ll continue to refine the rules and guardrails I use for my agents from community anecdotes (X, Reddit, Hacker News, Discord communities, etc) and best practices guides from Anthropic and frontier model purveyors. As more trust is built, more tasks will be delegated and autonomy will be given to act with minimal approvals and oversight from me.

Class-only SVGs that render everywhere

Tue, 29 Sep 2026 00:00:00 GMT

So I like to write myself.. this blog, books, my own notes, but I’m increasingly leveraging AI to author and maintain diagrams. The AI follows my ten-class contract and the lint I set up. I can pass in hand-drawn diagrams or text-based descriptions and have simple, effective, lightweight but impactful diagrams produced. I think these fit the web very well, they render cleanly in Obsidian, and I can version control them in ADRs (the ones that deserve a release cycle) and otherwise.

That only works if the diagram is markup, not a picture. So every diagram on this site is an SVG which contains no colors, no fonts, nothing. The site styles it, it’s git versioned, and lint checked in CI just like any other code.

This is new to me and frankly an experiment!

Note: Mermaid has always had a place in Markdown based authoring and last couple years a big uprising in usage since agents write and consume it very well and efficiently. Now that I have my agents writing diagrams for me, I’ve shifted to SVG.

The problem with pictures

Export an SVG from drawing tools and you get the fill, stroke and font-family baked in as presentation attributes on every elements. Looks great.. Then the site maybe gets a dark mode and you’ve got near-black lines on a near-black page, or you redesign and every old diagram is still wearing the old theme.

You can’t fix that from inside the file very well either. Presentation attributes can’t take CSS variables, so a hardcoded #1a1a1a can’t follow the theme tokens. And an SVG loaded via <img> is its own document which can’t see the page’s CSS, so even currentColor resolves against nothing useful. The SVG 2 styling chapter is a fun read if you want to go down the full cascade rabbit hole.

So this site (built with Astro) inlines each diagram at build time, with a little plugin in the Astro Markdown pipeline that swaps the embedded file for its <svg> markup, wraps it in a <figure>, and turns the alt text into the SVG’s <title> if the file doesn’t have one (so screen readers get a real description). Once it’s inline, it’s just part of the page, and page CSS reaches it. It took a few back and forths with Claude Code to nail down the experience I wanted for generation but I’m pleased with the results.

Classes only

The contract is ten classes, and they’re the only thing a diagram gets to say about how it looks.

dg             # on the root <svg>
dg-box         # a node
dg-box-accent  # the node the paragraph is about
dg-muted       # a background node
dg-group       # a boundary around nodes
dg-line        # connectors
dg-arrow       # arrowheads
dg-label       # text
dg-label-sm    # smaller text
dg-set         # a Venn set

No style attributes, no fill, stroke, or font-* anywhere in the file (with one exception, a fixed <style> block, more on that below). A box and its label look like this:

<rect class="dg-box-accent" x="136" y="8" width="208" height="48" rx="4"/>
<text class="dg-label" x="240" y="37" text-anchor="middle">Rules and fixtures</text>

diagram.css is the only place those classes get hooked up to the theme tokens (dg-box gets the paper fill + ink stroke, dg-set gets the accent at low opacity, etc) and each diagram file gets a copy of it, with a test keeping it in sync. Dark mode just follows the tokens, and if I ever redesign, one file and every diagram is restyled without touching a post.

The tradeoff is you can’t just make one box red because it feels important that day. There’s one accent class and the style guide says to use it on at most one or two nodes. For small explanatory diagrams that constraint is the point (it keeps them from turning into slide art).

Rendering outside the site

Obsidian doesn’t inline anything, it just shows ![[x.svg]] as an <img /> which hits the same limitation as above: page CSS can’t reach inside the image. On my iPhone, every class-only diagram came out as black boxes with black-on-black labels, and the connector lines were invisible.

So every diagram file has this one fixed <style> block which is diagram.css with the light/dark token values written out, plus system fonts (obviously an image can’t load the site’s web fonts). It sits right after the <title> in every file, the same bytes each time:

<svg class="dg" role="img" viewBox="0 0 480 160" xmlns="http://www.w3.org/2000/svg">
  <title>One rules file and its fixtures feed two checks</title>
  <style>.dg-box{fill:#f3f4f0;stroke:#17191b;stroke-width:1.5}…@media (prefers-color-scheme:dark){.dg-box{fill:#141719;stroke:#e0e3dd}…}</style>
  …
</svg>

The image picks up Obsidian’s theme rather than the phone’s theme (my iPhone is in dark mode but Obsidian was in light, and I got the light version), so the diagram doesn’t need to carry its own background.

The site build strips the block before inlining, so pages still use diagram.css alone. Each file pays ~850 bytes of repeated CSS for it. If the theme changes I update the block and find-and-replace it in the files, which is rare.

This is the first diagram as Obsidian renders it on my iPhone with the block in place.

Linted like code

The rules are in one versioned file (svg-lint.rules.json) which only contains data and parameters, so the site build and the vault check read from the same list. A trimmed excerpt:

{
  "version": 3,
  "viewBoxWidth": 480,
  "maxBytes": 20480,
  …
  "forbiddenAttrs": ["style", "fill", "stroke", …, "xlink:href"],
  …
  "classVocabulary": ["dg", "dg-box", "dg-box-accent", …, "dg-set"],
  …
}

xlink:href is on the forbidden list because Inkscape and most editors still write it. There’s a fixtures directory of passing and failing SVGs (each failing fixture named after the rule it breaks), and the site build runs the lint on every diagram (inline or embedded), with a failure naming the file and the rule. The lint requires the fallback block byte-for-byte right after </title> (and forbids it in inline SVG), and a drift test fails if the block stops matching diagram.css and tokens.css.

The cool part is there’s a second copy of the lint, in Python, that checks the note in my vault before it publishes through Enveloppe. Since both run every test fixture.. any drift becomes a failing test on both sides.

Checking what the picture looks like

Element and attribute checks tell you a diagram is clean, not that it’s readable (two diagrams passed the old lint and were unreadable), so the lint’s layout rules (v2, now merged) look at the picture itself.

An arrowhead drawn as its own loose shape fails the build:

<line class="dg-line" x1="96" y1="32" x2="200" y2="32"/>
<polygon class="dg-arrow" points="200,24 216,32 200,40"/>

The same arrow passes when the line points at a <marker> with marker-end:

<defs>
  <marker id="arrow" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto">
    <path class="dg-arrow" d="M0,0 L8,4 L0,8 Z"/>
  </marker>
</defs>
<line class="dg-line" x1="96" y1="32" x2="208" y2="32" marker-end="url(#arrow)"/>

A marker-end pointing at a marker that isn’t in the file fails, as does text outside of the viewBox, and a label centered in a box without text-anchor="middle".

Other things only warn, like an estimated label width (chars * font-size * 0.6) overflowing its box or the canvas, a corner radius other than 4, or rect and circle geometry off the 8-unit grid. Lines and text are exempt from the grid (connectors meet box centers, labels sit on baselines).

A problem fails the build on a live post but only warns on a future-dated one (so one broken draft doesn’t hold up other posts), and CI fails on warnings, so I get alerted when the post merges, days before its go-live build would break.

It’s just a text width estimate and a few coordinate checks, the same move as a type checker, turning “looks off to me” into a rule with a failing example. That matters more now that AI is authoring and maintaining a lot of this stuff, since “make it look nice” is a vague target for a model and a rules file with passing/failing fixtures is one CI can check.

The lint only checks the “how” of the diagram. The “what” is still captured by the first question on the style guide checklist: does it say something the prose can’t? If not, the diagram is dropped.

From a cron job to Sentinel: a resident agent that checks my infra

Mon, 28 Sep 2026 00:00:00 GMT

Sentinel started out as a nightly cron job. Back in April, it was a systemd timer that ran at 07:00 every day, executing claude --print on the infra-health skill, which in turn posted the health report. The first night, it failed after zero seconds because the model I had set for the job was its own fallback model, so the CLI refused to start. Fair enough, RTFM moment. I switched the fallback to haiku, and the second night it ran.

The nightly job was just a one way report. If something needed action taken it was a burden for me to handle. I wanted a full devsecops agent I could interact with, and not only health checks, but actions and approvals for fixes or improvements sitting in my queue to review if needed, with severity escalations built in. It made more sense as its own agent in NanoClaw, and the existing scripts ran on it unchanged.

In May I moved it to its own agent group in NanoClaw, a small agent runtime I run at home. Upstream NanoClaw and my fork are on GitHub. It’s been renamed twice now, from Ranger to Warden to Sentinel, and its mandate has expanded from health checks to remediation and deploy validation.

Switching from a nightly report to an always-on agent raised the question of cost. I realized that if I woke the model every 15 minutes just to confirm everything was fine, most tokens would be spent on green runs requiring no judgment.

As a token savings play and for speed and efficiency, I have automated non-LLM checks in bash every 15 minutes, and they only escalate to the LLM when something actually fails. NanoClaw exposes a wakeAgent property as part of scheduled task exit conditions, so that if any automated check fails, the agent can wake (wakeAgent = true) and examine the script output. The model is there for judgment, escalation and action, not for pinging services.

The script is called Infra Pulse. It has 82 checks across 9 categories (containers, DNS, disks, backups etc.). Several of them read existing container healthchecks. NanoClaw runs it before the agent, and the last line of output is JSON. If all pass, that’s all you get.. the task closes without calling the model.

{"wakeAgent": false}

If any check fails, that same line includes wakeAgent: true plus the findings, injected into Sentinel’s prompt. I decided to wake on any single failed check (instead of confirming, then waking) because, as I noted at the time: “Simpler, LLM decides if it’s noise.” Each finding is suppressed from re-alerting for an hour, so a flapping service won’t keep paging me. The pulse writes a last-run metric to VictoriaMetrics, and Grafana pages me if it goes stale.. a dead pulse otherwise wouldn’t make a sound.

Sentinel is read-only.. it observes and proposes. Any finding of medium+ severity that includes a root cause and rollback path triggers the request_approval tool, spawning an Approve/Reject card in the Slack channel. The fix only runs after approval. A short list of critical-only fixes can run unasked, but promotion to that list requires two past incidents and a seven-day trial where the rule only simulates action.

The TV tuner Pi was an exception I made by hand. My ruling was “tell it to reboot the pi if any issues”, so Sentinel got a restart-then-reboot rule for that device only, operator-approved as an exception to the usual two-incident bar.

The first time I ran pulse, Sentinel woke up with 10 findings, all false positives. Probes were looking for services where there weren’t any, or an auth-protected service returning 401 was being counted down. The house was fine, the checks were wrong. Same-day tuning got that down to 1 finding, and now 401/403 are considered alive. A slower variant: a weekly deep scan had been yellow for months, traced to a stale hand-kept list treating sidecars and exporters as user-facing. Correcting the list (not faking healthchecks) fixed it.

Alerts needed the same treatment. All house alerts go through one relay (built on Apprise), and a second Sentinel task reads new alerts through the same script gate, waking only for ones it hadn’t seen. A noise audit a week later showed 134 alerts in 30 days, of which 68 were from three alerts that got stuck firing and re-notifying every six hours. Approved fixes stopped re-paging for already-acknowledged conditions. Sentinel keeps a tuning log in the runbook, snoozes accepted findings, and the snoozes expire on their own.

Sure there have been false positives and stuck alerts paging me, but the agent has a self learning loop that tunes these out based on observations and feedback.

Since I first rolled this out, I have been extremely impressed with what is possible: automated actionable events surfacing when I fat fingered a deployment, a credential changed for a service, a port left open accidentally, etc. The value has been tremendous. It’s like having a 24/7 team on call to fix things, taking a lot of the server management overhead out.

I’m working on better alert triage and onboarding more into pulse scripts as time permits. And a better “allow list” of actions so I don’t have to manually approve everything.

In June, my AI sysadmin and security agents assessed the whole homelab.

Architecture decision records deserve a release cycle

Sun, 27 Sep 2026 00:00:00 GMT

Over the last several years professionally, I have been close to my customers’ architecture decision record (ADR) practices. In my day job, I play an integral role in authoring and shaping these and reviewing them as they go from draft to published. They are a foundational deliverable for me and I consider them one of my main work products. ADRs are more important than ever for human and AI consumption. In financial services they double as audit trail and change-management evidence (which is part of why the review and release discipline matters so much there), but the principle applies everywhere. Version control them and run them through a proper release and review cycle, similar to the SDLC or exactly like it.

What an ADR actually is

Michael Nygard proposed the format back in 2011. Five parts (title, status, context, decision, consequences), one decision per record, and short enough that people actually read them. Nygard put it well - “Nobody ever reads large documents, either.”

The status field is important. An ADR is immutable once accepted. You don’t go back and edit it. When a decision gets revisited, you write a new ADR that supersedes the old one. That history lets you trace why the system looks the way it does, not just what it looks like today.

Thoughtworks put lightweight ADRs in Adopt on their Technology Radar, recommending teams store them in source control rather than a wiki. Their take: “for most projects, we see no reason why you wouldn’t want to use this technique.” There are good templates and tooling collected at adr.github.io, and AWS Prescriptive Guidance publishes a solid guide on using ADRs for technical decisions.

Treat them like code

When your ADRs live in the repo, the cross-functional conversation happens in one place. Open a PR when you propose a decision and security, ops, application teams can weigh in right there on the decision itself before anything gets accepted. That conversation was going to happen anyway in Slack or a meeting, but now it’s captured and searchable instead of scattered.

What releasing an ADR looks like

I’ll give an example: consider a situation where you are deciding whether to adopt a queue for async processing between two services. You write an ADR with status proposed and open a PR. The usual reviewers weigh in there. Once the PR merges with status accepted, that decision is released. Months later if the team moves to a different approach, a new ADR supersedes the original, and the old record stays in the repo as history.

A CODEOWNERS entry keeps the right people in the review loop:

docs/decisions/    @architecture-team @security

You can keep a lightweight changelog in the decisions folder (even just a list of ADR numbers, dates, and one-line summaries) so anyone can scan what changed and when without reading every file.

For agents, point them at accepted ADRs only. A line in your CLAUDE.md or AGENTS.md can be as simple as:

Architectural decisions are in docs/decisions/. Follow all ADRs with status: accepted.

That way agents consume reviewed decisions rather than stale proposals or chat transcripts.

Why this matters more now

I run agents on my own projects too. Agents are stateless, so every session starts fresh without any history of past decisions. Git tells an agent what changed but not why. Without good documentation like ADRs in my workflow, I’ve had agents re-propose decisions that were already settled.

The Architecture Without Architects paper calls this “vibe architecting,” where agents make architectural decisions implicitly through prompt interpretation and prompt wording alone produces structurally different systems for the same task. The paper’s proposed remedies include review practices and decision records.

BrainGrid wrote a good piece on ADRs for AI coding agents that gets into why agents need the reasoning behind a decision, not just the final call. And someone in an r/ClaudeCode thread asked how you carry decisions across sessions without dragging along all the chat history - top-voted answer was ADRs.

Human in the loop on architecture

Human in the loop on ADRs is the foundation for successful agentic consumption and for human-to-human, cross-functional collaboration. The reviewed ADR is what your agent should consume. It went through real review and got sign-off from the people who own the decision, which is more than you can say for a prompt or a chat transcript.

I’m taking the enterprise practices I observe at my customers into my own projects and having a lot of success with it.

Why I dropped OpenClaw for a smaller agent runtime

Sat, 26 Sep 2026 00:00:00 GMT

The initial OpenClaw craze was incredible to experience and be a part of. My main agent Otto ran on it daily via Discord, and his personality and memory were stored in OpenClaw workspace files.

Unfortunately, the project itself was very unstable and very difficult to untangle and patch when issues came out and keep parity with the mainline. Their release cycle and testing strategy got increasingly burdensome for me as new features were gated behind upgrades that were sure to break things.

Reapers and memory caps

The most glaring example of this was a memory leak. A plugin would continue to spawn agent sessions and never clean them up. After 19 hours, I had 1,163 orphaned processes eating 43 GB of RAM on a server sitting at 30 of 31 GB used. The upstream fix to implement session time limits is still an open issue. I contained it by putting a hard memory cap on the service and writing a reaper on a 10-minute timer that killed processes like these older than 30 minutes. It worked, but I couldn’t help but wonder if I’d fixed anything or if I’d just built a fence around it.

Upgrades weren’t any easier. I had four different versions of OpenClaw over the course of ten days or so. One didn’t like my existing streaming configuration because the schema had changed underneath. Another had a plugin that flatly refused to load, even though the service was still reporting the old version number.

So I patched my way forward. I wanted to use Gemini on Google Vertex, but OpenClaw’s Vertex provider was just an alias for the regular Gemini one, which was making requests to a URL that returned a 404. So I wrote my own provider plugin. That was maybe 250 lines of code and four hours of work. Then all three of my model fallbacks broke at the same time, so I emptied out the fallback chain and made the failures loud. Each fix kept me on a platform that needed the next fix.

If you spend more time fixing your automation and harnesses than enjoying what they can do to automate your life, sometimes you need to take a step back and evaluate what you’re running.

One migration day

Thankfully I came upon NanoClaw, which is a smaller project and a very lightweight runtime (also extremely easy to modify), and so far I have been really happy with it.

The migration itself took me one day (about five hours of work). I took inventory of what I had set up in OpenClaw (five scheduled jobs, all of which were already failing.. a clear sign that I was spending more time keeping the old platform alive than actually using it). I wired 11 Discord channels to Otto, merged the separate personality and memory files into a single memory file, and made the changes in my public fork.. the same framework I described in Always be curious.

Each agent in its own container

Each agent group is its own container with its own memory and tools, so adding “Sentinel” weeks later didn’t touch my main agent “Otto”. This security isolation is indeed a huge feature: agents running in containers with specific grants to the things they need only, and strict agent-to-agent allowlists as well. And when something broke I could fix it in my fork instead of wrapping fixes in reapers and memory caps.

I applied the same thinking to notes: Otto has a read-only vault mount, with one writable inbox folder beneath it. The inbox-only write rule is enforced by the filesystem, not just instructions. An agent can only reach another agent through a deliberately wired link.

Tip · Read-only mounts don’t forget

If an agent should only write to one folder, mount everything else read-only. An instruction can be forgotten or talked around in a long session, but a read-only mount simply refuses the write.

I could also interact with a smaller community via Discord and found the community behind it was really intelligent, not only from a forward-thinking agentic perspective but from a security and operations perspective.

Building on it

I have built a memory layer based on OKF (Open Knowledge Format) and continue to build on top of NanoClaw. My custom agent runtime leverages Claude Code containerized in tmux sessions with a custom messaging bridge, roles and tools, and it has been very stable. Now that I also integrated Codex and Gemini agents into the fold, a few more specialist agents and further stability enhancements and quality of life improvements are on the horizon.

Before NanoClaw, I used Memento as my agent memory, which is just my Markdown notes in Obsidian. I’m still using plain Markdown for my OKF layer, but each agent group now has its own memory tree, with one concept per file. Next up: Sentinel, the agent that watches my infra.

Handoffs and carry: keeping continuity across agent sessions

Fri, 25 Sep 2026 00:00:00 GMT

Good ideas show up in the middle of building something, but chasing them wrecks the session’s context and flow. So I log them as carry blocks, and they feed the backlog where I can prioritize them properly.

Before I started using carry blocks, I used to pass open items as prose in my session handoffs, instructing each session to copy the list into the next handoff. I didn’t verify that they did. I should have checked instead of trusting the instruction. At one boundary, a backlog that had accumulated over months got squeezed down to 4 lines, losing 3 HIGH importance items.

The carry block is the check. I have one note per project in my Obsidian vault and read/write it through Memento, which already handles handoffs. Open tasks only.. facts stay in Claude Code’s memory files. An item leaves the Open list only 3 ways: done, dropped with a reason, or promoted to the backlog. Anything else, the validator rejects the commit removing it.. a silent drop becomes an error.

I considered putting a TODO.md in each repo, but that broke two of my rules: tasks live in the vault, and public repos can’t hold infrastructure details. Plus, any project without a repo wouldn’t have anywhere for it. I also considered limiting each block to 15 items (I’ve since removed that limit) and breaking them into sections by workstream, but having every agent see everything is more valuable than saving a few tokens.

The first block for each project was seeded from the full session history. home-infra (my homelab project) alone turned up 62 open items from March-July that were documented nowhere. It exposed prose handoffs leaking for months, well before the one known bad boundary.

The day after I shipped the blocks, I ran a session in one of my projects to see what a fresh session would get. Claude Code only passes the output of a hook inline up to 10,000 characters.. beyond that, the model only sees a short preview. Four of my eight blocks exceeded that limit, so they arrived at the model with zero open items. (My earlier check that supposedly confirmed all items were arriving was testing a simulated session start, and I’d believed it.)

Re-imposing a cap would have satisfied the limit, but home-infra would have had to drop ~100 of its 126 items. I settled on a summary plus the 10 newest items per session, with the full file readable on demand. A live check after the fix showed the session searching the file for an item outside the window.. exactly the behavior the summary was supposed to enable.

Checking the hook’s output isn’t the same as checking what the model saw. The rule now is to verify from the session’s own transcript. It was the best fix, since the limit was there whether I liked it or not.

This needs to work when I have multiple sessions open in the same project, as described in Running six to nine Claude Code sessions at once. If one session tries to create a commit based on a stale read of a block, it gets rejected.. one session can’t overwrite another’s changes.

The 60+ recovered items showed how much was getting lost before. Keep feeding the backlog from carry and doing a weekly prioritization pass. Using carry blocks to prioritize and keep me honest about follow ups and check-after dates while I let something soak for a while or need to step away from a project for a few weeks.

This opens up future for parallel workflows and side sessions during the main thread, using worktrees or similar.

My whole dev setup fits in my pocket

Thu, 24 Sep 2026 00:00:00 GMT

The sessions run on the server anyway. With mosh and tmux I can check in and answer a permission prompt from anywhere instead of being stuck at the desk.

Mosh is what makes this setup work on a phone. It’s based on UDP, so the session survives switching from Wi-Fi to cellular and the phone going to sleep, both of which would drop a plain SSH connection. On the server, every SSH login attaches to the same tmux session, so the phone shows me the same windows I left open on the laptop. (My setup at the desk is Terminal setup tmux Ghostty and the 1Password CLI.)

A push means the session wants something: an answer, feedback, or a response. Claude Code hooks send a push when a session asks for permission, asks a question, or finishes a task. The hook is installed from a Brewfile in my dotfiles, and paired per new machine with a one-time script, so any machine running a session can reach the phone. (My dotfile distribution is chezmoi one source of truth for every dotfile.)

I can also send screenshots from the phone into a session. For example, I was tracing the floorplan of our house and had some marked-up scans on the phone, so I sent them straight into the session as source images. They came in as temporary files, so I copied them into the repo before a reboot could lose them.

I do most of my work with my iPhone and moshi.app to be honest. I tried a lot of iOS SSH apps before I found Moshi, and one question I always had was: could a self-hosted solution ever match the experience? One app was real mosh, but no push notifications. One self-hosted web terminal claimed mosh but was just typing a mosh-server command in a regular SSH shell (still WebSocket in the browser, not UDP). Another app I tried and disliked. Moshi is the only one I’ve found that lets me swipe to switch tmux windows.

I open Moshi, swipe to that window and answer it. The swiping is the fiddliest part of my setup: Moshi finds windows by reading the tmux status bar, and my custom theme had added enough extra text that the swipe stopped working for no obvious reason. A cosmetic theme isn’t the first thing I’d blame for a broken gesture. I ended up hiding the theme text only when the phone is the client (a phone connection sets MOSHI_CLIENT=1, and tmux clears the left and right sides of the status bar when it sees that). My laptop keeps the theme, and the swipe has been solid on the phone since.

It’s been better than using Claude remote control since I like to be in the terminal, and my agents can a2a message each other in tmux and I can focus on outcomes and reviewing work. How that many-session setup runs is in Running six to nine Claude Code sessions at once. I’ve recently started using Claude artifacts to review and approve changes and plans more, this has been really nice on mobile.

Edited on 2026-10-24 to add reviewing plans in Claude artifacts.

LG laundry Live Activities (and why I care about wash temperature)

Wed, 23 Sep 2026 00:00:00 GMT

The lock screen card shows the current phase of the washer and a countdown to completion (4:13 PM on the first day I checked). No app launch required, and the card persists throughout the entire cycle.

My LG washer + dryer were already in Home Assistant via the native LG ThinQ integration, so I had the cycle data. I chose the native integration over the wideq library because wideq didn’t support real laundry, and the official one gets cloud-pushed updates from LG and exposes all laundry entities.

I created one card per machine, leveraging the Live Activities support in Home Assistant Companion app notifications. The card tracks the cycle from start to “Done - unload”. The Washer Done card also includes a reminder to move the load to the dryer. Tapping the card opens the Laundry dashboard in Home Assistant. Only one phone receives the card; the other phone gets a regular notification when the cycle finishes.

Note · One push per state change

Home Assistant sends an update to the card only when the machine’s state changes, never on every tick, because iOS throttles frequent Live Activity pushes and then starts dropping them. The phone counts the time down by itself between updates.

I assumed the finish time was fixed once the LG washer started, so initially I only updated the card when the phase changed. But during the first wash, the washer estimated 45 minutes when powered on, then changed to 149 minutes after weighing the load. So I added a second update type to re-send the card if the finish time differs significantly from the displayed value.

Done means done (eventually)

Getting the card to stick around for five minutes after the cycle and then clear itself required several rounds of fixes. The very first version only showed “Washer” and a 1% progress bar, without any phase. A more interesting bug was that the Done card (intended to last five minutes) vanished after about 30 seconds. It turned out that the washer powers itself off about 30 seconds after finishing, and Home Assistant was treating that power-off as a new state change, cancelling the Done card mid-wait. I hadn’t accounted for the machine doing anything after the cycle ended. So I learned to ignore the power-off immediately after cycle end, and now the Done card lasts a full five minutes.

Tip · Power-off cancels Done card

The washer turns itself off about 30 seconds after it’s Done. Home Assistant sees this as a new state change and cancels the Done card. We need to ignore the power off immediately after a cycle ends.

The stranger bug was when the card counted up four hours and survived two attempts to clear it manually. It turned out that when I pushed twice in less than a minute, each push created a new card. Home Assistant stores one token per card, so the first card got orphaned.. I couldn’t update it or end it. How do you clear a card nothing references? Turns out: you don’t. You prevent it from being created in the first place. Now every push is delayed for longer, so that if I make a burst of changes, only the final state ever makes it to the phone.

Tip · Rapid pushes orphan cards

If I push twice in less than a minute, it will create a new card for each push. Home Assistant only stores one token per card, so there is no way to update or end the first card. Delay every push so that only the final state makes it to the phone.

The hot-water prime

The part I’m proudest of has nothing to do with the phone. The washer wants hot water for the first fill, then cold water for rinses. But the hot and cold lines are bridged under the far fixture, so the recirculation pump running also warms the cold line, which the rinses draw from.

So when the washer is powered on, Home Assistant presses the water heater’s hot button once, forcing a single recirculation cycle. It waits for the cycle to end, turns recirculation off, and holds off until the wash ends, at which point it releases and recirculation returns to its normal schedule. The first time I ran a real wash, the outlet temperature rose from 114.8 to 125.6 °F in about a minute, and the water going into the first fill was 126 to 129 °F.

Wash temperature is important for color protection, whitening when it matters, stain treatment of certain types of stains (hot on some types of untreated stains may set the stain), and surfactant & enzyme performance.

Tip · LG hides the temperature setting

LG doesn’t expose the wash temperature setting to Home Assistant, so it can’t tell a cold-only cycle from a hot one, and a cold wash still gets the priming minute. And if the house empties and someone comes home during a wash, recirculation can resume, so one rinse may run warm.

Priming the washer is just one piece of a larger project I have around the water heater and recirculation loop. I’ll talk about the rest of the setup in my next post: Hot water at the tap with Home Assistant.

A lead session and a peer per project in Claude Code

Tue, 22 Sep 2026 00:00:00 GMT

I had a list of 21 things to do across a bunch of projects. Doing them one at a time would have taken so long. So I opened one Claude Code session at the root of my vault to act as the lead, while peer sessions were already running in the project repos, some of them in the middle of their own work.

Each project already has its own repo and context, so one session per project made sense, with one lead keeping the ledger and checking the work. The separation of duties is enforced at the user scoped claude.md file with instructions on how to spin up a new session in tmux if need be.

The ledger is a single vault note, each row includes the owning session and a status. By the end of that week, the status page showed nine tmux windows, one for each project. Two of them weren’t Claude but Codex and agy, but they sat alongside the others. Underpinning it all is the machine-wide setup I described in the previous part of this series, My Claude Code setup one chezmoi source for every machine.

Having the specialist agent for each project be responsible for its own project or delegation to another project agent actually seems more token efficient and quality efficient because of the domain knowledge the agent has.

The sessions also cross-checked each other’s work. Hours after Memento shipped its carry block code, the domains session raced two carry commands and lost the update in 60 of 60 runs. Both were writing the same temporary file, without a lock. The next Memento release had an exclusive lock, a separate temp file for each write, and a race test that failed on the old code. Would I have caught that on my own before it ate a carry block? The lead got caught making mistakes too, and caught the peers’ mistakes in turn. For more on the memory tool, see Memento agent memory I can read edit and prune.

Claude code has agent to agent messaging now (session messaging) which has proven to be a really effective cross agent collaboration mechanism. Sessions can find each other and talk directly. This is covered with the messaging tools in the Claude Code subagents docs.

This pattern played out in real life when I rolled out mini-splits into Home Assistant at home. I spun up three cooperating sessions (firmware/hardware, DHCP/Wi-Fi, Home Assistant adoption/cutovers) and got nine heads live in two days (see Nine Mitsubishi mini-splits in Home Assistant for 15 per head for the full build).

The rule that I approve things in each session’s own window stays. But the decision rights and approval gates are a whole other post (see Running an agent fleet like a small team).

Next is getting NanoClaw agents to message back and forth with host sessions. Each host session now registers its name, its tmux pane and whether it is busy, but the NanoClaw agents can’t reach that registry yet. This brings my always on agent fleet into my CLI agent fleet.

Edited on 2026-10-24 to add the mini-split rollout and the host-session registry.

So Good They Can't Ignore You and the passion that shows up later

Sat, 19 Sep 2026 00:00:00 GMT

Passion comes from being really good at something. And you only get really good at something by practicing it a lot and investing in it.

This is the core idea in So Good They Can’t Ignore You by Cal Newport, published in 2012. In that book, Newport argues that “follow your passion” is bad career advice. Instead, he says it’s better to build rare and valuable skills (or “career capital”), because passion tends to follow the acquisition of skills.

So let’s say you’re an accountant, and you’ve spent years hating every minute of staring at spreadsheets. You finally decide to quit for good, sell all your possessions and go to start a farm.

It’s most likely going to be disastrous. You’re not really prepared. You don’t know how hard it is. You’re falling in love with this sort of fantasy of what an authentic existence is, and that it really can throw you off into sort of a spiral of unhappiness trying to find your passion.

The fundamental error you’re making here is that you’re assuming that there’s an ideal life out there for you, just waiting. But as Newport points out in a 2018 post, most people don’t have a clear existing passion to discover.

Most people don’t become exceptional at something because you actually have to do uncomfortable things on a very regular basis over a long period of time.

That’s the whole trick. You can’t avoid being uncomfortable. It’s valuable to be able to do it because most people will quit when practice gets hard before they ever get to the payoff.

It’s okay for you to not absolutely love your job. And it’s okay for you to figure out how to be as valuable as possible within the constraints. Paying your rent and having autonomy and living on your own, I would say are more important than having the dream job when you’re 22 years old.

I think this is the kindest line in the whole argument. It speaks to people early in their career and says it’s okay to take a job that pays the rent and get good at that first.

It almost feels like he’s gone full circle, where he’s relabeling passion for mission.

Mission is the final rule of the book, and one that Newport says we only earn after years of building career capital. But is mission really that different from passion, or is it just passion that waited its turn?

Terminal setup: tmux, Ghostty and the 1Password CLI

Fri, 18 Sep 2026 00:00:00 GMT

I want the same tmux on every machine so muscle memory carries over, a terminal that’s fast, and secrets kept out of my dotfiles.

My tmux configuration is a single template in my chezmoi repo. Every machine has identical bindings and history after applying the configuration. The only per-host difference is the tmux status bar color: my laptop is blue, my home server is teal, and the always-on Mac is purple. I can glance at the bottom of the window and know which machine I’m on before running commands.

I’ve been running both Ghostty and iTerm2. The first problem I had with Ghostty was with SSH. My sessions to servers broke because they complained about unsuitable terminal: xterm-ghostty and echoed everything I typed in the shell. This was because Ghostty reports its own terminal type, but servers that don’t know about it won’t have a terminfo entry for it.

Ghostty’s terminfo page offers two solutions: either install the Ghostty terminfo on every server you connect to, or configure Ghostty to report a terminal type that all servers know about. I decided I didn’t need Ghostty’s own terminal type on the servers I was connecting to (which I was mostly reaching via tmux anyway), so I went with option 2. I lost a few Ghostty-only extras (styled underlines, extended key reporting), but I rarely need those in my day-to-day work. One line in my Ghostty config:

term = xterm-256color

The bigger round of tmux changes came from a research doc on Claude Code terminal configuration I’d compiled the day before. The agent checked my config against the doc point by point, read me the gaps it found, and I approved every change (“we can always revert”). Two mattered most: extended keys so Shift+Enter gets through to Claude Code (the rest of that setup: My Claude Code setup one chezmoi source for every machine), and a much longer scrollback history.

That same day, I added the tmux-resurrect and tmux-continuum plugins to restore sessions, so my Claude Code sessions would come back after a restart. But Continuum’s auto-restore restarted the saved claude processes without attaching a terminal, and every subsequent SSH attempt to attach failed with open terminal failed: not a terminal. Turns out restoring a shell is easy, but Claude Code needs an actual terminal at startup.. something I hadn’t considered when picking plugins. I had the plugins removed, and tmux has been running without them since.

1Password only pulls a token when something needs it. The agent that calls the API runs the 1Password CLI op command to read the token at the time it’s needed. Nothing is stored in a dotfile. The service account the agent layer uses only has access to a single vault. I handle secrets for services running on the home server differently, as I wrote about in 1Password CLI and Connect secrets without env files.

I’m exploring using 1Password environment files and the new 1Password agent integration in my workflows, but so far this scoped down vault access with op CLI has been extremely useful for secrets management in my agent layer.

With tmux mouse mode on, Shift-drag in Ghostty gets normal text selection back. I like the speed and limitless config-ability and customizations Ghostty offers. I’ll be sticking with it for now.

My Claude Code setup: one chezmoi source for every machine

Wed, 16 Sep 2026 00:00:00 GMT

All of it is worth showing, but the chezmoi piece is the base. Every machine gets the same Claude settings, skills and statusline, and my live choices don’t get reverted.

The Claude settings file is a chezmoi modify_ script instead of a plain template. That way chezmoi only patches the keys it owns in settings.json and leaves hooks and plugins to their own installers. The model and effort level get seeded only if they’re missing. (That rule came from a bug: every apply used to revert a mid-session /model switch to the template default, now my pick survives the next apply.) The bigger picture is chezmoi one source of truth for every dotfile.

My claude.md file has references to how to edit this and how to push to git so my agents know how to update configs and render diffs and applies when I ask of it.

When I want to change the Claude config, it’s usually just a request in an already-open session. The agent edits the chezmoi source, shows me the diff, and applies & pushes after I read it. All my non-plugin skills live in the same repo, so any new skill I add gets instantly available on every machine the next time I apply.

Most of my skills come from Uberpowers, which is a fork of superpowers by obra, via pcvelz’s fork. It’s where I keep the working loop that turns an idea into brainstorming, into a spec, into a plan.. all before writing a single line of code. In May I broke syncing with upstream permanently because I have my own roadmap, but the foundation is still there.

Sessions got heavy, and the number one thing on my list was finding the source of the context bloat. I had the agent measure every block it was injecting over 30 days of session logs. Turns out I was loading about 15,000 tokens of stuff before I’d even typed a word, and most of those blocks weren’t earning their place. The model wasn’t actually using hardly any of it. I cut the per-turn injections, removed a lookup hook that was searching memory on every prompt, and killed nine skills the model had never once used.

I decided full context was more important than token savings, so I kept one block intact: the per-project carry block from Memento. It tells a new session where the last one left off, with no size limit. I also kept the hooks that push prompts to my phone, but that’s a whole other post (see My whole dev setup fits in my pocket).

I told the agent “Think outside the box, you’ve concluded too quickly” the day my clicks died in Claude Code inside tmux on my home server, but on my laptop with the same settings, it worked. Turns out: I had a mouse setting sitting idle in my dotfiles for 2 months, the flag only exists from Claude Code v2.1.195+, and my home server had updated past it, activating the setting silently. Never thought a no-op setting could bite me later.

The agent’s first theories were wrong, but it wanted to save one to memory before it was proven. I told it “stop saving it to memory, I am not ready to conclude that yet.” A simple diff of the Claude environment variables across the two machines revealed the flag. Setting it to 0 brought back clicks. The missing version note is tracked in issue #71687.

In every session I have a statusline from cship which reads the data Claude Code passes to a statusline and shows me how full the context window is, how close I am to usage limits, etc. My fork adds the weekly Fable limit which Claude Code doesn’t pass, so my fork queries the usage endpoint.

The usage call had another surprise in store for me: I had many sessions open, and they were all retrying the endpoint every 30 seconds after it failed, which in turn meant they were collectively keeping the account throttled. So now my fork backs off for five minutes after a failure. This only surfaced because I was running many sessions at once, which brings me to the next part: Running six to nine Claude Code sessions at once.

Propane from a water heater's gas meter (and a 2.8x correction)

Tue, 15 Sep 2026 00:00:00 GMT

The dial on the propane tank says it’s at 60% and the estimated tank level on the Home Assistant dashboard says it’s at 71%. The monthly meter next to it says we used 95 gallons in September, but it was actually about 2 gallons. All of the numbers come from the lifetime gas counter on the Navien water heater. Fixing them revealed that the water heater is only about a third of the propane leaving the tank.

I built the propane tracker the day after I put the recirculation on a schedule, so I could see how much it affected our gas usage. You can read about how I got the heater into Home Assistant in the first place in Wiring our Navien water heater into Home Assistant and Hot water at the tap with Home Assistant. The Navien ESPHome component exposes a counter of lifetime gas used, in therms. I set up a template sensor sensor.navien_propane_used that multiplies therms by 1.0929 to get propane gallons (a gallon of propane ≈ 91,500 BTU), and marked it total_increasing so Home Assistant treats it as a count-up-only meter. Then I put two utility meters on top of that, one that resets every day and one that resets every month. I set them up with ha-mcp from my own spec.

Since the tank is 120 gallons, that means that each percent on the dial is 1.2 gallons. I start the estimate at the dial reading after delivery (77% when I set it up) and subtract the gallons used since then divided by 1.2 to get back to a percent. The re-anchor script just asks me for the new dial reading after each delivery. I set up an automation to alert me when the estimated amount gets below 35%.

I wanted to include the propane in my Energy dashboard, but the gas source doesn’t accept gallons as a unit (the gas documentation says it accepts cubic feet, CCF, MCF, cubic meters, liters, and energy units). So I created a liters twin sensor.navien_propane_used_l which I use to feed the Energy dashboard (on US units, HA displays cubic feet anyway).

From the beginning, the numbers seemed low. In the first six days, the tracker was running at a rate of about 4.8 gallons per month, the lifetime counter on the heater implied about 2.4 gallons per month, and our average propane deliveries were about 11 gallons per month. So I expected the estimate to be optimistic, but I had no way of knowing how wrong it was.

The phantom 95 gallons

The first thing that went wrong was the counter itself. One evening in early September, it reported unavailable for about seven minutes, followed by a single 0.0, then the real value two seconds later. Unfortunately, my template passed the zero through as a valid value, and the combination of total_increasing and the drop caused a meter reset, so the jump from zero to the real value was interpreted as new consumption. All downstream meters double-counted the heater’s entire lifetime. The gallons sensor jumped by 92.7 gallons in a single hourly long-term-statistics bucket, and the therms counter and liters twin jumped by the equivalent amount in their own units. The 95 gallons the meter showed for September were made up of this phantom 92.7 plus another two or so actual consumption.

An audit of my HA configuration weeks earlier had flagged these templates because I was using | float without a default, which logged a template error every time the heater’s ESPHome device dropped off. The audit noted “don’t fix with float(0) (that registers as a meter reset).. use an availability template and move them to tracked YAML”. I’d written down the fix but hadn’t gotten around to it when the bad zero hit.

The guard and the repair

I approved the fix on two conditions: it had to be tracked YAML, and it had to survive a real restart. The propane templates moved out of UI helpers (which are stored internally in HA and invisible to tracked config) into a template: block in configuration.yaml. Same entity ids, so the statistics stayed continuous, and each has an availability template rejecting non-numeric and values less than or equal to zero.

Why not just use float(0) as a default instead of an availability template? Because the meters don’t ignore 0. I wrote down “0 is exactly the failure; unavailable is ignored.” Utility meters and statistics ignore unavailable values but take 0 at face value, so a default of 0 would turn every dropout into the reset behind the phantom 92.7 gallons.

But the history needed repair, too. utility_meter.calibrate set the monthly meter to 2.33 gallons. Long-term statistics got recorder/adjust_sum_statistics (same operation as Developer tools > Statistics; recorder docs) to remove 92.70 gallons from the gallons counter’s bad hour and 84.81 therms from the therms counter’s. The liters twin was awkward: its statistics are stored in cubic feet even though it reports liters, so adjustment_unit_of_measurement had to match the stored unit, making the correction 12.39 cubic feet. When I restarted the Core that evening, both the guarded templates and the repaired numbers came back as expected.

Only a third of the draw

With the counter clean, the remaining difference between dial and estimate was real. Over the 46 days since the anchor reading, the dial had dropped 17 points (about 20.4 gallons of the 120-gallon tank) and the heater had metered 7.28 gallons. The gas range is on the same tank but unmetered, so the heater was only about a third of the tank’s outflow.

	Tank dial	Heater lifetime counter
Anchor after the delivery	77%	86.81 gal
46 days later	60%	94.09 gal
Used	17 points, about 20.4 gal	7.28 gal

The dial is the credible number: 20.4 gallons / 46 days ≈ 13 gallons/month, and it matches the deliveries’ ~11 gallons/month. But the uncorrected estimate was badly late: at the metered rate it would take ~10 months to reach 35%, not the under four the dial was on track to do. The low-tank alert would have fired ~six months after the tank really hit 35%.. no backstop at all.

Now the estimate multiplies the metered gallons by a correction factor input_number.propane_meter_correction_factor = 2.80, so that it reads 60.0% just like the dial. (The factor is the drop in the dial since the anchor, times 1.2 gallons per percent, divided by the gallons the heater metered over the same period.)

To keep the factor honest, I added a new script.propane_log_gauge_reading which stores the dial reading alongside the heater-metered usage since the anchor, and writes a logbook line with the implied factor. It runs whenever I read the dial, and before each delivery. The re-anchor script now also logs the discarded estimate, and the Water Heater dashboard tiles show the latest values. The logbook is purged per recorder retention, so a durable copy of each reading goes into the gauge log table in my notes.

The Grafana Home Assistant telemetry dashboard, which is fed by VictoriaMetrics, has had propane panels since I built the tracker: one for daily use (the one-day increase of the gallons sensor) and one for the tank estimate on a 0-100 scale, with red below 35%. That makes a step down to the dial value easy to spot.

Gotchas

The Energy dashboard won’t take gallons for a gas source.. a liters twin feeds it, and Home Assistant shows that twin in cubic feet on US units.
A single 0 from the counter looks like a meter reset to a total_increasing sensor, but unavailable is ignored.. so the guard rejects zeros instead of defaulting to them.
Templates created as UI helpers are invisible to tracked config.. the logic-bearing propane templates live in YAML now.
Long-term statistics can be stored in a different unit than what the sensor shows, and adjust_sum_statistics needs the stored unit passed explicitly.
After the templates moved to YAML, the Energy dashboard’s helper entity came back with a _2 suffix.. I removed the orphan and renamed the new one back.
A Core restart restored the anchor inputs that same day.. in the logbook it looked like a re-anchor until the history showed identical values before and after.

One pair isn’t enough

I fitted the factor from just one pair of readings, and the range’s share of the tank varies seasonally (cooking is about flat, hot water goes up in winter), so the factor drifts. The estimate is only as good as the latest dial reading, so I’ll read it monthly and before deliveries, and refit the factor when the implied value shifts. One pair fits both a multiplier (2.8x on heater use) and a flat offset (about 0.28 gallons/day for the range), so I need a second pair of readings to tell them apart.

Every notification my house sends goes through one relay

Sat, 12 Sep 2026 00:00:00 GMT

When I started, I wondered if I could see every house notification being sent from the sender to my phone, but nothing I had could answer that. I had more than a dozen different things sending me alerts (Grafana rules on my metrics database, Home Assistant, my router, disk and RAID checks, my UPS, Uptime Kuma, Backrest), and they were all calling Pushover in different ways, and none of them were logging what they were sending.

I could either set up a Pushover Open Client mirror (which would log everything that made it to my phone, but not change anything on the senders’ side) or route all alerts through an apprise relay, which I already had running on my home server for autoheal’s container alerts. I went with the relay because, in addition to logging, it can also route messages based on their urgency and fan them out to multiple destinations. The relay is called apprise-api and is a small web server that wraps the apprise library.

I decided that everything needs to go through the relay (no direct-to-Pushover fallback, not even for Home Assistant and the router) and spent about a day cutting over. A search through my repos turned up most things that talked to Pushover, but a sweep of my running services turned up 3 that I didn’t know about in git (including Uptime Kuma and Backrest). Turns out my repo wasn’t the complete picture of everything that talked to Pushover after all.

My favorite part is the routing: the sender chooses a relay key based on the urgency of the message. I have keys for everyday homelab, critical (Pushover’s siren sound), low-priority (Grafana query failures), emergency, and a separate life-safety key for Home Assistant. Each key maps to Pushover (at the right priority), a Slack channel, and a log. Since my router doesn’t have bash or jq, I wrote a little BusyBox shell helper.

I mapped the notify calls from Home Assistant to keys based on priority. I approved the switch with a “yes to both” and the emergency test page that followed made it to my phone, quotes and line breaks intact.

Each alert gets sent to Loki as a single line of JSON containing the title and message, tagged with the key that was used to send it. This answers my original question.. I can use Grafana Explore on the alerts log to see every notification the house has sent me, in order.

The log also feeds my monitoring agent. Every 15 min, the host check looks at the log first, so it only wakes the agent if there’s a new alert that the agent didn’t send itself. A quiet house costs nothing. The agent decides whether the alert is just noise or worth a look, but it will never page anyone.

The tradeoff is that my home server is now the single point of failure for alerts. If it’s down, nothing gets delivered, including the router’s page that the server is down. Apprise drops a failed send without retrying, and network-problem alerts are the most likely to fail this way. I have one status check outside the relay that watches the relay.

Next up is a retry queue in front of Pushover. So if a send fails because of a network drop, it’ll just wait and then go out when the connection comes back up instead of being lost.

Forcing every DNS query through AdGuard (and the LG TV)

Thu, 10 Sep 2026 00:00:00 GMT

While browsing the query log of AdGuard (which I set up in AdGuard Home and the query-log shipper), the LG TV in our living room quickly caught my eye. It made about 470 queries per week to LG’s ad, content-recognition, and tracking hosts, but my main blocklist HaGeZi Pro only matched exactly 1 subdomain. I assumed that the blocklist had me covered for the TV, but the device that is most interested in watching me was barely touched.

First, I wanted to fix this on the TV only, so I created 5 AdGuard rules (which only apply to that single client) and marked them as important, so no blocklist or allowlist can ever re-open those hosts. This doesn’t affect anything else on the LAN. I deliberately didn’t block the LG firmware/update hosts because I do want the TV to receive updates. The $client and $important modifiers are documented in AdGuard’s blocklist syntax docs.

'||<lg-acr-host>^$client=<tv>,important'
'||<lg-ads-host>^$client=<tv>,important'
'||<lg-fingerprint-host>^$client=<tv>,important'
'||<lg-beacon-host>^$client=<tv>,important'
'||<lg-iot-host>^$client=<tv>,important'
'||<lg-service-platform-host>^$client=<tv>,important'

The sixth rule I added a day later, when I noticed that the TV Home Dashboard was phoning home an inventory of devices to an LG service-platform host. I blocked that too, accepting the risk that it might break LG Channels or the ThinQ home-screen tiles. Again, this is scoped to the TV only, like the other five.

The real weak spot here is that client-scoped rules only work if the client queries AdGuard. A smart TV can hardcode its own resolver, and we know they do when blocking makes DNS look broken. I tested this by making one query from my home server directly to Google’s public resolver, getting the real answer. Nothing on my network was stopping a device from bypassing AdGuard, and I didn’t see any evidence of the TV doing it over IPv4. But what if it did? So my next fix was preventive.

Before I close that hole, I wanted to check who is bypassing AdGuard on purpose. I knew I had set up “direct DNS on some machines” before. That’s only for my Sonos speakers, which use Google DNS via DHCP, and that’s intentional. HaGeZi blocks some endpoints that the Sonos speakers need to be controlled. No one else bypasses it. Had I skipped this step, the intercept would have silently broken my speakers.

Tip · Find deliberate bypassers first

My Sonos speakers use Google DNS via DHCP because HaGeZi blocks some endpoints they need. Without the exemption, the intercept would have silently broken the speakers.

This catches every stray query. My router runs OpenWrt (see OpenWrt to the rescue), and it now redirects any plain DNS query from the LAN to its own AdGuard, regardless of what resolver the device named. The reply gets rewritten on the way back, so the device thinks Google answered. The query shows up in AdGuard under the device’s own address. DNS-over-TLS queries from the LAN to the internet get rejected instead of dropped, so they fail immediately instead of waiting for a timeout. My Sonos speakers are in the exemption set, so they go direct like before. Queries already addressed to the AdGuard replica on my home server go through untouched, so DHCP DNS failover still works. I built this from the redirect and ipset options in OpenWrt’s firewall docs. It’s deployed via Ansible like everything else on my router (see Ansible manages every config on my router and access points).

# fw4 redirect semantics: `dest_ip` is the address to DNAT TO; the ORIGINAL
# destination is matched by `src_dip` (single value, `!` allowed).
- name: Configure LAN DNS intercept (redirect stray DNS to AGH, refuse DoT)
  community.openwrt.command:
    cmd: "{{ item }}"
  loop:
    - uci set firewall.dns_intercept=redirect
    - uci set firewall.dns_intercept.src='lan'
    - uci set firewall.dns_intercept.src_dport='<dns-port>'
    - uci set firewall.dns_intercept.proto='tcp udp'
    - uci set firewall.dns_intercept.src_dip='!{{ openwrt_dns_intercept_passthrough }}'
    - uci set firewall.dns_intercept.dest_ip='{{ openwrt_dns_intercept_target }}'
    - uci set firewall.dns_intercept.dest_port='<dns-port>'
    - uci set firewall.dns_intercept.target='DNAT'
    - uci set firewall.dns_intercept.ipset='!agh_bypass'
    - uci set firewall.dot_block=rule
    - uci set firewall.dot_block.src='lan'
    - uci set firewall.dot_block.dest='wan'
    - uci set firewall.dot_block.dest_port='<dot-port>'
    - uci set firewall.dot_block.target='REJECT'
    - uci set firewall.dot_block.ipset='!agh_bypass'

As proof, here’s a test you can run on any network. I asked Google’s resolver for a blocklisted host and got 0.0.0.0 (which Google would never return). Then I asked Google for a LAN-only name and got the local answer (which only AdGuard could provide). Finally, a DNS-over-TLS query failed fast (connection refused).

dig +short @<google-dns> <blocklisted-host>   # 0.0.0.0 -> intercepted and filtered
dig +short +tls @<cloudflare-dns> example.com # fails fast: DoT refused
fw4 print 2>&1 >/dev/null                     # warnings a green Ansible run won't show

This rule took me three deploys. Ansible ran clean and green, but the firewall chain was missing. It turns out that OpenWrt’s fw4 silently skips a redirect section with a list of source addresses (where I had put the Sonos exemptions), so I had to put those into an ipset instead. And in a further surprise, on a redirect the field name dest_ip is where the query is sent, while the original destination is matched by src_dip.. almost the opposite of what you’d expect. The only command that tells you this is fw4 print, which shows the warnings Ansible hides. Two silent skips in one night mean a green fw4 run proves nothing until you inspect the chain and send a real query.

Warning · fw4 skips sections silently

fw4 silently ignores a redirect section with a list of source addresses, so put them in an ipset instead. When redirecting, dest_ip is where the query is sent, the original destination is matched by src_dip. fw4 print 2>&1 >/dev/null shows the warnings Ansible hides.

My first version was IPv4-only, but the TV quickly exposed that gap: it turns out the TV also has an IPv6 address and was querying LG hosts over v6. Those landed on the router’s AdGuard and were logged. I tested with a query to Google’s v6 resolver and got real answers back, so I had the same hole on another protocol. The next day I added the IPv6 twin: a redirect plus a DNS-over-TLS block. No pass-through, no exemptions.. the router now advertises only its own address over v6.

What I didn’t do: DNS-over-HTTPS still gets through (it uses the normal HTTPS port, so blocking that would break the web; stopping it is a blocklist job, not a firewall job). The IoT network isn’t intercepted (it’s built around talking directly upstream, not running its own AdGuard listener). It shipped with a runbook and a kill switch (one uci line per section, plus fw4 reload).

DNS is limited against the TV, but the TV’s own privacy settings are next: content recognition, viewing and voice information agreements, set on the TV itself. I may also move the TV to the IoT network, once I decide whether that network gets its own intercept.

Photos: iCloud is a sync service, not a backup

Wed, 09 Sep 2026 00:00:00 GMT

All my photos are in iCloud, but as we all should know well.. that isn’t a backup. iCloud is a sync service, so deletions replicate. If I delete a photo on one device, it’s gone everywhere. That’s how it’s supposed to work. But it means that if my account gets compromised, my billing lapses, or something happens on Apple’s side, all my copies can be lost at once. It’s really just one copy. I need a copy outside Apple’s reach that can’t be deleted by an upstream deletion. That’s the photo component of My backup strategy.

What does a copy I own look like? Two copies, built to fail in different ways. Copy one is a plain-file export with osxphotos. It writes regular .jpg/.heic files to a year/month folder tree, with XMP sidecars next to each one. It’s grow-only: if I delete a photo in iCloud, it stays in the archive. That archive gets shipped offsite to Backblaze B2 with the server’s other backups.

Copy 2 is an exact mirror of the Photos library bundle, for full-fidelity library restore. It mirrors deletions, so it’s protected by local snapshots (via sanoid). The two copies have opposite failure modes, and what keeps them apart is the mirror only deletes if that night’s export succeeded. (I missed that gate in the first build: it could have emptied the mirror from the wrong library in the same run that added nothing to the archive.) The fixed version is short:

# BUT --delete is gated on the export having succeeded. A failed export is the
# signal that something upstream is wrong (wrong library path, unmounted SD
# card, half-migrated library), and mirroring deletions from a wrong library
# would empty this copy in the same run that failed to add to the other.
DELETE_FLAG=""; [ "$RUN_OK" = "1" ] && DELETE_FLAG="--delete"
rsync -a $DELETE_FLAG --partial --timeout=600 \
  -e "ssh $SSH_OPTS" \
  "${PHOTOS_USER}@${PHOTOS_HOST}:${LIBRARY_PATH}" "$BUNDLE_DST"
rc=$?
# 24 = "source file vanished", routine on a library that is being written to.
[ "$rc" -eq 0 ] || [ "$rc" -eq 24 ] && BUNDLE_OK=1
# Either copy failing fails the unit
[ "$RUN_OK" = "1" ] && [ "$BUNDLE_OK" = "1" ] || exit 1

Extracting originals requires a Mac, since they’re only accessible through a local Photos library (no server-side API for originals). My summer research favored icloudpd on a Linux server (no Mac required), but I ultimately chose osxphotos on an always-on Mac (with icloudpd as a fallback) because maintaining the fragile two-factor dance was just too much. The Mac setup is intentionally minimal: just the library and one extractor script. The script takes a password on stdin and emits a single JSON line when osxphotos finishes.

Note · The library can’t live on the server

Photos refuses a volume with POSIX ownership disabled, and an SMB mount has none. The library’s database is also a live SQLite file that should not run over a network filesystem, and macFUSE tends to break on macOS updates.

The first version was a push model (launchd job on the Mac, own schedule), but I rebuilt it as a pull before even running it once, so now the server starts every run. With a push model, silence “is indistinguishable from ‘nothing to do’”.. a missed run looks exactly like a quiet night. With a pull model, a missed run looks like a failure, and the server can tell the difference between “the Mac was asleep” and “the extractor failed”. Also: one scheduler for everything, and no secret at rest on the Mac.

The first full export was 60,058 photos, 551.7 GB, 0 errors. I tested the restore by comparing md5sums of the archive file and the library original.. same bits, not a re-encode. Now a weekly job restores a sample from B2 and verifies the hashes the same way. The checks have already caught real issues: a freshly erased volume had ownership disabled, and the first real run failed on a systemd environment-file parsing error. The failures were reported clearly by metrics (Mac is fine, job is broken).

osxphotos reported 15,464 photos as “missing”.. not an error (exit code 0), not even a log line, just.. gone. All signals green, but that number told me otherwise. I’d been using a clean exit as proof everything landed, but this was proof it hadn’t. All 15,464 were iCloud Shared Album items with no local original to export, so nothing was lost. I was worried that if I switched the Mac to Optimize Mac Storage, I’d get a missing count that looks exactly the same. Now I get an alert if the missing count jumps more than 25% from the week before.

Warning · A clean run can still miss photos

A Shared Album item and a photo that Optimize Mac Storage pushed to the cloud both show up as “missing” in the export report. Only the trend in that count tells them apart, which is why the alert watches the jump and not the exit code.

The strongest objection to this approach is that Apple doesn’t lose photos, so it’s too much machinery for a non-problem. But the risk isn’t Apple’s disks.. sync obeys commands, including deletes, and account/billing failure ignores datacentre redundancy. The machinery is small where it counts: one script runs on the Mac, the server does the rest.

There are real costs, which I accept as trade-offs. A photo created and deleted between two nightly runs will never be captured. The Mac is a single point of extraction; if it’s down, nothing new will arrive until it’s back, but alerts flag that within 1.5 days.

Next up: retiring the old Parachute and Google One route after a full week green.

My backup strategy

Tue, 08 Sep 2026 00:00:00 GMT

I want to back up my private data properly, and that includes my cloud storage.. iMessage, Gmail, iCloud Drive. All my messages, email, and family photos are on someone else’s servers, so I wanted copies that I controlled.

Messages, Gmail, and iCloud Photos all have legs pulling them to the home server. The iMessage archive mirrors the Messages database and attachments to ZFS nightly, Gmail goes to OpenArchiver (which I wrote up in OpenArchiver a self-hosted Gmail archive), and Photos arrive via The always-on Mac. iCloud Drive is the only source without a leg yet.

The home server is the canonical copy. Everything I want to keep lands on a ZFS mirror, which sanoid snapshots every 15 minutes. Every dataset has a stated backup class, so nothing goes unbacked by accident. My laptop’s Time Machine backups also live here, on a Samba share (Samba for Time Machine five things that broke on day one).

The biggest surprise in the pool inventory was that the Time Machine share’s snapshots were pinned at ~550 GB. How could a backup of a backup hold more than half a terabyte? It turned out that the share was using the same long-lived snapshot template as the archive datasets. But Time Machine maintains its own history, so ZFS only needs to protect the sparsebundle from self-corruption. Matching the retention to that job dropped the snapshots to 27 GB.

The second tier consists of two local copies on different disks. A nightly Backrest run invokes restic to write to a USB disk. An old SSD I had lying around became the second local repo, on a different device and bus, so the two local restores share no failure. Restic needs no ZFS beneath it, so the SSD runs plain ext4.

Backblaze B2 is my only offsite, and the server can’t delete there. I gave the server an application key that allows adding backups but not deleting them, so even a totally exploded (or compromised) server couldn’t wipe the offsite copy. I’m manually pruning old snapshots once a quarter with a short-lived key that I create and then revoke immediately after. The pennies-per-quarter growth cost means skipping a quarter isn’t the end of the world, but leaving a delete key on the server certainly would be.

Having all my private data in its own bucket means I can move it to a colder tier or just drop it later without disturbing the main repo. At $3.50 a month or so for the offsite, it’s easy to weigh against the cost of losing everything.

Photos are the newest addition. These flow from iCloud to the always-on Mac, get exported with osxphotos, then up to the server and out to B2 (more in Photos iCloud is a sync service not a backup). I’ll do a post on restore testing across all tiers later.

iCloud Drive is next, following the same pattern. It’ll be pulled home from the same always-on Mac, into an append-only dataset on the server, then on to the same local and offsite tiers as everything else.

The AI argument at work is really about who's on call

Mon, 07 Sep 2026 00:00:00 GMT

See, the problem is that neither side is making it up. They are seeing really scary trends. They’re grappling with real hard problems that are getting worse.

If both sides are right, how can we ever hope to resolve the argument? In June, Charity Majors wrote AI enthusiasts are in a race against time, AI skeptics are in a race against entropy. The enthusiasts don’t want to get left behind. The skeptics see the costs of quality and reliability piling up.

One of the things that’s ironic though is that both of these sides feel like they are the tiny minority and they’re outmanned and they’re being suppressed and they are standing up for what is truth and valor in the face of the big AI folks or the big skeptics.

They both think of themselves as rebels, and somebody isn’t doing a good job of identifying who the empire is. I think a better way to divide people might not be by how optimistic they are, but by who gets paged when there are alerts.

This often starts to come down to the group that is on call and the group that is not.

The people who the buck stops with them, they are seeing melting mental models, they’re seeing slop, they’re seeing all their hard work just dissolve and they don’t see any end in sight.

It’s not surprising that there are skeptics. They get the 3 a.m. side of the productivity gain.

It’s always weird to me just how much software engineers really seem to believe that the world exists in the repo. It doesn’t. It’s production, you know?

The enthusiast often measures the repo (how many PRs got merged, how fast they’re landing). The skeptic measures production, where the cost of that speed eventually shows up.

If you’re shipping code written by an agent, I think you should be carrying the pager for it. It’s “you build it, you run it” for agents, and it makes the discussion honest pretty quickly.. the person counting merged pull requests ends up being the one reading the alerts.

Imagine an organization where a separate ops group carries the pager for all application teams. The app teams start using agents, merging code faster, and the ops team absorbs the consequences. There’s a long conversation about this dynamic on The Pragmatic Engineer. But who’s on call for the code that nobody read?

A CO alarm that turns on every light and talks on every Sonos

Sun, 06 Sep 2026 00:00:00 GMT

For a long time, Home Assistant lacked the ability to make indoor noise. I only had sirens, but those were Ring sirens, so they required the cloud and were outdoor-facing. I still haven’t paired my Z-Wave sirens. So while the detector alarm could page my phone, and a leak could automatically close the water valve (Leak to closed valve in 68 ms), nothing in the house itself would react to carbon monoxide.

This gap is more important for CO than for smoke: the first symptom of CO poisoning is drowsiness, so a sleepy person won’t do anything about a phone buzzing on a nightstand. The CDC’s page on carbon monoxide notes that sleepers can die before they experience any symptoms. So what would get someone out of bed? I decided that CO would get a physical whole-house response, while smoke would continue on its existing path.

When any CO detector goes from clear to detected, every light in the house turns on at full brightness. All eight Sonos speakers in the house then play a siren, followed by a spoken warning. This repeats every 30 seconds until the CO level is no longer detected. I’m using the Sonos integration and the announce option of the media player play_media service to play the clip over whatever audio is currently playing.

It’s all local. The siren+voice clip is an mp3 in Home Assistant’s www folder, streamed over the local network to the speakers. No internet usage, so it works even during an outage. That’s why I didn’t want to use the Ring sirens. Smoke is handled separately by the detectors’ own horns and critical phone pages.

The first play was at a quiet volume 8 on the master bedroom Sonos, just to prove the file loaded and the speaker announces. Then I played the clip myself on the parlour Playbar at louder levels to get a sense of the room experience. I tested the clip only; I’ve never triggered the full chain with real CO.

My favorite part of this is the Silence button. Once everyone is awake, the siren going off every 30 seconds is pointless. The way I handle that is to turn off the automation, then turn it back on, which stops any run in progress (the automation docs talk about turning one off to stop its actions). The automation only fires on a clear → CO transition, so a lingering CO level won’t restart it. But a new detection will restart it, so Silence never leaves the house unprotected for the next one.

Tip · How Silence stops the loop

Turning the automation off and then on will stop any run in progress. The trigger is clear → CO only, so a lingering CO level won’t restart it, but a new detection will.

The detectors do it differently: the SMCO410 manual says its CO silence is about 4 minutes, and “if CO levels remain or continue to increase, the horn will start sounding again.” I thought about copying that for the house, but decided against it: different jobs.. the horn nags its own room, the house alarm wakes everyone once, then gets ready for the next detection. The Silence silences only the house loop, the detectors’ horns are untouched.

The Silence tile started out on the kids’ dashboard.. I took it off, a house-wide alarm silencer doesn’t belong where kids tap things. The Z-Wave sirens are the durable answer for smoke and CO, and pairing them is the real fix for the missing indoor alarm. This version works now with existing lights and speakers. Next idea: use the Silence as an action button on a critical CO phone alert, so that whoever is paged can stop the loop without going to the dashboard.

Warning · Silence stops only the house loop

I had initially put the silence tile on the kids’ dashboard, but then took it off. The silence command only stops the house loop from playing, it doesn’t do anything to the detectors’ horns.

Choose Boring Technology and the LLM as an innovation token

Wed, 02 Sep 2026 00:00:00 GMT

It’s kind of like the LLM ate all the easy parts of the job, but all of those hard parts are still there. But because we now just have more bandwidth, the question is, have we just kind of all bought ourselves more innovation tokens? I’m inclined to say to a degree, yes.

This question comes from Choose Boring Technology, a great essay by Dan McKinley from 2015. The core idea in that essay is that every company gets like 3 innovation tokens. You should spend your innovation tokens on the stuff that differentiates your company, and use boring tools for everything else because you understand the failure modes of boring tools.

I actually think the introduction of large language models themselves is a form of innovation token usage.

Now, two things can be true at once. The model can free up your time to focus on the hard parts, but it’s also a new piece of technology that your team needs to learn and trust, which means it costs a token when it comes in the door. The talk version of this essay talks about how you need to account for the total cost of a piece of technology (maintenance costs vs velocity benefits) across the entire company.

I think a lot of large language models are oversold and that the maintenance costs, whether it’s cognitive load, whether it’s training, whether it’s the cost of, is the foundation model going to get banned by the US government, those maintenance costs can be quite high, and that if you’re not getting the velocity benefit out of it, you’ve now introduced a new thing that’s actually a net drag on your company.

I think that might be the thing, is some of this boring technology in a world of LLMs is how many innovation tokens of cognitive load do I have to take on because I’m choosing something that’s not in the training corpus of large language models.

The old meaning of boring was “we understand the failure modes”. The new added meaning of boring is “the model has been trained on a huge number of similar examples”. If your team starts choosing a framework largely because it’s what the coding agent deals with well, you get a quiet drift of everything towards the most common stack. The upside is you get known failure modes. The downside is maybe the better tool is the less common one.

I like this idea that you really need to justify what feels like a shiny new thing. Is it in the core capability? Is this something that your team’s willing to own for the next 10 years? If you make this decision and you can’t switch off of it, are you comfortable with it for the next 10 years?

The model is a dependency like any other, with vendor, price, and policy risk. Who on your team is going to own it for the next ten years?

1Password CLI and Connect: secrets without env files

Tue, 01 Sep 2026 00:00:00 GMT

On my home server, every stack had an adjacent file of real secret values, even though I’ve been using 1Password as the source of truth since May. As described in How a deploy works with Dockhand, the pattern was for a script to read each value with the 1Password CLI and PUT it into the Dockhand per-stack environment as a regular variable, which Dockhand persisted as a plaintext .env.dockhand file per stack on the ZFS data disk. This worked for 3 months. I said I’d talk more about the op CLI in chezmoi one source of truth for every dotfile, so here goes. The cost was a plaintext file of real values for every stack on the data disk.

A post on r/1Password about using 1Password Connect secrets in Docker Compose led me to alikhil’s docker-secrets-1password, a 1Password Connect provider for Docker’s Secrets Engine, which lets compose carry se:// references at runtime instead of values. The engine is experimental and requires Docker CE 29.2+. I did some analysis, then a spike with two dummy containers. The references resolved in both plain docker run and compose. Resolution happened at container creation, so the resolved value was stored in the container’s config on disk (Docker’s own metadata). The root client could resolve them too, so Dockhand didn’t need changes. A restart worked with the engine stopped, and a new create failed loudly. The engine’s install bounced dockerd via needrestart, and I was relieved live-restore carried all 84 containers without restarts.

I wrote down the self-assessed honest security delta: wins (pointer-only compose files, no env files on disk, no more relying on Dockhand’s replace-everything env API, rotation is just update a 1Password field and recreate the container), losses (plaintext moves into dockerd metadata, any Docker-socket client can resolve any secret the engine serves). (My note: judge it as plumbing hygiene, not a security boundary change, plan to wait until the engine leaves experimental.) I reversed myself the next morning and adopted it on experimental 0.9.0 anyway, deciding I could accept the risk as long as I only did one stack at a time and the Connect server was only ever deployed by hand. Kept a watch item for when the engine stabilizes.

The resolution chain

Before I put anything into production, I set up a couple of guardrails: first, a dummy container that resolved a reference with a plain docker run against the real setup; then a throwaway Dockhand git stack, because Dockhand’s own compose invocation was the last link I hadn’t tested. Both passed, and I tore them down. The first real stack I put into production was the query-log shipper from AdGuard Home and the query-log shipper, because that seemed like the lowest-stakes thing to start with. The chain I’d put in place was: pointers in the compose files in git, Dockhand runs compose, the Docker daemon while creating the container asks the Secrets Engine for each reference, the engine’s provider gets the value from a local Connect server, which syncs from 1Password.

The Connect stack itself runs two containers, connect-api and connect-sync (1.8.2), with the API bound to loopback only. The credentials file lives outside git and is backed up to 1Password as a document (I tested restoring it and got back a byte-identical file). The data directory is just a cache; I can always resync from scratch. I disabled autoheal and the deploy webhook for this stack because it holds the identity, so a sweep must never recreate it. The provider is alikhil’s 0.3.1 deb, running as a systemd system unit, with the token stored as a systemd-creds encrypted credential piped in (it never touches the command line). The Ansible playbook that codifies the host layer asserts the credential exists but never creates it. The bootstrap is checked for cycles: the Connect stack itself has no se:// references, and its two roots of trust (the credentials file and the encrypted token) are both restorable from 1Password.

# stacks/op-connect/compose.yaml (excerpt)
# IDENTITY-HOLDING: no autoheal. MANUAL-DEPLOY-ONLY (webhook disabled).
# API bound loopback-only: any client with the token can read the vault.
services:
  connect-api:
    image: 1password/connect-api:1.8.2
    ports:
      - "127.0.0.1:<port>:8080"
    volumes:
      - <credentials-file>:/home/opuser/.op/1password-credentials.json:ro
      - <data-dir>:/home/opuser/.op/data
  connect-sync:
    image: 1password/connect-sync:1.8.2
    volumes:
      - <credentials-file>:/home/opuser/.op/1password-credentials.json:ro
      - <data-dir>:/home/opuser/.op/data

The CLI had one more lesson for me before any of this even ran. I set up the Connect server from my laptop using my personal 1Password account, and got a 403 on the first attempt. I assumed the CLI was acting as my signed-in account, but it turned out that the service-account token I had in my shell environment was taking precedence (which took me a while to see). Service accounts can’t manage Connect, so every request was being refused as that token.

23 stacks in a day

For each value, I took the sha256 hash of the value Dockhand injected and compared it to the hash of the 1Password field. I probed the reference in a throwaway container before committing. After deploying, I checked the running container’s env by hash and the app itself by using it. Then I emptied the stack’s env in Dockhand, deleted the leftover .env.dockhand (which Dockhand never removes when the variable set goes empty), and verified it stayed gone after redeploy. The Compose variable value became a pointer plus a comment naming the thing.

# before: Dockhand injected the value from its per-stack env file
environment:
  - APP_PASS=${APP_PASS}

# after: a pointer the Docker daemon resolves when the container is created
environment:
  - APP_PASS=se://op/<vault-uuid>/<item-uuid>/<field>   # item "<name>" in 1Password

The friction came in small pieces throughout the day: item titles with spaces triggered invalid pattern from the provider’s matching logic (now all references use UUIDs plus a comment naming the item), the provider only matches on field labels (field ids and %20 didn’t work, so I renamed labels to avoid spaces), the reference has to be the entire variable value (my connection string embedding a password now lives whole as its own field, rotating with that password), and references don’t resolve in command: (put the value in the environment, have the shell read it, $$ escape for compose.. proven live on a Valkey server taking its password on the command line).

# stacks/<stack>/compose.yaml (excerpt)
environment:
  REDIS_PASSWORD: se://op/<vault-uuid>/<item-uuid>/<field>
command: ["sh", "-c", "exec valkey-server --requirepass \"$$REDIS_PASSWORD\""]

The CLI misled me twice: op item create - reads a JSON template from pipe only, not < file redirect (it complains about --category, but adding --category creates a blank item). My failed attempts to create a Connect token and store it in the same pipe still minted the token, so I deleted two orphans. The hash step revealed variables I’d injected for months that reached nothing, including one admin “user” variable that was literally the app’s default. By day’s end, all 23 stacks ran on se:// references (18 commits in) with zero .env.dockhand files on the server. I also found an orphaned deploy directory for a retired service that still had plaintext credentials in it, so I deleted that.

Locked-out nodes

The VPN sidecars had been keeping their Tailscale state in memory, so every restart/redeploy registered the node as brand new. I’d turned on Tailnet Lock around August, so every new node key needed signing before the node could talk. The container and proxy looked healthy, the deploy was green, but the tunnel was dead. I wondered how many other things were “fine” while broken. It bit me three times that day: after the migration, after a verify bounce, after a push that changed nothing but a comment. I signed each new node key by hand (nine signings).

I fixed this by persisting the sidecar’s state so each node has one identity, signed once. After the change, a restart kept the same node key and nothing got locked out. I didn’t want to use pre-signed auth keys because Tailscale’s docs say that’s equivalent to leaking your signing key, and I didn’t want to set up an auto-sign timer because that’s just a rubber stamp.

# stacks/<vpn-stack>/compose.yaml (excerpt)
     environment:
       - TS_AUTHKEY=se://op/<vault-uuid>/<item-uuid>/<field>
-      - "TS_STATE_DIR=mem:"
+      - TS_STATE_DIR=/var/lib/tailscale
     volumes:
+      - <state-dir>:/var/lib/tailscale

The gates and hashing on both ends made it safe to do a day’s worth of changes all at once: no value moved without proof of arrival. The Compose files only hold pointers, and rotation is a design property: change a 1Password field and recreate the container (a restart won’t do it). Losing the engine only costs new creates, not anything already running. What’s not better: the resolved value is still in Docker container metadata by design, and the experimental pieces have sharp edges and misleading CLI errors. Worst of all, the green deploy told me nothing about the dead tunnel or the dead secrets chain.

Both can hide behind a green deploy, so I changed how I monitor them. Now an uptime monitor watches Connect’s health endpoint, and separately the existing 15-minute health pulse gained two new checks: one that walks the entire secrets chain behind the daemon (engine, provider, Connect), and one locked-out-node check that looks for any node Tailnet Lock is holding unsigned. Both are proven by breaking things on purpose, and the reboot baseline script snapshots both. The old injection script stays dormant until the engine survives its first Docker CE upgrade. Separately, the next test is a full redeploy of the VPN sidecars. That’s where I’ll leave it.

SYMFONISK remotes to ZHA, and retiring the IKEA hub

Mon, 31 Aug 2026 00:00:00 GMT

I kept the IKEA DIRIGERA hub running for 2 devices only: a pair of SYMFONISK remotes on our coffee table for controlling the living room Sonos. This summer I’ve been consolidating all my vendor hubs into Home Assistant (I’ve already retired SmartThings, removing the integration and unplugging the hub), so this 2-remote hub seemed like the next to go. As always, the guiding question was: can Home Assistant do the job without me losing anything I actually use?

Yes, with one small loss. Both remotes are plain Zigbee devices: the round knob is a SYMFONISK Sound Controller (E1744, firmware 24.4.5) and the button remote is a gen2 sound remote (E2123, firmware 1.0.35). The hub also acts as a Thread border router, but I already have an Apple TV doing that, and my Home Assistant Thread and Matter integrations are loaded with zero devices. Core Home Assistant doesn’t have a DIRIGERA integration (only HACS), so once the remotes move, there’s literally nothing left on the hub I need to talk to. The only real loss is the two playlist presets I’d configured in the hub.. those don’t migrate.. but I pulled the hub that weekend.

I’m already running Zigbee natively on ZHA with a Sonoff Zigbee 3.0 dongle (I moved off Zigbee2MQTT earlier this month), so these remotes joined the same radio everything else is on. Now a remote press goes through the dongle, triggers a zha_event from ZHA, which Home Assistant automations pick up and send to the living room Sonos via the Sonos integration. Pairing was quick.. I reset each remote with four presses of the pair button, then permitted joining from Home Assistant for 254 seconds. Both joined on the first try and became direct children of the coordinator, the button remote at LQI 164 and the knob at 148. I’m happy with that.. one attempt each and they’re solid. I named them right away and put them in the Living Room area.

The button remote has just one automation. I set the mode to queued so that if I press rapidly, all presses run in order instead of dropping any. I trigger on the exact command because for some reason every press also generates a spurious attribute_updated in addition to the real toggle. The track buttons send step (mode 0 or 1 is next/previous). If you hold a volume button, the remote itself repeats move events about once per second, and each move increases/decreases the volume by one step, so holding ramps volume up/down for free.

The knob is in the same cluster and uses the same move command, but with the opposite meaning. It sends a single move at the beginning of a turn, then nothing until a stop at the end. If I wired it up the same way as the button remote, turning the knob would result in a single volume step, followed by idle.

The knob needed its own automation, which is the part of this project I’m most proud of. It uses restart mode, and on move uses a repeat loop to step the volume every 0.35 seconds until stop is received, which re-triggers the automation and ends the loop. I capped the loop at 20 steps so that a lost stop can’t crank the volume to max. I used a template on move_mode to determine direction instead of matching exactly on all arguments because the rate byte (195 when I captured it) might not be constant, and an exact-argument trigger that never matches doesn’t raise any log errors.. it just silently kills the knob.

One gotcha in this configuration that tripped me up is that Home Assistant wants the variables: block to be its own step in the action sequence. Putting it in the same dictionary as repeat: caused the automation to fail to load with “extra keys not allowed”. So I had to put the variables in a separate step before the loop.

There aren’t any details about the events in the documentation, so before setting up any trigger, I temporarily created an automation logging every zha_event from both remotes to the system log, pressed and held every button, turned and clicked the knob in every supported way, then deleted the logger. The log showed these shapes:

every gen2 press fires the real command and a spurious attribute_updated alongside it
gen2 play/pause is toggle (confirmed with a follow-up press)
gen2 track buttons send step, mode 0 or 1
holding a gen2 volume button repeats move, about once per second
turning the knob sends one move followed by a stop
the dot buttons fire press events on endpoints 2 and 3

Capturing the events exposed the spurious event and the volume difference between the remotes, guessing or reading Zigbee2MQTT’s docs would have gotten me wrong on both. It also resolved a previous worry I had, that early gen2 firmware was reported to not have distinct events for each button. I’ve now written into CLAUDE.md of my Home Assistant config repo to capture the events before wiring them up, so my coding agent will do that. The shapes of the events come from the device quirks in zha-device-handlers, which sits on top of the zigpy ZHA library, and those maintainers did the hard part.

I tested every gesture on Home Assistant 2026.7.2 and they all worked: taps and holds on the button remote, and turn/every press count up to triple on the knob. I unplugged the DIRIGERA the same day, and since I never gave it a DHCP reservation, there’s nothing to clean up on the router.

The remote batteries made it to metrics without any extra work.. they landed in VictoriaMetrics, and the Grafana battery panels use pattern queries, so the remotes showed up automagically. I deliberately excluded them from the battery watchdog (that’s reserved for fire/water/life safety devices).. a dead remote announces itself when someone reaches for the volume. Hue is the only vendor hub I have left, and it’s staying that way by design.

One loose end: the two dot buttons on the gen2 remote. I can see press events on endpoints 2 and 3 in the capture, so I’m ready to wire those in. I’ll leave them unmapped until I pick two playlists to replace the hub presets. Those will play as Sonos favorites from the button automation.

What Platform Engineering says about where to bet in 2026

Sun, 30 Aug 2026 00:00:00 GMT

If I were making a 2026 platform bet for a bank or a Fortune 500 company, I’d fund the internal platform to run as a product (with an owner and users to win over) and drop the portal project. I’m basing this on ten years working with large financial institutions and Fortune 500 companies, now as a principal cloud architect. Platform Engineering by Camille Fournier and Ian Nowland (O’Reilly, 2024) supports this framing; both authors have led platform organizations, including in finance.

The book’s diagnosis: an over-general swamp. Cloud and open source gave us unlimited building blocks, but teams chose differently, so a year on, the estate is mostly glue code, each service with its own pipeline and monitoring conventions. The explosion of choice coincided with rising demands: uptime, security, compliance, cost.

The book’s remedy: a curated product built on real software abstractions, serving a broad base of ordinary application developers and operated reliably enough for the business to depend on it. Without these four pillars, the authors argue, you’re not managing complexity, you’re just shifting it. A developer portal isn’t a platform, in my view.. it’s the front door that still opens into the swamp. In fact, I’d argue that a portal-first platform program is the single most common way to burn a year or more with zero adoption.

Timing matters, and AI is why. DORA’s 2025 research on platform engineering shows that nearly 90% of organizations now have some form of internal platform. But the more interesting finding was about quality: high-quality platforms turn AI adoption into a significant positive for organizational performance, while low-quality ones make it barely register.

The way I see it, in a regulated enterprise, the platform carries the security and compliance controls so teams don’t have to reinvent them. Bounded freedom, speed within paths that already contain what the risk function and change board would demand. Better than twenty teams each building a flawed cloud setup in shared accounts. It makes the platform a prerequisite for any AI productivity case, not a side project.

The new kind of platform user is agents.. they request environments and open pull requests at machine speed, walking the same golden paths developers do. Datadog’s post on golden paths for AI agents describes making capabilities machine-readable (typed schemas, parseable errors). The controls live outside the agent (approvals, policy checks, CI gates, scoped credentials, an audit trail for every run).

I’d build a single golden path for people and agents, with guardrails inside the path instead of bolted on per-agent. Eventually a regulator will ask who approved a given change, and “the agent did it” won’t fly with any risk function. If the approval is on the path, it doesn’t matter if a developer or an agent made the change.. the answer is the same. I talked about agents as platform users in my Team Topologies post, but this is the practical follow-up.

The strongest objection is that platform engineering is just layoffs rebranded. The platform team becomes a ticket queue, blocking other teams for weeks when something breaks.. a cost center that can’t prove its worth.

My answer is that this is an unfunded platform. You usually can’t prove value because leadership funded a team, not a product, so there’s no roadmap owner and no measurement of outcomes. Put a product owner in place and give teams outcome measures they can feel.. time to first deploy, change failure rate, recovery time, audit evidence on demand. Leadership must budget time for application teams to migrate (the book warns that migrations routinely cost two to three times the estimate).

If I were advising a CIO on next year’s platform budget, I’d make three calls. Fund the platform team as a product team, with an owner and a roadmap driven by developer needs. Stop funding portal projects as a goal, and bespoke pipelines that serve one team’s pet stack. Put agents on the roadmap this year as a platform user, with a path that goes through the same controls.

The book talks about how a well-loved platform can fade into the background, and how being called boring is the highest compliment. I’d like to apply that measure to the agent persona: the platform is working when the agent’s change passes the same approval and lands in the same audit trail as the developer’s change. Nobody in risk or compliance needs to ask how it got there.

Leak to closed valve in 68 ms

Tue, 25 Aug 2026 00:00:00 GMT

Why haven’t I gotten any notifications from Home Assistant, you might ask? Well, I assumed they were broken, but they weren’t.. it’s just that I had zero automations, so the notify service was never being called.

This question came to me just as HA’s telemetry started hitting my new metrics DB (VictoriaMetrics as the first metrics database on my home server), and the first thing I noticed was that both of my Phyn leak pucks and First Alert smoke/CO detectors had been unavailable for about a week.. and nobody had been alerted to that fact.

I already had the shutoff hardware in place (an EcoNet Z-Wave motor on a Bulldog ball valve on the main, on the network but not being used for anything), so the first automation I built closes that. When the leak sensor gets wet, HA closes the main first, then pages me, allowing the notification steps to fail without preventing the shutoff. Water never gets automatically restored.. restoring it is a human decision. Now the house can protect itself even if my phone is in another room.

Two design choices made this durable. First, I put all leak sensors in a group helper and triggered the automation from the group, not the device class. The Phyn sensors report device class “problem”, not “moisture”, so a moisture trigger would match nothing and protect nothing. When I later added two Zooz sensors (under the kitchen sink and in the laundry closet) to the group, I didn’t have to edit the automation.

Tip · Not all leak sensors report as moisture

A trigger on “any moisture sensor” only covers sensors that actually report as moisture. Grouping the sensors you care about means coverage is a list you can read, not a guess.

Second, the valve semantics were inverted. The Z-Wave switch showed off while water was flowing.. off meant open and on meant closed. That’s a vendor decision, not a Z-Wave convention. I wrapped the raw switch in a Change device type of a switch helper with invert on to get a true valve entity where close means close. The helper docs pointed me to that native option instead of a template valve. Downstream automations don’t know or care that the semantics are inverted.

Note · Check which way off goes

The Z-Wave switch showed off when water was flowing: off meant open, on meant closed. Change the device type of a switch helper with invert to get a valve entity where close means close.

From the beginning, I set up a standing rule to send water and leak alerts directly to my phone as iOS critical alerts, plus a Pushover message. I tested the valve by pressing its button once, and it closed while both alerts made it through. But then I asked about the night: would I sleep through it? Would it get re-pinged? I bumped both water alerts to Pushover’s emergency priority, which repeats every minute for up to an hour until acknowledged.

Warning · One press only on the valve button

On the EcoNet motor, three quick presses start Z-Wave inclusion or exclusion, and holding the button for 10 seconds factory resets it. A test only needs one isolated press.

Smoke and CO got the same upgrade, and moving them exposed a silent failure in my safety alerts. I moved the detectors from SmartThings to Z-Wave, and the obvious thing was to add the new entities to the existing smoke alert. That looked right, but it paged nobody because the Z-Wave detectors report “on”, never “detected”. So the alert got a second trigger for that state. The offline watchdog missed them, too: the pattern matched no smoke entities. Now one Life Safety label covers all safety sensors and the valve, and a native HA automation pages me if any labeled entity goes dark. I discovered that Siri could shut off the house water (“turn on the water valve” drove a raw inverted switch). The valve is permanently removed from HomeKit and renamed in a room-and-function style. (Naming things in Home Assistant)

The first live water test was real proof. The probe got wet at 19:09:23.685 and the automation started 1 millisecond later. The valve reported closed at 19:09:23.753.. 68 milliseconds after the probe got wet, and before any notification was sent. When the sensor dried, I reopened the valve from HA and nothing closed it again.

The test exposed two gaps: a second phone in the house had never been on any life-safety page since I had the automations, and every leak was triggering two emergency sirens instead of one. Both were fixed that same evening. Now, if HA closes the valve, I get a single quiet receipt, but if the valve closes on its own, both phones get an emergency alert.

Any new leak sensor is covered as soon as it joins the group, and the close-first pattern carries over to CO. Next up are the places without sensors. The bathrooms need leak sensors, the sump pit needs a Z-Wave sensor (the Eve sensor there doesn’t have Matter firmware), and CO is next on the build list: A CO alarm that turns on every light and talks on all 8 Sonos.

Config history for a live Home Assistant

Mon, 24 Aug 2026 00:00:00 GMT

HA is already backed up (ZFS snapshots + Backrest, so I can restore everything after a dead disk or a bad upgrade), but backups don’t tell you what changed and when. This gap has been on my list since July, but it became unacceptable once I had leak automation that could close the main water valve. If the automation had behaved differently, I could restore a point in time, replacing the older version, but never see the edits I made.

HA got its own git repo on self-hosted Forgejo. I split it from the home-infra repo. The compose file that runs HA will remain in home-infra, because the service is hosted there, but the project itself lives in its own repo. Moving the compose would mean a new credential and a stack repoint, with no benefit.

The key decision is the direction. Live is the source of truth, so I only sync the live config to the repo, never the other way around. HA rewrites its own YAML through the config API when I make changes in the UI, so the repo is a record, not a deployment. Nothing ever gets applied. I always track an explicit list of the handful of YAML files from the configuration directory.. never a glob. A glob risks accidentally committing a file containing secrets.

An hourly systemd timer runs the sync and pushes a commit if there are changes. I chose auto-commit over an alert waiting for action because a history that depends on someone remembering to run a script isn’t a history. Before copying anything, the script aborts if it finds a literal token or inline password value, but !secret references are allowed. I never track the secrets file or the entire .storage/ directory, so ZFS + Backrest remain the only recovery path.

Clean run, zero commits

The first version of the sync reported a clean run but didn’t commit anything. The problem was a single line where bash’s negation didn’t cover as much as I thought: I read ! as “if adding and committing fails”, but bash applies ! to the single pipeline next to it, not the entire list:

# as written (commit never runs after a successful add):
if ! git add config && git commit ...
# parses as:  (! git add config) && (git commit ...)
# fix: group with braces so the negation covers the whole add+commit

So after a successful add, the negated half was false, and the && short-circuited before it got to git commit. Grouping with braces fixed that. I only caught this because I was checking git log had moved instead of the script’s exit status (which was zero the whole time).

Note · Permanent drift from two bytes

HA writes scenes.yaml as the two bytes [] with no final newline, which the repo’s yamllint -d relaxed pre-commit hook rejects. Fixing only the stored copy would have made the comparison report drift forever, so the sync normalises newlines on both sides before comparing.

When the timer goes quiet

The wrapper pages on sync failure, but a stopped timer never fails, it just goes quiet, and a stale mirror is worse than no mirror at all because it looks authoritative. So each run writes a last-run timestamp as a node-exporter textfile metric, which lands in VictoriaMetrics. A file-provisioned Grafana rule alerts after three hours (three missed hourly runs):

  # 3h = three consecutive missed hourly runs, allowing for the timer's
  # RandomizedDelaySec=5m and a reboot, so a single skipped run is not a page.
  - orgId: 1
    name: home-assistant-config-mirror
    folder: Home Assistant
    interval: 30m
    rules:
      - uid: ha-config-sync-stale
        title: HA config mirror has not synced in 3h
        condition: A
        data:
          - refId: A
            model:
              expr: '(time() - ha_config_sync_last_run_timestamp_seconds > bool 10800) or absent(ha_config_sync_last_run_timestamp_seconds)'
              instant: true
        noDataState: OK

I tested the rule by forcing it to fire, not by watching it stay green. The results: a healthy mirror got me 0, a one-second threshold got me 1, and a bogus metric name got me 1 (via the absent() half). That last one is why I can safely set noDataState: OK.. a vanished metric still pages, but it doesn’t count as no data.

Tip · Force the alert to fire

A healthy mirror gave me 0, a one-second threshold gave me 1, and a bogus metric name gave me 1 (via the absent() half). That absent() is what makes noDataState: OK safe.

The dashboards with no history

That same afternoon I spent a long time cleaning up entities and dashboards, and doing some registry work like Naming things in Home Assistant. When I was done, the mirror had made zero commits, and the drift check had been reporting “no drift” the whole time (because the dashboards I’d created in the UI are storage-mode dashboards and live in the untracked .storage/). The four with no history were the floorplan, kid’s room, map, and water heater dashboards.

It turns out that quickly fixing this by loosening the .gitignore was the wrong approach: .storage contains plaintext credentials (the auth store with long-lived tokens, and every integration’s config entry). Instead, I’m syncing exports of each dashboard (just the view and card JSON) from the Lovelace API into config/dashboards/. The export lifts the config out of its wrapper and drops the version metadata, so an unrelated HA schema migration isn’t seen as a dashboard edit. It sorts object keys but keeps list order because the view/card order is the content. At the end of the week, I had six dashboards with history, and the drift check caught the one I’d deliberately edited as a test.

Running unattended

The timer ran hourly overnight unattended, zero drift. Every HA session I’ve had since then ended with the repo clean and pushed. It changed how I build things.. new sensors go into tracked YAML instead of UI helpers so they show up in the mirror.

Warning · Reduced, not closed

Entity renames, areas, labels and config entries still live only in .storage, so they have no diff history. ZFS and Backrest can restore them, but they still can’t say what changed.

The other weak spot is commit messages. Auto-generated messages tell you what, not why. For changes that need explaining, I run the sync manually and then git commit --amend to add a real message. So the history of something like the leak automation can say why as well as what.

A Zigbee sensor in a metal mailbox (and a home zone 523 m off)

Sun, 23 Aug 2026 00:00:00 GMT

I wanted to test an old SmartThings multipurpose sensor, CentraLite 3321-S, in a metal curb mailbox about 20 feet from the front door and still have it report via Zigbee. The sensor has a door contact and accelerometer, which ZHA exposes via a device quirk, so it should be enough to alert me to mail. My first attempt didn’t go well: I paired the sensor to ZHA and it worked fine on the bench two floors from the coordinator, but it went unavailable 6 minutes after I put it in the mailbox and stayed dark overnight. I suspected the box was eating the signal.

The internet consensus seemed to be “metal mailbox is a Faraday cage, give up, mount the sensor on the outside”. I didn’t want to accept that without a number, though.. surely the box was costing me some signal, not all of it? How much, and was something else interfering?

Before I started measuring the box, I looked at my mesh and got a first surprise: since commissioning, my ZHA network consisted of the SONOFF ZBDongle-E coordinator in the basement and one end device, with zero routers. I hadn’t noticed because everything worked, but of course a router-less mesh looks healthy until you try to add a device at the edge. My Sengled bulbs are end devices and never route, and my Hue bulbs only route on the Hue Bridge’s own network (invisible to ZHA).

I started by changing my setup: I added three Third Reality 3RSP02064Z plugs as my first routers, placing them in the living room (indoor), the office (indoor), and in the driveway (one with line of sight to the mailbox). Then I did a research pass and found 29 sources (mostly forum threads and zigbee2mqtt issues) that predicted: a steel box will cause 20-30 dB of signal loss; thin aluminium might be 10-15 dB; anything inside the box is a coin flip. Every long-running success story involved mounting the electronics outside the box, from this SmartThings accelerometer mailbox thread to this r/homeassistant question about a metal mailbox.

The first reading with the indoor plugs was a puzzle: -85 dBm inside the box, vs -83 dBm with no routers. Turns out end devices pick a parent once and never roam, so it was still using the indoor parent through the metal. I pulled the battery with the sensor in its final spot (lid open, because a rejoin scan from inside sealed metal backs off for minutes) and let it pick again. Tricky to catch because ZHA’s “via device” still showed the coordinator after it moved to the plug (it records that at join time), and the topology tables only refresh on scan.

One conclusion I drew mid-process was wrong and influenced subsequent readings. In a Claude Code session, I saw the LQI jump from 68 to 136 and assumed the plugs were working. The reading was accurate, but the conclusion was flawed.. I’d also brought the sensor indoors, so I’d changed two variables and only named one. From that point on, every measurement was taken with the sensor in its mounted position inside the box.

Once I had both of those nailed down, the readings looked like this:

Sensor position and mesh	LQI	RSSI
Bench indoors	136	-67 dBm
In the box, no routers	68	-83 dBm
In the box, still on its indoor parent	60	-85 dBm
In the box, driveway plug only	160	-60 dBm

I measured the cost of the metal enclosure and outdoor distance at about 18 dB, a bit less than the 20-30 dB for steel in the literature. I did a trial with the driveway plug as the only router, the sensor sealed in the closed box, and nine open/close cycles plus knock tests. The result was 18 of 18 transitions caught, zero drops. Very satisfying, and I cancelled the planned trials (each indoor plug alone, the spot under the porch). A side effect of isolating the driveway plug was that it unstuck the stalled firmware update, so now all three plugs are on the same version. I suspect the stalled Zigbee OTA and the unresolved route are the same fact.

What this means is that the metal was a cost, not a wall, and the blocker was the empty mesh. A mesh with no routers looks healthy until you put a device at the edge. I built on this a mailbox notification triggered by the door contact OR the accelerometer, with a 10-minute cooldown, attaching a front-door camera snapshot (the mailbox is at the left edge of the frame). I fire-tested it via the States API. It also fires when we check the mail ourselves, which is correct by design: nothing in this hardware can tell who opened the box, and the camera sees someone, not who. Ring motion is snapshot context only, never a trigger or condition.

The home zone 523 m from home

The same day I nailed down the second long guess: presence had been broken for months. I blamed the phone twice: the companion app seemed dead, then location permissions. Not the phone. My home zone was 523 m from where the phones rest at night, with a 100 m radius, so the house was 423 m outside its own zone and location reports lacked a zone. I measured the median of five consecutive overnight GPS fixes, 2.1-2.3 m accuracy, on the assumption that nobody is half a km from home at 3 am for twenty minutes. I validated that against a second phone in the house, registered independently the same day, which landed 0.5 m from the median. After fixing that, both people read home.

Two traps: reloading core config updated the API value but not the home zone itself (which is built at startup), and reloading zones only rebuilds those defined in YAML, so it needed a full restart.. (if I’d only checked the API, the reload would have looked like a success). Also, adding a person and registering their phone are separate steps, and Home Assistant didn’t link the new tracker to the person automatically. Once I had presence working, I could define a “someone home” sensor that’s on if any phone’s GPS or home Wi-Fi says so, and use it to pause hot-water recirculation when the house is empty, as I wrote up in Hot water at the tap with Home Assistant.

The box had more to show me

Eighteen of eighteen in a single trial means I got signal in one session, not multi-day reliability.. and the box still had more to show me. Over the next three days the sensor lost its parent three times counting the first night, offline 8 to 17 hours each time. When it was connected, signal was -64 to -70 dBm and LQI 112 to 144.. so signal strength wasn’t the issue. Two of the rejoins gave fake “mailbox opened” pages at 4-5 a.m.: contact on for 0.2 to 1 second, no accelerometer event. A real opening had the contact and accelerometer events about 110 ms apart, and a real delivery held the door open for 7 seconds. The trial went well, but for the purposes of resiliency and accuracy I opted to move the sensors to the exterior of the box.

I moved the sensor to the exterior back of the box, vibration only now. I removed the contact trigger from the notification and hid the contact entity (it stays open forever). I pulled the battery again in the new position, enclosure open, so the Zigbee parent was chosen from the real location. The 18 dB figure stands, and the research advice won on reliability. We’ll see if vibration alone pages on wind or a passing truck. The only test so far was a person shaking the box, 15 events over nine minutes. The silicone seal waits until the mount proves itself.

Edited on 2026-10-03 to link Hot water at the tap with Home Assistant.

Samsung Frames in local art mode, no SmartThings key

Wed, 19 Aug 2026 00:00:00 GMT

I’m trying to retire my SmartThings hub and figure out which automations I have for my Frame TVs will die as a result. I have two: a 2024 43-inch in the living room and a 2021 55-inch in the playroom. The stock Samsung TV integration in Home Assistant reports on regardless of whether the TV is displaying art or live TV. This means I can’t write an automation to “turn the art off at night” because there’s no way to differentiate between the two states.

The community integration ha-samsungtv-smart is a fork of ollo69’s original project and adds an art-mode switch and art-state sensor. I chose this one after reading the source code. The setup asks for a SmartThings key, but if you leave it empty, it connects locally to the TV. This is important because Samsung is moving the SmartThings API to paid tiers.

Initially, pairing attempts timed out after exactly 30 seconds because nobody was at the TV to press Allow. I fixed this by pairing each Frame while standing in front of it. Once both were paired, toggling art mode flipped the art sensor from off to on while keeping the TV entity set to on. This shows exactly what the stock integration couldn’t.

The second puzzle was the art library showing zero pieces despite the TV having its own collection. The art channel uses a separate connection from the TV control connection and remains silent during live TV playback. But after the TV went idle, the library count flipped from 0 to 15, and when I selected a Monet in Home Assistant, the picture on the wall changed.

Some folks have had luck with Samsung Tizen debloat, and this is a big thing now with reverse engineering and customizing with AI agents.

Edited on 2026-10-25 to add the Tizen debloat idea.

Naming things in Home Assistant

Tue, 18 Aug 2026 00:00:00 GMT

I needed to relocate some smoke/CO detectors, so I needed to give them proper names, and that’s when I noticed that all my other names in Home Assistant were.. well, a year after moving house, many of the entity IDs still had the rooms from the old house.

So the detector job turned into a whole-house naming cleanup. The Hue bridge went from 63 devices to 39, and I rebuilt the list of areas to actually reflect the floors in this house.

Named the detectors twice

I wrote the hardware ID on each detector in sharpie, so the first pass was to name them <Room> ID-X so I’d have the ID in every alert. I redid that as <Room> Smoke/CO to match the <area> <what it is> pattern from voice assistant best practices, moved the ID to a label, and decided I didn’t want ID-G in a 3am fire page, but that I could still filter for the ID with one label filter, and that the label would survive if I ever renamed the room (and rooms in this house demonstrably get renamed). Cost: 236 entity renames and one automation update.

I’ll do the ID-to-label move first next time. Labels are the best way to hold the ID.

The pretty ID was the dead one

This has come up a few times before, so I have codified protections against this now. In one case, the SmartThings copies of my Hue bulbs had clean IDs, but the real Hue entities were on _2. In another, a Hue room group was skipped by reconcile, same outcome. In the third, I re-registered my Samsung TV and left the old entities. The nice ID was always the dead one. Always check the state of the entity before renaming.

A few days later, I had the same pattern with 3 Ring devices where I’d replaced the hardware. ring-mqtt keys devices by MAC, so a replacement means remove then add. Home Assistant never prunes the removed device, so it won’t reuse the entity ID the ghost is holding, so the live replacement ends up on permanent _2 IDs, and my dashboards still point at the dead one. The first ghost was my Veranda Door doorbell, which looked broken in Home Assistant but was fine in the Ring app. The second was my Front Door camera. The third was my kitchen chime, which the dashboard said was offline even as it was audibly chiming. I deleted each ghost, then moved the live entities onto the freed IDs.

Warning · Replaced Ring devices land on

Since ring-mqtt keys devices by MAC, a replacement means remove then add. But home Assistant never prunes the removed device, so the live replacement ends up on _2 IDs. Delete the ghost, then move the live entities onto the freed IDs.

Not all duplicates are ghosts

My smoke/CO detectors had pairs of “Alarm silenced” entities, with one of each pair on _2. I thought this was more ghosts, but I was wrong. Each pair was from the same Z-Wave device, but reporting separately for the Smoke and CO notification classes. Deleting either one would mean losing life-safety telemetry from a working detector.

Warning

Before calling any entity an orphan, look at its platform and config entry in the entity registry. Two entities with nearly identical names on the same device can both be live and both be needed.

Renames don’t tell anyone

Renaming never updates referrers. I renamed a Ring old-house device and silently broke a Grafana alert exclusion for battery cameras. The regex still matched the nonexistent sensor.lounge_wireless, but the live sensor.back_bedroom_wireless started paging. The alerts run on the setup from VictoriaMetrics as the first metrics database on my home server. I changed lounge to back_bedroom in the exclusion and commented in the rule file to check on every rename. I also added an alert: if a Home Assistant entity is unavailable for 48 hours, the dead entities surface themselves. The first test firing was the Ring ghost I’d deleted hours earlier.

Scan first, then rename

After hand-writing the same rename checks for the third time, I decided to write them down once. I built a little rename skill around the read-only scan: I can now enter an entity ID and get a list of all automations, scripts, and dashboards that use it. I run this before every rename and patch the findings first.

The first real use case was the Hatch sound machine in a kid’s room. Three presets were all named “Ocean”, so the entity IDs use the Hatch preset number instead of _2 or _3. The number comes from the device itself, and it matches how we use it in the family (“we use 5 often”). Could I just rename the preset in the Hatch app instead? No.. the integration builds the scene identity from the favourite’s name in the app, so renaming it there would orphan the Home Assistant entities. Renaming the preset broke six dashboard scene references, but the scan caught all six, and a second scan came up clean after patching them.

Still fixing area stutter

The next thing I’m still working on fixing is area stutter, where new integrations will prefix the area to an entity name that already includes the area, resulting in media_player.living_room_living_room_tv. I renamed a batch of these in one session, including media_player.parlour_parlour. I left light.boiler_room_boiler_room alone because it’s well established and the cost of renaming it outweighs the stutter.

Poolsuite FM on every Sonos in the house, no AirPlay needed

Mon, 17 Aug 2026 00:00:00 GMT

When I tap the Poolsuite favorite in the Sonos app, the speaker starts playing Poolsuite FM from my home server. Anyone in the house can pick it, and we have 7 channels to choose from, from the main Poolsuite FM mix, to Tokyo Disco, Hangover Club, and more. And nothing is exposed to the internet.

The Poolsuite interface is amazing, but it’s web only, and for Sonos usage, the phone browser running audio in the background over AirPlay had proven to be problematic. So a way over-engineered solution was obviously the move here.

To get the stream from my home network into the Sonos app, I had to do a little trick. The new Sonos app doesn’t support custom music services anymore, so I couldn’t add my own. But I could use a TuneIn Custom URL, which only stores the address on TuneIn’s servers and has the speaker fetch the stream itself over the local network. I’d initially ruled out TuneIn, but then I realized that what I really needed was to get the stream off the internet, which the Custom URL does. One step missing from every guide I read was that I only saw the favorite after re-authenticating TuneIn in the Sonos app (Settings, Services, TuneIn).

The Poolsuite channels are curated playlists of tracks hosted on SoundCloud. skinnydip reads the list of tracks for each channel from Poolsuite, then uses yt-dlp to find the audio for each track on SoundCloud. Finally, ffmpeg stitches everything together into one continuous MP3 radio stream, complete with song titles. Both yt-dlp and ffmpeg sit idle when there are no listeners.

The VPN is part of the design: only the track lookups go through the gluetun VPN container I already run on the server for other services. The actual audio and the Poolsuite track lists come in direct, so the VPN only carries a trickle of lookup traffic. If the VPN goes down, the lookups just fail, back off, and retry automatically.

I made a handful of changes to my skinnydip setup. Song titles now display in the Sonos app, the stream cuts off a minute after the last listener leaves (so nothing plays to an empty house), and cold starts send silence until the music is ready (so the first tap works, no second try needed).

The biggest change was running the transcoder at real time with ffmpeg’s -re. Without it, ffmpeg would transcode at download speed and pile up audio in the listener’s buffer way faster than it could play. Now it paces itself, memory usage dropped from 7.9 GiB to about 120 MiB while streaming, and the Sonos song titles no longer run hours ahead of the actual audio.

It’s deployed the same way everything else is: a little Docker stack in my home-infra repo, deployed with Dockhand (see How a deploy works with Dockhand). It has an autoheal healthcheck that restarts it if it stops answering, and I get a Grafana alert if failed tracks start piling up in an hour. It’s stateless on purpose, so there’s nothing to back up.

This unlocks a few things: I don’t need AirPlay anymore, the speaker can grab the stream itself so no phone needs to stay awake or at home, and I only need my phone to be on the home Wi-Fi once to add a favorite.

Home Assistant gets the same stream via a script so I can play any channel on any speaker. I have 8 room buttons which set a known starting volume before playing. One more starts a synced Downstairs group: Living Room, Kitchen, Parlour, a Roam. Being a script, any automation can start Poolsuite in a room at a set volume. The TuneIn trick works for any LAN stream so any of them can become a Sonos favorite.

Next up, I need to figure out how to refresh playlists without a restart, since skinnydip only fetches them at boot. I also need to pin the yt-dlp version so a rebuild doesn’t change it. After that, I’m adding physical remotes to the speakers: SYMFONISK remotes to ZHA and retiring the IKEA hub.

BirdNET listening in the backyard

Thu, 13 Aug 2026 00:00:00 GMT

My backyard is full of birds that I can hear (the whole setup is in the project intro), but the first bird-listener I built didn’t pick up almost any of them. When I ran it, the list of detections only had 2 entries, both of which were Grey Catbird. So while the container was up and the web UI loaded, making it look like it was working, practically speaking it was almost totally deaf.

I set up a USB microphone connected to BirdNET-Go, a Go application by tphakala that identifies birds from a live audio stream. It runs Cornell Lab’s BirdNET model and Google’s Perch model, and I wanted to run both. I deployed it with Docker using an Ansible playbook, similar to how I described in Ansible manages every config on my router and access points.

Getting it to listen

First, I was mounting my configuration at /config, but the application was looking for its configuration in its home directory, so it didn’t find anything and generated a default configuration instead. The default audio source was the sysdefault audio device, which of course wasn’t available in the container. The settings I’d written weren’t being read. Mounting the configuration folder at /home/birdnet/.config/birdnet-go fixed this.

Warning · Mount config at home dir

The /config mount is ignored and BirdNET-Go generates a default config instead. Mount the config folder to /home/birdnet/.config/birdnet-go instead.

Once I had the config being read properly, I lowered the detection threshold from 0.8, because having several birds in the same three-second analysis window dilutes each confidence, so a high threshold filters out real calls. Within a couple of hours, I went from 2 detections to over 120 detections of 10 species.

The next issue was new detections being saved to the DB without an audio clip or spectrogram, and the UI showing loading spinners. This was due to another auto-generated config value: a loudness range of 0 caused the FFmpeg loudness normalizer to crash without a useful log. Setting loudnessrange to 7 brought the clips back.

Perch was installed but deaf

I had previously switched from the latest image to dev because latest had become a stale March build that lacked the ONNX runtime that Perch requires. I installed Perch v2 that evening, loaded it without restarting, and it showed as enabled. But when I woke up the next morning, Perch hadn’t received any audio. For 22 hours, all 1,638 detections in the DB were from BirdNET only.

There’s a difference between a model being enabled and a model actually getting audio. The top-level models.enabled are only the models that are available. But each audio source has its own models list that determines which models it sends audio to, and I’d only listed birdnet there. I also hadn’t turned on the false positive filter, so it was logging “False positive filtering is disabled current_level=0”. A clue was the maintainer’s Perch announcement thread on r/BirdNET_Analyzer, which said the filter must not be left at 0 if using Perch. I added Perch to the source’s model list, set the filter level to 1, and set the overlap to 2.0 (the minimum for level 1).

 birdnet:
     sensitivity: 1.2
     threshold: 0.5
-    overlap: 1.5
+    overlap: 2.0
 realtime:
           models:
             - birdnet
+            - perch_v2
     falsepositivefilter:
-        level: 0
+        level: 1

I went with the relatively lenient level 1 setting, even though the maintainer recommends moderate or higher for Perch. I wanted to get some baseline data before tightening it up. At first, this didn’t seem to make a difference, but that’s because docker compose restart was using the old cached config. Only a full recreate applied the change.

# docker compose restart keeps the cached config; recreate instead
docker compose down && docker compose up -d

Once I recreated the container, the logs started showing detections from both models. It discards individual hits until both models agree, and immediately Perch was detecting more than BirdNET. Each was picking up species the other wasn’t: Perch caught a Carolina Wren that BirdNET missed, and BirdNET caught a Red-bellied Woodpecker that Perch missed.

The nonsense from that night

The BirdNET-only night left me with nonsense data. The worst offender was a Eurasian Wigeon (a European duck) with 7 detections in the dark at 0.97 confidence. This matches a known upstream issue #2741 where car horns are identified as Trumpeter Swans. Low-confidence detections and out-of-range detections are flagged as unlikely (but not deleted, to keep an audit trail for reviewing single-model errors).

Clipping at the tail end

Another problem I caught by ear was distortion at the ends of clips, which was also visible on the spectrograms. It turned out that every sampled clip peaked at 0 dB full scale, because I’d set the export true peak limit to 0. Reducing that to -1.5 dBTP created some headroom.

             normalization:
                 enabled: true
                 targetlufs: -10
                 loudnessrange: 7
-                truepeak: 0
+                truepeak: -1.5

Two models, one rule

At this point I have it set up so that both models need to agree for a detection to count, so the false positive filter is really on. I’ve started recreating the container and checking the logs for detections by both models after any configuration change. I’ll watch the data for a day or two and then decide if I want to tighten the filter further.

Home Assistant already receives every detection via MQTT through Mosquitto, with both the species and confidence level exposed as entities. Next idea is to have it announce each new species aloud in the house.

BirdNET near the nature preserve

Wed, 12 Aug 2026 00:00:00 GMT

We live right next to a large nature preserve and birds pass through all the time, so I wanted to know what was actually out there. It started as a fleeting idea and turned into a full end to end thing, all the way from microphone to e-ink displays.

I don’t have a yard feeder, so all of these are birds passing through. I’m using a lapel mic costing less than $15, placed outdoors in a sheltered spot. The mic feeds into BirdNET-Go, which splits the audio stream into 3-second chunks and passes each chunk through two models: Cornell Lab’s BirdNET (trained on ~6,000 species) and Google’s Perch (trained on 14,795 species). The models cross-check each other. Once I had the config sorted, the first evening gave me more than 120 detections across 10 species in about 2 hours. That’s a lot more than I expected for a yard without a feeder.

The second model (Perch) wasn’t getting the audio stream. Everything looked fine in the config, Perch was installed and enabled, but it never got any audio for a day, maybe two. Enabled is not the same as actually listening, so check the logs that both models are hearing things.

This was because there’s one setting in two places: BirdNET-Go maintains a global list of enabled models, and a per-audio-source list of models. I’d added Perch to the global list but not the mic’s list, so it was still only running BirdNET for that stretch of detections. Adding Perch to the mic’s list got both models running on the same audio.

Tip

In BirdNET-Go, models.enabled just says which models are available. The models list on each audio source is what decides which ones actually hear it.

By August, we’d archived about 62,500 detections (about 800/day) and heard 105 species, 5 of which weren’t birds. The katydid was the sixth-most-detected species of all time with about 3,500 detections, earning it the largest tile on the e-ink collage. I originally wanted bugs on the screens, but the katydid made me think otherwise. Bugs are now removed from the displays, but the stats keep every detection.

Aviary (which I promised to write up in an earlier post) polls BirdNET-Go every minute and stores its own copy of each detection in its own database. I opted for a copy rather than reading directly from BirdNET-Go’s database so that the site keeps working if BirdNET-Go restarts, and because BirdNET-Go prunes old detections, so having a copy is a good long-term archive. For all the screens in the house, the site renders its own page as an image and the device itself only has to display the image. It’s the same approach I used for our family e-ink photo frame.

The e-ink frame and the old Pi screen rotate through the birds that have been around, so the data shows up in the house and not just on a dashboard somewhere. It’s a beautiful extension of the nature that surrounds us here at our home.

The old Pi screen is actually an original Raspberry Pi 1, which is too old to run a browser, so I’m rendering a fresh image every 3 minutes. The color e-ink frame is via Pigment, the app from the family photo frame post, with the birds being an extra album I called BirdNET. Aviary borrows from a few earlier projects out there.. the collage and stats ideas from AvianVisitors, the BirdNET-Go API pattern from birdnet_display, and the controller/display split from inky-bird-frame.

The birds on the wall are styled like pages from a field journal. inky-bird-frame comes with a free MIT-licensed catalog of pre-rendered plates, and 59 of my species were already covered, so no new art or paid image tool was needed. Generated art covers display images for 21 more species at a total of $2.20 for the batch.

Note

Aviary is public, so neighbors can see what has been around with a plain link and no app.

I’m pretty happy with how it turned out. Next up is some more art for the birds that don’t have a plate yet and maybe a touchscreen kiosk mode for the smaller 7 inch desk display I have on a Raspberry Pi. The next part of this project is BirdNET listening in the backyard where I dive deeper into the mic and tuning two models to work together.

The Value Flywheel Effect from the CIO's chair

Tue, 11 Aug 2026 00:00:00 GMT

Cloud waste is rising again. The Flexera 2026 State of the Cloud survey estimates wasted cloud spend at 29% (up from 27%), marking the first increase in five years, with AI workloads cited as the primary driver. The State of FinOps 2026 report shows 78% of FinOps practices now report into the CTO or CIO organization, but most of the easy savings have already been realized.

This puts a different spin on The Value Flywheel Effect today. Author David Anderson (with Mark McCann and Michael O’Reilly) published it with IT Revolution in 2022, when people talked about cloud in terms of speed. The thesis that cost and speed follow from strategy first lands harder now that people who own the cloud bill are also being asked about the value of the technology.

The model itself is broken into four phases: clarity of purpose, challenge and landscape, next best action, and long-term value. Each phase has a different leader, with the CEO being first. Each phase is worked with a Wardley map. Hanging off of those phases are twelve tenets. Two that a platform lead might underline are “code is a liability” and “frictionless developer experience.”

This pattern comes out of the serverless-first program the authors ran at Liberty Mutual. In Tech Lead Journal, Anderson says that program reduced run costs by around 90% (his number, unaudited) and improved time to market. The map in the book does the heavy lifting: it positions components along the genesis-to-commodity axis. The rule is to build differentiators and consume commodities.

Anderson’s definition of serverless-first is more narrow than the label might imply. In the same interview, he talks about managed services as the default choice, not turning every workload into a function, and doing the simplest thing that delivers value. That makes serverless-first more of a platform decision than an architecture style.

Gartner says 80% of large software engineering organizations will have platform teams by 2026, up from 45% in 2022. The “frictionless developer experience” tenet can work as a charter for such teams: the platform makes the managed default the easy path. This is close to the cognitive-load argument in Team Topologies is an AI agent a teammate or a platform.

Banks and insurers need to push back on this part. The managed-service design is harder to move. DORA has been in effect since January 2025, requiring EU financial firms to manage ICT concentration risk and plan for exiting providers. Late last year, European supervisors named the first critical ICT third-party providers for direct oversight. So now substitutability is something the regulator can ask the CIO about.

Commodity pricing can cut the other way: 37signals says it saves $2 million/year or so after moving its compute from public cloud to on premises in 2024. But neither of those objections is addressed by the slogan. The book’s own tool can do that: a map showing which components are commodities and where lock-in is worth paying for.

The authors have advanced their argument. Now that code generation is cheap, knowing what to build trumps building fast, and clarity of purpose moves back to the front of the flywheel. But it leaves the CIO wondering: can one Wardley map be the shared artifact for the FinOps team, the platform team, and the risk function?

A family e-ink photo frame in a day (Pigment)

Mon, 10 Aug 2026 00:00:00 GMT

I got the Seeed reTerminal E1004 with the color e-ink panel and battery. I wanted to be able to pick (and crop) the photos myself, from my own server, rather than use Seeed’s stock app or set up Immich like other owners have done. The frame wakes up and polls the server for the next picture, so nothing needs to push anything.. the build only needs to be able to answer on request.

Before I started coding, I looked around for what others had done before me. The r/trmnl thread “Byonk driving a E1004” and the r/immich thread “ReTerminal e1004 working with Immich” (which includes an eleven-step flashing walkthrough and how to hook up Byonk) got me most of the way on the map. Byonk is oetiker’s self-hosted TRMNL device server, so it already speaks the device protocol, and it comes with community-tuned dithering for this exact panel.

I decided against writing a standalone server because it would mean reimplementing the tuned dithering, and the built-in screens in Byonk don’t support photo cropping. So Pigment only does the curation.. it stores photos and crops, and Byonk handles all the communication with the frame and turning a photo into six ink colors. Suddenly, the hard part was already solved by someone else, and I was left with just picking and cropping photos.

I decided to call it Pigment because e-ink physically moves pigment particles to draw things, and “pigment print” is the archival standard for color photo prints, so it fits both the screen and the photos.

I wrote the spec in the afternoon and had a working app by the evening. It lets you upload a photo from your phone, crop it in the browser, and render it at the panel’s 1200x1600 portrait resolution. I designed it to store the original photo alongside its crop rectangle, and the render is derived from both, so you don’t have to re-upload if you want to change the crop. The release tool tagged the first cut 1.0.0, and I was like “we aren’t even close to 1.0”, so I retagged it 0.1.0. From design to five releases in a day.

Here’s a diagram of the path the photo takes from the phone to the glass, including the button presses that reach Pigment before the frame requests a picture from Byonk.

Flashing and first bugs

That same evening, I flashed the community firmware from TRMNL firmware PR #445, which was closed without being merged. I created my own fork, which maps three buttons to next, previous, and refresh, using the button pin map from Frans-Willem/epd-photoframe. When a button is pressed, the firmware communicates with Pigment first (step back or hold current photo) and then asks Byonk for a picture as usual, so Byonk doesn’t need to be changed at all.

I ran into two problems: First, the frame’s Wi-Fi setup page never saved the network join, failing the same way twice. I worked around it by writing the Wi-Fi settings directly into device storage over USB, where they survived later firmware flashes. Second, a crop bug squashed photos taken sideways because the server validated the crop against the image’s dimensions before EXIF rotation. I hadn’t thought about rotation until I saw one photo get distorted, but fixed it the same night.

The dither shootout

The next day, I turned to the picture itself. I scored seven Byonk dither settings on five real photos, rendering each with Byonk and measuring the color distance from the pre-dither original (average Oklab color difference). The chart below shows the mean score for each setting.

Unexpectedly, the live frame performed worst of the seven settings. This was because I’d hardcoded jarvis-judice-ninke in the script, overriding the config, and because I’d spelled “sierra-light” in the panel tuning block instead of “sierra-lite”, which Byonk expects, so the tuning was silently ignored (no error). I wondered how long the worst setting would have gone unnoticed had I not started measuring. A one-word rename in the panel config fixed it.

     colors: "#000000,#FFFFFF,#FF0000,#FFFF00,#0000FF,#00FF00"
     colors_actual: "#000000,#FFFFFF,#C20000,#FFE042,#2160C4,#00994D"
     dither:
-      sierra-light:
+      sierra-lite:
         error_clamp: 0.11
         noise_scale: 5

Numbers and eyes disagreed. Atkinson scored best but its output nudged mid-tones green, a hue drift the average hid. I trusted my eyes more than a single number for photos. I ranked them visually and picked sierra-lite tuned. It beat runner-up floyd-steinberg with a lower error clamp on all five photos, on average and at their worst pixels. Days later, on real glass, sierra-lite still looked good.

It’s now running on battery, changing either when I press a button or once a day by itself. It pages me when the battery gets low. It uses very little power: I thought the battery reading was flat and asked why, it turned out to have dropped from 98 to 95 percent over about four days off USB. I estimate it’ll run for about four months per charge, but that’s a ballpark figure from the first few days of readings, not measured. Since then I’ve added the ability to show a rendered image from another service in a Pigment album, such as the bird collage from BirdNET listening in the backyard.

Every iMessage since 2017, in one archive that only grows

Sun, 09 Aug 2026 00:00:00 GMT

I wanted to turn on Keep 1 Year to reclaim storage on every device, but I didn’t want to lose anything older than a year. So the idea was to capture everything before it aged out and keep it searchable for the long haul.

With Messages in iCloud turned on, this is a global setting across all your devices. So when you switch from Keep Forever to Keep 1 Year, it trims your message history on every device that syncs with iCloud. My message history goes back to 2017, so that’s years of conversations I’d lose across all my devices.

This included attachments and all conversations. v1 was a nightly job on my Mac that copied the Messages database using SQLite’s backup command, copied attachments to my home server, and converted the database to HTML using imessage-exporter. The nightly exports were merged into a single folder that never deleted anything, and imessage-exporter did the heavy lifting of turning Apple’s database into human-readable pages.

Append-only at the file level isn’t append-only at the content level. Every signal looked healthy while history drained out. Each night’s export was rebuilt from a database that by then held only one year, and the merge overwrote the files with the same names, so full-history pages were quietly replaced by one-year versions with no error and no log line.

I discovered this by doing a coverage check against the Messages database from my old 2019 laptop. Only about 12% of my messages from 2017 to 2019 were still in the folder. The result was so far off the design that I suspected the check itself first, but it held up. The weekly snapshot from May on the old server had all of it, and I recovered the missing history from the snapshot that same day. This is a great example of why testing backups with a real restore is so important.

Now the databases are the archive and the HTML is just a view I can rebuild on demand, or whatever format I need at the time.

Now I have a single master database that only ever grows. I can feed it from any Mac or old HTML export, and it keeps track of where every single message came from. I also feed the nightly message count into VictoriaMetrics so I get alerted if it ever drops, and keep an encrypted offsite copy of the archive with restic.

The newest source is The always-on Mac. The first time I pulled from it as a live source, it added 18,111 messages to the archive that the archive didn’t have. It makes me wonder where they were all along.

It seems that the Keep 1 Year trim only reduced the size of the Macs, not purged the older history from iCloud. The evidence is that the always-on Mac (which is still set to Keep Forever) has messages going back to 2017. Now I just need to finish cleaning up the old copies and start building the search on the archive.

Data hoarder mentality for sure, but the historical conversations might be useful in the future (ie finding an old text thread with my late father).

Edited on 2026-10-24 to add the message-count chart, the always-on Mac as a source and the Keep 1 Year/iCloud finding.

Team Topologies: is an AI agent a teammate or a platform?

Fri, 07 Aug 2026 00:00:00 GMT

If we start with the premise that a team has a limited mental capacity, what is the effect of adding an AI agent to that team? This question keeps bringing me back to Team Topologies by Matthew Skelton and Manuel Pais, a book that keeps coming up in discussions around platform engineering and SRE. The second edition, published by IT Revolution in 2025, includes case studies from ING and GovTech Singapore, and the afterword extends the concepts to AI-augmented organizations.

The premise is that since a team’s thinking capacity is finite, the work it owns should be sized to what it can carry. This leads to small, long-lived teams with boundaries drawn to minimize cognitive load. In chapter 3, the authors concede that there is no precise measure. “While there is no formula for cognitive load, we can assess the number and relative complexity (internal to the organization) of domains for which a given team is responsible.”

The model breaks cognitive load down into 3 types: Intrinsic (the difficulty of the problem itself), germane (the value-adding learning for the team), and Extraneous (all the bullshit the environment piles on, like “how do I deploy this again?”). The book wants to see the Extraneous engineered away so that there’s more room for the meaningful learning.

There are four team types built around reducing load. The stream-aligned team owns a work flow end to end, and the other three team types are mostly about relieving that load, either through a service they consume or through a temporary collaboration. The platform team is the clearest example of this.. their main job is to reduce the cognitive load of the stream-aligned teams who use the platform.

This book gets some pushback. Some people complain that organizations adopt the four labels but don’t adopt the conditions that make them work. Some people complain that platform teams become bottlenecks. Some people point out that politics and incentives are outside of the model. A recent thread on r/sre asked if a “team” is sometimes just a group of people who share a job title but work on unrelated epics.

This year, the discussion shifted to agents. At QCon London, Skelton argued that agents (like people) lose coherence when they operate outside a defined scope.. they need bounded agency. His question: why give an agent write access to every data store when no person would get it. Olivier Wulveryck mapped the four team types onto an agentic platform, where the platform absorbs technical complexity. Wulveryck also pointed out that agents give nearby humans “anticipation burden”.. since they respond instantly.

This made me think about how I use agents. I run one agent session per project, splitting responsibilities so each one stays small enough that I can keep up. It keeps things clean and aligns with how I think (more on that in Running an agent fleet like a small team). In the book’s terms, it’s a boundary decision about my own load as much as it is about the agents’.

I’m still not sure if an agent is a team member who needs a load boundary, a platform that lifts load off me, or just another tool. And does the answer change when the agent starts acting without asking first?

One antenna, live TV in Plex on every TV in the house

Thu, 06 Aug 2026 00:00:00 GMT

OTA TV and DVR without putting an antenna on every TV. One antenna on the 3rd floor pulls in NYC stations.. I got it working with a Pi and the old Hauppauge tuner first, then picked up a $40 tuner on eBay.

I started this on a Sunday in June, in my basement. I had an old Hauppauge WinTV-HVR-955Q USB tuner in my home server, with an antenna connected at the rack. The first scan locked 0 channels, but I blamed the weak signal on the antenna’s position, not the tuner. I moved the antenna and rescanned, locking 62 channels: the entire NYC market except ABC 7.

Tvheadend runs the tuner in a Docker stack and serves channels as streams. It doesn’t show anything on a TV, so the real decision is what app the house will watch in. I deployed Channels DVR Server first, but learned later it’s subscription-only and removed it the same day. The commit message for removing it was “Channels DVR is subscription-only; using Plex Pass (lifetime) + Threadfin’s HDHomeRun emulation instead.” I already owned a lifetime Plex Pass, so Plex Live TV and DVR came at no extra cost.

Plex doesn’t speak Tvheadend directly, so Threadfin bridges the gap. It’s a maintained fork of the abandoned xTeVe project, which takes Tvheadend’s channel list and pretends to be an HDHomeRun on the network. In Plex, I clicked “Have an HDHomeRun?”, added it like a store-bought network tuner, and Plex started supplying its own guide data. Plex has no idea it’s not real, and when it tunes a channel, Threadfin just redirects to the Tvheadend stream. The Threadfin status page shows 62 channels and 1118 programs.

The goal was “Live TV on the Apple TVs.” OTA video is MPEG-2, so Plex needs to transcode to H.264 for the Apple TVs, which it does via Intel QuickSync on the server’s i7-10700 at near-zero CPU cost. By the end of the first day, I had end-to-end live TV working in the Plex app on my iPhone (picture and sound).

ABC 7 was the only station I couldn’t get. They broadcast on VHF and the antenna I have is biased toward UHF, so I never locked the scan. I was curious if there was somewhere in the house where I’d have to put it to pick up channel 7. I was going to look up the towers on rabbitears.info and try to move the antenna to a window or outside.

The move took the next few weeks, and the antenna made it up to the 3rd floor, where I proved the setup with the old Hauppauge tuner on a Pi up there. By the time I had the DVR working in July, Plex could record as well as play live. The $40 eBay HDHomeRun Duo made it simple, since it was the real version of the network tuner Threadfin had imitated.

The family can watch and record live TV on any TV in the house now since Plex is available on all Apple TVs and iOS devices.

Thinking in Systems says headcount is the weakest lever

Mon, 03 Aug 2026 00:00:00 GMT

Thinking in Systems is a primer on stocks and feedback loops by environmental scientist Donella Meadows. It was published posthumously by Chelsea Green in 2008, but it continues to make the rounds on engineering and strategy reading lists. The value is in a single ranked list, adapted from her original essay Leverage Points: Places to Intervene in a System. The list ranks twelve places to push on a system, from least to most effective. You might be surprised which familiar levers rank low, and how far at the bottom you’ll find the numbers we usually adjust first. (“99 percent of our attention goes to parameters,” she writes.)

This list makes me uncomfortable when I think about it in terms of an enterprise transformation program. Parameters are headcount and budgets.. the dials leadership can turn from a slide, so that’s what programs turn. Higher up are information flows (who sees cost and incidents), rules (incentives and approval paths), and the goals the system is really optimizing for. I think most enterprise transformations fail at the leverage-point level before they even get to execution.. they change some numbers but leave the structure. Meadows also observes that the real leverage point in a company is often already being pushed on, just in the wrong direction.

Meadows talks about how cheap wins hide in information flows.. that adding (or restoring) information is often more powerful than rebuilding infrastructure. In my own homelab, I had an auto-restart loop that kept restarting a container with a perfectly fine service because the healthcheck was testing the wrong thing. I fixed it by fixing the information the loop was acting on, not the loop itself. (Autoheal opt-in and healthchecks that actually test something)

This lens also applies to AI. The 2025 DORA report shows how AI amplifies existing state.. the returns track internal platform quality and clear workflows, not how broadly you rolled out the tools. In Meadows’s terms, I’d describe AI adoption as a reinforcing loop that amplifies whatever it starts with. Organizations that already changed their rules and information flows get better results and invest more. Everyone else mostly buys more seats. If you’re leading an AI program, which dial is it turning?

Hybrid search over my Obsidian vault, always on and always current

Sat, 01 Aug 2026 00:00:00 GMT

As an Obsidian user I needed fast, accurate, always current search across my vault. I’d used qmd a lot at work, but my vault changes all day and gets searched all day by me and my agents, so I wanted something always up and watching the files.

qmd is a really good local search tool and I’ve borrowed a lot of ideas from it for my own build. But it suffers from a cold start. Local models need to be reloaded every launch, so it takes 15 seconds or so. The only way to avoid this is to have a server process that keeps the models in memory.

It’s called niko. This is an homage to a dog named Niko I had when I was growing up. He was an incredible black Labrador retriever. He always fetched brilliantly.

I have it running as a persistent service that watches my vault and reindexes, re-embeds, and keeps up with my ever changing vault so myself and my agents have instant accurate and up to date context when needed (which is often!) I expose a CLI with vs and an API both (via MCP or direct API) and I have my Claude Code wired up with a skill to it. I have plans to integrate future agentic projects with it too.

On each query, it performs keyword and fuzzy-title searches across 7,000+ notes, runs vector search, and merges the results into a single ranking. Keywords are handled by SQLite FTS5, and vectors by sqlite-vec. The caller here is the Memento agent, which asks niko for JSON responses so it can take action.

I wanted search and retrieval over my knowledge base that works close to the way I remember things: hand me the snippet I need, or point me (or an agent) at the whole file when that’s what it takes.

Fast and always on is only half the battle. I also wanted vault awareness. I wanted features that let me slice my corpus into buckets and provide dedicated instructions for each bucket of documents and files.

Collections are sections of the vault (Research, my Reddit saves, etc.) and each one carries a weight and a context description. Largely inspired by qmd. That context shows up on every result as a Context: line, so a positive hit from my Reddit saves gets flagged as third-party opinion, so an agent won’t mistake it for a decision I actually made or something I actually wrote. Filters on scope, tags, frontmatter and dates are soft boosts (nudging the ranking without hiding anything), and --strict makes them hard filters.

My Reddit saves collection is content that Stash imports into my vault. This is content that the agent should consider another person’s opinion, not my own notes.

The daily restarts… The reranker was blocking everything else, the health check failed and autoheal kept restarting it, and I only noticed by chance. Moving the reranker off the main thread fixed it, and it’s why I added alerting for things that fail quietly.

I assumed the model library was already doing inference off the main thread, but a small repro harness proved otherwise. Disabling reranking was an option, but I wanted to keep it, so I moved the reranking to a worker thread. Now the health check answers 300 of 300 requests while a three-minute rerank happens, so autoheal has no reason to intervene.

The resulting alerting covers two quiet failures: one for container CPU, and another that pages me when the file watcher goes quiet and stops logging file events. Both pages are delivered via my single alert relay.

Part 2 is how I get it more accurate: testing, measuring, and the model swaps that came out of it. After that, more of my agent projects plug into it, NanoClaw first.

When AI writes the code, the architect's job is the elevator ride

Mon, 27 Jul 2026 00:00:00 GMT

If AI agents are writing most of the code in the engine room, does the big company still need someone to ride between the engine room and the penthouse? This is a 2026 update to a question posed in Gregor Hohpe’s The Software Architect Elevator (O’Reilly, 2020) about the purpose of an architect in a large organization. My answer is yes, and riding is now the bulk of the job.

I’ve spent ten years working with large financial institutions and Fortune 500 companies. My current role as a principal cloud architect puts me on the elevator by definition. Before that I was a developer, development manager, infrastructure manager, sysadmin, small business owner, and consultant.

In Hohpe’s building, the executives work in the penthouse, the IT folks work in the engine room, and there are many management floors in between. Information takes the stairs, but it loses meaning on each floor (telephone game style), so the executives end up funding the wrong things and innovative engineering projects don’t get the support they need. The architect rides the elevator both ways, translating strategy into architecture and bringing execution learnings back up. And then keeps riding.

That’s all in his original essay on martinfowler.com, and one of the ideas people often call out is that an architect needs to be able to speak the language of every floor they interact with.

In the same essay, he writes “most of what architects have traditionally done should be done by developers, by tools, or not at all.” This was written years before coding agents, so the “by tools” part is now literal. If developers (and agents) can do the drawings and the boilerplate, then the architect is left with the part that can’t be delegated: the ride itself.

The case for the ride is even stronger when you consider what agents changed. Writing code is now practically free, but knowing for sure that a change is right hasn’t gotten any cheaper. Addy Osmani talks about this in Agentic Code Review.. how the constraint moved downstream to verification, and how review rigor should scale with the change’s blast radius. GitClear’s data, which Osmani cites, shows daily AI users writing ~4x more raw code for ~12 percent more delivered value.

This changes what the penthouse sees: a working demo in days. The temptation for executives is to read that as a solved problem and write off the architect as overhead. But I think the most valuable trip now is upward, carrying what the demo hides. Missing from the demo are the verification cost and how far failure spreads. Also absent are the run cost and the regulator’s questions.

Also carried up is the gap between what the business says it needs and what it actually needs. That’s the job. I think cheap code widens that gap: the agent builds what was asked, quickly and convincingly, even if the request was wrong.

The strongest pushback is that architects who stopped coding are an ivory tower anti-pattern, and senior engineers do architecture anyway, so AI just removes another layer of overhead. Another common complaint is that architects rarely have a manager’s mandate, so their word only counts if backed by leadership.

They’re right about architects only riding the top floors. An architect who doesn’t touch the engine room has stopped riding. In 2026, staying in the engine room means using the same agents and pipelines as your teams. The mandate point is fair.. having the answer is easy, but the elevator is worthless if you can’t decide and own, and that comes from leadership. I split the work: delegate low-level design, own the contracts between systems, dive in only where the risk lives.

If an engineering leader asks me if architects are worth the chair, I’d say: keep them, but measure them on translation, not diagrams. Did decisions get made faster? Did we reverse course less often? Did AI pilots get killed early or scaled for reasons we can state clearly? Expect them to use the same tools your teams do, hands-on. Give them a seat at the table where AI investment gets decided.

Hohpe says architecture is about selling options. Cheap code means cheap options.. your team can build several versions of almost anything in the time it took to build one before. Now the architect’s job is to price those options, decide which ones are worth verifying and running, and kill the rest before they hit production.

Autoheal opt-in, and healthchecks that actually test something

Sun, 26 Jul 2026 00:00:00 GMT

The first sign of trouble was when autoheal kept restarting a container even though the service in the container was fine. It turned out that the image didn’t include curl, so the curl-based healthcheck never succeeded. I fixed that by rewriting the healthcheck to use Python’s urllib instead. A few days later, the Z-Wave controller was restarted twice in the middle of its own soft reset because the healthcheck failed while the controller was busy. I fixed that by loosening the healthcheck to about three minutes.

Autoheal is a small container that watches Docker healthchecks and restarts containers that aren’t healthy, paging me via the Apprise relay to Pushover. I had it running label=all, which put every container with any healthcheck in scope, including hurried healthchecks I’d written myself and healthchecks baked into images I hadn’t read. The scope included the agents’ credential proxy, which holds OAuth grants and shouldn’t be restarted blindly. When I did a best-practices review of the server in June, label=all was the biggest operational red flag: the wrong healthcheck could trigger a restart loop, bouncing an identity-holding service unattended. The review advised me to do both: make autoheal opt-in and audit the depth of healthchecks. Autoheal is now opt-in, and the diagram below shows the bar a check must clear before it gets the label, with the old loop underneath.

The label

The switch was one env var: AUTOHEAL_CONTAINER_LABEL from all to autoheal, so only containers with autoheal=true are watched. I kept the same coverage: the label went on exactly the containers already watched (36 services, 29 compose files, including image-provided checks). Changing the label recreates the container, so I did all 36 labels in one commit and held them until a push window. The credential proxy stayed deliberately unlabeled, so its restart is manual.

  autoheal:
    image: willfarrell/autoheal
    # Watches every container with a HEALTHCHECK; restarts any that go unhealthy.
    environment:
      AUTOHEAL_CONTAINER_LABEL: all
      AUTOHEAL_INTERVAL: 300
      AUTOHEAL_START_PERIOD: 60
      WEBHOOK_URL: http://apprise:<port>/notify/<key>

  autoheal:
    image: willfarrell/autoheal
    # OPT-IN (2026-06-11, was label=all): only restarts unhealthy containers
    # carrying the label autoheal=true. A wrong/shallow HEALTHCHECK on an
    # unlabeled container can no longer cause a restart loop, and
    # identity-holding services are deliberately excluded.
    environment:
      AUTOHEAL_CONTAINER_LABEL: autoheal
      AUTOHEAL_INTERVAL: 300
      AUTOHEAL_START_PERIOD: 60

Once autoheal stopped covering for me, the hidden problem showed up: my two DNS query-log shippers from AdGuard Home and the query-log shipper went unhealthy once per boot. Their old autoheal restarts had coincided with every long boot, always six minutes or so in, and autoheal had fixed it every time, so I’d never seen it. The cause was Dockhand re-owning the entire stacks folder at startup, which flipped the owner of the shippers’ cursor file I’d kept in there. I moved the shippers’ runtime data to the storage pool and made a runbook rule: no runtime data ever inside the stacks tree. How a deploy works with Dockhand explains the folder and why Dockhand considers it its own.

deep checks that reach the real service

The depth audit changed what “healthy” means for some compose file or image checks. For Forgejo, it’s / (proves only the web listener) to /api/healthz (also checks the database and cache), which matters for the git server in Self-hosted git with Forgejo and why some repos never touch GitHub: the listener can answer even if the database is gone. For Home Assistant, it’s / to /api/, accepting either 200 or 401. Containers that lacked checks got new ones: ten total, across the observability stack from Metrics and logs from the network gear and the self-hosted archive app. The archive app’s Postgres now runs pg_isready, and its Valkey cache pings (both PONG and NOAUTH mean alive, both prove a live responding server). The cache’s password stays out of the healthcheck.

  forgejo:
    healthcheck:
      # /api/healthz checks DB + cache backends, not just the listener.
      test: ["CMD-SHELL", "wget -q --spider http://localhost:<port>/api/healthz || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
    labels:
      - "autoheal=true"

    # PONG|NOAUTH both prove a live, responding server; avoids putting the
    # password in the healthcheck (it's only on the command line, not in env).
    healthcheck:
      test: ["CMD-SHELL", "valkey-cli ping 2>&1 | grep -qE 'PONG|NOAUTH' || exit 1"]

Every command was run in a live container with docker exec before committing, which paid off immediately: the archive app’s localhost resolved to ::1 but the app listened on IPv4 only, so the checks now probe 127.0.0.1. Two compose files claimed in comments that they lacked curl or wget, but busybox wget was there.. docker exec beats a comment. Some images have no shell (Tempo, AdGuard’s sync container), so no check, but that’s covered by the target-down alert and Uptime Kuma.

Ship unlabeled, soak, then label

I’ve started deploying new checks without autoheal=true.. they earn the label after a quiet stretch. A wrong check with autoheal=true means an instant restart loop; without it, it just shows unhealthy. This paid off a month later when a Loki 3.4 → 3.7.3 image update broke the exec check (no /bin/sh, no wget), so Loki stayed permanently unhealthy but didn’t loop because it lacked the label. Very satisfying to see a rule I’d written specifically for this case work perfectly the first time it mattered. I removed the check (Loki is covered by the target-down alert and Kuma like other shell-less images), but the discipline held.

A passing check can still be broken

In a late July fleet health audit (a 48-hour sweep of all 78 containers’ logs, fanned out to read-only subagents in a Claude Code session), I found node-exporter logging ~1,230 errors/minute (~1.7 million lines/day into Loki) after an image bump. Container was healthy, Kuma was green. Turned out it was its own healthcheck: wget --spider closes the connection after headers, and version 1.12.1 logs a broken-pipe error for each metric family (~615/check). I’d never thought of a check being green and noisy at the same time. New repo rule: an exit-0 healthcheck can still trash the app’s log.. watch what the probe makes the app write, not just its exit code. A full-body fetch fixed it (errors → zero).

     healthcheck:
-      test: ["CMD-SHELL", "wget -q --spider http://127.0.0.1:<port>/metrics || exit 1"]
+      # Full-body fetch, not --spider: v1.12.1 logs one broken-pipe ERROR per
+      # metric family (~615/check) when the probe closes after headers.
+      test: ["CMD-SHELL", "wget -qO /dev/null http://127.0.0.1:<port>/metrics || exit 1"]

Two days later, the same rule caught Home Assistant: a check that counted a 401 from /api/ as healthy, but the HTTP integration logs every unauthenticated request as a http.ban warning (2,879 of them in 24 hours, plus a standing “Login attempt failed” notification) while the container read healthy the whole time. Both candidate probes were tried live in the container first.. /manifest.json gives a clean 200 without auth. After the recreate, the warning count went to zero.

     healthcheck:
-      test: ["CMD-SHELL", "code=$$(curl -s -o /dev/null -w '%{http_code}' http://localhost:<port>/api/); [ \"$$code\" = \"200\" ] || [ \"$$code\" = \"401\" ]"]
+      test: ["CMD-SHELL", "code=$$(curl -s -o /dev/null -w '%{http_code}' http://localhost:<port>/manifest.json); [ \"$$code\" = \"200\" ]"]
       interval: 30s

The placebo, replaced at the source

The fix I’m proudest of is the Reddit archiver from Stash 3 running my Reddit archive like a real service. The image’s healthcheck was python -c "sys.exit(0)" (always passes), and the run loop kept going after a failed run, so failures were invisible no matter how long they lasted. It’s my own app, so I fixed it there: the app now touches a heartbeat file at boot and after each successful run, and the check fails if the file is older than HEARTBEAT_MAX_AGE. One image, two cadences (via env var): the regular archiver now fails after two missed two-hour cycles, the refresh container after two missed daily cycles plus an hour of slack. Both lost their autoheal labels while the new check soaks.

       - SCHEDULE_INTERVAL=86400
+      # 24h cadence: 2 missed cycles + 1h slack
+      - HEARTBEAT_MAX_AGE=176400
     restart: always
-    labels:
-      - "autoheal=true"
+    # autoheal label removed while the success-gated heartbeat healthcheck
+    # soaks (two-step rule); re-add after ~a week healthy

The smoke test exposed another issue: a run without credentials reported “completed successfully”. The script returned an exit code of 0 on config validation failure. I thought the exit code was honest, but I was wrong. A success-gated check is only as honest as the wrapped exit code. This would have refreshed the heartbeat on failure, turning the check into another placebo. Now the script exits with 1 on config validation failure.

Still soaking

The bar now is a check that tests the real service, is smoke-tested in the live container, soaks unlabeled, and stays quiet in the app’s log. The label never goes on a credential holder. As of this writing, the June batch of soaks, which includes both archiver containers, is still unlabeled, well beyond the week I allotted for them. Removing the label was a single commit, but restoring it requires my return and judgment that the check earned it. Newer services like the Home Assistant MCP server and Matter server follow the same rule: labels deferred.

MCP is the API design we owed humans first

Sat, 25 Jul 2026 00:00:00 GMT

You have to call like seven APIs to get a VM in the cloud. Create a VM, a network, a storage device connected to a VPC and then attach credentials. That’s not intent based. I want a VM.

If you’ve ever provisioned a machine by hand, you’ll recognize this as a tedious series of steps. If you give an agent a bunch of raw calls like this, it’ll turn into a mess like a newcomer-on-day-one would make.

Kubernetes does the exact same thing. When you say give me an ingress or a service in Kubernetes, that reflects out to seven calls too, to create load balancers, SSL certificates, DNS setup.

The Kubernetes Service is a nice example of this outcome-declaring design. You declare the outcome you want (“give me a stable endpoint in front of this set of Pods”) and if you want a LoadBalancer, your cloud provider will build you a load balancer, and the controllers will take care of fanning out. (If you want to read the origin history of this, there’s a long discussion on The Pragmatic Engineer)

The fundamentals of this isn’t everything is going to change because of MCP. What the hell? A whole conference, guys? This is just API design.

The MCP specification defines the tools an agent can discover and call, which is useful plumbing. But it doesn’t help you design better tools. If you have a chatty API, putting it in MCP just gives it a new front door.

So what? Create VM v1, create VM v2, who cares? Because these are intent based APIs. So I think these new tools are reminding us of this.

People spend years arguing about the style of an API, but in the end most of it doesn’t matter as much as “does this call actually do what the caller wants?” Versioning can be solved. An agent guessing seven steps behind a VM is going to get some wrong. So is every engineer working from the same reference docs.

I’m watching people write these huge markdown files to give the agent context. How about you write documentation to give me context so I can have fully working examples.

Say the platform team exposes a single “give me an environment” call.. the agent and the engineer both use it. No wrapper. If my internal platform requires an agent to wrap it and a markdown file to use it, what does that say about the golden path I’m giving my own engineers?

The Mythical Man-Month asks whether agents count as manpower

Tue, 07 Jul 2026 00:00:00 GMT

When programs are late, you slip the date and add more headcount. In The Mythical Man-Month (1975), Frederick Brooks wrote that adding more people to a software project that is already late will make it even later. The 1995 anniversary edition includes the original text, as well as “No Silver Bullet” and a twenty-year retrospective.

Martin Fowler revisited the book this month and said the most valuable lesson for him was conceptual integrity. The reflex has a new form: a widely shared ExperiencedDevs thread described how managers assumed a 5x productivity increase from AI agents before work even started, turning a five-person, two-month plan into twelve days.

Brooks’s law has two costs: the time newcomers need to ramp up before they can contribute, and communication paths that grow as n(n-1)/2 (so four people need six channels, seven need twenty-one). The line that carries the argument in chapter 2 is “Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication among them.”

The law still holds for human teams. A recent ExperiencedDevs post described a team that was three weeks behind, added three developers, and finished five weeks behind. The original four spent about half their time explaining the rationale behind past decisions. No review tool could tell the newcomers why the auth layer had the shape it did.

Agents seem to avoid the first cost: they don’t ramp up like humans, and you don’t need to call an onboarding meeting. So the law seems optional. But Google Research measured multi-agent setups and found coordination improved by ~81% on work that could be cleanly split, but got 39 to 70% worse on sequential work. That’s Brooks’s partition condition, measured on silicon. And when agents operate independently without an orchestrator, errors get amplified up to 17 times.

The book’s other ideas matter more for large programs. Conceptual integrity means the design comes from one mind or a small group in agreement. A surgical team has one lead and supporting specialists, and the Hacker News discussion of Fowler’s post reframed it in terms of one person directing agents playing multiple roles. In regulated shops, the sequential delivery stages (architecture review, testing, security sign-off, change approval) don’t magically get faster just because the code got cheaper.

The DORA 2025 report says AI adoption raises throughput but delivery stability still drops, and the gains go to loosely coupled systems. McKinsey and Oxford say large IT programs already run 45% over budget and deliver 56% less value than planned. Modernization and bank transformation timelines.. do agents change the failing number, or just add another vendor stream to coordinate? Some vendor pitches promise decade-long mainframe programs in three years.

I’ve spent the last ten years working with large financial institutions and Fortune 500 companies, now as a principal cloud architect. Before that I was a developer, then a development manager. If agents are capacity, then the things Brooks protected become design work.. a small group holding the design, clean partitions, fewer channels. The human reviewers and approvers become the scarce channel all agent output must pass through. So when a program is late, do you add more agents? Or do you first reduce the number of things that have to talk to each other?

Designing Data-Intensive Applications, read from the cloud side

Mon, 06 Jul 2026 00:00:00 GMT

How much have nine years of cloud changed a book that focuses on fundamentals? I was prompted to think about this because this spring we got the 2nd edition of Designing Data-Intensive Applications! Martin Kleppmann wrote the 1st edition in 2017, and since then it’s been the default recommendation for data-systems reading. For the 2nd edition he brought on a co-author, Chris Riccomini, and O’Reilly published it. They did a staged release, where you could read the early-release chapters for months before the print edition shipped.

The discussion around the release gives you a sense for how people relate to the book.. the top reply in the r/dataengineering release thread was a joke that most people will buy the new edition but still not read it, which was upvoted by over eight hundred people in the thread.

The new chapter 1 (“Trade-Offs in Data Systems Architecture”) introduces the book through the lens of three trade-offs: operational vs analytical, cloud vs self-hosting, and single node vs distributed. The section on cloud, which The Pragmatic Engineer ran as an excerpt, talks about how in cloud-native systems, local disks are “more like an ephemeral cache” and how you instead build on dedicated storage services like object stores. This in turn means that storage and compute get separated out into distinct pieces.

That same chapter doesn’t just say “the cloud is great” either.. it says that if you have an experienced team and a predictable load, it’s often cheaper to buy your own machines and run the software yourself, which stood out to me given the new cloud-heavy framing of the chapter. It makes it sound like a deliberate cost decision to self-host.

The chapter also talks about what operations looks like in the cloud era: instead of tending to individual machines, it’s about choosing services and migrating to new ones when your needs change. The rest of the book keeps the same spine as the first edition.. there are still chapters on replication and consensus.. but the chapter on indexing now includes vector indexes like HNSW.

Readers seem to disagree about how much it’s changed.. one person said Kleppmann called the edition fairly incremental and that first-edition readers don’t necessarily need to re-read it, but another person said right after the official release that it reads like an entirely new book. Some worry it’s become more cloud-centric and say they’ll keep the first edition around for a more old school view.

One long-running critique of the book is that while it’s a decent survey, it doesn’t provide the theoretical background you’d get from a university course on distributed systems. People defending the book say that the breadth is the point.. it’s useful for people who don’t know what they don’t know and need to make buy decisions. People who want more depth on storage engines tend to point to Database Internals.

The authors have also moved on from some things in the first edition.. in the BugBash podcast with both of them, they talk about how the new edition is framed around the move away from local disks and towards object storage and databases being rebuilt on top of S3. They also revisit the CAP theorem and suggest “offline availability” as a more useful way to think about that trade-off.

My homelab is the self-hosting case study for this chapter: predictable load, one machine at home. And the Stash archive already separates the record from the views: an append-only record in JSON sidecars, with notes re-rendered from it. I described that in Stash 2 the refresh engine that keeps my Reddit archive alive. So I’m left wondering: if the cloud treats local disk as cache, what would I gain (and give up) by doing the same with my own disk on a one-machine home setup?

Metrics and logs from the network gear

Sat, 27 Jun 2026 00:00:00 GMT

One afternoon this month, I decided to onboard all my network gear into my existing VictoriaMetrics+Loki stack (which was already monitoring my home server) for metrics and logs, with Grafana dashboards on top and alerts via Pushover. Previously, my OpenWrt boxes and HP switch were only being monitored by Uptime Kuma ping checks, the first of which I’d added that same morning. Sure enough, the following day I got an alert that the switch had unexpectedly rebooted.

I started collecting logs before that, though. The system log on OpenWrt is a ring buffer in RAM, so older entries are lost when it rotates. Back in April, I started sending the router’s logd to rsyslog on my old server via UDP, creating one file per host per date, which made it possible to investigate events after the buffer had rotated. Later I switched logd to send to Grafana Alloy instead via TCP, which pushes to Loki with host and app labels.

I didn’t get it right the first time, though. The default syslog listener format in Alloy is RFC 5424, but OpenWrt’s logd sends the older BSD RFC 3164 format, so every message was rejected until I set the syslog format option to rfc3164. Once that was sorted, I added the same syslog task to the AP role in my Ansible setup, so now all three APs stream logs to Loki alongside the router.

I started small: I wanted SNMP monitoring for the switch and something similar for the router that wouldn’t eat resources. The router has 1 GB RAM, of which ~400 MB was free, and prometheus-node-exporter-lua is in OpenWrt’s 25.x feed, so that seemed like an obvious choice. I installed it with the base collectors and a NAT traffic collector, listening on LAN-only, via a new Ansible task toggled by a variable. My first check-mode dry run failed because the exporter’s uci config didn’t exist until after the package was installed, but that was easy to fix with a probe that’s safe in check mode. Once deployed, the endpoint returned 1,658 lines of metrics.

Halfway through the deployment, I realized the APs could export metrics too, so I looked at some community Grafana dashboards for this exporter to see what I was getting into. All three APs now run node-exporter-lua with the wifi and wifi_stations collectors (for per-station signal strength and rate), bringing the total to four OpenWrt devices I’m scraping. I’m using the pull model, so VictoriaMetrics scrapes each exporter directly, and Alloy just logs and traces (which had been the server convention for about a week at that point).

The switch is the odd one out: it’s an HP 1810G-8 running firmware P.2.22. I checked every page in its web UI.. no SNMP settings anywhere. But the SNMP agent answers anyway (v1 and v2c, with a fixed read-only community string, take it or leave it). Fine. Read-only is fine. LAN-only is fine. snmp_exporter v0.29.0 with the stock if_mib and system modules, scraped every 30 seconds by VictoriaMetrics.

Once the data was flowing, I could set up the network-gear alert group to send Pushover messages. First, if the switch becomes SNMP-unreachable for 10 minutes (which would also catch it falling back to its factory IP, which had happened earlier the same day when a power cycle reset the IP and password). Second, if the sysUpTime resets, indicating an unrequested reboot. Third and fourth, the most important switch ports: the router uplink and the living-room AP port. And last but not least, a per-device alert if the exporter goes down for 15 minutes.

One more alert came from the AP logs: since the office AP is on a DFS channel, I set up a Loki rule to match hostapd’s DFS-RADAR-DETECTED and DFS-NEW-CHANNEL lines. The match tokens needed to be anchored, because a loose “cac” also matches “cachesize” in the dnsmasq logs. I checked against seven days of logs and confirmed that there were zero radar events. I’m using Grafana dashboard 11147, which I provision into Grafana as “OpenWrt Fleet”.

I tripped over a few things:

snmp_exporter’s stock config already defines a system module. I defined it again in a second config file and the exporter crash-looped on the duplicate key.. two fix-forward commits to sort that out.
The switch-down alert fired while the switch was perfectly healthy. up == 0 returns no series when the target is actually up, and with no-data set to Alerting, that silence turned into a permanent alert. Caught it via Pushover within the hour.. the fix was adding an absent() check and setting the no-data state to OK.
Dashboard 11147 chains its variables and assumes instance labels in ip:port form, which mine don’t use. I replaced them with a single instance variable and rewrote 138 label matchers.
The overview panels on that dashboard only work with the exporter’s openwrt collector enabled, so I had to add it.
The dashboard is old enough (schemaVersion 19) that its datasource references are plain strings matched by name. Every panel fell back to Loki and sent it PromQL queries.. I converted 68 references to uid objects.
When only the scrape config changes and the VictoriaMetrics service itself doesn’t, nothing restarts it, so new scrape targets just don’t appear until you send a SIGHUP.

I considered NetFlow, but decided against it: while softflowd on the router itself is light enough, I would need to set up a collector stack on the server, which isn’t worth the added value. I already have nlbwmon running on the router for per-client bandwidth accounting, and node-exporter-lua now covers interface counters.

The next day, the switch reboot alert triggered. The sysUpTime graph shows about 29 hours of uptime and a single reset, taking the server uplink down for about 66 seconds. Unfortunately, the switch’s log lives in RAM and has no clock, so the reboot erased the reason. I configured switch syslog, but it only supports classic BSD syslog over UDP, and the port isn’t changeable, so I set up a second Alloy listener. A packet capture showed no packets when config and web interface events triggered, so I don’t know if it sends anything. The next link event will be the real test. I also added an alert independent of the switch: the carrier-change counter of the server’s own NIC, so I’ll see a flapping link even if the switch stays silent.

The same night, the switch went dark for hours. The server’s metrics revealed the uplink was down for about 6 hours 45 minutes, with 253 continuous local scrapes.. no gaps, so the outage was fully visible even though the switch itself reported nothing. The culprit: power. The switch was running on a passive PoE splitter stepping 48 V down to 12 V, which clearly couldn’t handle the load. The next day, I moved it back to its own 12 V adapter.

Samba for Time Machine: what broke under a green dashboard

Thu, 25 Jun 2026 00:00:00 GMT

The Samba container was healthy in Dockhand when I first deployed it, but it refused to login. I’m using it to do Mac Time Machine backups over the network to ZFS datasets on my home server, plus two more shares for file offloading and a 106 GB cold archive. I’d never had SMB on this server before, neither on the host nor in any container. I chose the ServerContainers/samba image because it includes vfs_fruit, a Samba module that makes the share behave the way macOS expects.

That green deploy hid five separate problems, none of which Dockhand told me about because it only said the container had started and its healthcheck was passing.

No user account.. every login was refused, but listing the shares as a guest still worked.
The hardening was silently ignored.. the shares had no host gate at all.
The healthcheck was green the whole time.
My home VPN clients were turned away.
Time Machine refused the destination.. no backup could even start.

Luckily nothing was lost because nothing was backed up yet and it was a brand-new service with no data. Also, the server doesn’t have a WAN interface, so the ungated shares were only accessible from inside the house and on the tailnet. No alert caught any of these, and I had to find them all myself by checking the running container or on my Mac (not the deploy status).

Minutes, a day and a loud error

These three surfaced within minutes on the evening of June 16 because I went straight to the running container and didn’t trust the status. The refused login exposed the missing account, and testparm -s in the container shows the config Samba is actually loaded, which didn’t include hosts allow. The nagging question was: if login is impossible, how could the healthcheck have been passing all along?

The VPN lockout sat until the next day, only discovered by asking if home VPN clients were allowed (they weren’t). The fix then hit a deploy that reported no success and never recreated the container (a Forgejo push webhook raced my forced deploy). Caught it by reading hosts allow back from the running container, then forcing deploy again. The last one, Time Machine error -1013, happened at Mac setup, but unlike the others it failed loudly.

Why they broke

Two of the five were from the image’s config format. It only takes env vars SAMBA_GLOBAL_CONFIG_* (global) and SAMBA_VOLUME_CONFIG_* (shares), which the entrypoint translates to smb.conf. Pre-creating a group with GROUP_<user> made busybox adduser fail “uid ‘1000’ in use”, so no account was ever created and valid users denied everyone. The second trap was global keys: the entrypoint only translates _SPACE_ to space and _COLON_ to colon. hosts_allow ended up in smb.conf as an unrecognized parameter, which Samba silently ignored along with the rest of the hardening I’d written the same way.

Tip · Check what Samba loaded

testparm -s in the container shows the config Samba actually loaded.

The healthcheck was wrong: smbclient -L lists shares and passes even without an account (guest fallback). The VPN lockout was a routing issue: my router was routing the AWG and OpenVPN subnets without masquerade, so Samba saw the real client source addresses and hosts deny = ALL blocked them (only LAN + tailnet allowed). The Time Machine error was just tmutil syntax mistakes: I’d forgotten the user name in the URL and put -p after the URL instead of before, so tmutil setdestination kept returning -1013 “No password/user”.

Two stack choices look like causes but aren’t. I publish the port on all interfaces.. binding to the specific Tailscale IP crash-loops on reboot if tailscaled hasn’t assigned the address before Docker starts the container. The gate is Samba’s hosts allow (no WAN side anyway). Bonjour discovery is off because matter-server already owns mDNS on the box, so I set the destination manually, which is more reliable than Bonjour even if it worked.

    ports:
      # NOT bound to specific IPs on purpose. A specific Tailscale-IP
      # bind crash-loops on reboot if tailscaled hasn't assigned
      # the IP before Docker starts this restart:unless-stopped container. Exposure is
      # gated at the SMB layer instead: `hosts allow` permits only LAN + tailnet +
      # loopback. The server has no WAN interface (behind the router's NAT).
    environment:
      # discovery off (mount by IP; mDNS port <port> owned by matter-server)
      - AVAHI_DISABLE=1
      - WSDD2_DISABLE=1

Fixing all five

I stopped pre-creating the group to fix the account issue. It turns out busybox adduser creates a primary group with the same name itself, and the pre-existing group caused the “uid in use” failure. I kept force user and force group on the shares so files still end up owned by uid/gid 1000.

# first deploy: pre-creating the group made busybox adduser fail ("uid in use"), so no account existed
      - ACCOUNT_<user>=${SMB_PASSWORD}
      - UID_<user>=1000
      - GROUP_<user>=1000          # group <user> gid 1000 (shares also `force group`)

# fix: let adduser make the same-name primary group itself
      # NB: do NOT pre-create GROUP_<user> - busybox adduser makes the user's
      # same-name primary group itself; a pre-existing one fails as "uid in use".
      - ACCOUNT_<user>=${SMB_PASSWORD}
      - UID_<user>=1000

For hardening, I rewrote all six global keys in entrypoint’s spelling, _SPACE_ where smb.conf has a space. Aside from the host gate, I enabled mandatory signing and set SMB2_10 as the minimum protocol. After redeploy, testparm -s showed all in the effective smb.conf, including hosts deny = ALL behind the allowed ranges.

# first deploy: a bare "_" stays literal, so Samba never saw "hosts allow"
      - "SAMBA_GLOBAL_CONFIG_hosts_allow=<lan-subnet> <tailnet-range> 127.0.0.1"
      - "SAMBA_GLOBAL_CONFIG_server_signing=mandatory"

# fix: the entrypoint only converts _SPACE_ -> space and _COLON_ -> colon
      - "SAMBA_GLOBAL_CONFIG_hosts_SPACE_allow=<lan-subnet> <tailnet-range> 127.0.0.1"
      - "SAMBA_GLOBAL_CONFIG_hosts_SPACE_deny=ALL"
      - "SAMBA_GLOBAL_CONFIG_server_SPACE_min_SPACE_protocol=SMB2_10"
      - "SAMBA_GLOBAL_CONFIG_server_SPACE_signing=mandatory"
      - "SAMBA_GLOBAL_CONFIG_load_SPACE_printers=no"
      - "SAMBA_GLOBAL_CONFIG_disable_SPACE_spoolss=yes"

The new healthcheck does a tree-connect to a real share and lists it, so it needs a real account. I smoke-tested that first: exit 0 for valid credentials, exit 1 for invalid. Only then did I add the autoheal=true label, so there was no restart from an unproven check.

# first deploy: passes via guest listing even when no account exists
      test: ["CMD-SHELL", "smbclient -L 127.0.0.1 -U <user>%$$ACCOUNT_<user> -m SMB3 >/dev/null 2>&1 || exit 1"]

# fix: connect to a real share and list it; autoheal turned on only after this was proven
      test: ["CMD-SHELL", "smbclient //127.0.0.1/<share> -U <user>%$$ACCOUNT_<user> -m SMB3 -c ls >/dev/null 2>&1 || exit 1"]
    labels:
      - "autoheal=true"

I added both subnets to hosts allow (IPv4 only, my choice) and forced a deploy again. I read the running config to confirm the new line was there. I found the working Mac command: sudo tmutil setdestination with -ap before the URL and the user name baked into the smb:// URL.

Tip · Time Machine destination syntax

sudo tmutil setdestination takes -ap before the URL. The user name goes in the smb:// URL. Otherwise tmutil setdestination returns -1013 “No password/user”.

      # LAN + home VPNs (AWG, OpenVPN - routed, NOT masqueraded by the router,
      # so the server sees the original VPN-subnet source) +
      # Tailnet (CGNAT, gated by Tailscale ACLs) + loopback.
      - "SAMBA_GLOBAL_CONFIG_hosts_SPACE_allow=<lan-subnet> <vpn-subnet-1> <vpn-subnet-2> <tailnet-range> 127.0.0.1"

The Time Machine share is just one environment variable full of raw smb.conf, with the entrypoint script turning each “; “ into a newline. That’s why the vfs_fruit settings are on the share instead of the global section. I pinned the image by digest and set up the stack to deploy on push to my own git server, as described in Self-hosted git with Forgejo and why some repos never touch GitHub.

; one env var, raw smb.conf; the entrypoint turns "; " into newlines
[<share>]
path = /shares/<share>
valid users = <user>
force user = <user>
force group = <user>
guest ok = no
read only = no
browseable = yes
vfs objects = catia fruit streams_xattr
fruit:metadata = stream
fruit:posix_rename = yes
fruit:nfs_aces = no
fruit:time machine = yes
fruit:time machine max size = 1T

First full backup

The Mac authenticated over SMB 3.1.1 with AES encryption, all three shares mounted. I wrote a .DS_Store file on each to confirm read-write. The first full backup started at 4 GB, I watched it grow, and it finished at 655 GB on the morning of June 19. Felt good after that rough first evening. The next incremental added 4 GB, and it’s been doing hourly backups since. To force one, I just run tmutil startbackup.. the --auto lets the system decide when.

What stuck

I kept the habit of verifying the effective config of the thing I’m running (testparm -s in the container, plus a tree-connect write test to prove the files actually land as 1000:1000) instead of just assuming the deploy worked. Same instinct as with my cloud architect habits at home.. the second rule was to prove the healthcheck before letting anything act on it (autoheal only got flipped on after the smoke test passed).

It’s now monitored by a Uptime Kuma TCP monitor (SMB doesn’t do HTTP).. if the container fails, autoheal restarts it and alerts me via apprise → Pushover. The offload and archive shares are in my nightly Backrest plan, and the Time Machine share is intentionally excluded (you don’t back up a backup). FileBrowser is still the web view for the same storage, but Samba gives me the native mount and Time Machine path.

One thing I still need to figure out is.. I set a 1 TB quota on the Time Machine share, and the Mac has ~829 GiB on it. For now, only a shallow history of older backups will fit until I bump that quota.

The Pragmatic Programmer and an agent that outruns its headlights

Mon, 22 Jun 2026 00:00:00 GMT

An agent can build an entire feature before anything is even run. If typing speed is no longer the limiting factor, what dictates the pace? This brings us to the old book The Pragmatic Programmer by David Thomas and Andrew Hunt, which keeps coming up in agent discussions.. specifically the 20th Anniversary Edition from 2019.

In the AI Engineer talk in April, Matt Pocock described how the agent can sometimes overreach, dumping lots of code and only then doing type checks or tests after. He called this “outrunning your headlights” (a phrase from the book) and put it as “the rate of feedback is your speed limit.” There’s a thread on r/ExperiencedDevs from spring asking if it’s still worth reading in 2026. Most of the replies were along the lines of yes, or at least learn its ideas.

The headlights thing is from the chapter on pragmatic paranoia. It says if you’re going faster than your headlights let you see the road, you’re in trouble. So take small deliberate steps and check the feedback before going further. The companion tip to that is “Only look ahead as far as you can see.” The small steps tip is new to the 2019 edition (about three quarters of the original text was changed). So the most-quoted rule in the agent talk is one of the youngest in a book first published in 1999.

Later chapters talk about tests in a similar way, describing them as a form of design feedback and the first user of your code, rather than a bug hunt. Pocock’s remedy for agents is test-driven development, so the agent gets a signal after every small step before moving on to the next one.

Not everyone found the book profound. Hacker News readers have long complained that the advice is obvious or Goldilocks advice (do enough, but not too much). Some say it only confirms lessons they learned the hard way. And perhaps that criticism is true for people. But Pocock counters that software agents by default are bad at taking small, deliberate steps, so even obvious advice must be built into them.

The agent discussion moved this idea from “personal habit” to “the tooling around the agent”. Ben Borgers writes without feedback loops, you’re the feedback loop. If the agent can’t test its own work, you end up being its QA. So give it the means to check itself (curl calls, browser automation), write a little, test a little.

My own setup gives me another perspective: I run several Claude Code sessions in parallel, and sometimes the relevant feedback comes from a different session. For example, one peer session raced two commands and lost the update in 60/60 runs. The very next Memento release included a lock and a race test that would have failed on the old code. I talk about this more in Running six to nine Claude Code sessions at once.

If feedback is the speed limit of your agent, who provides the headlights? Does the agent write its own tests? Do you run a second checking agent? Or is it a human approving the change before it ships?

The Hard Parts rule for data: whoever writes it owns it

Sun, 21 Jun 2026 00:00:00 GMT

One question that keeps coming up for me is.. if you split up a shared database, how do you decide who owns which tables? Software Architecture: The Hard Parts by Neal Ford, Mark Richards, Pramod Sadalage, and Zhamak Dehghani (O’Reilly, 2021) spends a big part of the first half of the book talking about how to separate out data, and they have a pretty short answer to this question: Dan Lebrero’s notes on the book summarize it as “the service that performs writes owns the data.”

The book classifies tables by the number of services that write to them. If there’s only one service that writes to the table, that’s single ownership, and that’s the easy case.. the table should just go with that service. If there are lots of services that write to the table, that’s common ownership, and you should fix it by making a dedicated service own the table and have all the other services access it through that. If there are a few services that write to the table, that’s joint ownership, and you should either split up the table so that each service writes to its own part, share a data domain, delegate writes to one service, or merge the services.

This rule is important because of the architecture quantum.. the independently deployable unit. If you share a database, you have static coupling. All the services that write to the same tables are part of the same quantum, no matter what your deployment diagram says. I think it’s ok to share a database instance to save costs, but sharing write access to the same tables is where you get into trouble. Milan Milanović’s review makes a similar point about the shared data domain pattern.. it can have good performance and fault tolerance, but ownership and security are difficult.

I think merging, the last option for joint ownership, doesn’t get enough credit. It might seem like a regression in your decomposition plan, but if you have two services that need to change the same rows, merging them is often more honest than building a protocol between them (the book’s definition already says they’re one quantum). One last question: If consolidation is one of the fixes in the book, why don’t we see it in decomposition plans more often?

Fundamentals of Software Architecture: is it all still trade-offs?

Fri, 19 Jun 2026 00:00:00 GMT

A book whose first law is that every architecture choice has a cost got a new edition, about 5 years after the first. The authors describe it as close to a rewrite. Fundamentals of Software Architecture 2nd edition by Mark Richards & Neal Ford (O’Reilly, March 2025) grows from ~450 pages to almost 600. If the core law hasn’t changed, what has changed in the last 5 years to warrant five new chapters?

The book itself is recurring at the beginning of the common path. When people ask questions about the journey from developer to architect on r/softwarearchitecture, replies often mention Fundamentals first, followed by its sequel, Software Architecture: The Hard Parts. In a thread for beginners earlier this year, commenters emphasized the order, noting that The Hard Parts is the sequel, so Fundamentals should come first.

The central idea is the First Law: “Everything in software architecture is a trade-off.” A corollary is that if something doesn’t seem to be a trade-off, it’s because you haven’t identified the trade-off yet. The goal is then to find the least worst architecture for the few characteristics the system must excel at, and the early chapters on architecture characteristics spend a lot of time defining those characteristics. The 2nd edition adds a Third Law: most architecture decisions aren’t binary choices but lie on a spectrum between two extremes.

The new chapters are about how architecture rarely stands on its own. In a Thoughtworks podcast about the new material, Richards talks about how the real goal is to align architecture with what’s around it. Chapter 26, Architectural Intersections, talks about data, infrastructure, team topologies, and generative AI. The modular monolith gets its own chapter (11), and Ford talks about how this fills a gap in the original that was opened up by a changed ecosystem. In the same podcast, Ford describes a measure of friction between teams: count the merge conflicts an enabling team pushes into a stream-aligned team (more conflicts = more imposition on the other).

Discussion pushes back on the over-neat generalizing. Each style chapter gives 1-5 stars for each characteristic.. a review of the second edition published this month points out that microkernel can have poor or excellent maintainability, so a single rating doesn’t say much. Critiques of the 1st edition made a similar point years ago: a book that preaches context shouldn’t make sweeping claims. The sharpest disagreement is on AI. The book says LLMs lack the context and wisdom to weigh architecture trade-offs. The reviewer says that view will date fast.. AI tools are already useful for researching trade-offs, and the book gives no credit for that.

That’s where the disagreement comes in, and where I find it thought-provoking. In Always be curious I wrote that curiosity is why and building is how, and the book’s Second Law says why is more important than how. My curiosity led me to a career in cloud architecture. So if agents start doing more of the how, does the why (the trade-off written down and owned by someone) become most of the job? And if so, can a printed list of intersections keep up in a field moving this fast?

chezmoi: one set of dotfiles across a Linux server and two Macs

Tue, 16 Jun 2026 00:00:00 GMT

Chezmoi let me simplify most of the per-machine special scripts and Makefiles I needed to build years ago when I was using stow.

For those who haven’t run into either one, GNU Stow is a great little tool which symlinks a folder of config files into your home directory, and chezmoi keeps a source copy of your dotfiles in a git repo and renders them onto each machine (I have a Linux server and two Macs that it now covers).

I had been a user of stow for years and combined with Makefiles, leaned on it to manage my dev environment and dotfiles across multiple machines.. using git versioned repos, and helping me keep consistency across environments. Recently, one of my colleagues mentioned they used chezmoi and gave me a peek at their configuration files, templates, and some things they were doing and I was intrigued. Since installing it, I have simplified my dotfiles using templates and can easily onboard and offboard configurations and manage everything with Claude Code.

The Linux machine was the biggest mess. In March, the dotfiles were 5 files pushed out by Ansible copy tasks, no templating, no secret injection at all. Starting March 19, chezmoi became the only thing writing dotfiles there (after I fixed a conflict with Ansible).

One template per config file

I love the logic that can be used (like conditionals) and the templates I can use to manage configuration files across environments (like personal, development, and work). The templates have been the biggest impact for my needs. One .gitconfig or one .zshrc and the logic is where it belongs in these file templates. Logic meaning the if/else blocks I use for gating what makes it into each rendered template on the target machine or environment.

A trimmed example with placeholder values, where the source file is dot_gitconfig.tmpl and each machine sets its own profile in its local chezmoi config. The templating docs cover the rest of the syntax.

# ~/.config/chezmoi/chezmoi.toml (one per machine, not in the repo)
[data]
    profile = "personal"

# ~/.local/share/chezmoi/dot_gitconfig.tmpl
[user]
    name = Your Name
{{- if eq .profile "work" }}
    email = you@work.example
{{- else }}
    email = you@example.com
{{- end }}

{{- if eq .chezmoi.os "darwin" }}
[credential]
    helper = osxkeychain
{{- end }}

Running chezmoi apply on a Mac with the personal profile gets the personal email and keychain helper. The server gets the same file without the macOS block.

Diff before apply!

Editing files managed (and not managed yet) by chezmoi and checking the diff (chezmoi diff) has been so useful, especially if handing modifications and tweaks off to an AI agent. Bringing a new environment or machine online is almost as simple as a git clone and chezmoi init + apply.

chezmoi add ~/.tmux.conf    # start managing a file
chezmoi edit ~/.zshrc       # edit the template (not the rendered file)
chezmoi diff                # see what would change
chezmoi apply               # write it out

The second laptop came online in a similar fashion on May 8. A couple weeks after that I moved the dotfiles repo to my self-hosted Forgejo and started templating my Claude Code settings per machine as well.

Another big win has been integrating with 1Password CLI, being able to render secrets files on demand when I run chezmoi apply (more on the 1Password op CLI in a future post). chezmoi has 1Password template functions that call the 1Password CLI, so the secret never sits in the repo.

Layers and separation of duties

On the server, chezmoi isn’t the only thing installing stuff. For now, ansible handles system packages through apt, and Homebrew on Linux covers my command line tools. I’m currently on a setup where each layer gets exactly one owner and this seems to be working well:

Owner	Owns	Examples	Why
apt (via Ansible)	System packages, daemons, kernel modules	docker, zfs, samba, tailscale, nut, smartmontools	Needs root, systemd integration, kernel coupling like ZFS DKMS
Linuxbrew	User-level CLI tools where the latest version matters	ripgrep, fd, jq, gh, fzf, bat, eza	Tracks upstream faster, user-scoped, no sudo
chezmoi	Dotfiles and shell config	.zshrc, .gitconfig, starship.toml, tmux.conf	One source, rendered per machine

~/.linuxbrew/bin comes before /usr/bin in my PATH, so brew “wins” if the tool exists in both places. I don’t really like to rely on that though, so if I notice a tool showing up in both places, I’ll pick one or the other and remove the other source.

Finally, Ansible is not allowed to deploy dotfiles directly. I tried that before and it broke a few things like credentials, so now Ansible is only allowed to manage system packages and daemons, and chezmoi is only allowed to write a dotfile.

I turned off band steering and the Wi-Fi got better

Mon, 15 Jun 2026 00:00:00 GMT

With 3 APs and a possible 4th in the short term, I wanted to take advantage of the hardware as best as I could and get the best bands and APs for my devices, especially the ones that move around all day.

As we covered in OpenWrt to the rescue, this network has been running band steering and AP steering since I came back to OpenWrt. The daemon, DAWN, would monitor each client and nudge them to the band and AP it deemed optimal. On June 11 I disabled DAWN on all three APs, meaning clients would now be solely responsible for making those decisions themselves, using 802.11r/k/v roaming, which was already supported by the APs.

DAWN wasn’t my first attempt. Initially I enabled usteer on all three APs with roam_trigger_snr=-65 and min_connect_snr=-70, but those thresholds were too aggressive and my IoT devices started churning on and off the network. The next day I backed usteer off to passive mode, set the SNR thresholds to zero, and only enabled band steering. That day’s research also revealed that transmit power is the primary lever for steering, and that DAWN conflicts with 802.11r (also described in this OpenWrt forum thread about aggressive DAWN kicking with 802.11r/k/v). When I moved to OpenWrt 25.12.2 in early April, I opted for DAWN over usteer anyway, as I prefer it. It’s in the 25.12.2 package repos and can discover other APs via umdns.

I tuned DAWN in rounds. In the first round, within days, I noticed that DAWN was kicking an idle smart sound machine every 10 seconds (≈ 8,640 kicks/day) because its receive rate matched bandwidth_threshold=6, so it was considered a client worth moving. I lowered that to 2 and the kicks stopped immediately. In the second round, a couple weeks later, I targeted 2.4 GHz because the kitchen AP’s 2.4 GHz channel peaked at 76.8% utilization. I lowered 2.4 GHz’s initial_score from 80 to 60 and increased 5 GHz’s rssi_center from -70 to -65 to push dual-band clients to 5 GHz. (The logs later showed that the “Apple TVs” on 2.4 GHz were actually other devices, so round 2 targeted the wrong clients.)

In round 3, a few days later, I drew from the research report (which references 22 sources) and adjusted kicking_threshold from 20 to 15 and min_number_to_kick from 3 to 2. For 2.4 GHz, I set low_rssi_val to -73 (from -80) and rssi_weight to 2 (from 0). During the rollout, I discovered that DAWN had been down on all three APs (for how long?) because an Ansible task with state: started only starts a service that was down, but doesn’t restart it after a config change. I added a restart handler to the role.

Since mid-May, my nightly network health check started flagging my iPhone: it was flapping between the router and the kitchen AP, registering up to 10 connect/disconnect events within 1.5 hours, all fast-transition roams. The same report also listed nightly DAWN beacon-request failures (52 the first night it was flagged, 38 a few weeks later), so I suspected DAWN was behind the roaming trouble.

The router and kitchen AP logs confirmed my suspicion: DAWN was sending 802.11k beacon requests to my iPhones (asking them to measure audible APs and report back), and they kept failing. My iPhones were using private Wi-Fi addresses, and the failed requests were followed by inactivity deauths that kicked the phone off the AP. Switching DAWN to passive mode didn’t help: kicking=0 prevents kicks, but not measurement requests.

A couple of months and multiple troubleshooting rounds later, after weird network behavior, DAWN ended up causing more headaches than it was solving. Even passive mode for DAWN didn’t stop it. I ended up simplifying and going back to 802.11r/k/v, and lo and behold: seamless roaming for the iPhones and Macs.

Disabling it was just two Ansible changes: setting openwrt_dawn: false for all three APs, and adding a teardown task (omitting the DAWN config alone doesn’t stop the daemon where it’s already installed). A quick post-run check confirmed DAWN was stopped and disabled at boot on all three. Roaming is now fully client-driven with 802.11r fast transition, hostapd’s 802.11k neighbor reports, and 802.11v BSS transition, all verified identical across the three APs. The options are documented in OpenWrt’s Wi-Fi configuration docs. Fast roaming between APs was already working fine, as evidenced by the fast-transition roams in the health check.

The next night’s check showed zero DAWN beacon-request failures. My iPhone still flapped at the same spot (SNR 14 / -78 dBm), but when I investigated that weak spot, it was just my iPhone in the bedroom connected to the kitchen AP on 5 GHz at -82 dBm (SNR 10).. the edge of the kitchen AP’s coverage, not a client stubbornly clinging to the wrong AP. So I closed the roaming check and left DAWN off. (This isn’t a clean before/after comparison: the same day I also put the new third-floor AP into service and turned off the router’s own radios, so multiple things changed at once.)

Without a steering daemon, clients that don’t support 802.11r might stay connected to an AP for too long. A day later, I noticed my Sonos Roam was stuck to the far living-room AP at -93 dBm, even though my iPhone was right next to it connected to the near AP. The Roam doesn’t support 802.11r, so my only options were to power-cycle it or deauth it to nudge it onto a better AP.

With a little more tweaking of power levels to let the client devices pick what’s best for them, I have found really good stability.

Next up.. getting metrics and logs off all the network gear.

My AI sysadmin and security agents assessed my homelab

Sat, 13 Jun 2026 00:00:00 GMT

An audit of my Ansible code vs the live hosts took most of a day. The end state: every playbook made zero changes, both on check run and real run. Along the way I defused a bunch of live landmines, like a WiFi package that kept reinstalling itself on every real run, bouncing WiFi every time. Once clean, I asked the AI agent (which had just touched every host) to grade the entire homelab against best practices. See also: Ansible manages every config on my router and access points

The result was a written review that started with a scorecard, then listed problems from urgent red flags through to tech debt, and closed with some lessons learned about how the work went. I doubted a review by the same agent who did the work would reveal anything new.

It gave me strong marks for deployment, backups, and docs, but weaker marks for security and drift detection. The summary described it as a “top-decile homelab” with weakness “concentrated in continuous assurance.” In other words, it’s well built, but nothing is actively ensuring it continues to be well built. The lowest mark was for security posture: a B-.

Note · The grader did the work

The grader had just done the audit itself, and the review says so in its own text. I read the grades with that in mind; the list of problems was the part I could check.

More important than the grades are the red flags, each of which is a concrete thing I can check, fix, or rule out. There were five of them:

No offsite backups.
Autoheal restarting every unhealthy container, not just the ones I picked.
Password SSH still on for the router.
The router as a single point of failure, with a recovery runbook I’d never actually tested.
Nothing alerting on host health.

The audit takeaway from that review that stuck with me more than any grade was: “unexercised Ansible rots invisibly.” Code that I ran regularly was fine, but the landmines were all in playbooks that hadn’t been run for a while. That made me wonder: what else in the lab is working only because nobody’s touched it lately?

The next morning I picked a few cheap things and started with the ones that had been hanging around the longest for no good reason. Removing the old server from the tailnet had been on the list for five work sessions, and it turned out to be a single call to the Tailscale API to delete the device. “Close the 30-second tasks” was one of the lessons from that review.. and I had to admit it was right. (see also: Audit the box before you wipe it)

Next up was autoheal. docker-autoheal had been watching every container, so I switched it to opt-in mode so it only restarts containers with a label. I left the auth proxy unlabeled on purpose, because a blind restart could mess up the login state it holds. (see also: Autoheal opt-in and healthchecks that actually test something for the longer story, and How a deploy works with Dockhand for the deploy side)

The “no host alerting” flag was.. wrong. I’d set up host alert rules (disk filling up, host going quiet) that went to my phone five days before this review, but the claim had outlived the fix by five days in my notes and the review. I proved that end to end by making a throwaway always-firing rule and putting it through a real Grafana notification policy. Both pings made it to my phone, so I deleted the test rule and marked that flag stale.

Tip · Check it’s still a gap

Before scheduling work on a gap from a review or from your own notes, check it against the live system. A note can say something is missing long after it was fixed.

Disabling the router’s password SSH took longer than you might think for something that size. Before making the change, I checked that I wouldn’t get locked out and discovered I hadn’t backed up any SSH keys in 1Password. My first attempt at backing them up saved the key in an unreadable form, which I only caught by testing the restore. Storing it as a document fixed that and let me restore it byte for byte. The router’s logs showed that my phone had been logging in with the main server’s key, not the one I thought it was using. So I based the change on the keys I knew were in use from the logs, not just the ones I had installed. That let me turn off password SSH in dropbear settings while keeping the web interface’s password login as a way back in. (see also: OpenWrt to the rescue for this router)

At the end of day two, I’d fixed two red flags, struck one as stale, and still had offsite backups and a router recovery drill to do. I’ll get to those soon. The weekly drift check, which is how I’ll try to avoid the kind of rot the audit uncovered, got parked on the backlog. The list is worth having, but every line needs a check against the real system before spending time on it. Next up to check and build: offsite backups.

If you keep one thing from The Staff Engineer's Path, keep the maps

Sun, 07 Jun 2026 00:00:00 GMT

The Staff Engineer’s Path by Tanya Reilly (O’Reilly, 2022) has three parts. The single strongest candidate for a takeaway is the set of three maps in part one (the big picture). “At the staff engineer level, everything you do has a high opportunity cost, so your work needs to be important.” Senior individual contributors and architects in large enterprises are judged on this question, and the maps are the tool this book gives you to answer it.

Map 1 (the locator map, or “you are here”): Understand your team’s purpose and how your team’s output affects the teams that depend on it. Map 2 (the topographic map): Understand the terrain between teams.. the friction points, where decisions are really made, the informal org chart that exists beyond the formal org chart. Reilly says you can learn this map by using open sources like shared calendars and chat channels. Map 3 (the treasure map, or “x marks the spot”): Understand where everyone wants to get to and why it’s important to them.

Part two, execution: This is where the picture has changed the most since 2022. Maxime Najim in LeadDev talks about how for staff-plus engineers, verification has replaced execution as the bottleneck. A staff engineer can bring a working prototype or a ready-to-go pull request instead of a ticket in the backlog. The treasure map is much easier: an agent can build an “x marks the spot” demo in an afternoon, and people can react to something real.

But the locator and topographic maps aren’t easier. The 2025 DORA report talks about how AI amplifies whatever system it enters, and the payoff from platform quality and team alignment is the terrain of those two maps. An agent can read your repository, but it can’t see the informal org chart, the risk committee that owns a decision, or which shared platform several business units depend on. For large regulated organizations like banks, that terrain is most of the work.

The trust numbers tell the same story: DORA found that about 30 percent of respondents had little or no trust in AI-generated code. So somebody needs to know where that output lands and who it affects. I’ve spent the last ten years working with large financial institutions and Fortune 500 companies, now as a principal cloud architect, and before that I was a developer and a sysadmin. Every role showed me a different slice of the same map: how code connects to the people and systems around it.

The idea strains in places. The book has been argued about for a long time, and some of that’s about how staff titles have been diluted, how it reads like a promotion guide for big tech, how somebody still has to fix the bugs. Part three, on leveling up, might be the hardest to apply right now. Stanford’s Digital Economy Lab has documented declines in early-career employment in AI-exposed jobs like software development, while experienced workers have stayed steady. That means fewer people are learning the terrain slowly. If maps are a scarce senior skill, who teaches the next generation how to draw them when the junior work that taught them goes to agents?

OpenArchiver: a self-hosted Gmail archive

Sat, 06 Jun 2026 00:00:00 GMT

Halfway through brainstorming where to put the Gmail archive, it occurred to me that I already have a ZFS pool in my home server, which is snapshotted and backed up nightly.

Once I had the storage sorted, I decided not to bother writing a spec. Instead, I had Claude build the stack on the storage I’d picked, while I made the connector call and ran the setup wizard myself, and it went from idea to running in the same session.

The app is OpenArchiver, an open-source self-hosted email archiving solution from LogicLabs. It pulls mail from your provider and stores each message as a standard .eml file, with full-text search on top. It needs a few helper containers alongside the main app, but I didn’t fork it.. I just adapted the compose file from their docker-compose.yml. I deployed it as a git-backed stack, just like everything else on the server (How a deploy works with Dockhand).

Getting Gmail into the system was the one real decision I had to make. I assumed the Google Workspace connector would be the natural choice (it’s built for Google, after all), but it turns out that relies on OAuth and requires domain-wide delegation.. a Google Workspace admin feature that isn’t available on personal Gmail accounts. So I went with the generic IMAP connector instead, using SSL and a Google app password. No admin console needed.

Tip · Personal Gmail? Use IMAP

The Workspace connector needs domain-wide delegation.. not available on a personal Gmail account. IMAP with a Google app password works fine.

That app password is the only secret that unlocks my mailbox, and it’s stored in one place and one place only: I type it into the OpenArchiver web UI, where it’s AES-encrypted in Postgres. It’s not in my git repo, not in my password manager, nowhere. Every other secret in this stack is injected at deploy time and kept well out of git, but this Gmail credential (being “per user”) belongs to the app.

I spent most of my time figuring out which parts of the stack are unrecoverable. The .eml files live on a ZFS dataset on the archive pool, backed by Sanoid snapshots and a nightly Backrest job. Postgres stores the metadata and encrypted credential in a named Docker volume, with a logical dump before each backup. The Meilisearch index can be rebuilt from the stored mail, so it’s not backed up. Valkey only holds the job queue, so it’s not backed up either. Apache Tika extracts text from attachments but has no state to back up. The result: I can focus my full backup effort on the mail itself.

Note · Mail is encrypted on disk

OpenArchiver encrypts the .eml files with its own storage key.. so whatever lands in snapshots and backups is encrypted too.

Backing up Postgres to a nightly dump is just one line in a registry file read by the pre-backup job. Using named volumes (instead of bind mounts) for all the stateful helper containers keeps me from having to chown host directories to match container user IDs.

openarchiver-postgres:open_archive:<role>

I needed to make it fit on a server where I already run many stacks. I kept the bridge network from upstream so services can discover each other by name (host networking would break that DNS). The upstream DB container is called plain postgres, which would clash on a shared host, so I renamed containers to e.g. openarchiver-postgres while keeping the service names.

Warning · Upstream container name clashes

The upstream DB container is called postgres, which will clash on a shared host. Rename containers (e.g. openarchiver-postgres), but keep the service names. Keep the upstream bridge network (host networking breaks service-name DNS).

The best moment was right after setup. I completed the setup wizard, then added a Gmail source myself, and it immediately started populating. The first-run check showed more than 580 .eml files, over 536 MB, and still growing. It was fun to watch because the stack was just an idea earlier in the same session. After the first pass, it syncs every 15 minutes. I ran it with the all-inclusive setting, so it’s even pulling in Junk and Trash. The next day’s audit showed it fully configured and ingesting (not pending).

    environment:
      SYNC_FREQUENCY: "*/15 * * * *"
      ALL_INCLUSIVE_ARCHIVE: "true"

That same evening, I added a tile to my Glance dashboard and set up Uptime Kuma monitors to alert me via Pushover, just like I did for the other stacks I set up that day, including VictoriaMetrics. A monitor is a good way to know if the archiver ever quietly stops.

This isn’t a light stack. Postgres, Meilisearch, and Tika (JVM) together use about 4 GB RAM on a 32 GB box, so it’s definitely a number to watch if the server comes under memory pressure.

VictoriaMetrics as the first metrics database on my home server

Fri, 05 Jun 2026 00:00:00 GMT

At home, I already had logs and traces flowing through Grafana Alloy to Loki and Tempo, but I had nowhere to put numbers. I couldn’t see when a CPU was getting hot or a container was stuck in a restart loop. I had no metrics being stored, so I couldn’t set up alerts. So I added VictoriaMetrics as the first metrics database on my server, and within a day it was feeding my dashboards and paging my phone. Beszel was left with nothing to do.

The gap first surfaced in my own app when wiring up observability for the vault search service: I was using Alloy to send logs to Loki and traces to Tempo, but dropping OTLP metrics because I had nowhere to put them. So the latency and lag numbers were stuck in log lines as fields. Grafana alerts on lag or request rate were out of reach until I had a metrics database. The Loki+Grafana side had been running for a while, see e.g. AdGuard Home and the query-log shipper, but Beszel was an island: a hub with zero agents, monitoring nothing.

The main thing I wanted from metrics was alerting. This is a small form factor box with limited cooling, so CPU temperature was the first thing I wanted on my phone. But then there’s memory, disk space, services that quietly stopped working.

I chose VictoriaMetrics over a full Prometheus server after comparing the two with Claude Code. VictoriaMetrics comes as a single small binary, uses little RAM, is Prometheus-compatible (so Grafana can query it with PromQL, and all exporters and community dashboards work without changes), and does a good job at compressing the data. I never considered Mimir because I’m running on a single machine at this scale.

I scoped this as a full metrics pillar, but didn’t write a spec. One of the key design choices was: who collects the data? Alloy is carrying all server logs, and I didn’t want to add any new mounts or config to that one container. So I decided to have VictoriaMetrics pull the data by scraping each exporter’s /metrics endpoint from its own scrape file. This keeps Alloy responsible for logs and traces only, so that a fault in the metrics system doesn’t bring down logs.

This pull model feels appropriate in a homelab context to me: almost all open-source services provide a Prometheus /metrics endpoint, but almost none support pushing OTLP data. This is a point the VictoriaMetrics blog makes when comparing OpenTelemetry and Prometheus. I’m still using Grafana Alloy for traces and logs, but metrics bypass it.

I added it as an extra container in my existing observability stack, rather than a new stack. It’s deployed from git using Dockhand, like everything else on the server (see Self-hosted git with Forgejo and why some repos never touch GitHub for the git side). I set the retention to 12 months with the single-node server’s -retentionPeriod flag, and configured it to read targets from a file using -promscrape.config. I put the data directory on a path that’s already backed up by my nightly Backrest job, so no new backup is needed.

  victoriametrics:
    image: victoriametrics/victoria-metrics:v1.144.0
    container_name: victoriametrics
    restart: unless-stopped
    network_mode: host
    volumes:
      - /storage/containers/victoriametrics:/victoria-metrics-data
      - ./victoriametrics/scrape.yml:/etc/vm/scrape.yml:ro
    command:
      - "-storageDataPath=/victoria-metrics-data"
      - "-retentionPeriod=12"
      - "-httpListenAddr=:<port>"
      - "-promscrape.config=/etc/vm/scrape.yml"

I set a 30-second scrape interval for everything in the scrape file, and added a header comment explaining why I chose this approach so anyone reading it next will know why Alloy isn’t involved. In Grafana, I set up a Prometheus-type datasource with a pinned id so the provisioned dashboards can refer to a stable name.

# VictoriaMetrics native scrape config (-promscrape.config). VM scrapes the
# exporters directly so Alloy (the log/trace pipeline) stays untouched.
global:
  scrape_interval: 30s
  external_labels:
    host: <host>

scrape_configs:
  - job_name: victoriametrics
    static_configs:
      - targets: ['localhost:<port>']
  - job_name: node
    static_configs:
      - targets: ['localhost:<port>']
  - job_name: cadvisor
    static_configs:
      - targets: ['localhost:<port>']
  # Observability-stack app metrics (self-monitoring the pillars). Only endpoints
  # verified live (HTTP 200 /metrics) are listed. Added 2026-06-05.
  - job_name: loki
  - job_name: tempo
  - job_name: grafana
  - job_name: alloy

My initial targets beyond the self-scrape were node-exporter for the host (CPU, memory, disk, network, temperature from /proc and /sys) and cAdvisor for per-container usage. I tuned cAdvisor down a bit since the box runs warm.. Docker containers only, every 30 seconds, skipping a bunch of metric groups. All targets came up the first night, and I added two community dashboards: Node Exporter Full and cAdvisor.

The next day I added the /metrics endpoints for Loki, Tempo, Grafana, and Alloy so the stack is monitoring itself. That brings us to seven total targets, all of which are up. I didn’t include Dockhand (the metrics path redirects) or Forgejo (metrics are off and 404) since they don’t provide a usable endpoint.

Alerting to my phone

This is the whole point, so I put these rules in right away. The temperature rule fires if the hottest core goes over 95 °C for five minutes. Memory is 92% for ten minutes. Disk is 90% for fifteen minutes. Then there are meta rules for a scrape target down for five minutes and container restarts over five in fifteen minutes.

But firing doesn’t mean paging! Initially, my rules hit the Grafana default contact point, so they fired but I didn’t get anything. I set up two Pushover contact points, one for critical alerts with high priority and a siren sound, and verified they were getting through.

The disk rule deliberately excludes RAM filesystems like tmpfs, because their usage doesn’t tell us anything about disks. But the same day /tmp (a 4 GB tmpfs) filled to 100% and broke some tooling with ENOSPC, and the disk rule didn’t catch it. So I added a sixth rule, just for /tmp, firing if it’s above 80% for ten minutes.

Warning · tmpfs escapes the disk rule

The disk rule specifically excludes RAM filesystems like tmpfs, but I had a case where /tmp (a 4 GB tmpfs) filled up to 100% and broke tooling with ENOSPC, so I have a sixth rule that fires if /tmp is above 80% for ten minutes.

A few gotchas

cAdvisor’s newest GitHub release (v0.57.0) doesn’t have a published container image.. the GitHub release tags and image tags are different. I pinned v0.55.1, the newest image that actually exists.
The VictoriaMetrics and node-exporter images don’t ship with curl, so a compose healthcheck won’t work and would leave them marked unhealthy for the autoheal sweep that watches every container. Uptime Kuma monitors them from outside instead, with alerts going to Pushover.
Community dashboards reference a ${DS_PROMETHEUS} input that file provisioning doesn’t fill in. I swapped it for the pinned datasource id and stripped the __inputs block.

node-exporter (for the host) and cAdvisor (for containers) made Beszel redundant, so I removed it from the server with the monitor and dashboard tiles. I kept the data directory in case I ever want to go back.

This means I have metrics alongside logs and traces in Grafana. A new watch target is just one scrape job in a file owned solely by VictoriaMetrics. Next up: Metrics and logs from the network gear where I’m scraping the network gear in the same way.

Self-hosted git with Forgejo (and why some repos never touch GitHub)

Thu, 04 Jun 2026 00:00:00 GMT

I set up my own Forgejo server for personal projects, mainly for the privacy aspect.. not to mention that I already ran an always on server. It made sense for my home lab to have that with excess compute and disk capacity instead of paying for a hosted solution through GitHub or GitLab. I had heard about the Forgejo project as it being a fork of Gitea, and I was really intrigued to be able to run my own actions with my own hosted runners on my own infra and experiment with hosting something like that, taking my own backups, and really having a private experimentation area.

One of my colleagues runs Forgejo and they spoke highly of it, and I actually met one of the maintainers at a conference in NYC. I was really impressed with their mission and their expertise, so I decided to go with it, and it has been great so far.

Forgejo forked from Gitea in 2022 after a for-profit company acquired it. Now it’s under the non-profit Codeberg e.V. and the comparison page tells the story. As a single user, the features are near-identical so it came down to governance for me.

I chose SQLite over PostgreSQL, and the database docs list all the options. Under 10 users, SQLite means one backup file and no maintenance. I can always migrate later.

I also chose the standard Docker image over the rootless one because it’s simpler for one user. The trade-off is every CLI command runs as the git user.

A one-commit deploy

The entire deploy is a single commit to my home infra repository: an Ansible playbook (189 lines) and a compose file (26 lines). The playbook handles everything from the ZFS dataset to the monitor, including an admin user with a generated password that I change on first login. All configuration happens through FORGEJO__section__KEY environment variables, so there’s no hand-edited app.ini file. Here’s the relevant snippet (with host/port/volume lines trimmed):

services:
  forgejo:
    image: codeberg.org/forgejo/forgejo:15
    container_name: forgejo
    network_mode: host
    restart: always
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - FORGEJO__server__START_SSH_SERVER=true
      - FORGEJO__server__LFS_START_SERVER=true
      - FORGEJO__database__DB_TYPE=sqlite3
      - FORGEJO__database__SQLITE_JOURNAL_MODE=WAL
      - FORGEJO__security__INSTALL_LOCK=true
      - FORGEJO__service__DISABLE_REGISTRATION=true
      - FORGEJO__cache__ADAPTER=twoqueue
      - 'FORGEJO__cache__HOST={"size":100, "recent_ratio":0.25, "ghost_ratio":0.5}'

I picked the web and SSH ports to avoid clashing with any existing services and the host’s sshd. The local DNS name is pointed at the server via an AdGuard Home rewrite, which I added manually in the UI. But when I first deployed, I got an error. Darn it.. Turns out Forgejo 15 doesn’t like running the admin CLI as root.

Forgejo is not supposed to be run as root.

The cost of the standard image: every docker exec forgejo ... command in the playbook runs as the git user, and the admin password is provided by Ansible, not me.

    - name: Create admin user
      shell: >
        docker exec --user git forgejo forgejo admin user create
        --admin
        --username <admin>
        --email <admin-email>
        --password '{{ forgejo_initial_password }}'
        --must-change-password

    - name: Generate initial admin password
      set_fact:
        forgejo_initial_password: "{{ lookup('password', '/dev/null chars=ascii_letters,digits length=24') }}"

Like every other service that runs on this box, Forgejo has its data on its own ZFS dataset rather than a Docker volume. Uptime Kuma checks it every 60 seconds and alerts me via Pushover if there are any issues. The Backrest pre-backup hook copies the database safely before restic runs.

When I looked at the monitor wiring, I got a surprise: the playbooks insert Uptime Kuma monitors via SQL, but the “default” notification flag doesn’t attach to those monitors. So the monitor_notification join table needs to be filled explicitly. Six monitors (Backrest, go2rtc, MagicMirror, Home Assistant, Mosquitto, Obsidian API) had never been bound to Pushover, so they were failing silently. Now the Forgejo playbook binds its own monitor and repairs any monitor missing Pushover.

Five days with a broken backup

The backup hook first flushes Forgejo’s queues, then uses sqlite3 .backup to make a copy that’s guaranteed consistent even while Forgejo is writing. I kept forgejo dump only as a secondary measure because of its known bugs.

# sqlite3 .backup produces a consistent snapshot even during writes.
# Output is self-contained (no WAL dependency).

# Flush Forgejo queues to ensure data is written to disk before safe copy
if command -v docker &>/dev/null; then
    if docker exec forgejo forgejo manager flush-queues 2>/dev/null; then
        echo "OK: Forgejo queues flushed"
    fi
fi

I came back to harden on May 11 and it was worth it: the first thing I noticed was that my clone URLs were showing localhost because I’d never set SSH_DOMAIN, which the config cheat sheet states is what sets the displayed clone URL.

       - FORGEJO__server__SSH_LISTEN_PORT=<ssh-port>
+      - FORGEJO__server__SSH_DOMAIN=forge.example
       - FORGEJO__server__SSH_PORT=<ssh-port>

The second issue is much worse. But.. ugh.. I had been pointing the hook at the wrong SQLite file since the day I deployed it. The database file for Forgejo is called gitea/gitea.db in the data directory. So the safe copy had been warning me every time it ran for five days that it couldn’t copy the file, and the dataset in question had zero ZFS snapshots as a fallback. Luckily, it was just a one-line fix in the hook and a sanoid snapshot policy for the dataset.

-        ["forgejo"]="/<pool-mount>/forgejo/forgejo.db"
+        ["forgejo"]="/<pool-mount>/forgejo/gitea/gitea.db"

[<pool>/forgejo]
  use_template = production

[template_production]
  hourly = 24
  daily = 7
  weekly = 4
  monthly = 0
  autosnap = yes
  autoprune = yes

Forgejo by default

From day one I knew the question of GitHub would be there. Whether I mirror or cut over completely would be a per-repo decision. Public repos might continue to live on GitHub for discoverability, for example. I figured I’d use Forgejo’s built-in GitHub migration tool and look at push mirrors for anything that stays. On May 14 I flipped the origin of my first existing repo to Forgejo. Three days later I did the opposite for my home infra repo: GitHub became the primary, with Forgejo mirroring it to guard against server loss. Two days after that I made my first repo Forgejo-only, with no GitHub copy at all.

At the end of May, when I moved to the new server, I decided enough was enough. There was no way I could clone private GitHub repos on the new box without any user intervention, so on June 1st I flipped the switch and made Forgejo-only the default, with the infra repo being the only exception that pushes to both. The very next day, I codified it into a rule: Forgejo is the default remote, and no GitHub push without checking first.

Currently, all of my repositories are privately hosted now on my own server. It makes me very comfortable that I can host and develop this way. There’s a few small exceptions, like this personal site is hosted on GitHub as it’s open-sourced. And some other community contributions that I make, I use GitHub because the open-source community is very important to me, and I like to contribute, give back on many projects. So GitHub ends up being somewhere where I do collaboration work, and anything that I feel needs to be released in the open-source community obviously gets put on GitHub.

Next up are runners. Forgejo Actions consumes GitHub Actions YAML, with some differences: you need a self-hosted runner, and the token is GITEA_TOKEN instead of GITHUB_TOKEN. I looked at deploying this in May, but the runner wasn’t production-ready yet so I held off. Running my own actions on my own runners was the original reason I got interested in Forgejo, so that’s what I’ll set up next.

Audit the box before you wipe it

Wed, 03 Jun 2026 00:00:00 GMT

Halfway through the migration (see The box and the move to the new server), I realized that I had never moved the message-archive pipeline. Its only copy (about 108 GB) was still on the old server, and the pipeline was dead. The service-by-service moves had told me what had moved, not what was left behind. I decided not to wipe anything until I had audited the entire box from top to bottom.

I labeled each artifact as either migrated, re-derivable, drop, or ACTION. The latter meant it existed nowhere else and I needed to make a call on it. I decided not to wipe anything until all ACTION items were cleared. Most of the results were boring: datasets that had already been copied, data that could be regenerated. Boring was what I wanted: it meant I could safely lose it.

The first ACTION item was a 106 GB dataset containing an old personal archive (documents and keys) that I didn’t have another copy of. It had the same name as a message-archive dataset I already had on the new server, so I’d have assumed it was safe based on the name, but the contents were different. I moved it into a child dataset under the existing archive parent on the new server, so it automatically inherits the recursive snapshots and nightly backup without any extra setup. I used a ZFS snapshot and zfs send into a receive, and verified that both sides had 903 files that were byte-identical.

The scan of my git repositories found the other three ACTIONs: one for a dashboard repo that had no remote (the old disk was the only copy), another for a repo with a single unpushed commit, and a third where I’d stashed some work in progress. All are now on my self-hosted git, and I used git stash branch to put the stash on its own branch (rather than merging it into the WIP commit I’d pushed, which might not match).

Not everything that was flagged got migrated. The old server had a copy of the bird-detection app from BirdNET listening in the backyard. I didn’t want to drop that unless I was sure the new server had everything the old one did.. had the old box picked up anything new that I’d missed? It turns out that both have the same earliest detection, but the new one has 22,304 detections compared to the old one’s 18,035. So the old copy went on the drop list.

Halfway through, the monitoring agent reported healthcheck zombies in 30 containers and suggested I set init: true on them all. It had never looked at the parent process of those zombies. Turns out all 16 recurring zombies were [git] <defunct> children of one container: forgejo-runner. The daemon is PID 1, so it has no init to reap the git commands it spawns for each CI job. Thirty containers sounded like a host-wide pattern, but it was really one container leaking in bursts.

I rejected the 30-container change and set init: true on the runner only, as Forgejo issue #7854 and Gitea issue #19077 recommend for runner-as-PID-1. Docker’s --init runs tini as PID 1 to reap. Zombies dropped from 16 to 0. The leak was slow, so this was hygiene, not rescue. I noted the global Docker daemon init setting as a next step if leaks ever show up across several containers. I also made the agent confirm script changes with me before applying them.

    container_name: forgejo-runner
    network_mode: host
    restart: always
    # PID 1 is `forgejo-runner daemon` (a Go process, no init). It shells out to
    # `git` per CI task and doesn't reap orphaned children → `[git] <defunct>`
    # zombies accumulate in bursts per job (slow PID-slot leak; cgroup pids.max).
    # tini as PID 1 reaps them. Exactly the fix Forgejo #7854 / Gitea #19077
    # recommend for the runner-as-PID-1 case. Single-process daemon → safe.
    init: true

With all ACTIONs cleared and nothing irreplaceable left, I exported the ZFS pool and powered the old server off. Then I discovered that the router’s watchdog from OpenWrt to the rescue was still watching it, pinging it every five minutes and trying to Wake-on-LAN it if it didn’t answer. I could tell because the “down” state file was stale. I removed it from the watch list so it stays down while the drives come out.

With the old server now dark and the pool exported, all that was left was for it to give up its drives. I’ll pull the two I want to keep and put them in the new server.

AdGuard Home and the query-log shipper

Tue, 02 Jun 2026 00:00:00 GMT

I was doing some troubleshooting (Apple TV and some apps not working correctly) and I realized the activity logs rotate out very quickly. Rightfully so, the Flint 2’s RAM is limited and we don’t want it hogging up the resources there. I needed more than a few hours of DNS resolution history to tune the allowlist, and what I needed wasn’t there.

After the router rebuild, I set up AdGuard Home as the first stop for all DNS queries in the house. It uses Cloudflare and Quad9 as upstreams via DNS over HTTPS, and filters with Hagezi’s Multi Pro and Threat Intelligence Feeds, plus the Hagezi referral allowlist. At the time, Multi Pro alone was about 226,000 rules. I used Ansible to template the router’s AdGuard config and pin its version. I also set up a replica AdGuard Home on my home server, using adguardhome-sync to copy the filters and client settings from the router every 15 minutes.

A few hours of memory

The short log history was a choice I made previously. The router has 1 GB of RAM, and the default file-based query log was using about 230 MB of tmpfs (which is RAM on OpenWrt). I solved that by disabling file logging and setting a 60,000-entry in-memory buffer, which used about 50 MB of RAM and at the time held about three to four hours’ worth of queries at the rate I was seeing.

That trade-off came back to bite me one night when a streaming app broke on the Apple TV. Disabling AdGuard protection fixed it, so the blocklist was to blame. But when I checked the query log for the app’s domains, the window during which it had failed was already out of the buffer. So I ended up creating a per-client filtering bypass for Apple TVs instead of per-domain allowlisting, because Multi Pro is aggressive and I couldn’t predict which app would trip it up next. All other devices are still filtered, and I still log Apple TV queries.

A few days before, when researching backup DNS, I had discovered that no DNS setup can truly unify query logs across instances. Each AdGuard has its own log, so the router and the replica each show a partial picture. My notes read “Loki and Grafana?”

No extra services on the router

The obvious idea is to run a log agent on the router. But when I double-checked the router’s memory situation, I realized that what looked like a shortage was actually just normal page cache usage, leaving about 436 MB of memory actually available. That still made most log agents a poor fit: Grafana Alloy typically uses 300 MB to 1 GiB of memory and has no OpenWrt build, and while Promtail is lighter (about 100 MB), it can spike and doesn’t have a maintained OpenWrt package. Even though something like Vector or rsyslog would fit, polling the AdGuard API from a machine that’s already running everything else doesn’t add anything to the router.

I wanted to get the query logs off, and I didn’t want any extra services running on the router’s hardware, so I decided to build a lightweight container service running on my main server that reads the API and grabs the query logs. If the service dies, I do alert on it, but it doesn’t impact the DNS service itself.

Building the shipper

The shipper is 164 lines of Python, using only the standard library. Every 60 seconds, it calls the query log endpoint of AdGuard’s HTTP API with Basic auth. It never modifies any AdGuard configuration, only reads. The log is returned in newest-first order, paged, so it keeps a timestamp cursor in a file and walks back through older pages on each cycle until it reaches the cursor. It pushes new entries to Loki as JSON lines, nudging duplicate timestamps forward to keep them strictly ascending.

The first run is intentionally quiet: the cursor starts at the newest entry, so nothing gets shipped. No backfill happens for what’s already in the buffer. I run one container per AdGuard instance (router, replica) as a git-driven stack on my home server. Each writes a heartbeat file, and the healthcheck fails if the file is older than three polls (180 seconds). The AdGuard admin login is injected at deploy time and never lands in git.

Two labels and 90 days

I only label Loki with job and host. Everything else (client, domain, upstream, matching rule, cached or not, duration) is in the log line itself, and queries extract them with Loki’s LogQL json parser. Keeping the label set small means fewer Loki streams. A high-cardinality label like client or domain would create a new stream for every value.

Loki’s default retention is 30 days. I added a per-stream retention rule to keep the AdGuard query log for 2160 hours (90 days), which Loki’s compactor enforces. I chose 90 days because I want a real DNS audit history: if I need to look into a problem, I want to be able to go back weeks, not hours.

The dashboard (and a regex bug)

I provision the Grafana dashboard from git. It shows query totals and the block rate at the top, followed by traffic by AdGuard host, top clients, and most-queried domains. At the bottom is a raw log panel for looking at individual queries.

The block rate was showing 99.4%, which would mean almost nothing was resolving. Turned out my filter for blocked queries matched the reason field against Filtered.* unanchored.. so it also caught NotFilteredNotFound, which is AdGuard’s reason for a normal allowed query. Anchoring it to ^Filtered.* fixed that, bringing the block rate down to about 8.4%. The router’s AdGuard instance is doing about 585 queries a minute, and the replica is idle.

I’m excited to extend this to leverage the data with AI to do blocklist and allowlist tuning, do per-device analysis, alert on DNS anomalies, and detect rogue devices (see the recent LG TV outcry about their TVs spying on users from inside their own networks and harvesting private data, which LG denies).

A used HP EliteDesk with a few mods runs my house

Mon, 01 Jun 2026 00:00:00 GMT

After 38 days of uptime, my old server had 87 pending package updates, including a kernel bump. The old hardware was a Dell OptiPlex with an i7-6700, which had been running 45 containers and storing all data on a single 2 TB Samsung SSD in a one-disk ZFS pool. I replaced it with a used HP EliteDesk 800 G6 SFF with an i7-10700, which cost me under $300. I decided to do a fresh install of Ubuntu Server instead of moving the old NVMe, because it would give Ansible a chance to replay its configuration and get rid of the unused GNOME desktop (which uses about 1 GB of RAM and consists of 41 packages) and the 12 to 15 GB of cruft in my home directory. More time up front for sure, but worth it.

ZFS mirror and the parallel run

For storage, I went with two 8 TB drives in a ZFS mirror instead of mdadm, because ZFS does block checksums (against bit rot), snapshots, and I can move the pool to another machine with zpool export and zpool import. I’m not using L2ARC or SLOG, because RAM ARC is sufficient and nothing is bandwidth-bound. Similarly, for memory, I planned 4x16 GB at 2133 (64 GB dual channel) instead of 32 GB at 3200, because capacity is more important than bandwidth for a ZFS box running 45 containers. For now it’s running 32 GB.

I decided to run both machines in parallel on the LAN and migrate services one by one. The old box was my fallback, so nothing got deleted, and the data directories were my rollback net. Throughout this, I had an open hardware question: I’d saved a build of the exact same machine on r/homelab, the same EliteDesk 800 G6 SFF, the same i7-10700, and it made me wonder: does the G6 SFF actually have two M.2 slots? I didn’t know for sure by the time I was done.

Benchmarks and the pool

The first benchmarks against the old box showed the CPU to be around 80% faster, and the NVMe drives were 50 to 75% faster. One Seagate Exos 8 TB was second-hand, with 54,000 power-on hours, zero errors, but a high G-sense count of 14,573. It ran at 55°C during the extended SMART test (about 13 hours). The second Exos was factory-fresh with zero hours. Both went into the mirror.

I created the pool with ashift=12 and lz4 compression. I had 14 datasets planned, but reduced that to 8. I set recordsize=8K for Postgres and 1M for media and Time Machine. I capped ARC at 8 GB of the 32 GB RAM. I didn’t create a backups dataset because a backup on the same disks as the protected data isn’t a backup.

I set a few defaults on the Docker daemon before starting any containers. Using my own address pool means Docker networks won’t collide with my VPN subnets. Enabling live-restore means containers will keep running through a daemon restart. I learned on the old box that having Ansible own everything leads to quick fixes that bypass it and accumulate drift. So on the new box, Ansible only owns the base system. Containers are deployed from git, see How a deploy works with Dockhand. The first service I deployed was a Grafana/Loki/Alloy/Tempo observability stack.

// ansible/templates/docker/daemon.json.j2 (excerpt)
{
  "log-opts": { "max-size": "10m", "max-file": "3", "compress": "true" },
  "exec-opts": ["native.cgroupdriver=systemd"],
  "live-restore": true,
  "userland-proxy": false,
  "shutdown-timeout": 30,
  "default-address-pools": [
    { "base": "<private /16>", "size": 24 }
  ],
  "bip": "<private /24>"
}

Reboot in place first

I rebooted it in place before moving it anywhere. That way I could separate a clean-boot recovery from any cable damage during transit, while still having remote access as a safety. A little script baselined the pre-reboot state and snapshotted the post-reboot state, listing running containers with a no restart policy and running-but-not-enabled system services. The diff showed what wouldn’t come back on its own. The rehearsal caught Uptime Kuma crash-looping because its empty uninitialized database was throwing no such table: setting. I stopped it until I could copy the real database from the old box.

Tip · Diff state across reboot

Take a baseline before rebooting, of running containers with a no restart policy, and running-but-not-enabled system services. Then take another snapshot after reboot, and diff them to see what won’t come back on its own.

The move and the mains pull

The real move was a ZFS snapshot of the pool, a graceful power-off, and a walk to the new spot. I put it on a 1000VA UPS, hooked up power and USB, plugged in the USB microphone for the bird detector, and set the BIOS to power on when AC returns. After it booted, 39 of 40 containers came back byte-identical, the mirror was intact with zero errors, and the Exos drives cooled from 55°C to 42 and 41°C.

I tested the UPS by pulling the mains power. Network UPS Tools reported it was on battery and discharging, and sent me an ONBATT push notification. Power was restored in less than a minute, so it went back online and started charging without shutting down. That’s by design: NUT only shuts the server down when the UPS reports low battery.

# ansible/files/nut/upsmon.conf.j2 (excerpt)
# Conservative: graceful shutdown only when the UPS asserts LOW BATTERY (LB).
RUN_AS_USER nut
MONITOR {{ nut_ups_name }}@localhost 1 upsmon <redacted> master
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
POWERDOWNFLAG /run/nut/killpower
NOTIFYCMD /etc/nut/notify-pushover.sh
NOTIFYFLAG ONLINE   SYSLOG+EXEC
NOTIFYFLAG ONBATT   SYSLOG+EXEC
NOTIFYFLAG LOWBATT  SYSLOG+EXEC

Cutovers and persisted config

The biggest cutover was my smart home stack: Home Assistant, Mosquitto, Z-Wave, Zigbee, Matter, ring-mqtt, go2rtc. I migrated the Z-Wave and Zigbee USB sticks myself, and my verdict on Home Assistant was “works, all devices there”. The API showed 833 entities. But then ring-mqtt kept using the old box’s MQTT broker after the cutover. Turns out its persisted config.json mqtt_url trumps the env variable, so changing the env var did nothing. Same story with Home Assistant .storage URLs and zwave-js-ui settings.. the saved file always beats the container startup config.

Warning · Persisted config beats env vars

In ring-mqtt’s case, the persisted config.json mqtt_url takes precedence over the env variable. Similarly, the Home Assistant .storage URLs and zwave-js-ui settings take precedence.

BirdNET-Go had a similar problem, but with hardware: the config pointed to sound card 1 (the mic on the old box), but the new box’s USB mic was card 0. I patched the config and pinned the card order with a modprobe option (because the UPS might power-cycle the box). I migrated 18,035 detections with clips (7.0 GB on the old box, 4.3 GB on ZFS thanks to lz4 compression, not data loss).

The things I dropped instead of migrating were MagicMirror, Termix, guacd, and the old box’s AdGuard Home replica. I retired the AdGuard replica by repointing the resolver first, so nothing depends on a DNS server about to vanish.

Nineteen healthy and a git split

All 19 migrated services are now healthy on the new box. Their old copies are set to restart=no, and I caught one straggler that was still unless-stopped. Every stack’s secrets are checked against 1Password, and none are left only on the old host. The last mistake was in git: a new-box clone pushed only to Forgejo and an old-box clone only to GitHub split the history. I merged them behind safety tags and a full git bundle, then gave both clones both push URLs as in Self-hosted git with Forgejo and why some repos never touch GitHub. The iperf3 server is moved and does 933 Mbit/s on the LAN. The old server is still running, but its copies are stopped, and two unknown LAN clients still have NFS mounts on it.

A clone is never sufficient

The lesson I wrote down at the time: a git clone (or zfs send, or a git-sync) is necessary but never sufficient. What I missed were host-local files under ~/.config/ and persisted app config pointing at the old box. Both failed quietly hours after cutover. Running both in parallel made mistakes cheap: the old box was intact with its data directories untouched, so fixes were config changes, not restores.

The old box is still up and still holds data, but I need to audit what’s on it before I wipe it.

An Elegant Puzzle's team sizing when output stops tracking headcount

Sun, 31 May 2026 00:00:00 GMT

One question I keep coming back to is.. how big should a team be, once you’ve decoupled output from headcount? A good place to start thinking about team sizing is this essay by Will Larson: Sizing engineering teams, which is also reprinted in the book An Elegant Puzzle (Stripe Press, 2019). The core rule of thumb in that essay is: a manager should manage 6-8 engineers, and a manager of managers should manage 4-6 managers.

There are two more rules from Larson that I think are relevant here: an on-call rotation should have about 8 engineers so that the frequency of getting paged is low enough that people don’t get burned out, and teams should have at least 4 people because otherwise they’ll behave more like a group of individuals than a team. (Larson on exceptions: “I have regretted it every single time.”)

These numbers make sense to me given the purpose of a manager’s span: a manager’s week is mostly spent with people, coaching each individual engineer and reviewing their work. Hiring and career conversations take their own hours. Underpinning all of this is the duty that if someone goes on vacation (or resigns), it shouldn’t bring down the service. None of this scales with lines of code shipped. It scales with headcount and the pager.

This book is often viewed as a playbook for big tech in the growth era. I think some of the hypergrowth-hiring parts haven’t aged well, but the thinking behind the sizing hasn’t. It’s about people all the way through.. you spend the same amount of time coaching people, and you need to cover for absences the same way.

This is why a recent number stood out to me. Fortune reported that Meta’s applied AI engineering group runs at about 50 engineers per manager.. about double the 25 to 1 that’s usually treated as the outer limit of span. I’d push back on this stretch: no one can coach or review 50 people. At that ratio, the manager processes people rather than manages them. Agents add to the review load for the manager, but the coaching load stays the same.

I keep coming back to the small team. If you have 4 people who can ship what 8 people did, those 4 people still need to rotate without burning out, and someone still needs time to review what the agents are doing. The pager and absences fall on fewer people. I’m still not settled on whether the smallest real team is now set by the on-call rotation instead of the workload.

How a deploy works with Dockhand

Fri, 29 May 2026 00:00:00 GMT

On the old server, I was managing each service in an Ansible playbook. That sounded tidy, but in practice I’d often make quick fixes by hand on the box instead of going through the playbook, so the running containers started to drift from what was in the playbook.

The new home server gave me a chance to split those two jobs. I’m using Ansible to manage the base system (like it already handles my network gear), and a dedicated tool for containers.

Last week I did a trial install of Dockhand on the old server and I’m happy with it. It’s lighter than Portainer and can import existing compose stacks. The source is on GitHub under the BSL 1.1 license, so you can read the code. The whole thing is built around its git integration, so kudos to the Finsys project for that.

The repo is the definition

I’m using a monorepo for home-infra, with a folder for each service: stacks/<service>/compose.yaml. The image: line pins the running version, so upgrading is just changing the tag and pushing. Dockhand clones the repo and runs compose from its own copy, so any hand edits on the host don’t matter. For example, I have a self-built search service with its tag pinned in my own Forgejo registry.

# stacks/vault-search/compose.yaml (excerpt)
services:
  vault-search:
    image: <registry>:<port>/<user>/vault-search:2.5.0
    container_name: vault-search
    restart: unless-stopped

And the same repo deploys Dockhand itself, which pins its own release the same way. So even the tool doing the deploying is defined by a git line.

# stacks/dockhand/compose.yaml (excerpt) - Dockhand itself, pinned the same way
services:
  dockhand:
    image: fnsys/dockhand:v1.0.29
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /storage/containers/dockhand:/storage/containers/dockhand
    environment:
      - DATA_DIR=/storage/containers/dockhand

I didn’t configure anything through the web UI. There were lots of forms, so I asked if they could be scripted. That got me making REST calls to a largely undocumented API. I figured out the field names the hard way: sshPrivateKey instead of sshKey, stackName instead of name. The repo URL needed the ssh:// form because Forgejo’s SSH runs on a non-standard port. I learned to push the stack folder before creating the stack, otherwise Dockhand wouldn’t find the compose file. It was fun to figure out, and everything ended up in a single helper script.

Push to redeploy

Each stack has its own webhook URL and its own HMAC-SHA256 secret. The helper script generates a secret with openssl rand -hex 20 and sets it on both ends: in the Dockhand stack and in the new push hook in Forgejo.

# .claude/skills/dockhand/scripts/deploy-stack.sh (excerpt)
SECRET="$(openssl rand -hex 20)"
curl -s -X PUT "$DH/api/git/stacks/$SID" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"webhookEnabled\":true,\"webhookSecret\":\"$SECRET\"}" >/dev/null

curl -s -X POST "$FORGEJO/api/v1/repos/<user>/home-infra/hooks" \
  -H "Authorization: token $FTOKEN" -H "Content-Type: application/json" \
  -d "{\"type\":\"gitea\",\"active\":true,\"events\":[\"push\"],\"config\":{\"url\":\"$DH/api/git/stacks/$SID/webhook\",\"content_type\":\"json\",\"secret\":\"$SECRET\"}}"

With a monorepo, a single push triggers every stack’s Forgejo webhook. Each stack checks whether its folder was changed, and if not, reports the deploy as skipped without restarting. This means I can put my Ansible files in the same repo: editing those triggers all hooks, but all stacks skip.

One wall, on the Forgejo side, is that it refuses webhooks to private addresses by default. That makes sense for a public instance, but it prevents my home server from talking to itself. I fixed that by adding Dockhand’s host to the ALLOWED_HOST_LIST under [webhook] in app.ini (config cheat sheet) and restarting Forgejo. Now the push path worked end to end.

Secrets from 1Password

My first approach to secrets was to check .env.op files into git with op:// references, then run op inject in a systemd oneshot at boot to render the real .env files next to them.

# stacks/observability/.env.op (deleted in 817397d)
GF_ADMIN_USER=admin
GF_ADMIN_PASSWORD=op://<vault>/<item>/password

# ansible/files/monitoring/refresh-secrets.sh (deleted in 817397d), systemd oneshot at boot
for envop in "$STACKS_DIR"/*/.env.op; do
    dir=$(dirname "$envop")
    if op inject -i "$envop" -o "$dir/.env" --force 2>/dev/null; then
        chmod 600 "$dir/.env"
    else
        ERRORS=$((ERRORS + 1))
    fi
done

I gave up on that plan by the afternoon. The oneshot was racing with Docker on boot and not retrying, so it was fragile. The commit message removing it says “Single-user homelab doesn’t need the op-inject indirection layer.”

The replacement is simpler: I read values from 1Password and send them by PUT into the Dockhand per-stack environment as regular variables. Dockhand stores them in an env file in its data directory on ZFS, so they persist across restarts. The compose file references only the variable name, using compose interpolation, never the value.

# stacks/observability/compose.yaml (grafana excerpt)
    environment:
      - GF_SECURITY_ADMIN_USER=${GF_ADMIN_USER:-admin}
      - GF_SECURITY_ADMIN_PASSWORD=${GF_ADMIN_PASSWORD:-admin}

This keeps 1Password as the source of truth, with Dockhand merely handling delivery. I avoided using Dockhand’s own “secret” variable type because those values are only stored in its database (encrypted, but still lost when Docker restarts).

The inject-secrets.sh script was born out of about 7 manual curl runs of this entire sequence. The environment PUT operation clobbers all variables, so the script reads the existing set, merges, and writes it back. It only ever outputs key names and value lengths, never the values themselves.

# .claude/skills/dockhand/scripts/inject-secrets.sh (excerpt)
case "$VALSPEC" in
  op://*) VAL="$(op read "$VALSPEC")";;
  lit:*)  VAL="${VALSPEC#lit:}";;
esac
new_json="$(jq -c --arg k "$KEY" --arg v "$VAL" '. + [{key:$k,value:$v,isSecret:false}]' <<<"$new_json")"

# PUT replaces the whole set, so merge with what's there first
EXIST="$(curl -s "$DH/api/stacks/$SVC/env?env=$ENVID" -H "Authorization: Bearer $TOKEN" | jq -c '(.variables // .)')"
MERGED="$(jq -c --argjson new "$new_json" \
  '($new | map(.key)) as $nk | (map(select((.key as $k | $nk | index($k)) | not))) + $new' <<<"$EXIST")"
curl -s -X PUT "$DH/api/stacks/$SVC/env?env=$ENVID" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"variables\":$MERGED}"

The ordering is important if your stack needs the secret to start. The helper creates the stack and wires the webhook without deploying (RUN_DEPLOY=0), injects the secrets, then deploys. Review caught the initial wrong ordering, which I hadn’t noticed.

Proving the push path

The first six stacks deployed via direct API calls, so I hadn’t actually exercised the webhook path. I wanted to see it work before trusting it. I tested by pushing a one-line comment to the compose file of Scrutiny (my disk SMART dashboard).

# chore: scrutiny compose comment (webhook path test)
--- a/stacks/scrutiny/compose.yaml
+++ b/stacks/scrutiny/compose.yaml
@@ -1,3 +1,4 @@
+# Disk SMART dashboard + history (complements smartd+Pushover alerting from Phase 2b).
 services:
   scrutiny:

As expected, Scrutiny redeployed and nothing else did. The injected Scrutiny secret survived the process too. It was satisfying to see the first full loop work as designed. I’ve now made the test push a standing step in my deployment runbook.. every new stack gets one before I consider it done.

But that check has a limit. A green deploy from Dockhand only means the compose command ran, not that the config you think should reach the container actually did. I’d mounted my observability stack’s configs with absolute paths from a second repo copy Dockhand never syncs, so my Loki limits edits never landed and it ran defaults. Switching to relative mounts fixed it and I added a “mounts must be relative” rule to my stack standard runbook. I also learned the hard way that while git stacks deploy via POST /api/git/stacks/{id}/deploy, /api/stacks/<name>/deploy?env=1 just returns an empty body.. now I check the actual container state and Dockhand’s sync status instead of assuming the response means anything.

Nine stacks from git

By the end of the first night, I had nine stacks running from git, each with its own Forgejo webhook. I migrated Dockhand’s data from a Docker named volume to a ZFS bind mount backed by Sanoid snapshots. I enabled authentication, backed up the API token and encryption key to 1Password, and the next day started building my own images through Forgejo’s container registry with pinned tags. Dockhand pulls them using its own stored registry credential (not the host’s docker login). Because git defines what should run, I now check the running container against it after every deploy. The registry is part of my self-hosted Forgejo, and I verified the stored credential works on the search service.

Next up: a commit and the running container that disagreed.

The SDLC isn't dead but the review queue might be

Wed, 20 May 2026 00:00:00 GMT

You want to get attention, you don’t say things are changing. You just say this is dead.

“Dead” refers to Boris Tane’s post The Software Development Lifecycle Is Dead, which argues that coding agents collapse the classic stages of the lifecycle into a single tight loop. There’s a good discussion of the post from both sides on the Changelog & Friends episode. I think the title is overstated, but the underlying argument is sound.

You’re assuming each of these is a massive undertaking. I always felt like it should be smaller, tighter, and loopier.

I always thought so.. the hand-offs between stages were the real problem.

We’ve ritualized and held sacred these steps along the way because we’ve had to and that’s been our comfortability. So as humans, we have this comfortability gate.

A lot of code review is precisely such a gate. It’s a way for the reviewer to gain confidence that someone looked at the change. That’s not the same as proving the code does what it’s supposed to do.

If you have an agent that files a few hundred PRs per day, and your team can review ten, your reviewers don’t suddenly become more careful. Instead, you get a backlog, and approvals degrade into clicks to clear the pile.

You’ll always care about some version of code quality, but the purpose of caring about code quality is, is the intent met? Does it pass tests? Does it actually work in development environments? Does it pass the smoke test where we spin it up 50 times and battle test it?

The human stays on question one, is the intent satisfied, because we need human judgment about what was really being asked for. Everything else gets automated into gates that every change has to pass through: run the test suite, spin it up repeatedly and do smoke runs.

I’m not sure exactly how, but I’m thinking code quality over code review. That’s where I’m leaning.

I lean the same direction, but there’s one prerequisite hurdle. In regulated organizations, the second-person approval doubles as audit evidence that a non-author verified the change before release. So the open questions are: what would an auditor accept in lieu of the approval click, and can a quality gate generate it?

Frictionless and the friction you only find by going faster

Mon, 18 May 2026 00:00:00 GMT

Imagine your agent successfully converts a ticket into a pull request before lunch. The code itself took minutes, but now the PR just sits there for a day or so because of a flaky pipeline or a reviewer with a similar stack of PRs. Frictionless, the book by Nicole Forsgren and Abi Noda, names it: cheap code-writing shifts the constraint from creating the code to validating and delivering it.

You won’t notice where something’s slowing you down until you actually speed up other parts of the system.

This is a pretty typical thing bottlenecks do in any system.. the slow review queue and the tests you didn’t trust were pre-existing problems that most people on the team had been dealing with for years, but the faster code writing removed the thing that was masking them.

So, what do you fix first.. the pipeline, the flaky tests, or the review queue? The book suggests you start before you even fix anything, with a listening tour. Talk to engineers and find out where they’re really losing time. Then, score each candidate by reach (how many people it affects) times impact, weighted by your confidence in those estimates, and divide by effort. You’ll usually find that the cheap fix that helps a lot of people, like the flaky test everyone has to rerun automatically, rises to the top.

The order is more important than the formula: asking first means you don’t waste a quarter rearchitecting your delivery pipeline when the daily universal pain is one unreliable test that one person can fix in an afternoon.

I really do believe that we’re at this fortunate moment in time with artificial intelligence that what’s good for AI is also good for humans. Fast builds, tests, documentation, clean code, all helps AI, but it’s also just easier for humans to reason about.

I see this as cheap code raising the urgency of an old developer experience backlog. The boring backlog now has an argument that executives will listen to. Fast builds and clear documentation used to be quality-of-life requests. Now they determine whether agent speed makes it to production.

I just think at some point someone’s going to have to answer for the code that’s written. And I think people are just optimistic that no, they won’t. That an AI agent will answer for the code that was written.

There’s a question underneath all of this what-to-fix-first stuff that nobody has a good answer to yet. Each of the fixes on this list makes it easier to move a ticket into production with fewer humans touching the change. If the pipeline is fast and trustworthy enough to approve the agent’s change on its own, whose name goes on it?

If agents write the tests, is Feathers' legacy code still legacy?

Fri, 15 May 2026 00:00:00 GMT

In 2004, Michael Feathers wrote a book called Working Effectively with Legacy Code which makes the following argument in one line: “To me, legacy code is simply code without tests.” The method in the book follows from that definition: find a seam (a place where you can change the program’s behavior without changing the code in that place), write characterization tests that pin down the current behavior (right or wrong), and only then change the code.

But in early 2026, Anthropic published a post on COBOL modernization that challenges this method. The company claims that the most expensive part of any COBOL migration is always understanding what the old code does. AI can map dependencies, trace execution paths, and generate tests to verify that the migrated code produces the same outputs as the COBOL. This matters because COBOL handles an estimated 95% of US ATM transactions. The post also says something crucial: people still need to decide if those tests are enough.

This question resonates with me because I’ve spent the last ten years working with large financial institutions and Fortune 500 companies, now as a principal cloud architect. Those estates run mainframe batch jobs and old Java code under the newest platforms, but their modernization plans keep stalling because of the exact problem this book names: code without tests can’t be changed safely.

Feathers still keeps the definition. In a 2024 interview on Tech Lead Journal, he also describes legacy code as code we don’t understand. He’s skeptical of tools that promise to write all the tests for you, but makes an exception for characterization tests, since they document the current behavior and can be thrown away after the job is done.

If tests are almost free, what’s still legacy? I think the definition holds, but the scarce part has changed. An agent pinning down a COBOL program’s behavior or an old Java service is way cheaper than getting a team to do it manually. But the agent can’t tell you which of that pinned behavior is a business rule and which is a bug that’s survived for decades in production because nobody dared fix it. That judgment is where the legacy cost now lives.

I’m saying agent-generated characterization tests are the cheapest on-ramp to modernization that financial services has ever had. They give a team the safety net to start changing code well before they can afford a full parallel run. Getting started is where such programs have always stalled.

But it’s only the on-ramp, and it doesn’t prove the new system. Pinned tests only cover the inputs someone thought to try. When you cut over a core system, you run the old and the new side by side on live inputs and switch over in small pieces. Anthropic’s post recommends the same.

The book’s question comes back: if cheap tests mean legacy is code nobody can explain, the hard part of modernization is the person deciding the pinned behavior is the requirement. Who signs off that the tests mean the right thing?

The Art of Action says brief the intent, even to an agent

Sat, 02 May 2026 00:00:00 GMT

When a plan fails to translate into the right action, our instinct is to add more detail. In The Art of Action (2011), Stephen Bungay identifies 3 gaps that explain this instinct: the knowledge gap (between plans and outcomes), the alignment gap (between plans and actions), and the effects gap (between actions and outcomes). We try to close the knowledge gap by adding more data, the alignment gap by issuing more detailed instructions, and the effects gap by introducing tighter controls. But, he says, adding more detail usually makes the gap you’re trying to close even bigger.

Bungay’s solution to this is to use a technique from the 19th-century Prussian army called mission command, or directed opportunism. You limit your direction to intent.. what must happen and why.. and each level comes up with its own route and reports back to the person who briefed them (a backbrief). If reality doesn’t match the plan, people are free to adjust their actions as long as they’re consistent with the intent. But you have to keep the brief short: “What has not been made simple cannot be made clear and what is not clear will not get done.”

I see the same reflex when I delegate work (writing code or changing infrastructure) to software agents. When the agent gets the task wrong, the natural reaction is to make the prompt longer, add more rules, or grow the list of things it should never do. That’s the alignment-gap reflex, just in a new setting. But as with human teams, making the list of instructions longer rarely fixes an agent that misunderstood the task.. it’s the same over-specification Bungay warns about. Telling the agent what needs to happen and why usually works better. Constantin Gonzalez makes a similar connection in his essay on delegating to models, where he describes prompting as delegation and lists commander’s intent as one of several frameworks that point in the same direction. His advice: tell the model what success looks like and why it matters.

The backbrief is where I land as the most useful part of Bungay’s approach to carry over to agents. Asking the agent to explain its plan before it makes any change is the one point where you see how your intent was read. A misread intent is cheapest to catch at the plan stage, before the effects gap opens on a real change that would need undoing.

I’m less sure about the freedom to adjust. If an officer adapts to the situation within the intent, he shares training and doctrine with the briefer and is accountable for his decision. The agent fills in the gaps with a confident guess. Does intent work without shared doctrine, or must the backbrief carry it alone?

Build, buy, or let AI write it: Khononov's subdomain test

Wed, 29 Apr 2026 00:00:00 GMT

Learning Domain-Driven Design by Vlad Khononov (O’Reilly, 2021) describes the strategic part as starting with a classification of subdomains, before you even start writing code. The core subdomains are what differentiate your company from its competitors. They’re complex and they’re constantly changing (if they were easy to do, your competitors would have copied you already). Generic subdomains are the hard problems that all companies share, but which have already been solved (like identity). Supporting subdomains are subdomains that you need to run your business, but which are simple and don’t give you a competitive edge.

Each type maps to a builder: core gets built in-house by your strongest people with careful business modeling; generic gets bought or adopted from open source; supporting gets built with simple patterns or outsourced to an external team. An informal test from book discussions: if someone would pay for the piece as a standalone, it’s likely core. Types can drift: core becomes generic through commoditization, supporting becomes core through growth, so the map needs periodic redrawing.

The sort is most valuable in modernization. McKinsey estimates up to 70% of Fortune 500 software is over 20 years old, and warns that translating code line-by-line merely relocates tech debt to a new platform. After ten years working with large financial institutions and Fortune 500 companies, now as a principal cloud architect, I believe the subdomain map is the first filter on a modernization budget. Most of a legacy estate is supporting or generic, so spending your best people on a faithful rebuild isn’t spending where the company competes.

AI moves the line again. In early 2026, software stocks dropped about 8% in a week on fears that agents would let enterprises build or automate what they licensed (ACS Information Age reported). Cheap generated code changes build-vs-buy most at the supporting and generic edges. I say generate supporting, keep buying generic, put your best people on core. Core’s hard part is the model of the business.. cheap code doesn’t make maintenance cheap. But if agents make supporting code nearly free, who owns it in five years?

Site Reliability Engineering, and the parts a bank can't copy

Tue, 28 Apr 2026 00:00:00 GMT

Site Reliability Engineering (O’Reilly, 2016, available online for free at sre.google) introduced the industry to the concepts of SLOs, error budget, and toil (repetitive manual work). The standing reaction from outsiders since the book’s publication has been “this only works if you have unlimited money”. Ben Treynor Sloss says in the introduction “SRE is what happens when you ask a software engineer to design an operations team.” This explains the tone of the book. If you work in IT at a bank or a Fortune 500 company, instead of asking “should we do SRE?” it’s more useful to ask “which parts of SRE will still work outside of Google?”

The enforcement mechanism travels the worst. Google’s typical policy is when a service’s error budget is exhausted, feature launches are frozen until reliability improves. This assumes releases are the primary cause of failures, and that the product team can stop shipping. In a large enterprise, unreliability usually comes from vendor integrations and shared infrastructure that no single team releases. A freeze collides with regulatory deadlines and change boards that run on their own calendar.

The operational work cap suffers from the same problem. The book says at least half of each SRE’s time should be on engineering projects, and toil beyond that point means excess ops load gets returned to developers. This assumes your organization is willing to give the pager back to dev teams when SREs get overwhelmed, which few enterprises are built to do. That’s the part I’d rule out.. adopting the whole package, launch freezes and all, with a cap enforced by shifting work between teams.

On the other side of the debate, many companies simply renamed their ops team and did nothing more. The title is now drifting toward “ops support” as platform teams absorb the engineering work. I’ve spent ten years working with large financial institutions and Fortune 500 companies, now as a principal cloud architect. The problem isn’t the title.. it’s the organization. If your SREs aren’t writing code and aren’t in the room when architecture decisions are made, they’re an ops team with a new badge. The first question for any company that has SREs: what does the word mean to them?

What is transferable is the idea behind each mechanism. An SLO is a reliability number your business agrees to and signs. Toil is something you measure and cap (and you can enforce the cap through planning, not a reorg). You don’t need Google’s org chart for either one. But they do let a large org talk about reliability before an outage forces the issue.

Regulated industries have an external version. The UK’s operational resilience rules (FCA PS21/3) tell banks and insurers to identify their important business services, set an impact tolerance for each (the maximum tolerable disruption), and prove they’re staying within it. The deadline for compliance was at the end of March 2025. The impact tolerance is the outer bound that contains the internal SLO. The regulator writes the target; it’s your job to set a tighter one for yourself.

This changes the purpose of the error budget. I think the error budget transfers least as a rule and most as a language. A bank can’t freeze launches based on the error budget alone, but it can make the SLO the number that product, operations, and risk argue about, as long as they stay within the regulator’s tolerance. The gap between the SLO and perfect reliability is what each side spends on change, and spending becomes a negotiation instead of a freeze.

Toil is worth watching because it hasn’t changed much at all. The Catchpoint SRE Report 2026 still shows practitioners spending about a third of their time on toil, roughly flat year on year. The AI-enabled toil reduction shows up mostly in managers’ answers, much less in engineers’ answers. If AI is the enterprise’s new automation budget, is anyone measuring toil before and after the way the book’s 50% cap assumed?

Ansible manages every config on my OpenWrt router and access points

Mon, 27 Apr 2026 00:00:00 GMT

My Flint 2 router is probably my most boring yet awesome piece of equipment. Several years old at this point, it still punches above its weight. It runs vanilla OpenWrt brilliantly with great support.

I manage all OpenWrt devices with Ansible from a single repository. The router uses the openwrt role, and the APs share the openwrt-ap role, each applied with a separate playbook. Getting rid of the Eeros (which are app-only, see OpenWrt to the rescue) was also a way to bring my network back under code control. The main lessons since then have been about the mismatch between what Ansible thinks it has written and what’s actually on the device.

With Eero (and before that, Orbi), everything was provisioned and configured with click ops. It was tough to reproduce and test things and to quickly roll back. This is another example of bringing my professional tendencies home: infrastructure as code and configuration as code are big deals to me.

I created the router role, replacing the Verizon router with Flint 2. It goes straight into the ONT now. The first session was 39 commits. It uses the community.openwrt collection (Galaxy page), which runs modules through the shell so you don’t need Python on the router. I designed it as a single role with a task file for each subsystem, which works well for a single device and scales to VLANs and Wi-Fi without refactoring. The per-subsystem tags (system, syslog, packages, network, firewall, dhcp, dns, wireless, vpn, uhttpd, acme, openvpn, tailscale, alerts, ddns, dawn, performance, storage, backup) let you apply just one area if needed. I love this. Per-device settings go in host vars (the router has over 40 DHCP reservations). Secrets (Wi-Fi passwords, VPN keys, DDNS keys, alert tokens) come from 1Password via the CLI at run time, so none are in the repo. Sweet.

The openwrt-ap role came in shortly after. It takes an M3000 with fresh vanilla OpenWrt and turns it into a dumb AP: bridge the WAN port into the LAN, set a static IP, disable DHCP and router advertisements, configure wireless and client steering. On April 7, I added performance tuning as code: hardware and software flow offloading, packet steering, and some sysctl changes so the APs can keep up with a gigabit WAN. The role stops (and disables) odhcpd on the APs, so the router is the only IPv6 control plane.

The first real test came with the OpenWrt 25.x upgrade on April 6-7, before the Eero cutover. OpenWrt 25.12 switched from opkg to apk, so the roles now test for /usr/bin/apk and branch on it. The new release also wants interface addresses as CIDR list ipaddr in UCI. I dropped WireGuard tasks, usteer tasks, and the bridge-loop watchdog (which only existed to guard the proxy-ARP VPN setup, already gone). I added DAWN for client steering and the new VPN, so the AP role grew DAWN and performance task files.

The first --check --diff dry run of the router role revealed seven check mode bugs.. because community.openwrt.command skips in check mode, so registered results were undefined downstream. I fixed those by marking detection tasks check_mode: false and adding | default(...) fallbacks. ansible.builtin.stat fails outright (no Python on the router), so file checks switched to test -x via the command module. This is working well so far.

The Ansible provisioning wasn’t without its quirks and pitfalls initially, getting it all solid. The DHCP and DNS issues we had out of the gate were a pain. On the surface everything was fine.. but the signals disagreed. It took a little bit of debugging, but we smoothed it out. Lesson learned: RTFM.

The first issue I had: when my iPhone was unable to join the network. I quickly discovered that dnsmasq on the router wasn’t running, and the log told me that was because of a duplicate dhcp-host IP address, which made it fail to start. Turns out that I had some stale DHCP entries from the April 6 subnet change, right next to their replacements: the old short names for two access points and two Phyn water sensors, as well as three old Orbi entries whose addresses collided with the Toniebox. Deleting those seven stale entries by hand brought DNS back up.

I was skeeved out by this, so I audited every UCI entry on all three devices against what Ansible had declared. The result was 4 critical, 8 high, 7 medium and 7 low findings. The VPN obfuscation keys were listed under option names that the driver ignores, so that config was inert. The Phyn sensors had their per-host DHCP options tripled because uci add_list appends on each run; deleting the list and then adding it fixed that, and it stayed single after two more runs. One AP listed the same bridge port four times. The roles contained dead usteer code, and the APs retained their factory WAN zone and firewall rules. Every add_list in the roles now follows delete-then-add.

On the same day, a run failed midway through the DNS tasks. The old dns.yml stopped AdGuard Home, then rendered a template with 1Password secrets. The Ansible server resolves DNS via AdGuard Home, so the 1Password lookup timed out, the task failed, and AdGuard was down for 2 minutes until I manually started it again. This happened more than once. The rewrite: first do a pre-flight DNS check on the server, cache the secret as a fact, render the config to a staging file while AdGuard is still up, then stop the service, atomically move it into place, and start it immediately (inside the task, not a handler, because handlers wait until play end).. ending with a DNS check with retries. A full router run afterward showed 191 tasks ok, 68 changed, 0 failed.

The second time dnsmasq went down, a couple of weeks later, was the same kind of mistake: I renamed two reservations labeled “Apple TV” that were actually the Wi-Fi modules of August smart locks, but left the old sections in place next to the new named ones. The stale anonymous section duplicated the Ring camera’s address and crashed dnsmasq into a loop until I cleaned it up manually. The fix in dhcp.yml is an orphan purge that compares the router’s host sections to the declared reservations and deletes extras. A pre-commit duplicate-IP check aborts the run, and a handler fails the run if dnsmasq isn’t running after restart. The gather+check tasks run in check mode too, so a dry run reports the same problems. The purge is untested on a real rename (the April 25 duplicates were removed by hand first), so I’m watching that closely.. I’ll update this post after the first real cleanup.

The same day, I discovered that DAWN was stopped on all three APs. The roles had been using state: started, which starts a stopped service but never restarts a running one, so none of the DAWN config changes were being applied. Both roles now have a Restart DAWN handler. I also audited the roles for secrets in their output: DDNS keys, Wi-Fi passwords, alert tokens, the AdGuard Home admin hash, VPN peer keys.. all printed to stdout during runs. Every task that handles a 1Password secret now has no_log.

Adding some robustness and resiliency to the deployment pipeline made sense given the issues. I’m very impressed with how easy the configuration is now, especially with agentic assistance.

Part of the AP role is scaffolding we expect to be deleted. The tasks to remove the factory WAN zone and firewall rules only exist because the vanilla OpenWrt image contains them. They wouldn’t be present in a custom AP image.

What’s next: I have automated backups and snapshots of the router dumping often as part of my obsessive backup strategy, but I think I could be a little bit better with config drift detection and remediation (drift from code), and I’m looking into a custom OpenWrt image building pipeline to make updates easier. The building blocks are there with Image Builder, Attended Sysupgrade server, and the firmware selector built on top of it.

Next up is AdGuard Home, the DNS server this post kept tripping over.

Running my Reddit archive as a real service

Sat, 25 Apr 2026 00:00:00 GMT

In part 1 I covered getting the saved posts out of Reddit and into the vault, and in part 2 the refresh engine that keeps it current. This part is about how it runs day to day on my Linux server, and what it took to get it to the point where I’d call it production.

Running it as a real service in my homelab has been a great success. I’m able to monitor the service and keep things running and syncing 24/7 with low overhead.

It runs as two containers built from one image on my Linux server, one handles ingest every 2 hours and the other runs the refresh sweep every 24 hours. I generally let a new service run for two days before I’d call it “production”. Thats what I did here. Almost 6000 posts refreshed in that window, score and comment-count history piling up, deletion markers landing with timestamps, new saves getting tagged roughly 20 seconds after they came in. I dropped tagging requests to the LLM endpoint to 3 a minute and after that, zero rate-limit errors. If something fails a refresh it gets picked up on the next run (17 in the first full run).

It took 64 commits to get there, the image size dropped from 10.1GB to 535MB after some optimizations. I tagged a rollback image before every risky change, and I kept a cutover/rollback runbook fresh so switching versions back and forth is documented step by step.

I may switch from saving posts to a system where I just send the post or comment URL (Apple Shortcut, hotkeys on macOS, etc) to an API and it handles the scraping, almost like a Raindrop type of thing. This would also allow me to send any URL or YouTube video to it and have the stash API figure out how and where to save it (and how and when to refresh it in perpetuity), but this is a little ways out. It has been a really fun project, that’s for sure!

The refresh engine that keeps my Reddit archive alive

Thu, 23 Apr 2026 00:00:00 GMT

My favorite feature of this has to be the refresh feature. The ability to keep pulling updates and adding them to the JSON sidecar allows me to build on top of this and keep the corpus “alive” with fresh data as time goes on. I have leveraged this body of content in my research, hobbies, even my work.

In part 1 I covered getting my saved posts out of Reddit and into the vault. In this post I’ll cover what happens to the posts once they are in the vault and why the refresh engine has been the most valuable feature.

JSON sidecars next to the markdown

I store Reddit posts in a sidecar JSON format (along with the original markdown) which allows me to query them on disk, append easily on changes to posts, keep records of deleted or edited posts, and all kinds of useful metadata about posts as soon as they are saved. This gives me a lot of flexibility in searching and recall activities and ensures I have the full history of the post and comments.

Trimmed example of a sidecar for one saved post (values made generic):

{
  "schema_version": 1,
  "id": "abc123",
  "type": "saved_post",
  "post": {
    "title": "...",
    "posted_at": "2025-10-11T19:50:37Z",
    "body_original": "...",
    "body_current": "...",
    "body_versions": [ ... ],
    "score_history": [ ... ],
    "deleted_at": null,
    "deleted_by": null
  },
  "comments": [
    {
      "id": "...",
      "depth": 0,
      "first_observed_at": "2025-10-12T14:02:11Z",
      "last_observed_alive_at": "2026-04-20T03:15:40Z",
      "body_original": "...",
      "body_current": "...",
      "body_versions": [],
      "score_history": [ ... ],
      "deleted_at": null
    }
  ],
  "capture": {
    "saved_at": "2025-10-12T14:02:11Z",
    "last_refreshed_at": "2026-04-20T03:15:40Z",
    "refresh_count": 3,
    "refresh_tier": "archived"
  },
  "recovery": {
    "attempts": [],
    "successful_provider": null
  },
  "derived": {
    "changed_since_prior_refresh": {
      "any_change": true,
      "post_body_edited": false,
      "new_comments": 0,
      "edited_comments": 0,
      "deleted_comments": 0,
      "removed_comments": 2
    },
    "latest_score": 31,
    "latest_num_comments": 91,
    "comment_stats": { "total": 92 }
  }
}

The original body is never replaced (body_original stays put, body_current moves, and all edits in between land in body_versions). Comments get a first_observed_at and a last_observed_alive_at, so if one disappears I know when it went. The derived block is the quick answer to “did anything change since last time?” without diffing the whole file (in this case, two comments were dropped since the prior refresh). I spent a lot of time working this out to work the way I wanted. I’m super happy with how it came out because it lets me leverage and visualize the data in really neat ways for my needs.

The refresh schedule

Merges are additive only, I don’t overwrite and store all deltas in the data type on disk. I do more frequent refreshes when the post is newer and taper that off to weekly, monthly, quarterly, and yearly as the post ages since activity is generally lower and we’ve already captured a lot of the changes during the post beginning. This is for performance and API respectfulness.

The tiers as I run them today:

Tier	Post age	Refresh every
fresh	first day or so	24 hours
active	after that	7 days
trailing	older	30 days
archived	older still	90 days
cold	oldest	365 days

The refreshes are all done through the Reddit API with PRAW and only the posts that are on their tier schedule are fetched, so the daily sweep is very small even though I have thousands of posts on disk.

I also have some logic that checks the Wayback Machine, PullPush, Reveddit and Reddit previews for any comments that were deleted (on older posts that I may save for whatever reason). These are logged in the sidecar recovery block as well, along with which provider was successful in getting the comment (if any).

Browsing the archive in Obsidian

Since I have them stored in my knowledge base vault, I’m able to search and browse interactively. The AI summary is available in markdown and this is useful for agents querying the dataset, but I’m able to search across the raw posts and comments using some queries on the JSON sidecars. I’m building an interface that lets me browse this on my phone similar to the Narwhal experience, but that is a little further away.

Who owns the code? Beyond Vibe Coding held up against 2026

Thu, 16 Apr 2026 00:00:00 GMT

In 2026, regulated engineering organizations have largely worked out permissioning for AI coding tools and developers are using them daily. The real question becomes what to standardize around their usage. It’s time for your bank or insurer to stop debating permissions and start standardizing AI-assisted engineering. Let’s standardize on vibe coding being confined to the sandbox, the merging engineer owning the change, tests and security scans being the contract, and provenance being recorded for every change. Let’s stop measuring adoption by the volume of code generated by tools.

From my perspective.. ten years working with large financial institutions and Fortune 500 companies, now as a principal cloud architect.. the clearest articulation of the split we should standardize around comes from Addy Osmani’s Beyond Vibe Coding (O’Reilly, 2025). Three ideas in particular hold up against what teams are actually doing in 2026, and each points to a standard we should adopt.

Vibe coding stays in the sandbox

Osmani makes a clear distinction between vibe coding and AI-assisted engineering. Vibe coding is about high-level prompting and accepting the output without deep review.. which he argues is fine for prototypes or learning. But AI-assisted engineering is about using those same tools as part of a normal engineering process, with review, tests, and a human ultimately accountable.

I’d argue that distinction has evolved from a description to a policy line, and in a regulated shop, the line is production. I’d recommend documenting where vibe coding is welcome (sandboxes, prototypes, spikes) and defining a single route into production where a demo-working prototype has to go through the same checks as any other change.

The hard 30%

The second idea is Osmani’s 70% problem.. AI gets you to 70% fast, but the last 30% (edge cases, integrating the new module with the rest of the system) is still hard. Security is in the last 30%. Veracode’s 2025 test of AI-generated code found that models picked the insecure option 45% of the time, and larger/newer models didn’t do any better than older ones.

So I’m concluding that tests (and security scanning) on every AI-assisted change are a contract. Write tests before code where possible. And stop counting generated code or PR volume as productivity.. the hard 30% is invisible in both.

You merge it, you own it

The third idea is the developer as reviewer/orchestrator. The book talks about the engineer directing the tools, reviewing their output, and deciding what gets shipped (Osmani’s shorthand elsewhere for this is “copilots, not autopilots”). I think this maps to a rule regulated enterprises already have: the merger owns the change, “the AI wrote it” is never acceptable in an incident review or audit. So review capacity becomes the bottleneck. I’d standardize small diffs and a failing test that existed before implementation.

Provenance also goes in the change record.. which tool was involved, which human edits/approvals happened before merge. This isn’t just for audit: the US Copyright Office said in early 2025 that prompts alone don’t confer authorship of AI output, protection hinges on the human-determined expressive elements. So ownership tracks to demonstrable human contribution, and the change record is where that’s evidenced.

The objection: the models will catch up

The strongest objection is obvious.. tools improve so quickly, a 2025 book is skating to where the puck was.. next year’s models may close the last 30% on their own. But even if generation improves, the accountability and audit requirements in regulated industries don’t shrink. Standardizing how you review and record pays dividends regardless of how good the models get.

One risk to watch is junior hiring. If you cut it because AI does the first 70%, you borrow against the future owners of the last 30%. Osmani makes the point that AI benefits experienced devs more than beginners.. so juniors need deliberate space to develop the judgment the hard part demands.

What I’d tell a CIO

So I’d advise a CIO to standardize four things: a sandbox/production line, merge-owner accountability, tests + security scans as the contract for every AI-assisted change, and provenance in the change record. Stop doing two things: adoption metrics on code volume, and rubber-stamp review of large AI-generated diffs. And keep hiring/growing juniors.. they’ll review the last 30% when today’s seniors move on.

Walking around for fast roaming's sake

Wed, 15 Apr 2026 00:00:00 GMT

I spent most of April 9th walking back and forth along the same loop in my house, watching the logs on the APs, trying to get 802.11r fast roaming working between my OpenWrt APs. I didn’t see the iPhone do a fast transition until the evening, and the settings that eventually worked weren’t the ones I’d changed in the morning.

Fast roaming mattered because the netgear orbi system and later the Eero mesh I replaced in OpenWrt to the rescue had it, and losing it showed up on calls. Our previous systems had fast roaming, and our Apple devices handled it beautifully as we moved around the house. Cell signal at home is spotty on both of our carriers, so our calls ride on Wi-Fi calling and FaceTime audio, and Wi-Fi has to be consistent and reliable. As we move between APs, the handoff should be seamless, with no dropouts. I also work from my home office from time to time, and strong Wi-Fi around the house is critical for that.

The walks

By keeping an eye on the AP association, signal-to-noise ratio (SNR), and packet loss while walking around the house, I feel confident in my choices for AP placement and power levels. I always walked the same route during my walks: starting in the kitchen, then going through the living room, master bedroom, a kid’s room, up to the office (third floor), and then back down.

Walks 1 and 2 were from that morning. In both, the iPhone does 14 full handshakes with 0 fast transitions (FT). That means every time it moves to a new AP, it needs to fully authenticate again instead of doing the shortened 802.11r exchange. At the end of the loop, you can see it ping-ponging between the kitchen and living room APs, doing 5 reassociations in 45 seconds.

My first wrong turn was a missing line. I searched the OpenWrt-generated hostapd config for ieee80211r=1, didn’t find it, and assumed 802.11r was off. That was a dead end. OpenWrt signals FT through FT entries in wpa_key_mgmt (like FT-PSK), not a separate line. It’s in the OpenWrt wireless documentation if you know where to look.

The second wrong turn took me longer to get out of. Each access point had been deriving its own NAS identifier, but my agent told me that all access points in a mobility domain needed to share one. So I set a single, shared nas_identifier on all three access points with Ansible, rolling them out in the following order: living room, kitchen, router. When I walked through the house for the third time, I was still at 0 fast transitions.

Good enough, until the master bath

I stopped here because all the important spots in my house were already at -65 dBm or better (living room: -35, kitchen: -53, master bedroom: -64, office: -55). The roam glitch is about 300 ms, which I only notice when I’m on a call and walking around. So I decided that was good enough for now and wrote up this status.

That afternoon, I noticed my iPhone flapping in the master bathroom. The best AP was only -72 dBm, the other two were -80 and -91. I made two changes: I changed all APs to WPA2 only with FT-PSK, dropping the SAE/WPA3 transition mode, and I increased the 5 GHz transmit power on my router from 14 to 17 dBm.

In walk 4, the flapping was fixed and the router kept my phone connected across floors, at a 1200 Mbit/s connection. There were still 0 fast transitions. The APs now ran the following configuration:

wpa=2
wpa_key_mgmt=WPA-PSK FT-PSK
mobility_domain=<domain>
ft_psk_generate_local=1
ft_over_ds=0
reassociation_deadline=20000

So my working theory at this point was that iOS will deauthenticate from the old AP before roaming, and that means that FT is impossible no matter what the APs offer.

The evening fix

I rejected that theory and went back to the sources myself, namely Apple’s page on Wi-Fi roaming support in Apple devices and the OpenWrt forum threads. The confident answer I’d been given was wrong (APs don’t need to share a NAS identifier, the community best practice is to configure a unique NAS-ID per AP) and during my review, I found an even bigger problem with the change I had made that afternoon: SAE requires Protected Management Frames (PMF), so my WPA3 transition mode had turned on PMF without me realizing, and when I switched to pure WPA2, PMF got turned off. And some iOS versions refuse 802.11r without PMF!

That evening, I made 6 changes and ran Ansible once to apply them all. I set the canary order so the AP my iPhone was connected to went last. The 6 settings I wanted were:

Setting	Why
`ieee80211w=1`	PMF in optional mode. Some iOS versions need it before they use FT, and clients without PMF can still connect.
`nas_identifier` unique per AP	Community best practice. Each access point has its own identity in the logs.
`proxy_arp=1`	The access point answers ARP for clients in power save, so the phone stays reachable on the LAN while it sleeps and when it roams.
`wnm_sleep_mode_no_keys=1`	A KRACK-family mitigation for WNM Sleep Mode key reinstallation (CVE-2017-13087).
`reassociation_deadline=20000`	About 20 seconds for the client to finish the FT reassociation. OpenWrt defaults to this already, but I set it explicitly so a future update can’t change it on me.
802.11k neighbor reports from DAWN	DAWN injects neighbor reports so clients know which access points are nearby before they roam.

After the run, I could successfully initiate FTs from both my iPhone and MacBook, and I observed 802.11k beacon reports flowing. Most importantly, all 31 stations remained connected without any PMF-induced connection drops.

Which change did it

I never isolated which of the six changes enabled FT because I shipped them all together, but my best guess is PMF.. it was previously supplied by SAE transition mode and got silently turned off when I switched to pure WPA2 in the afternoon. The per-AP NAS-ID is a best practice, but not the fix. With ft_psk_generate_local=1, hostapd derives the FT keys locally on every association from the identifier the client presents, so shared and unique NAS-IDs behave the same. That’s what the Ansible role’s comment says, after all. The other thing I learned is: have patience. iOS needs a warm-up before it starts using FT, so one walk with 0 fast transitions doesn’t mean FT is broken.

I needed a way to ensure consistent settings across my entire AP fleet, both now and in the future, that were tuned for performance and resiliency. Because of this, I now have all of my settings defined in Ansible, which is managing every aspect of the configuration on the router and APs. We’ll look at Ansible in the next part of this series.

OpenWrt to the rescue

Tue, 14 Apr 2026 00:00:00 GMT

Our home Wi-Fi is back on OpenWrt, with a GL.iNet Flint 2 as the router and two Cudy M3000 access points, all three running the official OpenWrt 25.12 release (25.12.2). For a few days in early April it was running on a three-node Eero Pro 6E mesh, and the Eeros were supposed to be the stable option…

The two weeks prior to that were rough on OpenWrt though, between a bridge loop at home, Apple TVs breaking with sae-mixed and 802.11r turned on, a Sonos outage, and a day of debugging a couple Phyn water sensor that kept failing TLS through the M3000 bridge.

I had heard great things about the Eero devices and they often run great deals. I was also keen on keeping this simple.. coming from almost 7 years of using a bulletproof Orbi mesh, simplicity and “it just works” seemed like a no brainer. So it seemed fitting to try. I was very bullish on them given what I heard. I chose the Pro 6E 3-pack variant for our 3,200 sq ft, 3-story home built in 1900.

Eero out of the box seemed like a slam dunk. Easy setup, short of manually migrating all DHCP leases over for 50+ devices. I subscribed to the Eero Plus subscription. It had immediate great performance, seemingly great Wi-Fi coverage, and an “it just works” posture I was hoping for. All three nodes were on wired backhaul.

Unfortunately, the honeymoon period quickly subsided with frequent dropouts on devices, vague issues with their adblock in the Plus tier, roaming issues, and most of all speed issues that seemed to hit after an auto update introduced a confirmed bug (which has since been fixed by Eero). The auto update functionality for me is a deal breaker. Not being able to pin to a specific version was very frustrating.

Even on a good day I had given things up. Eero has no VLANs, so our IoT VLAN went away. It’s managed from the app only, and it bugged me that I couldn’t provision the Eero devices with code. Adding all of that configuration by hand seemed daunting, and I’d rather spend my time elsewhere. DNS filtering went from AdGuard Home down to the basic filtering in Eero Secure, and presence detection (a tracker in LuCI, OpenWrt’s web interface) was gone. The VPN moved to a container on our server, and from the LAN it hairpinned through the Eeros at about 50 Mbps, where I got about 80 Mbps connecting over cell.

The fix came from OpenWrt itself. On April 6, OpenWrt 25.12.2 turned out to fix the M3000’s problem: its mt76 driver fixed a bridge TX bug on the MediaTek mt7981 chip, the same bug behind the Phyn sensor failures. With 25.12.2 on the AP, the Phyn sensors worked through it right away. I used that as the push to put all three OpenWrt devices on the official 25.12.2 build, with AdGuard Home back in front using DNS over HTTPS upstream and DAWN steering clients between the access points. HE160 (160 MHz channels) works now too.

Returned the Eeros. 25.12.2 fixed all issues.

FiOS comes into the Flint 2, which is doing DHCP, AdGuard Home and the VPN, and a switch feeds the two M3000s. I have an IoT VLAN again for all the IoT devices.

The cutover back on April 7 still had some gotchas, mostly DNS and IPv6:

With their WAN unplugged, the Eeros kept sending IPv6 router advertisements. Pulling their power fixed it.
odhcpd wasn’t advertising an RDNSS option (RFC 8106), so iPhones got no IPv6 DNS server at all.
The Global nameserver in the Tailscale admin console still pointed at the dead Eero gateway.
AdGuard Home’s upstream for PTR lookups was hanging, which hung DNS queries along with it.

Before the swap, a power-cycle test of the Flint and one M3000 had shown zero config drift.

The stability I wanted came from trusting open source, lots of debugging, and a lot of trial and error. Rock solid performance and stability is now real with this setup, which costs less than the Eero setup (but definitely not in terms of my time!)

This is the first part of a series of posts about our home network. I’ll talk about some walk tests I did around the house to tune 802.11r fast roaming, how I use Ansible to manage the config of all of these devices, and then a post about AdGuard Home.

Release It! read against operational resilience rules and AI

Thu, 09 Apr 2026 00:00:00 GMT

Since January 2025, banks and insurers in the EU must demonstrate operational resilience under the Digital Operational Resilience Act. The rules require resilience testing and maintaining a register of all ICT third-party dependencies, each with an exit plan.

Michael Nygard was arguing for this kind of thinking long before any regulator. The publisher’s pitch for the 2nd edition of Release It! (2018) was that 80% of a project’s lifecycle cost is in production, and not many books talk about that. He’s also the person who proposed architecture decision records.

Through the lens of a decade working with large financial institutions and Fortune 500 companies, now as a principal cloud architect, I’ll test three ideas from the book against 2026. This will show you where your enterprise should put its next effort into resilience, and what each choice delivers in a live production system.

Life after release

The second edition adds chaos engineering and cloud-native operations to a book already about post-release software. The DORA testing requirement is the same idea, but an obligation the supervisor can verify. Is the evidence the bank gives the supervisor about the same as the resilience of the system they’re actually running?

The missing bulkhead

Timeouts, circuit breakers, and bulkheads are named for service-to-service calls. The idea is that one slow or failing dependency shouldn’t be able to exhaust the threads of all callers. So you put a time limit on each call, and keep a separate pool for each group of callers.

In July 2024, a single faulty security update brought down about 8.5 million Windows machines, grinding airlines and banks to a halt. Breakers between your company’s own services wouldn’t have helped here.. the missing bulkhead was at the vendor level, where one update rolled out to every machine.

To me, this outage was as much a cynical-software failure as it was a failure of testing. A component blindly trusted an update, every machine applied it at the same time, and nothing stood between a single bad file and an entire fleet of machines.

Cynical software meets a model API

The best line from that book hasn’t aged a bit: “Cynical software expects bad things to happen and is never surprised when they do.” Nygard says integration points are the number-one killer of systems, and APIs for models are the newest integration points for most enterprises. Later in 2024, the Financial Stability Board put third-party dependencies and concentration of AI model providers at the top of the list of AI-related financial stability risks.

To me, a model API is just another integration point. It needs a timeout and a circuit breaker with a fallback, or an explicit sign-off that it’s allowed to run without one. Pretending it won’t fail like your database or a partner’s API does is exactly the kind of trusting attitude this book argues against.

You could produce more compliance artifacts (DORA mandates them anyway, so that keeps the supervisor happy), but an entry in a register won’t stop a slow call from exhausting your threads. You could integrate with a second provider, spreading the concentration risk the FSB describes, but now you’re building and testing two integrations instead of one.

Timeouts and breakers on every outbound call are changes you can make in your own code, without negotiating a contract first. You can build fallback options into your model calls, whether that’s a second provider via configuration or a degraded answer. But if you run without either, you need someone accountable to sign for it.

Memento: agent memory I can read, edit and prune

Tue, 07 Apr 2026 00:00:00 GMT

I have used Claude Code almost exclusively as my agentic development tool since my first Max plan subscription in April 2025 and haven’t looked back. Previously using Cline, Roo Code, etc.. I am much more comfortable on the command line. I do use Codex and opencode for various specific cases. Multi model reviews are one of those. I’m a firm believer in the right model for the job, and having a plethora of models available via OpenRouter, OpenAI 1p and Anthropic 1p lets me leverage the best tool for each task I’m working on.

Multi model review using Claude models in conjunction with OpenAI astra or sol is invaluable in development and planning phases, bug fixes, feature development, etc. Whichever tool I’m in.. every new session starts out not knowing anything about the last.

Problem: turns out long term agent memory is a hard problem to solve for. There are many many (many) open source projects (mem0 and claude-mem are two of the bigger ones) and many many approaches that do yield great results for most users. I have tried some claude-mem techniques, and I’ve used mem0-style memory in other setups. I found recalling stale information, or not enough information, was detrimental to my processes.

Solution: I wanted something simple that I could read, edit, prune, and augment easily. I also wanted to follow the best practices outlined by Claude Code product owner Boris Cherny and use fresh sessions a lot (avoid compaction) and stick sessions to a scoped concern (I am still trying to get the hang of this).

As an Obsidian user, markdown handoff documents (full featured summaries that tee up the next session) made so much sense to me. A simple skill of /handoff (and the inverse /onboard) lets me move onto fresh context sessions quickly and easily while not losing pace. It allows me to put a project down for a few days or weeks and onboard right back to where I was in a fresh context.

I worked with Claude Opus 4.6 (my favorite Claude model to date) to build this.

How it works

All memories belong to a project, and the same project resolves to the same id on every machine by design. Memento does this with a fallback chain in compute_project_id (first match wins):

vault://<path relative to the vault>       # the project lives in the vault
git://<sha12 of the origin remote url>     # otherwise, a git repo
path://<sha12 of the $HOME-relative path>  # otherwise, just the folder

I hash origin’s remote url, so the same repo cloned anywhere on any machine maps to the same memory file(s).

I run /handoff at the end of a session. It writes:

a continuation doc stamped with the session end time, tees up the next session
a short forward-looking retro: behavioral observations go to a retro log, each one hashed so repeats get recognized and promoted
task candidates for the technical loose ends (the next /onboard offers to turn these into real tasks)
the carry block: a structured list of what’s still open for that project, diffed against the previous carry

Note: I used to have automation via a SessionEnd hook that mined each transcript with an LLM, but I retired it in June: every run was its own claude -p on the bill, and it kept burning tokens re-synthesizing stuff it already summarized. I bagged this for a more manual approach and it works great for my workflow.

I love the carry block. My ever increasing backlog of ideas, todos, fleeting thoughts about projects I’m working on. I have to have dedicated routine reviews of this but a lot of my best ideas come during the flow where I can just say “backlog this”, or “carry this until next week” and this system helps me “remember” all of this effectively and have provenance to where that came from.the session where each idea first came up.

At session starts, a bootstrap hook reads the last 10 session files and turns them into one-liners: about 150 tokens of index in under 100 ms (file reads only, no model call). The project’s carry block gets injected right alongside it. Claude Code only keeps 10,000 characters of hook output, so the carry has to fit under that or it gets dropped. Anything deeper is a vault-search query on demand (BM25 plus semantic over the vault) and an MCP server handles ad-hoc reads.

Dream cycles and older memories fading out naturally were something I continue to evolve and leverage. Every night at 4 AM a run harvests memory files, uses Haiku to normalize relative dates (so “yesterday” becomes an actual date) and rebuilds the index. Costs about two cents. A weekly run on Sunday at 5 AM looks for patterns across sessions, about 15 cents.

This system is fantastic and it’s cross-machine, cross-environment (Obsidian Sync official handles the sync). Everything is markdown in the vault, config included: defaults first, then a shared layer in the vault, then machine-local overrides. I added memento install codex in June so Codex gets the same memory, and v0.12.0 (the carry block release) runs 628 tests.

Why it stays plain

Having tighter and easier control of context injection and keeping memory pure keeps the agent working more predictably and as a result more autonomously when needed. Durable long term memory needs to be written to locations where you can easily see the rules and edit them. Clean context is the responsibility of the human in the loop.

Clean Code, rewritten: which rules still matter when AI writes code?

Sun, 05 Apr 2026 00:00:00 GMT

A Sonar survey earlier this year of over 1,100 developers found that AI accounts for ~42% of committed code, but 96% of developers don’t fully trust the code generated by AI. This means the purpose of a coding standard changes from describing the code that people write to describing the code that people review, an increasing portion of which is generated by models.

At a time when the most argued-about software book comes back as a full rewrite, the second edition of Clean Code finally reaching readers in late 2025 (17 years after the first) lands in a world very different from 2008. And so it’s worth asking which of those rules still earn their place in your team’s standard when the model writes the first draft.

Robert Martin describes the new edition as a “complete rewrite” that delivers the same message in an entirely different presentation. The announcement post became one of the biggest book threads r/programming has ever had. The book is split into four parts and 37 chapters, includes examples in seven different programming languages (not just Java), and folds in material that used to be in The Clean Coder and Clean Architecture. The two most telling additions about the moment we’re living in: a chapter called “AIs, LLMs, and God Knows What” and an appendix titled “The Clean Code Debate” that takes a softer tone and says it’s okay to disagree.

Critics say the presentation is new but the style is the same. The Axolot’s Don’t Refactor Like Uncle Bob (Second Edition) review praises the use of modern Java and stepwise refactorings, then shows the same habits that drew criticism in the first edition: tiny functions, turning arguments into instance fields so methods can share state without parameters, and treating comments as a last resort.

The longest-running form of this argument is John Ousterhout’s written exchange with Martin about method length and comments. The first edition suggests functions of two to four lines, and Ousterhout argues that such fine decomposition displaces complexity rather than removing it. This argument deserves more space than I have here (more in A Philosophy of Software Design point and counterpoint).

As a principal cloud architect who has worked with large financial institutions and Fortune 500 companies, I see a coding standard as a review contract rather than a style guide. Banks and insurers often use code review as a form of change control; an auditor may ask “who approved this change and could they understand it?”. If a model is the first author, the reviewer is the only human who needs to understand the change.

The area where AI-written code struggles most is readability. CodeRabbit’s analysis of 470 open-source pull requests found that AI-authored code had about 1.7 times as many issues overall, but over 3 times as many readability issues.. the largest gap in their dataset. This typically manifests as code that looks tidy but doesn’t follow local conventions for naming and structure (CodeRabbit’s report). This is exactly the kind of thing a written standard should capture.

Through this lens, the rules in the book fall into two groups. The key question is: which of these rules would a reviewer block a merge for? The first group helps a second reader and can be checked in review: reveal intent with names, tests as repeatable proof (which get their own chapter in the new edition, “Repeatable Proof”). The second group is taste: functions should be two to four lines, don’t write comments. A linter can count the second group, but passing the linter doesn’t tell the reviewer or auditor anything about whether another person could understand the change.

The postscript to Martin’s own rewrite makes this point more sharply than any survey. At the end of Chapter 2, he fed the cleaned-up code to Grok 3. It returned a version that passed his tests but kept an unnecessary instance variable it had picked up from his code. The model copies the style it’s given, so the contents of your repo end up in every AI-written change, whether you enforce them or not. Which rules belong in the standard agents copy and auditors read?

Fowler's Refactoring catalog as a vocabulary for agents

Sat, 04 Apr 2026 00:00:00 GMT

I’ve found that asking an agent to “refactor” a module can lead to unpredictable results.. sometimes it’ll carefully clean up the code, and sometimes it’ll rewrite half the file. This unpredictability made me curious about what the term was intended to mean. For a precise definition, we can turn to Martin Fowler’s Refactoring. The 2nd edition (2018) is mostly a catalog of small named moves (Extract Function, Split Phase, Replace Loop with Pipeline, etc) with motivation and step-by-step mechanics for each one. All of the entries were redone with JavaScript examples for the 2nd edition. The defining property of all of these moves is that they preserve the observable behavior of the code.

Of course, the meaning of the word had drifted long before we had agents.. Back in 2004, Fowler wrote that if your system is broken for a couple of days during a “refactoring”, it’s almost certainly not refactoring. But today, asking an agent to refactor a module will often result in a rewrite, and losing a feature along the way is a common complaint among users of these tools.

I think the names are what survive automation. The mechanics were always going to be handled by tools. A shared vocabulary lets you ask for one change and check that only that change happened. “Split Phase on this function, no behavior change” instructs the agent and tells the reviewer what to look for in the diff: just that move. “Refactor this” doesn’t give you anything to check against. A 2025 study showed that instructions built on Fowler’s catalog let LLMs do every refactoring type in its benchmark while preserving behavior.

That same study found something unexpected: when the model was given the refactoring goal instead of the transformation to apply, it improved code quality more than when it was told exactly what to do. I’m landing on naming the move when I need to review the change, and naming the goal when exploring. For those using agents, the open question seems to be: if goal-led prompts produce better code, how much quality are you willing to trade for changes that are easy to review?

Radical Candor and the nice boss who never says it

Wed, 25 Mar 2026 00:00:00 GMT

It’s a disservice to everyone involved. They say the road to hell is paved with good intentions. I think in the moment you go, “Oh, you know what? They’re not meeting expectations, but let’s just give them another chance.” Or maybe if I wrote the ticket in a little more detailed way, they’d get it right. Or you just kind of go through and you realize, I’m not helping this person grow.

Kim Scott’s Radical Candor describes a grid version of this habit. The axes are “care personally” and “challenge directly”, and the resulting four quadrants are labeled. A boss who cares but never challenges is described in the book as ruinous empathy.

My biggest worry is that it goes unnoticed in the moment because the boss is so likeable, and the engineer only finds out about the missed expectations when they’re called into a termination meeting and told that the “chances” they were given in the previous year were actually conversations that were never had with them.

It’ll eat up a culture in a company from the inside, right? It’s like getting rust on some sort of piece of metal that’s not supposed to be exposed to the elements.

Scott herself said as much in 2015.. most management mistakes land in this quadrant.

Given working at a place that is too polite or too combative, I choose too polite. It’s really exhausting when everything becomes an argument. But you got to be careful that you don’t become ruinously empathetic.

Okay, you got me. Politeness is a totally sound default, which is why we drift that way. Each talk we avoid seems so kind by itself, and it feels kind until the person finds out how long everyone else knew.

I think radical candor kind of has to flow from the top down. And by flowing from the top down, sometimes you enable an environment where it can flow from the bottom up.

The book’s entry point is the boss. The leader asks for criticism before giving any, and when it starts coming in, listens without trying to rebut. (If the person at the top can’t take it, subordinates won’t risk offering it; they’ll see through the invitation.)

You can’t read this book in a weekend, decide that you’re going to adopt radical candor and then change the culture of the company. This will take work. This is one of those things where you can’t just cargo cult this. You can’t just say we declare that we’re doing this.

So if you can’t just declare it, what’s the first visible thing you do on Monday? That action is what signals to your team it’s real.

Will Larson's Staff Engineer and the authority you borrow

Tue, 24 Mar 2026 00:00:00 GMT

The title isn’t going to give you the authority. The track that you’re on is not going to give you the authority. You actually have to earn this authority from the bottom up. And I would say that the best CTOs and CEOs, yes, they have the title, but they also legitimately have earned it.

Staff Engineer: Leadership beyond the management track by Will Larson (2021) is about the IC career path beyond the senior engineer role. It’s split into two parts: the first part contains guides to the work of the role, and the second part contains interviews with staff engineers about how the role plays out in practice at their employers.

Larson’s guide to staying aligned with authority is candid about the source of that authority: it’s borrowed (typically from your direct manager), and the power granted to you can be revoked. That’s reasonable! Your title gets you a seat at the table, but your boss determines how much that seat weighs on any given day.

As a staff engineer, yes, you have some authority, but most of your authority, almost all of it, is derived from the managers. And to be an effective leader, you actually have to be a more effective follower.

Emulating this in a place where you don’t have authority actually has a lot of benefits, right? A staff engineer doesn’t hire and fire folks typically, so therefore you’re not in this sidestep of, oh, if I interact with this person, are they going to fire me?

But there’s a real freedom, even though you have a lender, right? People will tell you things as a staff engineer that they would never tell you if you were a director, because there’s no performance review on the line.

There’s an amount of fight picking you can do in private to hopefully change their opinions or determine direction. There is zero fight picking you can do in public because, especially as an individual contributor, you need to be aligned completely with what they do. That’s where you derive all of your authority. And so if at a certain point you don’t trust the decisions they’re making, it’s just game over.

You can dissent in private, but the cost of keeping the loan is public alignment. This idea raises the question: if your authority is borrowed from a single manager, what’s it worth when that manager leaves this week?

Automating Reddit saved content into my personal knowledge base

Mon, 23 Mar 2026 00:00:00 GMT

I’m a big Reddit user. I moderate subs and am active in many communities. Coming from the early days of BBS and later IRC, I love community engagement in these types of arenas. I find engaging with subject matter experts fascinating and it fuels my creative sparks. I learn so much this way.

One thing I had become frustrated with is reading a post or comment and then later seeing it deleted by a user or moderator or edited heavily.. losing the content I was looking for. I always used the save feature in Reddit as I read, as a “read it later” type of workflow. But.. ugh.. the “saved” interface on Reddit has no search, and old saves are unreachable past 1,000. I am sure many users would agree. It is nearly impossible to find older things you saved (recall, search, etc).

So in addition to saving posts, I started using Obsidian Web Clipper on some posts, as I actually already had done that for blogs and other web content to save in my vault (I’m a big Obsidian user). This worked as a snapshot, but I wondered what could I do that leveled this up a bit?

What if I could automate this? What if I could capture even deltas and changes and view them later? Search across everything. Point an LLM at it and use it as a deep knowledge base. Automate routine updates to saved posts, doing this for every post I’ve ever saved since the inception of Reddit. Analyzing how my saved posts trend, seeing my interests on a timeline, being able to have a personal AI agent see and interact with this context. This is how Reddit Stash was born.

Pulling every save out of Reddit

To get all of the posts that I have ever saved across my multiple Reddit accounts, I used Reddit’s GDPR export and some PRAW to get around the 1000 post limit that Reddit imposes on their API.

Each save is put into my vault as a markdown note with frontmatter. An LLM reads the post and writes the tags and a short summary (I have a couple free tier LLM subscriptions that let me use claude haiku for this and it’s been working great). The prompt instructs the LLM with a master tag list and keywords to choose from so that the same topics get the same tags across all notes. (example below, trimmed and with made up values)

---
sub: examplesub
title: "Which burr grinder for pour over under $200?"
type: saved_post
author: example_user
url: "https://reddit.com/r/examplesub/comments/abc123/which_burr_grinder/"
time: 2019-04-12 08:15:00
num_comments: 47
score: 112
tags:
  - discussion
keywords:
  - coffee
  - burr-grinders
  - pour-over
---

I also set the timestamps of each file to the date of the Reddit post. This allows me to sort by date in Obsidian, so that the notes appear in the order that they were posted on Reddit. This is what allows for the “timeline” idea to work.

The whole thing runs in a container on my Linux server every 2 hours, so new saves show up in the vault without me needing to do anything.

My Reddit saves are like a live activity stream of my interests over time. It is one of the most helpful resources when deep in my hobbies to have this ever growing archive of knowledge to mine from, and to be able to visually see how my interests and hobbies evolve over time by analyzing the save trends.

More details are coming soon about this and tangential projects as I’m really having fun with these and see a ton of value in how I leverage these day to day.

Sandi Metz's 99 Bottles of OOP and knowing when to stop

Thu, 12 Mar 2026 00:00:00 GMT

I just find myself thinking, geez Louise, you are complicating this simple problem like crazy, because then you start getting into things like, well, why are we hard coding bottle, because it might not always be bottles. It might be a flask. It might be a jug.

This is 99 Bottles of OOP by Sandi Metz and her co-authors Katrina Owen and TJ Stankus. The whole book is basically one extended refactoring of a single program that prints out the “99 Bottles of Beer” song.

One of the best pages early in the book is where they show the Shameless Green.. the straightforward version with a case statement and the verses. It doesn’t look very exciting, but it’s easy for anyone to read. It’s the right answer for a problem of this size, and the book makes sure to demonstrate that before going on to argue why we should go beyond it.

If you literally just need down and dirty, this was what’s been asked of you, Shameless Green is the correct answer, and I do agree with that.

But the book doesn’t stop there. It then goes on to talk about how you can avoid using conditionals by using the Liskov Substitution Principle, and it deconstructs the plain version of the song to make it adhere to this rule, even though it never needed to in the first place.

There’s a pathological avoidance of conditionals in this code. And again, this is an ideological thing.

I think this is where ideological zealotry can get you really twisted up into knots. A principle should serve its purpose. If a principle stops serving its purpose and it’s just a way for you to take what you’re doing and defend it without additional thought, I think you end up getting into these weird spots.

The weird thing about this is that Metz wrote an essay called The Wrong Abstraction where she talks about how duplication is way cheaper than the wrong abstraction. And getting ready to sing a song about beer for the day when a bottle becomes a jug is basically as clear an example of the wrong abstraction as I can imagine.

I think my last criticism is this book is missing the when to stop lesson, which is that just because I could doesn’t mean I should.

The back of the book promises that I’ll learn to recognize when code is “good enough”. But what would the stop sign look like in a code review?

Escaping the Build Trap when building costs almost nothing

Tue, 10 Mar 2026 00:00:00 GMT

At Y Combinator’s AI Startup School talk last summer, Andrew Ng argued that AI coding tools make engineers go much faster, but that deciding what to build doesn’t go any faster, so now product management is the bottleneck. One of his teams proposed a ratio of 1 PM per half an engineer (instead of the normal ratio of 1 PM : 4 engineers).

He also said an experienced product manager’s gut is a surprisingly good substitute for data when you have to decide quickly.

This made me think of a book. I’ve been on both sides of the backlog: first as a software developer, and later as a software development manager. On a lot of teams, the scoreboard is the number of features shipped. Melissa Perri, in Escaping the Build Trap (O’Reilly, 2018), calls this the build trap: when an organization measures its success by output (features shipped) and loses sight of the outcomes (the customer value those features create).

This idea predates the book.. the original post from 2014 describes the build trap. Perri writes, “Building is the easy part of the product development process. Figuring out what to build and how we are going to build it is the hard part.” She suggests that we stop gauging productivity based on output and instead reward teams for learning more about their customers through research and experiments.

I think what Ng’s talk represents is Perri’s line finally coming true.. building is now actually easy. But I don’t think that means the build trap goes away. I think if more product managers make decisions based on their gut, that’s still a build trap if the team keeps measuring how many things they shipped. I think it makes it easier to get into the build trap, because if you have a coding assistant that will just build the next feature you ask it to every time, and never question whether that feature should exist in the first place, then even though the output is cheaper to produce, if you’re still measuring the team the same way, they’ll just make more unnecessary work, but faster.

But cheapness is also the escape. Perri described the escape years before AI made building cheap. She said treat every solution like an experiment in service of an outcome. Experiments are easier if building a working prototype costs a day instead of a quarter. So for every cheap build, the question is: how will we know it worked? What outcome should this move, and what will we do if it doesn’t? Otherwise, we’ll just be efficiently ineffective, shipping fast toward nothing in particular.

One question remains, which Perri’s framing from 2014 doesn’t need to answer. If building is near-free, and any idea can become a working feature by the end of the week, is the hard part still deciding what to build? Or is it deciding what to delete?

The DevOps Handbook and the case studies where it always works

Sat, 28 Feb 2026 00:00:00 GMT

I think DevOps has a bad habit of overselling what it can do.

The DevOps Handbook is a non-fiction companion to The Phoenix Project (the novel that inspired The Phoenix Project and the work nobody planned), written by Gene Kim, Jez Humble, Patrick Debois, John Willis, and Nicole Forsgren. It’s structured around the Three Ways (flow, feedback, and continual learning), which are based on Lean and Agile. The second edition (IT Revolution, 2021) includes new case studies from adidas, American Airlines, Fannie Mae, Target, and the US Air Force. I think the ideas in the book are great.. I wouldn’t argue with any of the Three Ways.

The ideas seem fine.. I’d poke at the evidence. The case studies all share a common pattern: a big organization that was stuck, a 2-3 year transformation, and now they ship faster and their teams are calmer. There isn’t a single case study where things were worse at the end than they were at the beginning.

Instead, it’s almost like apocryphal stories about how DevOps always fixes every problem.

It’s a map made only from those who made it to the destination, and it’s silent about the routes where teams got lost or quietly quit. But it’s along those routes that most of the valuable warnings are found.

So you hire a DevOps team because of the handbook, and two years later you end up with a new silo with a fancier name. Sound familiar? This frustration is common among small and mid-sized companies that have fully committed to a dedicated DevOps team.

They’ve invested in something that itself becomes overly complicated and then there’s a lot of tech debt and then there’s a lot of resentment and you get back into this old dev versus ops situation, but now you have this very expensive DevOps team versus what you probably had before.

But you’re right back up against that wall again. The whole point of this book is to tear down the wall between dev and ops. If you put a team in place and call them DevOps, they can quickly become the newest, most expensive part of that wall.

What would a failed DevOps case study teach us that success stories can’t? Would any company be willing to write one up so others could read it before repeating the move?

What AI Engineering looks like to a model risk team

Thu, 26 Feb 2026 00:00:00 GMT

With foundation models, the demo is now the easy part. Any team can connect a hosted model to a prompt and produce a convincing result in a week or so. But most large organizations still struggle to turn that demo into a production system. Chip Huyen’s AI Engineering (O’Reilly, January 2025) focuses squarely on that gap. The book defines the job as adapting models trained by others, and its companion repo of chapter summaries has earned more than 18,000 GitHub stars.

One year after publication, the underlying question remains: is this a new discipline, or is it software engineering plus a dependency that just gives you different answers every time you ask? Against the backdrop of a large regulated enterprise, several hold up. More importantly, each lands on an existing governance function.

Evals are where the work is

Evaluation gets 2 whole chapters, so it’s clear where the book sees the work. Open-ended outputs don’t have a single correct answer, so the book suggests exact checks where the outputs can be verified, and AI judges where they can’t.. both in a per-application evaluation pipeline. In a companion post on common pitfalls when building generative AI applications, Huyen cautions against abandoning human review, suggesting that experts review a daily sample of 30 to 1,000 examples.

In a bank, this pipeline looks a lot like what a model validator would ask for. SR 11-7 is the Federal Reserve and OCC’s 2011 guidance requiring effective challenge of every model, including outcomes analysis of real performance. A paper on model risk management for generative AI in financial institutions notes that these models bring additional risks like hallucination and toxicity, and validation testing needs to expand to cover that. Banks are hiring: Evident’s 2025 index shows 42 of the 50 large banks it tracks have a senior responsible-AI lead, up from 33 the year before.

Start simple, finish slow

The second idea is the least glamorous piece of advice in the book: try prompting first, then retrieval, then finetuning.. don’t start with frameworks and vector databases. The number that stuck with me from that pitfalls post: LinkedIn got to 80% of the experience they wanted in 1 month, but it took 4 more months to get to more than 95%.

This curve is the enterprise production gap, just in small form. McKinsey’s state of AI survey says 88% of companies use AI in at least one function, but only a third have started to scale it, and most attribute less than 5% of EBIT to AI. The a16z enterprise CIO survey shows finetuning declining as prompting and longer context windows improve, which supports the book’s recommended order.

What you build around the model

Inference optimization gets its own chapter, and the final chapter shows how to build an application around the model layer by layer. Context first, guardrails on input and output, a router and gateway before the models, caching and monitoring after, and user feedback to close the loop. When you apply this to a regulated enterprise, most of the layers already have an owner.

The gateway is where a regulated enterprise will want to put controls and FinOps. That same a16z survey found that 37% of enterprises are running 5 or more models in production, and cost is weighing more heavily in model choice. Menlo Ventures estimates enterprise spending on generative AI at $37 billion in 2025, about triple the prior year. Gartner says over 40% of agentic AI projects will be canceled by the end of 2027 due to cost, unclear business value, and weak risk controls. If you’re running 5+ models, every model swap is a change that needs re-validation.

The book reads like software and systems engineering applied to a dependency that isn’t deterministic. It treats evals and the gateway as engineering hygiene, which is exactly what regulated firms will turn into controls. SR 11-7 assumed models built and specified by the bank, but pipeline models are third-party and may answer the same prompt differently each time. This means the eval pipeline may become the validation artifact.

Worse is Better and deciding which simplicity to pay for

Wed, 18 Feb 2026 00:00:00 GMT

All of us are in this push and pull between, oh, one day this thing might be used by a million users, and so therefore I should build the perfect abstraction, because perfectly modeling how I think that this thing could be used one day is the right way to do this.

The attraction towards worse-is-better has been around for a long time.. in 1990, Richard Gabriel (a Lisp person) wrote an essay trying to explain why Unix and C were winning and Lisp (“the right thing”) was losing. That essay was The Rise of Worse is Better, which splits the field into two camps: MIT and New Jersey.

The New Jersey camp (early Unix, C) made one specific trade: the essay says “implementation simplicity was more important than interface simplicity.” The MIT camp paid the opposite price, keeping the interface simple no matter what it cost to implement.

If you’re trying to gain customers and you want to make it simpler than your competition, you probably need to do the right thing from a simplicity standpoint. That’s where that extra effort comes in. If you need to just down and dirty get scrappy, the simplicity of implementation is probably more important.

These are two kinds of simplicity, and they charge it to different parties. Simple implementation means it’s cheap for the builder, easy to ship and extend. Simple interface means it’s cheap for the user, costly for the implementer. Which does this project pay for? I think you should say that out loud before you write any code. Saying one thing or the other leads to very different products.

Just because it’s worse doesn’t make it better. Just because you don’t think about some of these details doesn’t mean that you shouldn’t think about it. You can actually make worse code that no one will ever use or want or is so poorly designed it can’t be maintained. So this is not an excuse to just have poor craftsmanship.

That said, if you’re going to build a scrappy version, it still has to be good enough that people will want to use it and that someone else can maintain it after you. So “Worse” in this case is more about the scope of what you build. You still have to care about what you’re doing.

You can optimize for a set of trade-offs and you say, look, if we can move fast and keep it simpler, that’s better than being complete and consistent and correct. And really what worse is better is all about is valuing outcomes over process.

Gabriel never resolved the question himself, which I find oddly reassuring. He spent a decade arguing with his own essay, including writing a rebuttal under the pseudonym Nickieben Bourbaki. In his own history of the idea, he writes “I still can’t decide” and the page ends “Decide for yourselves.” Which is of course the real answer. The call is yours to make for your product, and to make again for the next one.

The one Tidy First? habit that makes AI changes reviewable

Mon, 09 Feb 2026 00:00:00 GMT

Kent Beck, in Tidy First? (O’Reilly, 2023), a short book with a question in the title, offers an idea worth taking away. Every time we change code, we change either its structure or its behavior. Beck says these two kinds of changes belong in separate commits or separate pull requests. He designs around that line: “Software design is preparation for change; change of behavior.”

Beck’s argument for splitting is economic: behavior change makes you money now; structure change buys options on the next change. Most software cost is the cost of changing it, and coupling makes that cost go up. A structure-only change is small and easy to reverse. He suggests structure-only changes (“tidyings”) merit lighter review, since review is a fixed cost that deters tidying at all.

I think this idea of a split is the most useful idea in this book for people who are under change control. I’ve spent 10 years working with large financial institutions and Fortune 500 companies, now as a principal cloud architect. In regulated shops, change boards and approvers want to know what you’re doing and why. A big-bang refactor rarely makes it through the approval process. A mixed diff can’t be confidently signed off. A diff that provably makes no behavior change is the easiest to approve. Getting in the habit of shipping such small changes is how modernization continues to happen between big programs.

AI makes this split even more important. DORA’s research links more AI adoption to a 7.2% drop in delivery stability, likely because of larger batch sizes. GitClear found that for the first time in the latest year measured, copy/pasted code exceeded moved code, suggesting skipped tidying. I’d argue this rule applies to agents too: ask for two diffs.. structure first, behavior second. Verifying the first changes nothing is much faster than untangling mixed change. If an agent can tidy for nearly free, is “tidy first?” even worth asking anymore?

Building Evolutionary Architectures: are fitness functions just tests?

Thu, 05 Feb 2026 00:00:00 GMT

Is a fitness function just a test by another name? Building Evolutionary Architectures by Neal Ford, Rebecca Parsons, Patrick Kua, and Pramod Sadalage revolves around this single term. The 2nd edition (2022) is subtitled Automated Software Governance. The book defines a fitness function as something that “provides an objective integrity assessment of some architectural characteristic(s).” A skeptical view might be that this is just testing, repackaged for consultants. The book kind of agrees.. it says that a fitness function can be a unit test, a metric or a monitor.

The difference is what the check is aimed at. In a Thoughtworks podcast episode on architecture as code, Ford describes them as unit-test-like checks aimed at architecture. He says most aren’t written in the application’s language, but instead are usually a pipeline job or a production alert. For example, you might have a build that fails when two packages create a dependency cycle. That’s not usually written as a behavior test (no user will ever notice the cycle, but the architecture will notice it a year later when you can’t change the packages separately). I think the “these are just tests” people are right about how it works, but wrong about what it’s doing. The check itself isn’t new, but what it’s checking is.

Every pull request to my site has to score at least 90 on Lighthouse in every category. That’s a fitness function in all but name.. and I think the name earns its keep. If a characteristic has a name, it has a threshold. If it has a threshold, it can fail the build. If it only exists as a principle, it has no threshold, you argue about it in a review meeting, and you don’t check it on every change. What characteristic in your system is only checked in someone’s head?

Hypermedia Systems and the complexity budget you can't split

Wed, 04 Feb 2026 00:00:00 GMT

I find myself more interested in the argument of Hypermedia Systems than the tool they’re selling. The book (by Carson Gross, Adam Stepinski, and Deniz Akşimşek) is available online for free, and makes the case that we should be sending HTML over the wire, doing all the thinking on the server, and keeping the browser thin and mostly responsible for rendering the response.

The underlying idea is older, and (to me) the best part: Gross talks about a complexity budget in this essay, the idea that every project spends its complexity somewhere, so you should spend it where the application earns its keep. This is an idea that stands on its own, independent of htmx, and I think it would survive even if htmx itself disappeared tomorrow.

Imagine an internal admin tool owned by a back-end team, full of forms and tables and a search box. The budget argument is strongest here: keep the complexity on the familiar server, keep the front end thin and un-needy.

The book is candid that this approach isn’t for everyone. It concedes that some highly interactive applications may require a single-page framework, and offers Alpine.js and hyperscript as scripting add-ons for extra browser behavior. That’s a fair concession, but it enables my biggest worry: a hybrid setup where a team keeps React for the dashboard but uses htmx for settings pages.

The complexity budget argument goes out the window, because even if the subsection of the app is theoretically simpler than it could be in React, the overhead of switching in between two frameworks in the same project is just going to wipe out any gains you make from making one half simpler.

So instead of saving your budget by only having to spend it once, you’ve spent it twice because you have two mental models and two sets of conventions to keep straight. I suspect most teams that try htmx will end up in this half-and-half state, which means that the best idea in the book is the one that’s hardest to actually benefit from where people are testing it out, since teams rarely start with a blank slate.. React is usually already there.

React is the incumbent, HTMX is the upstart. When you’re the upstart, you need to offer some really compelling reason to switch away from React.

But the complexity budget itself raises a question: is “simpler” by itself reason enough to leave React? Or does htmx, despite its good ideas, only really win when there was no React to replace in the first place?

The Agile Manifesto won and became the process it fought

Thu, 22 Jan 2026 00:00:00 GMT

I think the Agile Manifesto has been completely a victim of its own success. This was absolutely a contrarian movement when it started. From what I understand about how software used to be developed, everything about Agile seems to be an improvement over it, from these massive waterfall projects.

The Manifesto for Agile Software Development is a single page. It was written by 17 people in a ski lodge in Snowbird, Utah, in February 2001. It contains 4 pairs of values, the first of which is “Individuals and interactions over processes and tools.” The entire argument is contained in the single line beneath them: “While there is value in the items on the right, we value the items on the left more.”

I think like all revolutionaries, they’re very good at critique and very, I would say, debatable on implementation of their utopian future.

This kind of fits.. The manifesto is a critique of how we build software, and it’s a good critique. But nobody at the lodge has documented how a large company with hundreds of engineers actually runs this on a Monday morning.

People say true Agile has never been tried. There is some truth to that, but there’s also truth to the idea that if it hasn’t been tried yet and it’s been 20 years and everyone and their dog says that they use Agile, then maybe there is no such thing as true Agile.

If everyone claims it and almost nobody does it, I think the definition is the problem.

Let’s say you have a bad release that brings down production on a Friday. Your company decides to put a change board in place that approves every deploy. A year later, you have a sign-off form attached to the board. Nobody on your team remembers which outage that form was supposed to prevent.

The organization gets burned some point in its past, and they make a rule so that they can protect themselves from getting hurt. But that rule actually gets in the way of the fact that you’re now harming all of the autonomous and self-actualized software engineers that could actually be trusted.

This is where the “processes and tools” side gets back in charge, but with an Agile badge. People inside the movement saw this coming within ten years. After the Snowbird reunion, Philippe Kruchten wrote Agile’s Teenage Crisis?, describing how Agile had become like a teenager adopting fads and jargon, and offering criticisms including how commercial interests can mask failure. It reads as the movement running a retrospective on itself.

The rebels won. The rebels overthrew the empire and are in some ways the empire now.

And once the rebellion becomes the rule book, I always wonder what a team needs to get back from it. Do we need a new manifesto, or do we just need the nerve to delete the one rule that’s still guarding a wound that healed years ago?

The Twelve-Factor App and the staging environment finance wants to cut

Mon, 19 Jan 2026 00:00:00 GMT

I don’t like when we write code and throw it over the fence to the ops engineers. In my opinion, every developer is an ops engineer. I hear people say, oh, management is cheap, they just want people who can do front end, back end, and devops, and all of this at the same time. And I think there’s trade-offs in everything we do. I also think part of your job as an engineer is understanding how what you wrote actually gets served and used by the public. And if it breaks, how do you fix it?

That fence has a name: the personnel gap. It’s from The Twelve-Factor App, a methodology put together by some Heroku engineers in 2011 for building SaaS apps.

This has become in a lot of ways the sane defaults. We talk about best practices, but this is the sane defaults.

Putting all your configuration in environment variables and writing logs to stdout is just what people do when they run containers now, it’s not called a methodology anymore. There’s still the dev/prod parity thing to negotiate, which asks that you keep your environments as similar as possible and close the time/personnel/tools gaps between your laptop and production.

When you’re looking at your cloud bill every quarter and wondering why staging is such a big line item when it’s never served a single customer, you have some choices. You could shrink it down to run on cheaper instances, keep it the same size as production, or get rid of it entirely and just test in production behind feature flags.

Sometimes it is cost prohibitive to have a dev or a staging environment that’s in parity with production. It’s typically far less expensive than the actual engineering staff that is having to deal with the fact that emerging problems come up, or the potential cost of outages.

The bill for staging is easy to see (it’s monthly, and it has a number).. the incident it prevents never has one.

There are now ways that you can kind of cheat production parity in that maybe your system is so big and so complex and there’s no way to actually reproduce a dev or a staging version of it, but there’s clever ways that you can actually honor this while not having to actually double your spend.

You can use feature flags, canary releases where you send a small slice of real traffic to your new code before you roll it out, but the worst thing you can do is to shrink your staging environment until it’s a different system.

Before I say something’s a production release, I should test it on the equivalent of production hardware.

If your staging environment has drifted so far from production that it quietly fails the standard, deploys keep passing through it.. yet the release only meets real production hardware for the first time on release day. Is anyone still testing there, or is it just a ritual with a line item?

Cloud Strategy prices lock-in instead of fearing it

Thu, 08 Jan 2026 00:00:00 GMT

We usually talk about lock-in as something to be avoided, full stop. But one idea I think is worth keeping from Cloud Strategy by Gregor Hohpe is that it shouldn’t always be treated that way.

That book (Leanpub, 2020, still being updated) describes itself as a decision-based approach to cloud migration. The chapter “Don’t Get Locked Up Into Avoiding Lock-In” was first published as an essay on martinfowler.com. It argues that lock-in comes in degrees and has a price. There are many kinds of lock-in: vendor, product, version, architecture, platform, skills, legal, and mental. Open source mostly addresses the vendor kind, but leaves the others untouched.

This model prices both sides of the decision: the likelihood and cost of switching, and the unique utility of the locked-in choice that you forfeit by staying generic. When the utility is high and the switching cost is high, lock-in is something you may want to accept deliberately. When the utility is low and the switching cost is high, you need to look hard. Avoiding lock-in has its own costs: the effort to build the abstraction and the new dependency on the layer you own. Hohpe says complexity may be the biggest cost. “Minimizing switching costs may not be the most economical choice.”

I think the most common overpayment is the speculative abstraction layer for a migration that never comes. You pay for it every day, and it only pays off on the day (if ever) you migrate. My other view is that the lock-in that nobody puts a price on is skills and organization. Choosing a platform for its portability is still lock-in, to the platform and its experts. That reframing prompted me to ask: how many portability decisions have I priced, and how many have I assumed come for free?

This turns a cloud migration back into a decision problem, with technology choice following decisions. For each component, ask what lock-in you accept and why. Record it while it’s still a choice and the switching cost is easy to estimate. One last question: Does lock-in that you drifted into ever get priced before you need to exit?

Slow Productivity and the work that keeps making more work

Wed, 07 Jan 2026 00:00:00 GMT

You realize that blue green deploy really means the way that we structure our repos has to change. Oh, and actually the build process for the machine images has to change, and oh, we have all these dependencies that have to get updated for us to update the build process, and then all of a sudden you get this task list that is nothing to do with blue green but everything to do with blue green.

Cal Newport calls that the task engine in Slow Productivity, his 2024 book aimed at knowledge workers. The three principles of slow productivity are: do fewer things, work at a natural pace, and obsess over quality. The task engine is part of the first principle.

Newport gives the example of having to choose between two jobs: research a hard topic and write it up, or organize a one-day conference (like a team on-site). Most people would pick the event, because it seems easier.

Organizing the onsite is going to continue generating task after task after task, because you’re going to have to organize schedules, figure out hotels, talk to caterers, come up with an agenda, clear the agenda with other members of the team, things like that. It’s just going to keep generating tasks.

And then you just hand over the document and you’re done. And the event still totally wins because all of the individual tasks are small and you can just do them before lunch and feel productive the whole time.

I have a paper trail of my productivity, but you’re like, did you do something meaningful? Did you actually do something you’re excited about working on? Is it gonna move the needle on the business at the company? And that’s a much harder standard to hold yourself to.

Newport’s term for this paper trail is “pseudo-productivity”.. visible activity that stands in for real productive effort. This is why it’s so hard to stop doing busy work: it looks like progress.

The fifth most important thing in your day is the greatest enemy to the first most important thing.

The task engine is usually the fifth priority, but it doesn’t seem that way until it eats into the week you were supposed to spend on the first priority.

Slow productivity is not the antithesis to accomplishing big cool stuff. Cal Newport argues it is the way to accomplish big cool stuff.

Newport floated the idea of slow before, in a 2021 essay on the anti-busyness revolution. But the task engine only helps once you can see it. So how do you spot the task engine before you agree to it?

Looks Good to Me and the case for writing down review rules

Mon, 05 Jan 2026 00:00:00 GMT

You put up a big PR that touches most of your service. Someone on your team approves it a few minutes after you put it up with the comment “LGTM”. There is no way to read the diff in a few minutes, so you kind of doubt that they actually reviewed it. But you can’t really say that it wasn’t a review, because nobody documented what the purpose of a review is.

This is where Adrienne Braganza’s Looks Good to Me starts. She says that every team has a review process, whether it’s designed or not. The difference is that it’s not a process you chose, and you have nothing to point to when a defect slips through.

The real decision is what to do about the process you already have. Do you leave it unwritten so people keep guessing? Do you implement a lot of heavy policy (mandatory checklists and sign-offs)? Or do you write a short team working agreement that the team owns and can edit as it learns?

I think there are some people who kind of take a badge of honor in not having any sort of process. And a lot of times you just need process.

For teams who think they’re too fast for process: I’d argue that a working agreement is the lightweight option. The heavy option is unwritten process, where every reviewer has their own private set of rules and disputes start by arguing about whose rules apply. An agreement can delegate nitpicks to linters/formatters so you can spend review time on intent.

The unwritten cost is ambiguity. Who can block a merge? When is it okay to self-approve a PR (overnight outage)? Braganza says the agreement should anticipate emergencies and edge cases. She also describes it as a living document that evolves with the team.

Having repeatable processes that anyone can step in and do doesn’t mean that you’re replaceable. It just means that the organization has reduced their risk.

If you write down the rules, then the one person who knows them can actually go on vacation. Let’s say you have a rule that’s intentionally bad, like “in all disagreements, the developer always wins over the architect”. I would argue about that in the first meeting, probably at length, but at least the whole team would know where they stood.

I would argue that that’s dysfunctional, but at least it wouldn’t be ambiguous.

The open edge is that it’s only as good as the last time the team got around to revising it. If it’s a living document that doesn’t get edited, it’ll turn into an unwritten process plus a file, and you’ll go back to quickly LGTMing. What makes a team want to reopen it? Is it a bad incident that got through review, or is it a new hire asking why things are done this way?

The Phoenix Project and the work nobody planned

Fri, 19 Dec 2025 00:00:00 GMT

There’s a sysadmin thread where people were arguing that The Phoenix Project was a book all IT should read. Then there’s a DevOps thread a couple years later where people were arguing that it was a terrible book with insufferable characters and dialogue heavy on corporate lingo. The interesting thing to me was that both threads were about the same novel, and the people arguing about it did roughly the same job.

The Phoenix Project is a business novel by Gene Kim, Kevin Behr, and George Spafford, published in January 2013. Bill Palmer becomes the VP of IT Operations at Parts Unlimited, an auto parts company whose survival depends on the late and troubled project Phoenix. With the help of his mentor Erik, he learns ideas adapted from manufacturing. It gradually becomes clear that every outage and urgent fix goes through one engineer, Brent.

The people who hate this novel are totally right about the characters and the lingo, but I don’t think that matters too much. It’s not for engineers who are living this every week. But as an allegory? It’s great. The audience isn’t the engineers, it’s the people who work next to the engineers and don’t understand why the team is always behind.

The idea that I think survives the eye-rolls is that there are four types of work: business projects, internal IT projects, changes, and unplanned work. These are laid out in IT Revolution’s summary. Bill can name the first three types pretty quickly, but the fourth one takes him the longest. The summary says that this type of work “is highly disruptive and prevents IT from focusing on strategic initiatives.” Unplanned work is still work.. it takes real hours from real people, but since it’s not on anybody’s list, it makes the team look slow to the people watching the list.

If I were picking one book for a non-engineer who couldn’t figure out where the team’s time was going, the obvious alternative is the non-fiction. The DevOps Handbook and Accelerate describe the same ideas in a more precise way and with a lot less melodrama, but I wouldn’t give those to this person. A reference book requires a patience that someone who’s never been paged doesn’t often have, and they’d probably stop reading after the first chapter.

I’d give them the novel. The characters are caricatures, but they’re caricatures of people you’ve met, which helps the four types of work stick. It’s written as a retelling of Goldratt’s The Goal, but with Brent as the constraint. The person who’s never carried a pager will recognize Brent faster than they will a definition of a bottleneck, and then they’ll notice the queue of interruptions in front of him.

You can reframe the eye-rolls.. engineers know unplanned work exists, so the story’s neat fixes feel too easy. But at least now they have a name for it that others will accept. For any team, the question is whether the unplanned work is on the board next to the planned work, or whether it’s only in the heads of the people getting paged about it.

The Clean Coder and when saying no isn't enough

Tue, 16 Dec 2025 00:00:00 GMT

I do fear if you read this book really early in your career, you could take it as gospel, and that if you take this advice without nuance, you may not understand and appreciate where he came from.

The Clean Coder by Robert C. Martin (2011) is a code of conduct for professional programmers. It talks about what to do when you have tough deadlines and your manager is asking you to do something unreasonable. It has a very loud chapter about knowing when to say no and how to phrase it.

Your manager wants you to build a feature by tomorrow, but it’s impossible. “I’ll try” sounds helpful, but it’s a promise you know you’ll break. “No” is honest, but leaves your manager empty-handed. The book says: say no, but offer a smaller deliverable you can do by tomorrow. I like that.

You can’t just show up and have all the fun parts of being a professional and say no to your manager and push back on deadlines and redefine requirements and things like that without doing all the other things that make you a professional.

But there’s a catch, and it’s important to remember: that no needs to be earned. Martin’s later Programmer’s Oath says something similar from another angle: be honest with your estimates, don’t make promises you can’t be certain of keeping.

I consider a professional organization one that doesn’t throw someone under the bus or put too much pressure on the very bottom rungs, and there wasn’t an emphasis of that in this book. And I don’t think that’s what the purpose of this book was.

Blameless postmortems have a rationale: when something goes wrong, it’s useful to ask which systemic factor allowed it to happen. But this book, addressed to individual programmers, never really asks that question.

You should hold yourself to this high standard, and again, I agree with that. But putting all of that on you and saying that if the organization fails it’s because you’re the problem, that’s the part where I was like, I don’t know.

You can’t fix a workplace that punishes honest estimates with an individual code of conduct. You can tell them the real number, offer to deliver something smaller, and still be told to try harder. Then you get blamed anyway when the deadline slips.

I wish there was a section here that was elaborated more on when to walk away. There are some environments in which you can’t thrive.

This is something I still don’t know how to do. Some workplaces make it hard to say no, but you can get better at it with practice. Others will never accept it, no matter how honest your estimate. How can you tell the difference before spending years finding out the hard way?

The Practice of Programming and the style fight nobody should win

Mon, 08 Dec 2025 00:00:00 GMT

If you get a pull request back with 30 comments and most of them are about where the brace goes, or that some lines are more than 80 characters, or whatever, it’s just a waste of everyone’s time. Kernighan and Pike tried to put an end to this nonsense in 1999 with The Practice of Programming, but I think the “preferably ours” is the most honest part of the sentence “Pick one style, preferably ours, use it consistently, and don’t waste time arguing.”

People took this advice literally. gofmt is one style, built into the Go toolchain. One of the Go proverbs from Rob Pike is “Gofmt’s style is no one’s favorite, yet gofmt is everyone’s favorite.”

If somebody just makes an opinionated choice, throws it into the build pipelines and set expectations, all of a sudden you’re just thinking about the code you’re writing and you’re not fighting over 80 columns versus 120 columns.

Arguing about style is the cheapest fight to lose, and I’d rather lose it to a tool. It also changes who has to deliver the bad news in the review.

The linter is fair, the linter is dispassionate, and the linter is not a human. And I think the linter is the perfect tool for nitpicking style.

The linter tells you to add a brace and nobody cares.. the reviewer can just focus on the code.

The other half of the advice in the book (which people often don’t mention) is that when working on a program written by someone else, you should maintain the style that’s already there.

Most of the people in this industry are not dumb. And so anything that was built a certain way often had a good reason for being built that way, even if that reason was we didn’t have enough time or we didn’t have enough money.

Respecting the existing style is a soft skill masquerading as a hard skill. If you reformat someone else’s codebase in your first week, you burn through credibility you’ll need to make a change that actually matters.

The book’s reader is the next programmer. Increasingly, the next reader is a coding agent that doesn’t care where you put your braces. Is the formatter the last surviving style guide, or does consistency matter more now that the agent mimics the patterns around it?

Architecture Modernization is an org design project first

Thu, 04 Dec 2025 00:00:00 GMT

When modernizing an old system, at some point someone needs to decide what kind of project it is. That decision shapes everything that comes after. Usually, I think, this kind of project gets funded as a technology project (rewrite! migrate to a new platform!) with a budget and an end date. But what if we funded it instead as an org design project, and the first thing we did was redraw the ownership?

Architecture Modernization by Nick Tune with Jean-Georges Perrin (Manning, 2024) answers the subtitle question with socio-technical alignment of software, strategy and structure. Tune writes that architecture includes how the business works and how the team is organized, so modernizing must update outdated thinking in those areas as well as in the code.

Technology-first leaves teams and boundaries in place and replaces the stack underneath. It seems faster and easier to budget for. In the InfoQ podcast, Tune describes the typical outcome: the new world gets built in isolation, collides with the old world’s constraints late (network policies, compliance rules), and must bend around them. Often the old funding models and business processes need to be restructured too, and that’s beyond the technology budget.

If you rewrite the same systems with the same teams along the same boundaries, you’ll mostly recreate the old coupling on new tech. Conway’s law doesn’t care what the stack is. If two teams had to coordinate every release in the old system, chances are high that in the new system you’ll still have two services that need to coordinate every release.

The org-design-first option starts with the business, not the stack. It maps strategy with Wardley maps and EventStorming, aligns loosely coupled boundaries with business domains, and shapes teams around them using Team Topologies value streams. Only then do we choose the tech for each boundary, by the teams who own them.

Tune’s vehicle for this is the Architecture Modernization Enabling Team. It facilitates sessions like EventStorming, determines what support each team needs to stay on course, and avoids making architecture decisions centrally. There’s a real cost to this approach.. it takes longer to show results, and Tune is explicit about that: deliver less now to do more later.

I’d rather do the org design first, because if you set up boundaries that match the business, you can make smaller, easier tech decisions (each team can make decisions for their own domain instead of having to make a single platform decision for all of it at once).

Tune says that modernization doesn’t really finish.. “It’s not like a party at the end where it’s all done, it just people stop talking about it.” If that’s the case, I think that kind of open-ended work maps better to a standing capability (the enabling team that stays) than a closing project. If there’s no finish line, does modernization belong in a project budget, or is the enabling team what gets funded?

Domain-Driven Design: draw the bounded context before the glossary

Wed, 26 Nov 2025 00:00:00 GMT

Most of the DDD that I see reaching codebases is the tactical half: aggregates, repositories, a folder for each “domain”, and the team calls the result DDD. I think the tactical half will be less useful when coding agents write a lot of code. An agent can generate a repository in seconds. But it can’t on its own decide where one model stops and the next one starts, which is the strategic half of Eric Evans’ book.

A common move right now is to drop a glossary into the agent’s instructions and feel responsible. But if “account” means a ledger with a balance in billing and a user with a password and session for login, a single glossary needs to pick one meaning and the agent will happily spread that meaning across every part of the system it touches, even where it’s wrong.

I’d argue to first set up a bounded context. A glossary without a boundary is just a word list the agent will apply everywhere. But once you draw that boundary, each context gets its own short term set. “Account” can mean a ledger in one context and a login in another, and the agent won’t get confused.

Evans seems to agree on the ordering too. Ubiquitous language is early in the book, but bounded contexts aren’t until chapter 14. In a 2009 talk, he called that placement “one of the few regrets he has about the way he wrote the book” and said “the reality is that there is always more than one model”. So if the person who named the pattern wishes he’d put boundaries first, I’m comfortable putting them first in a prompt. (As an aside, he later compared a trained language model to a bounded context in a 2024 keynote)

The reason the tactical patterns matter the least to me right now is that the agent can generate aggregates and repositories all day. The strategic part (where to put the lines, what the words inside mean) is the work that’s left to people. One question I keep coming back to is how people keep the language up to date inside each context when agents are writing most of the code inside it.

Facilitating Software Architecture and who gets to decide

Tue, 25 Nov 2025 00:00:00 GMT

In large organizations, architecture decisions often queue up behind a small number of senior people or a standing review board. If you work in a regulated shop, eventually someone will ask you who made a decision, on what basis, and who they consulted. Facilitating Software Architecture by Andrew Harmel-Law (O’Reilly, December 2024) tackles this question directly, framing it as a design problem: how do we design the way the organization makes decisions?

The usual answers are both costly: the approving review board becomes a gate, a bottleneck where delivery teams either wait or quietly find ways around it; consensus spreads the decision across everyone in the room, so nobody owns the outcome when it needs to be revisited.

The alternative that Harmel-Law proposes is the advice process, which they first described in an article on martinfowler.com a few years before the book was published. The rule is simple: “anyone can make an architectural decision.” But before you make the decision, you need to seek advice from everyone who will be meaningfully affected by the decision, as well as people with relevant expertise. You don’t need to follow the advice you receive, but you do need to request it and record it. The emphasis here is on gathering many inputs and voices before a single person makes a decision, but not on reaching consensus.

Where I land is the advice process, every decision carried by an adr/architecture decision record. Harmel-Law’s version of the record includes context and options weighed, plus a section for advice received. That changes the purpose of the record. A written decision with advice recorded is better evidence of governance than an approval stamp in the minutes of a board meeting. It answers the questions an auditor might have right there on the page, with the name of the decider on it and the advice they gathered next to their own reasoning. A decision not documented is as good as not made when audit time comes.

My doubt is about scope. There are decisions that shouldn’t be taken alone, like adopting a new technology or adding a third-party integration. I’d keep a short written list of the types of decisions that have to go through an open advisory forum. Per Harmel-Law, that’s a weekly hour where anyone can show up. Everything else I’d leave to the team that has to live with the result. Even for the listed decisions, the forum advises, the decider still decides.

The forum’s job changes from gate to conversation. I suspect the change is bigger for architects than for teams. Architects go from people who say yes to people whose advice is worth seeking. That’s a different skill and a different place in the org. An agent can draft the context and options for a decision record well enough. The process only holds if the named person asked the people affected before making the call themself.

Project to Product and the work a feature roadmap hides

Fri, 21 Nov 2025 00:00:00 GMT

You’re looking at your team’s end-of-quarter done column. All the cards are features. You advanced your roadmap, you successfully demoed. But is this a product team? As Mik Kersten describes in Project to Product, look at what’s not in your column. The Flow Framework describes all software work as four flow items: customer-pulled features, defects, risks (security and compliance work), and debts. If your column only has features, it means you didn’t fix any defects, you didn’t deliver any risk work, you didn’t pay down any debts. Or you did, but you’re not counting it anywhere.

I think the headline idea in this book (projects end, products don’t) has largely aged into our vocabulary. We won it as a way of talking, but lost it as a way of funding teams. I would keep the flow distribution as a measure to determine if that shift really happened. The four items all share from the same fixed capacity, so a larger share of features means a smaller share for the other three. Your roadmap only lists features.

Kersten warns of this in his essay how to budget for technical debt. As more new functionality gets delivered, less time is available for fixing defects and doing security work. A shrinking share of work on technical debt is a leading indicator for defects and delays that haven’t yet become visible. The question for me is: when you are in a planning meeting, who argues for the share of technical debt when features are being demoed? Kersten says you need to reserve time and money for technical debt every financial year. If you don’t, the debt will remain invisible to the business, just like it did for Nokia with Symbian.

The cheap features of AI coding agents make this problem even more acute. If an agent can build a feature in an afternoon, the share of features will grow unprompted, while risk and debt only get their share if deliberately protected. The trade-off gets worse before it gets better. If agents can also fix defects and pay down debt, does the trade-off go away, or does the fixed capacity shift to people reviewing the agent work?

Always be curious

Thu, 13 Nov 2025 00:00:00 GMT

I’ve been poking around with computers since the BBS days. I started with a Tandy 1000SX with no magnetic storage, eventually had a fleet of budget Pentiums running Slackware in my basement, graduated to rack mounts and co-locations, and eventually turned it into a career in cloud architecture. Hell, I ended up co-writing a book, the AWS Cookbook, about how curious builders can use the cloud to do really amazing stuff.

Curiosity got me here, it’s just how I’m wired.. I’ll be honest, it does take conscious effort to stay that way daily in my adulthood, and it is an effort I look forward to every day with my family and my own hobbies. In my own hobbies, always be building became the motto, but it’s all rooted in this insatiable curiosity. Curiosity is the why.. building is the how.. they are both parts of my story. One without the other and the momentum just fizzles out.

I still run Linux machines in my basement to this day, and they are the foundation for my home lab. Yes, I rent cloud compute for things like ML model training and other side quests, but self-hosting in my basement is just home to me, no pun intended.

This is where the magic happens. My home lab is the home of the heat pump telemetry hunt, the fast roaming tests, and many other fun projects. It’s also where the “always be building” motto is most easily misinterpreted as a nightly obligation.

Always doesn’t mean every night.. it means a steady habit. If the baby is napping on me, I’ll pick up a Claude Code session and fix some bugs in my Home Assistant plugins or push an update to my e-ink firmware. Maybe cruise the ESP32 subreddit for inspiration on future projects.

I’m genuinely excited to write about what I’ve been working on because it turns fleeting thoughts and productivity blasts into embedded, durable knowledge.. there is science to this. Writing as you experiment and learn beds in those concepts, and so does sharing with your peers, colleagues, strangers, internet passers-by, and AI agents stopping by here. Psychologists call the first part the generation effect, where information you produce yourself is remembered better than information you only read, and the testing effect shows that pulling something back out of memory makes it stick longer than reviewing it again.

In Why I built this site I talked about my decision to start writing here, and this post is about the underlying reason for that. Here are the next projects I’m going to write up, as that’s where my curiosity is currently pointed.

I’m super excited to write about my Aviary project, an extension of my BirdNET-Go deployment, my e-ink management software called Pigment, and my agentic framework forked from NanoClaw that I run fleets of agents in, powered by Claude Code and tmux. More strategic takes on technology and my day job. How I see the bleeding edge of this technology shaping humanity, and the exciting new things we need to adapt to. Book reviews, rants, raves, anecdotes, write-ups, write-downs, and more.

Since this space is literally just a rendered extension of my Obsidian vault, I can easily bring new posts here with minimal friction to share. See How this site is built for more on that.

Stay curious & keep building.

Edited on 2026-10-10 to link Walking around for fast roaming’s sake, The Mitsubishi heat pump telemetry hunt and My cloud architect habits at home.

How this site is built

Fri, 31 Oct 2025 00:00:00 GMT

Short version: it’s a static site on Cloudflare Pages, hosting costs nothing, and I publish posts from Obsidian without opening a terminal.

For years my “website” was a LinkedIn profile and a résumé on VisualCV. Recently I replaced both with this site. (Why I bothered is its own post: Why I built this site.)

It’s built with Astro. There’s no server and no database, just HTML files that Cloudflare Pages serves from its edge. The source lives in a private GitHub repo with a mirror on my home server. Every pull request gets its own preview URL and runs the checks: linting, type checks, tests against the built pages, an HTML validator, a link checker, and Lighthouse, which has to score at least 90 in every category.

I looked at sixteen design directions before picking one, and the first version leaned on an album liner notes idea, with numbered tracks and a small catalog mark. It looked clever in a mockup. On the live page it was noise, so I took it out. What’s left is two typefaces and one column. Most people who land here want my résumé or a way to reach me, and I’d rather not make them hunt for either.

The fonts are Newsreader, Barlow and DM Mono, all served from this site instead of Google Fonts. I started with the variable version of Newsreader, which weighed 280 KB and pushed the largest paint on the page to almost three seconds. Static cuts fixed that. The one thing I missed was the heavier headline, so the h1 now has its own 21 KB file, cut from the variable font at the 72-point optical size.

Posts start as notes in Obsidian, where I already write everything else. A post is a note with a title, a one-line description and a date. When it’s ready I flip one property, share: true, from my laptop or my phone, and that’s the last thing I touch.

Obsidian Sync carries the note to my home server, which runs Obsidian in a container. Once an hour a small script looks for posts that changed, and when it finds one it tells Enveloppe, an Obsidian plugin, to publish it. Enveloppe opens a pull request with the Markdown, the pull request merges itself, and Cloudflare builds the site.

I’m particularly excited about edits.. if I want to add something to a post, clarify or correct something, I simply make the change with Obsidian or any text editor, save it, and it goes out with the next hourly check. Or right away if I tell Enveloppe to push and merge.

The date is how scheduling works. The build leaves out any post dated in the future, so it isn’t in the index, the RSS feed or anywhere else until its day. A GitHub Action starts a fresh build every morning, and whatever is dated that day shows up. Days are New York days, so a post dated Tuesday goes live Tuesday morning here, not at midnight UTC.

Each post also gets its own preview card for LinkedIn and anywhere else that shows one. The build draws it as a PNG from the post title, in the same fonts as the site.

Is it a lot of machinery for a personal site? Probably a little. But it runs itself, it costs nothing to host, and I can publish from my phone, which is all I really wanted.

Why I built this site

Wed, 29 Oct 2025 00:00:00 GMT

I’m always learning, always reading and always writing. Most of that writing lives in my Obsidian vault, where it helps me think but doesn’t help anybody else. I wanted a place for the pieces worth sharing, so I built one.

For a long time my presence online was a LinkedIn profile and a résumé on VisualCV. Both are fine when someone wants my work history. Neither is a good place to talk about a book I’m reading or a tool that surprised me. I’ve done long-form writing before as a co-author of the AWS Cookbook, and I’d like to write more often, in smaller pieces.

The other reason is that building it was fun. I got to geek out on Astro, Cloudflare Pages, publishing straight from Obsidian and a bunch of AI agent workflows, and I learned something from every one of them. Learning by building a thing I’ll actually use is right up my alley. The details are in How this site is built.

So what goes here? Cloud architecture and software engineering, since that’s my work, plus the stuff I geek out on outside of it: books, music, and the projects running on my homelab. Some posts will be quick raves, some will be longer write-ups (and probably a rant or two).