Michael Zazon

Reliability & operations · Take

When a bad certificate chain meets agents that work this fast

Eight weak eMudhra intermediates made cert forgery easy for anyone with agents, so I used mine to patch my container trust stores within days.

Bold to assume a super budget esp32 IoT device would check for revoked certificates. These companies crank these things out very cheaply and move on to the next.. software updates and security patches are far and few between.

I’ve been stuck on this reaction ever since the October eMudhra advisory, which revealed that they’d issued 8 intermediate certificates with weak keys that anyone can use to forge a certificate for any domain. There’s a test page that shows whether your client still trusts them.

Most IoT devices wouldn’t really be able to handle this case anyway.. virtually none check CRLs or OCSP, and they don’t bundle intermediate CAs to mark them as invalid

The CAs are baked into the firmware pretty much all the time..

The trust store on the device is what it shipped with. It can only be changed by a firmware update from the vendor. Actions the device owner takes on their network can’t change what certificates the device believes in.

A lot of embedded devices (e.g. ESP32) only include the certificate chains for the servers they contact, so this won’t be useful in those cases. Anything running Linux will probably be using ca-certificates though, which will have it.

This was true for the host, which had already been patched by unattended upgrades, but I’d assumed it covered more than it did: most containers ship with their own CA store from the day the image was built. How many images were running around with a frozen copy of that list? I switched to a single shared read-only CA bundle, refreshed from the host’s, mounted into containers. The agent did the inventory and edits, and I approved each batch before it rolled out.. the same ask-before-fixing habit I’d gotten used to with Sentinel. Node’s system-CA flag was a no-go because it only adds trust, it can’t remove a bad root.

No need to chuck tokens at it, just use mitmproxy

Most of the discussion seems to be about people scanning their own devices for fun, which is cheap these days whether you use agents or not. But that speed is available to anyone, and Anthropic reported an AI-orchestrated espionage campaign. In the past, such flaws would have waited until someone bothered to exploit them. Now, with near-zero cost to scan and try, the gap is human-speed patching. That’s why I used agents defensively, and the shared bundle was rolled out within days of the advisory.

So I made two decisions. First, I made my containers’ trust stores host-owned instead of frozen to each image at build time. Second, I left the agent on the inventory so that next time there’s an advisory, it’ll be a batch job for me to approve.