1Password CLI and Connect: secrets without env files

On my home server, every stack had an adjacent file of real secret values, even though I’ve been using 1Password as the source of truth since May. As described in How a deploy works with Dockhand, the pattern was for a script to read each value with the 1Password CLI and PUT it into the Dockhand per-stack environment as a regular variable, which Dockhand persisted as a plaintext .env.dockhand file per stack on the ZFS data disk. This worked for 3 months. I said I’d talk more about the op CLI in chezmoi one source of truth for every dotfile, so here goes. The cost was a plaintext file of real values for every stack on the data disk.

A post on r/1Password about using 1Password Connect secrets in Docker Compose led me to alikhil’s docker-secrets-1password, a 1Password Connect provider for Docker’s Secrets Engine, which lets compose carry se:// references at runtime instead of values. The engine is experimental and requires Docker CE 29.2+. I did some analysis, then a spike with two dummy containers. The references resolved in both plain docker run and compose. Resolution happened at container creation, so the resolved value was stored in the container’s config on disk (Docker’s own metadata). The root client could resolve them too, so Dockhand didn’t need changes. A restart worked with the engine stopped, and a new create failed loudly. The engine’s install bounced dockerd via needrestart, and I was relieved live-restore carried all 84 containers without restarts.

I wrote down the self-assessed honest security delta: wins (pointer-only compose files, no env files on disk, no more relying on Dockhand’s replace-everything env API, rotation is just update a 1Password field and recreate the container), losses (plaintext moves into dockerd metadata, any Docker-socket client can resolve any secret the engine serves). (My note: judge it as plumbing hygiene, not a security boundary change, plan to wait until the engine leaves experimental.) I reversed myself the next morning and adopted it on experimental 0.9.0 anyway, deciding I could accept the risk as long as I only did one stack at a time and the Connect server was only ever deployed by hand. Kept a watch item for when the engine stabilizes.

The resolution chain

Before I put anything into production, I set up a couple of guardrails: first, a dummy container that resolved a reference with a plain docker run against the real setup; then a throwaway Dockhand git stack, because Dockhand’s own compose invocation was the last link I hadn’t tested. Both passed, and I tore them down. The first real stack I put into production was the query-log shipper from AdGuard Home and the query-log shipper, because that seemed like the lowest-stakes thing to start with. The chain I’d put in place was: pointers in the compose files in git, Dockhand runs compose, the Docker daemon while creating the container asks the Secrets Engine for each reference, the engine’s provider gets the value from a local Connect server, which syncs from 1Password.

The Connect stack itself runs two containers, connect-api and connect-sync (1.8.2), with the API bound to loopback only. The credentials file lives outside git and is backed up to 1Password as a document (I tested restoring it and got back a byte-identical file). The data directory is just a cache; I can always resync from scratch. I disabled autoheal and the deploy webhook for this stack because it holds the identity, so a sweep must never recreate it. The provider is alikhil’s 0.3.1 deb, running as a systemd system unit, with the token stored as a systemd-creds encrypted credential piped in (it never touches the command line). The Ansible playbook that codifies the host layer asserts the credential exists but never creates it. The bootstrap is checked for cycles: the Connect stack itself has no se:// references, and its two roots of trust (the credentials file and the encrypted token) are both restorable from 1Password.

# stacks/op-connect/compose.yaml (excerpt)
# IDENTITY-HOLDING: no autoheal. MANUAL-DEPLOY-ONLY (webhook disabled).
# API bound loopback-only: any client with the token can read the vault.
services:
  connect-api:
    image: 1password/connect-api:1.8.2
    ports:
      - "127.0.0.1:<port>:8080"
    volumes:
      - <credentials-file>:/home/opuser/.op/1password-credentials.json:ro
      - <data-dir>:/home/opuser/.op/data
  connect-sync:
    image: 1password/connect-sync:1.8.2
    volumes:
      - <credentials-file>:/home/opuser/.op/1password-credentials.json:ro
      - <data-dir>:/home/opuser/.op/data

The CLI had one more lesson for me before any of this even ran. I set up the Connect server from my laptop using my personal 1Password account, and got a 403 on the first attempt. I assumed the CLI was acting as my signed-in account, but it turned out that the service-account token I had in my shell environment was taking precedence (which took me a while to see). Service accounts can’t manage Connect, so every request was being refused as that token.

23 stacks in a day

For each value, I took the sha256 hash of the value Dockhand injected and compared it to the hash of the 1Password field. I probed the reference in a throwaway container before committing. After deploying, I checked the running container’s env by hash and the app itself by using it. Then I emptied the stack’s env in Dockhand, deleted the leftover .env.dockhand (which Dockhand never removes when the variable set goes empty), and verified it stayed gone after redeploy. The Compose variable value became a pointer plus a comment naming the thing.

# before: Dockhand injected the value from its per-stack env file
environment:
  - APP_PASS=${APP_PASS}

# after: a pointer the Docker daemon resolves when the container is created
environment:
  - APP_PASS=se://op/<vault-uuid>/<item-uuid>/<field>   # item "<name>" in 1Password

The friction came in small pieces throughout the day: item titles with spaces triggered invalid pattern from the provider’s matching logic (now all references use UUIDs plus a comment naming the item), the provider only matches on field labels (field ids and %20 didn’t work, so I renamed labels to avoid spaces), the reference has to be the entire variable value (my connection string embedding a password now lives whole as its own field, rotating with that password), and references don’t resolve in command: (put the value in the environment, have the shell read it, $$ escape for compose.. proven live on a Valkey server taking its password on the command line).

# stacks/<stack>/compose.yaml (excerpt)
environment:
  REDIS_PASSWORD: se://op/<vault-uuid>/<item-uuid>/<field>
command: ["sh", "-c", "exec valkey-server --requirepass \"$$REDIS_PASSWORD\""]

The CLI misled me twice: op item create - reads a JSON template from pipe only, not < file redirect (it complains about --category, but adding --category creates a blank item). My failed attempts to create a Connect token and store it in the same pipe still minted the token, so I deleted two orphans. The hash step revealed variables I’d injected for months that reached nothing, including one admin “user” variable that was literally the app’s default. By day’s end, all 23 stacks ran on se:// references (18 commits in) with zero .env.dockhand files on the server. I also found an orphaned deploy directory for a retired service that still had plaintext credentials in it, so I deleted that.

Locked-out nodes

The VPN sidecars had been keeping their Tailscale state in memory, so every restart/redeploy registered the node as brand new. I’d turned on Tailnet Lock around August, so every new node key needed signing before the node could talk. The container and proxy looked healthy, the deploy was green, but the tunnel was dead. I wondered how many other things were “fine” while broken. It bit me three times that day: after the migration, after a verify bounce, after a push that changed nothing but a comment. I signed each new node key by hand (nine signings).

I fixed this by persisting the sidecar’s state so each node has one identity, signed once. After the change, a restart kept the same node key and nothing got locked out. I didn’t want to use pre-signed auth keys because Tailscale’s docs say that’s equivalent to leaking your signing key, and I didn’t want to set up an auto-sign timer because that’s just a rubber stamp.

# stacks/<vpn-stack>/compose.yaml (excerpt)
     environment:
       - TS_AUTHKEY=se://op/<vault-uuid>/<item-uuid>/<field>
-      - "TS_STATE_DIR=mem:"
+      - TS_STATE_DIR=/var/lib/tailscale
     volumes:
+      - <state-dir>:/var/lib/tailscale

The gates and hashing on both ends made it safe to do a day’s worth of changes all at once: no value moved without proof of arrival. The Compose files only hold pointers, and rotation is a design property: change a 1Password field and recreate the container (a restart won’t do it). Losing the engine only costs new creates, not anything already running. What’s not better: the resolved value is still in Docker container metadata by design, and the experimental pieces have sharp edges and misleading CLI errors. Worst of all, the green deploy told me nothing about the dead tunnel or the dead secrets chain.

Both can hide behind a green deploy, so I changed how I monitor them. Now an uptime monitor watches Connect’s health endpoint, and separately the existing 15-minute health pulse gained two new checks: one that walks the entire secrets chain behind the daemon (engine, provider, Connect), and one locked-out-node check that looks for any node Tailnet Lock is holding unsigned. Both are proven by breaking things on purpose, and the reboot baseline script snapshots both. The old injection script stays dormant until the engine survives its first Docker CE upgrade. Separately, the next test is a full redeploy of the VPN sidecars. That’s where I’ll leave it.