infra/docs/plans/2026-07-03-vault-token-self-heal-design.md

Vault Token Renewer Self-Heal Design

Date: 2026-07-03 Status: Approved (brainstorm complete; implementation pending) Owner: wizard@devvm Supersedes: the "version-only, no self-heal" scope choice recorded in docs/runbooks/vault-token-renew-devvm.md (2026-06-07)

Problem

wizard@devvm holds a maintenance-free periodic Vault token (token-devvm-wizard, period=768h, renewed daily by the vault-token-renew user timer) precisely so no weekly re-login is needed. But ~/.vault-token is the Vault CLI's default token sink, so any vault login -method=oidc — which the infra docs themselves instruct before applies — overwrites it with a 7-day OIDC token. The renewer's drift guard (deliberately detect-only) then refuses to renew the foreign token and fails the unit daily, into a log nobody watches.

Observed consequence: a self-perpetuating weekly-expiry loop. The OIDC token expires after 7 days → Vault 403s → the natural response is another vault login -method=oidc → clobbers again. Drift persisted unnoticed 2026-06-18 → 06-26 and 2026-06-29 → 07-03 (memory #7121); Viktor experienced it as "the token expires maybe once a week".

Goal: vault login -method=oidc becomes harmless on devvm. The renewer converts any admin-capable clobber back into the permanent periodic token, unattended. (Chosen over "never log in" doc-fixes and over instant path-unit healing — see Alternatives.)

Decisions

#	Decision	Notes
1	Heal in the existing renewer's drift branch, at its nightly run	~20-line diff to an already-tested script; no new units. A few-hours window holding the 7-day OIDC token is harmless (heal window 24h ≪ 7d TTL)
2	Heal = attempt re-mint using the foreign token itself; let Vault's 403 decide	No policy-list guessing — identity-vs-token-policies burned us before (memory #4211). OIDC tokens carry vault-admin via identity_policies, so the create succeeds
3	Weak foreign token (create denied) → keep today's loud DRIFT failure	A read-only clobber (e.g. the 2026-06-05 kubernetes-woodpecker-default incident) signals a misbehaving agent flow; auto-papering over it would hide the offender. Log gains a "heal denied — investigate what wrote it" suffix
4	Do NOT revoke the clobbering OIDC token	It may still back the user's live login session; it ages out in 7 days on its own
5	After a successful heal, revoke stale token-devvm-wizard accessors	Anti-sprawl: each heal would otherwise strand the previous periodic admin token server-side for up to 32 days. Walk auth/token/accessors, revoke every display_name=token-devvm-wizard except the just-minted one. Runs only on heal (rare), never on the happy path
6	Minted-token sanity check before writing the file	Look up the new token; require display_name=token-devvm-wizard. Write via temp file + mv + chmod 600 so a failed mint can never truncate ~/.vault-token
7	Keep timer cadence (daily) and all happy-path behavior unchanged
8	No notification plumbing in this change	devvm alerting is tracked separately (beads code-aslh). Heal events are logged; heal-denied/FAIL still fail the unit

Behavior matrix

Token found in ~/.vault-token	Before	After
Our periodic token	renew-self, log OK	unchanged
Foreign, admin-capable (OIDC login)	log DRIFT, exit 1	re-mint periodic token with it, sanity-check, atomic write, revoke stale periodic accessors, log HEALED: re-minted from foreign dn=<dn> (revoked N stale), exit 0
Foreign, weak (read-only k8s clobber)	log DRIFT, exit 1	log DRIFT … heal denied — foreign token lacks create authority; investigate what wrote it, exit 1
Vault unreachable / lookup fails	log FAIL, exit 1	unchanged

Re-mint command (identical to the manual recovery the DRIFT log already prescribes):

vault token create -orphan -period=768h \
  -policy=vault-admin -policy=sops-admin -display-name=devvm-wizard

Testing

• Unit (scripts/test-vault-token-renew.sh, existing source-the-functions harness): new pure functions for (a) the stale-accessor revoke filter (match on display_name, exclude the current accessor) and (b) the minted-token sanity predicate; regression cases for the existing drift predicate stay green.
• Live, post-deploy (on devvm):
  1. Mint a fake 1h admin token (-display-name=fake-oidc, -policy=vault-admin -policy=sops-admin), write to ~/.vault-token, start the service → expect HEALED, file holds token-devvm-wizard.
  2. Mint a fake 10m no-privilege token (-policy=default), write it, start the service → expect DRIFT … heal denied, unit failed; restore real token.
  3. Revoke both fakes; one-off sweep of stale periodic accessors left by the June 26 / July 3 manual re-mints.

Docs & rollout

• Same commit rewrites the runbook's "Drift guard & recovery" section: self-heal is the recovery for admin-capable clobbers; manual re-mint remains only for weak clobbers (or a dead token with no admin-capable replacement in the file).
• vault login -method=oidc instructions across the docs stay as-is — the login is now harmless by design.
• Deploy per the runbook's manual model: install -m 0755 to ~/.local/bin/vault-token-renew. Units unchanged — no daemon-reload.
• After landing: update memories #4204/#4211 (gotcha now self-healing).

Alternatives considered

• Instant heal (systemd path unit + protected source-copy of the token): strictly more capable (seconds-latency, heals weak clobbers too, zero re-minting), but 2 new units + a second secret file + inotify re-trigger edge cases — machinery disproportionate to the residual risk. Revisit only if the few-hour heal window ever bites.
• Vault CLI token_helper interception: right interception point in theory, but a helper bug breaks every vault CLI call, Terraform reads ~/.vault-token natively anyway, and it adds latency inside login. Rejected.
• Docs-only ("never log in"): rejected by user — the login should keep working, not become forbidden knowledge.
• Raise the OIDC role's 7-day token_max_ttl: shared role, affects every OIDC user; rejected previously for the same reason (memory #4205).