infra/docs/runbooks/breakglass-ui.md

Runbook: devvm breakglass UI (claude-breakglass)

Last updated: 2026-06-12

What this is

breakglass.viktorbarzin.me — an in-cluster Claude-driven web UI for recovering the devvm (Proxmox VM 102) when it is wedged but the cluster is healthy (the warm case). You chat with a Claude agent that SSHes into the devvm to diagnose/repair it, and there are manual buttons that power-cycle the VM via the Proxmox host even if the Anthropic API is down.

This is NOT the cold breakglass. If the cluster or PVE host is down, this UI is down too (it's a cluster workload). For that case use the cold path:

• ssh -p 52222 root@<wan> → qm stop 102 && qm start 102 (docs/runbooks/breakglass-ssh.md)
• server-lifecycle iDRAC CLI (192.168.1.4) to power-cycle the whole host.

Architecture

browser ─► Cloudflare ─► Traefik ─► auth-proxy (Authentik, basic-auth fallback)
                                      └─► claude-breakglass Service (in-cluster)
claude-breakglass pod (ns claude-breakglass, own SA, NO Vault role):
  • app.breakglass.server (FastAPI) serves the Svelte UI + /api
  • chat → claude -p --agent breakglass (stream-json → SSE)
  • ssh-agent holds the breakglass key (synced by ESO, never on disk)
  • ssh devvm  → breakglass@10.0.10.10 (full sudo)         [diagnose/repair]
  • ssh pve <verb> → root@192.168.1.127 forced-command     [VM 102 power verbs]

Image: forgejo.viktorbarzin.me/viktor/claude-agent-service:latest (shared with claude-agent-service; the deployment overrides the command with /srv/docker-entrypoint-breakglass.sh). Code: claude-agent-service/app/breakglass/. Stack: stacks/claude-breakglass/. ADR: claude-agent-service/docs/adr/0001-*.

Auth (how to get in)

• Normal: Authentik SSO (you're already logged in to the SSO).
• Authentik down: the auth-proxy falls back to HTTP basic-auth ("Emergency Access"). Username admin; password is the shared auth_fallback_htpasswd (Vault secret/platform). This same credential gates every auth="required" app. Rotate: regenerate the htpasswd, vault kv patch secret/platform auth_fallback_htpasswd=..., apply the traefik stack (the auth-proxy rolls on the checksum/auth-proxy-htpasswd annotation).

The PVE forced-command (the reset path)

The breakglass SSH key's entry in PVE /root/.ssh/authorized_keys is pinned to command="/usr/local/bin/breakglass-pve",restrict,from="192.168.1.2". It only accepts the bare verbs status | forensics | reset | stop | start | cycle against VM 102 — anything else is rejected and logged to /var/log/breakglass-pve.log. Every mutating verb captures forensics first.

• cycle = stop→start (fresh QEMU, applies staged config) — the fix for a QEMU I/O stall (2026-06-11). If a clean stop fails, it kills the wedged QEMU PID then starts. Prefer cycle over reset for a wedged VM.
• reset is a warm reset (reuses QEMU) — only for a normal guest hang.

Script source: stacks/claude-breakglass/files/breakglass-pve (deploy via scp … root@192.168.1.127:/usr/local/bin/breakglass-pve).

NAT quirks (why from= differs per host)

Discovered during bring-up — both verified from a real in-cluster pod:

• pod → PVE (192.168.1.127): pfSense SNATs inter-VLAN traffic to its 192.168.1.2 interface, so PVE sees 192.168.1.2 for ALL cluster (and devvm) SSH. Hence the PVE key uses from="192.168.1.2". The devvm itself is NOT a permitted source (it's the box being recovered).
• pod → devvm (10.0.10.10): the devvm sees the Calico-SNAT node IP (10.0.20.0/24). Hence the devvm key uses from="10.0.20.0/24".

Host bootstrap (one-time; redo on devvm rebuild / key rotation)

The keypair lives in Vault secret/claude-breakglass/ssh_key (private_key/public_key). To re-provision after a rebuild:

PUB=$(vault kv get -field=public_key secret/claude-breakglass/ssh_key)

# devvm (full-sudo recovery user):
sudo useradd -m -s /bin/bash breakglass 2>/dev/null || true
sudo install -d -m700 -o breakglass -g breakglass /home/breakglass/.ssh
printf 'from="10.0.20.0/24" %s\n' "$PUB" | sudo tee /home/breakglass/.ssh/authorized_keys
sudo chown breakglass:breakglass /home/breakglass/.ssh/authorized_keys
sudo chmod 600 /home/breakglass/.ssh/authorized_keys
echo 'breakglass ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/breakglass && sudo chmod 440 /etc/sudoers.d/breakglass

# PVE (forced-command power verbs):
scp stacks/claude-breakglass/files/breakglass-pve root@192.168.1.127:/usr/local/bin/breakglass-pve
ssh root@192.168.1.127 chmod 0755 /usr/local/bin/breakglass-pve
# then append to /root/.ssh/authorized_keys on PVE:
#   command="/usr/local/bin/breakglass-pve",restrict,from="192.168.1.2" <PUB>

Host-key checking is OFF in the pod's ssh config (a devvm rebuild rotates the host key; strict checking would lock the breakglass out mid-incident — trusted internal LAN, key auth stands).

Verify

kubectl -n claude-breakglass get pods                 # Running
kubectl -n claude-breakglass logs deploy/claude-breakglass | grep -i ssh-add
curl -sk https://breakglass.viktorbarzin.me/health    # (through the edge)
# from a pod, the PVE path:  ssh pve status  → "status: running"

Isolation (why a separate deployment)

The shared claude-agent pod runs agents that ingest untrusted input (recruiter emails, nextcloud todos) with Bash. Co-locating the root-on-devvm key there would let a prompt injection exfiltrate it. The breakglass runs in its own namespace with its own SA and no Vault role (ESO syncs only its key); the terraform-state Vault policy is explicitly DENIED secret/claude-breakglass/*.