infra/docs/runbooks/chrome-service-snapshot.md

Runbook — chrome-service snapshot pipeline

Operational playbook for the hourly cookie-snapshot pipeline that warms external Claude Code sessions on the dev box. Architecture in architecture/chrome-service.md.

At a glance

Component	Where	When	What
chrome-service Deployment	chrome-service ns	always-on	headed chromium, CDP :9222, persistent /profile/chromium-data
snapshot-server sidecar	same pod	always-on	serves /api/snapshot, bearer-gated, port 8088
snapshot-harvester CronJob	chrome-service ns	23 * * * *	dumps storage_state() via CDP → /profile/snapshots/storage-state.json
dev-box refresh timer	each dev box, per OS user	hourly (*:28)	playwright-snapshot-refresh@<user>.timer curls chrome.viktorbarzin.me/api/snapshot → ~/.cache/playwright-shared-storage-state.json
dev-box playwright-mcp@<user>.service	each dev box, per OS user	always-on	pinned @playwright/mcp@<ver> --isolated --storage-state=… on the user's PLAYWRIGHT_PORT; per-MCP-connection (per-session) contexts

Provisioning (reproducible from git)

The dev-box side is per-OS-user and fully reproducible — no hand-setup. Each user gets their own isolated @playwright/mcp server (multiple concurrent Claude sessions per user, isolated by --isolated), wired into their Claude in every directory via a user-scope ~/.claude.json entry (playwright → http://localhost:<PLAYWRIGHT_PORT>/mcp).

• System-level template units (NOT systemd --user, so no linger needed): playwright-mcp@.service + playwright-snapshot-refresh@.{service,timer}, sourced from infra/scripts/workstation/playwright/, installed to /etc/systemd/system/ by setup-devvm.sh (§9e). User=%i; per-user PLAYWRIGHT_PORT from /etc/t3-serve/playwright-<user>.env.
• Port allocation: roster_engine.py (PLAYWRIGHT_BASE_PORT=8931, sticky) — emitted in the derive JSON, written per-user by t3-provision-users.sh (§5c).
• Snapshot token: setup-devvm.sh (§8c) stages Vault secret/chrome-service api_bearer_token → root file /etc/t3-serve/chrome-service-token; the provisioner copies it (if-absent, 0600) to each user's ~/.config/playwright/token (the hourly root reconcile has no Vault token, hence the staging — mirrors the Claude OAuth token in §8a).
• MCP wiring + enablement: t3-provision-users.sh install_playwright() runs claude mcp add --scope user … playwright AS the user (clobber-proof, if-absent) and systemctl enable --now the system instances. Idempotent; never restarts a running instance or rewrites an existing ~/.claude.json entry.
• Pinned version: bump @playwright/mcp@<ver> in scripts/workstation/playwright/playwright-mcp@.service (the @latest → silent-fleet-roll footgun is why; see the T3_PIN rationale in setup-devvm.sh).

Day-to-day

Log into a new site (warm the profile)

1. Open https://chrome.viktorbarzin.me/ (Authentik will gate).
2. The noVNC view of the in-cluster headed chromium loads. Click on the browser window, navigate, log in.
3. Cookies land in /profile/chromium-data/Default/Cookies on the PVC.
4. Within ≤60 min, the snapshot-harvester CronJob picks them up and writes the snapshot. Within ≤60 min after that, dev boxes pull the new file. New Claude Code sessions see the new cookies.
5. To skip the wait: trigger the harvester now (next section).

Trigger snapshot harvester manually

kubectl -n chrome-service create job \
  --from=cronjob/chrome-service-snapshot-harvester \
  snapshot-harvest-$(date +%s)

# Watch logs
kubectl -n chrome-service logs -f -l job-name=$(kubectl -n chrome-service get jobs -o name | tail -1 | cut -d/ -f2)

Expected: wrote snapshot (… bytes) to /profile/snapshots/storage-state.json.

Trigger dev-box refresh manually

# On the dev box, refresh a specific user's snapshot (system template instance):
sudo systemctl start playwright-snapshot-refresh@<user>.service

# Or run the script directly AS that user:
sudo -u <user> /usr/local/bin/playwright-snapshot-refresh

# Verify
sudo ls -la /home/<user>/.cache/playwright-shared-storage-state.json

Inspect the current snapshot

# In-cluster (from any pod with kubectl exec into the chrome-service pod):
kubectl -n chrome-service exec deploy/chrome-service -c snapshot-server -- \
  cat /profile/snapshots/storage-state.json | jq '.cookies | length'

# Externally (via the bearer-gated endpoint):
TOKEN=$(vault kv get -field=api_bearer_token secret/chrome-service)
curl -fsSL -H "Authorization: Bearer $TOKEN" \
  https://chrome.viktorbarzin.me/api/snapshot | jq '.cookies | length'

Failure modes

"no browser contexts found"

The harvester reports no browser contexts found — chrome-service may not have launched a persistent context yet and exits non-zero.

Cause: chromium just started and hasn't created its default context yet, or it crashed.

Fix: check chrome-service pod logs (kubectl -n chrome-service logs deploy/chrome-service -c chrome-service). The next hourly run will retry. If chromium is wedged: kubectl -n chrome-service rollout restart deploy/chrome-service (strategy = Recreate, brief downtime).

"connect_over_cdp failed"

Harvester or any in-cluster caller can't reach the CDP endpoint.

Cause: chrome-service pod not Ready, NetworkPolicy doesn't admit the caller's namespace, or chromium isn't listening on :9222.

Diagnose:

kubectl -n chrome-service get pods
kubectl -n chrome-service describe networkpolicy chrome-service-ws-ingress

# From inside the cluster (e.g. a debug pod in chrome-service ns):
nc -zv chrome-service.chrome-service.svc.cluster.local 9222
curl -fsSL http://chrome-service.chrome-service.svc.cluster.local:9222/json/version

Fix: depends on the diagnosis. NetworkPolicy needs the caller's namespace label or an explicit name-fallback. If chromium isn't binding, check the container logs.

Dev-box playwright-snapshot-refresh returns 401

The bearer token in ~/.config/playwright/token doesn't match the server's. Almost always means the Vault secret was rotated and the local cache is stale.

Fix (re-stage centrally so a rebuild stays correct, then re-copy to the user):

vault login -method=oidc  # if needed
sudo install -m 0600 <(vault kv get -field=api_bearer_token secret/chrome-service) \
  /etc/t3-serve/chrome-service-token
sudo install -o <user> -g <user> -m 0600 \
  /etc/t3-serve/chrome-service-token /home/<user>/.config/playwright/token
sudo systemctl start playwright-snapshot-refresh@<user>.service

Dev-box playwright-snapshot-refresh returns 404 with "snapshot not yet available"

The harvester hasn't run successfully yet (fresh cluster, or all recent runs failed). Trigger it manually (see "Trigger snapshot harvester manually").

Claude Code sessions still see old cookies

The MCP server reads the snapshot file at process start and seeds each new context with it. Existing MCP sessions don't hot-reload — they keep the cookies they were seeded with at session start. New sessions get the fresh snapshot.

Fix: restart the user's MCP server on the dev box to pick up the new file:

sudo systemctl restart playwright-mcp@<user>.service

Snapshot file is suspiciously small or empty cookies array

The persistent chromium context isn't holding any cookies. Probably means the user hasn't logged into anything via noVNC, or chromium was relaunched without preserving /profile/chromium-data.

Diagnose:

kubectl -n chrome-service exec deploy/chrome-service -c chrome-service -- \
  ls -la /profile/chromium-data/Default/Cookies

A populated Cookies SQLite file should be several hundred KB once real logins exist. If it's missing or empty, log in via noVNC.

Token rotation

# Rotate Vault secret (32-byte URL-safe random).
vault kv put secret/chrome-service \
  api_bearer_token=$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')

# Reloader auto-restarts chrome-service pod (snapshot-server picks up new token).

# On EVERY dev box: re-stage the root file, then overwrite each user's copy
# (the provisioner's per-user copy is if-absent, so a ROTATION must overwrite).
sudo install -m 0600 <(vault kv get -field=api_bearer_token secret/chrome-service) \
  /etc/t3-serve/chrome-service-token
for u in $(ls /etc/t3-serve/playwright-*.env 2>/dev/null | sed 's#.*/playwright-##;s#\.env##'); do
  sudo install -o "$u" -g "$u" -m 0600 \
    /etc/t3-serve/chrome-service-token /home/"$u"/.config/playwright/token
done

# Verify the next refresh succeeds for a user:
sudo systemctl start playwright-snapshot-refresh@<user>.service
sudo journalctl -u playwright-snapshot-refresh@<user>.service -n 20

Restore from a backup tarball

The 6-hourly backup CronJob writes tar -czf /backup/YYYY_MM_DD_HH.tar.gz -C /profile . to NFS at /srv/nfs/chrome-service-backup/. To restore the entire profile:

# 1. Scale chrome-service down so its lock is released.
kubectl -n chrome-service scale deploy/chrome-service --replicas=0

# 2. Mount the PVC in a helper pod and restore.
kubectl -n chrome-service apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata: {name: restore-helper, namespace: chrome-service}
spec:
  containers:
  - name: helper
    image: alpine:3.20
    command: [sleep, infinity]
    volumeMounts:
    - {name: profile, mountPath: /profile}
    - {name: backup, mountPath: /backup, readOnly: true}
  volumes:
  - name: profile
    persistentVolumeClaim: {claimName: chrome-service-profile-encrypted}
  - name: backup
    persistentVolumeClaim: {claimName: chrome-service-backup-host}
  restartPolicy: Never
EOF

kubectl -n chrome-service wait --for=condition=ready pod/restore-helper

kubectl -n chrome-service exec restore-helper -- sh -c '
  rm -rf /profile/chromium-data /profile/snapshots &&
  tar -xzf /backup/2026_06_04_18.tar.gz -C /profile
'

# 3. Cleanup helper, scale chrome-service back up.
kubectl -n chrome-service delete pod restore-helper
kubectl -n chrome-service scale deploy/chrome-service --replicas=1