infra/docs/runbooks/t3-version-bump.md

Runbook: bump the pinned t3 version (e.g. 0.0.24 → 0.0.25)

t3 on the devvm is pinned (T3_PIN, default 0.0.24) and held there by the t3-autoupdate enforcer. t3 is pre-1.0 and ships breaking changes between builds, so a bump is a deliberate, verified, reversible step — never an auto-update. This runbook makes it calm. Background: post-mortem 2026-06-09-t3-nightly-autoupdate-auth-outage.md.

What a bump actually touches

1. Pairing API — t3 renamed POST /api/auth/bootstrap → /api/auth/browser-session in 0.0.25. t3-dispatch is now version-agnostic (tries browser-session, falls back to bootstrap; see pairEndpoints in scripts/t3-dispatch/main.go), so 0.0.24↔0.0.25 needs no dispatch change. If a future build renames it again, add the new path to pairEndpoints, rebuild, redeploy first.
2. Schema — 0.0.25+ migrate every ~/.t3/userdata/state.sqlite forward (auth_pairing_links/auth_sessions role→scopes, +proof_key_thumbprint). This is a one-way door: a binary downgrade alone will NOT roll it back — you must restore the DB. Hence the mandatory pre-bump backup below.

Pre-flight (no downtime)

# 1. Confirm the dispatch already speaks the new version's pairing API.
#    Install the candidate to an isolated prefix (does NOT touch the global pin):
npm install --prefix /tmp/t3-cand t3@<new>           # e.g. t3@0.0.25
BIN=/tmp/t3-cand/node_modules/.bin/t3; D=$(mktemp -d)
"$BIN" serve --host 127.0.0.1 --port 3796 --base-dir "$D" >/tmp/cand.log 2>&1 &
CRED=$("$BIN" auth pairing create --base-dir "$D" --ttl 5m --json | sed -n 's/.*"credential":"\([^"]*\)".*/\1/p')
#    Try the dispatch's endpoints; one must give 200 + Set-Cookie: t3_session.
for ep in /api/auth/browser-session /api/auth/bootstrap; do
  curl -s -i -X POST -H 'Content-Type: application/json' -d "{\"credential\":\"$CRED\"}" \
    "http://127.0.0.1:3796$ep" | grep -iE 'HTTP/|set-cookie: t3_session'; done
kill %1; rm -rf "$D" /tmp/t3-cand
# If NO endpoint yields a t3_session cookie -> the API changed again; update
# pairEndpoints in main.go + rebuild the dispatch BEFORE proceeding.

# 2. Dispatch unit tests still green:
( cd ~/code/infra/scripts/t3-dispatch && go test ./... )

The bump

NEW=0.0.25
# 1. PRE-BUMP BACKUP — the rollback safety net. Per user, stop the serve (so the
#    copy is consistent + fast), copy state.sqlite, restart. Do the ACTIVE admin
#    instance last / from OUTSIDE its own t3 session (you can't restart the serve
#    you're running inside).
for u in $(awk -F= '!/^[[:space:]]*#/&&NF==2{gsub(/ /,"",$2);print $2}' /etc/ttyd-user-map | sort -u); do
  src=/home/$u/.t3/userdata/state.sqlite; [ -f "$src" ] || continue
  sudo systemctl stop t3-serve@$u
  sudo install -d -o "$u" -g "$u" -m700 /var/backups/t3-state/$u
  sudo cp -a "$src" /var/backups/t3-state/$u/state-prebump-$NEW-$(date +%Y%m%d-%H%M%S).sqlite
  sudo systemctl start t3-serve@$u
done
# (t3-backup-state also runs daily; this captures a guaranteed snapshot at T-0.)

# 2. Move the pin in BOTH places (keep them in sync):
sed -i "s/T3_PIN:-[0-9.]*/T3_PIN:-$NEW/" ~/code/infra/scripts/t3-autoupdate.sh \
                                          ~/code/infra/scripts/workstation/setup-devvm.sh
sudo install -m0755 ~/code/infra/scripts/t3-autoupdate.sh /usr/local/bin/t3-autoupdate

# 3. Run the enforcer. It installs t3@$NEW, then HEALTH-CHECKS the real pairing
#    handshake (mint -> browser-session/bootstrap -> t3_session). If pairing is
#    broken in $NEW, it AUTO-ROLLS-BACK to the previous version and exits non-zero.
sudo /usr/local/bin/t3-autoupdate    # restarts idle instances; defers active ones

# 4. Restart any instance the enforcer deferred (active agent), when it's idle.
#    The wizard/admin instance: restart from OUTSIDE its own session, or it picks
#    up $NEW on its next natural restart (the unit runs the global /usr/bin/t3).

Verify

for u in vbarzin emil.barzin ancaelena98; do
  curl -sI -H "X-authentik-username: $u" http://10.0.10.10:3780/ | grep -iE 'HTTP/|set-cookie: t3_session'
done   # each must be 302 + t3_session
t3 --version    # == $NEW

Rollback (if pairing breaks or $NEW misbehaves)

The enforcer auto-rolls-back the binary if its health-check fails. But if a problem surfaces after serves migrated their DBs forward, the binary alone won't fix it — restore the DBs:

sed -i "s/T3_PIN:-[0-9.]*/T3_PIN:-0.0.24/" ~/code/infra/scripts/t3-autoupdate.sh ~/code/infra/scripts/workstation/setup-devvm.sh
sudo install -m0755 ~/code/infra/scripts/t3-autoupdate.sh /usr/local/bin/t3-autoupdate
sudo npm i -g t3@0.0.24
for u in $(awk -F= '!/^[[:space:]]*#/&&NF==2{gsub(/ /,"",$2);print $2}' /etc/ttyd-user-map | sort -u); do
  bak=$(sudo ls -1t /var/backups/t3-state/$u/state-prebump-* 2>/dev/null | head -1)
  [ -n "$bak" ] || continue
  sudo systemctl stop t3-serve@$u
  sudo install -o "$u" -g "$u" -m600 "$bak" /home/$u/.t3/userdata/state.sqlite
  sudo rm -f /home/$u/.t3/userdata/state.sqlite-wal /home/$u/.t3/userdata/state.sqlite-shm
  sudo systemctl start t3-serve@$u
done
# verify 302 + t3_session as above

(The 2026-06-09 incident had no pre-bump backup, so rollback meant per-user sqlite surgery. With the backup, it's a restore.)