diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index da5f0f51..7b6a2c00 100755 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -38,7 +38,7 @@ Violations cause state drift, which causes future applies to break or silently r - **DNS**: `dns_type = "proxied"` (Cloudflare CDN) or `"non-proxied"` (direct A/AAAA). DNS records are auto-created — no need to edit `config.tfvars`. Smoke-test target: `echo.viktorbarzin.me` (auth=public, header-reflecting backend). - **Anubis PoW challenge** (`modules/kubernetes/anubis_instance/`): per-site reverse proxy that issues a 30-day JWT cookie after a tiny PoW solve. Use for **public, content-bearing sites without app-level auth** (blog, docs, wikis, static landing pages). Pattern: declare `module "anubis" { source = "../../modules/kubernetes/anubis_instance"; name = "X"; namespace = ...; target_url = "http://..svc.cluster.local" }`, then in `ingress_factory` set `service_name = module.anubis.service_name`, `port = module.anubis.service_port`, `anti_ai_scraping = false`. Shared ed25519 key in Vault `secret/viktor` -> `anubis_ed25519_key`; cookie scoped to `viktorbarzin.me` so one solve covers all Anubis-fronted subdomains. **DO NOT put Anubis in front of Git/API/WebDAV/CLI endpoints** — clients without JS can't solve PoW. **Replicas default to 1** because Anubis stores in-flight challenges in process memory; a challenge issued by pod A and solved against pod B errors with `store: key not found` (HTTP 500). Bumping replicas requires wiring a shared Redis store (TODO). For path-level carve-outs (e.g. wrongmove has `/` behind Anubis but `/api` direct, blog has `/net-diag.sh` direct), declare a second `ingress_factory` with `ingress_path = ["/"]` pointing at the bare backend service. Active on: blog (except `/net-diag.sh`), www, kms, travel, f1, cc, json, pb (privatebin), home (homepage), wrongmove (UI only). See `.claude/reference/patterns.md` "Anti-AI Scraping" for full layering. - **Docker images**: Always build for `linux/amd64`. SHA-tag rule is being phased out — see `docs/plans/2026-05-16-auto-upgrade-apps-{design,plan}.md`. New model: CI pushes `:latest` (optionally also `:<8-char-sha>` for traceability), Keel polls and triggers rollouts. Cache-staleness concern from the old rule is resolved at the nginx layer (URL-split — manifests pass through, blobs cached). Until Phase 1 of the migration completes (per the plan), follow the SHA-tag rule for new services to match existing pattern. -- **Private registry**: `forgejo.viktorbarzin.me/viktor/` (Forgejo packages, OAuth-style PAT auth). Use `image: forgejo.viktorbarzin.me/viktor/:` + `imagePullSecrets: [{name: registry-credentials}]`. Kyverno auto-syncs the Secret to all namespaces. Containerd `hosts.toml` on every node redirects to in-cluster Traefik LB `10.0.20.203` (with `skip_verify = true`, since the node dials Traefik by IP but the cert is for `forgejo.viktorbarzin.me`) to avoid hairpin NAT. That redirect covers **kubelet pulls** only — in-cluster pods (notably Woodpecker buildkit build pods pushing images) resolve `forgejo.viktorbarzin.me` via a CoreDNS `rewrite name exact ... traefik.traefik.svc.cluster.local` (Corefile in `stacks/technitium/modules/technitium/main.tf`), since they do NOT use the node containerd mirror; without it, buildkit pushes intermittently timed out on the public-IP hairpin (added 2026-06-04, beads code-yh33). **Was `.200` until 2026-06-01** — Traefik's 2026-05-30 move to its dedicated `.203` left this redirect pointing at the now-dead `.200:443`, silently breaking every *fresh* forgejo pull (cached images kept running, so it stayed hidden until a new image tag was pulled). Redirect source lives in `modules/create-template-vm/k8s-node-containerd-setup.sh` (new nodes) and `scripts/setup-forgejo-containerd-mirror.sh` (existing nodes). Push-side: viktor PAT in Vault `secret/ci/global/forgejo_push_token` (Forgejo container packages are scoped per-user; only the package owner can push, ci-pusher cannot write to viktor/*). Pull-side: cluster-puller PAT in Vault `secret/viktor/forgejo_pull_token`. Retention CronJob (`forgejo-cleanup` in `forgejo` ns, daily 04:00) keeps newest 10 versions + always `:latest`; integrity probed every 15min by `forgejo-integrity-probe` in `monitoring` ns (catalog walk + manifest HEAD on every blob). See `docs/plans/2026-05-07-forgejo-registry-consolidation-{design,plan}.md` for the migration history. Pull-through caches for upstream registries (DockerHub, GHCR, Quay, k8s.gcr, Kyverno) stay on the registry VM at `10.0.20.10` ports 5000/5010/5020/5030/5040 — the old port-5050 R/W private registry was decommissioned 2026-05-07. +- **Private registry**: `forgejo.viktorbarzin.me/viktor/` (Forgejo packages, OAuth-style PAT auth). Use `image: forgejo.viktorbarzin.me/viktor/:` + `imagePullSecrets: [{name: registry-credentials}]`. Kyverno auto-syncs the Secret to all namespaces. Containerd `hosts.toml` on every node redirects to in-cluster Traefik LB `10.0.20.203` (with `skip_verify = true`, since the node dials Traefik by IP but the cert is for `forgejo.viktorbarzin.me`) to avoid hairpin NAT. That redirect covers **kubelet pulls** only — in-cluster pods (notably Woodpecker buildkit build pods pushing images) resolve `forgejo.viktorbarzin.me` via a CoreDNS `rewrite name exact ... traefik.traefik.svc.cluster.local` (Corefile in `stacks/technitium/modules/technitium/main.tf`), since they do NOT use the node containerd mirror; without it, buildkit pushes intermittently timed out on the public-IP hairpin (added 2026-06-04, beads code-yh33). **Was `.200` until 2026-06-01** — Traefik's 2026-05-30 move to its dedicated `.203` left this redirect pointing at the now-dead `.200:443`, silently breaking every *fresh* forgejo pull (cached images kept running, so it stayed hidden until a new image tag was pulled). Redirect source lives in `modules/create-template-vm/k8s-node-containerd-setup.sh` (new nodes) and `scripts/setup-forgejo-containerd-mirror.sh` (existing nodes). Push-side: viktor PAT in Vault `secret/ci/global/forgejo_push_token` (Forgejo container packages are scoped per-user; only the package owner can push, ci-pusher cannot write to viktor/*). Pull-side: cluster-puller PAT in Vault `secret/viktor/forgejo_pull_token`. Retention CronJob (`forgejo-cleanup` in `forgejo` ns, daily 04:00) keeps newest 10 versions + always `:latest` + any buildkit `*cache*` tag (so `--cache-from`/`--cache-to` refs survive retention — added 2026-06-09); **went live (DRY_RUN=false) 2026-06-09** after verifying 0 running images on the delete set — the registry PVC is at its 50Gi autoresize ceiling on the HDD (we did NOT move it to SSD, see beads code-oflt), so live retention is what keeps it from filling. Integrity probed every 15min by `forgejo-integrity-probe` in `monitoring` ns (catalog walk + manifest HEAD on every blob). See `docs/plans/2026-05-07-forgejo-registry-consolidation-{design,plan}.md` for the migration history. Pull-through caches for upstream registries (DockerHub, GHCR, Quay, k8s.gcr, Kyverno) stay on the registry VM at `10.0.20.10` ports 5000/5010/5020/5030/5040 — the old port-5050 R/W private registry was decommissioned 2026-05-07. - **LinuxServer.io containers**: `DOCKER_MODS` runs apt-get on every start — bake slow mods into a custom image (`RUN /docker-mods || true` then `ENV DOCKER_MODS=`). Set `NO_CHOWN=true` to skip recursive chown that hangs on NFS mounts. - **Node memory changes**: When changing VM memory on any k8s node, update kubelet `systemReserved`, `kubeReserved`, and eviction thresholds accordingly. Config: `/var/lib/kubelet/config.yaml`. Template: `stacks/infra/main.tf`. Current values: systemReserved=512Mi, kubeReserved=512Mi, evictionHard=500Mi, evictionSoft=1Gi. - **Node OS disk tuning** (in `stacks/infra/main.tf`): kubelet `imageGCHighThresholdPercent=70` (was 85), `imageGCLowThresholdPercent=60` (was 80), ext4 `commit=60` in fstab (was default 5s), journald `SystemMaxUse=200M` + `MaxRetentionSec=3day`. diff --git a/.claude/reference/service-catalog.md b/.claude/reference/service-catalog.md index 633b227f..c8022fa1 100644 --- a/.claude/reference/service-catalog.md +++ b/.claude/reference/service-catalog.md @@ -32,7 +32,7 @@ |---------|-------------|-------| | k8s-dashboard | Kubernetes dashboard at `k8s.viktorbarzin.me`. **Forward-auth + auto-injected SA token** (apiserver OIDC blocked, see design §12). nginx token-injector (`dashboard_injector.tf`) maps `X-authentik-username` → the user's `dashboard-` SA token (ns admin + read-only on namespace-list/nodes only via `dashboard-nav-readonly` — no cross-tenant reads, `rbac/.../dashboard-sa.tf`; admins → cluster-admin SA) and sets `Authorization: Bearer` → no token-paste, dashboard auto-authenticates per user. Forward-auth admits `kubernetes-*` groups for this host (`stacks/authentik/admin-services-restriction.tf`). oauth2-proxy + `k8s-dashboard` OIDC app built but idle. | k8s-dashboard | | reverse-proxy | Generic reverse proxy | reverse-proxy | -| t3code | Multi-user coding-agent GUI at t3.viktorbarzin.me. `auth=required` (Authentik) → DevVM `t3-dispatch` service (`10.0.10.10:3780`, unprivileged user) maps `X-authentik-username` → that user's own `t3-serve@` instance (file perms enforced by uid; wizard→:3773, emo→:3774; unmapped→403) and **auto-injects the t3 session on first visit** (mints via the root `t3-mint` wrapper, scoped sudoers → `/api/auth/bootstrap` `t3_session` cookie). **Source of truth = `infra/scripts/workstation/roster.yaml`** (os_user → authentik_user/k8s_user/tier/namespaces); `roster_engine.py` (pytest-covered) derives desired state and `t3-provision-users` (hourly systemd timer) applies it — constrained accounts, additive per-tier groups, `t3-serve@` instances, and **regenerating** `/etc/ttyd-user-map` + `dispatch.json` (those two are now GENERATED — do not hand-edit). New non-admins inherit wizard's Claude config (machine-wide managed `claudeMd` in `/etc/claude-code/managed-settings.json` + per-user `~/.claude/{skills,rules}` symlinks seeded by `/etc/skel`) and get a **writable git-crypt-LOCKED** infra clone at `~/code` (code plaintext, secret files ciphertext). Tiers: admin / power-user (cluster-wide read-only) / namespace-owner. **Add a user:** one entry in `roster.yaml` → reconcile. Per-user OIDC kubeconfig, the `oidc-power-user-readonly` ClusterRole, and the Authentik `T3 Users` edge gate are applied (the gate is live — only `T3 Users` members reach t3); the emo cutover to his own locked clone is the remaining gated step. DevVM artifacts versioned in `infra/scripts/` (`t3-serve@.service`, `t3-provision-users` + `workstation/{roster.yaml,roster_engine.py,setup-devvm.sh,managed-settings.json,skel/}`, `t3-dispatch/`, `t3-mint`, `sudoers-t3-autopair`, `t3-autoupdate.*`); TF (`stacks/t3code`) owns only the ingress + Endpoints→:3780. **t3 binary tracks `nightly`** via `t3-autoupdate` (daily systemd timer; health-check + auto-rollback on a bad build; restarts only idle instances) — so new models (e.g. Opus 4.8) land as t3 ships them. Native app/app.t3.codes unsupported (cross-origin) — deferred until published. Design: `docs/plans/2026-06-01-t3-auto-provision-*`. | t3code | +| t3code | Multi-user coding-agent GUI at t3.viktorbarzin.me. `auth=required` (Authentik) → DevVM `t3-dispatch` service (`10.0.10.10:3780`, unprivileged user) maps `X-authentik-username` → that user's own `t3-serve@` instance (file perms enforced by uid; wizard→:3773, emo→:3774; unmapped→403) and **auto-injects the t3 session on first visit** (mints via the root `t3-mint` wrapper, scoped sudoers → `/api/auth/bootstrap` `t3_session` cookie). **Source of truth = `infra/scripts/workstation/roster.yaml`** (os_user → authentik_user/k8s_user/tier/namespaces); `roster_engine.py` (pytest-covered) derives desired state and `t3-provision-users` (hourly systemd timer) applies it — constrained accounts, additive per-tier groups, `t3-serve@` instances, and **regenerating** `/etc/ttyd-user-map` + `dispatch.json` (those two are now GENERATED — do not hand-edit). New non-admins inherit wizard's Claude config (machine-wide managed `claudeMd` in `/etc/claude-code/managed-settings.json` + per-user `~/.claude/{skills,rules}` symlinks seeded by `/etc/skel`) and get a **writable git-crypt-LOCKED** infra clone at `~/code` (code plaintext, secret files ciphertext). Tiers: admin / power-user (cluster-wide read-only) / namespace-owner. **Add a user:** one entry in `roster.yaml` → reconcile. Per-user OIDC kubeconfig, the `oidc-power-user-readonly` ClusterRole, and the Authentik `T3 Users` edge gate are applied (the gate is live — only `T3 Users` members reach t3); the emo cutover to his own locked clone is the remaining gated step. DevVM artifacts versioned in `infra/scripts/` (`t3-serve@.service`, `t3-provision-users` + `workstation/{roster.yaml,roster_engine.py,setup-devvm.sh,managed-settings.json,skel/}`, `t3-dispatch/`, `t3-mint`, `sudoers-t3-autopair`, `t3-autoupdate.*`); TF (`stacks/t3code`) owns only the ingress + Endpoints→:3780. **t3 binary is PINNED** (`T3_PIN`, currently `0.0.24`) — `t3-autoupdate` is a daily *enforcer* that re-asserts the pin (a no-op when correct; restarts only idle instances), NOT a nightly tracker. It used to track `nightly`, but on 2026-06-09 a nightly bump migrated every `~/.t3/state.sqlite` forward (`role`→`scopes`) and changed the bootstrap API, breaking pairing for ALL users (post-mortem `2026-06-09-t3-nightly-autoupdate-auth-outage.md`). t3 is pre-1.0, so moving the pin requires first verifying `t3-dispatch`'s bootstrap flow against the new build (expect 302 + `t3_session`). Pin set in `t3-autoupdate.sh` + `setup-devvm.sh` (keep in sync). Native app/app.t3.codes unsupported (cross-origin) — deferred until published. Design: `docs/plans/2026-06-01-t3-auto-provision-*`. | t3code | ## Active Use | Service | Description | Stack | @@ -116,7 +116,7 @@ | status-page | Status page | status-page | | plotting-book | Book plotting/world-building app | plotting-book | | tripit | Self-hosted TripIt-clone travel-itinerary PWA (FastAPI + SvelteKit SPA, same-origin). CNPG (`tripit` db, Vault static role `pg-tripit`) + RWX NFS trip-doc vault (`/srv/nfs/tripit-documents`) + RWO `proxmox-lvm-encrypted` personal-document vault `tripit-personal-documents` (passports/IDs — AES-256-GCM app-layer envelope, master key `DOCUMENT_ENCRYPTION_KEY` in `secret/tripit`). `auth=required` (Authentik forward-auth, reads `X-authentik-email`); second `auth=none` ingress on `/api/calendar` for HMAC-token-gated `.ics` feed. Email-ingest CronJob `tripit-ingest-plans` (`*/15`) is the SOLE inbound path — forward a booking to plans@viktorbarzin.me (catch-all → spam@), polled read-only and routed ONLY to a registered user / verified linked address (no default-owner fallback; strangers ignored), parsed by local LLM (`qwen3vl-4b`), and the sender is emailed the outcome (Added to trip / Couldn't import). Plus `tripit-poll-flights`, `tripit-run-reminders`, `tripit-transport-nudge`, `tripit-weather-brief`. (The old Gmail-scrape `tripit-ingest-mail` CronJob was removed 2026-06-05.) App secrets in Vault `secret/tripit`. | tripit | -| stem95su | STEM educational platform for **95. СУ „Проф. Иван Шишманов"** (Sofia school) at stem95su.viktorbarzin.me. Public **open** static site (`auth=none` — CrowdSec + ai-bot-block, no login). Stock `nginx:1.28-alpine` serving content **straight off PVE host NFS** `/srv/nfs/stem-site` (RWX `nfs_volume`, mounted read-only) — **NOT** image-baked, so the externally-authored (Gemini-exported) HTML/media updates with no rebuild; auto-backed-up offsite by `nfs-mirror`. **Content source = Google Drive folder "claude"** (id `1cmOI2jRyBJdnrVPgbr4kx2cx_4DY6pm_`, shared Valentina→vbarzin@gmail.com). **Deploy is ON-DEMAND, no scheduled job** (deliberate — short-term content, avoid rotting artifacts): mirror Drive→NFS via a throwaway `rclone/rclone` container using the existing `google_workspace` OAuth creds in Vault `secret/viktor` (`google_workspace_mcp_token_json`) → rsync to `/srv/nfs/stem-site` (empty-source guard). Just ask Claude to "sync stem95su from Drive" (recipe in claude-memory). Nextcloud "PVE NFS Pool"/rsync still works as a manual fallback. Dashboard `stem_board.html` served at `/` via a small nginx ConfigMap (`index`). No DB, no in-cluster secrets. Reference impl for the NFS-backed static-site pattern (see patterns.md). | stem95su | +| stem95su | STEM educational platform for **95. СУ „Проф. Иван Шишманов"** (Sofia school) at stem95su.viktorbarzin.me. Public **open** static site (`auth=none` — CrowdSec + ai-bot-block, no login). Stock `nginx:1.28-alpine` serving content **straight off PVE host NFS** `/srv/nfs/stem-site` (RWX `nfs_volume`, mounted read-only) — **NOT** image-baked, so the externally-authored (Gemini-exported) HTML/media updates with no rebuild; auto-backed-up offsite by `nfs-mirror`. **Content source = Google Drive folder "claude"** (id `1cmOI2jRyBJdnrVPgbr4kx2cx_4DY6pm_`, shared Valentina→vbarzin@gmail.com). **Deploy = scheduled mirror** (since 2026-06-09, reversed the earlier on-demand-only call once content went active): CronJob `stem95su-gdrive-sync` (`*/10`, `stacks/stem95su/gdrive-sync.tf`) mounts the content PVC RW and `rclone sync`s the Drive folder onto it (`docker.io/rclone/rclone:1.74.3`, `scope=drive.readonly` — Drive is READ-ONLY; empty-source guard + `--max-delete 25` so a partial listing can't wipe the site). rclone creds (OAuth refresh-token) in Vault `secret/stem95su` (`rclone_conf`) → ESO secret `stem95su-rclone`. **Requires the GCP OAuth app (project home-lab-1700868541205) published to "Production"** or the refresh token expires ~weekly (re-mint + `vault kv put secret/stem95su rclone_conf=…` after publishing); a dead token surfaces as a failed Job. Manual on-demand sync still possible (throwaway rclone container from devvm; recipe in claude-memory). Nextcloud "PVE NFS Pool"/rsync is a manual fallback. Dashboard `stem_board.html` served at `/` via a small nginx ConfigMap (`index`). No DB, no in-cluster secrets. Reference impl for the NFS-backed static-site pattern (see patterns.md). | stem95su | | trek | **TRIAL (2026-06-05)** — self-hosted group-trip planner (upstream [TREK](https://github.com/mauriceboe/TREK), `mauriceboe/trek:3.0.22`, AGPL-3.0). Solo evaluation behind Authentik forward-auth (`auth=required`) before deciding build-vs-adopt; covers collaborative trip planning + accommodation records + activities + per-person budget splitting on free OpenStreetMap (no paid maps key). SQLite + uploads on `proxmox-lvm-encrypted` (`trek-data-encrypted` 2Gi, `trek-uploads-encrypted` 5Gi). For the trial only: `ENCRYPTION_KEY` is TREK-auto-generated onto the data PVC and the bootstrap admin (`admin@trek.local`) is printed to pod logs — NO Vault/ESO wiring (graduation TODO: move key to `secret/trek` + ESO, add an app-level SQLite backup CronJob since host file-backup can't read the LUKS PVC, wire TREK↔Authentik OIDC). Pinned image, TF-managed (no CI/Keel). Availability-poll companion (Rallly) deferred. Teardown: `tg destroy` in `stacks/trek`. | trek | ## Cloudflare Domains diff --git a/docs/plans/2026-06-01-t3-auto-provision-design.md b/docs/plans/2026-06-01-t3-auto-provision-design.md index 1c34e8a8..ca9a7e1f 100644 --- a/docs/plans/2026-06-01-t3-auto-provision-design.md +++ b/docs/plans/2026-06-01-t3-auto-provision-design.md @@ -85,12 +85,14 @@ Replaces the in-cluster nginx `t3-dispatch` (the session-mint needs `sudo` + loc Per request (Authentik forward-auth has injected a trustworthy `X-authentik-username`): 1. Resolve `X-authentik-username` → OS user via `/etc/ttyd-user-map`. No mapping → **403**. -2. **Has a valid t3 session cookie?** → reverse-proxy (incl. WebSocket upgrade) to `127.0.0.1:`. (Steady state — the common path.) -3. **No cookie** (first visit / expired) → auto-pair: +2. **Has a valid t3 session cookie?** → reverse-proxy (incl. WebSocket upgrade) to `127.0.0.1:`. (Steady state — the common path.) Sub-requests (XHR/asset/WebSocket) take the cookie at face value; on a **top-level document navigation** the cookie is verified against the instance's `GET /api/auth/session` so a present-but-dead cookie doesn't slip through. +3. **No cookie, or an invalid cookie on a document navigation** (first visit / expired / server-side session wiped) → auto-pair: - `sudo -u t3 auth pairing create --base-dir /home//.t3 --ttl 5m --json` → one-time token. - exchange it at the instance's `POST /api/auth/bootstrap` → capture the returned `Set-Cookie`. - relay that `Set-Cookie` to the browser + `302 /`. Browser now holds the t3 session cookie → next request is the steady-state path. **Login → straight in.** +> **As-built note (2026-06-09):** the first implementation re-paired only on an *absent* cookie. After an auth-schema rollback wiped every server-side session, browsers still held live-looking-but-dead 30-day `t3_session` cookies, which the dispatcher proxied straight through → t3 rendered its pair page (the "all users must pair again" incident). Fixed by validating a present cookie via `/api/auth/session` and re-pairing on `authenticated:false` — **gated to document navigations** (`isDocumentNav`: trust `Sec-Fetch-Dest: document`, else fall back to `Accept: text/html`) so XHR/asset/WebSocket sub-requests are never answered with a `302`, and **fail-open** (proxy through) on any validation error so no new failure mode is introduced. See `scripts/t3-dispatch/main.go` (`sessionValid`, `isDocumentNav`) + `main_test.go`. + Implementation: a small reverse proxy that supports WebSocket upgrade (Go `httputil.ReverseProxy`, or Python aiohttp) — chosen at plan time. ### 4. Terraform — `stacks/t3code` shrinks diff --git a/docs/plans/2026-06-09-workstation-authentik-membership-plan.md b/docs/plans/2026-06-09-workstation-authentik-membership-plan.md new file mode 100644 index 00000000..1803e04d --- /dev/null +++ b/docs/plans/2026-06-09-workstation-authentik-membership-plan.md @@ -0,0 +1,469 @@ +# Workstation Membership v2 — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax. This is **infra** work: the engine tasks are real pytest TDD; the host/Authentik tasks "verify" via an idempotent re-run + a smoke check with expected output. Honor the Terraform-only rule for cluster/Authentik changes (`scripts/tg apply`); devvm host scripts are the accepted exception. Claim `host:devvm` before host mutations and `stack:authentik` before applying Authentik. + +**Goal:** Make the Authentik `T3 Users` group membership the single source of truth for who gets a devvm workstation account, identified by email; retire `roster.yaml`. + +**Architecture:** The provisioner reads `T3 Users` members from the Authentik API (read-only token) instead of `roster.yaml`. A pure engine derives the Linux `os_user` from each member's email (or an `os_user` Authentik attribute override) and produces the same desired-state shape v1 already applies. Workstation access stays fully decoupled from cluster RBAC (`k8s_users` untouched). wizard is special-cased as the admin/owner. + +**Tech Stack:** Python (pure engine, pytest) + Bash (provisioner) + `jq`/`curl` (Authentik API) + Terraform (`stacks/authentik`: read-only token, drop HCL members). + +**Design:** `infra/docs/plans/2026-06-09-workstation-authentik-membership-design.md`. + +--- + +## File structure + +- Modify: `infra/scripts/workstation/roster_engine.py` — add `derive_os_user()` + `roster_from_members()` (pure). +- Modify: `infra/scripts/workstation/test_roster_engine.py` — tests for the two new functions. +- Modify: `infra/scripts/t3-provision-users.sh` — source members from the Authentik API instead of `roster.yaml`. +- Modify: `infra/scripts/workstation/setup-devvm.sh` — drop the read-only Authentik token to `/etc/t3-serve/authentik-token`. +- Create: `infra/stacks/authentik/t3-provision-token.tf` — read-only service account + API token. +- Modify: `infra/stacks/authentik/t3-users.tf` — drop the HCL `users` list (membership becomes Authentik-managed). +- Delete: `infra/scripts/workstation/roster.yaml` (Task 7). +- Modify: `infra/.claude/reference/service-catalog.md`, `infra/docs/architecture/multi-tenancy.md` (Task 7). + +--- + +## Task 1: Engine — `derive_os_user()` + +**Files:** Modify `infra/scripts/workstation/roster_engine.py`; Test `infra/scripts/workstation/test_roster_engine.py` + +- [ ] **Step 1: Write the failing tests** (append to `test_roster_engine.py`) + +```python +# --- derive_os_user: email/attribute -> Linux username (v2) --- + +def test_derive_os_user_sanitizes_email_local_part(): + assert eng.derive_os_user("emil.barzin@gmail.com", None) == "emil_barzin" + + +def test_derive_os_user_attribute_overrides(): + assert eng.derive_os_user("emil.barzin@gmail.com", "emo") == "emo" + + +def test_derive_os_user_lowercases_and_replaces_unsafe_runs(): + assert eng.derive_os_user("Weird.Name+tag@x.com", None) == "weird_name_tag" + + +def test_derive_os_user_truncates_to_32(): + long = ("a" * 40) + "@x.com" + assert eng.derive_os_user(long, None) == "a" * 32 + + +def test_derive_os_user_blank_attribute_is_ignored(): + assert eng.derive_os_user("emil.barzin@gmail.com", "") == "emil_barzin" +``` + +- [ ] **Step 2: Run to verify they fail** + +Run: `cd infra/scripts/workstation && python3 -m pytest test_roster_engine.py -k derive_os_user -q` +Expected: FAIL — `AttributeError: module 'roster_engine' has no attribute 'derive_os_user'` + +- [ ] **Step 3: Implement** (add to `roster_engine.py`, after `RosterError`) + +```python +import re + +_MAX_USERNAME = 32 + + +def derive_os_user(email: str, os_user_attr: str | None) -> str: + """Linux username for a workstation member: the explicit `os_user` Authentik + attribute if set, else the email local-part sanitized to a valid username + (lowercase; runs of non [a-z0-9_-] -> '_'; stripped; <=32 chars).""" + if os_user_attr: + return os_user_attr + local = email.split("@", 1)[0].lower() + cleaned = re.sub(r"[^a-z0-9_-]+", "_", local).strip("_") + return cleaned[:_MAX_USERNAME] +``` + +- [ ] **Step 4: Run to verify they pass** + +Run: `python3 -m pytest test_roster_engine.py -k derive_os_user -q` +Expected: PASS (5 passed) + +- [ ] **Step 5: Commit** + +```bash +cd /home/wizard/code/infra +git add scripts/workstation/roster_engine.py scripts/workstation/test_roster_engine.py +git commit -m "workstation: engine derive_os_user (email/attribute -> Linux username)" +``` + +--- + +## Task 2: Engine — `roster_from_members()` + +Builds a `Roster` (the v1 type `derive_desired_state` already consumes) from the Authentik member list, so the existing tested derivation is reused unchanged. + +**Files:** Modify `roster_engine.py`; Test `test_roster_engine.py` + +- [ ] **Step 1: Write the failing tests** + +```python +# --- roster_from_members: Authentik members -> Roster (v2) --- + +MEMBERS = [ + {"email": "vbarzin@gmail.com", "os_user": "wizard"}, + {"email": "emil.barzin@gmail.com", "os_user": "emo"}, + {"email": "ancaelena98@gmail.com", "os_user": "ancamilea"}, +] +ADMINS = {"vbarzin@gmail.com"} + + +def test_roster_from_members_maps_identity_fields(): + r = eng.roster_from_members(MEMBERS, ADMINS) + u = r.users["emo"] + assert u.os_user == "emo" + assert u.authentik_user == "emil.barzin" # email local-part = t3-dispatch key + assert u.k8s_user == "emil.barzin@gmail.com" # email = identity + assert u.tier == "power-user" # non-admin + + +def test_roster_from_members_admin_by_email(): + r = eng.roster_from_members(MEMBERS, ADMINS) + assert r.users["wizard"].tier == "admin" + + +def test_roster_from_members_derives_os_user_when_no_override(): + r = eng.roster_from_members([{"email": "jane.doe@x.com", "os_user": None}], set()) + assert "jane_doe" in r.users + assert r.users["jane_doe"].tier == "power-user" + + +def test_roster_from_members_raises_on_os_user_collision(): + members = [{"email": "a@x.com", "os_user": "dup"}, {"email": "b@y.com", "os_user": "dup"}] + with pytest.raises(eng.RosterError, match="collision"): + eng.roster_from_members(members, set()) + + +def test_roster_from_members_reuses_derive_desired_state(): + r = eng.roster_from_members(MEMBERS, ADMINS) + ds = eng.derive_desired_state(r, {"wizard": 3773, "emo": 3774, "ancamilea": 3775}) + assert ds.dispatch["emil.barzin"] == {"os_user": "emo", "port": 3774} + assert ds.accounts["wizard"].groups == ("code-shared", "docker", "sudo") + assert ds.accounts["emo"].groups == () +``` + +- [ ] **Step 2: Run to verify they fail** + +Run: `python3 -m pytest test_roster_engine.py -k roster_from_members -q` +Expected: FAIL — `AttributeError: ... 'roster_from_members'` + +- [ ] **Step 3: Implement** (add to `roster_engine.py`) + +```python +def roster_from_members(members: list[dict], admin_emails: set[str]) -> Roster: + """Build a Roster from Authentik `T3 Users` members. Each member dict has + `email` and optional `os_user`. tier = admin iff the email is in admin_emails, + else power-user (a non-admin workstation: no groups, locked clone). Raises on + an os_user collision (two emails resolving to the same Linux username).""" + users: dict[str, User] = {} + for m in members: + email = m["email"] + os_user = derive_os_user(email, m.get("os_user")) + if os_user in users: + raise RosterError( + f"os_user collision: {email!r} and {users[os_user].k8s_user!r} " + f"both resolve to {os_user!r} (set an os_user attribute to disambiguate)" + ) + tier = "admin" if email in admin_emails else "power-user" + users[os_user] = User( + os_user=os_user, + authentik_user=email.split("@", 1)[0], + k8s_user=email, + tier=tier, + namespaces=(), + ) + return Roster(users) +``` + +- [ ] **Step 4: Run the whole suite** + +Run: `python3 -m pytest test_roster_engine.py -q && ruff check roster_engine.py test_roster_engine.py` +Expected: PASS (all, incl. the v1 tests) + ruff clean + +- [ ] **Step 5: Commit** + +```bash +git add scripts/workstation/roster_engine.py scripts/workstation/test_roster_engine.py +git commit -m "workstation: engine roster_from_members (Authentik members -> Roster, reuses derive)" +``` + +--- + +## Task 3: Read-only Authentik token (Terraform) + +**Files:** Create `infra/stacks/authentik/t3-provision-token.tf` + +- [ ] **Step 1: Write the resources** (service account + API token + view permissions) + +```hcl +# Read-only service account whose token the devvm provisioner uses to list +# "T3 Users" members. View-only: it can read users + groups, nothing else. +resource "authentik_user" "t3_provision" { + username = "t3-provision-bot" + name = "T3 Provision (read-only)" + type = "service_account" + path = "service-accounts" +} + +resource "authentik_token" "t3_provision" { + identifier = "t3-provision-readonly" + user = authentik_user.t3_provision.id + intent = "api" + description = "devvm t3-provision-users: read T3 Users membership" + retrieve_key = true +} + +# Global view permissions for the service account (users + groups read only). +resource "authentik_rbac_permission_user" "t3_provision_view_user" { + user = authentik_user.t3_provision.id + permission = "authentik_core.view_user" +} + +resource "authentik_rbac_permission_user" "t3_provision_view_group" { + user = authentik_user.t3_provision.id + permission = "authentik_core.view_group" +} + +output "t3_provision_token" { + value = authentik_token.t3_provision.key + sensitive = true +} +``` + +- [ ] **Step 2: Apply** (claim first) + +```bash +~/code/scripts/presence claim stack:authentik --purpose "v2: read-only t3-provision token" +export VAULT_ADDR=https://vault.viktorbarzin.me && vault login -method=oidc +cd /home/wizard/code/infra/stacks/authentik && ../../scripts/tg apply -target=authentik_user.t3_provision -target=authentik_token.t3_provision -target=authentik_rbac_permission_user.t3_provision_view_user -target=authentik_rbac_permission_user.t3_provision_view_group --non-interactive +``` +Expected: 4 added. (If the `authentik_rbac_permission_user` resource/permission codename differs in the installed provider, run `../../scripts/tg console` / check the provider docs and adjust the codename; verify in Step 3.) + +- [ ] **Step 3: Store the token in Vault + verify it is read-only** + +```bash +TOK=$(../../scripts/tg output -raw t3_provision_token) +vault kv patch secret/authentik t3_provision_token="$TOK" +# verify: can LIST T3 Users members... +curl -sk -H "Authorization: Bearer $TOK" "https://authentik.viktorbarzin.me/api/v3/core/users/?groups_by_name=T3%20Users" | jq -r '.results[].email' +# ...but CANNOT write (expect 403): +curl -sk -o /dev/null -w '%{http_code}\n' -X PATCH -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' -d '{"name":"x"}' "https://authentik.viktorbarzin.me/api/v3/core/users/14/" +``` +Expected: the three emails listed; the PATCH returns `403`. + +- [ ] **Step 4: Commit** + +```bash +git add stacks/authentik/t3-provision-token.tf +git commit -m "workstation: read-only Authentik token for the t3-provision membership query" +``` + +--- + +## Task 4: setup-devvm.sh — stage the token for the root provisioner + +**Files:** Modify `infra/scripts/workstation/setup-devvm.sh` + +- [ ] **Step 1: Add a token-staging step** (after step 6, before the final `log "OK"`). The hourly provisioner runs as root with no Vault token, so `setup-devvm.sh` (run by wizard, who can read Vault) drops it to a root-only file. + +```bash +# 8) stage the read-only Authentik token for the root provisioner's membership query. +if command -v vault >/dev/null; then + export VAULT_ADDR="${VAULT_ADDR:-https://vault.viktorbarzin.me}" + if tok="$(vault kv get -field=t3_provision_token secret/authentik 2>/dev/null)"; then + install -m 0600 /dev/stdin /etc/t3-serve/authentik-token <<<"$tok" + log "staged /etc/t3-serve/authentik-token (read-only Authentik API)" + else + log "WARN: t3_provision_token not in Vault -> Authentik membership query will be skipped" + fi +fi +``` + +- [ ] **Step 2: Run + verify** + +Run: `sudo bash /home/wizard/code/infra/scripts/workstation/setup-devvm.sh 2>&1 | grep -E 'authentik-token|OK'` then `sudo stat -c '%a %U' /etc/t3-serve/authentik-token` +Expected: "staged ... authentik-token" + `OK`; perms `600 root`. + +- [ ] **Step 3: Commit** + +```bash +git add scripts/workstation/setup-devvm.sh +git commit -m "workstation: setup-devvm.sh stages the read-only Authentik token (root-only)" +``` + +--- + +## Task 5: Provisioner — source members from Authentik (replace roster.yaml) + +**Files:** Modify `infra/scripts/t3-provision-users.sh` + +- [ ] **Step 1: Add a members-fetch + swap the engine call.** Replace the roster-read/derive block. Fetch members from Authentik (best-effort); build the members JSON `[{email, os_user}]`; pass to the engine via a new `--members-json` mode on `derive`. + +First extend the engine CLI (`roster_engine.py` `_main`): add `derive-members` that reads a members JSON + ports JSON + admin emails and emits the same desired-state JSON. + +```python +# in _main(), add a subparser: + pm = sub.add_parser("derive-members", help="desired state from an Authentik member list") + pm.add_argument("--members-json", required=True) + pm.add_argument("--ports-json", required=True) + pm.add_argument("--admin-emails", default="", help="comma-separated admin emails") + # ...in the dispatch: + if args.cmd == "derive-members": + with open(args.members_json, encoding="utf-8") as fh: + members = json.load(fh) + with open(args.ports_json, encoding="utf-8") as fh: + ports = json.load(fh) + admins = {e for e in args.admin_emails.split(",") if e} + ds = derive_desired_state(roster_from_members(members, admins), ports) + json.dump(_desired_state_to_dict(ds), sys.stdout, indent=2, sort_keys=True) + sys.stdout.write("\n") + return 0 +``` + +In `t3-provision-users.sh`, replace the `ROSTER`/validate/derive section with: + +```bash +AUTHENTIK_URL="${AUTHENTIK_URL:-https://authentik.viktorbarzin.me}" +TOKEN_FILE="${TOKEN_FILE:-/etc/t3-serve/authentik-token}" +T3_GROUP="${T3_GROUP:-T3 Users}" +ADMIN_EMAILS="${WORKSTATION_ADMIN_EMAILS:-vbarzin@gmail.com}" + +members_file="$(mktemp)"; trap 'rm -f "$ports_file" "$members_file" "${desired_file:-}"' EXIT +if [[ -r "$TOKEN_FILE" ]]; then + tok="$(cat "$TOKEN_FILE")" + if curl -sf -H "Authorization: Bearer $tok" --get \ + --data-urlencode "groups_by_name=$T3_GROUP" \ + "$AUTHENTIK_URL/api/v3/core/users/" \ + | jq -c '[.results[] | select(.is_active) | {email: .email, os_user: (.attributes.os_user // null)}]' \ + > "$members_file" && [[ -s "$members_file" ]]; then + : + else + log "WARN: Authentik membership query failed -> no membership change this run"; echo '[]' > "$members_file" + SKIP_RECONCILE=1 + fi +else + log "WARN: $TOKEN_FILE absent -> no membership change this run"; echo '[]' > "$members_file"; SKIP_RECONCILE=1 +fi + +if [[ "${SKIP_RECONCILE:-0}" == 1 ]]; then log "reconcile skipped (no Authentik membership)"; exit 0; fi + +desired_file="$(mktemp)" +python3 "$ENGINE" derive-members --members-json "$members_file" --ports-json "$ports_file" --admin-emails "$ADMIN_EMAILS" > "$desired_file" +jq -e . "$desired_file" >/dev/null || { echo "[t3-provision] derive-members produced invalid JSON" >&2; exit 1; } +``` + +(Keep steps 4-6 of the existing script — accounts/groups/clone/kubeconfig, .env/enable, regen map/dispatch — unchanged; they consume `$desired_file`.) + +- [ ] **Step 2: shellcheck + DRY_RUN** (with the staged token present) + +Run: `cd /home/wizard/code/infra/scripts && shellcheck -S warning t3-provision-users.sh && sudo DRY_RUN=1 bash t3-provision-users.sh 2>&1 | grep -iE 'clone|kubeconfig|reconcile|WARN'` +Expected: shellcheck clean; dry-run lists the current members, no account creations (all exist), "reconcile complete (DRY-RUN)". + +- [ ] **Step 3: Real run + verify it reproduces current state** + +Run: `sudo jq -S . /etc/t3-serve/dispatch.json > /tmp/d1; sudo DRY_RUN=0 bash t3-provision-users.sh >/dev/null 2>&1; sudo jq -S . /etc/t3-serve/dispatch.json > /tmp/d2; diff /tmp/d1 /tmp/d2 && echo SAME; id -nG emo` +Expected: `SAME` (dispatch content unchanged); emo groups unchanged. Redeploy: `sudo install -m0755 t3-provision-users.sh /usr/local/bin/t3-provision-users`. + +- [ ] **Step 4: Commit** + +```bash +git add scripts/t3-provision-users.sh scripts/workstation/roster_engine.py scripts/workstation/test_roster_engine.py +git commit -m "workstation: provisioner sources members from Authentik T3 Users (replaces roster.yaml)" +``` + +--- + +## Task 6: Authentik — Authentik-managed membership + legacy os_user attributes + +**Files:** Modify `infra/stacks/authentik/t3-users.tf`; set user attributes via API. + +- [ ] **Step 1: Set the legacy os_user attributes** (the 3 existing accounts don't derive from their emails). Read-merge-write so existing attributes are preserved (Authentik PATCH replaces the `attributes` dict). + +```bash +export VAULT_ADDR=https://vault.viktorbarzin.me +TOK=$(vault kv get -field=tf_api_token secret/authentik) +A=https://authentik.viktorbarzin.me/api/v3 +set_os_user() { # $1=username $2=os_user + local pk attrs + pk=$(curl -sk -H "Authorization: Bearer $TOK" "$A/core/users/?username=$1" | jq '.results[0].pk') + attrs=$(curl -sk -H "Authorization: Bearer $TOK" "$A/core/users/$pk/" | jq -c --arg o "$2" '.attributes + {os_user:$o}') + curl -sk -X PATCH -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \ + -d "{\"attributes\":$attrs}" "$A/core/users/$pk/" | jq -r '.username + " os_user=" + .attributes.os_user' +} +set_os_user "vbarzin@gmail.com" wizard +set_os_user "emil.barzin@gmail.com" emo +set_os_user "ancaelena98@gmail.com" ancamilea +``` +Expected: three lines confirming `os_user=` each. + +- [ ] **Step 2: Drop the HCL `users` list** so membership is Authentik-managed. Edit `t3-users.tf`: remove the `users = [...]` argument from `resource "authentik_group" "t3_users"` (keep the `data "authentik_user"` lookups removed too if now unused). Leave the group resource (name only). + +```hcl +resource "authentik_group" "t3_users" { + name = "T3 Users" + # Membership is managed in Authentik (UI/API), not Terraform — the devvm + # provisioner reconciles workstation accounts from this group's members. +} +``` + +- [ ] **Step 3: Apply + verify members unchanged** + +```bash +cd /home/wizard/code/infra/stacks/authentik && ../../scripts/tg apply -target=authentik_group.t3_users --non-interactive +curl -sk -H "Authorization: Bearer $TOK" "$A/core/groups/?search=T3%20Users" | jq -r '.results[0].users_obj[].username' +``` +Expected: apply shows the group updated (no member change / the `users` field no longer managed); the 3 members still listed. + +- [ ] **Step 4: Commit** + +```bash +git add stacks/authentik/t3-users.tf +git commit -m "workstation: T3 Users membership is Authentik-managed (drop HCL member list)" +``` + +--- + +## Task 7: Retire roster.yaml + update docs + +**Files:** Delete `infra/scripts/workstation/roster.yaml`; modify `service-catalog.md`, `multi-tenancy.md`. + +- [ ] **Step 1: Confirm nothing reads roster.yaml anymore** + +Run: `grep -rn 'roster.yaml\|roster_engine.*roster\b' /home/wizard/code/infra/scripts /home/wizard/code/infra/docs | grep -v 'load_roster\|test_\|design.md\|-plan.md'` +Expected: no live references in the provisioner (the engine keeps `load_roster` for tests, that's fine). + +- [ ] **Step 2: Delete it + update the service-catalog t3code row** — change "Source of truth = roster.yaml" to "Source of truth = the Authentik `T3 Users` group (members → accounts via the read-only API token); `os_user` from the email or a per-user `os_user` attribute". Update the multi-tenancy Workstation section's "single source of truth" line likewise. + +```bash +git rm scripts/workstation/roster.yaml +# (edit service-catalog.md + multi-tenancy.md per above) +``` + +- [ ] **Step 3: Commit** + +```bash +git add scripts/workstation/roster.yaml .claude/reference/service-catalog.md docs/architecture/multi-tenancy.md +git commit -m "workstation: retire roster.yaml — Authentik T3 Users group is the membership SSoT" +``` + +--- + +## Task 8: End-to-end smoke (add + remove a throwaway member) + +- [ ] **Step 1: Add a throwaway test member** to `T3 Users` in Authentik (a test user, or temporarily add an existing one), set no `os_user` attribute. Run `sudo /usr/local/bin/t3-provision-users` and confirm an account `` is created (`id `), with a locked `~/code` (secret file shows `GITCRYPT`) and `~/.kube/config`. +- [ ] **Step 2: Remove the test member** from the group; run the reconcile; confirm they drop out of `/etc/ttyd-user-map` + `dispatch.json` (the reversible cut). Leave `userdel` to the gated offboarding runbook. +- [ ] **Step 3: Verify the 3 real users are intact** — `id emo` (groups unchanged), emo/ancamilea/wizard still in `dispatch.json`, their `t3-serve@` active, emo's locked clone + ancamilea's intact. + +--- + +## Self-review + +- **Spec coverage:** Authentik-as-SSoT (Tasks 5,6) · email identity + os_user derive/override (Tasks 1,6) · provisioner reads the API (Task 5) · read-only token for the root timer (Tasks 3,4) · roster.yaml retires (Task 7) · k8s_users/cluster untouched (no task touches it) · wizard special-cased (admin_emails, Task 2). All covered. +- **Type consistency:** `derive_os_user(email, os_user_attr)` and `roster_from_members(members, admin_emails)` used consistently; `members` dicts are `{email, os_user}`; reuses the existing `User`/`Roster`/`derive_desired_state`/`DesiredState`. +- **apiserver-OIDC:** out of scope here (kubectl auth method only) — flagged in the design; the generic kubeconfig task is unchanged from v1. +- **Open risk:** the `authentik_rbac_permission_user` resource name / permission codenames may differ in the installed provider version (Task 3) — Step 3 verifies read-works/write-403 and says to adjust if needed. diff --git a/docs/post-mortems/2026-06-09-t3-nightly-autoupdate-auth-outage.md b/docs/post-mortems/2026-06-09-t3-nightly-autoupdate-auth-outage.md new file mode 100644 index 00000000..e8f0d2d5 --- /dev/null +++ b/docs/post-mortems/2026-06-09-t3-nightly-autoupdate-auth-outage.md @@ -0,0 +1,138 @@ +# Post-Mortem: t3 Nightly Auto-Update (0.0.25) Migrated `state.sqlite` Forward → mint/pairing Broke for All Devvm Users + +## Summary + +The devvm t3 auto-updater (`t3-autoupdate.timer`) pulled the `t3@nightly` +build `0.0.25-nightly.20260608.497`. That build ran two forward schema +migrations on every per-user `~/.t3/userdata/state.sqlite` (renaming +`role`→`scopes` in `auth_pairing_links` + `auth_sessions`, adding +`proof_key_thumbprint`) **and** changed the bootstrap API. The result was a +binary-vs-schema mismatch that broke `t3-mint` (pairing-credential issuance) +for **all** users — every fresh login landed on the t3 pairing prompt instead +of an authenticated session. + +## Impact + +- **Who:** every devvm t3 user — `wizard` (Viktor), `emo`, `ancamilea`. +- **What:** `t3 auth pairing create` failed (`AuthControlPlaneError: + Failed to create pairing link` → `PersistenceSqlError` on + `auth_pairing_links`), so `t3-dispatch` auto-pair returned 500/502 and the + browser showed the pairing prompt. Existing *already-authenticated* sessions + kept working (validated against `auth_sessions`, not the pairing path). +- **When:** ~13:56 (bad nightly installed) → ~15:16 (all users verified 302). +- **Trigger of the report:** Anca could not log in ("gets the pair prompt, + session broken"). + +## Timeline (devvm clock) + +- **13:56** — `t3-provision-users` step 5b ran `systemctl enable --now + t3-autoupdate.timer`. The timer is `OnCalendar=04:00 … Persistent=true`; + `--now` + a missed 04:00 schedule fired the daily job **immediately**. +- **13:56** — updater installed `t3@nightly` = `0.0.25-nightly.20260608.497` + (was `0.0.24`). The `GET / → 200` health-check **passed** (it never + exercises mint/bootstrap), so no auto-rollback. It restarted *idle* serves + (emo) onto 0.0.25 and deferred *active* ones (wizard, ancamilea). +- **~14:38** — `t3-mint` (now global 0.0.25) ran migrations 31 + (`AuthAuthorizationScopes`) + 32 (`AuthPairingProofKeyThumbprint`) against + each `state.sqlite` it touched → schemas moved to "level 32". +- **~14:40** — first recovery action rolled the **binary** back to `0.0.24`. + This did **not** help: the DBs were still at level 32, so the level-30 + binary's INSERT hit `no column named role` / `NOT NULL constraint failed: + scopes`. (Downgrading a binary after a forward migration is not a rollback.) +- **~15:01–15:16** — diagnosed the binary-vs-schema mismatch, confirmed + `0.0.25` *stable* is **also** dispatch-incompatible (auto-pair → 502, the + bootstrap API moved), pinned to `0.0.24`, reset the two new users' disposable + DBs, surgically reverted wizard's two auth tables to level 30. All three + users verified 302 + `Set-Cookie: t3_session`. + +## Root Cause + +Three compounding factors: + +1. **Auto-tracking a pre-1.0 tool's nightly.** `t3-autoupdate.sh` ran + `npm i -g t3@nightly`. t3 ships breaking schema-migration and bootstrap-API + changes between builds; our `t3-dispatch` (Go) speaks a fixed bootstrap + contract (`POST /api/auth/bootstrap {"credential":…}` → `Set-Cookie`). +2. **`enable --now` on a `Persistent=true` timer.** The provisioner's + re-assertion of the timer didn't just *arm* the schedule — it fired the + missed daily job on the spot, mid-afternoon, with users active. +3. **A health-check that proves nothing about auth.** The smoke test only + probes `GET / → 200`. The 0.0.25 server answers 200 while its pairing/mint + path is incompatible, so the "auto-rollback on bad build" never triggered. + +Forward migrations + a binary downgrade = a DB the old binary can't write. +`state.sqlite` also holds the precious projection tables (session history), so +a blanket "delete and re-pair" was only safe for the brand-new users. + +## Detection + +User report (Anca on the pairing prompt). No alert fired — the auto-updater's +own health-check is the only automated gate and it passed. **Gap:** nothing +monitors the end-to-end pairing flow. + +## Fixes & Mitigations + +### 1. Pin t3, stop tracking nightly (DONE) + +`infra/scripts/t3-autoupdate.sh` is now a **pinned-version enforcer**: +`T3_PIN="${T3_PIN:-0.0.24}"`, `npm i -g "t3@$T3_PIN"`. It re-asserts the pin +(a no-op when already correct) instead of chasing nightly. Unit `Description`s +updated. To move the pin: bump `T3_PIN` **and first** verify `t3-dispatch`'s +bootstrap flow against the new build (`curl` the dispatch → expect 302 + +`Set-Cookie: t3_session`). + +### 2. Drop `--now` from the provisioner (DONE) + +`infra/scripts/t3-provision-users.sh` step 5b now runs `systemctl enable +t3-autoupdate.timer` (no `--now`) — it arms the 04:00 schedule without firing a +missed job immediately. + +### 3. Pinned install at machine setup (DONE) + +`infra/scripts/workstation/setup-devvm.sh` installs `t3@$T3_PIN` directly, so a +fresh box has the pinned t3 immediately rather than depending on the enforcer's +first run. + +### 4. Recovery actions taken on the host (DONE) + +- Global `t3` rolled to `0.0.24`; enforcer redeployed + timer re-enabled + (verified the enforcer is a no-op at the pin). +- New users (`emo` 0 threads, `ancamilea` 1 trivial thread): `state.sqlite` + parked aside; serve restarted → fresh level-30 DB. +- `wizard` (96 threads, and the serve hosting the recovery session — cannot be + restarted): the two auth tables were atomically rebuilt to the level-30 + schema (copied from a fresh DB) and migration records 31/32 removed. + `auth_sessions` had 0 rows and the 0.0.24 serve never reads `scopes`, so the + live session and all projection history were untouched. Backup: + `/home/wizard/.t3/userdata/auth-backup-*.sql`. + +### 5. End-to-end pairing health-check (DEFERRED) + +The smoke test should exercise mint→bootstrap→cookie, not just `GET /`. Not +done here (the pin makes it moot for the known-good build); needed before the +enforcer is ever pointed at a new version. A blackbox probe on the dispatch +auto-pair (expect 302 + `t3_session`) would have alerted within minutes. + +## Lessons + +- **Don't auto-track a pre-1.0 tool's nightly.** Pin to a known-good, + contract-verified build; upgrades are a deliberate, tested act. +- **`enable --now` on a `Persistent=true` timer fires the missed job now.** + Use plain `enable` to arm a schedule without a surprise immediate run. +- **A liveness probe (`GET /`) is not a readiness/correctness probe.** If a + feature (auth/pairing) can break while `/` stays 200, the health-check must + exercise that feature or it gives false confidence. +- **A binary downgrade is not a schema rollback.** Once a forward migration + runs, the data is migrated; the old binary now mismatches its own DB. +- **Separate disposable state from precious state before resetting.** t3's + `state.sqlite` mixes ephemeral auth (`auth_pairing_links`, `auth_sessions`) + with precious history (`projection_*`); surgical table-level repair + preserved 8k+ messages that a blanket reset would have destroyed. + +## References + +- `infra/scripts/t3-autoupdate.sh` (pinned enforcer), `.service`, `.timer` +- `infra/scripts/t3-provision-users.sh` step 5b +- `infra/scripts/workstation/setup-devvm.sh` step 2b +- `infra/.claude/reference/service-catalog.md` (t3 serving layer) +- Backup of wizard's pre-repair auth tables: `/home/wizard/.t3/userdata/auth-backup-*.sql` diff --git a/scripts/t3-autoupdate.service b/scripts/t3-autoupdate.service index d3306da7..7b043f13 100644 --- a/scripts/t3-autoupdate.service +++ b/scripts/t3-autoupdate.service @@ -1,5 +1,5 @@ [Unit] -Description=Track latest t3 nightly (health-checked, idle-only restart) +Description=Enforce pinned t3 version (health-checked, idle-only restart) After=network-online.target Wants=network-online.target diff --git a/scripts/t3-autoupdate.sh b/scripts/t3-autoupdate.sh index 962f3fc4..836605f0 100644 --- a/scripts/t3-autoupdate.sh +++ b/scripts/t3-autoupdate.sh @@ -1,20 +1,30 @@ #!/usr/bin/env bash -# Track the latest t3 nightly — with a health-check + auto-rollback (lesson from -# the Keel auto-update incidents: never blindly trust a new build) and idle-only -# restarts (never kill an in-flight coding session). Runs as root via the unit. +# Enforce the PINNED t3 version ($T3_PIN) across the box — NOT "latest/nightly". +# t3 is pre-1.0 and ships breaking schema-migration + bootstrap-API changes between +# builds that our t3-dispatch can't follow blind. 2026-06-09: a nightly auto-update +# (0.0.25) migrated every ~/.t3 state.sqlite forward (auth_pairing_links/auth_sessions +# role->scopes) AND changed the bootstrap API, breaking mint/pairing for ALL users. +# So we PIN; this unit just re-asserts the pin (a no-op when already correct) with a +# health-check + auto-rollback and idle-only restarts (never kill an in-flight session). +# To move the pin: bump T3_PIN AND first verify t3-dispatch's bootstrap flow against the +# new build (curl the dispatch -> expect 302 + Set-Cookie t3_session). See post-mortem +# 2026-06-09-t3-nightly-autoupdate-auth-outage.md. +# CAVEAT: the health-check below only probes GET / (200) — it does NOT exercise the +# mint/bootstrap/pairing path, so it will NOT catch an auth regression on its own. set -uo pipefail +T3_PIN="${T3_PIN:-0.0.24}" # known-good, t3-dispatch-compatible (2026-06-09 post-mortem) LOG() { logger -t t3-autoupdate "$*"; echo "t3-autoupdate: $*"; } ver() { t3 --version 2>/dev/null | awk '{print $NF}' | sed 's/^v//'; } -before=$(ver); LOG "current: ${before:-unknown}" -npm i -g t3@nightly >/dev/null 2>&1 || { LOG "npm install failed; staying on ${before:-current}"; exit 0; } +before=$(ver); LOG "current: ${before:-unknown}; pin: $T3_PIN" +npm i -g "t3@$T3_PIN" >/dev/null 2>&1 || { LOG "npm install failed; staying on ${before:-current}"; exit 0; } after=$(ver) if [[ -z "$after" || "$after" == "$before" ]]; then - LOG "already latest (${before:-?}); nothing to do"; exit 0 + LOG "already at pin $T3_PIN (${before:-?}); nothing to do"; exit 0 fi -LOG "installed $after (was $before); health-checking…" +LOG "re-pinned to $after (was $before); health-checking…" # Health-check the NEW binary on a throwaway port/base-dir before trusting it. SMOKE_PORT=3799; SMOKE_DIR=$(mktemp -d) diff --git a/scripts/t3-autoupdate.timer b/scripts/t3-autoupdate.timer index a59135f7..ccdbd4c6 100644 --- a/scripts/t3-autoupdate.timer +++ b/scripts/t3-autoupdate.timer @@ -1,5 +1,5 @@ [Unit] -Description=Daily t3 nightly auto-update +Description=Daily t3 pinned-version enforcer (re-asserts T3_PIN; no-op when correct) [Timer] OnCalendar=*-*-* 04:00:00 diff --git a/scripts/t3-dispatch/main.go b/scripts/t3-dispatch/main.go index cebcc119..401b0edb 100644 --- a/scripts/t3-dispatch/main.go +++ b/scripts/t3-dispatch/main.go @@ -59,6 +59,60 @@ func lookup(ak string) (entry, bool) { return e, ok } +// mintToken mints a one-time pairing token for osUser via the scoped sudoers +// entry (the dispatch service can invoke nothing else). Indirected through a var +// so tests can stub the privileged exec. +var mintToken = func(osUser string) ([]byte, error) { + return exec.Command("sudo", "-n", "/usr/local/bin/t3-mint", osUser).Output() +} + +var sessionClient = &http.Client{Timeout: 5 * time.Second} + +// sessionValid asks the user's instance whether the presented t3_session cookie +// is still valid. Server-side sessions can be wiped/expired independently of the +// 30-day cookie (e.g. an auth-schema rollback drops every session row), leaving +// the browser with a live-looking but dead cookie. Fails OPEN: any error/non-200/ +// parse failure returns true so the request still proxies — a re-pair is forced +// only on a definitive authenticated:false. +func sessionValid(e entry, c *http.Cookie) bool { + req, err := http.NewRequest(http.MethodGet, + fmt.Sprintf("http://127.0.0.1:%d/api/auth/session", e.Port), nil) + if err != nil { + return true + } + req.AddCookie(c) + resp, err := sessionClient.Do(req) + if err != nil { + return true + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return true + } + var s struct { + Authenticated bool `json:"authenticated"` + } + if json.NewDecoder(resp.Body).Decode(&s) != nil { + return true + } + return s.Authenticated +} + +// isDocumentNav reports whether r is a top-level browser document navigation, as +// opposed to an XHR/fetch/asset/WebSocket sub-request. Only such requests are +// safe to answer with a re-pair 302 — redirecting a sub-resource would corrupt +// the SPA's fetch/WebSocket contract. Trust Sec-Fetch-Dest when present (all +// modern browsers send it); fall back to the Accept header otherwise. +func isDocumentNav(r *http.Request) bool { + if r.Method != http.MethodGet { + return false + } + if dest := r.Header.Get("Sec-Fetch-Dest"); dest != "" { + return dest == "document" + } + return strings.Contains(r.Header.Get("Accept"), "text/html") +} + // autoPair mints a one-time pairing token for the user's instance (as that OS // user, via the scoped sudoers entry) and exchanges it at the instance's // /api/auth/bootstrap, relaying the returned t3_session Set-Cookie to the browser. @@ -66,7 +120,7 @@ func autoPair(e entry, w http.ResponseWriter, r *http.Request) { // t3-mint (root, via scoped sudoers) validates the OS user is in // /etc/ttyd-user-map, then mints as that user. The dispatch service itself // runs unprivileged and can invoke nothing else. - out, err := exec.Command("sudo", "-n", "/usr/local/bin/t3-mint", e.OsUser).Output() + out, err := mintToken(e.OsUser) if err != nil { log.Printf("mint for %s failed: %v", e.OsUser, err) http.Error(w, "pairing mint failed", http.StatusInternalServerError) @@ -111,7 +165,16 @@ func handler(w http.ResponseWriter, r *http.Request) { http.Error(w, "no t3 instance provisioned for this user", http.StatusForbidden) return } - if _, err := r.Cookie(cookieName); err != nil { + c, err := r.Cookie(cookieName) + if err != nil { + autoPair(e, w, r) + return + } + // A present cookie can still be server-side-invalid (sessions wiped/expired + // while the 30-day cookie lingers). On a top-level navigation, verify it and + // re-pair if dead — otherwise the instance just renders its pair page. Gated + // to document navs so we never 302 an XHR/asset/WebSocket sub-request. + if isDocumentNav(r) && !sessionValid(e, c) { autoPair(e, w, r) return } diff --git a/scripts/t3-dispatch/main_test.go b/scripts/t3-dispatch/main_test.go new file mode 100644 index 00000000..81ca26a9 --- /dev/null +++ b/scripts/t3-dispatch/main_test.go @@ -0,0 +1,200 @@ +package main + +import ( + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "testing" +) + +func portOf(t *testing.T, ts *httptest.Server) int { + t.Helper() + u, err := url.Parse(ts.URL) + if err != nil { + t.Fatalf("parse %s: %v", ts.URL, err) + } + p, err := strconv.Atoi(u.Port()) + if err != nil { + t.Fatalf("port %s: %v", u.Port(), err) + } + return p +} + +func TestIsDocumentNav(t *testing.T) { + cases := []struct { + name string + method string + headers map[string]string + want bool + }{ + {"GET sec-fetch-dest document", "GET", map[string]string{"Sec-Fetch-Dest": "document"}, true}, + {"GET accept html (no sec-fetch)", "GET", map[string]string{"Accept": "text/html,application/xhtml+xml"}, true}, + {"GET xhr empty dest beats accept", "GET", map[string]string{"Sec-Fetch-Dest": "empty", "Accept": "text/html"}, false}, + {"GET json", "GET", map[string]string{"Accept": "application/json"}, false}, + {"POST html", "POST", map[string]string{"Accept": "text/html"}, false}, + {"GET no headers", "GET", map[string]string{}, false}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + r, _ := http.NewRequest(c.method, "/", nil) + for k, v := range c.headers { + r.Header.Set(k, v) + } + if got := isDocumentNav(r); got != c.want { + t.Errorf("isDocumentNav = %v, want %v", got, c.want) + } + }) + } +} + +func sessionServer(status int, body string) *httptest.Server { + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/api/auth/session" { + http.NotFound(w, r) + return + } + w.WriteHeader(status) + _, _ = w.Write([]byte(body)) + })) +} + +func TestSessionValid(t *testing.T) { + ck := &http.Cookie{Name: cookieName, Value: "x"} + + t.Run("authenticated true -> valid", func(t *testing.T) { + ts := sessionServer(200, `{"authenticated":true}`) + defer ts.Close() + if !sessionValid(entry{Port: portOf(t, ts)}, ck) { + t.Fatal("want valid (true) for authenticated:true") + } + }) + t.Run("authenticated false -> invalid", func(t *testing.T) { + ts := sessionServer(200, `{"authenticated":false}`) + defer ts.Close() + if sessionValid(entry{Port: portOf(t, ts)}, ck) { + t.Fatal("want invalid (false) for authenticated:false") + } + }) + t.Run("500 -> fail-open valid", func(t *testing.T) { + ts := sessionServer(500, `boom`) + defer ts.Close() + if !sessionValid(entry{Port: portOf(t, ts)}, ck) { + t.Fatal("want fail-open true on 500") + } + }) + t.Run("malformed json -> fail-open valid", func(t *testing.T) { + ts := sessionServer(200, `not json`) + defer ts.Close() + if !sessionValid(entry{Port: portOf(t, ts)}, ck) { + t.Fatal("want fail-open true on unparseable body") + } + }) + t.Run("unreachable -> fail-open valid", func(t *testing.T) { + ts := sessionServer(200, `{"authenticated":false}`) + p := portOf(t, ts) + ts.Close() // nothing listening now + if !sessionValid(entry{Port: p}, ck) { + t.Fatal("want fail-open true on connection refused") + } + }) +} + +// fakeInstance serves the three endpoints the dispatcher touches: the session +// check, the bootstrap exchange, and a catch-all standing in for the proxied app. +func fakeInstance(authenticated bool, bootstrapCalled *bool) *httptest.Server { + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/api/auth/session": + if authenticated { + _, _ = w.Write([]byte(`{"authenticated":true}`)) + } else { + _, _ = w.Write([]byte(`{"authenticated":false}`)) + } + case "/api/auth/bootstrap": + if bootstrapCalled != nil { + *bootstrapCalled = true + } + http.SetCookie(w, &http.Cookie{Name: cookieName, Value: "fresh", Path: "/"}) + _, _ = w.Write([]byte(`{"authenticated":true}`)) + default: + _, _ = w.Write([]byte("APP")) + } + })) +} + +func setTable(port int) { + mu.Lock() + table = map[string]entry{"vbarzin": {OsUser: "wizard", Port: port}} + mu.Unlock() +} + +func TestHandlerRepairsOnInvalidCookieDocNav(t *testing.T) { + called := false + ts := fakeInstance(false, &called) + defer ts.Close() + setTable(portOf(t, ts)) + + orig := mintToken + mintToken = func(string) ([]byte, error) { return []byte(`{"credential":"tok"}`), nil } + defer func() { mintToken = orig }() + + r := httptest.NewRequest("GET", "/", nil) + r.Header.Set("X-authentik-username", "vbarzin@gmail.com") + r.Header.Set("Sec-Fetch-Dest", "document") + r.AddCookie(&http.Cookie{Name: cookieName, Value: "stale"}) + w := httptest.NewRecorder() + + handler(w, r) + + if w.Code != http.StatusFound { + t.Fatalf("stale cookie on doc-nav should re-pair (302), got %d body=%q", w.Code, w.Body.String()) + } + if !called { + t.Fatal("expected bootstrap to be called during re-pair") + } + cookies := w.Result().Cookies() + if len(cookies) == 0 || cookies[0].Value != "fresh" { + t.Fatalf("expected fresh t3_session relayed, got %+v", cookies) + } +} + +func TestHandlerProxiesOnValidCookie(t *testing.T) { + ts := fakeInstance(true, nil) + defer ts.Close() + setTable(portOf(t, ts)) + + r := httptest.NewRequest("GET", "/", nil) + r.Header.Set("X-authentik-username", "vbarzin@gmail.com") + r.Header.Set("Sec-Fetch-Dest", "document") + r.AddCookie(&http.Cookie{Name: cookieName, Value: "good"}) + w := httptest.NewRecorder() + + handler(w, r) + + if w.Code != http.StatusOK || w.Body.String() != "APP" { + t.Fatalf("valid cookie should proxy (200 APP), got %d %q", w.Code, w.Body.String()) + } +} + +func TestHandlerProxiesXHREvenIfCookieInvalid(t *testing.T) { + called := false + ts := fakeInstance(false, &called) // session would say invalid, but XHR must NOT be re-paired + defer ts.Close() + setTable(portOf(t, ts)) + + r := httptest.NewRequest("GET", "/api/threads", nil) + r.Header.Set("X-authentik-username", "vbarzin@gmail.com") + r.Header.Set("Sec-Fetch-Dest", "empty") // XHR/fetch, not a document nav + r.AddCookie(&http.Cookie{Name: cookieName, Value: "stale"}) + w := httptest.NewRecorder() + + handler(w, r) + + if called { + t.Fatal("must NOT re-pair (302) a non-document sub-request — would corrupt the SPA fetch contract") + } + if w.Code != http.StatusOK || w.Body.String() != "APP" { + t.Fatalf("XHR should proxy through, got %d %q", w.Code, w.Body.String()) + } +} diff --git a/scripts/t3-provision-users.sh b/scripts/t3-provision-users.sh index 33edf3fd..37689153 100644 --- a/scripts/t3-provision-users.sh +++ b/scripts/t3-provision-users.sh @@ -95,6 +95,39 @@ EOF log "wrote OIDC kubeconfig -> $user:~/.kube/config" } +# Idempotently set KEY=VALUE in a t3-serve env file, PRESERVING other lines — so writing +# T3_PORT never clobbers an injected CLAUDE_CODE_OAUTH_TOKEN, and vice-versa. Mode 0600. +env_set() { + local file="$1" key="$2" val="$3" + if [[ "$DRY_RUN" == 1 ]]; then echo "[dry-run] set $key -> $file"; return 0; fi + install -d -m 0755 "$(dirname "$file")" + if [[ -f "$file" ]] && grep -q "^${key}=" "$file"; then + grep -qx "${key}=${val}" "$file" || sed -i "s|^${key}=.*|${key}=${val}|" "$file" + else + printf '%s=%s\n' "$key" "$val" >> "$file" + fi + chmod 600 "$file" +} + +# Share the admin's Claude subscription with a non-admin: inject CLAUDE_CODE_OAUTH_TOKEN +# (the staged long-lived token) into their t3-serve env — ONLY if they have neither their +# own ~/.claude/.credentials.json (own login) nor an existing token. Never clobbers. The +# agent picks it up when its t3-serve@ instance (re)starts. +install_user_claude_token() { + local user="$1" home envf tok + local token_file="${CLAUDE_TOKEN_FILE:-/etc/t3-serve/claude-oauth-token}" + home="$(getent passwd "$user" | cut -d: -f6)" + [[ -z "$home" ]] && return 0 + [[ -f "$home/.claude/.credentials.json" ]] && return 0 # has own login -> leave it + [[ -r "$token_file" ]] || return 0 + envf="${ENVDIR:-/etc/t3-serve}/$user.env" + grep -q '^CLAUDE_CODE_OAUTH_TOKEN=' "$envf" 2>/dev/null && return 0 # already shared + if [[ "$DRY_RUN" == 1 ]]; then echo "[dry-run] share Claude token -> $envf"; return 0; fi + tok="$(cat "$token_file")" + env_set "$envf" CLAUDE_CODE_OAUTH_TOKEN "$tok" + log "shared Claude token -> $user (t3-serve env; restart needed to take effect)" +} + [[ $EUID -eq 0 ]] || { echo "t3-provision-users: must run as root" >&2; exit 1; } for bin in python3 jq; do command -v "$bin" >/dev/null || { echo "missing $bin" >&2; exit 1; }; done [[ -f "$ROSTER" && -f "$ENGINE" ]] || { echo "roster/engine not under $WORKSTATION_DIR" >&2; exit 1; } @@ -144,21 +177,27 @@ while IFS=$'\t' read -r os_user tier shell groups_csv; do log "add $os_user -> group $g"; run gpasswd -a "$os_user" "$g" >/dev/null done fi - if [[ "$tier" != admin ]]; then # non-admins: locked ~/code clone + OIDC kubeconfig + if [[ "$tier" != admin ]]; then # non-admins: locked clone + kubeconfig + shared Claude token install_locked_clone "$os_user" install_user_kubeconfig "$os_user" + install_user_claude_token "$os_user" fi done < <(jq -r '.accounts[] | [.os_user, .tier, .shell, (.groups|join(","))] | @tsv' "$desired_file") # 5) per-user .env (sticky port) + enable t3-serve@ while IFS=$'\t' read -r os_user port; do envf="$ENVDIR/$os_user.env" - if [[ ! -f "$envf" ]] || ! grep -qx "T3_PORT=$port" "$envf"; then - run bash -c "printf 'T3_PORT=%s\n' '$port' > '$envf'" - fi + env_set "$envf" T3_PORT "$port" # update-or-append; preserves CLAUDE_CODE_OAUTH_TOKEN id "$os_user" >/dev/null 2>&1 && run systemctl enable --now "t3-serve@$os_user.service" >/dev/null 2>&1 || true done < <(jq -r '.ports | to_entries[] | [.key, .value] | @tsv' "$desired_file") +# 5b) machine-wide (once, not per-user): keep the t3 pinned-version ENFORCER enabled (it +# re-asserts T3_PIN daily; a no-op when already correct). NOT --now: with Persistent=true +# a `--now` enable fires the missed daily job IMMEDIATELY, which on 2026-06-09 pulled a +# breaking nightly mid-day and took out auth for everyone. `enable` (no --now) just arms +# the 04:00 schedule; fresh boxes get t3 from setup-devvm.sh's pinned install, not here. +run systemctl enable t3-autoupdate.timer >/dev/null 2>&1 || true + # 6) regenerate /etc/ttyd-user-map + dispatch.json from the desired state (SSoT: # a roster entry removed here DISAPPEARS, which is what the offboarding cut relies on) if [[ "$DRY_RUN" == 1 ]]; then diff --git a/scripts/workstation/setup-devvm.sh b/scripts/workstation/setup-devvm.sh index 9e0284b6..faf7b7bc 100755 --- a/scripts/workstation/setup-devvm.sh +++ b/scripts/workstation/setup-devvm.sh @@ -33,6 +33,16 @@ if [[ $need_node -eq 1 ]]; then fi command -v claude >/dev/null || { log "npm: installing @anthropic-ai/claude-code"; npm install -g @anthropic-ai/claude-code >/dev/null; } +# 2b) t3 (the per-user coding surface) — PINNED, never nightly/latest. t3 is pre-1.0 and +# ships breaking auth-schema + bootstrap-API changes our t3-dispatch can't follow blind +# (2026-06-09 outage: a nightly auto-update broke pairing for ALL users). The daily +# t3-autoupdate ENFORCER re-asserts this same pin; install it here so a fresh box has t3 +# immediately. Keep T3_PIN in sync with t3-autoupdate.sh. +T3_PIN="${T3_PIN:-0.0.24}" +if [[ "$(t3 --version 2>/dev/null | awk '{print $NF}' | sed 's/^v//')" != "$T3_PIN" ]]; then + log "npm: installing pinned t3@$T3_PIN"; npm install -g "t3@$T3_PIN" >/dev/null +fi + # 3) kubelogin (kubectl oidc-login) system-wide — NOT the apt 'kubelogin' (= Azure tool) if [[ ! -x /usr/local/bin/kubelogin ]]; then log "kubelogin: installing int128/kubelogin" @@ -77,4 +87,21 @@ if [[ -d "$ADMIN_CODE" ]]; then log "hardened $ADMIN_CODE (o-rx — not world-readable)" fi +# 8) stage the shared Claude subscription OAuth token (long-lived sk-ant-oat01) to a +# root-readable file the provisioner injects into non-admins' t3-serve env, so they +# share the admin's Claude subscription (only those without their own ~/.claude login). +if command -v vault >/dev/null; then + export VAULT_ADDR="${VAULT_ADDR:-https://vault.viktorbarzin.me}" + # setup-devvm runs as root (no ~/.vault-token); borrow the admin's token to read Vault. + if [[ -z "${VAULT_TOKEN:-}" && -r /home/wizard/.vault-token ]]; then + VAULT_TOKEN="$(cat /home/wizard/.vault-token)"; export VAULT_TOKEN + fi + if claude_tok="$(vault kv get -field=claude_oauth_token secret/workstation 2>/dev/null)"; then + install -m 0600 /dev/stdin /etc/t3-serve/claude-oauth-token <<<"$claude_tok" + log "staged /etc/t3-serve/claude-oauth-token (shared Claude subscription)" + else + log "WARN: secret/workstation claude_oauth_token absent -> non-admins won't share Claude auth" + fi +fi + log "OK (idempotent)" diff --git a/stacks/forgejo/cleanup.tf b/stacks/forgejo/cleanup.tf index add332d3..4c019c83 100644 --- a/stacks/forgejo/cleanup.tf +++ b/stacks/forgejo/cleanup.tf @@ -4,9 +4,17 @@ # it's per-user runtime state inside the Forgejo DB. Driving retention from # a CronJob hitting the public API keeps the policy versioned in this repo. # -# Auth: a write:package PAT belonging to ci-pusher (same user that pushes -# from CI). DELETE on packages requires write:package scope. PAT lives in -# Vault at secret/viktor/forgejo_cleanup_token. +# Auth: a write:package PAT belonging to VIKTOR (the package OWNER). PAT +# lives in Vault at secret/viktor/forgejo_cleanup_token. +# +# CORRECTION 2026-06-09: this previously said the PAT belonged to ci-pusher. +# That was wrong and silently broke retention — Forgejo container packages +# are scoped per-user, so ci-pusher gets HTTP 403 on DELETE of viktor/* +# (the dry-run only does GETs, which DO work, so the 403 stayed hidden until +# the first live run). DELETE requires a write:package PAT owned by viktor. +# forgejo_cleanup_token is therefore set to viktor's write:package PAT (today +# the same value as secret/ci/global/forgejo_push_token). IF that push token +# is ever regenerated, re-mirror it here or retention silently 403s again. data "vault_kv_secret_v2" "forgejo_viktor" { mount = "secret" @@ -14,8 +22,12 @@ data "vault_kv_secret_v2" "forgejo_viktor" { } locals { - # Flip to false after first 7 days of dry-run logs look correct. - forgejo_cleanup_dry_run = true + # Activated 2026-06-09 after verifying a dry-run delete list against all + # running viktor/* images cluster-wide: 0 running images on the delete set + # (would prune 317 stale versions, keeping newest 10 + latest + cache tags). + # Live retention is what keeps the registry PVC from filling on the HDD + # (we deliberately did NOT move Forgejo to SSD — see beads code-oflt). + forgejo_cleanup_dry_run = false } resource "kubernetes_config_map" "forgejo_cleanup_script" { diff --git a/stacks/forgejo/files/cleanup.sh b/stacks/forgejo/files/cleanup.sh index 61bb7a96..5efc6cce 100644 --- a/stacks/forgejo/files/cleanup.sh +++ b/stacks/forgejo/files/cleanup.sh @@ -2,8 +2,13 @@ # Forgejo container-package retention. # # For each container package owned by ${FORGEJO_OWNER}, keep newest -# ${KEEP_LAST_N} versions + always keep tag "latest". Deletes the rest via +# ${KEEP_LAST_N} versions + always keep tag "latest" + always keep any +# buildkit cache tag (matches "cache", e.g. tripit:cache — these back +# --cache-from/--cache-to and must survive retention or every build is a +# cold rebuild). Deletes the rest via # DELETE /api/v1/packages/{owner}/container/{name}/{version}. +# (Note: an 8-char SHA tag is pure hex and cannot contain "cache" — 'h' is +# not a hex digit — so the cache match never catches a real image tag.) # # DRY_RUN=true logs what would be deleted but issues no DELETE calls. # @@ -72,9 +77,11 @@ for NAME in $NAMES; do N_VERSIONS=$(jq 'length' "$TMPDIR/$NAME.json") echo "[$NAME] $N_VERSIONS version(s)" - # Build the keep set: top $KEEP + anything tagged 'latest'. + # Build the keep set: top $KEEP + always 'latest' + any buildkit cache tag. jq -r --argjson keep "$KEEP" ' - [.[0:$keep][].version] + [.[] | select(.version == "latest") | .version] + [.[0:$keep][].version] + + [.[] | select(.version == "latest") | .version] + + [.[] | select(.version | test("cache"; "i")) | .version] | unique | .[] ' "$TMPDIR/$NAME.json" > "$TMPDIR/$NAME.keep" diff --git a/stacks/forgejo/main.tf b/stacks/forgejo/main.tf index 2778f555..e1b8c351 100644 --- a/stacks/forgejo/main.tf +++ b/stacks/forgejo/main.tf @@ -9,7 +9,7 @@ resource "kubernetes_namespace" "forgejo" { name = "forgejo" labels = { "istio-injection" : "disabled" - tier = local.tiers.edge + tier = local.tiers.edge "keel.sh/enrolled" = "true" } } @@ -94,7 +94,7 @@ resource "kubernetes_deployment" "forgejo" { fs_group = 1000 } container { - name = "forgejo" + name = "forgejo" # Pinned to 11.0.14 (latest 11.x as of 2026-05-12) — was on # floating `:11`. On 2026-05-24T15:35:37Z Keel force-policy # rewrote the tag from `11.0.14 → 1.18` (Gitea-era Forgejo @@ -168,13 +168,19 @@ resource "kubernetes_deployment" "forgejo" { name = "data" mount_path = "/data" } + # Bumped 1Gi -> 3Gi 2026-06-09: Forgejo was OOMKilled (exit 137) + # under registry-push load from in-cluster CI builds (tripit + # buildkit pushes large layers into the OCI registry). VPA + # upperBound reads ~1.5Gi, but that's suppressed by the 1Gi cap it + # kept OOMing against — size for the push spike, not steady-state. + # requests=limits (Guaranteed QoS) per the repo memory convention. resources { requests = { cpu = "15m" - memory = "1Gi" + memory = "3Gi" } limits = { - memory = "1Gi" + memory = "3Gi" } } port { @@ -202,7 +208,7 @@ resource "kubernetes_deployment" "forgejo" { metadata[0].annotations["keel.sh/match-tag"], metadata[0].annotations["keel.sh/trigger"], metadata[0].annotations["keel.sh/pollSchedule"], # KYVERNO_LIFECYCLE_V2 - spec[0].template[0].spec[0].container[0].image, # KEEL_IGNORE_IMAGE — Keel manages tag updates + spec[0].template[0].spec[0].container[0].image, # KEEL_IGNORE_IMAGE — Keel manages tag updates metadata[0].annotations["kubernetes.io/change-cause"], metadata[0].annotations["deployment.kubernetes.io/revision"], spec[0].template[0].metadata[0].annotations["keel.sh/update-time"], diff --git a/stacks/kyverno/modules/kyverno/resource-governance.tf b/stacks/kyverno/modules/kyverno/resource-governance.tf index 855128f1..bcbaece0 100644 --- a/stacks/kyverno/modules/kyverno/resource-governance.tf +++ b/stacks/kyverno/modules/kyverno/resource-governance.tf @@ -15,6 +15,15 @@ locals { governance_tiers = ["0-core", "1-cluster", "2-gpu", "3-edge", "4-aux"] excluded_namespaces = ["kube-system", "metallb-system", "kyverno", "calico-system", "calico-apiserver"] + + # GPU-priority injection exclude list. Adds `tts` to the base set so the + # `inject-gpu-workload-priority` policy does NOT stamp the immich-equal + # gpu-workload (1,200,000) priority on Chatterbox-TTS pods. Chatterbox is a + # best-effort off-peak batch tenant on the shared T4: it must keep its + # tier-2-gpu (600,000) priority so it is ALWAYS the pod evicted under GPU-node + # pressure, never immich-ml/frigate/llama-swap. See the tts stack + # (stacks/tts/) + docs/plans/2026-06-08-chatterbox-tts-infra.md §3. + gpu_priority_excluded_namespaces = concat(local.excluded_namespaces, ["tts"]) } # ----------------------------------------------------------------------------- @@ -905,7 +914,10 @@ resource "kubectl_manifest" "mutate_gpu_priority" { any = [ { resources = { - namespaces = local.excluded_namespaces + # tts added so Chatterbox-TTS keeps tier-2-gpu priority (it's a + # best-effort off-peak batch tenant — must be evicted first, + # not promoted to immich-equal gpu-workload). See locals above. + namespaces = local.gpu_priority_excluded_namespaces } } ] diff --git a/stacks/stem95su/gdrive-sync.tf b/stacks/stem95su/gdrive-sync.tf new file mode 100644 index 00000000..9f870205 --- /dev/null +++ b/stacks/stem95su/gdrive-sync.tf @@ -0,0 +1,119 @@ +# Automatic Google Drive -> site sync (added 2026-06-09; supersedes the +# earlier on-demand-only model now that content is actively maintained). +# +# A CronJob mirrors the READ-ONLY Drive folder "claude" (servable content in +# subfolder "stem claude/files/") onto the NFS content volume every 10 min via +# rclone. rclone is delta-aware: an unchanged run lists ~33 files' metadata and +# transfers nothing, so the schedule is cheap (not a 24MB re-download). nginx +# keeps serving the same volume read-only; updates appear within ~5s (actimeo). +# +# Drive is treated strictly READ-ONLY: scope=drive.readonly and rclone only ever +# reads the remote (sync gdrive: -> /data), never writes back. +# +# TOKEN LONGEVITY: the GCP OAuth app (project home-lab-1700868541205) MUST be +# published to "Production" or its refresh token expires ~weekly and this job +# fails. After publishing, re-mint the token and refresh +# `secret/stem95su.rclone_conf`. A failed run surfaces as a failed Job. + +resource "kubernetes_manifest" "rclone_external_secret" { + manifest = { + apiVersion = "external-secrets.io/v1beta1" + kind = "ExternalSecret" + metadata = { + name = "stem95su-rclone" + namespace = kubernetes_namespace.stem95su.metadata[0].name + } + spec = { + refreshInterval = "1h" + secretStoreRef = { + name = "vault-kv" + kind = "ClusterSecretStore" + } + target = { name = "stem95su-rclone" } + data = [{ + secretKey = "rclone.conf" + remoteRef = { + key = "stem95su" + property = "rclone_conf" + } + }] + } + } + depends_on = [kubernetes_namespace.stem95su] +} + +resource "kubernetes_cron_job_v1" "gdrive_sync" { + metadata { + name = "stem95su-gdrive-sync" + namespace = kubernetes_namespace.stem95su.metadata[0].name + labels = { run = "stem95su", component = "gdrive-sync" } + } + spec { + schedule = "*/10 * * * *" + concurrency_policy = "Forbid" + successful_jobs_history_limit = 2 + failed_jobs_history_limit = 3 + job_template { + metadata {} + spec { + backoff_limit = 1 + ttl_seconds_after_finished = 86400 + template { + metadata { labels = { run = "stem95su", component = "gdrive-sync" } } + spec { + restart_policy = "OnFailure" + container { + name = "rclone" + image = "docker.io/rclone/rclone:1.74.3" + # Mirror Drive folder -> /data. Guard: hard-fail on auth/list error + # (so an expired token is visible); skip quietly if the source is + # empty / missing the dashboard (never wipe the live site); + # --max-delete caps catastrophic deletes from a partial listing. + command = ["/bin/sh", "-c", <<-EOT + set -eu + cp /config/rclone.conf /tmp/rc.conf + SRC="gdrive:stem claude/files" + LIST=$(rclone --config /tmp/rc.conf lsf "$SRC" --files-only) || { echo "FATAL: Drive list failed (auth/network)"; exit 1; } + N=$(printf '%s\n' "$LIST" | grep -c . || true) + if [ "$N" -lt 1 ] || ! printf '%s\n' "$LIST" | grep -qx "stem_board.html"; then + echo "GUARD: source N=$N / stem_board.html missing -- skipping, site untouched"; exit 0 + fi + echo "source OK ($N files) -- mirroring to /data" + rclone --config /tmp/rc.conf sync "$SRC" /data --exclude ".DS_Store" --fast-list --transfers 4 --max-delete 25 -v + EOT + ] + resources { + requests = { cpu = "10m", memory = "64Mi" } + limits = { memory = "192Mi" } + } + volume_mount { + name = "rclone-config" + mount_path = "/config" + read_only = true + } + volume_mount { + name = "content" + mount_path = "/data" + } + } + volume { + name = "rclone-config" + secret { secret_name = "stem95su-rclone" } + } + volume { + name = "content" + persistent_volume_claim { + claim_name = module.nfs_content.claim_name + } + } + } + } + } + } + } + lifecycle { + # KYVERNO_LIFECYCLE_V1: Kyverno admission webhook mutates dns_config with ndots=2 + ignore_changes = [spec[0].job_template[0].spec[0].template[0].spec[0].dns_config] + } + depends_on = [kubernetes_manifest.rclone_external_secret] +} diff --git a/stacks/tripit/main.tf b/stacks/tripit/main.tf index 197a80d0..6e79ed8c 100644 --- a/stacks/tripit/main.tf +++ b/stacks/tripit/main.tf @@ -65,6 +65,15 @@ locals { SMTP_USER = "spam@viktorbarzin.me" SMTP_FROM = "plans@viktorbarzin.me" PUBLIC_BASE_URL = "https://tripit.viktorbarzin.me" + # Narrator audio (ADR-0004): Chatterbox via the in-cluster `tts` stack. + # OpenAI-compatible /v1/audio/speech; the bake POSTs best-effort synth + # requests, so a down/Pending Chatterbox is a clean skip (browser-TTS + # fallback), never a bake error. ClusterIP-only → no token. Note: the mode + # is `openai_compatible` (tripit renamed it from `chatterbox`); TTS_MODEL is + # still the `chatterbox` family string tripit sends as the OpenAI `model`. + TTS_MODE = "openai_compatible" + TTS_BASE_URL = "http://chatterbox-tts.tts.svc.cluster.local:8000" + TTS_MODEL = "chatterbox" } } diff --git a/stacks/tts/README.md b/stacks/tts/README.md new file mode 100644 index 00000000..f5a972dd --- /dev/null +++ b/stacks/tts/README.md @@ -0,0 +1,149 @@ +# tts — Chatterbox TTS (tripit narration) + +In-cluster text-to-speech for tripit's "Tour guide". Runs the +[devnen/Chatterbox-TTS-Server](https://github.com/devnen/Chatterbox-TTS-Server) +(Resemble AI Chatterbox under an OpenAI-compatible HTTP server) as a single +Deployment + ClusterIP Service `chatterbox-tts.tts.svc.cluster.local:8000`, +requesting **one time-slice** of the shared Tesla T4 (`nvidia.com/gpu: 1`). + +Full design + rationale (Option-A off-peak control, OOM analysis, ADR links): +`docs/plans/2026-06-08-chatterbox-tts-infra.md` (in the tripit-tour-guide repo) +and `infra/docs/post-mortems/2026-06-02-immich-ml-ttl-gpu-oom-recruiter.md`. + +> This stack mirrors `infra/stacks/llama-cpp/`. The scaffolding files +> (`backend.tf`, `providers.tf`, `cloudflare_provider.tf`, `tiers.tf`, +> `.terraform.lock.hcl`) are **generated by Terragrunt** on `init` and are +> git-ignored — only `main.tf`, `terragrunt.hcl` and this README are tracked. + +--- + +## What this stack creates + +- `kubernetes_namespace.tts` — tier `2-gpu`, keel-enrolled, istio off. +- `module.nfs_models` — RWX NFS-SSD PVC at `/srv/nfs-ssd/chatterbox`, mounted at + `/data` (predefined voices, narrator reference WAVs, **and** the HuggingFace + model cache via `HF_HOME=/data/hf_cache`, so weights download once and persist + across the per-window pod recreation). +- `kubernetes_config_map.chatterbox_config` — `config.yaml`: `server.port=8004`, + `model.repo_id=chatterbox-multilingual`, `tts_engine.device=cuda`, voices / + reference paths under `/data`. +- `kubernetes_deployment.chatterbox` — **starts at `replicas=0`**; the off-peak + CronJobs own the replica count at runtime. `TTS_BF16=off` (T4 = Turing, no + bf16). `priority_class_name=tier-2-gpu` (the polite-tenant demotion). +- `kubernetes_service.chatterbox` — ClusterIP, **`port 8000 → targetPort 8004`** + so tripit's default `TTS_BASE_URL` works unchanged. Prometheus scrape + annotations. +- **Off-peak control** (SA + Role + RoleBinding + 3 CronJobs): see below. + +## Off-peak control (Option A — window + free-VRAM gate) + +The T4 is time-sliced with **zero VRAM isolation** (post-mortem 2026-06-02), so +`nvidia.com/gpu: 1` buys a scheduling turn, NOT memory. Chatterbox must only +allocate VRAM when the card is actually free. Implemented as three CronJobs +(all `Europe/London`), each a `bitnami/kubectl` pod using the namespace SA: + +| CronJob | Schedule (default) | Action | +|---|---|---| +| `chatterbox-window-up` | `0 2 * * *` | **Preflight**: scrape `gpu_pod_memory_used_bytes` from `gpu-pod-exporter.nvidia.svc:80/metrics`, compute `free = 16 GiB − Σused`; scale to **1 only if** `free ≥ vram_free_floor_bytes`. | +| `chatterbox-vram-guard` | `*/5 2-5 * * *` | **Guard**: every 5 min in-window, scale to **0** if `free < floor` (a resident woke; yield the card mid-bake). | +| `chatterbox-window-down` | `0 6 * * *` | **Window end**: scale to **0** unconditionally. | + +`tripit`'s bake is best-effort + cached-forever (ADR-0002/0004) — a skipped or +aborted window simply backfills on the next one. No latency SLA. + +### The free-VRAM floor — YOU MUST MEASURE THIS + +`var.vram_free_floor_bytes` defaults to **6 GiB** (a conservative guess: +~4 GiB assumed multilingual FP16 peak + ~2 GiB headroom for the +read→`cudaMalloc` race). **The real T4 peak of `chatterbox-multilingual` is not +published upstream.** Capture it during the first bake: + +```bash +# while a real synth is running on the freed T4: +kubectl -n monitoring exec deploy/prometheus -- \ + promtool query instant http://localhost:9090 \ + 'sum(gpu_pod_memory_used_bytes{namespace="tts"})' +# or read the gauge straight from the exporter: +kubectl -n nvidia exec ds/gpu-pod-exporter -- \ + sh -c 'curl -s localhost:9401/metrics | grep "namespace=\"tts\""' +``` + +Then set the floor to `measured_peak + ~2 GiB` (pass `-var` or add to the stack +tfvars). If the peak is too high to coexist even off-peak, switch +`model.repo_id` in `main.tf` to `chatterbox` (English, lighter) or +`chatterbox-turbo`, or escalate to Option B (scale `immich-machine-learning` to +0 for the window). + +--- + +## Build + push the image (do this BEFORE the first apply) + +`devnen/Chatterbox-TTS-Server` ships **no published image** — build from the +repo's **cu128** target (matches the cluster's pinned 570.195.03 / CUDA 12.8 +driver) and push to the private Forgejo registry. The devvm docker is pre-authed +to `forgejo.viktorbarzin.me`. Run on the devvm (large CUDA image — needs disk + +bandwidth): + +```bash +# 1. Clone the upstream server repo (outside the monorepo). +git clone https://github.com/devnen/Chatterbox-TTS-Server /tmp/chatterbox-tts-server +cd /tmp/chatterbox-tts-server + +# 2. Build the cu128 variant (Dockerfile.cu128 — PyTorch 2.9.0+cu128, the target +# the repo's docker-compose-cu128.yml uses) for linux/amd64. +SHA="$(git rev-parse --short=8 HEAD)" +docker build \ + --platform linux/amd64 \ + --build-arg RUNTIME=nvidia \ + -f Dockerfile.cu128 \ + -t forgejo.viktorbarzin.me/viktor/chatterbox-tts:latest \ + -t "forgejo.viktorbarzin.me/viktor/chatterbox-tts:${SHA}" \ + . + +# 3. Push both tags. (If docker isn't authed: log in with the viktor push PAT +# from Vault — `vault kv get -field=forgejo_push_token secret/ci/global` — +# `docker login forgejo.viktorbarzin.me -u viktor`.) +docker push forgejo.viktorbarzin.me/viktor/chatterbox-tts:latest +docker push "forgejo.viktorbarzin.me/viktor/chatterbox-tts:${SHA}" +``` + +> If `Dockerfile.cu128` is not a clean `docker build` target (e.g. it relies on +> build args defined only in `docker-compose-cu128.yml`), lift those args onto +> the `docker build` line or `docker compose -f docker-compose-cu128.yml build` +> then `docker tag` the resulting `chatterbox-tts-server:cu128` image to the +> Forgejo ref above before pushing. + +--- + +## Apply (admin-gated — run in order) + +```bash +vault login -method=oidc +~/code/scripts/presence claim node:k8s-node1 --purpose "chatterbox-tts first apply (GPU)" +~/code/scripts/presence claim stack:tts --purpose "chatterbox-tts stack apply" + +# 1. The polite-tenant hardening (exclude tts from gpu-workload priority). +~/code/scripts/tg plan --stack kyverno +~/code/scripts/tg apply --stack kyverno + +# 2. This stack. +~/code/scripts/tg plan --stack tts +~/code/scripts/tg apply --stack tts # apply does NOT wake the GPU (replicas=0) + +# 3. Flip tripit narration on. +~/code/scripts/tg plan --stack tripit +~/code/scripts/tg apply --stack tripit +``` + +See `docs/plans/2026-06-08-chatterbox-tts-infra.md` §5 for the full go-live +checklist (seed voices on NFS-SSD, smoke-test a synth, watch the neighbours). + +## Rollback (instant, no data loss) + +- **Narration off:** set `TTS_MODE=none` (or drop the three `TTS_*` lines) in + `stacks/tripit/main.tf` → `tg apply --stack tripit`. The bake makes no audio; + playback falls back to browser TTS. Cached `story_audio` rows are harmless. +- **Chatterbox off the GPU:** `kubectl -n tts scale deploy/chatterbox-tts + --replicas=0` (transient) and/or `tg destroy --stack tts`. Best-effort synth + means tripit bakes keep running audio-less — no error. +- Neither touches the resident GPU tenants (Option A never modifies them). diff --git a/stacks/tts/main.tf b/stacks/tts/main.tf new file mode 100644 index 00000000..2afdfad8 --- /dev/null +++ b/stacks/tts/main.tf @@ -0,0 +1,474 @@ +variable "image_tag" { + type = string + default = "latest" + description = "chatterbox-tts image tag. Use the 8-char git SHA in CI; :latest for local trials." +} + +# ───────────────────────────────────────────────────────────────────────────── +# Option-A off-peak control (see docs/plans/2026-06-08-chatterbox-tts-infra.md §3). +# The Deployment sits at replicas=0; a CronJob scales it to 1 at the window start +# ONLY IF a free-VRAM preflight passes, and another scales it back to 0 at window +# end. A guard CronJob yields the card mid-window if free VRAM drops below the +# floor (a resident woke up). tripit's bake is best-effort + idempotent, so a +# skipped/aborted window simply backfills on the next one (ADR-0002/0004). +# ───────────────────────────────────────────────────────────────────────────── + +variable "vram_free_floor_bytes" { + type = number + # OPEN ITEM — must be measured (§5 smoke test / §3.X). This is the minimum free + # VRAM the preflight requires before it will scale Chatterbox up, and the floor + # the guard yields below. Default = 6 GiB ≈ (a conservative guess for + # chatterbox-multilingual FP16 peak ~4 GiB + ~2 GiB headroom for the + # read→cudaMalloc race). RAISE/LOWER once the real T4 peak is captured from + # gpu_pod_memory_used_bytes{namespace="tts"} during a real synth. + default = 6442450944 + description = "Minimum free GPU VRAM (bytes) required before scaling Chatterbox up; guard yields below it." +} + +variable "gpu_total_bytes" { + type = number + default = 17179869184 # Tesla T4 = 16 GiB + description = "Total VRAM on the shared GPU. Free = this minus sum(gpu_pod_memory_used_bytes)." +} + +variable "offpeak_window_up_schedule" { + type = string + default = "0 2 * * *" # 02:00 Europe/London (see timezone on the CronJob) + description = "Cron schedule that fires the free-VRAM preflight + scale-up at window start." +} + +variable "offpeak_window_down_schedule" { + type = string + default = "0 6 * * *" # 06:00 Europe/London + description = "Cron schedule that scales Chatterbox back to 0 at window end." +} + +variable "offpeak_guard_schedule" { + type = string + default = "*/5 2-5 * * *" # every 5 min inside the 02:00–06:00 window + description = "Cron schedule for the mid-window guard that yields the card if free VRAM drops." +} + +locals { + namespace = "tts" + labels = { app = "chatterbox-tts" } + image = "forgejo.viktorbarzin.me/viktor/chatterbox-tts:${var.image_tag}" + + # config.yaml rendered into a ConfigMap, mounted at /app/config.yaml (the + # server's WORKDIR is /app). Voices, reference audio and the HF model cache + # all live on the NFS-SSD PVC (mounted at /data) so weights persist across + # restarts and load fast. server.port stays at the devnen default 8004; the + # Service remaps 8000->8004 so tripit's default TTS_BASE_URL works unchanged. + # + # model.repo_id = chatterbox-multilingual (ADR-0004; 23 languages for + # worldwide place-names). If the measured T4 VRAM peak is too high to coexist + # even off-peak, fall back to "chatterbox" (English, lighter) — a one-line + # change here (§3.X / §6 decision 3). + chatterbox_config = yamlencode({ + server = { + host = "0.0.0.0" + port = 8004 + } + model = { + repo_id = "chatterbox-multilingual" + } + tts_engine = { + device = "cuda" + predefined_voices_path = "/data/voices" + reference_audio_path = "/data/reference_audio" + } + }) + + # Shared script for the off-peak CronJobs. Reads the in-cluster + # gpu_pod_memory_used_bytes gauge (the per-namespace gauge the 2026-06-02 + # post-mortem built — host-PID attribution, no new exporter needed), sums it, + # and computes free = GPU_TOTAL - used. Pure POSIX + awk; curl is baked into + # the curl image. ACTION is "up" | "down" | "guard". + # up — scale to 1 ONLY IF free >= FLOOR (positive admission). + # guard — scale to 0 IF free < FLOOR (a resident woke mid-window; yield). + # down — scale to 0 unconditionally (window end). + # Heredoc escaping: only `$${...}` (literal `${...}`) is escaped — Terraform + # would otherwise try to interpolate it. Bare `$(...)`, `$((...))` and awk's + # `$NF` are literal `$` and pass through unescaped. + vram_gate_script = <<-EOT + set -eu + : "$${ACTION:?}" "$${FLOOR:?}" "$${GPU_TOTAL:?}" + METRICS_URL="http://gpu-pod-exporter.nvidia.svc.cluster.local:80/metrics" + + # Sum gpu_pod_memory_used_bytes across all pods. Missing metric / empty + # scrape => used=0 (card idle). -f so a non-200 scrape is a hard error we + # treat conservatively (skip scale-up). + if ! BODY="$(curl -sf -m 10 "$${METRICS_URL}")"; then + echo "WARN: could not scrape $${METRICS_URL}" + if [ "$${ACTION}" = "up" ]; then + echo "preflight: scrape failed -> NOT scaling up (fail-safe)"; exit 0 + fi + # For down/guard a failed scrape must NOT block yielding the card. + BODY="" + fi + USED="$(printf '%s\n' "$${BODY}" \ + | awk '/^gpu_pod_memory_used_bytes\{/ { s += $NF } END { printf "%d", s }')" + USED="$${USED:-0}" + FREE="$(( GPU_TOTAL - USED ))" + echo "GPU VRAM: used=$${USED} free=$${FREE} floor=$${FLOOR} (total=$${GPU_TOTAL})" + + case "$${ACTION}" in + up) + if [ "$${FREE}" -ge "$${FLOOR}" ]; then + echo "preflight PASS: free >= floor -> scaling chatterbox-tts to 1" + kubectl -n tts scale deploy/chatterbox-tts --replicas=1 + else + echo "preflight SKIP: free < floor -> leaving chatterbox-tts at 0 (retry next window)" + fi + ;; + guard) + if [ "$${FREE}" -lt "$${FLOOR}" ]; then + echo "guard TRIP: free < floor -> yielding the card, scaling chatterbox-tts to 0" + kubectl -n tts scale deploy/chatterbox-tts --replicas=0 + else + echo "guard OK: free >= floor -> chatterbox-tts may keep running" + fi + ;; + down) + echo "window end -> scaling chatterbox-tts to 0" + kubectl -n tts scale deploy/chatterbox-tts --replicas=0 + ;; + esac + EOT + + # Common spec for the three off-peak CronJobs. Each runs one bitnami/kubectl + # pod (in-cluster SA, no kubeconfig) executing the shared gate script with a + # different ACTION. timezone pins the window to Europe/London regardless of + # node TZ. + offpeak_cronjobs = { + chatterbox-window-up = { + schedule = var.offpeak_window_up_schedule + action = "up" + } + chatterbox-window-down = { + schedule = var.offpeak_window_down_schedule + action = "down" + } + chatterbox-vram-guard = { + schedule = var.offpeak_guard_schedule + action = "guard" + } + } +} + +resource "kubernetes_namespace" "tts" { + metadata { + name = local.namespace + labels = { + tier = local.tiers.gpu + "istio-injection" = "disabled" + "keel.sh/enrolled" = "true" + } + } + lifecycle { + ignore_changes = [metadata[0].labels["goldilocks.fairwinds.com/vpa-update-mode"]] + } +} + +# Model weights + voices on NFS-SSD (fast load), RWX so a seed Job / kubectl cp +# can write the predefined voices + narrator reference WAV while the Deployment +# mounts it. Path /srv/nfs-ssd/chatterbox on the Proxmox host. Mirrors +# llama-cpp's nfs_models. First start downloads the model into /data/hf_cache +# (HF_HOME below), so weights persist across pod restarts. +module "nfs_models" { + source = "../../modules/kubernetes/nfs_volume" + name = "chatterbox-models" + namespace = kubernetes_namespace.tts.metadata[0].name + nfs_server = "192.168.1.127" + nfs_path = "/srv/nfs-ssd/chatterbox" + storage = "20Gi" # multilingual weights + HF cache + voices headroom +} + +resource "kubernetes_config_map" "chatterbox_config" { + metadata { + name = "chatterbox-config" + namespace = kubernetes_namespace.tts.metadata[0].name + labels = local.labels + } + data = { + "config.yaml" = local.chatterbox_config + } +} + +# Single Deployment running the devnen Chatterbox-TTS-Server (OpenAI-compatible +# /v1/audio/speech). Sits at replicas=0 — the off-peak CronJobs below scale it +# to 1 only when the free-VRAM preflight passes (Option A), and back to 0 at +# window end. wait_for_rollout=false so apply never blocks on a pod that is +# intentionally scaled to 0. +resource "kubernetes_deployment" "chatterbox" { + metadata { + name = "chatterbox-tts" + namespace = kubernetes_namespace.tts.metadata[0].name + labels = merge(local.labels, { tier = local.tiers.gpu }) + } + wait_for_rollout = false + spec { + # Off-peak control owns the replica count at runtime (CronJobs scale 0<->1). + # Declare 0 here so a plain `tg apply` outside the window doesn't wake the + # card. ignore_changes on replicas (below) stops apply from fighting the + # CronJob's scale. + replicas = 0 + strategy { type = "Recreate" } + selector { + match_labels = { app = "chatterbox-tts" } + } + template { + metadata { + labels = { app = "chatterbox-tts" } + annotations = { + "checksum/config" = sha256(local.chatterbox_config) + } + } + spec { + node_selector = { "nvidia.com/gpu.present" = "true" } + toleration { + key = "nvidia.com/gpu" + operator = "Equal" + value = "true" + effect = "NoSchedule" + } + # C-hardening (§3.RECOMMENDATION.3): Chatterbox is a polite, best-effort + # batch tenant — give it the regular tier-2-gpu priority (600000) so it + # is ALWAYS the pod evicted under GPU-node pressure, never immich-ml / + # frigate / llama-swap. This relies on the `tts` namespace being EXCLUDED + # from the Kyverno `inject-gpu-workload-priority` policy (which would + # otherwise stamp the immich-equal gpu-workload=1,200,000 priority on any + # nvidia.com/gpu pod). That exclusion is the two-line edit to the kyverno + # stack flagged in the PR. Without it, this priority_class_name is + # overwritten on pod CREATE and Chatterbox would compete as an equal. + priority_class_name = "tier-2-gpu" + + image_pull_secrets { name = "registry-credentials" } + + container { + name = "chatterbox-tts" + image = local.image + port { + container_port = 8004 + name = "http" + } + + # T4 is Turing — NO bf16 (ADR-0004). Pin off; run FP16/FP32. + env { + name = "TTS_BF16" + value = "off" + } + # Park the HuggingFace cache on the NFS-SSD PVC so model weights + # download once and persist across pod restarts (the pod is recreated + # every window). The devnen compose mounts HF cache at /app/hf_cache; + # point HF_HOME at the PVC instead. + env { + name = "HF_HOME" + value = "/data/hf_cache" + } + env { + name = "HF_HUB_CACHE" + value = "/data/hf_cache" + } + + volume_mount { + name = "config" + mount_path = "/app/config.yaml" + sub_path = "config.yaml" + } + volume_mount { + name = "models" + mount_path = "/data" + } + + # /v1/audio/voices is cheap and only 200s once the model is loaded — + # so it gates real readiness. First start downloads the model, which + # is slow; the generous failure_threshold absorbs that. + readiness_probe { + http_get { + path = "/v1/audio/voices" + port = 8004 + } + initial_delay_seconds = 20 + period_seconds = 15 + failure_threshold = 12 + } + liveness_probe { + http_get { + path = "/v1/audio/voices" + port = 8004 + } + initial_delay_seconds = 120 + period_seconds = 30 + failure_threshold = 5 + } + resources { + requests = { + cpu = "200m" + memory = "2Gi" + } + limits = { + memory = "8Gi" + "nvidia.com/gpu" = "1" # ONE time-slice (operator advertises 100), NOT the whole card + } + } + } + + volume { + name = "config" + config_map { + name = kubernetes_config_map.chatterbox_config.metadata[0].name + } + } + volume { + name = "models" + persistent_volume_claim { + claim_name = module.nfs_models.claim_name + } + } + } + } + } + lifecycle { + ignore_changes = [ + # Off-peak CronJobs own the replica count — don't let apply reset it. + spec[0].replicas, + spec[0].template[0].spec[0].dns_config, # KYVERNO_LIFECYCLE_V1 + spec[0].template[0].spec[0].container[0].image, # KEEL_IGNORE_IMAGE + metadata[0].annotations["keel.sh/match-tag"], + metadata[0].annotations["keel.sh/policy"], + metadata[0].annotations["keel.sh/trigger"], + metadata[0].annotations["keel.sh/pollSchedule"], # KYVERNO_LIFECYCLE_V2 + metadata[0].annotations["kubernetes.io/change-cause"], + metadata[0].annotations["deployment.kubernetes.io/revision"], + spec[0].template[0].metadata[0].annotations["keel.sh/update-time"], + ] + } +} + +resource "kubernetes_service" "chatterbox" { + metadata { + name = "chatterbox-tts" + namespace = kubernetes_namespace.tts.metadata[0].name + labels = local.labels + annotations = { + # Prometheus annotation-based scrape (mirrors tripit). The devnen server + # has no /metrics; this monitors liveness via the blackbox path and keeps + # the Service in the scrape set if a /metrics endpoint is added later. + "prometheus.io/scrape" = "true" + "prometheus.io/path" = "/v1/audio/voices" + "prometheus.io/port" = "8000" + } + } + spec { + type = "ClusterIP" # in-cluster only — never ingressed (no token needed) + selector = { app = "chatterbox-tts" } + port { + name = "http" + port = 8000 # tripit's default TTS_BASE_URL port + target_port = 8004 # the devnen server's actual listen port + } + } +} + +# ───────────────────────────────────────────────────────────────────────────── +# Option-A off-peak control: SA + Role (scale the Deployment) + RoleBinding + +# three CronJobs (window-up preflight, mid-window guard, window-down). Mirrors +# the nextcloud-watchdog in-cluster-kubectl pattern (SA → Role → bitnami/kubectl +# CronJob, no kubeconfig). +# ───────────────────────────────────────────────────────────────────────────── + +resource "kubernetes_service_account" "offpeak" { + metadata { + name = "chatterbox-offpeak" + namespace = kubernetes_namespace.tts.metadata[0].name + } +} + +resource "kubernetes_role" "offpeak" { + metadata { + name = "chatterbox-offpeak" + namespace = kubernetes_namespace.tts.metadata[0].name + } + # get + patch on the deployment scale subresource is all the gate needs. + rule { + api_groups = ["apps"] + resources = ["deployments", "deployments/scale"] + verbs = ["get", "patch"] + } +} + +resource "kubernetes_role_binding" "offpeak" { + metadata { + name = "chatterbox-offpeak" + namespace = kubernetes_namespace.tts.metadata[0].name + } + role_ref { + api_group = "rbac.authorization.k8s.io" + kind = "Role" + name = kubernetes_role.offpeak.metadata[0].name + } + subject { + kind = "ServiceAccount" + name = kubernetes_service_account.offpeak.metadata[0].name + namespace = kubernetes_namespace.tts.metadata[0].name + } +} + +resource "kubernetes_cron_job_v1" "offpeak" { + for_each = local.offpeak_cronjobs + + metadata { + name = each.key + namespace = kubernetes_namespace.tts.metadata[0].name + labels = local.labels + } + spec { + schedule = each.value.schedule + timezone = "Europe/London" + concurrency_policy = "Forbid" + starting_deadline_seconds = 120 + successful_jobs_history_limit = 1 + failed_jobs_history_limit = 3 + job_template { + metadata { labels = local.labels } + spec { + backoff_limit = 1 + active_deadline_seconds = 120 + ttl_seconds_after_finished = 300 + template { + metadata { labels = local.labels } + spec { + service_account_name = kubernetes_service_account.offpeak.metadata[0].name + restart_policy = "Never" + container { + name = "vram-gate" + image = "bitnami/kubectl:latest" + command = ["/bin/bash", "-c", local.vram_gate_script] + env { + name = "ACTION" + value = each.value.action + } + env { + name = "FLOOR" + value = tostring(var.vram_free_floor_bytes) + } + env { + name = "GPU_TOTAL" + value = tostring(var.gpu_total_bytes) + } + resources { + requests = { cpu = "20m", memory = "64Mi" } + limits = { memory = "128Mi" } + } + } + } + } + } + } + } + lifecycle { + # KYVERNO_LIFECYCLE_V1: Kyverno mutates dns_config with ndots=2 on CronJobs. + ignore_changes = [spec[0].job_template[0].spec[0].template[0].spec[0].dns_config] + } +} diff --git a/stacks/tts/terragrunt.hcl b/stacks/tts/terragrunt.hcl new file mode 100644 index 00000000..6bc02966 --- /dev/null +++ b/stacks/tts/terragrunt.hcl @@ -0,0 +1,36 @@ +include "root" { + path = find_in_parent_folders() +} + +dependency "platform" { + config_path = "../platform" + skip_outputs = true +} + +dependency "vault" { + config_path = "../vault" + skip_outputs = true +} + +# tts: in-cluster text-to-speech for tripit's "Tour guide" narration. +# One Deployment of `forgejo.viktorbarzin.me/viktor/chatterbox-tts` (devnen +# Chatterbox-TTS-Server, OpenAI-compatible /v1/audio/speech) at a single +# ClusterIP Service `chatterbox-tts.tts.svc:8000` (server listens on 8004; +# the Service remaps). Requests ONE time-slice of the shared T4 +# (nvidia.com/gpu=1) — a slice, not the card. +# +# OOM-avoidance (Option A, docs/plans/2026-06-08-chatterbox-tts-infra.md §3): +# the Deployment sits at replicas=0; an off-peak CronJob scales it to 1 at the +# 02:00–06:00 Europe/London window ONLY IF a free-VRAM preflight passes +# (gpu_pod_memory_used_bytes from gpu-pod-exporter), a guard CronJob yields the +# card mid-window if a resident wakes, and a window-down CronJob scales back to +# 0. tripit's bake is best-effort + cached-forever (ADR-0002/0004), so a +# skipped/aborted window simply backfills next time — no latency SLA. +# +# Polite-tenant hardening: the `tts` namespace must be EXCLUDED from the kyverno +# `inject-gpu-workload-priority` policy (a separate two-line edit to the kyverno +# stack) so Chatterbox keeps tier-2-gpu priority (600000) and is always the pod +# evicted under pressure — never immich-ml/frigate/llama-swap. +# +# Image is built from the devnen repo + pushed to Forgejo — see this stack's +# README.md for the exact docker build + push commands.