From c5bda77731d8ddca3b85bb4b055f8a38a01ce7b9 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 14:23:33 +0000 Subject: [PATCH 01/24] forgejo: survive CI-build registry-push storms (mem 3Gi + working retention) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Heavy in-cluster builds (e.g. tripit buildkit) were taking Forgejo down via two vectors. Fixes both, without moving Forgejo off the sdc HDD (code-oflt deferred): - Memory 1Gi -> 3Gi (requests=limits). Forgejo was OOMKilled (exit 137) under registry-push load; VPA upperBound ~1.5Gi was suppressed by the 1Gi cap it kept OOMing against. Size for the push spike. - Activate registry retention (DRY_RUN false). Verified the delete list against all running viktor/* images first: 0 running images affected. Pruned 478 -> 161 package versions; PVC was at its 50Gi autoresize ceiling. - FIX broken retention auth: the cleanup PAT was ci-pusher's, but Forgejo scopes container packages per-user, so DELETE on viktor/* returned 403 (the dry-run only did GETs, hiding it). Repointed forgejo_cleanup_token to viktor's write:package PAT. Retention had never actually worked. - Protect buildkit *cache* tags from retention (cleanup.sh keep-set) so the gentler-builds layer cache survives daily pruning. [ci skip] — already applied via scripts/tg. Co-Authored-By: Claude Opus 4.8 --- .claude/CLAUDE.md | 2 +- stacks/forgejo/cleanup.tf | 22 +++++++++++++++++----- stacks/forgejo/files/cleanup.sh | 13 ++++++++++--- stacks/forgejo/main.tf | 16 +++++++++++----- 4 files changed, 39 insertions(+), 14 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index da5f0f51..7b6a2c00 100755 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -38,7 +38,7 @@ Violations cause state drift, which causes future applies to break or silently r - **DNS**: `dns_type = "proxied"` (Cloudflare CDN) or `"non-proxied"` (direct A/AAAA). DNS records are auto-created — no need to edit `config.tfvars`. Smoke-test target: `echo.viktorbarzin.me` (auth=public, header-reflecting backend). - **Anubis PoW challenge** (`modules/kubernetes/anubis_instance/`): per-site reverse proxy that issues a 30-day JWT cookie after a tiny PoW solve. Use for **public, content-bearing sites without app-level auth** (blog, docs, wikis, static landing pages). Pattern: declare `module "anubis" { source = "../../modules/kubernetes/anubis_instance"; name = "X"; namespace = ...; target_url = "http://..svc.cluster.local" }`, then in `ingress_factory` set `service_name = module.anubis.service_name`, `port = module.anubis.service_port`, `anti_ai_scraping = false`. Shared ed25519 key in Vault `secret/viktor` -> `anubis_ed25519_key`; cookie scoped to `viktorbarzin.me` so one solve covers all Anubis-fronted subdomains. **DO NOT put Anubis in front of Git/API/WebDAV/CLI endpoints** — clients without JS can't solve PoW. **Replicas default to 1** because Anubis stores in-flight challenges in process memory; a challenge issued by pod A and solved against pod B errors with `store: key not found` (HTTP 500). Bumping replicas requires wiring a shared Redis store (TODO). For path-level carve-outs (e.g. wrongmove has `/` behind Anubis but `/api` direct, blog has `/net-diag.sh` direct), declare a second `ingress_factory` with `ingress_path = ["/"]` pointing at the bare backend service. Active on: blog (except `/net-diag.sh`), www, kms, travel, f1, cc, json, pb (privatebin), home (homepage), wrongmove (UI only). See `.claude/reference/patterns.md` "Anti-AI Scraping" for full layering. - **Docker images**: Always build for `linux/amd64`. SHA-tag rule is being phased out — see `docs/plans/2026-05-16-auto-upgrade-apps-{design,plan}.md`. New model: CI pushes `:latest` (optionally also `:<8-char-sha>` for traceability), Keel polls and triggers rollouts. Cache-staleness concern from the old rule is resolved at the nginx layer (URL-split — manifests pass through, blobs cached). Until Phase 1 of the migration completes (per the plan), follow the SHA-tag rule for new services to match existing pattern. -- **Private registry**: `forgejo.viktorbarzin.me/viktor/` (Forgejo packages, OAuth-style PAT auth). Use `image: forgejo.viktorbarzin.me/viktor/:` + `imagePullSecrets: [{name: registry-credentials}]`. Kyverno auto-syncs the Secret to all namespaces. Containerd `hosts.toml` on every node redirects to in-cluster Traefik LB `10.0.20.203` (with `skip_verify = true`, since the node dials Traefik by IP but the cert is for `forgejo.viktorbarzin.me`) to avoid hairpin NAT. That redirect covers **kubelet pulls** only — in-cluster pods (notably Woodpecker buildkit build pods pushing images) resolve `forgejo.viktorbarzin.me` via a CoreDNS `rewrite name exact ... traefik.traefik.svc.cluster.local` (Corefile in `stacks/technitium/modules/technitium/main.tf`), since they do NOT use the node containerd mirror; without it, buildkit pushes intermittently timed out on the public-IP hairpin (added 2026-06-04, beads code-yh33). **Was `.200` until 2026-06-01** — Traefik's 2026-05-30 move to its dedicated `.203` left this redirect pointing at the now-dead `.200:443`, silently breaking every *fresh* forgejo pull (cached images kept running, so it stayed hidden until a new image tag was pulled). Redirect source lives in `modules/create-template-vm/k8s-node-containerd-setup.sh` (new nodes) and `scripts/setup-forgejo-containerd-mirror.sh` (existing nodes). Push-side: viktor PAT in Vault `secret/ci/global/forgejo_push_token` (Forgejo container packages are scoped per-user; only the package owner can push, ci-pusher cannot write to viktor/*). Pull-side: cluster-puller PAT in Vault `secret/viktor/forgejo_pull_token`. Retention CronJob (`forgejo-cleanup` in `forgejo` ns, daily 04:00) keeps newest 10 versions + always `:latest`; integrity probed every 15min by `forgejo-integrity-probe` in `monitoring` ns (catalog walk + manifest HEAD on every blob). See `docs/plans/2026-05-07-forgejo-registry-consolidation-{design,plan}.md` for the migration history. Pull-through caches for upstream registries (DockerHub, GHCR, Quay, k8s.gcr, Kyverno) stay on the registry VM at `10.0.20.10` ports 5000/5010/5020/5030/5040 — the old port-5050 R/W private registry was decommissioned 2026-05-07. +- **Private registry**: `forgejo.viktorbarzin.me/viktor/` (Forgejo packages, OAuth-style PAT auth). Use `image: forgejo.viktorbarzin.me/viktor/:` + `imagePullSecrets: [{name: registry-credentials}]`. Kyverno auto-syncs the Secret to all namespaces. Containerd `hosts.toml` on every node redirects to in-cluster Traefik LB `10.0.20.203` (with `skip_verify = true`, since the node dials Traefik by IP but the cert is for `forgejo.viktorbarzin.me`) to avoid hairpin NAT. That redirect covers **kubelet pulls** only — in-cluster pods (notably Woodpecker buildkit build pods pushing images) resolve `forgejo.viktorbarzin.me` via a CoreDNS `rewrite name exact ... traefik.traefik.svc.cluster.local` (Corefile in `stacks/technitium/modules/technitium/main.tf`), since they do NOT use the node containerd mirror; without it, buildkit pushes intermittently timed out on the public-IP hairpin (added 2026-06-04, beads code-yh33). **Was `.200` until 2026-06-01** — Traefik's 2026-05-30 move to its dedicated `.203` left this redirect pointing at the now-dead `.200:443`, silently breaking every *fresh* forgejo pull (cached images kept running, so it stayed hidden until a new image tag was pulled). Redirect source lives in `modules/create-template-vm/k8s-node-containerd-setup.sh` (new nodes) and `scripts/setup-forgejo-containerd-mirror.sh` (existing nodes). Push-side: viktor PAT in Vault `secret/ci/global/forgejo_push_token` (Forgejo container packages are scoped per-user; only the package owner can push, ci-pusher cannot write to viktor/*). Pull-side: cluster-puller PAT in Vault `secret/viktor/forgejo_pull_token`. Retention CronJob (`forgejo-cleanup` in `forgejo` ns, daily 04:00) keeps newest 10 versions + always `:latest` + any buildkit `*cache*` tag (so `--cache-from`/`--cache-to` refs survive retention — added 2026-06-09); **went live (DRY_RUN=false) 2026-06-09** after verifying 0 running images on the delete set — the registry PVC is at its 50Gi autoresize ceiling on the HDD (we did NOT move it to SSD, see beads code-oflt), so live retention is what keeps it from filling. Integrity probed every 15min by `forgejo-integrity-probe` in `monitoring` ns (catalog walk + manifest HEAD on every blob). See `docs/plans/2026-05-07-forgejo-registry-consolidation-{design,plan}.md` for the migration history. Pull-through caches for upstream registries (DockerHub, GHCR, Quay, k8s.gcr, Kyverno) stay on the registry VM at `10.0.20.10` ports 5000/5010/5020/5030/5040 — the old port-5050 R/W private registry was decommissioned 2026-05-07. - **LinuxServer.io containers**: `DOCKER_MODS` runs apt-get on every start — bake slow mods into a custom image (`RUN /docker-mods || true` then `ENV DOCKER_MODS=`). Set `NO_CHOWN=true` to skip recursive chown that hangs on NFS mounts. - **Node memory changes**: When changing VM memory on any k8s node, update kubelet `systemReserved`, `kubeReserved`, and eviction thresholds accordingly. Config: `/var/lib/kubelet/config.yaml`. Template: `stacks/infra/main.tf`. Current values: systemReserved=512Mi, kubeReserved=512Mi, evictionHard=500Mi, evictionSoft=1Gi. - **Node OS disk tuning** (in `stacks/infra/main.tf`): kubelet `imageGCHighThresholdPercent=70` (was 85), `imageGCLowThresholdPercent=60` (was 80), ext4 `commit=60` in fstab (was default 5s), journald `SystemMaxUse=200M` + `MaxRetentionSec=3day`. diff --git a/stacks/forgejo/cleanup.tf b/stacks/forgejo/cleanup.tf index add332d3..4c019c83 100644 --- a/stacks/forgejo/cleanup.tf +++ b/stacks/forgejo/cleanup.tf @@ -4,9 +4,17 @@ # it's per-user runtime state inside the Forgejo DB. Driving retention from # a CronJob hitting the public API keeps the policy versioned in this repo. # -# Auth: a write:package PAT belonging to ci-pusher (same user that pushes -# from CI). DELETE on packages requires write:package scope. PAT lives in -# Vault at secret/viktor/forgejo_cleanup_token. +# Auth: a write:package PAT belonging to VIKTOR (the package OWNER). PAT +# lives in Vault at secret/viktor/forgejo_cleanup_token. +# +# CORRECTION 2026-06-09: this previously said the PAT belonged to ci-pusher. +# That was wrong and silently broke retention — Forgejo container packages +# are scoped per-user, so ci-pusher gets HTTP 403 on DELETE of viktor/* +# (the dry-run only does GETs, which DO work, so the 403 stayed hidden until +# the first live run). DELETE requires a write:package PAT owned by viktor. +# forgejo_cleanup_token is therefore set to viktor's write:package PAT (today +# the same value as secret/ci/global/forgejo_push_token). IF that push token +# is ever regenerated, re-mirror it here or retention silently 403s again. data "vault_kv_secret_v2" "forgejo_viktor" { mount = "secret" @@ -14,8 +22,12 @@ data "vault_kv_secret_v2" "forgejo_viktor" { } locals { - # Flip to false after first 7 days of dry-run logs look correct. - forgejo_cleanup_dry_run = true + # Activated 2026-06-09 after verifying a dry-run delete list against all + # running viktor/* images cluster-wide: 0 running images on the delete set + # (would prune 317 stale versions, keeping newest 10 + latest + cache tags). + # Live retention is what keeps the registry PVC from filling on the HDD + # (we deliberately did NOT move Forgejo to SSD — see beads code-oflt). + forgejo_cleanup_dry_run = false } resource "kubernetes_config_map" "forgejo_cleanup_script" { diff --git a/stacks/forgejo/files/cleanup.sh b/stacks/forgejo/files/cleanup.sh index 61bb7a96..5efc6cce 100644 --- a/stacks/forgejo/files/cleanup.sh +++ b/stacks/forgejo/files/cleanup.sh @@ -2,8 +2,13 @@ # Forgejo container-package retention. # # For each container package owned by ${FORGEJO_OWNER}, keep newest -# ${KEEP_LAST_N} versions + always keep tag "latest". Deletes the rest via +# ${KEEP_LAST_N} versions + always keep tag "latest" + always keep any +# buildkit cache tag (matches "cache", e.g. tripit:cache — these back +# --cache-from/--cache-to and must survive retention or every build is a +# cold rebuild). Deletes the rest via # DELETE /api/v1/packages/{owner}/container/{name}/{version}. +# (Note: an 8-char SHA tag is pure hex and cannot contain "cache" — 'h' is +# not a hex digit — so the cache match never catches a real image tag.) # # DRY_RUN=true logs what would be deleted but issues no DELETE calls. # @@ -72,9 +77,11 @@ for NAME in $NAMES; do N_VERSIONS=$(jq 'length' "$TMPDIR/$NAME.json") echo "[$NAME] $N_VERSIONS version(s)" - # Build the keep set: top $KEEP + anything tagged 'latest'. + # Build the keep set: top $KEEP + always 'latest' + any buildkit cache tag. jq -r --argjson keep "$KEEP" ' - [.[0:$keep][].version] + [.[] | select(.version == "latest") | .version] + [.[0:$keep][].version] + + [.[] | select(.version == "latest") | .version] + + [.[] | select(.version | test("cache"; "i")) | .version] | unique | .[] ' "$TMPDIR/$NAME.json" > "$TMPDIR/$NAME.keep" diff --git a/stacks/forgejo/main.tf b/stacks/forgejo/main.tf index 2778f555..e1b8c351 100644 --- a/stacks/forgejo/main.tf +++ b/stacks/forgejo/main.tf @@ -9,7 +9,7 @@ resource "kubernetes_namespace" "forgejo" { name = "forgejo" labels = { "istio-injection" : "disabled" - tier = local.tiers.edge + tier = local.tiers.edge "keel.sh/enrolled" = "true" } } @@ -94,7 +94,7 @@ resource "kubernetes_deployment" "forgejo" { fs_group = 1000 } container { - name = "forgejo" + name = "forgejo" # Pinned to 11.0.14 (latest 11.x as of 2026-05-12) — was on # floating `:11`. On 2026-05-24T15:35:37Z Keel force-policy # rewrote the tag from `11.0.14 → 1.18` (Gitea-era Forgejo @@ -168,13 +168,19 @@ resource "kubernetes_deployment" "forgejo" { name = "data" mount_path = "/data" } + # Bumped 1Gi -> 3Gi 2026-06-09: Forgejo was OOMKilled (exit 137) + # under registry-push load from in-cluster CI builds (tripit + # buildkit pushes large layers into the OCI registry). VPA + # upperBound reads ~1.5Gi, but that's suppressed by the 1Gi cap it + # kept OOMing against — size for the push spike, not steady-state. + # requests=limits (Guaranteed QoS) per the repo memory convention. resources { requests = { cpu = "15m" - memory = "1Gi" + memory = "3Gi" } limits = { - memory = "1Gi" + memory = "3Gi" } } port { @@ -202,7 +208,7 @@ resource "kubernetes_deployment" "forgejo" { metadata[0].annotations["keel.sh/match-tag"], metadata[0].annotations["keel.sh/trigger"], metadata[0].annotations["keel.sh/pollSchedule"], # KYVERNO_LIFECYCLE_V2 - spec[0].template[0].spec[0].container[0].image, # KEEL_IGNORE_IMAGE — Keel manages tag updates + spec[0].template[0].spec[0].container[0].image, # KEEL_IGNORE_IMAGE — Keel manages tag updates metadata[0].annotations["kubernetes.io/change-cause"], metadata[0].annotations["deployment.kubernetes.io/revision"], spec[0].template[0].metadata[0].annotations["keel.sh/update-time"], From 1f23ba692954812f040f27111c320e821bcc2027 Mon Sep 17 00:00:00 2001 From: viktor Date: Tue, 9 Jun 2026 18:18:13 +0000 Subject: [PATCH 02/24] tripit: build off-infra via GHA -> GHCR (private), pull via scoped ghcr-credentials Switches tripit image from forgejo.viktorbarzin.me (in-cluster buildkit, sdc load) to ghcr.io/viktorbarzin/tripit built by GitHub Actions (Forgejo push-mirror -> private GitHub -> GHA -> GHCR). Adds a tripit-ns-scoped GHCR pull secret (github_pat, interim). Verified: deploy on :c8dfb5cb ready, ingest-plans CronJob pulled :latest + Succeeded. [ci skip] --- stacks/tripit/main.tf | 51 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 50 insertions(+), 1 deletion(-) diff --git a/stacks/tripit/main.tf b/stacks/tripit/main.tf index 197a80d0..2c099fc6 100644 --- a/stacks/tripit/main.tf +++ b/stacks/tripit/main.tf @@ -15,7 +15,12 @@ variable "tls_secret_name" { locals { namespace = "tripit" - image = "forgejo.viktorbarzin.me/viktor/tripit:${var.image_tag}" + # Image now built OFF-INFRA by GitHub Actions, pushed to GHCR (private), 2026-06-09: + # Forgejo viktor/tripit push-mirrors -> private ViktorBarzin/tripit GitHub repo -> + # GHA builds + pushes ghcr.io/viktorbarzin/tripit. Removes both the build IO and the + # Forgejo/sdc registry push from the homelab. Running tag is set via `kubectl set + # image` (image is KEEL_IGNORE_IMAGE below); CronJobs track :latest. + image = "ghcr.io/viktorbarzin/tripit:${var.image_tag}" labels = { app = "tripit" } @@ -65,6 +70,15 @@ locals { SMTP_USER = "spam@viktorbarzin.me" SMTP_FROM = "plans@viktorbarzin.me" PUBLIC_BASE_URL = "https://tripit.viktorbarzin.me" + # Narrator audio (ADR-0004): Chatterbox via the in-cluster `tts` stack. + # OpenAI-compatible /v1/audio/speech; the bake POSTs best-effort synth + # requests, so a down/Pending Chatterbox is a clean skip (browser-TTS + # fallback), never a bake error. ClusterIP-only → no token. Note: the mode + # is `openai_compatible` (tripit renamed it from `chatterbox`); TTS_MODEL is + # still the `chatterbox` family string tripit sends as the OpenAI `model`. + TTS_MODE = "openai_compatible" + TTS_BASE_URL = "http://chatterbox-tts.tts.svc.cluster.local:8000" + TTS_MODEL = "chatterbox" } } @@ -84,6 +98,35 @@ resource "kubernetes_namespace" "tripit" { } } +# GHCR pull secret (tripit ns only) for the private ghcr.io/viktorbarzin/tripit image +# now built off-infra by GitHub Actions. Uses viktor's github_pat as the pull +# credential — admin-scoped, accepted as an interim (rotate to a fine-grained +# read:packages token later). Scoped to this namespace to limit the broad token's +# blast radius (deliberately NOT folded into the cluster-wide registry-credentials). +data "vault_kv_secret_v2" "viktor" { + mount = "secret" + name = "viktor" +} + +resource "kubernetes_secret" "ghcr_credentials" { + metadata { + name = "ghcr-credentials" + namespace = kubernetes_namespace.tripit.metadata[0].name + } + type = "kubernetes.io/dockerconfigjson" + data = { + ".dockerconfigjson" = jsonencode({ + auths = { + "ghcr.io" = { + username = "ViktorBarzin" + password = data.vault_kv_secret_v2.viktor.data["github_pat"] + auth = base64encode("ViktorBarzin:${data.vault_kv_secret_v2.viktor.data["github_pat"]}") + } + } + }) + } +} + # App secrets — seed these in Vault before applying: # secret/tripit # VAPID_PUBLIC_KEY — Web Push (VAPID) public key for push subscriptions @@ -277,6 +320,9 @@ resource "kubernetes_deployment" "tripit" { image_pull_secrets { name = "registry-credentials" } + image_pull_secrets { + name = kubernetes_secret.ghcr_credentials.metadata[0].name + } init_container { name = "alembic-migrate" @@ -533,6 +579,9 @@ resource "kubernetes_cron_job_v1" "tripit_worker" { image_pull_secrets { name = "registry-credentials" } + image_pull_secrets { + name = kubernetes_secret.ghcr_credentials.metadata[0].name + } container { name = "worker" image = local.image From 8eb0bb244f0bc84d1f1a8bff2bb83c0321f2d602 Mon Sep 17 00:00:00 2001 From: viktor Date: Tue, 9 Jun 2026 18:20:54 +0000 Subject: [PATCH 03/24] docs(ci-cd): add off-infra GHA->GHCR build pattern for private Forgejo repos (tripit pilot) [ci skip] --- .claude/CLAUDE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 7b6a2c00..601e8d2f 100755 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -114,6 +114,7 @@ images. **Migrated to GHA** (9): Website, k8s-portal, claude-memory-mcp, apple-health-data, audiblez-web, plotting-book, insta2spotify, audiobook-search, council-complaints **Woodpecker-native owned-app build** (Forgejo registry, build->deploy in one `.woodpecker.yml`): tuya_bridge, job-hunter, f1-stream (extracted to viktor/f1-stream 2026-06-05; Woodpecker repo id 166; the old github source is archived + its GHA repo-id-10 deactivated) **Woodpecker-only**: travel_blog (1.4GB content too large for GHA), infra pipelines (terragrunt apply, certbot, build-cli — need cluster access) +**Private Forgejo repo → off-infra GHA → GHCR** (NEW 2026-06-09 — gentler builds: keeps build IO **and** the registry push OFF the homelab/sdc; replaces in-cluster Woodpecker buildkit for private repos): **tripit** is the pilot. Forgejo `viktor/tripit` (canonical) push-mirrors → PRIVATE `ViktorBarzin/tripit` GitHub repo (`sync_on_commit`); `.github/workflows/build.yml` (committed on Forgejo, mirrors over) builds + pushes `ghcr.io/viktorbarzin/tripit:+latest` on GHA (free, ~2min, GHA-native cache). Cluster pulls the PRIVATE image via a **tripit-ns-scoped** `ghcr-credentials` dockerconfigjson (interim cred = viktor's admin `github_pat`; rotate to a fine-grained read:packages token). Deploy = `kubectl set image` (image is KEEL_IGNORE_IMAGE); worker CronJobs track `:latest`. GitHub default branch must be `master`. **Replicate to f1-stream, tuya_bridge, job-hunter** (currently Woodpecker-native in-cluster builds). Mirror + workflow-file commits are done via the Forgejo API over the internal Traefik LB (`curl --resolve forgejo.viktorbarzin.me:443:10.0.20.203`) since the devvm can't reach forgejo's public hairpin. **Per-project files**: - `.github/workflows/build-and-deploy.yml` — GHA: checkout, build, push DockerHub, POST Woodpecker API From 64f405db3662c4958e97f4d018387739ae826b3f Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 18:31:27 +0000 Subject: [PATCH 04/24] workstation: default Claude model = claude-fable-5 for all devvm users Adds a model key (claude-fable-5) to the machine-wide managed-settings.json (installed to /etc/claude-code/ by setup-devvm.sh). Sets the default model for every Claude Code session on the devvm (CLI + t3 web) at top settings precedence; per-session /model and explicit --model flags still override. The org claudeMd block is unchanged. Co-Authored-By: Claude Opus 4.8 --- scripts/workstation/managed-settings.json | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/workstation/managed-settings.json b/scripts/workstation/managed-settings.json index aa259ff7..224c49bd 100644 --- a/scripts/workstation/managed-settings.json +++ b/scripts/workstation/managed-settings.json @@ -1,3 +1,4 @@ { - "claudeMd": "# Viktor Barzin homelab — shared multi-user Claude Code Workstation (devvm)\n\nYou are running as a specific OS user on a SHARED devvm Workstation, not as the admin. These org-wide rules apply to EVERY user and sit at the top of settings precedence (they cannot be overridden by a user's own config):\n\n- Respect your permission tier. Your kubectl, Vault, and infra access are scoped to your RBAC tier (admin / power-user / namespace-owner). Do not attempt to escalate privileges or reach another user's resources.\n- Secrets are per-user. Never read another user's home directory, credentials, tokens, or ~/.claude secrets. Your own secrets live in your home at mode 600.\n- Infrastructure changes go through Terraform/Terragrunt (scripts/tg apply) — never direct kubectl apply/edit/patch. Pushing to git does NOT deploy; applies are manual and admin-gated, so your edits cannot take effect without an admin apply.\n- Follow the engineering rules in ~/.claude/rules/ (execution, planning, quality) and every CLAUDE.md in the repo tree.\n- The monorepo is at ~/code. Non-admins get a git-crypt-LOCKED clone: secret files read as ciphertext — that is expected, not an error." + "claudeMd": "# Viktor Barzin homelab — shared multi-user Claude Code Workstation (devvm)\n\nYou are running as a specific OS user on a SHARED devvm Workstation, not as the admin. These org-wide rules apply to EVERY user and sit at the top of settings precedence (they cannot be overridden by a user's own config):\n\n- Respect your permission tier. Your kubectl, Vault, and infra access are scoped to your RBAC tier (admin / power-user / namespace-owner). Do not attempt to escalate privileges or reach another user's resources.\n- Secrets are per-user. Never read another user's home directory, credentials, tokens, or ~/.claude secrets. Your own secrets live in your home at mode 600.\n- Infrastructure changes go through Terraform/Terragrunt (scripts/tg apply) — never direct kubectl apply/edit/patch. Pushing to git does NOT deploy; applies are manual and admin-gated, so your edits cannot take effect without an admin apply.\n- Follow the engineering rules in ~/.claude/rules/ (execution, planning, quality) and every CLAUDE.md in the repo tree.\n- The monorepo is at ~/code. Non-admins get a git-crypt-LOCKED clone: secret files read as ciphertext — that is expected, not an error.", + "model": "claude-fable-5" } From 68a237faf7dc41867026cb7940de9d254905edb5 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 19:35:29 +0000 Subject: [PATCH 05/24] workstation: skel start-claude.sh inherits managed default model (drop hardcoded --model) The per-user launcher hardcoded --model claude-opus-4-8; an explicit --model flag overrides the managed default in /etc/claude-code/managed-settings.json (claude-fable-5). Dropping it lets emo and all new accounts inherit the org default (per-session /model still works). Deployed to /etc/skel and emo live copy in the same change. Co-Authored-By: Claude Opus 4.8 --- scripts/workstation/skel/start-claude.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/workstation/skel/start-claude.sh b/scripts/workstation/skel/start-claude.sh index 7c86cc58..fa21aa36 100755 --- a/scripts/workstation/skel/start-claude.sh +++ b/scripts/workstation/skel/start-claude.sh @@ -31,7 +31,9 @@ launch() { # Deliberately not `exec` so we can branch on the exit code: clean quit ends the # pane (ttyd closes the terminal); a crash drops to a shell so the tmux session # isn't destroyed-and-recreated in a ttyd auto-reconnect loop. -launch --dangerously-skip-permissions --model claude-opus-4-8 "${name_args[@]}" +# No --model flag: inherit the org-wide default from /etc/claude-code/managed-settings.json +# (an explicit --model would override that managed default for every launched session). +launch --dangerously-skip-permissions "${name_args[@]}" code=$? [ "$code" -eq 0 ] && exit 0 From b1a6391a4dabe4a009b37943eebf088ce5171c2c Mon Sep 17 00:00:00 2001 From: viktor Date: Tue, 9 Jun 2026 19:41:08 +0000 Subject: [PATCH 06/24] docs(ci-cd): tripit auto-deploy (GHA->Woodpecker 167) + svu semver in GHA [ci skip] --- .claude/CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 601e8d2f..e3e59fe6 100755 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -114,7 +114,7 @@ images. **Migrated to GHA** (9): Website, k8s-portal, claude-memory-mcp, apple-health-data, audiblez-web, plotting-book, insta2spotify, audiobook-search, council-complaints **Woodpecker-native owned-app build** (Forgejo registry, build->deploy in one `.woodpecker.yml`): tuya_bridge, job-hunter, f1-stream (extracted to viktor/f1-stream 2026-06-05; Woodpecker repo id 166; the old github source is archived + its GHA repo-id-10 deactivated) **Woodpecker-only**: travel_blog (1.4GB content too large for GHA), infra pipelines (terragrunt apply, certbot, build-cli — need cluster access) -**Private Forgejo repo → off-infra GHA → GHCR** (NEW 2026-06-09 — gentler builds: keeps build IO **and** the registry push OFF the homelab/sdc; replaces in-cluster Woodpecker buildkit for private repos): **tripit** is the pilot. Forgejo `viktor/tripit` (canonical) push-mirrors → PRIVATE `ViktorBarzin/tripit` GitHub repo (`sync_on_commit`); `.github/workflows/build.yml` (committed on Forgejo, mirrors over) builds + pushes `ghcr.io/viktorbarzin/tripit:+latest` on GHA (free, ~2min, GHA-native cache). Cluster pulls the PRIVATE image via a **tripit-ns-scoped** `ghcr-credentials` dockerconfigjson (interim cred = viktor's admin `github_pat`; rotate to a fine-grained read:packages token). Deploy = `kubectl set image` (image is KEEL_IGNORE_IMAGE); worker CronJobs track `:latest`. GitHub default branch must be `master`. **Replicate to f1-stream, tuya_bridge, job-hunter** (currently Woodpecker-native in-cluster builds). Mirror + workflow-file commits are done via the Forgejo API over the internal Traefik LB (`curl --resolve forgejo.viktorbarzin.me:443:10.0.20.203`) since the devvm can't reach forgejo's public hairpin. +**Private Forgejo repo → off-infra GHA → GHCR** (NEW 2026-06-09 — gentler builds: keeps build IO **and** the registry push OFF the homelab/sdc; replaces in-cluster Woodpecker buildkit for private repos): **tripit** is the pilot. Forgejo `viktor/tripit` (canonical) push-mirrors → PRIVATE `ViktorBarzin/tripit` GitHub repo (`sync_on_commit`); `.github/workflows/build.yml` (committed on Forgejo, mirrors over) builds + pushes `ghcr.io/viktorbarzin/tripit:+latest` on GHA (free, ~2min, GHA-native cache). Cluster pulls the PRIVATE image via a **tripit-ns-scoped** `ghcr-credentials` dockerconfigjson (interim cred = viktor's admin `github_pat`; rotate to a fine-grained read:packages token). **Auto-deploy** (verified 2026-06-09): the GHA `deploy` job POSTs `ci.viktorbarzin.me/api/repos/167/pipelines` (Woodpecker repo **167** = the GitHub mirror, registered github-forge; GHA secret `WOODPECKER_TOKEN`) with `IMAGE_TAG`+`IMAGE_NAME` → `.woodpecker/deploy.yml` (event:**manual** ONLY, so the Forgejo→GitHub mirror's raw pushes don't fire a tag-less deploy) runs `kubectl set image deployment/tripit tripit=… alembic-migrate=…` in-cluster (woodpecker-agent SA = cluster-admin, no kubeconfig). Image is KEEL_IGNORE_IMAGE so the SHA tag sticks; worker CronJobs track `:latest`. **Semver** (parallel layer): the GHA `build` job runs `svu` v3.4.1 over conventional commits, auto-cuts the next `vX.Y.Z` git tag pushed to CANONICAL Forgejo (GHA secret `FORGEJO_GIT_TOKEN` = write:repository PAT, NOT the package-scoped push token) and bakes `VERSION` → app reports it at `/api/version` (verified 0.2.1). Deploy tag stays the 8-char SHA. The old in-cluster `.woodpecker/build.yml` was DELETED (only `.woodpecker/deploy.yml` remains). GitHub default branch must be `master`. **Replicate to f1-stream, tuya_bridge, job-hunter** (currently Woodpecker-native in-cluster builds). Mirror + workflow-file commits are done via the Forgejo API over the internal Traefik LB (`curl --resolve forgejo.viktorbarzin.me:443:10.0.20.203`) since the devvm can't reach forgejo's public hairpin. **Per-project files**: - `.github/workflows/build-and-deploy.yml` — GHA: checkout, build, push DockerHub, POST Woodpecker API From 48013a4a929bf19ad4caa8172d2479bfed243dcc Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 07:30:19 +0000 Subject: [PATCH 07/24] feat(tts): Chatterbox TTS stack + off-peak T4 gate, wire tripit narration [ci skip] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New `infra/stacks/tts/` deploys devnen/Chatterbox-TTS-Server (OpenAI-compatible /v1/audio/speech) as ClusterIP `chatterbox-tts.tts.svc:8000` (server listens on 8004; Service remaps), requesting ONE T4 time-slice. Mirrors stacks/llama-cpp/. Option A off-peak control (no VRAM isolation on the time-sliced T4 — see post-mortem 2026-06-02): Deployment sits at replicas=0; three Europe/London CronJobs own the replica count — `chatterbox-window-up` scales to 1 at 02:00 ONLY IF a free-VRAM preflight passes (sum gpu_pod_memory_used_bytes from gpu-pod-exporter; free = 16GiB - used >= floor), `chatterbox-vram-guard` yields the card mid-window if a resident wakes, `chatterbox-window-down` scales to 0 at 06:00. tripit's bake is best-effort + cached-forever (ADR-0002/0004) so a skipped/aborted window backfills next time. SA+Role+RoleBinding grant the CronJobs deployments/scale (nextcloud-watchdog pattern). Polite-tenant hardening: kyverno `inject-gpu-workload-priority` now excludes the `tts` namespace (new `gpu_priority_excluded_namespaces` local) so Chatterbox keeps tier-2-gpu priority (600k) and is always evicted first under GPU pressure — never immich-ml/frigate/llama-swap. The LimitRange-fallback policy still uses the base exclude list (tts untouched there). tripit: add TTS_MODE=openai_compatible, TTS_BASE_URL, TTS_MODEL=chatterbox to local.app_env (no token — ClusterIP only). No tripit code change. Image build is documented in stacks/tts/README.md (devnen cu128 target -> forgejo.viktorbarzin.me/viktor/chatterbox-tts) — build is impractical inline (large CUDA image + needs the upstream repo). NOT APPLIED — review branch only. Free-VRAM floor (var.vram_free_floor_bytes, default 6GiB) must be set from the measured chatterbox-multilingual T4 peak during the first bake. Co-Authored-By: Claude Opus 4.8 --- .../modules/kyverno/resource-governance.tf | 14 +- stacks/tts/README.md | 149 ++++++ stacks/tts/main.tf | 474 ++++++++++++++++++ stacks/tts/terragrunt.hcl | 36 ++ 4 files changed, 672 insertions(+), 1 deletion(-) create mode 100644 stacks/tts/README.md create mode 100644 stacks/tts/main.tf create mode 100644 stacks/tts/terragrunt.hcl diff --git a/stacks/kyverno/modules/kyverno/resource-governance.tf b/stacks/kyverno/modules/kyverno/resource-governance.tf index 855128f1..bcbaece0 100644 --- a/stacks/kyverno/modules/kyverno/resource-governance.tf +++ b/stacks/kyverno/modules/kyverno/resource-governance.tf @@ -15,6 +15,15 @@ locals { governance_tiers = ["0-core", "1-cluster", "2-gpu", "3-edge", "4-aux"] excluded_namespaces = ["kube-system", "metallb-system", "kyverno", "calico-system", "calico-apiserver"] + + # GPU-priority injection exclude list. Adds `tts` to the base set so the + # `inject-gpu-workload-priority` policy does NOT stamp the immich-equal + # gpu-workload (1,200,000) priority on Chatterbox-TTS pods. Chatterbox is a + # best-effort off-peak batch tenant on the shared T4: it must keep its + # tier-2-gpu (600,000) priority so it is ALWAYS the pod evicted under GPU-node + # pressure, never immich-ml/frigate/llama-swap. See the tts stack + # (stacks/tts/) + docs/plans/2026-06-08-chatterbox-tts-infra.md §3. + gpu_priority_excluded_namespaces = concat(local.excluded_namespaces, ["tts"]) } # ----------------------------------------------------------------------------- @@ -905,7 +914,10 @@ resource "kubectl_manifest" "mutate_gpu_priority" { any = [ { resources = { - namespaces = local.excluded_namespaces + # tts added so Chatterbox-TTS keeps tier-2-gpu priority (it's a + # best-effort off-peak batch tenant — must be evicted first, + # not promoted to immich-equal gpu-workload). See locals above. + namespaces = local.gpu_priority_excluded_namespaces } } ] diff --git a/stacks/tts/README.md b/stacks/tts/README.md new file mode 100644 index 00000000..f5a972dd --- /dev/null +++ b/stacks/tts/README.md @@ -0,0 +1,149 @@ +# tts — Chatterbox TTS (tripit narration) + +In-cluster text-to-speech for tripit's "Tour guide". Runs the +[devnen/Chatterbox-TTS-Server](https://github.com/devnen/Chatterbox-TTS-Server) +(Resemble AI Chatterbox under an OpenAI-compatible HTTP server) as a single +Deployment + ClusterIP Service `chatterbox-tts.tts.svc.cluster.local:8000`, +requesting **one time-slice** of the shared Tesla T4 (`nvidia.com/gpu: 1`). + +Full design + rationale (Option-A off-peak control, OOM analysis, ADR links): +`docs/plans/2026-06-08-chatterbox-tts-infra.md` (in the tripit-tour-guide repo) +and `infra/docs/post-mortems/2026-06-02-immich-ml-ttl-gpu-oom-recruiter.md`. + +> This stack mirrors `infra/stacks/llama-cpp/`. The scaffolding files +> (`backend.tf`, `providers.tf`, `cloudflare_provider.tf`, `tiers.tf`, +> `.terraform.lock.hcl`) are **generated by Terragrunt** on `init` and are +> git-ignored — only `main.tf`, `terragrunt.hcl` and this README are tracked. + +--- + +## What this stack creates + +- `kubernetes_namespace.tts` — tier `2-gpu`, keel-enrolled, istio off. +- `module.nfs_models` — RWX NFS-SSD PVC at `/srv/nfs-ssd/chatterbox`, mounted at + `/data` (predefined voices, narrator reference WAVs, **and** the HuggingFace + model cache via `HF_HOME=/data/hf_cache`, so weights download once and persist + across the per-window pod recreation). +- `kubernetes_config_map.chatterbox_config` — `config.yaml`: `server.port=8004`, + `model.repo_id=chatterbox-multilingual`, `tts_engine.device=cuda`, voices / + reference paths under `/data`. +- `kubernetes_deployment.chatterbox` — **starts at `replicas=0`**; the off-peak + CronJobs own the replica count at runtime. `TTS_BF16=off` (T4 = Turing, no + bf16). `priority_class_name=tier-2-gpu` (the polite-tenant demotion). +- `kubernetes_service.chatterbox` — ClusterIP, **`port 8000 → targetPort 8004`** + so tripit's default `TTS_BASE_URL` works unchanged. Prometheus scrape + annotations. +- **Off-peak control** (SA + Role + RoleBinding + 3 CronJobs): see below. + +## Off-peak control (Option A — window + free-VRAM gate) + +The T4 is time-sliced with **zero VRAM isolation** (post-mortem 2026-06-02), so +`nvidia.com/gpu: 1` buys a scheduling turn, NOT memory. Chatterbox must only +allocate VRAM when the card is actually free. Implemented as three CronJobs +(all `Europe/London`), each a `bitnami/kubectl` pod using the namespace SA: + +| CronJob | Schedule (default) | Action | +|---|---|---| +| `chatterbox-window-up` | `0 2 * * *` | **Preflight**: scrape `gpu_pod_memory_used_bytes` from `gpu-pod-exporter.nvidia.svc:80/metrics`, compute `free = 16 GiB − Σused`; scale to **1 only if** `free ≥ vram_free_floor_bytes`. | +| `chatterbox-vram-guard` | `*/5 2-5 * * *` | **Guard**: every 5 min in-window, scale to **0** if `free < floor` (a resident woke; yield the card mid-bake). | +| `chatterbox-window-down` | `0 6 * * *` | **Window end**: scale to **0** unconditionally. | + +`tripit`'s bake is best-effort + cached-forever (ADR-0002/0004) — a skipped or +aborted window simply backfills on the next one. No latency SLA. + +### The free-VRAM floor — YOU MUST MEASURE THIS + +`var.vram_free_floor_bytes` defaults to **6 GiB** (a conservative guess: +~4 GiB assumed multilingual FP16 peak + ~2 GiB headroom for the +read→`cudaMalloc` race). **The real T4 peak of `chatterbox-multilingual` is not +published upstream.** Capture it during the first bake: + +```bash +# while a real synth is running on the freed T4: +kubectl -n monitoring exec deploy/prometheus -- \ + promtool query instant http://localhost:9090 \ + 'sum(gpu_pod_memory_used_bytes{namespace="tts"})' +# or read the gauge straight from the exporter: +kubectl -n nvidia exec ds/gpu-pod-exporter -- \ + sh -c 'curl -s localhost:9401/metrics | grep "namespace=\"tts\""' +``` + +Then set the floor to `measured_peak + ~2 GiB` (pass `-var` or add to the stack +tfvars). If the peak is too high to coexist even off-peak, switch +`model.repo_id` in `main.tf` to `chatterbox` (English, lighter) or +`chatterbox-turbo`, or escalate to Option B (scale `immich-machine-learning` to +0 for the window). + +--- + +## Build + push the image (do this BEFORE the first apply) + +`devnen/Chatterbox-TTS-Server` ships **no published image** — build from the +repo's **cu128** target (matches the cluster's pinned 570.195.03 / CUDA 12.8 +driver) and push to the private Forgejo registry. The devvm docker is pre-authed +to `forgejo.viktorbarzin.me`. Run on the devvm (large CUDA image — needs disk + +bandwidth): + +```bash +# 1. Clone the upstream server repo (outside the monorepo). +git clone https://github.com/devnen/Chatterbox-TTS-Server /tmp/chatterbox-tts-server +cd /tmp/chatterbox-tts-server + +# 2. Build the cu128 variant (Dockerfile.cu128 — PyTorch 2.9.0+cu128, the target +# the repo's docker-compose-cu128.yml uses) for linux/amd64. +SHA="$(git rev-parse --short=8 HEAD)" +docker build \ + --platform linux/amd64 \ + --build-arg RUNTIME=nvidia \ + -f Dockerfile.cu128 \ + -t forgejo.viktorbarzin.me/viktor/chatterbox-tts:latest \ + -t "forgejo.viktorbarzin.me/viktor/chatterbox-tts:${SHA}" \ + . + +# 3. Push both tags. (If docker isn't authed: log in with the viktor push PAT +# from Vault — `vault kv get -field=forgejo_push_token secret/ci/global` — +# `docker login forgejo.viktorbarzin.me -u viktor`.) +docker push forgejo.viktorbarzin.me/viktor/chatterbox-tts:latest +docker push "forgejo.viktorbarzin.me/viktor/chatterbox-tts:${SHA}" +``` + +> If `Dockerfile.cu128` is not a clean `docker build` target (e.g. it relies on +> build args defined only in `docker-compose-cu128.yml`), lift those args onto +> the `docker build` line or `docker compose -f docker-compose-cu128.yml build` +> then `docker tag` the resulting `chatterbox-tts-server:cu128` image to the +> Forgejo ref above before pushing. + +--- + +## Apply (admin-gated — run in order) + +```bash +vault login -method=oidc +~/code/scripts/presence claim node:k8s-node1 --purpose "chatterbox-tts first apply (GPU)" +~/code/scripts/presence claim stack:tts --purpose "chatterbox-tts stack apply" + +# 1. The polite-tenant hardening (exclude tts from gpu-workload priority). +~/code/scripts/tg plan --stack kyverno +~/code/scripts/tg apply --stack kyverno + +# 2. This stack. +~/code/scripts/tg plan --stack tts +~/code/scripts/tg apply --stack tts # apply does NOT wake the GPU (replicas=0) + +# 3. Flip tripit narration on. +~/code/scripts/tg plan --stack tripit +~/code/scripts/tg apply --stack tripit +``` + +See `docs/plans/2026-06-08-chatterbox-tts-infra.md` §5 for the full go-live +checklist (seed voices on NFS-SSD, smoke-test a synth, watch the neighbours). + +## Rollback (instant, no data loss) + +- **Narration off:** set `TTS_MODE=none` (or drop the three `TTS_*` lines) in + `stacks/tripit/main.tf` → `tg apply --stack tripit`. The bake makes no audio; + playback falls back to browser TTS. Cached `story_audio` rows are harmless. +- **Chatterbox off the GPU:** `kubectl -n tts scale deploy/chatterbox-tts + --replicas=0` (transient) and/or `tg destroy --stack tts`. Best-effort synth + means tripit bakes keep running audio-less — no error. +- Neither touches the resident GPU tenants (Option A never modifies them). diff --git a/stacks/tts/main.tf b/stacks/tts/main.tf new file mode 100644 index 00000000..2afdfad8 --- /dev/null +++ b/stacks/tts/main.tf @@ -0,0 +1,474 @@ +variable "image_tag" { + type = string + default = "latest" + description = "chatterbox-tts image tag. Use the 8-char git SHA in CI; :latest for local trials." +} + +# ───────────────────────────────────────────────────────────────────────────── +# Option-A off-peak control (see docs/plans/2026-06-08-chatterbox-tts-infra.md §3). +# The Deployment sits at replicas=0; a CronJob scales it to 1 at the window start +# ONLY IF a free-VRAM preflight passes, and another scales it back to 0 at window +# end. A guard CronJob yields the card mid-window if free VRAM drops below the +# floor (a resident woke up). tripit's bake is best-effort + idempotent, so a +# skipped/aborted window simply backfills on the next one (ADR-0002/0004). +# ───────────────────────────────────────────────────────────────────────────── + +variable "vram_free_floor_bytes" { + type = number + # OPEN ITEM — must be measured (§5 smoke test / §3.X). This is the minimum free + # VRAM the preflight requires before it will scale Chatterbox up, and the floor + # the guard yields below. Default = 6 GiB ≈ (a conservative guess for + # chatterbox-multilingual FP16 peak ~4 GiB + ~2 GiB headroom for the + # read→cudaMalloc race). RAISE/LOWER once the real T4 peak is captured from + # gpu_pod_memory_used_bytes{namespace="tts"} during a real synth. + default = 6442450944 + description = "Minimum free GPU VRAM (bytes) required before scaling Chatterbox up; guard yields below it." +} + +variable "gpu_total_bytes" { + type = number + default = 17179869184 # Tesla T4 = 16 GiB + description = "Total VRAM on the shared GPU. Free = this minus sum(gpu_pod_memory_used_bytes)." +} + +variable "offpeak_window_up_schedule" { + type = string + default = "0 2 * * *" # 02:00 Europe/London (see timezone on the CronJob) + description = "Cron schedule that fires the free-VRAM preflight + scale-up at window start." +} + +variable "offpeak_window_down_schedule" { + type = string + default = "0 6 * * *" # 06:00 Europe/London + description = "Cron schedule that scales Chatterbox back to 0 at window end." +} + +variable "offpeak_guard_schedule" { + type = string + default = "*/5 2-5 * * *" # every 5 min inside the 02:00–06:00 window + description = "Cron schedule for the mid-window guard that yields the card if free VRAM drops." +} + +locals { + namespace = "tts" + labels = { app = "chatterbox-tts" } + image = "forgejo.viktorbarzin.me/viktor/chatterbox-tts:${var.image_tag}" + + # config.yaml rendered into a ConfigMap, mounted at /app/config.yaml (the + # server's WORKDIR is /app). Voices, reference audio and the HF model cache + # all live on the NFS-SSD PVC (mounted at /data) so weights persist across + # restarts and load fast. server.port stays at the devnen default 8004; the + # Service remaps 8000->8004 so tripit's default TTS_BASE_URL works unchanged. + # + # model.repo_id = chatterbox-multilingual (ADR-0004; 23 languages for + # worldwide place-names). If the measured T4 VRAM peak is too high to coexist + # even off-peak, fall back to "chatterbox" (English, lighter) — a one-line + # change here (§3.X / §6 decision 3). + chatterbox_config = yamlencode({ + server = { + host = "0.0.0.0" + port = 8004 + } + model = { + repo_id = "chatterbox-multilingual" + } + tts_engine = { + device = "cuda" + predefined_voices_path = "/data/voices" + reference_audio_path = "/data/reference_audio" + } + }) + + # Shared script for the off-peak CronJobs. Reads the in-cluster + # gpu_pod_memory_used_bytes gauge (the per-namespace gauge the 2026-06-02 + # post-mortem built — host-PID attribution, no new exporter needed), sums it, + # and computes free = GPU_TOTAL - used. Pure POSIX + awk; curl is baked into + # the curl image. ACTION is "up" | "down" | "guard". + # up — scale to 1 ONLY IF free >= FLOOR (positive admission). + # guard — scale to 0 IF free < FLOOR (a resident woke mid-window; yield). + # down — scale to 0 unconditionally (window end). + # Heredoc escaping: only `$${...}` (literal `${...}`) is escaped — Terraform + # would otherwise try to interpolate it. Bare `$(...)`, `$((...))` and awk's + # `$NF` are literal `$` and pass through unescaped. + vram_gate_script = <<-EOT + set -eu + : "$${ACTION:?}" "$${FLOOR:?}" "$${GPU_TOTAL:?}" + METRICS_URL="http://gpu-pod-exporter.nvidia.svc.cluster.local:80/metrics" + + # Sum gpu_pod_memory_used_bytes across all pods. Missing metric / empty + # scrape => used=0 (card idle). -f so a non-200 scrape is a hard error we + # treat conservatively (skip scale-up). + if ! BODY="$(curl -sf -m 10 "$${METRICS_URL}")"; then + echo "WARN: could not scrape $${METRICS_URL}" + if [ "$${ACTION}" = "up" ]; then + echo "preflight: scrape failed -> NOT scaling up (fail-safe)"; exit 0 + fi + # For down/guard a failed scrape must NOT block yielding the card. + BODY="" + fi + USED="$(printf '%s\n' "$${BODY}" \ + | awk '/^gpu_pod_memory_used_bytes\{/ { s += $NF } END { printf "%d", s }')" + USED="$${USED:-0}" + FREE="$(( GPU_TOTAL - USED ))" + echo "GPU VRAM: used=$${USED} free=$${FREE} floor=$${FLOOR} (total=$${GPU_TOTAL})" + + case "$${ACTION}" in + up) + if [ "$${FREE}" -ge "$${FLOOR}" ]; then + echo "preflight PASS: free >= floor -> scaling chatterbox-tts to 1" + kubectl -n tts scale deploy/chatterbox-tts --replicas=1 + else + echo "preflight SKIP: free < floor -> leaving chatterbox-tts at 0 (retry next window)" + fi + ;; + guard) + if [ "$${FREE}" -lt "$${FLOOR}" ]; then + echo "guard TRIP: free < floor -> yielding the card, scaling chatterbox-tts to 0" + kubectl -n tts scale deploy/chatterbox-tts --replicas=0 + else + echo "guard OK: free >= floor -> chatterbox-tts may keep running" + fi + ;; + down) + echo "window end -> scaling chatterbox-tts to 0" + kubectl -n tts scale deploy/chatterbox-tts --replicas=0 + ;; + esac + EOT + + # Common spec for the three off-peak CronJobs. Each runs one bitnami/kubectl + # pod (in-cluster SA, no kubeconfig) executing the shared gate script with a + # different ACTION. timezone pins the window to Europe/London regardless of + # node TZ. + offpeak_cronjobs = { + chatterbox-window-up = { + schedule = var.offpeak_window_up_schedule + action = "up" + } + chatterbox-window-down = { + schedule = var.offpeak_window_down_schedule + action = "down" + } + chatterbox-vram-guard = { + schedule = var.offpeak_guard_schedule + action = "guard" + } + } +} + +resource "kubernetes_namespace" "tts" { + metadata { + name = local.namespace + labels = { + tier = local.tiers.gpu + "istio-injection" = "disabled" + "keel.sh/enrolled" = "true" + } + } + lifecycle { + ignore_changes = [metadata[0].labels["goldilocks.fairwinds.com/vpa-update-mode"]] + } +} + +# Model weights + voices on NFS-SSD (fast load), RWX so a seed Job / kubectl cp +# can write the predefined voices + narrator reference WAV while the Deployment +# mounts it. Path /srv/nfs-ssd/chatterbox on the Proxmox host. Mirrors +# llama-cpp's nfs_models. First start downloads the model into /data/hf_cache +# (HF_HOME below), so weights persist across pod restarts. +module "nfs_models" { + source = "../../modules/kubernetes/nfs_volume" + name = "chatterbox-models" + namespace = kubernetes_namespace.tts.metadata[0].name + nfs_server = "192.168.1.127" + nfs_path = "/srv/nfs-ssd/chatterbox" + storage = "20Gi" # multilingual weights + HF cache + voices headroom +} + +resource "kubernetes_config_map" "chatterbox_config" { + metadata { + name = "chatterbox-config" + namespace = kubernetes_namespace.tts.metadata[0].name + labels = local.labels + } + data = { + "config.yaml" = local.chatterbox_config + } +} + +# Single Deployment running the devnen Chatterbox-TTS-Server (OpenAI-compatible +# /v1/audio/speech). Sits at replicas=0 — the off-peak CronJobs below scale it +# to 1 only when the free-VRAM preflight passes (Option A), and back to 0 at +# window end. wait_for_rollout=false so apply never blocks on a pod that is +# intentionally scaled to 0. +resource "kubernetes_deployment" "chatterbox" { + metadata { + name = "chatterbox-tts" + namespace = kubernetes_namespace.tts.metadata[0].name + labels = merge(local.labels, { tier = local.tiers.gpu }) + } + wait_for_rollout = false + spec { + # Off-peak control owns the replica count at runtime (CronJobs scale 0<->1). + # Declare 0 here so a plain `tg apply` outside the window doesn't wake the + # card. ignore_changes on replicas (below) stops apply from fighting the + # CronJob's scale. + replicas = 0 + strategy { type = "Recreate" } + selector { + match_labels = { app = "chatterbox-tts" } + } + template { + metadata { + labels = { app = "chatterbox-tts" } + annotations = { + "checksum/config" = sha256(local.chatterbox_config) + } + } + spec { + node_selector = { "nvidia.com/gpu.present" = "true" } + toleration { + key = "nvidia.com/gpu" + operator = "Equal" + value = "true" + effect = "NoSchedule" + } + # C-hardening (§3.RECOMMENDATION.3): Chatterbox is a polite, best-effort + # batch tenant — give it the regular tier-2-gpu priority (600000) so it + # is ALWAYS the pod evicted under GPU-node pressure, never immich-ml / + # frigate / llama-swap. This relies on the `tts` namespace being EXCLUDED + # from the Kyverno `inject-gpu-workload-priority` policy (which would + # otherwise stamp the immich-equal gpu-workload=1,200,000 priority on any + # nvidia.com/gpu pod). That exclusion is the two-line edit to the kyverno + # stack flagged in the PR. Without it, this priority_class_name is + # overwritten on pod CREATE and Chatterbox would compete as an equal. + priority_class_name = "tier-2-gpu" + + image_pull_secrets { name = "registry-credentials" } + + container { + name = "chatterbox-tts" + image = local.image + port { + container_port = 8004 + name = "http" + } + + # T4 is Turing — NO bf16 (ADR-0004). Pin off; run FP16/FP32. + env { + name = "TTS_BF16" + value = "off" + } + # Park the HuggingFace cache on the NFS-SSD PVC so model weights + # download once and persist across pod restarts (the pod is recreated + # every window). The devnen compose mounts HF cache at /app/hf_cache; + # point HF_HOME at the PVC instead. + env { + name = "HF_HOME" + value = "/data/hf_cache" + } + env { + name = "HF_HUB_CACHE" + value = "/data/hf_cache" + } + + volume_mount { + name = "config" + mount_path = "/app/config.yaml" + sub_path = "config.yaml" + } + volume_mount { + name = "models" + mount_path = "/data" + } + + # /v1/audio/voices is cheap and only 200s once the model is loaded — + # so it gates real readiness. First start downloads the model, which + # is slow; the generous failure_threshold absorbs that. + readiness_probe { + http_get { + path = "/v1/audio/voices" + port = 8004 + } + initial_delay_seconds = 20 + period_seconds = 15 + failure_threshold = 12 + } + liveness_probe { + http_get { + path = "/v1/audio/voices" + port = 8004 + } + initial_delay_seconds = 120 + period_seconds = 30 + failure_threshold = 5 + } + resources { + requests = { + cpu = "200m" + memory = "2Gi" + } + limits = { + memory = "8Gi" + "nvidia.com/gpu" = "1" # ONE time-slice (operator advertises 100), NOT the whole card + } + } + } + + volume { + name = "config" + config_map { + name = kubernetes_config_map.chatterbox_config.metadata[0].name + } + } + volume { + name = "models" + persistent_volume_claim { + claim_name = module.nfs_models.claim_name + } + } + } + } + } + lifecycle { + ignore_changes = [ + # Off-peak CronJobs own the replica count — don't let apply reset it. + spec[0].replicas, + spec[0].template[0].spec[0].dns_config, # KYVERNO_LIFECYCLE_V1 + spec[0].template[0].spec[0].container[0].image, # KEEL_IGNORE_IMAGE + metadata[0].annotations["keel.sh/match-tag"], + metadata[0].annotations["keel.sh/policy"], + metadata[0].annotations["keel.sh/trigger"], + metadata[0].annotations["keel.sh/pollSchedule"], # KYVERNO_LIFECYCLE_V2 + metadata[0].annotations["kubernetes.io/change-cause"], + metadata[0].annotations["deployment.kubernetes.io/revision"], + spec[0].template[0].metadata[0].annotations["keel.sh/update-time"], + ] + } +} + +resource "kubernetes_service" "chatterbox" { + metadata { + name = "chatterbox-tts" + namespace = kubernetes_namespace.tts.metadata[0].name + labels = local.labels + annotations = { + # Prometheus annotation-based scrape (mirrors tripit). The devnen server + # has no /metrics; this monitors liveness via the blackbox path and keeps + # the Service in the scrape set if a /metrics endpoint is added later. + "prometheus.io/scrape" = "true" + "prometheus.io/path" = "/v1/audio/voices" + "prometheus.io/port" = "8000" + } + } + spec { + type = "ClusterIP" # in-cluster only — never ingressed (no token needed) + selector = { app = "chatterbox-tts" } + port { + name = "http" + port = 8000 # tripit's default TTS_BASE_URL port + target_port = 8004 # the devnen server's actual listen port + } + } +} + +# ───────────────────────────────────────────────────────────────────────────── +# Option-A off-peak control: SA + Role (scale the Deployment) + RoleBinding + +# three CronJobs (window-up preflight, mid-window guard, window-down). Mirrors +# the nextcloud-watchdog in-cluster-kubectl pattern (SA → Role → bitnami/kubectl +# CronJob, no kubeconfig). +# ───────────────────────────────────────────────────────────────────────────── + +resource "kubernetes_service_account" "offpeak" { + metadata { + name = "chatterbox-offpeak" + namespace = kubernetes_namespace.tts.metadata[0].name + } +} + +resource "kubernetes_role" "offpeak" { + metadata { + name = "chatterbox-offpeak" + namespace = kubernetes_namespace.tts.metadata[0].name + } + # get + patch on the deployment scale subresource is all the gate needs. + rule { + api_groups = ["apps"] + resources = ["deployments", "deployments/scale"] + verbs = ["get", "patch"] + } +} + +resource "kubernetes_role_binding" "offpeak" { + metadata { + name = "chatterbox-offpeak" + namespace = kubernetes_namespace.tts.metadata[0].name + } + role_ref { + api_group = "rbac.authorization.k8s.io" + kind = "Role" + name = kubernetes_role.offpeak.metadata[0].name + } + subject { + kind = "ServiceAccount" + name = kubernetes_service_account.offpeak.metadata[0].name + namespace = kubernetes_namespace.tts.metadata[0].name + } +} + +resource "kubernetes_cron_job_v1" "offpeak" { + for_each = local.offpeak_cronjobs + + metadata { + name = each.key + namespace = kubernetes_namespace.tts.metadata[0].name + labels = local.labels + } + spec { + schedule = each.value.schedule + timezone = "Europe/London" + concurrency_policy = "Forbid" + starting_deadline_seconds = 120 + successful_jobs_history_limit = 1 + failed_jobs_history_limit = 3 + job_template { + metadata { labels = local.labels } + spec { + backoff_limit = 1 + active_deadline_seconds = 120 + ttl_seconds_after_finished = 300 + template { + metadata { labels = local.labels } + spec { + service_account_name = kubernetes_service_account.offpeak.metadata[0].name + restart_policy = "Never" + container { + name = "vram-gate" + image = "bitnami/kubectl:latest" + command = ["/bin/bash", "-c", local.vram_gate_script] + env { + name = "ACTION" + value = each.value.action + } + env { + name = "FLOOR" + value = tostring(var.vram_free_floor_bytes) + } + env { + name = "GPU_TOTAL" + value = tostring(var.gpu_total_bytes) + } + resources { + requests = { cpu = "20m", memory = "64Mi" } + limits = { memory = "128Mi" } + } + } + } + } + } + } + } + lifecycle { + # KYVERNO_LIFECYCLE_V1: Kyverno mutates dns_config with ndots=2 on CronJobs. + ignore_changes = [spec[0].job_template[0].spec[0].template[0].spec[0].dns_config] + } +} diff --git a/stacks/tts/terragrunt.hcl b/stacks/tts/terragrunt.hcl new file mode 100644 index 00000000..6bc02966 --- /dev/null +++ b/stacks/tts/terragrunt.hcl @@ -0,0 +1,36 @@ +include "root" { + path = find_in_parent_folders() +} + +dependency "platform" { + config_path = "../platform" + skip_outputs = true +} + +dependency "vault" { + config_path = "../vault" + skip_outputs = true +} + +# tts: in-cluster text-to-speech for tripit's "Tour guide" narration. +# One Deployment of `forgejo.viktorbarzin.me/viktor/chatterbox-tts` (devnen +# Chatterbox-TTS-Server, OpenAI-compatible /v1/audio/speech) at a single +# ClusterIP Service `chatterbox-tts.tts.svc:8000` (server listens on 8004; +# the Service remaps). Requests ONE time-slice of the shared T4 +# (nvidia.com/gpu=1) — a slice, not the card. +# +# OOM-avoidance (Option A, docs/plans/2026-06-08-chatterbox-tts-infra.md §3): +# the Deployment sits at replicas=0; an off-peak CronJob scales it to 1 at the +# 02:00–06:00 Europe/London window ONLY IF a free-VRAM preflight passes +# (gpu_pod_memory_used_bytes from gpu-pod-exporter), a guard CronJob yields the +# card mid-window if a resident wakes, and a window-down CronJob scales back to +# 0. tripit's bake is best-effort + cached-forever (ADR-0002/0004), so a +# skipped/aborted window simply backfills next time — no latency SLA. +# +# Polite-tenant hardening: the `tts` namespace must be EXCLUDED from the kyverno +# `inject-gpu-workload-priority` policy (a separate two-line edit to the kyverno +# stack) so Chatterbox keeps tier-2-gpu priority (600000) and is always the pod +# evicted under pressure — never immich-ml/frigate/llama-swap. +# +# Image is built from the devnen repo + pushed to Forgejo — see this stack's +# README.md for the exact docker build + push commands. From fbcc3302141a536a4bd2ec5f64527da5016af50d Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 12:09:14 +0000 Subject: [PATCH 08/24] workstation: v2 membership implementation plan [ci skip] 8 tasks: engine derive_os_user + roster_from_members (TDD); read-only Authentik token (TF); setup-devvm.sh stages it; provisioner sources T3 Users members from the Authentik API (replaces roster.yaml); Authentik-managed membership + legacy os_user attributes; retire roster.yaml; e2e add/remove smoke. Pairs with the 2026-06-09 design doc. Co-Authored-By: Claude Opus 4.8 --- ...9-workstation-authentik-membership-plan.md | 469 ++++++++++++++++++ 1 file changed, 469 insertions(+) create mode 100644 docs/plans/2026-06-09-workstation-authentik-membership-plan.md diff --git a/docs/plans/2026-06-09-workstation-authentik-membership-plan.md b/docs/plans/2026-06-09-workstation-authentik-membership-plan.md new file mode 100644 index 00000000..1803e04d --- /dev/null +++ b/docs/plans/2026-06-09-workstation-authentik-membership-plan.md @@ -0,0 +1,469 @@ +# Workstation Membership v2 — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax. This is **infra** work: the engine tasks are real pytest TDD; the host/Authentik tasks "verify" via an idempotent re-run + a smoke check with expected output. Honor the Terraform-only rule for cluster/Authentik changes (`scripts/tg apply`); devvm host scripts are the accepted exception. Claim `host:devvm` before host mutations and `stack:authentik` before applying Authentik. + +**Goal:** Make the Authentik `T3 Users` group membership the single source of truth for who gets a devvm workstation account, identified by email; retire `roster.yaml`. + +**Architecture:** The provisioner reads `T3 Users` members from the Authentik API (read-only token) instead of `roster.yaml`. A pure engine derives the Linux `os_user` from each member's email (or an `os_user` Authentik attribute override) and produces the same desired-state shape v1 already applies. Workstation access stays fully decoupled from cluster RBAC (`k8s_users` untouched). wizard is special-cased as the admin/owner. + +**Tech Stack:** Python (pure engine, pytest) + Bash (provisioner) + `jq`/`curl` (Authentik API) + Terraform (`stacks/authentik`: read-only token, drop HCL members). + +**Design:** `infra/docs/plans/2026-06-09-workstation-authentik-membership-design.md`. + +--- + +## File structure + +- Modify: `infra/scripts/workstation/roster_engine.py` — add `derive_os_user()` + `roster_from_members()` (pure). +- Modify: `infra/scripts/workstation/test_roster_engine.py` — tests for the two new functions. +- Modify: `infra/scripts/t3-provision-users.sh` — source members from the Authentik API instead of `roster.yaml`. +- Modify: `infra/scripts/workstation/setup-devvm.sh` — drop the read-only Authentik token to `/etc/t3-serve/authentik-token`. +- Create: `infra/stacks/authentik/t3-provision-token.tf` — read-only service account + API token. +- Modify: `infra/stacks/authentik/t3-users.tf` — drop the HCL `users` list (membership becomes Authentik-managed). +- Delete: `infra/scripts/workstation/roster.yaml` (Task 7). +- Modify: `infra/.claude/reference/service-catalog.md`, `infra/docs/architecture/multi-tenancy.md` (Task 7). + +--- + +## Task 1: Engine — `derive_os_user()` + +**Files:** Modify `infra/scripts/workstation/roster_engine.py`; Test `infra/scripts/workstation/test_roster_engine.py` + +- [ ] **Step 1: Write the failing tests** (append to `test_roster_engine.py`) + +```python +# --- derive_os_user: email/attribute -> Linux username (v2) --- + +def test_derive_os_user_sanitizes_email_local_part(): + assert eng.derive_os_user("emil.barzin@gmail.com", None) == "emil_barzin" + + +def test_derive_os_user_attribute_overrides(): + assert eng.derive_os_user("emil.barzin@gmail.com", "emo") == "emo" + + +def test_derive_os_user_lowercases_and_replaces_unsafe_runs(): + assert eng.derive_os_user("Weird.Name+tag@x.com", None) == "weird_name_tag" + + +def test_derive_os_user_truncates_to_32(): + long = ("a" * 40) + "@x.com" + assert eng.derive_os_user(long, None) == "a" * 32 + + +def test_derive_os_user_blank_attribute_is_ignored(): + assert eng.derive_os_user("emil.barzin@gmail.com", "") == "emil_barzin" +``` + +- [ ] **Step 2: Run to verify they fail** + +Run: `cd infra/scripts/workstation && python3 -m pytest test_roster_engine.py -k derive_os_user -q` +Expected: FAIL — `AttributeError: module 'roster_engine' has no attribute 'derive_os_user'` + +- [ ] **Step 3: Implement** (add to `roster_engine.py`, after `RosterError`) + +```python +import re + +_MAX_USERNAME = 32 + + +def derive_os_user(email: str, os_user_attr: str | None) -> str: + """Linux username for a workstation member: the explicit `os_user` Authentik + attribute if set, else the email local-part sanitized to a valid username + (lowercase; runs of non [a-z0-9_-] -> '_'; stripped; <=32 chars).""" + if os_user_attr: + return os_user_attr + local = email.split("@", 1)[0].lower() + cleaned = re.sub(r"[^a-z0-9_-]+", "_", local).strip("_") + return cleaned[:_MAX_USERNAME] +``` + +- [ ] **Step 4: Run to verify they pass** + +Run: `python3 -m pytest test_roster_engine.py -k derive_os_user -q` +Expected: PASS (5 passed) + +- [ ] **Step 5: Commit** + +```bash +cd /home/wizard/code/infra +git add scripts/workstation/roster_engine.py scripts/workstation/test_roster_engine.py +git commit -m "workstation: engine derive_os_user (email/attribute -> Linux username)" +``` + +--- + +## Task 2: Engine — `roster_from_members()` + +Builds a `Roster` (the v1 type `derive_desired_state` already consumes) from the Authentik member list, so the existing tested derivation is reused unchanged. + +**Files:** Modify `roster_engine.py`; Test `test_roster_engine.py` + +- [ ] **Step 1: Write the failing tests** + +```python +# --- roster_from_members: Authentik members -> Roster (v2) --- + +MEMBERS = [ + {"email": "vbarzin@gmail.com", "os_user": "wizard"}, + {"email": "emil.barzin@gmail.com", "os_user": "emo"}, + {"email": "ancaelena98@gmail.com", "os_user": "ancamilea"}, +] +ADMINS = {"vbarzin@gmail.com"} + + +def test_roster_from_members_maps_identity_fields(): + r = eng.roster_from_members(MEMBERS, ADMINS) + u = r.users["emo"] + assert u.os_user == "emo" + assert u.authentik_user == "emil.barzin" # email local-part = t3-dispatch key + assert u.k8s_user == "emil.barzin@gmail.com" # email = identity + assert u.tier == "power-user" # non-admin + + +def test_roster_from_members_admin_by_email(): + r = eng.roster_from_members(MEMBERS, ADMINS) + assert r.users["wizard"].tier == "admin" + + +def test_roster_from_members_derives_os_user_when_no_override(): + r = eng.roster_from_members([{"email": "jane.doe@x.com", "os_user": None}], set()) + assert "jane_doe" in r.users + assert r.users["jane_doe"].tier == "power-user" + + +def test_roster_from_members_raises_on_os_user_collision(): + members = [{"email": "a@x.com", "os_user": "dup"}, {"email": "b@y.com", "os_user": "dup"}] + with pytest.raises(eng.RosterError, match="collision"): + eng.roster_from_members(members, set()) + + +def test_roster_from_members_reuses_derive_desired_state(): + r = eng.roster_from_members(MEMBERS, ADMINS) + ds = eng.derive_desired_state(r, {"wizard": 3773, "emo": 3774, "ancamilea": 3775}) + assert ds.dispatch["emil.barzin"] == {"os_user": "emo", "port": 3774} + assert ds.accounts["wizard"].groups == ("code-shared", "docker", "sudo") + assert ds.accounts["emo"].groups == () +``` + +- [ ] **Step 2: Run to verify they fail** + +Run: `python3 -m pytest test_roster_engine.py -k roster_from_members -q` +Expected: FAIL — `AttributeError: ... 'roster_from_members'` + +- [ ] **Step 3: Implement** (add to `roster_engine.py`) + +```python +def roster_from_members(members: list[dict], admin_emails: set[str]) -> Roster: + """Build a Roster from Authentik `T3 Users` members. Each member dict has + `email` and optional `os_user`. tier = admin iff the email is in admin_emails, + else power-user (a non-admin workstation: no groups, locked clone). Raises on + an os_user collision (two emails resolving to the same Linux username).""" + users: dict[str, User] = {} + for m in members: + email = m["email"] + os_user = derive_os_user(email, m.get("os_user")) + if os_user in users: + raise RosterError( + f"os_user collision: {email!r} and {users[os_user].k8s_user!r} " + f"both resolve to {os_user!r} (set an os_user attribute to disambiguate)" + ) + tier = "admin" if email in admin_emails else "power-user" + users[os_user] = User( + os_user=os_user, + authentik_user=email.split("@", 1)[0], + k8s_user=email, + tier=tier, + namespaces=(), + ) + return Roster(users) +``` + +- [ ] **Step 4: Run the whole suite** + +Run: `python3 -m pytest test_roster_engine.py -q && ruff check roster_engine.py test_roster_engine.py` +Expected: PASS (all, incl. the v1 tests) + ruff clean + +- [ ] **Step 5: Commit** + +```bash +git add scripts/workstation/roster_engine.py scripts/workstation/test_roster_engine.py +git commit -m "workstation: engine roster_from_members (Authentik members -> Roster, reuses derive)" +``` + +--- + +## Task 3: Read-only Authentik token (Terraform) + +**Files:** Create `infra/stacks/authentik/t3-provision-token.tf` + +- [ ] **Step 1: Write the resources** (service account + API token + view permissions) + +```hcl +# Read-only service account whose token the devvm provisioner uses to list +# "T3 Users" members. View-only: it can read users + groups, nothing else. +resource "authentik_user" "t3_provision" { + username = "t3-provision-bot" + name = "T3 Provision (read-only)" + type = "service_account" + path = "service-accounts" +} + +resource "authentik_token" "t3_provision" { + identifier = "t3-provision-readonly" + user = authentik_user.t3_provision.id + intent = "api" + description = "devvm t3-provision-users: read T3 Users membership" + retrieve_key = true +} + +# Global view permissions for the service account (users + groups read only). +resource "authentik_rbac_permission_user" "t3_provision_view_user" { + user = authentik_user.t3_provision.id + permission = "authentik_core.view_user" +} + +resource "authentik_rbac_permission_user" "t3_provision_view_group" { + user = authentik_user.t3_provision.id + permission = "authentik_core.view_group" +} + +output "t3_provision_token" { + value = authentik_token.t3_provision.key + sensitive = true +} +``` + +- [ ] **Step 2: Apply** (claim first) + +```bash +~/code/scripts/presence claim stack:authentik --purpose "v2: read-only t3-provision token" +export VAULT_ADDR=https://vault.viktorbarzin.me && vault login -method=oidc +cd /home/wizard/code/infra/stacks/authentik && ../../scripts/tg apply -target=authentik_user.t3_provision -target=authentik_token.t3_provision -target=authentik_rbac_permission_user.t3_provision_view_user -target=authentik_rbac_permission_user.t3_provision_view_group --non-interactive +``` +Expected: 4 added. (If the `authentik_rbac_permission_user` resource/permission codename differs in the installed provider, run `../../scripts/tg console` / check the provider docs and adjust the codename; verify in Step 3.) + +- [ ] **Step 3: Store the token in Vault + verify it is read-only** + +```bash +TOK=$(../../scripts/tg output -raw t3_provision_token) +vault kv patch secret/authentik t3_provision_token="$TOK" +# verify: can LIST T3 Users members... +curl -sk -H "Authorization: Bearer $TOK" "https://authentik.viktorbarzin.me/api/v3/core/users/?groups_by_name=T3%20Users" | jq -r '.results[].email' +# ...but CANNOT write (expect 403): +curl -sk -o /dev/null -w '%{http_code}\n' -X PATCH -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' -d '{"name":"x"}' "https://authentik.viktorbarzin.me/api/v3/core/users/14/" +``` +Expected: the three emails listed; the PATCH returns `403`. + +- [ ] **Step 4: Commit** + +```bash +git add stacks/authentik/t3-provision-token.tf +git commit -m "workstation: read-only Authentik token for the t3-provision membership query" +``` + +--- + +## Task 4: setup-devvm.sh — stage the token for the root provisioner + +**Files:** Modify `infra/scripts/workstation/setup-devvm.sh` + +- [ ] **Step 1: Add a token-staging step** (after step 6, before the final `log "OK"`). The hourly provisioner runs as root with no Vault token, so `setup-devvm.sh` (run by wizard, who can read Vault) drops it to a root-only file. + +```bash +# 8) stage the read-only Authentik token for the root provisioner's membership query. +if command -v vault >/dev/null; then + export VAULT_ADDR="${VAULT_ADDR:-https://vault.viktorbarzin.me}" + if tok="$(vault kv get -field=t3_provision_token secret/authentik 2>/dev/null)"; then + install -m 0600 /dev/stdin /etc/t3-serve/authentik-token <<<"$tok" + log "staged /etc/t3-serve/authentik-token (read-only Authentik API)" + else + log "WARN: t3_provision_token not in Vault -> Authentik membership query will be skipped" + fi +fi +``` + +- [ ] **Step 2: Run + verify** + +Run: `sudo bash /home/wizard/code/infra/scripts/workstation/setup-devvm.sh 2>&1 | grep -E 'authentik-token|OK'` then `sudo stat -c '%a %U' /etc/t3-serve/authentik-token` +Expected: "staged ... authentik-token" + `OK`; perms `600 root`. + +- [ ] **Step 3: Commit** + +```bash +git add scripts/workstation/setup-devvm.sh +git commit -m "workstation: setup-devvm.sh stages the read-only Authentik token (root-only)" +``` + +--- + +## Task 5: Provisioner — source members from Authentik (replace roster.yaml) + +**Files:** Modify `infra/scripts/t3-provision-users.sh` + +- [ ] **Step 1: Add a members-fetch + swap the engine call.** Replace the roster-read/derive block. Fetch members from Authentik (best-effort); build the members JSON `[{email, os_user}]`; pass to the engine via a new `--members-json` mode on `derive`. + +First extend the engine CLI (`roster_engine.py` `_main`): add `derive-members` that reads a members JSON + ports JSON + admin emails and emits the same desired-state JSON. + +```python +# in _main(), add a subparser: + pm = sub.add_parser("derive-members", help="desired state from an Authentik member list") + pm.add_argument("--members-json", required=True) + pm.add_argument("--ports-json", required=True) + pm.add_argument("--admin-emails", default="", help="comma-separated admin emails") + # ...in the dispatch: + if args.cmd == "derive-members": + with open(args.members_json, encoding="utf-8") as fh: + members = json.load(fh) + with open(args.ports_json, encoding="utf-8") as fh: + ports = json.load(fh) + admins = {e for e in args.admin_emails.split(",") if e} + ds = derive_desired_state(roster_from_members(members, admins), ports) + json.dump(_desired_state_to_dict(ds), sys.stdout, indent=2, sort_keys=True) + sys.stdout.write("\n") + return 0 +``` + +In `t3-provision-users.sh`, replace the `ROSTER`/validate/derive section with: + +```bash +AUTHENTIK_URL="${AUTHENTIK_URL:-https://authentik.viktorbarzin.me}" +TOKEN_FILE="${TOKEN_FILE:-/etc/t3-serve/authentik-token}" +T3_GROUP="${T3_GROUP:-T3 Users}" +ADMIN_EMAILS="${WORKSTATION_ADMIN_EMAILS:-vbarzin@gmail.com}" + +members_file="$(mktemp)"; trap 'rm -f "$ports_file" "$members_file" "${desired_file:-}"' EXIT +if [[ -r "$TOKEN_FILE" ]]; then + tok="$(cat "$TOKEN_FILE")" + if curl -sf -H "Authorization: Bearer $tok" --get \ + --data-urlencode "groups_by_name=$T3_GROUP" \ + "$AUTHENTIK_URL/api/v3/core/users/" \ + | jq -c '[.results[] | select(.is_active) | {email: .email, os_user: (.attributes.os_user // null)}]' \ + > "$members_file" && [[ -s "$members_file" ]]; then + : + else + log "WARN: Authentik membership query failed -> no membership change this run"; echo '[]' > "$members_file" + SKIP_RECONCILE=1 + fi +else + log "WARN: $TOKEN_FILE absent -> no membership change this run"; echo '[]' > "$members_file"; SKIP_RECONCILE=1 +fi + +if [[ "${SKIP_RECONCILE:-0}" == 1 ]]; then log "reconcile skipped (no Authentik membership)"; exit 0; fi + +desired_file="$(mktemp)" +python3 "$ENGINE" derive-members --members-json "$members_file" --ports-json "$ports_file" --admin-emails "$ADMIN_EMAILS" > "$desired_file" +jq -e . "$desired_file" >/dev/null || { echo "[t3-provision] derive-members produced invalid JSON" >&2; exit 1; } +``` + +(Keep steps 4-6 of the existing script — accounts/groups/clone/kubeconfig, .env/enable, regen map/dispatch — unchanged; they consume `$desired_file`.) + +- [ ] **Step 2: shellcheck + DRY_RUN** (with the staged token present) + +Run: `cd /home/wizard/code/infra/scripts && shellcheck -S warning t3-provision-users.sh && sudo DRY_RUN=1 bash t3-provision-users.sh 2>&1 | grep -iE 'clone|kubeconfig|reconcile|WARN'` +Expected: shellcheck clean; dry-run lists the current members, no account creations (all exist), "reconcile complete (DRY-RUN)". + +- [ ] **Step 3: Real run + verify it reproduces current state** + +Run: `sudo jq -S . /etc/t3-serve/dispatch.json > /tmp/d1; sudo DRY_RUN=0 bash t3-provision-users.sh >/dev/null 2>&1; sudo jq -S . /etc/t3-serve/dispatch.json > /tmp/d2; diff /tmp/d1 /tmp/d2 && echo SAME; id -nG emo` +Expected: `SAME` (dispatch content unchanged); emo groups unchanged. Redeploy: `sudo install -m0755 t3-provision-users.sh /usr/local/bin/t3-provision-users`. + +- [ ] **Step 4: Commit** + +```bash +git add scripts/t3-provision-users.sh scripts/workstation/roster_engine.py scripts/workstation/test_roster_engine.py +git commit -m "workstation: provisioner sources members from Authentik T3 Users (replaces roster.yaml)" +``` + +--- + +## Task 6: Authentik — Authentik-managed membership + legacy os_user attributes + +**Files:** Modify `infra/stacks/authentik/t3-users.tf`; set user attributes via API. + +- [ ] **Step 1: Set the legacy os_user attributes** (the 3 existing accounts don't derive from their emails). Read-merge-write so existing attributes are preserved (Authentik PATCH replaces the `attributes` dict). + +```bash +export VAULT_ADDR=https://vault.viktorbarzin.me +TOK=$(vault kv get -field=tf_api_token secret/authentik) +A=https://authentik.viktorbarzin.me/api/v3 +set_os_user() { # $1=username $2=os_user + local pk attrs + pk=$(curl -sk -H "Authorization: Bearer $TOK" "$A/core/users/?username=$1" | jq '.results[0].pk') + attrs=$(curl -sk -H "Authorization: Bearer $TOK" "$A/core/users/$pk/" | jq -c --arg o "$2" '.attributes + {os_user:$o}') + curl -sk -X PATCH -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \ + -d "{\"attributes\":$attrs}" "$A/core/users/$pk/" | jq -r '.username + " os_user=" + .attributes.os_user' +} +set_os_user "vbarzin@gmail.com" wizard +set_os_user "emil.barzin@gmail.com" emo +set_os_user "ancaelena98@gmail.com" ancamilea +``` +Expected: three lines confirming `os_user=` each. + +- [ ] **Step 2: Drop the HCL `users` list** so membership is Authentik-managed. Edit `t3-users.tf`: remove the `users = [...]` argument from `resource "authentik_group" "t3_users"` (keep the `data "authentik_user"` lookups removed too if now unused). Leave the group resource (name only). + +```hcl +resource "authentik_group" "t3_users" { + name = "T3 Users" + # Membership is managed in Authentik (UI/API), not Terraform — the devvm + # provisioner reconciles workstation accounts from this group's members. +} +``` + +- [ ] **Step 3: Apply + verify members unchanged** + +```bash +cd /home/wizard/code/infra/stacks/authentik && ../../scripts/tg apply -target=authentik_group.t3_users --non-interactive +curl -sk -H "Authorization: Bearer $TOK" "$A/core/groups/?search=T3%20Users" | jq -r '.results[0].users_obj[].username' +``` +Expected: apply shows the group updated (no member change / the `users` field no longer managed); the 3 members still listed. + +- [ ] **Step 4: Commit** + +```bash +git add stacks/authentik/t3-users.tf +git commit -m "workstation: T3 Users membership is Authentik-managed (drop HCL member list)" +``` + +--- + +## Task 7: Retire roster.yaml + update docs + +**Files:** Delete `infra/scripts/workstation/roster.yaml`; modify `service-catalog.md`, `multi-tenancy.md`. + +- [ ] **Step 1: Confirm nothing reads roster.yaml anymore** + +Run: `grep -rn 'roster.yaml\|roster_engine.*roster\b' /home/wizard/code/infra/scripts /home/wizard/code/infra/docs | grep -v 'load_roster\|test_\|design.md\|-plan.md'` +Expected: no live references in the provisioner (the engine keeps `load_roster` for tests, that's fine). + +- [ ] **Step 2: Delete it + update the service-catalog t3code row** — change "Source of truth = roster.yaml" to "Source of truth = the Authentik `T3 Users` group (members → accounts via the read-only API token); `os_user` from the email or a per-user `os_user` attribute". Update the multi-tenancy Workstation section's "single source of truth" line likewise. + +```bash +git rm scripts/workstation/roster.yaml +# (edit service-catalog.md + multi-tenancy.md per above) +``` + +- [ ] **Step 3: Commit** + +```bash +git add scripts/workstation/roster.yaml .claude/reference/service-catalog.md docs/architecture/multi-tenancy.md +git commit -m "workstation: retire roster.yaml — Authentik T3 Users group is the membership SSoT" +``` + +--- + +## Task 8: End-to-end smoke (add + remove a throwaway member) + +- [ ] **Step 1: Add a throwaway test member** to `T3 Users` in Authentik (a test user, or temporarily add an existing one), set no `os_user` attribute. Run `sudo /usr/local/bin/t3-provision-users` and confirm an account `` is created (`id `), with a locked `~/code` (secret file shows `GITCRYPT`) and `~/.kube/config`. +- [ ] **Step 2: Remove the test member** from the group; run the reconcile; confirm they drop out of `/etc/ttyd-user-map` + `dispatch.json` (the reversible cut). Leave `userdel` to the gated offboarding runbook. +- [ ] **Step 3: Verify the 3 real users are intact** — `id emo` (groups unchanged), emo/ancamilea/wizard still in `dispatch.json`, their `t3-serve@` active, emo's locked clone + ancamilea's intact. + +--- + +## Self-review + +- **Spec coverage:** Authentik-as-SSoT (Tasks 5,6) · email identity + os_user derive/override (Tasks 1,6) · provisioner reads the API (Task 5) · read-only token for the root timer (Tasks 3,4) · roster.yaml retires (Task 7) · k8s_users/cluster untouched (no task touches it) · wizard special-cased (admin_emails, Task 2). All covered. +- **Type consistency:** `derive_os_user(email, os_user_attr)` and `roster_from_members(members, admin_emails)` used consistently; `members` dicts are `{email, os_user}`; reuses the existing `User`/`Roster`/`derive_desired_state`/`DesiredState`. +- **apiserver-OIDC:** out of scope here (kubectl auth method only) — flagged in the design; the generic kubeconfig task is unchanged from v1. +- **Open risk:** the `authentik_rbac_permission_user` resource name / permission codenames may differ in the installed provider version (Task 3) — Step 3 verifies read-works/write-403 and says to adjust if needed. From eeadf0f85df5cf87881518a12f453cd5385eb2ca Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 14:05:44 +0000 Subject: [PATCH 09/24] workstation: share admin Claude subscription with non-admins (CLAUDE_CODE_OAUTH_TOKEN) Non-admins without their own ~/.claude login get the shared long-lived sk-ant-oat01 token injected into their t3-serve env, so their agent authenticates against the admin's subscription. setup-devvm.sh stages it from Vault secret/workstation.claude_oauth_token (root-readable); the provisioner's install_user_claude_token injects per-user, if-absent (never clobbers emo's own login). Live-fixed anca (verified AUTHOK); this codifies it for reproducibility + future users. NOT pushed (shared-tree divergence hold). Co-Authored-By: Claude Opus 4.8 --- scripts/t3-provision-users.sh | 27 ++++++++++++++++++++++++++- scripts/workstation/setup-devvm.sh | 17 +++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/scripts/t3-provision-users.sh b/scripts/t3-provision-users.sh index 33edf3fd..27a1beda 100644 --- a/scripts/t3-provision-users.sh +++ b/scripts/t3-provision-users.sh @@ -95,6 +95,26 @@ EOF log "wrote OIDC kubeconfig -> $user:~/.kube/config" } +# Share the admin's Claude subscription with a non-admin: inject CLAUDE_CODE_OAUTH_TOKEN +# (the staged long-lived token) into their t3-serve env — ONLY if they have neither their +# own ~/.claude/.credentials.json (own login) nor an existing token. Never clobbers. The +# agent picks it up when its t3-serve@ instance (re)starts. +install_user_claude_token() { + local user="$1" home envf tok + local token_file="${CLAUDE_TOKEN_FILE:-/etc/t3-serve/claude-oauth-token}" + home="$(getent passwd "$user" | cut -d: -f6)" + [[ -z "$home" ]] && return 0 + [[ -f "$home/.claude/.credentials.json" ]] && return 0 # has own login -> leave it + [[ -r "$token_file" ]] || return 0 + envf="${ENVDIR:-/etc/t3-serve}/$user.env" + grep -q '^CLAUDE_CODE_OAUTH_TOKEN=' "$envf" 2>/dev/null && return 0 # already shared + if [[ "$DRY_RUN" == 1 ]]; then echo "[dry-run] share Claude token -> $envf"; return 0; fi + tok="$(cat "$token_file")" + printf 'CLAUDE_CODE_OAUTH_TOKEN=%s\n' "$tok" >> "$envf" + chmod 600 "$envf" + log "shared Claude token -> $user (t3-serve env; restart needed to take effect)" +} + [[ $EUID -eq 0 ]] || { echo "t3-provision-users: must run as root" >&2; exit 1; } for bin in python3 jq; do command -v "$bin" >/dev/null || { echo "missing $bin" >&2; exit 1; }; done [[ -f "$ROSTER" && -f "$ENGINE" ]] || { echo "roster/engine not under $WORKSTATION_DIR" >&2; exit 1; } @@ -144,9 +164,10 @@ while IFS=$'\t' read -r os_user tier shell groups_csv; do log "add $os_user -> group $g"; run gpasswd -a "$os_user" "$g" >/dev/null done fi - if [[ "$tier" != admin ]]; then # non-admins: locked ~/code clone + OIDC kubeconfig + if [[ "$tier" != admin ]]; then # non-admins: locked clone + kubeconfig + shared Claude token install_locked_clone "$os_user" install_user_kubeconfig "$os_user" + install_user_claude_token "$os_user" fi done < <(jq -r '.accounts[] | [.os_user, .tier, .shell, (.groups|join(","))] | @tsv' "$desired_file") @@ -159,6 +180,10 @@ while IFS=$'\t' read -r os_user port; do id "$os_user" >/dev/null 2>&1 && run systemctl enable --now "t3-serve@$os_user.service" >/dev/null 2>&1 || true done < <(jq -r '.ports | to_entries[] | [.key, .value] | @tsv' "$desired_file") +# 5b) machine-wide (once, not per-user): keep the t3 nightly auto-updater enabled so it +# self-heals hourly — a `disabled` timer silently freezes every instance on an old build. +run systemctl enable --now t3-autoupdate.timer >/dev/null 2>&1 || true + # 6) regenerate /etc/ttyd-user-map + dispatch.json from the desired state (SSoT: # a roster entry removed here DISAPPEARS, which is what the offboarding cut relies on) if [[ "$DRY_RUN" == 1 ]]; then diff --git a/scripts/workstation/setup-devvm.sh b/scripts/workstation/setup-devvm.sh index 9e0284b6..f929b30a 100755 --- a/scripts/workstation/setup-devvm.sh +++ b/scripts/workstation/setup-devvm.sh @@ -77,4 +77,21 @@ if [[ -d "$ADMIN_CODE" ]]; then log "hardened $ADMIN_CODE (o-rx — not world-readable)" fi +# 8) stage the shared Claude subscription OAuth token (long-lived sk-ant-oat01) to a +# root-readable file the provisioner injects into non-admins' t3-serve env, so they +# share the admin's Claude subscription (only those without their own ~/.claude login). +if command -v vault >/dev/null; then + export VAULT_ADDR="${VAULT_ADDR:-https://vault.viktorbarzin.me}" + # setup-devvm runs as root (no ~/.vault-token); borrow the admin's token to read Vault. + if [[ -z "${VAULT_TOKEN:-}" && -r /home/wizard/.vault-token ]]; then + VAULT_TOKEN="$(cat /home/wizard/.vault-token)"; export VAULT_TOKEN + fi + if claude_tok="$(vault kv get -field=claude_oauth_token secret/workstation 2>/dev/null)"; then + install -m 0600 /dev/stdin /etc/t3-serve/claude-oauth-token <<<"$claude_tok" + log "staged /etc/t3-serve/claude-oauth-token (shared Claude subscription)" + else + log "WARN: secret/workstation claude_oauth_token absent -> non-admins won't share Claude auth" + fi +fi + log "OK (idempotent)" From fad10a8707d679dcb372d4a198e00cef3e253afc Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 14:15:03 +0000 Subject: [PATCH 10/24] =?UTF-8?q?workstation:=20fix=20new-user=20.env=20cl?= =?UTF-8?q?obber=20=E2=80=94=20env=5Fset=20preserves=20CLAUDE=5FCODE=5FOAU?= =?UTF-8?q?TH=5FTOKEN?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The port-write used '>' (overwrite), wiping the token injected earlier in the same run for a NEW user (existing users like anca survived only because their .env already had the T3_PORT line). New env_set() does update-or-append per key, preserving others. Verified end-to-end: throwaway t3probe provisioned from scratch -> .env has both T3_PORT + CLAUDE_CODE_OAUTH_TOKEN -> claude -p AUTHOK. So all new non-admins now authenticate automatically. NOT pushed (shared-tree divergence hold). Co-Authored-By: Claude Opus 4.8 --- scripts/t3-provision-users.sh | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/scripts/t3-provision-users.sh b/scripts/t3-provision-users.sh index 27a1beda..8c269bdd 100644 --- a/scripts/t3-provision-users.sh +++ b/scripts/t3-provision-users.sh @@ -95,6 +95,20 @@ EOF log "wrote OIDC kubeconfig -> $user:~/.kube/config" } +# Idempotently set KEY=VALUE in a t3-serve env file, PRESERVING other lines — so writing +# T3_PORT never clobbers an injected CLAUDE_CODE_OAUTH_TOKEN, and vice-versa. Mode 0600. +env_set() { + local file="$1" key="$2" val="$3" + if [[ "$DRY_RUN" == 1 ]]; then echo "[dry-run] set $key -> $file"; return 0; fi + install -d -m 0755 "$(dirname "$file")" + if [[ -f "$file" ]] && grep -q "^${key}=" "$file"; then + grep -qx "${key}=${val}" "$file" || sed -i "s|^${key}=.*|${key}=${val}|" "$file" + else + printf '%s=%s\n' "$key" "$val" >> "$file" + fi + chmod 600 "$file" +} + # Share the admin's Claude subscription with a non-admin: inject CLAUDE_CODE_OAUTH_TOKEN # (the staged long-lived token) into their t3-serve env — ONLY if they have neither their # own ~/.claude/.credentials.json (own login) nor an existing token. Never clobbers. The @@ -110,8 +124,7 @@ install_user_claude_token() { grep -q '^CLAUDE_CODE_OAUTH_TOKEN=' "$envf" 2>/dev/null && return 0 # already shared if [[ "$DRY_RUN" == 1 ]]; then echo "[dry-run] share Claude token -> $envf"; return 0; fi tok="$(cat "$token_file")" - printf 'CLAUDE_CODE_OAUTH_TOKEN=%s\n' "$tok" >> "$envf" - chmod 600 "$envf" + env_set "$envf" CLAUDE_CODE_OAUTH_TOKEN "$tok" log "shared Claude token -> $user (t3-serve env; restart needed to take effect)" } @@ -174,9 +187,7 @@ done < <(jq -r '.accounts[] | [.os_user, .tier, .shell, (.groups|join(","))] | @ # 5) per-user .env (sticky port) + enable t3-serve@ while IFS=$'\t' read -r os_user port; do envf="$ENVDIR/$os_user.env" - if [[ ! -f "$envf" ]] || ! grep -qx "T3_PORT=$port" "$envf"; then - run bash -c "printf 'T3_PORT=%s\n' '$port' > '$envf'" - fi + env_set "$envf" T3_PORT "$port" # update-or-append; preserves CLAUDE_CODE_OAUTH_TOKEN id "$os_user" >/dev/null 2>&1 && run systemctl enable --now "t3-serve@$os_user.service" >/dev/null 2>&1 || true done < <(jq -r '.ports | to_entries[] | [.key, .value] | @tsv' "$desired_file") From 2125651aaac8aaeb68e05e9eb2a566b76f57c730 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 15:51:08 +0000 Subject: [PATCH 11/24] t3-dispatch: re-pair on present-but-invalid t3_session cookie The dispatcher only re-paired on an ABSENT cookie. After the 2026-06-09 auth-schema rollback wiped all server-side sessions, browsers kept dead 30-day t3_session cookies; the dispatcher proxied them straight through and t3 rendered its pair page ("all users must pair again"). Now a present cookie on a top-level document navigation is validated via the instance's /api/auth/session and re-paired on authenticated:false. Gated to document navs (Sec-Fetch-Dest: document, else Accept: text/html) so XHR/asset/WebSocket sub-requests are never answered with a 302; fails open (proxy through) on any validation error. Unit + handler tests added. [ci skip] Co-Authored-By: Claude --- .../2026-06-01-t3-auto-provision-design.md | 6 +- scripts/t3-dispatch/main.go | 67 +++++- scripts/t3-dispatch/main_test.go | 200 ++++++++++++++++++ 3 files changed, 269 insertions(+), 4 deletions(-) create mode 100644 scripts/t3-dispatch/main_test.go diff --git a/docs/plans/2026-06-01-t3-auto-provision-design.md b/docs/plans/2026-06-01-t3-auto-provision-design.md index 1c34e8a8..ca9a7e1f 100644 --- a/docs/plans/2026-06-01-t3-auto-provision-design.md +++ b/docs/plans/2026-06-01-t3-auto-provision-design.md @@ -85,12 +85,14 @@ Replaces the in-cluster nginx `t3-dispatch` (the session-mint needs `sudo` + loc Per request (Authentik forward-auth has injected a trustworthy `X-authentik-username`): 1. Resolve `X-authentik-username` → OS user via `/etc/ttyd-user-map`. No mapping → **403**. -2. **Has a valid t3 session cookie?** → reverse-proxy (incl. WebSocket upgrade) to `127.0.0.1:`. (Steady state — the common path.) -3. **No cookie** (first visit / expired) → auto-pair: +2. **Has a valid t3 session cookie?** → reverse-proxy (incl. WebSocket upgrade) to `127.0.0.1:`. (Steady state — the common path.) Sub-requests (XHR/asset/WebSocket) take the cookie at face value; on a **top-level document navigation** the cookie is verified against the instance's `GET /api/auth/session` so a present-but-dead cookie doesn't slip through. +3. **No cookie, or an invalid cookie on a document navigation** (first visit / expired / server-side session wiped) → auto-pair: - `sudo -u t3 auth pairing create --base-dir /home//.t3 --ttl 5m --json` → one-time token. - exchange it at the instance's `POST /api/auth/bootstrap` → capture the returned `Set-Cookie`. - relay that `Set-Cookie` to the browser + `302 /`. Browser now holds the t3 session cookie → next request is the steady-state path. **Login → straight in.** +> **As-built note (2026-06-09):** the first implementation re-paired only on an *absent* cookie. After an auth-schema rollback wiped every server-side session, browsers still held live-looking-but-dead 30-day `t3_session` cookies, which the dispatcher proxied straight through → t3 rendered its pair page (the "all users must pair again" incident). Fixed by validating a present cookie via `/api/auth/session` and re-pairing on `authenticated:false` — **gated to document navigations** (`isDocumentNav`: trust `Sec-Fetch-Dest: document`, else fall back to `Accept: text/html`) so XHR/asset/WebSocket sub-requests are never answered with a `302`, and **fail-open** (proxy through) on any validation error so no new failure mode is introduced. See `scripts/t3-dispatch/main.go` (`sessionValid`, `isDocumentNav`) + `main_test.go`. + Implementation: a small reverse proxy that supports WebSocket upgrade (Go `httputil.ReverseProxy`, or Python aiohttp) — chosen at plan time. ### 4. Terraform — `stacks/t3code` shrinks diff --git a/scripts/t3-dispatch/main.go b/scripts/t3-dispatch/main.go index cebcc119..401b0edb 100644 --- a/scripts/t3-dispatch/main.go +++ b/scripts/t3-dispatch/main.go @@ -59,6 +59,60 @@ func lookup(ak string) (entry, bool) { return e, ok } +// mintToken mints a one-time pairing token for osUser via the scoped sudoers +// entry (the dispatch service can invoke nothing else). Indirected through a var +// so tests can stub the privileged exec. +var mintToken = func(osUser string) ([]byte, error) { + return exec.Command("sudo", "-n", "/usr/local/bin/t3-mint", osUser).Output() +} + +var sessionClient = &http.Client{Timeout: 5 * time.Second} + +// sessionValid asks the user's instance whether the presented t3_session cookie +// is still valid. Server-side sessions can be wiped/expired independently of the +// 30-day cookie (e.g. an auth-schema rollback drops every session row), leaving +// the browser with a live-looking but dead cookie. Fails OPEN: any error/non-200/ +// parse failure returns true so the request still proxies — a re-pair is forced +// only on a definitive authenticated:false. +func sessionValid(e entry, c *http.Cookie) bool { + req, err := http.NewRequest(http.MethodGet, + fmt.Sprintf("http://127.0.0.1:%d/api/auth/session", e.Port), nil) + if err != nil { + return true + } + req.AddCookie(c) + resp, err := sessionClient.Do(req) + if err != nil { + return true + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return true + } + var s struct { + Authenticated bool `json:"authenticated"` + } + if json.NewDecoder(resp.Body).Decode(&s) != nil { + return true + } + return s.Authenticated +} + +// isDocumentNav reports whether r is a top-level browser document navigation, as +// opposed to an XHR/fetch/asset/WebSocket sub-request. Only such requests are +// safe to answer with a re-pair 302 — redirecting a sub-resource would corrupt +// the SPA's fetch/WebSocket contract. Trust Sec-Fetch-Dest when present (all +// modern browsers send it); fall back to the Accept header otherwise. +func isDocumentNav(r *http.Request) bool { + if r.Method != http.MethodGet { + return false + } + if dest := r.Header.Get("Sec-Fetch-Dest"); dest != "" { + return dest == "document" + } + return strings.Contains(r.Header.Get("Accept"), "text/html") +} + // autoPair mints a one-time pairing token for the user's instance (as that OS // user, via the scoped sudoers entry) and exchanges it at the instance's // /api/auth/bootstrap, relaying the returned t3_session Set-Cookie to the browser. @@ -66,7 +120,7 @@ func autoPair(e entry, w http.ResponseWriter, r *http.Request) { // t3-mint (root, via scoped sudoers) validates the OS user is in // /etc/ttyd-user-map, then mints as that user. The dispatch service itself // runs unprivileged and can invoke nothing else. - out, err := exec.Command("sudo", "-n", "/usr/local/bin/t3-mint", e.OsUser).Output() + out, err := mintToken(e.OsUser) if err != nil { log.Printf("mint for %s failed: %v", e.OsUser, err) http.Error(w, "pairing mint failed", http.StatusInternalServerError) @@ -111,7 +165,16 @@ func handler(w http.ResponseWriter, r *http.Request) { http.Error(w, "no t3 instance provisioned for this user", http.StatusForbidden) return } - if _, err := r.Cookie(cookieName); err != nil { + c, err := r.Cookie(cookieName) + if err != nil { + autoPair(e, w, r) + return + } + // A present cookie can still be server-side-invalid (sessions wiped/expired + // while the 30-day cookie lingers). On a top-level navigation, verify it and + // re-pair if dead — otherwise the instance just renders its pair page. Gated + // to document navs so we never 302 an XHR/asset/WebSocket sub-request. + if isDocumentNav(r) && !sessionValid(e, c) { autoPair(e, w, r) return } diff --git a/scripts/t3-dispatch/main_test.go b/scripts/t3-dispatch/main_test.go new file mode 100644 index 00000000..81ca26a9 --- /dev/null +++ b/scripts/t3-dispatch/main_test.go @@ -0,0 +1,200 @@ +package main + +import ( + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "testing" +) + +func portOf(t *testing.T, ts *httptest.Server) int { + t.Helper() + u, err := url.Parse(ts.URL) + if err != nil { + t.Fatalf("parse %s: %v", ts.URL, err) + } + p, err := strconv.Atoi(u.Port()) + if err != nil { + t.Fatalf("port %s: %v", u.Port(), err) + } + return p +} + +func TestIsDocumentNav(t *testing.T) { + cases := []struct { + name string + method string + headers map[string]string + want bool + }{ + {"GET sec-fetch-dest document", "GET", map[string]string{"Sec-Fetch-Dest": "document"}, true}, + {"GET accept html (no sec-fetch)", "GET", map[string]string{"Accept": "text/html,application/xhtml+xml"}, true}, + {"GET xhr empty dest beats accept", "GET", map[string]string{"Sec-Fetch-Dest": "empty", "Accept": "text/html"}, false}, + {"GET json", "GET", map[string]string{"Accept": "application/json"}, false}, + {"POST html", "POST", map[string]string{"Accept": "text/html"}, false}, + {"GET no headers", "GET", map[string]string{}, false}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + r, _ := http.NewRequest(c.method, "/", nil) + for k, v := range c.headers { + r.Header.Set(k, v) + } + if got := isDocumentNav(r); got != c.want { + t.Errorf("isDocumentNav = %v, want %v", got, c.want) + } + }) + } +} + +func sessionServer(status int, body string) *httptest.Server { + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/api/auth/session" { + http.NotFound(w, r) + return + } + w.WriteHeader(status) + _, _ = w.Write([]byte(body)) + })) +} + +func TestSessionValid(t *testing.T) { + ck := &http.Cookie{Name: cookieName, Value: "x"} + + t.Run("authenticated true -> valid", func(t *testing.T) { + ts := sessionServer(200, `{"authenticated":true}`) + defer ts.Close() + if !sessionValid(entry{Port: portOf(t, ts)}, ck) { + t.Fatal("want valid (true) for authenticated:true") + } + }) + t.Run("authenticated false -> invalid", func(t *testing.T) { + ts := sessionServer(200, `{"authenticated":false}`) + defer ts.Close() + if sessionValid(entry{Port: portOf(t, ts)}, ck) { + t.Fatal("want invalid (false) for authenticated:false") + } + }) + t.Run("500 -> fail-open valid", func(t *testing.T) { + ts := sessionServer(500, `boom`) + defer ts.Close() + if !sessionValid(entry{Port: portOf(t, ts)}, ck) { + t.Fatal("want fail-open true on 500") + } + }) + t.Run("malformed json -> fail-open valid", func(t *testing.T) { + ts := sessionServer(200, `not json`) + defer ts.Close() + if !sessionValid(entry{Port: portOf(t, ts)}, ck) { + t.Fatal("want fail-open true on unparseable body") + } + }) + t.Run("unreachable -> fail-open valid", func(t *testing.T) { + ts := sessionServer(200, `{"authenticated":false}`) + p := portOf(t, ts) + ts.Close() // nothing listening now + if !sessionValid(entry{Port: p}, ck) { + t.Fatal("want fail-open true on connection refused") + } + }) +} + +// fakeInstance serves the three endpoints the dispatcher touches: the session +// check, the bootstrap exchange, and a catch-all standing in for the proxied app. +func fakeInstance(authenticated bool, bootstrapCalled *bool) *httptest.Server { + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/api/auth/session": + if authenticated { + _, _ = w.Write([]byte(`{"authenticated":true}`)) + } else { + _, _ = w.Write([]byte(`{"authenticated":false}`)) + } + case "/api/auth/bootstrap": + if bootstrapCalled != nil { + *bootstrapCalled = true + } + http.SetCookie(w, &http.Cookie{Name: cookieName, Value: "fresh", Path: "/"}) + _, _ = w.Write([]byte(`{"authenticated":true}`)) + default: + _, _ = w.Write([]byte("APP")) + } + })) +} + +func setTable(port int) { + mu.Lock() + table = map[string]entry{"vbarzin": {OsUser: "wizard", Port: port}} + mu.Unlock() +} + +func TestHandlerRepairsOnInvalidCookieDocNav(t *testing.T) { + called := false + ts := fakeInstance(false, &called) + defer ts.Close() + setTable(portOf(t, ts)) + + orig := mintToken + mintToken = func(string) ([]byte, error) { return []byte(`{"credential":"tok"}`), nil } + defer func() { mintToken = orig }() + + r := httptest.NewRequest("GET", "/", nil) + r.Header.Set("X-authentik-username", "vbarzin@gmail.com") + r.Header.Set("Sec-Fetch-Dest", "document") + r.AddCookie(&http.Cookie{Name: cookieName, Value: "stale"}) + w := httptest.NewRecorder() + + handler(w, r) + + if w.Code != http.StatusFound { + t.Fatalf("stale cookie on doc-nav should re-pair (302), got %d body=%q", w.Code, w.Body.String()) + } + if !called { + t.Fatal("expected bootstrap to be called during re-pair") + } + cookies := w.Result().Cookies() + if len(cookies) == 0 || cookies[0].Value != "fresh" { + t.Fatalf("expected fresh t3_session relayed, got %+v", cookies) + } +} + +func TestHandlerProxiesOnValidCookie(t *testing.T) { + ts := fakeInstance(true, nil) + defer ts.Close() + setTable(portOf(t, ts)) + + r := httptest.NewRequest("GET", "/", nil) + r.Header.Set("X-authentik-username", "vbarzin@gmail.com") + r.Header.Set("Sec-Fetch-Dest", "document") + r.AddCookie(&http.Cookie{Name: cookieName, Value: "good"}) + w := httptest.NewRecorder() + + handler(w, r) + + if w.Code != http.StatusOK || w.Body.String() != "APP" { + t.Fatalf("valid cookie should proxy (200 APP), got %d %q", w.Code, w.Body.String()) + } +} + +func TestHandlerProxiesXHREvenIfCookieInvalid(t *testing.T) { + called := false + ts := fakeInstance(false, &called) // session would say invalid, but XHR must NOT be re-paired + defer ts.Close() + setTable(portOf(t, ts)) + + r := httptest.NewRequest("GET", "/api/threads", nil) + r.Header.Set("X-authentik-username", "vbarzin@gmail.com") + r.Header.Set("Sec-Fetch-Dest", "empty") // XHR/fetch, not a document nav + r.AddCookie(&http.Cookie{Name: cookieName, Value: "stale"}) + w := httptest.NewRecorder() + + handler(w, r) + + if called { + t.Fatal("must NOT re-pair (302) a non-document sub-request — would corrupt the SPA fetch contract") + } + if w.Code != http.StatusOK || w.Body.String() != "APP" { + t.Fatalf("XHR should proxy through, got %d %q", w.Code, w.Body.String()) + } +} From 5ea238c7077a58e0d6905a2d069cee40c52a8253 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 16:08:44 +0000 Subject: [PATCH 12/24] t3: pin t3@0.0.24 + stop nightly auto-update (auth-outage fix) [ci skip] The t3-autoupdate timer (re-enabled by the provisioner's step 5b with `--now`, which fires the missed daily job immediately on a Persistent timer) pulled t3@nightly 0.0.25 mid-day. That build ran forward schema migrations on every ~/.t3 state.sqlite (auth_pairing_links/auth_sessions role->scopes, +proof_key_thumbprint) AND changed the bootstrap API, breaking t3-mint/pairing for ALL devvm users (pair prompt, no session). - t3-autoupdate.sh: now a pinned-version ENFORCER (T3_PIN=0.0.24), not a nightly tracker -- re-asserts the pin (a no-op when correct). - t3-provision-users.sh step 5b: drop `--now` (it triggered the immediate missed-job run that pulled the bad build). - setup-devvm.sh: install pinned t3@0.0.24 at machine setup. - unit Descriptions + service-catalog reflect the pin. - post-mortem: 2026-06-09-t3-nightly-autoupdate-auth-outage.md. Host already reconciled out-of-band: rolled back to 0.0.24, re-enabled the (now-pinned) enforcer, reset the 2 new users' disposable DBs, surgically reverted wizard's auth tables to level-30 (96 threads + live session preserved). All users verified 302 + t3_session. Co-Authored-By: Claude Opus 4.8 --- .claude/reference/service-catalog.md | 2 +- ...06-09-t3-nightly-autoupdate-auth-outage.md | 138 ++++++++++++++++++ scripts/t3-autoupdate.service | 2 +- scripts/t3-autoupdate.sh | 24 ++- scripts/t3-autoupdate.timer | 2 +- scripts/t3-provision-users.sh | 9 +- scripts/workstation/setup-devvm.sh | 10 ++ 7 files changed, 174 insertions(+), 13 deletions(-) create mode 100644 docs/post-mortems/2026-06-09-t3-nightly-autoupdate-auth-outage.md diff --git a/.claude/reference/service-catalog.md b/.claude/reference/service-catalog.md index 0ba680cb..c8022fa1 100644 --- a/.claude/reference/service-catalog.md +++ b/.claude/reference/service-catalog.md @@ -32,7 +32,7 @@ |---------|-------------|-------| | k8s-dashboard | Kubernetes dashboard at `k8s.viktorbarzin.me`. **Forward-auth + auto-injected SA token** (apiserver OIDC blocked, see design §12). nginx token-injector (`dashboard_injector.tf`) maps `X-authentik-username` → the user's `dashboard-` SA token (ns admin + read-only on namespace-list/nodes only via `dashboard-nav-readonly` — no cross-tenant reads, `rbac/.../dashboard-sa.tf`; admins → cluster-admin SA) and sets `Authorization: Bearer` → no token-paste, dashboard auto-authenticates per user. Forward-auth admits `kubernetes-*` groups for this host (`stacks/authentik/admin-services-restriction.tf`). oauth2-proxy + `k8s-dashboard` OIDC app built but idle. | k8s-dashboard | | reverse-proxy | Generic reverse proxy | reverse-proxy | -| t3code | Multi-user coding-agent GUI at t3.viktorbarzin.me. `auth=required` (Authentik) → DevVM `t3-dispatch` service (`10.0.10.10:3780`, unprivileged user) maps `X-authentik-username` → that user's own `t3-serve@` instance (file perms enforced by uid; wizard→:3773, emo→:3774; unmapped→403) and **auto-injects the t3 session on first visit** (mints via the root `t3-mint` wrapper, scoped sudoers → `/api/auth/bootstrap` `t3_session` cookie). **Source of truth = `infra/scripts/workstation/roster.yaml`** (os_user → authentik_user/k8s_user/tier/namespaces); `roster_engine.py` (pytest-covered) derives desired state and `t3-provision-users` (hourly systemd timer) applies it — constrained accounts, additive per-tier groups, `t3-serve@` instances, and **regenerating** `/etc/ttyd-user-map` + `dispatch.json` (those two are now GENERATED — do not hand-edit). New non-admins inherit wizard's Claude config (machine-wide managed `claudeMd` in `/etc/claude-code/managed-settings.json` + per-user `~/.claude/{skills,rules}` symlinks seeded by `/etc/skel`) and get a **writable git-crypt-LOCKED** infra clone at `~/code` (code plaintext, secret files ciphertext). Tiers: admin / power-user (cluster-wide read-only) / namespace-owner. **Add a user:** one entry in `roster.yaml` → reconcile. Per-user OIDC kubeconfig, the `oidc-power-user-readonly` ClusterRole, and the Authentik `T3 Users` edge gate are applied (the gate is live — only `T3 Users` members reach t3); the emo cutover to his own locked clone is the remaining gated step. DevVM artifacts versioned in `infra/scripts/` (`t3-serve@.service`, `t3-provision-users` + `workstation/{roster.yaml,roster_engine.py,setup-devvm.sh,managed-settings.json,skel/}`, `t3-dispatch/`, `t3-mint`, `sudoers-t3-autopair`, `t3-autoupdate.*`); TF (`stacks/t3code`) owns only the ingress + Endpoints→:3780. **t3 binary tracks `nightly`** via `t3-autoupdate` (daily systemd timer; health-check + auto-rollback on a bad build; restarts only idle instances) — so new models (e.g. Opus 4.8) land as t3 ships them. Native app/app.t3.codes unsupported (cross-origin) — deferred until published. Design: `docs/plans/2026-06-01-t3-auto-provision-*`. | t3code | +| t3code | Multi-user coding-agent GUI at t3.viktorbarzin.me. `auth=required` (Authentik) → DevVM `t3-dispatch` service (`10.0.10.10:3780`, unprivileged user) maps `X-authentik-username` → that user's own `t3-serve@` instance (file perms enforced by uid; wizard→:3773, emo→:3774; unmapped→403) and **auto-injects the t3 session on first visit** (mints via the root `t3-mint` wrapper, scoped sudoers → `/api/auth/bootstrap` `t3_session` cookie). **Source of truth = `infra/scripts/workstation/roster.yaml`** (os_user → authentik_user/k8s_user/tier/namespaces); `roster_engine.py` (pytest-covered) derives desired state and `t3-provision-users` (hourly systemd timer) applies it — constrained accounts, additive per-tier groups, `t3-serve@` instances, and **regenerating** `/etc/ttyd-user-map` + `dispatch.json` (those two are now GENERATED — do not hand-edit). New non-admins inherit wizard's Claude config (machine-wide managed `claudeMd` in `/etc/claude-code/managed-settings.json` + per-user `~/.claude/{skills,rules}` symlinks seeded by `/etc/skel`) and get a **writable git-crypt-LOCKED** infra clone at `~/code` (code plaintext, secret files ciphertext). Tiers: admin / power-user (cluster-wide read-only) / namespace-owner. **Add a user:** one entry in `roster.yaml` → reconcile. Per-user OIDC kubeconfig, the `oidc-power-user-readonly` ClusterRole, and the Authentik `T3 Users` edge gate are applied (the gate is live — only `T3 Users` members reach t3); the emo cutover to his own locked clone is the remaining gated step. DevVM artifacts versioned in `infra/scripts/` (`t3-serve@.service`, `t3-provision-users` + `workstation/{roster.yaml,roster_engine.py,setup-devvm.sh,managed-settings.json,skel/}`, `t3-dispatch/`, `t3-mint`, `sudoers-t3-autopair`, `t3-autoupdate.*`); TF (`stacks/t3code`) owns only the ingress + Endpoints→:3780. **t3 binary is PINNED** (`T3_PIN`, currently `0.0.24`) — `t3-autoupdate` is a daily *enforcer* that re-asserts the pin (a no-op when correct; restarts only idle instances), NOT a nightly tracker. It used to track `nightly`, but on 2026-06-09 a nightly bump migrated every `~/.t3/state.sqlite` forward (`role`→`scopes`) and changed the bootstrap API, breaking pairing for ALL users (post-mortem `2026-06-09-t3-nightly-autoupdate-auth-outage.md`). t3 is pre-1.0, so moving the pin requires first verifying `t3-dispatch`'s bootstrap flow against the new build (expect 302 + `t3_session`). Pin set in `t3-autoupdate.sh` + `setup-devvm.sh` (keep in sync). Native app/app.t3.codes unsupported (cross-origin) — deferred until published. Design: `docs/plans/2026-06-01-t3-auto-provision-*`. | t3code | ## Active Use | Service | Description | Stack | diff --git a/docs/post-mortems/2026-06-09-t3-nightly-autoupdate-auth-outage.md b/docs/post-mortems/2026-06-09-t3-nightly-autoupdate-auth-outage.md new file mode 100644 index 00000000..e8f0d2d5 --- /dev/null +++ b/docs/post-mortems/2026-06-09-t3-nightly-autoupdate-auth-outage.md @@ -0,0 +1,138 @@ +# Post-Mortem: t3 Nightly Auto-Update (0.0.25) Migrated `state.sqlite` Forward → mint/pairing Broke for All Devvm Users + +## Summary + +The devvm t3 auto-updater (`t3-autoupdate.timer`) pulled the `t3@nightly` +build `0.0.25-nightly.20260608.497`. That build ran two forward schema +migrations on every per-user `~/.t3/userdata/state.sqlite` (renaming +`role`→`scopes` in `auth_pairing_links` + `auth_sessions`, adding +`proof_key_thumbprint`) **and** changed the bootstrap API. The result was a +binary-vs-schema mismatch that broke `t3-mint` (pairing-credential issuance) +for **all** users — every fresh login landed on the t3 pairing prompt instead +of an authenticated session. + +## Impact + +- **Who:** every devvm t3 user — `wizard` (Viktor), `emo`, `ancamilea`. +- **What:** `t3 auth pairing create` failed (`AuthControlPlaneError: + Failed to create pairing link` → `PersistenceSqlError` on + `auth_pairing_links`), so `t3-dispatch` auto-pair returned 500/502 and the + browser showed the pairing prompt. Existing *already-authenticated* sessions + kept working (validated against `auth_sessions`, not the pairing path). +- **When:** ~13:56 (bad nightly installed) → ~15:16 (all users verified 302). +- **Trigger of the report:** Anca could not log in ("gets the pair prompt, + session broken"). + +## Timeline (devvm clock) + +- **13:56** — `t3-provision-users` step 5b ran `systemctl enable --now + t3-autoupdate.timer`. The timer is `OnCalendar=04:00 … Persistent=true`; + `--now` + a missed 04:00 schedule fired the daily job **immediately**. +- **13:56** — updater installed `t3@nightly` = `0.0.25-nightly.20260608.497` + (was `0.0.24`). The `GET / → 200` health-check **passed** (it never + exercises mint/bootstrap), so no auto-rollback. It restarted *idle* serves + (emo) onto 0.0.25 and deferred *active* ones (wizard, ancamilea). +- **~14:38** — `t3-mint` (now global 0.0.25) ran migrations 31 + (`AuthAuthorizationScopes`) + 32 (`AuthPairingProofKeyThumbprint`) against + each `state.sqlite` it touched → schemas moved to "level 32". +- **~14:40** — first recovery action rolled the **binary** back to `0.0.24`. + This did **not** help: the DBs were still at level 32, so the level-30 + binary's INSERT hit `no column named role` / `NOT NULL constraint failed: + scopes`. (Downgrading a binary after a forward migration is not a rollback.) +- **~15:01–15:16** — diagnosed the binary-vs-schema mismatch, confirmed + `0.0.25` *stable* is **also** dispatch-incompatible (auto-pair → 502, the + bootstrap API moved), pinned to `0.0.24`, reset the two new users' disposable + DBs, surgically reverted wizard's two auth tables to level 30. All three + users verified 302 + `Set-Cookie: t3_session`. + +## Root Cause + +Three compounding factors: + +1. **Auto-tracking a pre-1.0 tool's nightly.** `t3-autoupdate.sh` ran + `npm i -g t3@nightly`. t3 ships breaking schema-migration and bootstrap-API + changes between builds; our `t3-dispatch` (Go) speaks a fixed bootstrap + contract (`POST /api/auth/bootstrap {"credential":…}` → `Set-Cookie`). +2. **`enable --now` on a `Persistent=true` timer.** The provisioner's + re-assertion of the timer didn't just *arm* the schedule — it fired the + missed daily job on the spot, mid-afternoon, with users active. +3. **A health-check that proves nothing about auth.** The smoke test only + probes `GET / → 200`. The 0.0.25 server answers 200 while its pairing/mint + path is incompatible, so the "auto-rollback on bad build" never triggered. + +Forward migrations + a binary downgrade = a DB the old binary can't write. +`state.sqlite` also holds the precious projection tables (session history), so +a blanket "delete and re-pair" was only safe for the brand-new users. + +## Detection + +User report (Anca on the pairing prompt). No alert fired — the auto-updater's +own health-check is the only automated gate and it passed. **Gap:** nothing +monitors the end-to-end pairing flow. + +## Fixes & Mitigations + +### 1. Pin t3, stop tracking nightly (DONE) + +`infra/scripts/t3-autoupdate.sh` is now a **pinned-version enforcer**: +`T3_PIN="${T3_PIN:-0.0.24}"`, `npm i -g "t3@$T3_PIN"`. It re-asserts the pin +(a no-op when already correct) instead of chasing nightly. Unit `Description`s +updated. To move the pin: bump `T3_PIN` **and first** verify `t3-dispatch`'s +bootstrap flow against the new build (`curl` the dispatch → expect 302 + +`Set-Cookie: t3_session`). + +### 2. Drop `--now` from the provisioner (DONE) + +`infra/scripts/t3-provision-users.sh` step 5b now runs `systemctl enable +t3-autoupdate.timer` (no `--now`) — it arms the 04:00 schedule without firing a +missed job immediately. + +### 3. Pinned install at machine setup (DONE) + +`infra/scripts/workstation/setup-devvm.sh` installs `t3@$T3_PIN` directly, so a +fresh box has the pinned t3 immediately rather than depending on the enforcer's +first run. + +### 4. Recovery actions taken on the host (DONE) + +- Global `t3` rolled to `0.0.24`; enforcer redeployed + timer re-enabled + (verified the enforcer is a no-op at the pin). +- New users (`emo` 0 threads, `ancamilea` 1 trivial thread): `state.sqlite` + parked aside; serve restarted → fresh level-30 DB. +- `wizard` (96 threads, and the serve hosting the recovery session — cannot be + restarted): the two auth tables were atomically rebuilt to the level-30 + schema (copied from a fresh DB) and migration records 31/32 removed. + `auth_sessions` had 0 rows and the 0.0.24 serve never reads `scopes`, so the + live session and all projection history were untouched. Backup: + `/home/wizard/.t3/userdata/auth-backup-*.sql`. + +### 5. End-to-end pairing health-check (DEFERRED) + +The smoke test should exercise mint→bootstrap→cookie, not just `GET /`. Not +done here (the pin makes it moot for the known-good build); needed before the +enforcer is ever pointed at a new version. A blackbox probe on the dispatch +auto-pair (expect 302 + `t3_session`) would have alerted within minutes. + +## Lessons + +- **Don't auto-track a pre-1.0 tool's nightly.** Pin to a known-good, + contract-verified build; upgrades are a deliberate, tested act. +- **`enable --now` on a `Persistent=true` timer fires the missed job now.** + Use plain `enable` to arm a schedule without a surprise immediate run. +- **A liveness probe (`GET /`) is not a readiness/correctness probe.** If a + feature (auth/pairing) can break while `/` stays 200, the health-check must + exercise that feature or it gives false confidence. +- **A binary downgrade is not a schema rollback.** Once a forward migration + runs, the data is migrated; the old binary now mismatches its own DB. +- **Separate disposable state from precious state before resetting.** t3's + `state.sqlite` mixes ephemeral auth (`auth_pairing_links`, `auth_sessions`) + with precious history (`projection_*`); surgical table-level repair + preserved 8k+ messages that a blanket reset would have destroyed. + +## References + +- `infra/scripts/t3-autoupdate.sh` (pinned enforcer), `.service`, `.timer` +- `infra/scripts/t3-provision-users.sh` step 5b +- `infra/scripts/workstation/setup-devvm.sh` step 2b +- `infra/.claude/reference/service-catalog.md` (t3 serving layer) +- Backup of wizard's pre-repair auth tables: `/home/wizard/.t3/userdata/auth-backup-*.sql` diff --git a/scripts/t3-autoupdate.service b/scripts/t3-autoupdate.service index d3306da7..7b043f13 100644 --- a/scripts/t3-autoupdate.service +++ b/scripts/t3-autoupdate.service @@ -1,5 +1,5 @@ [Unit] -Description=Track latest t3 nightly (health-checked, idle-only restart) +Description=Enforce pinned t3 version (health-checked, idle-only restart) After=network-online.target Wants=network-online.target diff --git a/scripts/t3-autoupdate.sh b/scripts/t3-autoupdate.sh index 962f3fc4..836605f0 100644 --- a/scripts/t3-autoupdate.sh +++ b/scripts/t3-autoupdate.sh @@ -1,20 +1,30 @@ #!/usr/bin/env bash -# Track the latest t3 nightly — with a health-check + auto-rollback (lesson from -# the Keel auto-update incidents: never blindly trust a new build) and idle-only -# restarts (never kill an in-flight coding session). Runs as root via the unit. +# Enforce the PINNED t3 version ($T3_PIN) across the box — NOT "latest/nightly". +# t3 is pre-1.0 and ships breaking schema-migration + bootstrap-API changes between +# builds that our t3-dispatch can't follow blind. 2026-06-09: a nightly auto-update +# (0.0.25) migrated every ~/.t3 state.sqlite forward (auth_pairing_links/auth_sessions +# role->scopes) AND changed the bootstrap API, breaking mint/pairing for ALL users. +# So we PIN; this unit just re-asserts the pin (a no-op when already correct) with a +# health-check + auto-rollback and idle-only restarts (never kill an in-flight session). +# To move the pin: bump T3_PIN AND first verify t3-dispatch's bootstrap flow against the +# new build (curl the dispatch -> expect 302 + Set-Cookie t3_session). See post-mortem +# 2026-06-09-t3-nightly-autoupdate-auth-outage.md. +# CAVEAT: the health-check below only probes GET / (200) — it does NOT exercise the +# mint/bootstrap/pairing path, so it will NOT catch an auth regression on its own. set -uo pipefail +T3_PIN="${T3_PIN:-0.0.24}" # known-good, t3-dispatch-compatible (2026-06-09 post-mortem) LOG() { logger -t t3-autoupdate "$*"; echo "t3-autoupdate: $*"; } ver() { t3 --version 2>/dev/null | awk '{print $NF}' | sed 's/^v//'; } -before=$(ver); LOG "current: ${before:-unknown}" -npm i -g t3@nightly >/dev/null 2>&1 || { LOG "npm install failed; staying on ${before:-current}"; exit 0; } +before=$(ver); LOG "current: ${before:-unknown}; pin: $T3_PIN" +npm i -g "t3@$T3_PIN" >/dev/null 2>&1 || { LOG "npm install failed; staying on ${before:-current}"; exit 0; } after=$(ver) if [[ -z "$after" || "$after" == "$before" ]]; then - LOG "already latest (${before:-?}); nothing to do"; exit 0 + LOG "already at pin $T3_PIN (${before:-?}); nothing to do"; exit 0 fi -LOG "installed $after (was $before); health-checking…" +LOG "re-pinned to $after (was $before); health-checking…" # Health-check the NEW binary on a throwaway port/base-dir before trusting it. SMOKE_PORT=3799; SMOKE_DIR=$(mktemp -d) diff --git a/scripts/t3-autoupdate.timer b/scripts/t3-autoupdate.timer index a59135f7..ccdbd4c6 100644 --- a/scripts/t3-autoupdate.timer +++ b/scripts/t3-autoupdate.timer @@ -1,5 +1,5 @@ [Unit] -Description=Daily t3 nightly auto-update +Description=Daily t3 pinned-version enforcer (re-asserts T3_PIN; no-op when correct) [Timer] OnCalendar=*-*-* 04:00:00 diff --git a/scripts/t3-provision-users.sh b/scripts/t3-provision-users.sh index 8c269bdd..37689153 100644 --- a/scripts/t3-provision-users.sh +++ b/scripts/t3-provision-users.sh @@ -191,9 +191,12 @@ while IFS=$'\t' read -r os_user port; do id "$os_user" >/dev/null 2>&1 && run systemctl enable --now "t3-serve@$os_user.service" >/dev/null 2>&1 || true done < <(jq -r '.ports | to_entries[] | [.key, .value] | @tsv' "$desired_file") -# 5b) machine-wide (once, not per-user): keep the t3 nightly auto-updater enabled so it -# self-heals hourly — a `disabled` timer silently freezes every instance on an old build. -run systemctl enable --now t3-autoupdate.timer >/dev/null 2>&1 || true +# 5b) machine-wide (once, not per-user): keep the t3 pinned-version ENFORCER enabled (it +# re-asserts T3_PIN daily; a no-op when already correct). NOT --now: with Persistent=true +# a `--now` enable fires the missed daily job IMMEDIATELY, which on 2026-06-09 pulled a +# breaking nightly mid-day and took out auth for everyone. `enable` (no --now) just arms +# the 04:00 schedule; fresh boxes get t3 from setup-devvm.sh's pinned install, not here. +run systemctl enable t3-autoupdate.timer >/dev/null 2>&1 || true # 6) regenerate /etc/ttyd-user-map + dispatch.json from the desired state (SSoT: # a roster entry removed here DISAPPEARS, which is what the offboarding cut relies on) diff --git a/scripts/workstation/setup-devvm.sh b/scripts/workstation/setup-devvm.sh index f929b30a..faf7b7bc 100755 --- a/scripts/workstation/setup-devvm.sh +++ b/scripts/workstation/setup-devvm.sh @@ -33,6 +33,16 @@ if [[ $need_node -eq 1 ]]; then fi command -v claude >/dev/null || { log "npm: installing @anthropic-ai/claude-code"; npm install -g @anthropic-ai/claude-code >/dev/null; } +# 2b) t3 (the per-user coding surface) — PINNED, never nightly/latest. t3 is pre-1.0 and +# ships breaking auth-schema + bootstrap-API changes our t3-dispatch can't follow blind +# (2026-06-09 outage: a nightly auto-update broke pairing for ALL users). The daily +# t3-autoupdate ENFORCER re-asserts this same pin; install it here so a fresh box has t3 +# immediately. Keep T3_PIN in sync with t3-autoupdate.sh. +T3_PIN="${T3_PIN:-0.0.24}" +if [[ "$(t3 --version 2>/dev/null | awk '{print $NF}' | sed 's/^v//')" != "$T3_PIN" ]]; then + log "npm: installing pinned t3@$T3_PIN"; npm install -g "t3@$T3_PIN" >/dev/null +fi + # 3) kubelogin (kubectl oidc-login) system-wide — NOT the apt 'kubelogin' (= Azure tool) if [[ ! -x /usr/local/bin/kubelogin ]]; then log "kubelogin: installing int128/kubelogin" From bccaa08d8e1ce25b984026b53236d6de34b6df6d Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 20:00:11 +0000 Subject: [PATCH 13/24] =?UTF-8?q?t3:=20prepare=20to=20adopt=200.0.25=20?= =?UTF-8?q?=E2=80=94=20version-agnostic=20dispatch=20+=20real=20pairing=20?= =?UTF-8?q?health-check=20+=20state=20backup=20[ci=20skip]?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Investigated the 0.0.25 break: it is ONLY an endpoint rename (/api/auth/bootstrap -> /api/auth/browser-session). The rest of the pairing contract (credential payload, t3_session cookie, /api/auth/session) is byte-identical, verified in isolated 0.0.24-vs-0.0.25 sandbox serves. So a future pin bump is now safe + reversible (pin STAYS 0.0.24 — this is prep): - t3-dispatch: autoPair tries /api/auth/browser-session, falls back to /api/auth/bootstrap on 404 — one binary pairs across both versions and any rolling-restart skew. TDD via TestAutoPairAcrossVersions (red on 0.0.25 before, green after). Built, deployed, verified live on 0.0.24 (all three users still 302 + t3_session via the fallback). - t3-autoupdate.sh: health-check now exercises the REAL mint->credential->cookie handshake (was GET / -> 200, which passed the pairing-broken nightly). A bad build now auto-rolls-back. Validated against both versions. - t3-backup-state.{sh,service,timer}: daily online VACUUM INTO of each ~/.t3 state.sqlite (was the only copy, unbacked) -> the one-way forward schema migration becomes a restore, not sqlite surgery. timeout-guarded. - runbooks/t3-version-bump.md: the reversible cutover checklist. - post-mortem #5 (health-check) DONE + #6 added; service-catalog updated. Co-Authored-By: Claude Opus 4.8 --- .claude/reference/service-catalog.md | 2 +- ...06-09-t3-nightly-autoupdate-auth-outage.md | 27 ++++- docs/runbooks/t3-version-bump.md | 105 ++++++++++++++++++ scripts/t3-autoupdate.sh | 33 ++++-- scripts/t3-backup-state.service | 6 + scripts/t3-backup-state.sh | 43 +++++++ scripts/t3-backup-state.timer | 10 ++ scripts/t3-dispatch/main.go | 44 +++++++- scripts/t3-dispatch/main_test.go | 60 ++++++++++ 9 files changed, 311 insertions(+), 19 deletions(-) create mode 100644 docs/runbooks/t3-version-bump.md create mode 100644 scripts/t3-backup-state.service create mode 100644 scripts/t3-backup-state.sh create mode 100644 scripts/t3-backup-state.timer diff --git a/.claude/reference/service-catalog.md b/.claude/reference/service-catalog.md index c8022fa1..f33430be 100644 --- a/.claude/reference/service-catalog.md +++ b/.claude/reference/service-catalog.md @@ -32,7 +32,7 @@ |---------|-------------|-------| | k8s-dashboard | Kubernetes dashboard at `k8s.viktorbarzin.me`. **Forward-auth + auto-injected SA token** (apiserver OIDC blocked, see design §12). nginx token-injector (`dashboard_injector.tf`) maps `X-authentik-username` → the user's `dashboard-` SA token (ns admin + read-only on namespace-list/nodes only via `dashboard-nav-readonly` — no cross-tenant reads, `rbac/.../dashboard-sa.tf`; admins → cluster-admin SA) and sets `Authorization: Bearer` → no token-paste, dashboard auto-authenticates per user. Forward-auth admits `kubernetes-*` groups for this host (`stacks/authentik/admin-services-restriction.tf`). oauth2-proxy + `k8s-dashboard` OIDC app built but idle. | k8s-dashboard | | reverse-proxy | Generic reverse proxy | reverse-proxy | -| t3code | Multi-user coding-agent GUI at t3.viktorbarzin.me. `auth=required` (Authentik) → DevVM `t3-dispatch` service (`10.0.10.10:3780`, unprivileged user) maps `X-authentik-username` → that user's own `t3-serve@` instance (file perms enforced by uid; wizard→:3773, emo→:3774; unmapped→403) and **auto-injects the t3 session on first visit** (mints via the root `t3-mint` wrapper, scoped sudoers → `/api/auth/bootstrap` `t3_session` cookie). **Source of truth = `infra/scripts/workstation/roster.yaml`** (os_user → authentik_user/k8s_user/tier/namespaces); `roster_engine.py` (pytest-covered) derives desired state and `t3-provision-users` (hourly systemd timer) applies it — constrained accounts, additive per-tier groups, `t3-serve@` instances, and **regenerating** `/etc/ttyd-user-map` + `dispatch.json` (those two are now GENERATED — do not hand-edit). New non-admins inherit wizard's Claude config (machine-wide managed `claudeMd` in `/etc/claude-code/managed-settings.json` + per-user `~/.claude/{skills,rules}` symlinks seeded by `/etc/skel`) and get a **writable git-crypt-LOCKED** infra clone at `~/code` (code plaintext, secret files ciphertext). Tiers: admin / power-user (cluster-wide read-only) / namespace-owner. **Add a user:** one entry in `roster.yaml` → reconcile. Per-user OIDC kubeconfig, the `oidc-power-user-readonly` ClusterRole, and the Authentik `T3 Users` edge gate are applied (the gate is live — only `T3 Users` members reach t3); the emo cutover to his own locked clone is the remaining gated step. DevVM artifacts versioned in `infra/scripts/` (`t3-serve@.service`, `t3-provision-users` + `workstation/{roster.yaml,roster_engine.py,setup-devvm.sh,managed-settings.json,skel/}`, `t3-dispatch/`, `t3-mint`, `sudoers-t3-autopair`, `t3-autoupdate.*`); TF (`stacks/t3code`) owns only the ingress + Endpoints→:3780. **t3 binary is PINNED** (`T3_PIN`, currently `0.0.24`) — `t3-autoupdate` is a daily *enforcer* that re-asserts the pin (a no-op when correct; restarts only idle instances), NOT a nightly tracker. It used to track `nightly`, but on 2026-06-09 a nightly bump migrated every `~/.t3/state.sqlite` forward (`role`→`scopes`) and changed the bootstrap API, breaking pairing for ALL users (post-mortem `2026-06-09-t3-nightly-autoupdate-auth-outage.md`). t3 is pre-1.0, so moving the pin requires first verifying `t3-dispatch`'s bootstrap flow against the new build (expect 302 + `t3_session`). Pin set in `t3-autoupdate.sh` + `setup-devvm.sh` (keep in sync). Native app/app.t3.codes unsupported (cross-origin) — deferred until published. Design: `docs/plans/2026-06-01-t3-auto-provision-*`. | t3code | +| t3code | Multi-user coding-agent GUI at t3.viktorbarzin.me. `auth=required` (Authentik) → DevVM `t3-dispatch` service (`10.0.10.10:3780`, unprivileged user) maps `X-authentik-username` → that user's own `t3-serve@` instance (file perms enforced by uid; wizard→:3773, emo→:3774; unmapped→403) and **auto-injects the t3 session on first visit** (mints via the root `t3-mint` wrapper, scoped sudoers → `/api/auth/bootstrap` `t3_session` cookie). **Source of truth = `infra/scripts/workstation/roster.yaml`** (os_user → authentik_user/k8s_user/tier/namespaces); `roster_engine.py` (pytest-covered) derives desired state and `t3-provision-users` (hourly systemd timer) applies it — constrained accounts, additive per-tier groups, `t3-serve@` instances, and **regenerating** `/etc/ttyd-user-map` + `dispatch.json` (those two are now GENERATED — do not hand-edit). New non-admins inherit wizard's Claude config (machine-wide managed `claudeMd` in `/etc/claude-code/managed-settings.json` + per-user `~/.claude/{skills,rules}` symlinks seeded by `/etc/skel`) and get a **writable git-crypt-LOCKED** infra clone at `~/code` (code plaintext, secret files ciphertext). Tiers: admin / power-user (cluster-wide read-only) / namespace-owner. **Add a user:** one entry in `roster.yaml` → reconcile. Per-user OIDC kubeconfig, the `oidc-power-user-readonly` ClusterRole, and the Authentik `T3 Users` edge gate are applied (the gate is live — only `T3 Users` members reach t3); the emo cutover to his own locked clone is the remaining gated step. DevVM artifacts versioned in `infra/scripts/` (`t3-serve@.service`, `t3-provision-users` + `workstation/{roster.yaml,roster_engine.py,setup-devvm.sh,managed-settings.json,skel/}`, `t3-dispatch/`, `t3-mint`, `sudoers-t3-autopair`, `t3-autoupdate.*`); TF (`stacks/t3code`) owns only the ingress + Endpoints→:3780. **t3 binary is PINNED** (`T3_PIN`, currently `0.0.24`) — `t3-autoupdate` is a daily *enforcer* that re-asserts the pin (a no-op when correct; restarts only idle instances), NOT a nightly tracker. It used to track `nightly`, but on 2026-06-09 a nightly bump migrated every `~/.t3/state.sqlite` forward (`role`→`scopes`) and changed the bootstrap API, breaking pairing for ALL users (post-mortem `2026-06-09-t3-nightly-autoupdate-auth-outage.md`). t3 is pre-1.0, so moving the pin is a deliberate, reversible step via `docs/runbooks/t3-version-bump.md` (pre-bump `state.sqlite` backup → bump `T3_PIN` → enforcer install with a REAL pairing health-check that auto-rolls-back → verify → restore). Pin set in `t3-autoupdate.sh` + `setup-devvm.sh` (keep in sync). `t3-dispatch` is **version-agnostic** (2026-06-09): `autoPair` tries `/api/auth/browser-session` (0.0.25) then falls back to `/api/auth/bootstrap` (0.0.24), so 0.0.24↔0.0.25 needs no dispatch change. `~/.t3` is backed up daily by `t3-backup-state` (online `VACUUM INTO`; previously unbacked — it's the only copy). Native app/app.t3.codes unsupported (cross-origin) — deferred until published. Design: `docs/plans/2026-06-01-t3-auto-provision-*`. | t3code | ## Active Use | Service | Description | Stack | diff --git a/docs/post-mortems/2026-06-09-t3-nightly-autoupdate-auth-outage.md b/docs/post-mortems/2026-06-09-t3-nightly-autoupdate-auth-outage.md index e8f0d2d5..4aa4cb77 100644 --- a/docs/post-mortems/2026-06-09-t3-nightly-autoupdate-auth-outage.md +++ b/docs/post-mortems/2026-06-09-t3-nightly-autoupdate-auth-outage.md @@ -106,12 +106,29 @@ first run. live session and all projection history were untouched. Backup: `/home/wizard/.t3/userdata/auth-backup-*.sql`. -### 5. End-to-end pairing health-check (DEFERRED) +### 5. End-to-end pairing health-check (DONE — 2026-06-09 follow-up) -The smoke test should exercise mint→bootstrap→cookie, not just `GET /`. Not -done here (the pin makes it moot for the known-good build); needed before the -enforcer is ever pointed at a new version. A blackbox probe on the dispatch -auto-pair (expect 302 + `t3_session`) would have alerted within minutes. +`t3-autoupdate.sh`'s smoke test now exercises the REAL handshake — mint → +`POST` the credential (trying `browser-session` then `bootstrap`) → require +`200` + a `t3_session` cookie — not just `GET / → 200`. A build that renames or +breaks the pairing API now fails the check and **auto-rolls-back**, instead of +shipping a pairing-broken binary to everyone. + +### 6. Version-agnostic dispatch + reversible bumps (DONE — "prepare for 0.0.25") + +So the pin can move without another outage: +- **`t3-dispatch` is now version-agnostic** — `autoPair` tries + `/api/auth/browser-session` (0.0.25) and falls back to `/api/auth/bootstrap` + (0.0.24), so one binary pairs across the rename and through rolling-restart + skew. Covered by `TestAutoPairAcrossVersions`. Investigation confirmed the + 0.0.25 break was *only* this endpoint rename — the rest of the contract + (credential payload, `t3_session` cookie, `/api/auth/session`) is byte-identical. +- **`~/.t3` state is now backed up** — `t3-backup-state` (daily timer, online + `VACUUM INTO`, timeout-guarded) snapshots each user's `state.sqlite` (previously + the only copy, unbacked). This turns the one-way forward migration into a + *restore*, not sqlite surgery. +- **Cutover is a checklist** — `docs/runbooks/t3-version-bump.md` (pre-flight + verify, pre-bump backup, enforcer install + auto-rollback, verify, restore). ## Lessons diff --git a/docs/runbooks/t3-version-bump.md b/docs/runbooks/t3-version-bump.md new file mode 100644 index 00000000..ce0fbcc1 --- /dev/null +++ b/docs/runbooks/t3-version-bump.md @@ -0,0 +1,105 @@ +# Runbook: bump the pinned t3 version (e.g. 0.0.24 → 0.0.25) + +t3 on the devvm is **pinned** (`T3_PIN`, default `0.0.24`) and held there by the +`t3-autoupdate` enforcer. t3 is pre-1.0 and ships breaking changes between +builds, so a bump is a **deliberate, verified, reversible** step — never an +auto-update. This runbook makes it calm. Background: post-mortem +`2026-06-09-t3-nightly-autoupdate-auth-outage.md`. + +## What a bump actually touches + +1. **Pairing API** — t3 renamed `POST /api/auth/bootstrap` → `/api/auth/browser-session` + in 0.0.25. `t3-dispatch` is now **version-agnostic** (tries `browser-session`, + falls back to `bootstrap`; see `pairEndpoints` in `scripts/t3-dispatch/main.go`), + so 0.0.24↔0.0.25 needs **no dispatch change**. If a *future* build renames it + again, add the new path to `pairEndpoints`, rebuild, redeploy first. +2. **Schema** — 0.0.25+ migrate every `~/.t3/userdata/state.sqlite` **forward** + (`auth_pairing_links`/`auth_sessions` `role`→`scopes`, `+proof_key_thumbprint`). + This is a **one-way door**: a binary downgrade alone will NOT roll it back — + you must restore the DB. Hence the mandatory pre-bump backup below. + +## Pre-flight (no downtime) + +```bash +# 1. Confirm the dispatch already speaks the new version's pairing API. +# Install the candidate to an isolated prefix (does NOT touch the global pin): +npm install --prefix /tmp/t3-cand t3@ # e.g. t3@0.0.25 +BIN=/tmp/t3-cand/node_modules/.bin/t3; D=$(mktemp -d) +"$BIN" serve --host 127.0.0.1 --port 3796 --base-dir "$D" >/tmp/cand.log 2>&1 & +CRED=$("$BIN" auth pairing create --base-dir "$D" --ttl 5m --json | sed -n 's/.*"credential":"\([^"]*\)".*/\1/p') +# Try the dispatch's endpoints; one must give 200 + Set-Cookie: t3_session. +for ep in /api/auth/browser-session /api/auth/bootstrap; do + curl -s -i -X POST -H 'Content-Type: application/json' -d "{\"credential\":\"$CRED\"}" \ + "http://127.0.0.1:3796$ep" | grep -iE 'HTTP/|set-cookie: t3_session'; done +kill %1; rm -rf "$D" /tmp/t3-cand +# If NO endpoint yields a t3_session cookie -> the API changed again; update +# pairEndpoints in main.go + rebuild the dispatch BEFORE proceeding. + +# 2. Dispatch unit tests still green: +( cd ~/code/infra/scripts/t3-dispatch && go test ./... ) +``` + +## The bump + +```bash +NEW=0.0.25 +# 1. PRE-BUMP BACKUP — the rollback safety net. Per user, stop the serve (so the +# copy is consistent + fast), copy state.sqlite, restart. Do the ACTIVE admin +# instance last / from OUTSIDE its own t3 session (you can't restart the serve +# you're running inside). +for u in $(awk -F= '!/^[[:space:]]*#/&&NF==2{gsub(/ /,"",$2);print $2}' /etc/ttyd-user-map | sort -u); do + src=/home/$u/.t3/userdata/state.sqlite; [ -f "$src" ] || continue + sudo systemctl stop t3-serve@$u + sudo install -d -o "$u" -g "$u" -m700 /var/backups/t3-state/$u + sudo cp -a "$src" /var/backups/t3-state/$u/state-prebump-$NEW-$(date +%Y%m%d-%H%M%S).sqlite + sudo systemctl start t3-serve@$u +done +# (t3-backup-state also runs daily; this captures a guaranteed snapshot at T-0.) + +# 2. Move the pin in BOTH places (keep them in sync): +sed -i "s/T3_PIN:-[0-9.]*/T3_PIN:-$NEW/" ~/code/infra/scripts/t3-autoupdate.sh \ + ~/code/infra/scripts/workstation/setup-devvm.sh +sudo install -m0755 ~/code/infra/scripts/t3-autoupdate.sh /usr/local/bin/t3-autoupdate + +# 3. Run the enforcer. It installs t3@$NEW, then HEALTH-CHECKS the real pairing +# handshake (mint -> browser-session/bootstrap -> t3_session). If pairing is +# broken in $NEW, it AUTO-ROLLS-BACK to the previous version and exits non-zero. +sudo /usr/local/bin/t3-autoupdate # restarts idle instances; defers active ones + +# 4. Restart any instance the enforcer deferred (active agent), when it's idle. +# The wizard/admin instance: restart from OUTSIDE its own session, or it picks +# up $NEW on its next natural restart (the unit runs the global /usr/bin/t3). +``` + +## Verify + +```bash +for u in vbarzin emil.barzin ancaelena98; do + curl -sI -H "X-authentik-username: $u" http://10.0.10.10:3780/ | grep -iE 'HTTP/|set-cookie: t3_session' +done # each must be 302 + t3_session +t3 --version # == $NEW +``` + +## Rollback (if pairing breaks or $NEW misbehaves) + +The enforcer auto-rolls-back the **binary** if its health-check fails. But if a +problem surfaces *after* serves migrated their DBs forward, the binary alone +won't fix it — restore the DBs: + +```bash +sed -i "s/T3_PIN:-[0-9.]*/T3_PIN:-0.0.24/" ~/code/infra/scripts/t3-autoupdate.sh ~/code/infra/scripts/workstation/setup-devvm.sh +sudo install -m0755 ~/code/infra/scripts/t3-autoupdate.sh /usr/local/bin/t3-autoupdate +sudo npm i -g t3@0.0.24 +for u in $(awk -F= '!/^[[:space:]]*#/&&NF==2{gsub(/ /,"",$2);print $2}' /etc/ttyd-user-map | sort -u); do + bak=$(sudo ls -1t /var/backups/t3-state/$u/state-prebump-* 2>/dev/null | head -1) + [ -n "$bak" ] || continue + sudo systemctl stop t3-serve@$u + sudo install -o "$u" -g "$u" -m600 "$bak" /home/$u/.t3/userdata/state.sqlite + sudo rm -f /home/$u/.t3/userdata/state.sqlite-wal /home/$u/.t3/userdata/state.sqlite-shm + sudo systemctl start t3-serve@$u +done +# verify 302 + t3_session as above +``` + +(The 2026-06-09 incident had no pre-bump backup, so rollback meant per-user +sqlite surgery. With the backup, it's a restore.) diff --git a/scripts/t3-autoupdate.sh b/scripts/t3-autoupdate.sh index 836605f0..4eac8ddf 100644 --- a/scripts/t3-autoupdate.sh +++ b/scripts/t3-autoupdate.sh @@ -9,8 +9,10 @@ # To move the pin: bump T3_PIN AND first verify t3-dispatch's bootstrap flow against the # new build (curl the dispatch -> expect 302 + Set-Cookie t3_session). See post-mortem # 2026-06-09-t3-nightly-autoupdate-auth-outage.md. -# CAVEAT: the health-check below only probes GET / (200) — it does NOT exercise the -# mint/bootstrap/pairing path, so it will NOT catch an auth regression on its own. +# The health-check below exercises the REAL pairing handshake (mint -> credential +# exchange -> t3_session cookie), mirroring t3-dispatch's endpoint fallback — so a +# build that renames or breaks the pairing API fails the check and auto-rolls-back +# (closes the 2026-06-09 miss, where a GET / probe passed a pairing-broken build). set -uo pipefail T3_PIN="${T3_PIN:-0.0.24}" # known-good, t3-dispatch-compatible (2026-06-09 post-mortem) LOG() { logger -t t3-autoupdate "$*"; echo "t3-autoupdate: $*"; } @@ -27,17 +29,34 @@ fi LOG "re-pinned to $after (was $before); health-checking…" # Health-check the NEW binary on a throwaway port/base-dir before trusting it. +# Gate 1 = liveness (GET / -> 200); Gate 2 = the REAL pairing handshake t3-dispatch +# performs (mint -> POST credential -> 200 + t3_session cookie), trying the same +# endpoint fallback. Gate 2 catches a bootstrap-API rename / pairing regression. SMOKE_PORT=3799; SMOKE_DIR=$(mktemp -d) t3 serve --host 127.0.0.1 --port "$SMOKE_PORT" --base-dir "$SMOKE_DIR" >/dev/null 2>&1 & -smoke=$!; ok=0 +smoke=$!; live=0; pair_ok=0 for _ in $(seq 1 15); do - [[ "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "http://127.0.0.1:$SMOKE_PORT/" 2>/dev/null)" == "200" ]] && { ok=1; break; } + [[ "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "http://127.0.0.1:$SMOKE_PORT/" 2>/dev/null)" == "200" ]] && { live=1; break; } sleep 2 done +if [[ "$live" == "1" ]]; then + cred=$(t3 auth pairing create --base-dir "$SMOKE_DIR" --ttl 5m --json 2>/dev/null \ + | tr -d '\n ' | sed -n 's/.*"credential":"\([^"]*\)".*/\1/p') + if [[ -n "$cred" ]]; then + for ep in /api/auth/browser-session /api/auth/bootstrap; do # mirror t3-dispatch's fallback + hdr=$(curl -s -i --max-time 5 -X POST -H 'Content-Type: application/json' \ + -d "{\"credential\":\"$cred\"}" "http://127.0.0.1:$SMOKE_PORT$ep" 2>/dev/null) + code=$(printf '%s' "$hdr" | sed -n '1s#.* \([0-9][0-9][0-9]\).*#\1#p') + [[ "$code" == "404" ]] && continue # endpoint absent in this build — try the next + printf '%s' "$hdr" | grep -qi '^set-cookie:[[:space:]]*t3_session=' && pair_ok=1 + break + done + fi +fi kill "$smoke" 2>/dev/null; wait "$smoke" 2>/dev/null; rm -rf "$SMOKE_DIR" -if [[ "$ok" != "1" ]]; then - LOG "HEALTH-CHECK FAILED for $after — rolling back to $before" +if [[ "$live" != "1" || "$pair_ok" != "1" ]]; then + LOG "HEALTH-CHECK FAILED for $after (live=$live pair=$pair_ok) — rolling back to $before" if [[ -n "$before" ]] && npm i -g "t3@$before" >/dev/null 2>&1; then LOG "rolled back to $before" else @@ -45,7 +64,7 @@ if [[ "$ok" != "1" ]]; then fi exit 1 fi -LOG "health OK; restarting idle instances" +LOG "health OK (live + pairing handshake); restarting idle instances" # Restart only IDLE per-user instances; defer any with an active agent child. for unit in $(systemctl list-units --type=service --state=running --no-legend 't3-serve@*' | awk '{print $1}'); do diff --git a/scripts/t3-backup-state.service b/scripts/t3-backup-state.service new file mode 100644 index 00000000..5f590942 --- /dev/null +++ b/scripts/t3-backup-state.service @@ -0,0 +1,6 @@ +[Unit] +Description=Consistent backup of per-user t3 ~/.t3 state.sqlite (history + auth) + +[Service] +Type=oneshot +ExecStart=/usr/local/bin/t3-backup-state diff --git a/scripts/t3-backup-state.sh b/scripts/t3-backup-state.sh new file mode 100644 index 00000000..7ade2892 --- /dev/null +++ b/scripts/t3-backup-state.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# Consistent online backup of each t3 user's ~/.t3 state.sqlite (chat/session +# history AND auth tables). ~/.t3 lives on the devvm local disk — NOT a K8s PVC and +# NOT in the 3-2-1 pipeline — so without this it is the only copy and a rebuild +# loses it. It also makes a t3 version bump REVERSIBLE: 0.0.25+ migrate the schema +# FORWARD (a one-way door), so a clean pre-bump backup turns rollback into a restore +# instead of per-user sqlite surgery (see runbooks/t3-version-bump.md). Runs as root +# via t3-backup-state.timer; the per-user .backup runs AS the owning user so the live +# WAL/-shm files keep their owner and the running t3-serve is never perturbed. +set -uo pipefail +DEST="${T3_BACKUP_DEST:-/var/backups/t3-state}" +KEEP="${T3_BACKUP_KEEP:-14}" +MAP=/etc/ttyd-user-map +LOG() { logger -t t3-backup-state "$*"; echo "t3-backup-state: $*"; } + +ts=$(date +%Y%m%d-%H%M%S) +# RHS of each non-comment "authentik=os_user" line = an OS user owning a ~/.t3. +mapfile -t users < <(awk -F= '!/^[[:space:]]*#/ && NF==2 { gsub(/[[:space:]]/,"",$2); print $2 }' "$MAP" 2>/dev/null | sort -u) +[[ ${#users[@]} -gt 0 ]] || { LOG "no users in $MAP; nothing to back up"; exit 0; } + +rc=0 +for u in "${users[@]}"; do + src="/home/$u/.t3/userdata/state.sqlite" + if [[ ! -f "$src" ]]; then LOG "skip $u (no state.sqlite)"; continue; fi + out="$DEST/$u"; dst="$out/state-$ts.sqlite" + install -d -o "$u" -g "$u" -m 0700 "$out" + # VACUUM INTO takes a consistent read-snapshot copy — unlike .backup it does NOT + # restart when the source is written mid-copy, so it finishes in a single pass even + # for the actively-used instance (the admin's own live session, which .backup would + # loop on forever). Run as the owning user so WAL access keeps the live serve happy. + # timeout caps a pathologically-slow copy (huge DB + concurrent writes on a contended + # disk) so the daily run can never wedge — it just logs + retries next cycle. The + # daily 03:30 slot normally finds instances idle, where even a large DB copies fast. + if runuser -u "$u" -- timeout "${T3_BACKUP_TIMEOUT:-900}" sqlite3 "$src" "VACUUM INTO '$dst'" 2>/dev/null && [[ -s "$dst" ]]; then + LOG "backed up $u -> $dst ($(stat -c%s "$dst" 2>/dev/null) bytes)" + else + LOG "WARN: backup FAILED for $u ($src)"; rc=1; rm -f "$dst" + fi + # retention: keep newest $KEEP per user + ls -1t "$out"/state-*.sqlite 2>/dev/null | tail -n +$((KEEP+1)) | xargs -r rm -f +done +LOG "done (rc=$rc)" +exit $rc diff --git a/scripts/t3-backup-state.timer b/scripts/t3-backup-state.timer new file mode 100644 index 00000000..72ac48e5 --- /dev/null +++ b/scripts/t3-backup-state.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Daily t3 state.sqlite backup (the only copy of ~/.t3; enables version-bump rollback) + +[Timer] +OnCalendar=*-*-* 03:30:00 +RandomizedDelaySec=20m +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/scripts/t3-dispatch/main.go b/scripts/t3-dispatch/main.go index 401b0edb..a36e2da0 100644 --- a/scripts/t3-dispatch/main.go +++ b/scripts/t3-dispatch/main.go @@ -113,9 +113,42 @@ func isDocumentNav(r *http.Request) bool { return strings.Contains(r.Header.Get("Accept"), "text/html") } +// pairEndpoints are the instance's session-bootstrap paths in preference order. +// t3 renamed /api/auth/bootstrap -> /api/auth/browser-session in 0.0.25; trying the +// new name first and falling back to the old lets ONE dispatch binary pair against +// either version — so the t3 pin can move forward (and survive a rolling-restart +// skew where some instances are already on the new version) without a 502 storm. +var pairEndpoints = []string{"/api/auth/browser-session", "/api/auth/bootstrap"} + +// exchangeCredential POSTs the pairing credential to the user's instance, trying +// each pairEndpoint in turn. A 404 means "absent in this t3 version" -> try the +// next; any other status is that endpoint's verdict, returned as-is. Caller owns +// resp.Body. +func exchangeCredential(port int, credential string) (*http.Response, error) { + body, _ := json.Marshal(map[string]string{"credential": credential}) + var lastErr error + for _, ep := range pairEndpoints { + resp, err := http.Post(fmt.Sprintf("http://127.0.0.1:%d%s", port, ep), + "application/json", bytes.NewReader(body)) + if err != nil { + lastErr = err + continue + } + if resp.StatusCode == http.StatusNotFound { + resp.Body.Close() // endpoint absent in this t3 version — try the next + continue + } + return resp, nil + } + if lastErr != nil { + return nil, lastErr + } + return nil, fmt.Errorf("no pairing endpoint accepted the request (all returned 404)") +} + // autoPair mints a one-time pairing token for the user's instance (as that OS -// user, via the scoped sudoers entry) and exchanges it at the instance's -// /api/auth/bootstrap, relaying the returned t3_session Set-Cookie to the browser. +// user, via the scoped sudoers entry) and exchanges it at the instance's pairing +// endpoint, relaying the returned t3_session Set-Cookie to the browser. func autoPair(e entry, w http.ResponseWriter, r *http.Request) { // t3-mint (root, via scoped sudoers) validates the OS user is in // /etc/ttyd-user-map, then mints as that user. The dispatch service itself @@ -133,16 +166,15 @@ func autoPair(e entry, w http.ResponseWriter, r *http.Request) { http.Error(w, "unparseable pairing output", http.StatusInternalServerError) return } - body, _ := json.Marshal(map[string]string{"credential": pc.Credential}) - resp, err := http.Post(fmt.Sprintf("http://127.0.0.1:%d/api/auth/bootstrap", e.Port), - "application/json", bytes.NewReader(body)) + resp, err := exchangeCredential(e.Port, pc.Credential) if err != nil { + log.Printf("pairing exchange for %s failed: %v", e.OsUser, err) http.Error(w, "bootstrap request failed", http.StatusBadGateway) return } defer resp.Body.Close() if resp.StatusCode != http.StatusOK { - log.Printf("bootstrap for %s returned %d", e.OsUser, resp.StatusCode) + log.Printf("pairing for %s returned %d", e.OsUser, resp.StatusCode) http.Error(w, "bootstrap rejected", http.StatusBadGateway) return } diff --git a/scripts/t3-dispatch/main_test.go b/scripts/t3-dispatch/main_test.go index 81ca26a9..ee43266e 100644 --- a/scripts/t3-dispatch/main_test.go +++ b/scripts/t3-dispatch/main_test.go @@ -117,6 +117,8 @@ func fakeInstance(authenticated bool, bootstrapCalled *bool) *httptest.Server { } http.SetCookie(w, &http.Cookie{Name: cookieName, Value: "fresh", Path: "/"}) _, _ = w.Write([]byte(`{"authenticated":true}`)) + case "/api/auth/browser-session": + http.NotFound(w, r) // models a 0.0.24 instance: the 0.0.25 endpoint is absent default: _, _ = w.Write([]byte("APP")) } @@ -198,3 +200,61 @@ func TestHandlerProxiesXHREvenIfCookieInvalid(t *testing.T) { t.Fatalf("XHR should proxy through, got %d %q", w.Code, w.Body.String()) } } + +// pairInstance simulates a t3 instance that exposes pairing at exactly one path +// (200 + t3_session) and 404s the other known path — modeling the 0.0.25 rename of +// /api/auth/bootstrap -> /api/auth/browser-session. records which path was hit. +func pairInstance(pairPath string, hit *string) *httptest.Server { + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/api/auth/browser-session", "/api/auth/bootstrap": + if r.URL.Path != pairPath { + http.NotFound(w, r) // endpoint absent in this t3 version + return + } + if hit != nil { + *hit = r.URL.Path + } + http.SetCookie(w, &http.Cookie{Name: cookieName, Value: "fresh", Path: "/"}) + _, _ = w.Write([]byte(`{"authenticated":true}`)) + default: + http.NotFound(w, r) + } + })) +} + +// TestAutoPairAcrossVersions: one dispatch binary must pair against BOTH the +// 0.0.24 endpoint (/api/auth/bootstrap) and the 0.0.25 one (/api/auth/browser-session), +// so the pin can move forward (and survive rolling-restart skew) without a 502 storm. +func TestAutoPairAcrossVersions(t *testing.T) { + orig := mintToken + mintToken = func(string) ([]byte, error) { return []byte(`{"credential":"tok"}`), nil } + defer func() { mintToken = orig }() + + for _, tc := range []struct{ name, pairPath string }{ + {"0.0.25 browser-session", "/api/auth/browser-session"}, + {"0.0.24 bootstrap", "/api/auth/bootstrap"}, + } { + t.Run(tc.name, func(t *testing.T) { + var hit string + ts := pairInstance(tc.pairPath, &hit) + defer ts.Close() + setTable(portOf(t, ts)) + + r := httptest.NewRequest("GET", "/", nil) + r.Header.Set("X-authentik-username", "vbarzin@gmail.com") // no cookie -> autoPair + w := httptest.NewRecorder() + handler(w, r) + + if w.Code != http.StatusFound { + t.Fatalf("want 302 re-pair, got %d body=%q", w.Code, w.Body.String()) + } + if hit != tc.pairPath { + t.Fatalf("want pairing via %s, hit=%q", tc.pairPath, hit) + } + if cs := w.Result().Cookies(); len(cs) == 0 || cs[0].Value != "fresh" { + t.Fatalf("want fresh t3_session relayed, got %+v", cs) + } + }) + } +} From 7fc4caefe34b3225af6551d7f459ce7441755ece Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 20:55:47 +0000 Subject: [PATCH 14/24] t3: bump pin 0.0.24 -> 0.0.26 (fable-5) [ci skip] Completes the 0.0.26 adoption prepared in fcb84ce0 (version-agnostic dispatch browser-session/bootstrap fallback + Gate-2 real pairing health-check + per-user state.sqlite backup). 0.0.26 verified end-to-end on the devvm: emo + ancamilea auto-pair via t3-dispatch (302 + Set-Cookie t3_session) after migrating state.sqlite 30->32; pre-cutover backups in /var/backups/t3-state. Brings claude-fable-5 into the t3 model picker. Co-Authored-By: Claude Opus 4.8 --- scripts/t3-autoupdate.sh | 2 +- scripts/workstation/setup-devvm.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/t3-autoupdate.sh b/scripts/t3-autoupdate.sh index 4eac8ddf..88ccc766 100644 --- a/scripts/t3-autoupdate.sh +++ b/scripts/t3-autoupdate.sh @@ -14,7 +14,7 @@ # build that renames or breaks the pairing API fails the check and auto-rolls-back # (closes the 2026-06-09 miss, where a GET / probe passed a pairing-broken build). set -uo pipefail -T3_PIN="${T3_PIN:-0.0.24}" # known-good, t3-dispatch-compatible (2026-06-09 post-mortem) +T3_PIN="${T3_PIN:-0.0.26}" # known-good, t3-dispatch-compatible (2026-06-09 post-mortem) LOG() { logger -t t3-autoupdate "$*"; echo "t3-autoupdate: $*"; } ver() { t3 --version 2>/dev/null | awk '{print $NF}' | sed 's/^v//'; } diff --git a/scripts/workstation/setup-devvm.sh b/scripts/workstation/setup-devvm.sh index faf7b7bc..526e7470 100755 --- a/scripts/workstation/setup-devvm.sh +++ b/scripts/workstation/setup-devvm.sh @@ -38,7 +38,7 @@ command -v claude >/dev/null || { log "npm: installing @anthropic-ai/claude-code # (2026-06-09 outage: a nightly auto-update broke pairing for ALL users). The daily # t3-autoupdate ENFORCER re-asserts this same pin; install it here so a fresh box has t3 # immediately. Keep T3_PIN in sync with t3-autoupdate.sh. -T3_PIN="${T3_PIN:-0.0.24}" +T3_PIN="${T3_PIN:-0.0.26}" if [[ "$(t3 --version 2>/dev/null | awk '{print $NF}' | sed 's/^v//')" != "$T3_PIN" ]]; then log "npm: installing pinned t3@$T3_PIN"; npm install -g "t3@$T3_PIN" >/dev/null fi From 83f418159a74dd4399f5705a63468f82ca83f438 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 21:22:34 +0000 Subject: [PATCH 15/24] =?UTF-8?q?backup:=20image-level=20vzdump=20of=20han?= =?UTF-8?q?d-managed=20VMs=20(devvm)=20=E2=80=94=20close=20no-VM-backup=20?= =?UTF-8?q?DR=20gap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The hand-managed Linux VMs (not in Terraform) were never imaged: the PVC/NFS/pfSense/PVE-config scripts cover cluster data but no VM disk. A lost devvm disk = unrecoverable home dirs + local-only git repos (monorepo root has no remote). vzdump-vms.{sh,service,timer}: daily 01:00 live `vzdump --mode snapshot` of VZDUMP_VMIDS (default 102=devvm) -> /mnt/backup/vzdump (Copy 2), keep 3; the monthly offsite-sync full pass mirrors it to Synology (Copy 3). Guest agent enabled -> fs-consistent. Nice/idle-ionice so it never starves etcd. Pushgateway job vzdump-backup. Deployed live to PVE + timer enabled. Docs updated: backup-dr.md (new VM-image layer + protection matrix), infra CLAUDE.md, AGENTS.md. [ci skip] Co-Authored-By: Claude Opus 4.8 --- .claude/CLAUDE.md | 1 + AGENTS.md | 3 +- docs/architecture/backup-dr.md | 28 +++++++- scripts/vzdump-vms.service | 16 +++++ scripts/vzdump-vms.sh | 117 +++++++++++++++++++++++++++++++++ scripts/vzdump-vms.timer | 14 ++++ 6 files changed, 177 insertions(+), 2 deletions(-) create mode 100644 scripts/vzdump-vms.service create mode 100644 scripts/vzdump-vms.sh create mode 100644 scripts/vzdump-vms.timer diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index e3e59fe6..3fb81fc2 100755 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -293,6 +293,7 @@ resource "kubernetes_persistent_volume_claim" "data_encrypted" { - `/usr/local/bin/daily-backup` — Daily 05:00. Mounts LVM thin snapshots ro → rsyncs FILES to `/mnt/backup/pvc-data////` with `--link-dest` versioning (4 weeks). Auto SQLite backup (magic number check, `?mode=ro`). Also backs up pfSense (config.xml + tar), PVE config. Prunes snapshots >7d. **Skip-list (2026-06-01)**: `nextcloud/nextcloud-data-proxmox` (orphaned pre-encryption PV). - `/usr/local/bin/offsite-sync-backup` — Daily 06:00 (After=daily-backup). Step 1: sda → Synology `pve-backup/` (incremental via manifest; monthly full `rsync --delete` days 1–7). Step 2: NFS direct → Synology — **immich-only on BOTH `nfs/` and `nfs-ssd/` (2026-06-01)**; ollama/llamacpp on the SSD no longer ship offsite. - `/usr/local/bin/lvm-pvc-snapshot` — Daily 03:00. Thin snapshots of all PVCs except dbaas+monitoring. 7-day retention. Instant restore: `lvm-pvc-snapshot restore `. +- `/usr/local/bin/vzdump-vms` — Daily 01:00. Live `vzdump --mode snapshot` of hand-managed VMs (the ones NOT in Terraform) → `/mnt/backup/vzdump/`, keep 3 per VMID. `VZDUMP_VMIDS` default `102` (devvm) — **the only VM imaged today** (its per-user home dirs + local-only git repos, incl. the no-remote monorepo root, are otherwise irreplaceable). devvm has the guest agent (`agent: 1`) so dumps are fs-consistent. Deliberately NOT in the incremental offsite manifest (would balloon Synology); the monthly offsite full pass (days 1-7) mirrors `/mnt/backup/vzdump/`. Pushgateway job `vzdump-backup`. Added 2026-06-09 (closed the silent "VMs never imaged" DR gap). Restore: `qmrestore /mnt/backup/vzdump/vzdump-qemu--.vma.zst `. - `nfs-change-tracker.service` — Continuous inotifywait on `/srv/nfs` + `/srv/nfs-ssd`. Logs changed file paths to `/mnt/backup/.nfs-changes.log`. Consumed by offsite-sync-backup for incremental rsync (completes in seconds instead of 30+ minutes). **Synology layout** (`192.168.1.13:/volume1/Backup/Viki/`): diff --git a/AGENTS.md b/AGENTS.md index 009c5c99..984e98d6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -109,7 +109,8 @@ Terragrunt-based homelab managing a Kubernetes cluster (5 nodes, v1.34.2) on Pro - **SQLite on NFS is unreliable** (fsync issues) — always use proxmox-lvm or local disk for databases. - **NFS mount options**: Always `soft,timeo=30,retrans=3` to prevent uninterruptible sleep (D state). - **NFS export directory must exist** on the Proxmox host before Terraform can create the PV. -- **Backup (3-2-1)**: Copy 1 = live PVCs on sdc. Copy 2 = sda `/mnt/backup` (PVC file backups, auto SQLite backups, pfSense, PVE config). Copy 3 = Synology offsite (two-tier: sda→`pve-backup/`, NFS→`nfs/`+`nfs-ssd/` via inotify change tracking). +- **Backup (3-2-1)**: Copy 1 = live PVCs on sdc. Copy 2 = sda `/mnt/backup` (PVC file backups, auto SQLite backups, pfSense, PVE config, **VM images via `vzdump-vms`**). Copy 3 = Synology offsite (two-tier: sda→`pve-backup/`, NFS→`nfs/`+`nfs-ssd/` via inotify change tracking). +- **vzdump-vms** (Daily 01:00): live `vzdump --mode snapshot` of hand-managed VMs (NOT in TF) → `/mnt/backup/vzdump/`, keep 3/VMID. `VZDUMP_VMIDS` default `102` (devvm) — the only VM imaged today; before this (2026-06-09) no VM was ever imaged. NOT in the incremental offsite manifest; monthly full pass mirrors it. See `docs/architecture/backup-dr.md`. - **daily-backup** (Daily 05:00): Auto-discovered BACKUP_DIRS (glob), auto SQLite backup (magic number + `?mode=ro`), pfSense, PVE config. No NFS mirror step (NFS syncs directly to Synology via inotify). - **offsite-sync-backup** (Daily 06:00): Step 1: sda→Synology `pve-backup/`. Step 2: NFS→Synology `nfs/`+`nfs-ssd/` via `rsync --files-from` (inotify change log). Monthly full `--delete`. - **nfs-change-tracker.service**: inotifywait on `/srv/nfs` + `/srv/nfs-ssd`, logs to `/mnt/backup/.nfs-changes.log`. Incremental syncs complete in seconds. diff --git a/docs/architecture/backup-dr.md b/docs/architecture/backup-dr.md index 60c1c77d..93c292dc 100644 --- a/docs/architecture/backup-dr.md +++ b/docs/architecture/backup-dr.md @@ -77,6 +77,8 @@ The **bypass list** (leg 2) is just `/srv/nfs/immich/` — too big for sda (1.5 - `Synology/Backup/Viki/nfs/` — immich only (post-2026-05-26) - `Synology/Backup/Viki/nfs-ssd/` — **immich-ML only (2026-06-01)**; ollama/llamacpp dropped (re-pullable models, live-only on the SSD) +**VM image backups (added 2026-06-09)**: the hand-managed Linux VMs (those NOT in Terraform — see `compute.md`) were historically **not imaged at all** — only their *contents* reached backup if they happened to host a PVC/NFS path. `vzdump-vms` now takes a daily live `vzdump --mode snapshot` of each configured VMID → `/mnt/backup/vzdump/` (Copy 2), carried offsite by the monthly offsite-sync full pass (Copy 3). **Currently enabled for VMID 102 (devvm)** — the shared workstation, whose per-user home dirs + local-only git repos are otherwise irreplaceable. Extend via `VZDUMP_VMIDS` in the unit. See "VM Image Backups (vzdump)" under How It Works. + ## Architecture Diagram ### Data Routing — where each path goes (post-2026-05-26) @@ -208,13 +210,14 @@ graph LR T0000["00:00 LVM thin snapshots
(lvm-pvc-snapshot)
sdc PVCs CoW"] T0015["00:15 PostgreSQL per-DB dumps
(CronJob)"] T0045["00:45 MySQL per-DB dumps
(CronJob)"] + T0100["01:00 vzdump-vms
live image of hand-managed VMs
(devvm) → sda /mnt/backup/vzdump/"] T0200["02:00 nfs-mirror (daily)
sdc /srv/nfs/* → sda /mnt/backup//
~10-20 min steady state"] T0500["05:00 daily-backup
mount LVM snapshots ro
rsync PVC files → /mnt/backup/pvc-data/
+ sqlite + pfsense + pve-config"] T0600["06:00 offsite-sync-backup
Step 1: sda → Synology /Viki/pve-backup/
Step 2: sdc/immich + nfs-ssd → /Viki/nfs[-ssd]/"] T1200["12:00 LVM thin snapshots (midday)
second daily snapshot"] end - T0000 --> T0015 --> T0045 --> T0200 --> T0500 --> T0600 --> T1200 + T0000 --> T0015 --> T0045 --> T0100 --> T0200 --> T0500 --> T0600 --> T1200 INO -.->|change events feed Step 2| T0600 style Nightly fill:#ffe0b2 @@ -322,6 +325,7 @@ graph LR | NFS Change Tracker | Continuous (inotifywait) | PVE host: `nfs-change-tracker.service` | Logs changed NFS file paths to `/mnt/backup/.nfs-changes.log` | | pfSense Backup | Daily 05:00 + daily-backup | PVE host: SSH + API | config.xml + full filesystem tar | | Offsite Sync | Daily 06:00 (after daily-backup) | PVE host: `offsite-sync-backup` | Two-step: sda→pve-backup + NFS→nfs/nfs-ssd via inotify | +| VM Image Backup (vzdump) | Daily 01:00, keep 3 | PVE host: `vzdump-vms` | Live `vzdump` of hand-managed VMs (devvm) → `/mnt/backup/vzdump/` | | PostgreSQL Backup (full) | Daily 00:00, 14d retention | CronJob in `dbaas` namespace | pg_dumpall for all databases | | PostgreSQL Backup (per-db) | Daily 00:15, 14d retention | CronJob in `dbaas` namespace | pg_dump -Fc per database → `/backup/per-db//` | | MySQL Backup (full) | Daily 00:30, 14d retention | CronJob in `dbaas` namespace | mysqldump --all-databases | @@ -352,6 +356,19 @@ Native LVM thin snapshots provide crash-consistent point-in-time recovery for 62 **Restore**: `lvm-pvc-snapshot restore ` — auto-discovers K8s workload, scales down, swaps LVs, scales back up. See `docs/runbooks/restore-lvm-snapshot.md`. +### VM Image Backups (vzdump) + +The hand-managed Linux VMs are **intentionally not in Terraform** (telmate/bpg provider bugs — see `compute.md`) and were historically **not imaged at all**: nothing took a whole-disk backup of the VM itself. For most that is acceptable — k8s nodes are reprovisioned from cloud-init and their data lives in PVCs covered above. But **devvm** (the shared multi-user Claude Code workstation, VMID 102) holds irreplaceable state that lives nowhere else: per-user home dirs (`~/.claude`, `~/.t3`, shell history), manually-installed tooling, and **local-only git repos** — the monorepo root at `/home/wizard/code` has no git remote. A lost devvm disk = unrecoverable. + +**Script**: `/usr/local/bin/vzdump-vms` on PVE host (source: `infra/scripts/vzdump-vms.sh`). Deploy: `scp infra/scripts/vzdump-vms.sh root@192.168.1.127:/usr/local/bin/vzdump-vms` + `scp infra/scripts/vzdump-vms.{service,timer} root@192.168.1.127:/etc/systemd/system/`, then `systemctl daemon-reload && systemctl enable --now vzdump-vms.timer`. +**Schedule**: Daily 01:00 via systemd timer — ahead of the other backup jobs so the fresh image is on sda before offsite-sync runs. +**Mode**: `vzdump --mode snapshot` — live, no downtime. devvm has the qemu guest agent enabled (`agent: 1`), so the snapshot is **filesystem-consistent** (fs-freeze) rather than merely crash-consistent. Runs `Nice=10` + `IOSchedulingClass=idle` + `--ionice 7` so it never starves etcd on the contended sdc IO domain. +**Scope**: VMIDs in `VZDUMP_VMIDS` (default `102` = devvm). Add VMIDs there to image other hand-managed VMs. +**Retention**: `KEEP=3` newest dumps per VMID on sda (`/mnt/backup/vzdump/`); each devvm image is ~35-50 GB zstd. +**Offsite**: deliberately **NOT** appended to the incremental offsite manifest — it never deletes, so daily multi-GB images would accumulate unbounded on Synology. Instead the **monthly offsite-sync full pass (days 1-7)** mirrors all of `/mnt/backup` (including `vzdump/`) to Synology with `--delete`, bounded to local retention. So Copy 2 (sda) refreshes **daily**; Copy 3 (Synology) refreshes **monthly**. +**Monitoring**: pushes `vzdump_last_run_timestamp` / `vzdump_last_status` / `vzdump_last_success_timestamp` to Pushgateway job `vzdump-backup`. A `VzdumpBackupStale` / `VzdumpBackupFailing` alert in `stacks/monitoring` (mirroring the LVM/pfSense backup alerts) is the recommended next addition. +**Restore**: on the PVE host, `qmrestore /mnt/backup/vzdump/vzdump-qemu--.vma.zst ` — restore to a spare VMID first if the original still exists, then swap disks; or use the PVE UI (add `/mnt/backup` as a dir storage with content=backup → Restore). + ### Layer 2: Weekly File-Level Backup (sda Backup Disk) **Backup disk**: sda (1.1TB RAID1 SAS) → VG `backup` → LV `data` → ext4 → mounted at `/mnt/backup` on PVE host. Dedicated backup disk, independent of live storage. @@ -527,12 +544,16 @@ The btrfs cleaner thread reclaims async — `df` may lag the snapshot-delete by | `/usr/local/bin/lvm-pvc-snapshot` | PVE host: LVM snapshot creation + restore | | `/usr/local/bin/daily-backup` | PVE host: PVC file copy + auto SQLite backup + pfSense | | `/usr/local/bin/offsite-sync-backup` | PVE host: two-step rsync to Synology (sda + NFS via inotify) | +| `/usr/local/bin/vzdump-vms` | PVE host: daily live `vzdump` image of hand-managed VMs (devvm) → `/mnt/backup/vzdump/` | | `/mnt/backup/` | PVE host: sda mount point (1.1TB backup disk) | +| `/mnt/backup/vzdump/` | PVE host: vzdump VM images (keep 3 per VMID), mirrored offsite monthly | | `/mnt/backup/.nfs-changes.log` | NFS change log from inotifywait, consumed by offsite-sync | | `/etc/systemd/system/nfs-change-tracker.service` | inotifywait watcher for `/srv/nfs` + `/srv/nfs-ssd` | | `/etc/systemd/system/lvm-pvc-snapshot.timer` | Daily 03:00 (LVM snapshots) | | `/etc/systemd/system/daily-backup.timer` | Daily 05:00 (file backup) | | `/etc/systemd/system/offsite-sync-backup.timer` | Daily 06:00 (offsite sync) | +| `/etc/systemd/system/vzdump-vms.timer` | Daily 01:00 (VM image backup) | +| `/etc/systemd/system/vzdump-vms.service` | oneshot: `vzdump-vms` (source `infra/scripts/vzdump-vms.{sh,service,timer}`) | | `/usr/local/bin/nfs-mirror` | PVE host: daily 02:00 mirror of /srv/nfs/* → sda /mnt/backup// (Layer 3a) | | `/etc/systemd/system/nfs-mirror.timer` | Daily 02:00 (NFS local mirror to sda) | | `stacks/dbaas/` | Terraform: PostgreSQL/MySQL backup CronJobs | @@ -911,6 +932,9 @@ the 2026-04-22 backup_offsite_sync FAIL (node3 kubelet hiccup at | Uptime Kuma | ✓ | ✓ | — | ✓ | proxmox-lvm | | **Other apps not enumerated above** | ✓¹ | ✓¹ | varies | ✓ | proxmox-lvm / proxmox-lvm-encrypted | | **Postiz** (bundled bitnami PG on local-path) | — | — | ✓ daily pg_dump → NFS | ✓ | local-path + NFS | +| **Hand-managed VMs (not in Terraform)** | +| devvm (workstation, VMID 102) | — | — | ✓ daily vzdump image | ✓ monthly | local-lvm (sdc) | +| Other hand-managed VMs (HA 103, registry 220, k8s nodes) | — | — | — gap² | — | local-lvm — see note² | | **Media (NFS)** | | Immich (~800GB) | — | — | — | ✓ | NFS | | Audiobookshelf | — | — | — | ✓ | NFS | @@ -924,6 +948,8 @@ the 2026-04-22 backup_offsite_sync FAIL (node3 kubelet hiccup at **Note**: All proxmox-lvm and proxmox-lvm-encrypted PVCs get LVM snapshots (except `dbaas` and `monitoring` namespaces, excluded for write-amplification reasons) + file-level backup. NFS-backed media syncs directly to Synology `nfs/` and `nfs-ssd/` via inotify change tracking. +² **Hand-managed VMs** — only **devvm (102)** is imaged today (`vzdump-vms`, `VZDUMP_VMIDS=102`). The k8s nodes are deliberately uncovered (reprovisioned from cloud-init; their data lives in the PVCs already backed up above). **home-assistant (103) and docker-registry (220) are a documented gap** — add their VMIDs to `VZDUMP_VMIDS` to image them (registry content is also re-pullable from upstreams; HA has its own add-on backups). pfSense (101) is covered separately by `daily-backup` (config.xml + weekly tar). + ¹ **"Other apps not enumerated above"** — the table only enumerates services worth calling out. The default backup posture for any service using `proxmox-lvm` or `proxmox-lvm-encrypted` (outside `dbaas`/`monitoring`) is **automatic** Layer 1 (LVM thin snapshots, 7d retention) + Layer 2 (file backup, 4 weekly versions on sda) + Layer 3 (offsite to Synology). Auto-discovery is by LV name pattern (`vm-*-pvc-*`), so adding a new service to the cluster gets it covered without any explicit registration. Run `ssh root@192.168.1.127 lvs --noheadings -o lv_name pve | grep '^vm-.*-pvc-' | grep -v _snap_ | wc -l` to see the live count. **Known gaps** — services with PVCs not on the proxmox-lvm path lose Layer 1+2: diff --git a/scripts/vzdump-vms.service b/scripts/vzdump-vms.service new file mode 100644 index 00000000..32ac8f96 --- /dev/null +++ b/scripts/vzdump-vms.service @@ -0,0 +1,16 @@ +[Unit] +Description=vzdump image backup of hand-managed VMs (devvm, …) to /mnt/backup +Documentation=https://forgejo.viktorbarzin.me/viktor/infra/src/branch/main/docs/architecture/backup-dr.md +After=network-online.target +Wants=network-online.target +RequiresMountsFor=/mnt/backup + +[Service] +Type=oneshot +ExecStart=/usr/local/bin/vzdump-vms +# Be gentle on the contended PVE IO domain (sdc) — backup must never starve etcd. +Nice=10 +IOSchedulingClass=idle +# Reading a ~77 GB disk + zstd can run long under IO contention; well above +# normal (~15-30 min) but bounded so a hung run can't wedge the timer forever. +TimeoutStartSec=4h diff --git a/scripts/vzdump-vms.sh b/scripts/vzdump-vms.sh new file mode 100644 index 00000000..fabb45e7 --- /dev/null +++ b/scripts/vzdump-vms.sh @@ -0,0 +1,117 @@ +#!/usr/bin/env bash +# vzdump-vms — image-level backup of hand-managed Proxmox VMs (NOT in Terraform). +# Deploy to PVE host at /usr/local/bin/vzdump-vms (strip the .sh). +# Schedule: Daily 01:00 via systemd timer. +# +# WHY: the hand-managed Linux VMs (devvm, …) have NO image backup. nfs-mirror / +# daily-backup / offsite-sync cover cluster PVCs, NFS, pfSense and PVE config — +# but never the VM disks themselves. A lost devvm disk = unrecoverable home dirs +# + local-only git repos (the monorepo root has no remote). This takes a live +# `vzdump --mode snapshot` of each configured VMID to /mnt/backup/vzdump (sda = +# Copy 2). The monthly offsite-sync full pass (days 1-7) mirrors /mnt/backup — +# including this dir — to Synology with --delete (Copy 3), bounded to local +# retention. We deliberately do NOT append to the incremental manifest: it never +# deletes, so daily multi-GB images would accumulate unbounded on Synology. +# +# RESTORE: pick a dump under /mnt/backup/vzdump, then on the PVE host: +# qmrestore /mnt/backup/vzdump/vzdump-qemu--.vma.zst +# (restore to a fresh VMID first if the original still exists, then swap), or use +# the PVE UI (Datacenter → Storage → upload dir → Restore). See backup-dr.md. +set -euo pipefail + +# systemd oneshot units get a minimal PATH (/usr/bin:/bin) — qm and vzdump live +# in /usr/sbin, so set an explicit PATH or the script silently can't find them. +export PATH="/usr/sbin:/usr/bin:/sbin:/bin:${PATH:-}" + +# --- Configuration --- +VMIDS="${VZDUMP_VMIDS:-102}" # space-separated. 102 = devvm. Add VMIDs here. +DUMPDIR="${VZDUMP_DUMPDIR:-/mnt/backup/vzdump}" +KEEP="${VZDUMP_KEEP:-3}" # retain N newest dumps per VMID on sda +COMPRESS="${VZDUMP_COMPRESS:-zstd}" +BACKUP_ROOT="/mnt/backup" +PUSHGATEWAY="${VZDUMP_PUSHGATEWAY:-http://10.0.20.100:30091}" +PUSHGATEWAY_JOB="vzdump-backup" +LOCKFILE="/run/vzdump-vms.lock" + +# --- Logging --- +log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"; } +warn() { log "WARN: $*" >&2; } + +# --- Metrics (always returns 0 so it never trips set -e) --- +push_metrics() { + local status="${1:-0}" bytes="${2:-0}" now + now=$(date +%s) + { + echo "vzdump_last_run_timestamp ${now}" + echo "vzdump_last_status ${status}" + echo "vzdump_last_bytes ${bytes}" + [ "${status}" -eq 0 ] && echo "vzdump_last_success_timestamp ${now}" + } | curl -s --connect-timeout 5 --max-time 10 --data-binary @- \ + "${PUSHGATEWAY}/metrics/job/${PUSHGATEWAY_JOB}" 2>/dev/null || true + return 0 +} + +# --- Locking (push a non-success metric if systemd kills us mid-run) --- +KILLED="" +cleanup() { + rm -f "${LOCKFILE}" + [ -n "${KILLED}" ] && push_metrics 2 0 +} +trap cleanup EXIT +trap 'KILLED=1; exit 143' TERM INT + +if ! ( set -o noclobber; echo $$ > "${LOCKFILE}" ) 2>/dev/null; then + warn "Another instance running (PID $(cat "${LOCKFILE}" 2>/dev/null || echo unknown)) — exiting" + exit 0 +fi + +# --- Preconditions --- +if ! mountpoint -q "${BACKUP_ROOT}"; then + warn "${BACKUP_ROOT} not mounted — aborting"; push_metrics 1 0; exit 1 +fi +mkdir -p "${DUMPDIR}" + +# --- Main --- +log "=== vzdump-vms starting (VMIDs: ${VMIDS}, keep ${KEEP}) ===" +STATUS=0 +TOTAL_BYTES=0 + +for vmid in ${VMIDS}; do + if ! qm status "${vmid}" >/dev/null 2>&1; then + warn "VMID ${vmid} not found on this node — skipping" + STATUS=1 + continue + fi + + log "--- vzdump ${vmid} ($(qm config "${vmid}" 2>/dev/null | sed -n 's/^name: //p')) ---" + if vzdump "${vmid}" \ + --dumpdir "${DUMPDIR}" \ + --mode snapshot \ + --compress "${COMPRESS}" \ + --ionice 7 \ + --quiet 1; then + newest=$(ls -t "${DUMPDIR}"/vzdump-qemu-"${vmid}"-*.vma.* 2>/dev/null | grep -v '\.notes$' | head -1 || true) + if [ -n "${newest}" ]; then + sz=$(stat -c%s "${newest}" 2>/dev/null || echo 0) + TOTAL_BYTES=$((TOTAL_BYTES + sz)) + log " OK: $(basename "${newest}") ($(numfmt --to=iec "${sz}" 2>/dev/null || echo "${sz}B"))" + fi + else + warn "vzdump ${vmid} failed (rc=$?)" + STATUS=1 + fi + + # Retention: keep newest ${KEEP} per VMID (archive + its .log + .notes siblings). + mapfile -t archives < <(ls -t "${DUMPDIR}"/vzdump-qemu-"${vmid}"-*.vma.* 2>/dev/null | grep -v '\.notes$' || true) + if [ "${#archives[@]}" -gt "${KEEP}" ]; then + for old in "${archives[@]:${KEEP}}"; do + prefix="${old%.vma.*}" # …/vzdump-qemu--- + log " prune: $(basename "${prefix}")" + rm -f "${prefix}".vma.* "${prefix}".log 2>/dev/null || true + done + fi +done + +log "=== vzdump-vms complete (status=${STATUS}, $(numfmt --to=iec "${TOTAL_BYTES}" 2>/dev/null || echo "${TOTAL_BYTES}B")) ===" +push_metrics "${STATUS}" "${TOTAL_BYTES}" +exit "${STATUS}" diff --git a/scripts/vzdump-vms.timer b/scripts/vzdump-vms.timer new file mode 100644 index 00000000..5cfefd92 --- /dev/null +++ b/scripts/vzdump-vms.timer @@ -0,0 +1,14 @@ +[Unit] +Description=Daily vzdump image backup of hand-managed VMs (devvm, …) +Documentation=https://forgejo.viktorbarzin.me/viktor/infra/src/branch/main/docs/architecture/backup-dr.md + +[Timer] +# 01:00 — ahead of nfs-mirror (02:00), lvm-pvc-snapshot (03:00), daily-backup +# (05:00) and offsite-sync (06:00), so the fresh image is on sda before the +# monthly full offsite pass mirrors /mnt/backup to Synology. +OnCalendar=*-*-* 01:00:00 +RandomizedDelaySec=10min +Persistent=true + +[Install] +WantedBy=timers.target From bc37b16815bad061e2b1911d4681cbdb4b8e1bef Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 9 Jun 2026 21:26:23 +0000 Subject: [PATCH 16/24] =?UTF-8?q?backup:=20fix=20vzdump-vms=20exit=20code?= =?UTF-8?q?=20=E2=80=94=20EXIT-trap=20&&=20short-circuit=20falsely=20faile?= =?UTF-8?q?d=20OK=20runs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First live run produced a valid 40G dump and logged status=0, but the service exited 1/FAILURE: cleanup() used `[ -n "$KILLED" ] && push_metrics 2 0`, and a bash EXIT trap whose LAST command returns non-zero overrides the script's `exit 0`. With KILLED empty the && short-circuits -> returns 1 -> a successful backup is marked failed (would trip a vzdump staleness/failure alert). Switch to daily-backup's `if…fi` idiom (returns 0 when not killed). Bug reproduced + fix verified locally; redeployed to PVE + reset-failed. [ci skip] Co-Authored-By: Claude Opus 4.8 --- scripts/vzdump-vms.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/vzdump-vms.sh b/scripts/vzdump-vms.sh index fabb45e7..8954d0e5 100644 --- a/scripts/vzdump-vms.sh +++ b/scripts/vzdump-vms.sh @@ -55,7 +55,10 @@ push_metrics() { KILLED="" cleanup() { rm -f "${LOCKFILE}" - [ -n "${KILLED}" ] && push_metrics 2 0 + # NB: must be `if…fi`, NOT `[ … ] && …` — a bash EXIT trap whose LAST command + # returns non-zero overrides the script's `exit 0`, so the `&&` short-circuit + # (when KILLED is empty) would falsely mark a successful backup as failed. + if [ -n "${KILLED}" ]; then push_metrics 2 0; fi } trap cleanup EXIT trap 'KILLED=1; exit 143' TERM INT From 5d9417fbaabe745a0258b4bd7f5ec49fd6e7f747 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Wed, 10 Jun 2026 09:30:41 +0000 Subject: [PATCH 17/24] workstation: emo contribute access + Phase-5 cutover done; gate master (push=apply) [ci skip] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-0004's premise was wrong: pushing master fires the Woodpecker apply pipeline (require_approval=forks only), so master pushes ARE deploys. Added Forgejo branch protection on master (push/merge whitelist=viktor, deploy keys allowed); non-admins contribute via branches + PRs. emo (ebarzin): write collaborator on viktor/infra, PAT in ~/.git-credentials, forgejo remote + upstream in his locked clone. Phase-5 finished: code-shared removed; ~/.claude symlinks kept (they ARE the skel shared-base mechanism — plan step 4c obsolete). Offboard runbook: revoke PAT + collaborator + group steps added. Co-Authored-By: Claude Fable 5 --- docs/architecture/multi-tenancy.md | 11 +++++++++-- .../2026-06-07-multi-user-workstation-design.md | 1 + .../2026-06-07-multi-user-workstation-plan.md | 2 ++ docs/runbooks/offboard-user.md | 16 +++++++++++++++- 4 files changed, 27 insertions(+), 3 deletions(-) diff --git a/docs/architecture/multi-tenancy.md b/docs/architecture/multi-tenancy.md index 2e66ae21..14744c89 100644 --- a/docs/architecture/multi-tenancy.md +++ b/docs/architecture/multi-tenancy.md @@ -543,9 +543,16 @@ Separate from the in-cluster namespace-owner model above, the **devvm** (`10.0.1 **Config inheritance (live):** wizard authors the base (his chezmoi-versioned `~/.claude`). Two native layers carry it to every user — the enforced org `claudeMd` in `/etc/claude-code/managed-settings.json` (top precedence, all sessions) and per-user `~/.claude/{skills,rules,…}` **symlinks** to the base (seeded via `/etc/skel`; edits propagate live). Secrets stay per-user at mode 600, never symlinked. -**Infra access:** non-admins get their own **writable, git-crypt-LOCKED** clone of the (public) infra repo at `~/code` — code/docs plaintext, secret files (`*.tfvars`, `secrets/**`) stay ciphertext. Changes are ungated (push ≠ apply); the real boundary is apply-time (`scripts/tg apply` needs an admin Vault token + cluster RBAC). +**Infra access:** non-admins get their own **writable, git-crypt-LOCKED** clone of the (public) infra repo at `~/code` — code/docs plaintext, secret files (`*.tfvars`, `secrets/**`) stay ciphertext. The provisioner clones anonymously from the public GitHub mirror; **contribute access is wired per-user on top** (see below). The apply boundary still holds (`scripts/tg apply` needs an admin Vault token + cluster RBAC), but **pushing `master` is NOT inert** — the Forgejo→Woodpecker webhook fires `.woodpecker/default.yml` (`event: push, branch: master`, `require_approval: forks` only), which terragrunt-applies changed stacks. `master` is therefore **branch-protected on Forgejo** (push + merge whitelists = `viktor`, deploy keys allowed): non-admins contribute via `/` branches + PRs, and only an admin merge lands (and thus applies) their change. -**Status (2026-06-08):** built + verified on the live host — capacity (8 GiB swap), config inheritance, roster-driven provisioner, per-user locked clone, **per-user OIDC kubeconfig + the `oidc-power-user-readonly` ClusterRole + emo's `k8s_users` entry (applied + impersonation-verified), and the Authentik `T3 Users` edge gate (applied + verified)**. **Remaining (held / future):** the emo cutover to his own locked clone (Phase 5), the offboarding apply-side (Phase 7), per-user MCP/auth injection, and roster-reconciled `T3 Users` membership. See `../runbooks/offboard-user.md` for deprovisioning. +**Contribute access (per non-admin, manual — the anca/tripit PAT precedent):** +1. Add their Forgejo user as a **write** collaborator on `viktor/infra` (`PUT /api/v1/repos/viktor/infra/collaborators/`). +2. Mint a PAT — the admin REST endpoint 404s here, use the in-pod CLI: `kubectl -n forgejo exec deploy/forgejo -- su -s /bin/sh git -c "forgejo admin user generate-access-token --username --token-name devvm-infra-git --scopes 'write:repository'"`. +3. Install it in their `~/.git-credentials` (`https://:@forgejo.viktorbarzin.me`, mode 600) + `git config --global credential.helper store`, set `user.name`/`user.email`. +4. In their clone: `git remote add forgejo https://forgejo.viktorbarzin.me/viktor/infra.git` and `git branch --set-upstream-to=forgejo/master master` (origin stays the anonymous GitHub mirror). +5. Verify: branch push succeeds; a push to `master` is rejected with `Not allowed to push to protected branch`. + +**Status (2026-06-10):** built + verified on the live host — capacity (8 GiB swap), config inheritance, roster-driven provisioner, per-user locked clone, per-user OIDC kubeconfig + the `oidc-power-user-readonly` ClusterRole + emo's `k8s_users` entry (applied + impersonation-verified), the Authentik `T3 Users` edge gate, **the emo Phase-5 cutover (own clone + launcher repoint + `code-shared` removal, completed 2026-06-10) and emo's contribute access (`ebarzin` write collaborator + PAT + protected `master`)**. Per the live `/etc/skel` design, non-admin `~/.claude/{rules,skills}` symlinks into the admin base are **kept** (they ARE the shared-base delivery mechanism — the plan's step to remove them is obsolete). **Remaining (held / future):** the offboarding apply-side (Phase 7), per-user MCP/auth injection, and roster-reconciled `T3 Users` membership. See `../runbooks/offboard-user.md` for deprovisioning. ## Related diff --git a/docs/plans/2026-06-07-multi-user-workstation-design.md b/docs/plans/2026-06-07-multi-user-workstation-design.md index 2148ae27..7267cc8c 100644 --- a/docs/plans/2026-06-07-multi-user-workstation-design.md +++ b/docs/plans/2026-06-07-multi-user-workstation-design.md @@ -166,6 +166,7 @@ Design principle: **every bit of devvm setup is an idempotent git script** — n - **ADR-0002 — devvm Linux users, not K8s ephemeral pods.** Re-platforming is overkill at this scale; config-push is easier on one host. - **ADR-0003 — Config inheritance via native machine-wide layers + per-user override.** Rejected: periodic sync, OverlayFS (no live lowerdir edits), Nix (rebuild not live). - **ADR-0004 — Infra access via per-user writable git-crypt-locked clones (changes ungated).** Each non-admin gets their own writable, keyless (locked) clone — read + edit + push freely, no PR gate. Safe because infra apply is manual + admin-only (push ≠ apply, id=4355) and the clone can't decrypt secrets. Rejected: the shared read-only mirror (gated changes) and the shared unlocked tree (secret leak + commit entanglement). Trade: repo-local CLAUDE.md updates via pull, not live (global config inheritance stays live via §4). + - **AMENDED 2026-06-10 — the "push ≠ apply" premise was WRONG.** The Forgejo→Woodpecker webhook on `viktor/infra` fires `.woodpecker/default.yml` on `push` to `master` (`require_approval: forks` only), which terragrunt-applies changed stacks — so an ungated master push IS a deploy. Enforcement added instead of dropping the ADR: Forgejo **branch protection on `master`** (push + merge whitelists = `viktor`, deploy keys allowed). Non-admins keep free branch pushes + PRs; only admin merges land on master. "No PR gate" is thereby reversed for non-admins; the rest of the ADR (per-user locked clones) stands. As-built: `../architecture/multi-tenancy.md` → "Contribute access". - **ADR-0005 — Power-user = cluster-wide read-only (no Secrets), via a NEW dedicated ClusterRole.** Re-widens cross-tenant READ for the trusted power-user tier only — but via a NEW `oidc-power-user-readonly` ClusterRole (get/list/watch, NO `secrets`), NOT the existing `oidc-power-user` (which grants read+write+Secrets and is unbound). Bound to the user's OIDC identity (kubelogin) — the apiserver accepts Authentik OIDC for the `kubernetes` audience; the dashboard's SA-token pattern is for the dashboard UI only. - **ADR-0006 — The roster is the single source of truth for the FULL lifecycle.** `roster.yaml` drives onboard *and* offboard; `/etc/ttyd-user-map`, `dispatch.json`, and Authentik `T3 Users` membership are *derived* from it, and tier is *validated* against `k8s_users` (fail-loud on mismatch). Rejected: hand-maintaining the four membership lists in parallel (guaranteed drift). Offboarding is first-class + staged (reversible cut → cluster revoke → gated `userdel`), not an afterthought. - **ADR-0007 — Add swap + a capacity budget to the devvm before onboarding active users.** A shared 24 GB / **0-swap** host OOM-kills live sessions under multi-user load (wizard alone runs ~20). Swap + a max-concurrent ceiling are prerequisites, not follow-ups. diff --git a/docs/plans/2026-06-07-multi-user-workstation-plan.md b/docs/plans/2026-06-07-multi-user-workstation-plan.md index 1bd3275c..f98580c7 100644 --- a/docs/plans/2026-06-07-multi-user-workstation-plan.md +++ b/docs/plans/2026-06-07-multi-user-workstation-plan.md @@ -171,6 +171,8 @@ users: ### Task 5.1: Cut emo over to his own writable locked clone (opt-in, reversible) +> **DONE 2026-06-10** (staged across 06-08 → 06-10), with two deviations: (1) step 4(c) **skipped deliberately** — the live `/etc/skel` shared base delivers `~/.claude/{rules,skills}` AS symlinks into the admin base, so emo's existing symlinks match the as-built design and were kept; (2) push access was **added** (not in this plan): `ebarzin` = write collaborator on Forgejo `viktor/infra` + PAT in `~/.git-credentials` + `forgejo` remote, with `master` branch-protected (see ADR-0004 amendment — push to master auto-applies via Woodpecker, so it is whitelist-gated to `viktor`). Verified: branch push OK, master push rejected, `code-shared` removed, admin tree unreadable as emo. + **Files:** none (host state; an explicit one-time action — NOT the routine reconcile) - [ ] **Step 1: Prereqs.** Confirm emo inherits config (Phase 1) + has his scoped kubeconfig (Phase 2). (Phase 3 deliberately SKIPPED emo — his clone is created *here*.) diff --git a/docs/runbooks/offboard-user.md b/docs/runbooks/offboard-user.md index 104f4fcd..269e2f1a 100644 --- a/docs/runbooks/offboard-user.md +++ b/docs/runbooks/offboard-user.md @@ -29,7 +29,21 @@ gated `userdel_archive`, which is **never** auto-applied). sudo systemctl disable --now t3-serve@.service sudo passwd -l ``` -4. **Verify:** they can no longer reach `t3.viktorbarzin.me` (302 → Authentik, then +4. **Revoke git + group access** *(manual)*: + ```bash + # legacy secret-bearing group, if they were ever in it + sudo gpasswd -d code-shared + # drop write access to the infra repo + curl -X DELETE -H "Authorization: token " \ + https://forgejo.viktorbarzin.me/api/v1/repos/viktor/infra/collaborators/ + # revoke their devvm git PAT (token name: devvm-infra-git; admin PAT may + # manage other users' tokens — verified 2026-06-10; the CLI has no delete) + curl -X DELETE -H "Authorization: token " \ + https://forgejo.viktorbarzin.me/api/v1/users//tokens/devvm-infra-git + ``` + Note: their already-running sessions keep dropped groups until cycled — restart + `t3-serve@` to enforce immediately. +5. **Verify:** they can no longer reach `t3.viktorbarzin.me` (302 → Authentik, then denied once removed from the `T3 Users` group — Part C) and cannot log in. Nothing is deleted; re-adding the roster entry + reconcile fully restores them. From 2e5af5dc0e5187324ce687ead3ee1f1a373afa8d Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Wed, 10 Jun 2026 09:41:38 +0000 Subject: [PATCH 18/24] workstation: keep non-admin infra clones fresh (hourly + at launch) [ci skip] Non-admins (emo) need current master without manual pulls. Two layers: - t3-provision-users reconcile gains refresh_locked_clone: fetch all remotes + ff-only master, guarded (on master, clean tree, upstream set); dirty/diverged clones are left alone with a WARN. - start-claude.sh freshens ~/code at session launch, 15s-capped so an offline remote never delays the session. Verified live on emo's clone: stale clone ff'd to tip by the reconciler; launcher snippet ff's when clean and refuses while a dirty file exists. Deployed to /usr/local/bin/t3-provision-users, /etc/skel/start-claude.sh, and emo's launcher. Co-Authored-By: Claude Fable 5 --- docs/architecture/multi-tenancy.md | 2 +- scripts/t3-provision-users.sh | 22 +++++++++++++++++++++- scripts/workstation/skel/start-claude.sh | 13 +++++++++++++ 3 files changed, 35 insertions(+), 2 deletions(-) diff --git a/docs/architecture/multi-tenancy.md b/docs/architecture/multi-tenancy.md index 14744c89..387611b7 100644 --- a/docs/architecture/multi-tenancy.md +++ b/docs/architecture/multi-tenancy.md @@ -543,7 +543,7 @@ Separate from the in-cluster namespace-owner model above, the **devvm** (`10.0.1 **Config inheritance (live):** wizard authors the base (his chezmoi-versioned `~/.claude`). Two native layers carry it to every user — the enforced org `claudeMd` in `/etc/claude-code/managed-settings.json` (top precedence, all sessions) and per-user `~/.claude/{skills,rules,…}` **symlinks** to the base (seeded via `/etc/skel`; edits propagate live). Secrets stay per-user at mode 600, never symlinked. -**Infra access:** non-admins get their own **writable, git-crypt-LOCKED** clone of the (public) infra repo at `~/code` — code/docs plaintext, secret files (`*.tfvars`, `secrets/**`) stay ciphertext. The provisioner clones anonymously from the public GitHub mirror; **contribute access is wired per-user on top** (see below). The apply boundary still holds (`scripts/tg apply` needs an admin Vault token + cluster RBAC), but **pushing `master` is NOT inert** — the Forgejo→Woodpecker webhook fires `.woodpecker/default.yml` (`event: push, branch: master`, `require_approval: forks` only), which terragrunt-applies changed stacks. `master` is therefore **branch-protected on Forgejo** (push + merge whitelists = `viktor`, deploy keys allowed): non-admins contribute via `/` branches + PRs, and only an admin merge lands (and thus applies) their change. +**Infra access:** non-admins get their own **writable, git-crypt-LOCKED** clone of the (public) infra repo at `~/code` — code/docs plaintext, secret files (`*.tfvars`, `secrets/**`) stay ciphertext. The provisioner clones anonymously from the public GitHub mirror; **contribute access is wired per-user on top** (see below). The apply boundary still holds (`scripts/tg apply` needs an admin Vault token + cluster RBAC), but **pushing `master` is NOT inert** — the Forgejo→Woodpecker webhook fires `.woodpecker/default.yml` (`event: push, branch: master`, `require_approval: forks` only), which terragrunt-applies changed stacks. `master` is therefore **branch-protected on Forgejo** (push + merge whitelists = `viktor`, deploy keys allowed): non-admins contribute via `/` branches + PRs, and only an admin merge lands (and thus applies) their change. **Clones stay fresh automatically** (2026-06-10): the hourly `t3-provision-users` reconcile runs `refresh_locked_clone` (fetch all remotes + fast-forward `master`, ONLY when on master with a clean tree and an upstream — dirty trees and local commits are left alone with a WARN), and `start-claude.sh` does the same freshen at session launch (15s-capped so an offline remote never stalls the session). **Contribute access (per non-admin, manual — the anca/tripit PAT precedent):** 1. Add their Forgejo user as a **write** collaborator on `viktor/infra` (`PUT /api/v1/repos/viktor/infra/collaborators/`). diff --git a/scripts/t3-provision-users.sh b/scripts/t3-provision-users.sh index 37689153..52a27015 100644 --- a/scripts/t3-provision-users.sh +++ b/scripts/t3-provision-users.sh @@ -45,6 +45,25 @@ install_locked_clone() { runuser -u "$user" -- git -C "$home/code" checkout --quiet master } +# Keep an EXISTING non-admin clone fresh (the admin's tree is never touched): fetch +# all remotes, then fast-forward master only when that is provably safe — on master, +# clean tree, upstream configured. Never rebases/merges; a non-ff master (local +# commits) is the user's to reconcile and is only WARNed about. Fetch failures +# (offline, missing credentials) are non-fatal: freshness is best-effort. +refresh_locked_clone() { + local user="$1" home + home="$(getent passwd "$user" | cut -d: -f6)" + [[ -n "$home" && -d "$home/code/.git" ]] || return 0 + if [[ "$DRY_RUN" == 1 ]]; then echo "[dry-run] refresh clone -> $user:$home/code"; return 0; fi + runuser -u "$user" -- env GIT_TERMINAL_PROMPT=0 git -C "$home/code" fetch --all --prune --quiet 2>/dev/null \ + || { log "WARN: clone fetch failed for $user (offline/credentials?) — skipped"; return 0; } + [[ "$(runuser -u "$user" -- git -C "$home/code" symbolic-ref --short -q HEAD)" == master ]] || return 0 + [[ -z "$(runuser -u "$user" -- git -C "$home/code" status --porcelain)" ]] || return 0 + runuser -u "$user" -- git -C "$home/code" rev-parse --verify -q 'master@{upstream}' >/dev/null || return 0 + runuser -u "$user" -- git -C "$home/code" merge --ff-only 'master@{upstream}' >/dev/null 2>&1 \ + || log "WARN: $user master not fast-forwardable (local commits?) — left as-is" +} + # Per-user OIDC kubeconfig (kubelogin/PKCE — the `kubernetes` Authentik client is # public, no secret). Identical for all users: identity comes from each user's own # interactive OIDC login, which the apiserver maps (email claim) to their RBAC. @@ -177,8 +196,9 @@ while IFS=$'\t' read -r os_user tier shell groups_csv; do log "add $os_user -> group $g"; run gpasswd -a "$os_user" "$g" >/dev/null done fi - if [[ "$tier" != admin ]]; then # non-admins: locked clone + kubeconfig + shared Claude token + if [[ "$tier" != admin ]]; then # non-admins: locked clone (kept fresh) + kubeconfig + shared Claude token install_locked_clone "$os_user" + refresh_locked_clone "$os_user" install_user_kubeconfig "$os_user" install_user_claude_token "$os_user" fi diff --git a/scripts/workstation/skel/start-claude.sh b/scripts/workstation/skel/start-claude.sh index fa21aa36..1a630366 100755 --- a/scripts/workstation/skel/start-claude.sh +++ b/scripts/workstation/skel/start-claude.sh @@ -19,6 +19,19 @@ fi cd "$HOME/code" 2>/dev/null || cd "$HOME" +# Freshen ~/code at session start so the user begins on current upstream state +# (the hourly t3-provision-users reconcile does the same in the background). +# Fast-forward only, and only when safe (on master + clean tree); hard 15s cap so +# an offline remote never stalls the launch. No-op for repos without remotes. +if [ -d "$HOME/code/.git" ]; then + GIT_TERMINAL_PROMPT=0 timeout 15 git -C "$HOME/code" fetch --all --prune --quiet 2>/dev/null || true + if [ "$(git -C "$HOME/code" symbolic-ref --short -q HEAD)" = master ] \ + && [ -z "$(git -C "$HOME/code" status --porcelain 2>/dev/null)" ] \ + && git -C "$HOME/code" rev-parse --verify -q 'master@{upstream}' >/dev/null 2>&1; then + git -C "$HOME/code" merge --ff-only 'master@{upstream}' >/dev/null 2>&1 || true + fi +fi + # Prefer the system-wide `claude` (installed by setup-devvm.sh); fall back to npx. launch() { if command -v claude >/dev/null 2>&1; then From 6d8773573c1878fad199ef8a44ca2c5c95dd0ca9 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Wed, 10 Jun 2026 10:12:26 +0000 Subject: [PATCH 19/24] workstation: agent-driven contribute flow for non-technical users [ci skip] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit emo can't use git — his agent must do all VCS mechanics invisibly. Managed claudeMd (every session, top precedence) now instructs agents: commit -> push / branch -> open PR via Forgejo API (user's PAT from ~/.git-credentials) -> back to clean master -> tell the user in plain words it's submitted for review. AGENTS.md carries the full recipe with the curl call. Verified live as emo: PR #1 opened (HTTP 201, write:repository scope suffices) and closed via his PAT. Deployed to /etc/claude-code/managed-settings.json; codex AGENTS.md mirrors for emo + ancamilea regenerated from the new claudeMd. Co-Authored-By: Claude Fable 5 --- AGENTS.md | 27 +++++++++++++++++++++++ scripts/workstation/managed-settings.json | 2 +- 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 984e98d6..585b3da7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -226,6 +226,33 @@ Per-workload opt-out: add the label `keel.sh/policy: never` on the Deployment me 4. Viktor reviews → CI applies → Slack notification 5. Portal: `https://k8s-portal.viktorbarzin.me/onboarding` for full guide +### Non-admin workstation users — the AGENT does the git work + +Non-admin devvm users (power-user / namespace-owner tiers) may not know git at +all. Their agent handles every version-control step silently — never ask them +to commit, push, pull, or open a PR, and never surface git jargon at them. +Their `~/code` clone arrives preconfigured: git identity, a `forgejo` remote +authenticated via `~/.git-credentials`, and `master` tracking `forgejo/master` +(auto-freshened hourly and at session launch, fast-forward only). + +To land a finished change from such a clone: + +1. `git checkout -b / master` — always branch off fresh master +2. Commit with a clear message (identity is preconfigured) +3. `git push forgejo /` +4. Open the PR with the user's own PAT (`write:repository` suffices — verified 2026-06-10): + ```bash + TOK=$(sed -E 's#https://[^:]+:([^@]+)@.*#\1#' ~/.git-credentials) + curl -X POST -H "Authorization: token $TOK" -H 'Content-Type: application/json' \ + https://forgejo.viktorbarzin.me/api/v1/repos/viktor/infra/pulls \ + -d '{"title":"","head":"<os-user>/<short-topic>","base":"master","body":"<what + why>"}' + ``` +5. `git checkout master` — leave the clone clean so auto-refresh keeps working +6. Tell the user in plain language that the change is submitted for Viktor's review + +Direct pushes to `master` are rejected by branch protection; merging (and the +CI apply a master push triggers) is admin-only. + ## Common Operations - **Deploy new service**: Use `stacks/<existing-service>/` as template. Create stack, add DNS in tfvars, apply platform then service. - **Fix crashed pods**: Run healthcheck first. Safe to delete evicted/failed pods and CrashLoopBackOff pods with >10 restarts. diff --git a/scripts/workstation/managed-settings.json b/scripts/workstation/managed-settings.json index 224c49bd..77bfdbab 100644 --- a/scripts/workstation/managed-settings.json +++ b/scripts/workstation/managed-settings.json @@ -1,4 +1,4 @@ { - "claudeMd": "# Viktor Barzin homelab — shared multi-user Claude Code Workstation (devvm)\n\nYou are running as a specific OS user on a SHARED devvm Workstation, not as the admin. These org-wide rules apply to EVERY user and sit at the top of settings precedence (they cannot be overridden by a user's own config):\n\n- Respect your permission tier. Your kubectl, Vault, and infra access are scoped to your RBAC tier (admin / power-user / namespace-owner). Do not attempt to escalate privileges or reach another user's resources.\n- Secrets are per-user. Never read another user's home directory, credentials, tokens, or ~/.claude secrets. Your own secrets live in your home at mode 600.\n- Infrastructure changes go through Terraform/Terragrunt (scripts/tg apply) — never direct kubectl apply/edit/patch. Pushing to git does NOT deploy; applies are manual and admin-gated, so your edits cannot take effect without an admin apply.\n- Follow the engineering rules in ~/.claude/rules/ (execution, planning, quality) and every CLAUDE.md in the repo tree.\n- The monorepo is at ~/code. Non-admins get a git-crypt-LOCKED clone: secret files read as ciphertext — that is expected, not an error.", + "claudeMd": "# Viktor Barzin homelab — shared multi-user Claude Code Workstation (devvm)\n\nYou are running as a specific OS user on a SHARED devvm Workstation, not as the admin. These org-wide rules apply to EVERY user and sit at the top of settings precedence (they cannot be overridden by a user's own config):\n\n- Respect your permission tier. Your kubectl, Vault, and infra access are scoped to your RBAC tier (admin / power-user / namespace-owner). Do not attempt to escalate privileges or reach another user's resources.\n- Secrets are per-user. Never read another user's home directory, credentials, tokens, or ~/.claude secrets. Your own secrets live in your home at mode 600.\n- Infrastructure changes go through Terraform/Terragrunt (scripts/tg apply) — never direct kubectl apply/edit/patch. Applies are manual and admin-gated: a non-admin's edits cannot take effect without an admin merge + apply.\n- Non-admins land changes via branch + PR, and the AGENT does ALL git/PR mechanics silently — the user may not know git, so never ask them to commit, push, pull, or open anything. When you finish a change in ~/code: commit it, push to a branch named <os-user>/<short-topic> on the forgejo remote, open a PR to master via the Forgejo API (token = the password field in ~/.git-credentials), then check out clean master again so background auto-refresh keeps working. Tell the user in plain words that the change is submitted for Viktor's review. Direct pushes to master are rejected (branch protection). Full recipe: AGENTS.md → 'Non-admin workstation users' in ~/code.\n- Follow the engineering rules in ~/.claude/rules/ (execution, planning, quality) and every CLAUDE.md in the repo tree.\n- The monorepo is at ~/code. Non-admins get a git-crypt-LOCKED clone: secret files read as ciphertext — that is expected, not an error.", "model": "claude-fable-5" } From a49d1eadf6aa525f6c4b6dc8b4b36fb7eedeeb40 Mon Sep 17 00:00:00 2001 From: Viktor Barzin <vbarzin@gmail.com> Date: Wed, 10 Jun 2026 14:53:43 +0000 Subject: [PATCH 20/24] =?UTF-8?q?workstation:=20emo=20direct=20master=20pu?= =?UTF-8?q?sh=20=E2=80=94=20allow-then-audit=20[ci=20skip]?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Viktor: emo may make any change; what matters is tracking what changed and why. ebarzin added to master push+merge whitelists (force-push stays disabled — append-only history). Tracking enforced three ways: - agent instructions (managed claudeMd + AGENTS.md): commit body MUST carry the user's plain-language intent; commits land on master directly; [ci skip] forbidden for non-admins - new notify-nonadmin-push step in .woodpecker/default.yml: Slack message for every non-admin master push (admin pushes silent) - PR flow remains the fallback for non-whitelisted users Accepted consequence (informed): emo's pushes auto-apply changed stacks via CI. Offboard runbook gains whitelist-removal step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --- .woodpecker/default.yml | 20 +++++++++ AGENTS.md | 44 +++++++++++++------ docs/architecture/multi-tenancy.md | 5 ++- ...026-06-07-multi-user-workstation-design.md | 1 + docs/runbooks/offboard-user.md | 5 +++ scripts/workstation/managed-settings.json | 2 +- 6 files changed, 60 insertions(+), 17 deletions(-) diff --git a/.woodpecker/default.yml b/.woodpecker/default.yml index 5661bccd..0df7500e 100644 --- a/.woodpecker/default.yml +++ b/.woodpecker/default.yml @@ -24,6 +24,26 @@ clone: backoff: 10s steps: + # Audit feed for the allow-then-audit contribution model: any master push by + # a NON-admin author is surfaced in Slack (Viktor's own pushes are not). + # Runs before apply and never blocks it. Note: [ci skip] commits never reach + # this step (Woodpecker skips the whole pipeline) — hence the rule that + # non-admins must not use [ci skip]. + - name: notify-nonadmin-push + image: curlimages/curl + environment: + SLACK_WEBHOOK: + from_secret: slack_webhook + commands: + - | + case "$CI_COMMIT_AUTHOR" in + viktor|ViktorBarzin|wizard) echo "admin push — no notify"; exit 0 ;; + esac + SUBJECT=$(echo "$CI_COMMIT_MESSAGE" | head -1 | tr -d '"\\') + curl -s -X POST -H 'Content-type: application/json' \ + --data "{\"text\":\"📝 infra master push by *$CI_COMMIT_AUTHOR*: $SUBJECT\n$CI_REPO_URL/commit/$CI_COMMIT_SHA\"}" \ + "$SLACK_WEBHOOK" || true + - name: apply image: forgejo.viktorbarzin.me/viktor/infra-ci:latest pull: true diff --git a/AGENTS.md b/AGENTS.md index 585b3da7..7559d276 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -235,23 +235,39 @@ Their `~/code` clone arrives preconfigured: git identity, a `forgejo` remote authenticated via `~/.git-credentials`, and `master` tracking `forgejo/master` (auto-freshened hourly and at session launch, fast-forward only). +The model is **allow-then-audit** (Viktor, 2026-06-10): whitelisted users (emo) +push straight to `master` — no PR gate — and the record of *what changed and +why* is what matters. Force-push is disabled for everyone, so master history +is append-only. + To land a finished change from such a clone: -1. `git checkout -b <os-user>/<short-topic> master` — always branch off fresh master -2. Commit with a clear message (identity is preconfigured) -3. `git push forgejo <os-user>/<short-topic>` -4. Open the PR with the user's own PAT (`write:repository` suffices — verified 2026-06-10): - ```bash - TOK=$(sed -E 's#https://[^:]+:([^@]+)@.*#\1#' ~/.git-credentials) - curl -X POST -H "Authorization: token $TOK" -H 'Content-Type: application/json' \ - https://forgejo.viktorbarzin.me/api/v1/repos/viktor/infra/pulls \ - -d '{"title":"<title>","head":"<os-user>/<short-topic>","base":"master","body":"<what + why>"}' - ``` -5. `git checkout master` — leave the clone clean so auto-refresh keeps working -6. Tell the user in plain language that the change is submitted for Viktor's review +1. Commit on `master`. **The commit message is the audit trail** — this matters + more than the change itself: + - subject: what changed, specific ("ha-sofia: lower fan curve bias to -5") + - body: WHY, in plain words — paraphrase the user's actual request and any + reasoning ("Emil asked for quieter fans in the evening; curve was + overshooting after the 2026-06-08 redesign") +2. `git push forgejo master`. If rejected non-fast-forward: `git pull --rebase + forgejo master` and push again. +3. **Never use `[ci skip]`** as a non-admin — it hides the change from the + Slack audit feed; a no-op CI apply on a docs-only commit is harmless. +4. Leave the clone on clean `master` so auto-refresh keeps working. +5. Tell the user in plain language what happened. Stack changes are + auto-applied by CI — verify the live result with the user's read-only + kubectl before saying "it's live". -Direct pushes to `master` are rejected by branch protection; merging (and the -CI apply a master push triggers) is admin-only. +If a push to `master` is rejected by branch protection (user not on the +whitelist — e.g. new users before Viktor grants it), fall back to a +`<os-user>/<short-topic>` branch + PR with the user's own PAT +(`write:repository` suffices — verified 2026-06-10): + +```bash +TOK=$(sed -E 's#https://[^:]+:([^@]+)@.*#\1#' ~/.git-credentials) +curl -X POST -H "Authorization: token $TOK" -H 'Content-Type: application/json' \ + https://forgejo.viktorbarzin.me/api/v1/repos/viktor/infra/pulls \ + -d '{"title":"<title>","head":"<os-user>/<short-topic>","base":"master","body":"<what + why>"}' +``` ## Common Operations - **Deploy new service**: Use `stacks/<existing-service>/` as template. Create stack, add DNS in tfvars, apply platform then service. diff --git a/docs/architecture/multi-tenancy.md b/docs/architecture/multi-tenancy.md index 387611b7..067cdcaf 100644 --- a/docs/architecture/multi-tenancy.md +++ b/docs/architecture/multi-tenancy.md @@ -543,14 +543,15 @@ Separate from the in-cluster namespace-owner model above, the **devvm** (`10.0.1 **Config inheritance (live):** wizard authors the base (his chezmoi-versioned `~/.claude`). Two native layers carry it to every user — the enforced org `claudeMd` in `/etc/claude-code/managed-settings.json` (top precedence, all sessions) and per-user `~/.claude/{skills,rules,…}` **symlinks** to the base (seeded via `/etc/skel`; edits propagate live). Secrets stay per-user at mode 600, never symlinked. -**Infra access:** non-admins get their own **writable, git-crypt-LOCKED** clone of the (public) infra repo at `~/code` — code/docs plaintext, secret files (`*.tfvars`, `secrets/**`) stay ciphertext. The provisioner clones anonymously from the public GitHub mirror; **contribute access is wired per-user on top** (see below). The apply boundary still holds (`scripts/tg apply` needs an admin Vault token + cluster RBAC), but **pushing `master` is NOT inert** — the Forgejo→Woodpecker webhook fires `.woodpecker/default.yml` (`event: push, branch: master`, `require_approval: forks` only), which terragrunt-applies changed stacks. `master` is therefore **branch-protected on Forgejo** (push + merge whitelists = `viktor`, deploy keys allowed): non-admins contribute via `<user>/<topic>` branches + PRs, and only an admin merge lands (and thus applies) their change. **Clones stay fresh automatically** (2026-06-10): the hourly `t3-provision-users` reconcile runs `refresh_locked_clone` (fetch all remotes + fast-forward `master`, ONLY when on master with a clean tree and an upstream — dirty trees and local commits are left alone with a WARN), and `start-claude.sh` does the same freshen at session launch (15s-capped so an offline remote never stalls the session). +**Infra access:** non-admins get their own **writable, git-crypt-LOCKED** clone of the (public) infra repo at `~/code` — code/docs plaintext, secret files (`*.tfvars`, `secrets/**`) stay ciphertext. The provisioner clones anonymously from the public GitHub mirror; **contribute access is wired per-user on top** (see below). The apply boundary still holds (`scripts/tg apply` needs an admin Vault token + cluster RBAC), but **pushing `master` is NOT inert** — the Forgejo→Woodpecker webhook fires `.woodpecker/default.yml` (`event: push, branch: master`, `require_approval: forks` only), which terragrunt-applies changed stacks. `master` is **branch-protected on Forgejo** (force-push disabled for everyone — history is append-only; push + merge whitelists = `viktor` + explicitly granted users, deploy keys allowed). **Allow-then-audit (Viktor, 2026-06-10):** `ebarzin` (emo) is on the whitelist and pushes straight to `master` — no PR gate. The tracking burden moves to: (a) **commit messages that record what + why** (the agent instructions in AGENTS.md and the managed claudeMd require the body to paraphrase the user's request), (b) the **`notify-nonadmin-push` Slack audit step** in `.woodpecker/default.yml` — every master push by a non-admin author is posted to Slack (admin pushes are not), and (c) non-admins **never use `[ci skip]`** so every change fires the pipeline (and thus the audit feed). Users NOT on the whitelist fall back to `<user>/<topic>` branches + PRs. **Clones stay fresh automatically** (2026-06-10): the hourly `t3-provision-users` reconcile runs `refresh_locked_clone` (fetch all remotes + fast-forward `master`, ONLY when on master with a clean tree and an upstream — dirty trees and local commits are left alone with a WARN), and `start-claude.sh` does the same freshen at session launch (15s-capped so an offline remote never stalls the session). **Contribute access (per non-admin, manual — the anca/tripit PAT precedent):** 1. Add their Forgejo user as a **write** collaborator on `viktor/infra` (`PUT /api/v1/repos/viktor/infra/collaborators/<login>`). 2. Mint a PAT — the admin REST endpoint 404s here, use the in-pod CLI: `kubectl -n forgejo exec deploy/forgejo -- su -s /bin/sh git -c "forgejo admin user generate-access-token --username <login> --token-name devvm-infra-git --scopes 'write:repository'"`. 3. Install it in their `~/.git-credentials` (`https://<login>:<token>@forgejo.viktorbarzin.me`, mode 600) + `git config --global credential.helper store`, set `user.name`/`user.email`. 4. In their clone: `git remote add forgejo https://forgejo.viktorbarzin.me/viktor/infra.git` and `git branch --set-upstream-to=forgejo/master master` (origin stays the anonymous GitHub mirror). -5. Verify: branch push succeeds; a push to `master` is rejected with `Not allowed to push to protected branch`. +5. (Optional — Viktor's call per user) Grant direct master push: add their login to the `master` branch-protection push + merge whitelists (`PATCH /api/v1/repos/viktor/infra/branch_protections/master`). Done for `ebarzin` 2026-06-10. +6. Verify: branch push succeeds; a `master` push succeeds for whitelisted users and is rejected with `Not allowed to push to protected branch` otherwise. **Status (2026-06-10):** built + verified on the live host — capacity (8 GiB swap), config inheritance, roster-driven provisioner, per-user locked clone, per-user OIDC kubeconfig + the `oidc-power-user-readonly` ClusterRole + emo's `k8s_users` entry (applied + impersonation-verified), the Authentik `T3 Users` edge gate, **the emo Phase-5 cutover (own clone + launcher repoint + `code-shared` removal, completed 2026-06-10) and emo's contribute access (`ebarzin` write collaborator + PAT + protected `master`)**. Per the live `/etc/skel` design, non-admin `~/.claude/{rules,skills}` symlinks into the admin base are **kept** (they ARE the shared-base delivery mechanism — the plan's step to remove them is obsolete). **Remaining (held / future):** the offboarding apply-side (Phase 7), per-user MCP/auth injection, and roster-reconciled `T3 Users` membership. See `../runbooks/offboard-user.md` for deprovisioning. diff --git a/docs/plans/2026-06-07-multi-user-workstation-design.md b/docs/plans/2026-06-07-multi-user-workstation-design.md index 7267cc8c..8e54fa95 100644 --- a/docs/plans/2026-06-07-multi-user-workstation-design.md +++ b/docs/plans/2026-06-07-multi-user-workstation-design.md @@ -167,6 +167,7 @@ Design principle: **every bit of devvm setup is an idempotent git script** — n - **ADR-0003 — Config inheritance via native machine-wide layers + per-user override.** Rejected: periodic sync, OverlayFS (no live lowerdir edits), Nix (rebuild not live). - **ADR-0004 — Infra access via per-user writable git-crypt-locked clones (changes ungated).** Each non-admin gets their own writable, keyless (locked) clone — read + edit + push freely, no PR gate. Safe because infra apply is manual + admin-only (push ≠ apply, id=4355) and the clone can't decrypt secrets. Rejected: the shared read-only mirror (gated changes) and the shared unlocked tree (secret leak + commit entanglement). Trade: repo-local CLAUDE.md updates via pull, not live (global config inheritance stays live via §4). - **AMENDED 2026-06-10 — the "push ≠ apply" premise was WRONG.** The Forgejo→Woodpecker webhook on `viktor/infra` fires `.woodpecker/default.yml` on `push` to `master` (`require_approval: forks` only), which terragrunt-applies changed stacks — so an ungated master push IS a deploy. Enforcement added instead of dropping the ADR: Forgejo **branch protection on `master`** (push + merge whitelists = `viktor`, deploy keys allowed). Non-admins keep free branch pushes + PRs; only admin merges land on master. "No PR gate" is thereby reversed for non-admins; the rest of the ADR (per-user locked clones) stands. As-built: `../architecture/multi-tenancy.md` → "Contribute access". + - **AMENDED AGAIN 2026-06-10 (later) — allow-then-audit.** Viktor granted emo (`ebarzin`) direct master push ("he's allowed to make any change; what matters is tracking what changed and why"). The PR gate is dropped FOR WHITELISTED USERS; tracking is enforced instead: agent-written commit messages must carry the user's plain-language intent (the WHY), a `notify-nonadmin-push` Slack step in `.woodpecker/default.yml` surfaces every non-admin master push, `[ci skip]` is forbidden for non-admins, and force-push stays disabled (append-only history). Accepted consequence: emo's pushes auto-apply changed stacks via CI. Branch protection + the PR fallback remain for non-whitelisted users. - **ADR-0005 — Power-user = cluster-wide read-only (no Secrets), via a NEW dedicated ClusterRole.** Re-widens cross-tenant READ for the trusted power-user tier only — but via a NEW `oidc-power-user-readonly` ClusterRole (get/list/watch, NO `secrets`), NOT the existing `oidc-power-user` (which grants read+write+Secrets and is unbound). Bound to the user's OIDC identity (kubelogin) — the apiserver accepts Authentik OIDC for the `kubernetes` audience; the dashboard's SA-token pattern is for the dashboard UI only. - **ADR-0006 — The roster is the single source of truth for the FULL lifecycle.** `roster.yaml` drives onboard *and* offboard; `/etc/ttyd-user-map`, `dispatch.json`, and Authentik `T3 Users` membership are *derived* from it, and tier is *validated* against `k8s_users` (fail-loud on mismatch). Rejected: hand-maintaining the four membership lists in parallel (guaranteed drift). Offboarding is first-class + staged (reversible cut → cluster revoke → gated `userdel`), not an afterthought. - **ADR-0007 — Add swap + a capacity budget to the devvm before onboarding active users.** A shared 24 GB / **0-swap** host OOM-kills live sessions under multi-user load (wizard alone runs ~20). Swap + a max-concurrent ceiling are prerequisites, not follow-ups. diff --git a/docs/runbooks/offboard-user.md b/docs/runbooks/offboard-user.md index 269e2f1a..05c0c5a8 100644 --- a/docs/runbooks/offboard-user.md +++ b/docs/runbooks/offboard-user.md @@ -36,6 +36,11 @@ gated `userdel_archive`, which is **never** auto-applied). # drop write access to the infra repo curl -X DELETE -H "Authorization: token <admin_pat>" \ https://forgejo.viktorbarzin.me/api/v1/repos/viktor/infra/collaborators/<forgejo_login> + # if they were whitelisted for direct master push, remove them from the + # branch-protection whitelists (PATCH with the remaining usernames) + curl -X PATCH -H "Authorization: token <admin_pat>" -H 'Content-Type: application/json' \ + https://forgejo.viktorbarzin.me/api/v1/repos/viktor/infra/branch_protections/master \ + -d '{"push_whitelist_usernames":["viktor"],"merge_whitelist_usernames":["viktor"]}' # revoke their devvm git PAT (token name: devvm-infra-git; admin PAT may # manage other users' tokens — verified 2026-06-10; the CLI has no delete) curl -X DELETE -H "Authorization: token <admin_pat>" \ diff --git a/scripts/workstation/managed-settings.json b/scripts/workstation/managed-settings.json index 77bfdbab..d9a7ccaf 100644 --- a/scripts/workstation/managed-settings.json +++ b/scripts/workstation/managed-settings.json @@ -1,4 +1,4 @@ { - "claudeMd": "# Viktor Barzin homelab — shared multi-user Claude Code Workstation (devvm)\n\nYou are running as a specific OS user on a SHARED devvm Workstation, not as the admin. These org-wide rules apply to EVERY user and sit at the top of settings precedence (they cannot be overridden by a user's own config):\n\n- Respect your permission tier. Your kubectl, Vault, and infra access are scoped to your RBAC tier (admin / power-user / namespace-owner). Do not attempt to escalate privileges or reach another user's resources.\n- Secrets are per-user. Never read another user's home directory, credentials, tokens, or ~/.claude secrets. Your own secrets live in your home at mode 600.\n- Infrastructure changes go through Terraform/Terragrunt (scripts/tg apply) — never direct kubectl apply/edit/patch. Applies are manual and admin-gated: a non-admin's edits cannot take effect without an admin merge + apply.\n- Non-admins land changes via branch + PR, and the AGENT does ALL git/PR mechanics silently — the user may not know git, so never ask them to commit, push, pull, or open anything. When you finish a change in ~/code: commit it, push to a branch named <os-user>/<short-topic> on the forgejo remote, open a PR to master via the Forgejo API (token = the password field in ~/.git-credentials), then check out clean master again so background auto-refresh keeps working. Tell the user in plain words that the change is submitted for Viktor's review. Direct pushes to master are rejected (branch protection). Full recipe: AGENTS.md → 'Non-admin workstation users' in ~/code.\n- Follow the engineering rules in ~/.claude/rules/ (execution, planning, quality) and every CLAUDE.md in the repo tree.\n- The monorepo is at ~/code. Non-admins get a git-crypt-LOCKED clone: secret files read as ciphertext — that is expected, not an error.", + "claudeMd": "# Viktor Barzin homelab — shared multi-user Claude Code Workstation (devvm)\n\nYou are running as a specific OS user on a SHARED devvm Workstation, not as the admin. These org-wide rules apply to EVERY user and sit at the top of settings precedence (they cannot be overridden by a user's own config):\n\n- Respect your permission tier. Your kubectl, Vault, and infra access are scoped to your RBAC tier (admin / power-user / namespace-owner). Do not attempt to escalate privileges or reach another user's resources.\n- Secrets are per-user. Never read another user's home directory, credentials, tokens, or ~/.claude secrets. Your own secrets live in your home at mode 600.\n- Infrastructure changes go through Terraform/Terragrunt — never direct kubectl apply/edit/patch. Committed stack changes are auto-applied by CI on push to master; you can verify the live result with your read-only kubectl.\n- The AGENT does ALL git mechanics silently — the user may not know git, so never ask them to commit, push, pull, or open anything, and never surface git jargon. When you finish a change in ~/code: commit it ON master and push to the forgejo remote. THE COMMIT MESSAGE IS THE AUDIT TRAIL — subject says WHAT changed; body says WHY in plain words (paraphrase the user's actual request) — this matters more than the change itself. Never use [ci skip] as a non-admin (it would hide the change from the audit feed; harmless no-op applies are fine). If the push is rejected non-fast-forward, git pull --rebase forgejo master and push again. If it is rejected by branch protection (user not whitelisted), fall back to a <os-user>/<topic> branch + PR via the Forgejo API (token = password field in ~/.git-credentials). Keep ~/code on a clean master when done so background auto-refresh keeps working. Tell the user in plain words what happened ('done — your change is live/recorded'). Full recipe: AGENTS.md → 'Non-admin workstation users' in ~/code.\n- Follow the engineering rules in ~/.claude/rules/ (execution, planning, quality) and every CLAUDE.md in the repo tree.\n- The monorepo is at ~/code. Non-admins get a git-crypt-LOCKED clone: secret files read as ciphertext — that is expected, not an error.", "model": "claude-fable-5" } From 0f45585b5304a8d3b5afe4198066f4fb01c64c98 Mon Sep 17 00:00:00 2001 From: Emil Barzin <emil.barzin@gmail.com> Date: Wed, 10 Jun 2026 14:54:04 +0000 Subject: [PATCH 21/24] test: verify emo direct master push (allow-then-audit) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Viktor granted emo direct push to master on 2026-06-10 — any change allowed, tracked via commit messages + the Slack audit feed. This empty commit verifies the whitelist and exercises the new notify-nonadmin-push CI step end-to-end. From 619b7608fac360f146292acde9637ac29515dba5 Mon Sep 17 00:00:00 2001 From: Emil Barzin <emil.barzin@gmail.com> Date: Wed, 10 Jun 2026 15:03:48 +0000 Subject: [PATCH 22/24] test: verify audit pipeline fires on emo push Second verification: the Forgejo->Woodpecker webhook was timing out on the public-IP hairpin (first test push fired no pipeline), so it now targets the in-cluster Woodpecker service. This push should produce a pipeline with the notify-nonadmin-push Slack step. From 63161ef3a537bdae62b45f63cec425d209d25de3 Mon Sep 17 00:00:00 2001 From: Emil Barzin <emil.barzin@gmail.com> Date: Wed, 10 Jun 2026 15:07:15 +0000 Subject: [PATCH 23/24] test: final audit-pipeline verification Repo-82 Woodpecker secrets were missing (repo-1 set cloned over) and the webhook now targets the in-cluster service. This push should run the full pipeline: Slack audit ping + no-op apply. From a34f9ff3b8f7dd2a7788fae611fdf33591cfae8c Mon Sep 17 00:00:00 2001 From: Viktor Barzin <vbarzin@gmail.com> Date: Wed, 10 Jun 2026 15:09:17 +0000 Subject: [PATCH 24/24] =?UTF-8?q?docs:=20infra=20Woodpecker=20repo-82=20op?= =?UTF-8?q?s=20=E2=80=94=20in-cluster=20webhook,=20secret=20parity,=20empt?= =?UTF-8?q?y-commit=20gotcha=20[ci=20skip]?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Emo's first direct pushes surfaced three latent CI issues, all fixed out-of-band today and recorded here: webhook deliveries to ci.viktorbarzin.me timing out on the public-IP hairpin (hook now targets the in-cluster woodpecker-server service), repo 82 registered without the repo-scoped secret set (cloned from repo 1 in the DB), and empty commits compiling every workflow so missing secrets hard-error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --- docs/architecture/ci-cd.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/architecture/ci-cd.md b/docs/architecture/ci-cd.md index 8a5990b6..e44df43d 100644 --- a/docs/architecture/ci-cd.md +++ b/docs/architecture/ci-cd.md @@ -197,6 +197,34 @@ steps: - Keeps Woodpecker global secrets in sync with Vault - Runs in `woodpecker` namespace +## Infra repo CI (Woodpecker repo 82 — Forgejo forge) + +The infra repo itself runs on Woodpecker via the **Forgejo** forge (repo id 82, +registered 2026-06-08; the GitHub-side repo id 1 also remains registered). +Pushes to `master` fire `.woodpecker/default.yml` (changed-stacks terragrunt +apply) plus the `notify-nonadmin-push` Slack audit step (allow-then-audit +contribution model — see `multi-tenancy.md`). Operational facts (2026-06-10): + +- **Webhook URL is the IN-CLUSTER service**: `http://woodpecker-server.woodpecker.svc.cluster.local/api/hook?...` + (PATCHed via the Forgejo API). The Woodpecker-generated default + (`https://ci.viktorbarzin.me/...`) resolves to the non-proxied public A + record from pods → NAT hairpin → intermittent `context deadline exceeded`, + silently dropping push events (found when a push produced no pipeline). + If Woodpecker ever "repairs" the repo it will rewrite the hook back to + `ci.viktorbarzin.me` — re-apply the in-cluster URL (or pin `ci.viktorbarzin.me` + in the CoreDNS pod carve-out alongside forgejo). +- **Repo-scoped secrets must exist on BOTH repos**: pipelines reference + repo-level secrets (`registry_ssh_key`, `pve_ssh_key`, `CLOUDFLARE_TOKEN`, + …). Repo 82 was registered without them and every all-workflow compile + errored with `secret "registry_ssh_key" not found`. Fixed by cloning repo-1 + rows to repo 82 in the Woodpecker DB (`insert into secrets … select … where + repo_id=1`). When registering a new forge repo for infra, clone the secret + set too. +- **Empty commits defeat path filters**: a commit with no changed files makes + Woodpecker include ALL workflow files (path conditions can't exclude), so + every repo secret must resolve. Normal commits with real files only compile + the matching workflows. + ## Decisions & Rationale ### Why GitHub Actions + Woodpecker?