k8s-upgrade: reclaim+auto-prune kubeadm /etc/kubernetes/tmp leak; correct crash root cause to etcd IO (not OIDC)

Digging into "why did the apiserver crash" disproved the earlier OIDC
explanation. An isolated v1.35.6 apiserver repro with authentik reachable
initialises OIDC cleanly (oidc.go:313, no error) and runs fine — so the
--authentication-config -> --oidc-* revert is NOT what crashed it. etcd's
surviving crash-window log is the real cause: 1180 "apply request took too long"
warnings in 16 min, individual applies up to 4.3s (healthy <100ms) right as
kubeadm tried to bring up the new apiserver. That's etcd IO starvation on the
shared sdc HDD (beads code-oflt).

A big contributor + the reason master root fs sat at 73%: kubeadm dumps a full
~400MB etcd DB backup into /etc/kubernetes/tmp/kubeadm-backup-etcd-<ts>/ before
every etcd upgrade and never cleans it up — 145 dirs / 28GB had accumulated,
driving image-GC churn and extra write-IO onto etcd's spindle. Reclaimed live
(73% -> 23%) and added a preflight prune (>3 days) so it can't re-accumulate.

Also corrected the OIDC handling: the kubeadm-config drift is real but only
breaks dashboard/kubectl SSO AFTER a successful upgrade (recoverable via the
chain's restore.sh + the kubeadm-config reconciliation) — it does not crash the
apiserver. So the preflight check is now an ALERT, not a block (was added on the
wrong hypothesis). Post-mortem, runbook, and apiserver-oidc.tf header corrected.

Per Viktor: reclaim the disk and automate so the manual cleanup never recurs;
the durable IO fix remains code-oflt (etcd off the shared HDD).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/post-mortems/2026-06-24-kubeadm-oidc-drift-apiserver-upgrade-stall.md

@@ -1,4 +1,8 @@
-# Post-mortem: kubeadm-config OIDC drift crash-looped the v1.35 apiserver upgrade (2026-06-24)
+# Post-mortem: k8s 1.34→1.35 upgrade stalled — etcd IO starvation (2026-06-24)
+
+> Filename kept for inbound links. The originally-suspected cause (kubeadm-config
+> OIDC drift) turned out **not** to be the crash — see "Correction" below. The OIDC
+> drift was a real *separate* latent bug fixed in the same change.
 
 **Impact:** The autonomous k8s-version-upgrade chain (23:00 UTC nightly) reached
 the master control-plane phase for the first time — preflight passed, etcd
@@ -6,85 +10,88 @@ snapshot taken, master cordoned + drained, etcd upgraded 3.6.5→3.6.6 — then
 kube-apiserver upgrade to v1.35.6 **crash-looped**. kubeadm waited its 5-minute
 static-pod-hash window across all internal retries, then auto-rolled-back to
 v1.34.9. The cluster stayed healthy on 1.34.9 (apiserver, all 7 nodes Ready), but
-the run left **k8s-master cordoned** and the chain **wedged on `in_flight=1`**
+the run left **k8s-master cordoned** and the chain **wedged on `in_flight=1`**.
-(which correctly blocks subsequent runs). No data loss; no user-facing outage
+No data loss; no user-facing outage (the master carries control-plane taints, so
-(the master carries control-plane taints, so no workloads were displaced).
+no workloads were displaced).
 
-**Trigger:** the first *minor* upgrade the chain ever attempted (1.34→1.35).
+**Trigger:** the first *minor* upgrade the chain ever attempted (1.34→1.35) — the
-Patch upgrades never hit this because the apiserver manifest content is identical
+first time kubeadm upgrades etcd (3.6.5→3.6.6) and regenerates the control-plane
-across patches; a minor upgrade is the first time kubeadm regenerates the
+static pods, i.e. the first time the upgrade pushes real write-IO at etcd.
-manifest with a new image.
 
-## Root cause
+## Root cause — etcd IO starvation on the shared HDD
 
-apiserver authentication was configured in **two** places that were allowed to
+The new kube-apiserver could not establish/keep a working connection to etcd
-drift from a **third**:
+during the upgrade because **etcd was IO-starved**. etcd's surviving container log
+from the crash window (`/var/log/pods/.../etcd/0.log`, 23:04–23:20 UTC) shows:
 
-1. `/etc/kubernetes/pki/auth-config.yaml` — a structured `AuthenticationConfiguration`
+- **1,180** `apply request took too long` warnings in 16 minutes;
-   (apiserver.config.k8s.io/v1) carrying **two** JWT issuers (`kubernetes` for
+- individual applies of **4.3s / 2.9s / 2.7s / 1.8s** (healthy is <100ms),
-   kubectl/kubelogin + `k8s-dashboard` for the dashboard's oauth2-proxy), added
+  clustered at **23:18:51 UTC** — exactly when kubeadm's final attempt was trying
-   2026-06-19 (`docs/plans/2026-06-04-k8s-dashboard-sso-design.md`).
+  to bring the new apiserver up.
-2. the **live** kube-apiserver static-pod manifest — referenced it via
-   `--authentication-config=/etc/kubernetes/pki/auth-config.yaml`.
-3. the **kubeadm-config `ClusterConfiguration` ConfigMap** — still carried the
-   **legacy single-issuer `--oidc-*` extraArgs** (`oidc-issuer-url`,
-   `oidc-client-id`, `oidc-username-claim`, `oidc-groups-claim`). Never updated
-   when (1)+(2) switched to structured auth.
 
-`kubeadm upgrade apply` **regenerates the static-pod manifests from
+A reproduced 1.35.6 apiserver with no etcd dies with
-kubeadm-config**. So it dropped `--authentication-config` and re-added the four
+`F instance.go:233 Error creating leases: error creating storage factory: context
-`--oidc-*` flags. Proven by `kubeadm upgrade diff v1.35.6`:
+deadline exceeded` — the same failure mode a multi-second etcd produces. etcd
+lives on the contended `sdc` HDD (**beads code-oflt**: "etcd/critical VM disks on
+shared sdc HDD — recurring IO-storm root cause"). The upgrade itself piled IO onto
+that spindle:
 
-```diff
+1. etcd's own upgrade-restart + WAL/db re-read (it restarted ~23:04, re-elected);
--    - --authentication-config=/etc/kubernetes/pki/auth-config.yaml
+2. kubeadm dumping a full **~400MB etcd DB backup** to
-+    - --oidc-issuer-url=https://authentik.viktorbarzin.me/application/o/kubernetes/
+   `/etc/kubernetes/tmp/kubeadm-backup-etcd-<ts>/` (on the same HDD) before the
-+    - --oidc-client-id=kubernetes
+   etcd upgrade — and **145 of these had accumulated to 28GB** (kubeadm never
-+    - --oidc-username-claim=email
+   cleans them up), pushing master root fs to **73%**, above the 70% kubelet
-+    - --oidc-groups-claim=groups
+   image-GC threshold, so image GC churned during the drain too;
-```
+3. master-drain pod evictions.
 
-The regenerated apiserver crash-looped (`CrashLoopBackOff`, `back-off 10s`, 8
+### Correction — it was NOT the OIDC flag swap
-probe failures in the kubelet journal) — it exited within seconds, repeatedly, so
-kubeadm's hash-watch never saw a stable new pod and timed out → rollback. (The
-`--oidc-*` flags are NOT removed in 1.35; the crash is the auth-config swap in the
-live control-plane environment, the only functional delta in the diff. Image
-pull, etcd, OOM, and disk were all ruled out: all v1.35.6 images were pre-pulled,
-etcd upgraded cleanly, no OOM, master root disk at 73%.)
 
-**Why the existing safety net missed it:** `stacks/rbac/modules/rbac/apiserver-oidc.tf`
+`kubeadm upgrade diff v1.35.6` showed the regenerated manifest also swaps
-already *knew* kubeadm drops `--authentication-config` and published a
+`--authentication-config` (structured multi-issuer OIDC) back to legacy
-`apiserver-oidc-restore` ConfigMap for the chain to re-run **after** the upgrade.
+single-issuer `--oidc-*` flags (kubeadm-config drift, see secondary finding). That
-But the apiserver crashes *during* `kubeadm upgrade apply`, which never returns
+was the *first* hypothesis — but an isolated repro of the 1.35.6 apiserver with
-success, so the post-upgrade restore step is never reached.
+those exact `--oidc-*` flags **and authentik reachable** initialised OIDC cleanly
+(`oidc.go:313`, no error) and ran fine until it hit the (deliberately dead) test
+etcd. So the auth swap does **not** crash the apiserver; it was a red herring for
+the crash. Image pull (all v1.35.6 images pre-pulled), OOM (none), and disk-full
+were also ruled out.
+
+## Secondary finding (real, fixed separately) — kubeadm-config OIDC drift
+
+apiserver auth is configured in three places that must agree:
+(1) `/etc/kubernetes/pki/auth-config.yaml` (structured, two issuers: `kubernetes`
++ `k8s-dashboard`, added 2026-06-19); (2) the live static-pod manifest
+(`--authentication-config`); (3) the kubeadm-config `ClusterConfiguration` CM —
+which still carried the legacy `--oidc-*` extraArgs. `kubeadm upgrade` regenerates
+the manifest from (3), so it would have reverted structured auth → **dashboard +
+kubectl SSO break after a successful upgrade** (recoverable: the chain's
+post-master `restore.sh` re-adds the flag). This is a real bug, just not the crash.
 
 ## Resolution
 
-1. **Reconciled kubeadm-config live** (2026-06-24, zero cluster impact — the CM is
+1. **Reclaimed the 28GB kubeadm scratch** on master (`/etc/kubernetes/tmp/kubeadm-backup-*`) — root fs 73% → 23%.
-   only read during an upgrade): rewrote `apiServer.extraArgs` to drop the
+2. **Reconciled kubeadm-config live** (zero cluster impact — CM only read at upgrade time): dropped `--oidc-*`, added `--authentication-config` via `kubeadm init phase upload-config kubeadm`. `kubeadm upgrade diff` then shows only the control-plane image bumps.
-   `--oidc-*` args and add `--authentication-config`, via `kubeadm init phase
+3. **Recovered:** uncordoned k8s-master, cleared the stuck `in_flight` gauge + annotation, deleted last night's Complete/Failed `1-35-6` phase jobs (a Complete preflight would otherwise make the detector idempotent-skip the re-run).
-   upload-config kubeadm`. `kubeadm upgrade diff v1.35.6` then showed **only** the
-   control-plane image bumps — no auth-flag changes.
-2. **Recovered:** uncordoned k8s-master, cleared the stuck `in_flight` gauge +
-   namespace annotation.
 
-## Prevention (all landed in this change)
+## Prevention (landed in this change)
 
 | Gap | Fix |
 |-----|-----|
-| kubeadm-config not managed alongside the live manifest | `apiserver-oidc.tf`'s remote script now **also** reconciles kubeadm-config (`kubeadm init phase upload-config`). It reaches the cluster two ways: the published `apiserver-oidc-restore` ConfigMap (a plain k8s resource — CI applies it with no ssh) which the chain's `phase_master` re-runs, and a local `-replace` apply with `TF_VAR_ssh_private_key`. (The null_resource trigger deliberately does NOT hash the script: CI has no ssh key, so it must stay a no-op on a plain CI apply.) |
+| kubeadm leaks ~400MB etcd-DB backups into `/etc/kubernetes/tmp` forever (→ disk fills, image-GC churn, write-IO on etcd's spindle) | **`upgrade-step.sh` preflight now prunes** `/etc/kubernetes/tmp/kubeadm-backup-*` + `kubeadm-upgraded-manifests*` older than 3 days on master, every run. Best-effort, never aborts. |
-| The chain drained the master into a crash with no pre-check | new **preflight gate 4b** in `upgrade-step.sh`: runs `kubeadm upgrade diff v$TARGET` and `block`s (k8s_upgrade_blocked=1 → K8sUpgradeBlocked alert) BEFORE snapshot/in-flight/drain if a `-` line would drop `--authentication-config`. Fails safe — blocks only on a positive drift signal. |
+| kubeadm-config drift would silently break SSO after an upgrade | `apiserver-oidc.tf`'s remote script now **also reconciles kubeadm-config** (`kubeadm init phase upload-config`), delivered via the `apiserver-oidc-restore` ConfigMap the chain re-runs (CI needs no ssh) or a local `-replace` apply. Preflight **alerts** (not blocks — SSO drift is recoverable) if `kubeadm upgrade diff` would still drop `--authentication-config`. |
-| The live fix had to be applied out-of-band (only `default` Vault policy on the workstation; CI can't ssh) | kubeadm-config reconciled live via `kubeadm init phase upload-config` on the master (2026-06-24); the committed code makes it durable for future upgrades. |
+| etcd on the contended `sdc` HDD starves under upgrade IO | **Durable fix is beads code-oflt** (move etcd/critical VM disks off `sdc`). Not in this change. Mitigations above reduce the upgrade's own IO; reclaimed disk removes the image-GC variable. |
 
 ## Lessons
 
-- **Out-of-band control-plane edits must be written back to kubeadm-config.**
+- **Capture the failing component's own logs before concluding.** The `kubeadm
-  Anything that edits a static-pod manifest directly (auth, admission, audit, API
+  upgrade diff` made the OIDC swap look like the cause; only etcd's log (multi-second
-  flags) is silently reverted on the next `kubeadm upgrade` unless kubeadm-config
+  applies) + an isolated apiserver repro showed the truth (etcd IO). A clean diff is
-  itself carries it. `kubeadm upgrade diff <target>` is the authoritative
+  "what config changes," not "why it crashed."
-  pre-flight check for "what will the upgrade change?" and is non-mutating.
+- **etcd on shared HDD is the cluster's recurring fragility** (immich IO storm
-- **A post-upgrade fixup can't repair something that breaks the upgrade itself.**
+  2026-05-25, this stall). Upgrades concentrate IO (etcd restart + kubeadm's 400MB
-  The restore-after-upgrade design assumed the apiserver would come up (degraded)
+  backup copy + drain) onto that spindle. code-oflt is the real fix.
-  and be fixed afterward; it actually crash-looped, so the fix has to be in
+- **Tools that leave per-operation scratch must be reaped.** kubeadm's
-  kubeadm-config *before* `apply`, plus a preflight gate.
+  `/etc/kubernetes/tmp` etcd backups are throwaway (real backups → NFS) but never
-- **Minor upgrades exercise manifest regeneration; patch upgrades don't.** First
+  GC'd; 28GB had silently accumulated.
-  minor bump is where this whole class of drift surfaces.
+- **Out-of-band control-plane edits must be written back to kubeadm-config** — else
+  `kubeadm upgrade` silently reverts them (here: SSO; could be admission/audit/API flags).

docs/runbooks/k8s-version-upgrade.md

@@ -41,7 +41,8 @@ Job 0 — preflight       (pinned: k8s-node1)
   ├── halt-on-alert (kured-style ignore-list)
   ├── 24h-quiet baseline (no Ready transitions <24h ago)
   ├── kubeadm upgrade plan matches target (skipped when master already at target — partial-resume)
-  ├── apiserver-OIDC drift gate: kubeadm upgrade diff must NOT drop --authentication-config (else BLOCK+alert)
+  ├── apiserver-OIDC drift check: kubeadm upgrade diff drops --authentication-config? → Slack WARN (recoverable; not a block)
+  ├── reclaim kubeadm scratch: prune /etc/kubernetes/tmp/kubeadm-backup-* >3d on master (kubeadm leaks ~400MB etcd-db backups)
   ├── Push k8s_upgrade_in_flight=1, k8s_upgrade_started_timestamp=$(date +%s)
   ├── Trigger backup-etcd Job, wait, verify snapshot byte count
   ├── SSH master: containerd skew fix (if master < workers)
@@ -229,10 +230,10 @@ Exposed in K8s via ExternalSecret `k8s-upgrade-creds` in the `k8s-upgrade` names
 from kubeadm-config**. apiserver auth uses a structured multi-issuer
 `--authentication-config` (kubectl + dashboard SSO), but kubeadm-config used to
 still carry the legacy single-issuer `--oidc-*` extraArgs — so every upgrade
-reverted the flag. On the **1.34→1.35** bump that regenerated apiserver
+reverted the flag, **silently breaking SSO after the upgrade** (the apiserver does
-**crash-looped and stalled the whole upgrade mid-flight** (master cordoned, etcd
+NOT crash on this — verified by isolated repro; it's recoverable via the restore
-already bumped); the post-upgrade restore below never ran because `kubeadm
+script below). NB: the **1.34→1.35 stall on 2026-06-24 was a *separate* issue —
-upgrade apply` itself never returned success. Post-mortem:
+etcd IO starvation**, not this drift; post-mortem:
 `docs/post-mortems/2026-06-24-kubeadm-oidc-drift-apiserver-upgrade-stall.md`.
 
 **Primary fix (2026-06-24):** `stacks/rbac/modules/rbac/apiserver-oidc.tf` now
@@ -243,9 +244,9 @@ upgrades with a pure image bump — `kubeadm upgrade diff <target>` shows only t
 image change. Zero live impact (the CM is read only during an upgrade).
 
 **Backstops:**
-- **Preflight gate 4b** runs `kubeadm upgrade diff` and BLOCKs (k8s_upgrade_blocked=1
+- **Preflight check 4b** runs `kubeadm upgrade diff` and **alerts** (Slack WARN, does
-  → alert) BEFORE draining the master if `--authentication-config` would still be
+  NOT block — the drift only breaks SSO, which is recoverable) if
-  dropped — so this can never again drain into a crash.
+  `--authentication-config` would still be dropped.
 - The `rbac` stack still publishes its restore script to the
   `kube-system/apiserver-oidc-restore` ConfigMap, and `phase_master` re-runs it on
   master right after `kubeadm upgrade apply` (idempotent, `/livez`-gated with

stacks/k8s-version-upgrade/scripts/upgrade-step.sh

@@ -416,25 +416,39 @@ phase_preflight() {
     fi
   fi
 
-  # 4b. apiserver-OIDC drift gate (backstop for the rbac stack's kubeadm-config
+  # 4b. apiserver-OIDC drift check (backstop for the rbac stack's kubeadm-config
   # reconciliation). A `kubeadm upgrade` REGENERATES the apiserver manifest from
   # kubeadm-config; if kubeadm-config still carries the legacy single-issuer
   # --oidc-* args instead of --authentication-config, the regenerated apiserver
-  # reverts structured multi-issuer auth and CRASH-LOOPS — stalling the chain
+  # loses structured multi-issuer auth → kubectl + dashboard SSO break AFTER the
-  # mid-flight with the master cordoned and etcd already bumped (the 2026-06-24
+  # upgrade. This is RECOVERABLE (the apiserver does NOT crash — verified by an
-  # v1.35 stall; docs/post-mortems/2026-06-24-kubeadm-oidc-drift-apiserver-upgrade-stall.md).
+  # isolated repro 2026-06-24; the chain's post-master restore.sh re-adds the flag,
-  # `kubeadm upgrade diff` shows exactly what the manifest regen will change; a
+  # and the rbac stack reconciles kubeadm-config so it won't recur) — so this is an
-  # '-' line dropping --authentication-config means the drift is still present.
+  # ALERT, not a block. (NB the 2026-06-24 stall was NOT this — it was etcd IO
-  # Skip on an at-target master (resume — no apiserver regen). Best-effort: blocks
+  # starvation; see docs/post-mortems/2026-06-24-kubeadm-oidc-drift-apiserver-upgrade-stall.md.)
-  # only on a POSITIVE drift signal, never merely because diff is unavailable.
+  # Skip on an at-target master (resume — no apiserver regen).
   if [ "$master_kubelet_v" != "$TARGET_VERSION" ]; then
     local apiserver_diff
     apiserver_diff=$(ssh "${SSH_OPTS[@]}" "$(ssh_target k8s-master)" "sudo kubeadm upgrade diff v$TARGET_VERSION 2>/dev/null" || true)
     if echo "$apiserver_diff" | grep -qE '^-[[:space:]].*--authentication-config'; then
-      block "kubeadm upgrade would DROP --authentication-config from kube-apiserver (kubeadm-config OIDC drift → apiserver crash-loop). Re-apply the rbac stack (apiserver-oidc.tf reconciles kubeadm-config), then retry. Master NOT drained."
+      slack "WARN preflight — kubeadm upgrade will DROP --authentication-config (kubeadm-config OIDC drift). SSO breaks post-upgrade until restore.sh re-adds it; re-apply the rbac stack to reconcile kubeadm-config. Proceeding (recoverable, not a crash)."
     fi
   fi
 
+  # 4c. Reclaim kubeadm scratch on master. `kubeadm upgrade apply` dumps a full
+  # ~400MB etcd DB backup into /etc/kubernetes/tmp/kubeadm-backup-etcd-<ts>/ before
+  # every etcd upgrade and NEVER cleans it up — 145 dirs / 28GB had accumulated by
+  # 2026-06-24, pushing master root fs to 73% (image-GC churn + extra write IO on
+  # the shared HDD where etcd lives — a contributor to the etcd IO starvation that
+  # stalled that run, see post-mortem). Real etcd backups go to NFS, so these are
+  # throwaway. Prune ones >3 days old (keeps a short rollback window). Best-effort;
+  # never aborts the chain.
+  if [ "$master_kubelet_v" != "$TARGET_VERSION" ]; then
+    ssh "${SSH_OPTS[@]}" "$(ssh_target k8s-master)" \
+      "sudo find /etc/kubernetes/tmp -maxdepth 1 -type d \( -name 'kubeadm-backup-*' -o -name 'kubeadm-upgraded-manifests*' \) -mtime +3 -exec rm -rf {} + 2>/dev/null; echo -n 'master root after prune: '; df -h / | awk 'NR==2{print \$5\" used, \"\$4\" free\"}'" \
+      || echo "kubeadm-scratch prune skipped (ssh/df failed) — non-fatal"
+  fi
+
   # 5. Push in-flight + started_timestamp metrics + ns annotations
   $KUBECTL annotate ns "$NS" \
     "viktorbarzin.me/k8s-upgrade-in-flight=$(date -u +%FT%TZ)" \

stacks/rbac/modules/rbac/apiserver-oidc.tf

@@ -18,13 +18,16 @@
 #   3. the kubeadm-config ClusterConfiguration CM   — what kubeadm regenerates from
 # Originally only (1)+(2) were managed, so every kubeadm upgrade rewrote the
 # manifest from the STALE CM, reverting --authentication-config to single-issuer
-# --oidc-* flags. On k8s 1.35 that regenerated apiserver CRASH-LOOPED and stalled
+# --oidc-* flags. The consequence is SSO breakage AFTER the upgrade: kubectl +
-# the whole upgrade mid-flight (master cordoned, etcd already bumped) — see
+# dashboard lose multi-issuer auth (the apiserver does NOT crash on this — verified
-# docs/post-mortems/2026-06-24-kubeadm-oidc-drift-apiserver-upgrade-stall.md. The
+# by an isolated repro 2026-06-24; the 2026-06-24 v1.35 upgrade *stall* was a
+# separate etcd IO-starvation issue, see
+# docs/post-mortems/2026-06-24-kubeadm-oidc-drift-apiserver-upgrade-stall.md). The
 # remote script below now ALSO reconciles (3) via `kubeadm init phase
 # upload-config`, so a future kubeadm upgrade regenerates a CORRECT manifest. The
-# k8s-version-upgrade chain additionally GATES on `kubeadm upgrade diff` in
+# k8s-version-upgrade chain additionally ALERTS (does not block — SSO drift is
-# preflight and blocks+alerts if --authentication-config would still be dropped.
+# recoverable) via `kubeadm upgrade diff` in preflight if --authentication-config
+# would still be dropped.
 #
 # SAFETY: the remote script health-gates on /livez and AUTO-ROLLS-BACK the
 # manifest from a timestamped backup if the apiserver does not recover, so a