diff --git a/.claude/reference/proxmox-inventory.md b/.claude/reference/proxmox-inventory.md index 2b9ab3fe..5e308aee 100644 --- a/.claude/reference/proxmox-inventory.md +++ b/.claude/reference/proxmox-inventory.md @@ -92,7 +92,7 @@ Channel 3: A4 [32G] ──── A8 [32G] ──── A12[ 8G ] = 72 GB | VMID | Name | Status | CPUs | RAM | Network | Disk | Notes | |------|------|--------|------|-----|---------|------|-------| | 101 | pfsense | running | 8 | 4GB | vmbr0, vmbr1:vlan10, vmbr1:vlan20 | 32G | Gateway/firewall | -| 102 | devvm | running | 16 | 24GB | vmbr1:vlan10 | 100G | Development VM + t3code Workstation host. 14G swap (8G /swapfile + 6G /swapfile2, grown 2026-06-10; swappiness=10). Capacity budget: ~4-5G RAM/active user, max ~3-4 concurrent active Claude sessions. NOT Terraform-managed. | +| 102 | devvm | running | 16 | 24GB | vmbr1:vlan10 | 100G | Development VM + t3code Workstation host. 14G swap (8G /swapfile + 6G /swapfile2, grown 2026-06-10; swappiness=10). Capacity budget: ~4-5G RAM/active user, max ~3-4 concurrent active Claude sessions. NOT Terraform-managed. Disk controller: `virtio-scsi-single` + `scsi0 iothread=1,aio=threads` staged 2026-06-11 after the QEMU I/O stall (was `scsihw: lsi`, the only VM on the legacy path — see `docs/post-mortems/2026-06-11-devvm-qemu-io-stall.md`); applies at next cold stop→start. | | 103 | home-assistant | running | 8 | 8GB | vmbr0 | 64G | HA Sofia, net0(vlan10) disabled, SSH: vbarzin@192.168.1.8 | | 105 | pbs | stopped | 16 | 8GB | vmbr1:vlan10 | 32G | Proxmox Backup (unused) | | 200 | k8s-master | running | 8 | 32GB | vmbr1:vlan20 | 64G | Control plane (10.0.20.100) | diff --git a/docs/architecture/compute.md b/docs/architecture/compute.md index 0b808b55..0f6ff092 100644 --- a/docs/architecture/compute.md +++ b/docs/architecture/compute.md @@ -101,7 +101,12 @@ graph TB > PVE host (sources in `infra/scripts/`, install pattern per > `architecture/backup-dr.md`). Timer fires `OnBootSec=5min` + > `OnCalendar=hourly`, so any drift (config restore, manual `qm -> set`, fresh clone) self-heals within the hour. Current caps: +> set`, fresh clone) self-heals within the hour. The script compares +> *normalized option sets*, so an unchanged config is a true no-op — +> until 2026-06-11 a raw string compare (defeated by `qm config`'s +> canonical key order) re-issued `qm set` hourly against running VMs, +> live-rewriting QEMU throttle state via QMP (implicated in the devvm +> I/O stall; see `post-mortems/2026-06-11-devvm-qemu-io-stall.md`). Current caps: > 102 devvm 60/60, 103 home-assistant 40/40, 200 k8s-master 100/60, > 201 k8s-node1 150/120, 202 k8s-node2 150/120, 203 k8s-node3 150/120, > 204 k8s-node4 150/120, 220 docker-registry 40/40. diff --git a/docs/post-mortems/2026-06-11-devvm-qemu-io-stall.md b/docs/post-mortems/2026-06-11-devvm-qemu-io-stall.md new file mode 100644 index 00000000..d62ed5ff --- /dev/null +++ b/docs/post-mortems/2026-06-11-devvm-qemu-io-stall.md @@ -0,0 +1,116 @@ +# 2026-06-11 — devvm dead ~90 min: QEMU-internal I/O stall on the legacy LSI disk path + +## Impact + +- devvm (VM 102, the shared multi-user Claude Code workstation) effectively + dead 15:21–16:48 UTC (18:21–19:48 EEST): all ssh/tmux and t3 sessions for + wizard/emo/anca lost, every in-flight agent killed. +- Detection was human (~90 min) — no `up{instance="devvm"} == 0` alert + exists (follow-up below). +- Recovery was manual: kill of the wedged QEMU process + `qm start` (the + kill left no autopsy — see "What we could not prove"). + +## Timeline (UTC; host journal runs EEST = UTC+3) + +- **15:01** — hourly `apply-mbps-caps` run live-rewrites VM 102's scsi0 + throttle via `qm set` (as it had done every hour for weeks — see Root + cause #4). +- **15:18–15:20** — guest healthy by every metric: CPU 7–16% of 16 vCPUs, + load 1.4, 17 GiB MemAvailable, swap flat at 2.0 GiB, host `sdc` 2–8% + utilized. Heavy claude/bwrap sandbox activity (normal workload). +- **15:19:08** — last journal line the guest ever writes (mid normal + traffic, zero kernel distress — not even a hung-task warning). +- **15:21** — host RRD (pvestatd polling QEMU over QMP once a minute) shows + `diskwrite` drop to **exactly 0 and stay 0 for 87 minutes** — not even + journal flushes. netout collapses 380K→7K/s. **QEMU keeps answering QMP + the whole time** — the process and its main loop are alive; only the + block path is dead. +- **15:21→15:39** — guest CPU (host's view) ramps 11% → ~50% and plateaus: + processes progressively piling up behind dead storage (dirty-page + writeback stuck → direct reclaim spins). Classic starvation cascade, not + a panic (a panic halts or spins flat from t=0). +- **16:47:42** — QMP socket resets: the wedged QEMU is killed out-of-band + (root shell; no PVE task, no snoopy line — shell-builtin `kill`). +- **16:48:31** — `qmstart` task; guest boots clean on kernel 6.8.0-124 + (wedged boot ran 6.8.0-117). + +## Ruled out (evidence, not vibes) + +- **Guest CPU/memory/swap pressure** — healthy at last scrape (Prometheus) + and per-minute host RRD. +- **Host storage** — `pve` thin pool 68% data / 15.5% meta; zero kernel + I/O errors on the host all day; `sdc` quiet through the window. +- **Host-side kill/OOM** — no OOM-killer lines, no segfault, no QEMU crash + log; 113 of 114 monitored targets stayed up. Only the devvm died. +- **Guest kernel panic** — would not keep QMP-visible blockstats frozen at + 0 while netout ACKs trickle; and the guest kernel logged nothing. + +## Root cause + +**Class pinned, exact line unprovable** (see below): the devvm's disk I/O +stalled *inside the QEMU process* — below the guest kernel (all guest I/O +froze simultaneously with nothing logged) and above host storage (host +clean, neighbors fine, QEMU main loop responsive). Contributing stack, +unique to this VM: + +1. **`scsihw: lsi`** — the emulated LSI 53C895A (1997 chip, QEMU's legacy + default for OSes without virtio drivers). The devvm was the **only VM + on the host** running its disk through this path; every healthy + neighbor uses `virtio-scsi-pci`. The LSI model is documented as + hang-prone under intensive I/O. +2. **No `iothread`** — all disk emulation ran on QEMU's single main event + loop, sharing it with timers and QMP. +3. **QEMU-level mbps throttle (60/60)** — a token bucket inside QEMU whose + queued I/O completes only when its re-arm timer fires. +4. **Hourly live throttle rewrites** — `apply-mbps-caps.sh`'s idempotency + check compared raw config strings, but `qm config` prints keys in its + own canonical order, so the check **never matched** and the script + re-issued `qm set` (→ live QMP `block_set_io_throttle` against the + running QEMU) every hour, 24×/day, for weeks — each poke a chance to + race the throttle machinery while queued I/O is in flight. The wedge + came 20 min after the 15:01 poke. + +## What we could not prove + +Whether the stuck queue was the LSI device model, the throttle-group +timer, or their interaction. The discriminating evidence (QMP +`query-block`, a stack trace of the QEMU process) existed in RAM at 16:47 +and was destroyed by the recovery kill. If a wedge recurs **autopsy before +shooting**: `qm guest exec` will fail but `qm monitor`/QMP `query-block`, +`query-status`, and `gdb -p -batch -ex 'thread apply all bt'` on the +kvm process pin it to the line. + +## Fixes + +| Status | Fix | +|---|---| +| shipped (this commit) | `apply-mbps-caps.sh` compares **normalized option sets** — hourly runs are now true no-ops; running VMs' throttle state is no longer rewritten 24×/day. Verified: reordered-key configs compare equal, real drift still triggers `qm set`, post-restart iothread configs compare equal. | +| staged, awaiting Viktor's cold stop→start | VM 102: `scsihw: virtio-scsi-single` + `scsi0 …,iothread=1,aio=threads` — replaces the LSI path with the paravirt controller all healthy VMs use, moves disk emulation off the main loop, swaps io_uring for boring thread-pool AIO. Guest pre-flight passed (`CONFIG_SCSI_VIRTIO=y` built-in; fstab on LVM dm-uuid/UUID). Must be a **full stop→start** — a guest reboot reuses the old QEMU process. | + +## Open follow-ups (discussed 2026-06-11, not yet built) + +- `DevvmDown` alert (`up{job="devvm"} == 0 for 3m` → Slack) — closes the + 90-min detection gap. +- Freeze forensics: netconsole → pve listener, serial console, + `kernel.panic=60`, and a capture-before-kill runbook (above) so any + recurrence is pinned, not mourned. +- The recurring *crawl* class (agent storms → swap-thrash; journald + watchdog-killed 3× on 2026-06-10) is a separate failure mode — + ssh/tmux sessions remain memory-uncontained by explicit decision + (swap-only, 2026-06-10). + +## Lessons + +- **A VM can die of QEMU-userspace causes that no guest or host kernel log + will ever show.** The host's per-VM RRD (pvestatd's QMP polls) is the + only witness — `diskwrite=0` with a live QMP socket is the signature. +- **"Idempotent" reconcilers must prove idempotency against the system's + canonical output format**, not against the string they themselves + constructed. A compare that never matches turns a safety net into a + 24×/day fault injector — and its own journal said `updating scsi0` + every hour, in plain sight, for weeks. +- The May-26 mbps caps fixed the sdc-saturation freeze class and + introduced this one's trigger surface. Layered mitigations fail in + layers — audit what a fix *adds*, not only what it removes. +- pve host logs are **EEST (UTC+3)**; guest logs are UTC. Every + cross-machine correlation in this incident initially looked 3h off. diff --git a/scripts/apply-mbps-caps.sh b/scripts/apply-mbps-caps.sh index 39bfb36b..5358f7db 100755 --- a/scripts/apply-mbps-caps.sh +++ b/scripts/apply-mbps-caps.sh @@ -27,6 +27,12 @@ TARGETS=( "220:scsi0:40:40" # docker-registry ) +# Sort a disk spec's comma-separated options so two specs with the same +# option set but different key order compare equal. +normalized() { + tr ',' '\n' <<<"$1" | LC_ALL=C sort | paste -sd, - +} + apply_one() { local spec="$1" local vmid slot rd wr @@ -49,8 +55,13 @@ apply_one() { newvalue="${cleaned},mbps_rd=${rd},mbps_wr=${wr}" # Skip the qm-set call entirely when state already matches — keeps - # journal noise low under the hourly timer. - if [[ "$current" == "$newvalue" ]]; then + # journal noise low under the hourly timer. Compare option SETS, not raw + # strings: `qm config` prints keys in its own canonical order, so a raw + # compare never matched and every hourly run re-issued `qm set`, which + # live-rewrites the running VM's QEMU throttle state via QMP (implicated + # in the 2026-06-11 devvm I/O stall — see + # docs/post-mortems/2026-06-11-devvm-qemu-io-stall.md). + if [[ "$(normalized "$current")" == "$(normalized "$newvalue")" ]]; then echo "vmid $vmid: $slot already at mbps_rd=${rd},mbps_wr=${wr} — no-op" return 0 fi