infra/docs/post-mortems/2026-06-11-devvm-qemu-io-stall.md

2026-06-11 — devvm dead ~90 min: QEMU-internal I/O stall on the legacy LSI disk path

Impact

• devvm (VM 102, the shared multi-user Claude Code workstation) effectively dead 15:21–16:48 UTC (18:21–19:48 EEST): all ssh/tmux and t3 sessions for wizard/emo/anca lost, every in-flight agent killed.
• Detection was human (~90 min) — no up{instance="devvm"} == 0 alert exists (follow-up below).
• Recovery was manual: kill of the wedged QEMU process + qm start (the kill left no autopsy — see "What we could not prove").

Timeline (UTC; host journal runs EEST = UTC+3)

• 15:01 — hourly apply-mbps-caps run live-rewrites VM 102's scsi0 throttle via qm set (as it had done every hour for weeks — see Root cause #4).
• 15:18–15:20 — guest healthy by every metric: CPU 7–16% of 16 vCPUs, load 1.4, 17 GiB MemAvailable, swap flat at 2.0 GiB, host sdc 2–8% utilized. Heavy claude/bwrap sandbox activity (normal workload).
• 15:19:08 — last journal line the guest ever writes (mid normal traffic, zero kernel distress — not even a hung-task warning).
• 15:21 — host RRD (pvestatd polling QEMU over QMP once a minute) shows diskwrite drop to exactly 0 and stay 0 for 87 minutes — not even journal flushes. netout collapses 380K→7K/s. QEMU keeps answering QMP the whole time — the process and its main loop are alive; only the block path is dead.
• 15:21→15:39 — guest CPU (host's view) ramps 11% → ~50% and plateaus: processes progressively piling up behind dead storage (dirty-page writeback stuck → direct reclaim spins). Classic starvation cascade, not a panic (a panic halts or spins flat from t=0).
• 16:47:42 — QMP socket resets: the wedged QEMU is killed out-of-band (root shell; no PVE task, no snoopy line — shell-builtin kill).
• 16:48:31 — qmstart task; guest boots clean on kernel 6.8.0-124 (wedged boot ran 6.8.0-117).

Ruled out (evidence, not vibes)

• Guest CPU/memory/swap pressure — healthy at last scrape (Prometheus) and per-minute host RRD.
• Host storage — pve thin pool 68% data / 15.5% meta; zero kernel I/O errors on the host all day; sdc quiet through the window.
• Host-side kill/OOM — no OOM-killer lines, no segfault, no QEMU crash log; 113 of 114 monitored targets stayed up. Only the devvm died.
• Guest kernel panic — would not keep QMP-visible blockstats frozen at 0 while netout ACKs trickle; and the guest kernel logged nothing.

Root cause

Class pinned, exact line unprovable (see below): the devvm's disk I/O stalled inside the QEMU process — below the guest kernel (all guest I/O froze simultaneously with nothing logged) and above host storage (host clean, neighbors fine, QEMU main loop responsive). Contributing stack, unique to this VM:

1. scsihw: lsi — the emulated LSI 53C895A (1997 chip, QEMU's legacy default for OSes without virtio drivers). The devvm was the only VM on the host running its disk through this path; every healthy neighbor uses virtio-scsi-pci. The LSI model is documented as hang-prone under intensive I/O.
2. No iothread — all disk emulation ran on QEMU's single main event loop, sharing it with timers and QMP.
3. QEMU-level mbps throttle (60/60) — a token bucket inside QEMU whose queued I/O completes only when its re-arm timer fires.
4. Hourly live throttle rewrites — apply-mbps-caps.sh's idempotency check compared raw config strings, but qm config prints keys in its own canonical order, so the check never matched and the script re-issued qm set (→ live QMP block_set_io_throttle against the running QEMU) every hour, 24×/day, for weeks — each poke a chance to race the throttle machinery while queued I/O is in flight. The wedge came 20 min after the 15:01 poke.

What we could not prove

Whether the stuck queue was the LSI device model, the throttle-group timer, or their interaction. The discriminating evidence (QMP query-block, a stack trace of the QEMU process) existed in RAM at 16:47 and was destroyed by the recovery kill. If a wedge recurs autopsy before shooting: qm guest exec will fail but qm monitor/QMP query-block, query-status, and gdb -p <pid> -batch -ex 'thread apply all bt' on the kvm process pin it to the line.

Fixes

Status	Fix
shipped (this commit)	apply-mbps-caps.sh compares normalized option sets — hourly runs are now true no-ops; running VMs' throttle state is no longer rewritten 24×/day. Verified: reordered-key configs compare equal, real drift still triggers qm set, post-restart iothread configs compare equal.
staged, awaiting Viktor's cold stop→start	VM 102: scsihw: virtio-scsi-single + scsi0 …,iothread=1,aio=threads — replaces the LSI path with the paravirt controller all healthy VMs use, moves disk emulation off the main loop, swaps io_uring for boring thread-pool AIO. Guest pre-flight passed (CONFIG_SCSI_VIRTIO=y built-in; fstab on LVM dm-uuid/UUID). Must be a full stop→start — a guest reboot reuses the old QEMU process.

Open follow-ups (discussed 2026-06-11, not yet built)

• DevvmDown alert (up{job="devvm"} == 0 for 3m → Slack) — closes the 90-min detection gap.
• Freeze forensics: netconsole → pve listener, serial console, kernel.panic=60, and a capture-before-kill runbook (above) so any recurrence is pinned, not mourned.
• The recurring crawl class (agent storms → swap-thrash; journald watchdog-killed 3× on 2026-06-10) is a separate failure mode — ssh/tmux sessions remain memory-uncontained by explicit decision (swap-only, 2026-06-10).

Lessons

• A VM can die of QEMU-userspace causes that no guest or host kernel log will ever show. The host's per-VM RRD (pvestatd's QMP polls) is the only witness — diskwrite=0 with a live QMP socket is the signature.
• "Idempotent" reconcilers must prove idempotency against the system's canonical output format, not against the string they themselves constructed. A compare that never matches turns a safety net into a 24×/day fault injector — and its own journal said updating scsi0 every hour, in plain sight, for weeks.
• The May-26 mbps caps fixed the sdc-saturation freeze class and introduced this one's trigger surface. Layered mitigations fail in layers — audit what a fix adds, not only what it removes.
• pve host logs are EEST (UTC+3); guest logs are UTC. Every cross-machine correlation in this incident initially looked 3h off.