infra/docs/post-mortems/2026-06-22-devvm-mem-io-overload-containment.md

# 2026-06-22 — devvm memory/IO overload: per-user containment + OOM backstop

## Impact

- devvm (VM 102, the shared multi-user Claude Code workstation) became
  unresponsive under combined memory + IO pressure and had to be **hard-killed +
  rebooted** by the admin on 2026-06-22 (morning). All ssh/tmux + t3 sessions for
  wizard/emo/anca lost, in-flight agents killed.
- Signature on the last htop before the kill: **load avg ~60** on 32 vCPU, **RAM
  22.5/23.5G**, **swap 13.9/14.0G (full)**, a wall of **D-state** (uninterruptible
  IO-wait) processes, and a single `ugrep` in emo's tmux holding **~10G RES /
  64% CPU**. Many `claude --effort max/xhigh` sessions + playwright-chrome MCP
  instances across three users on top.

## This is the "crawl" class, not the QEMU-stall class

The 2026-06-11 post-mortem (`2026-06-11-devvm-qemu-io-stall.md`) fixed a
*different* failure mode — a QEMU-userspace block-path wedge on the legacy LSI
controller. That fix shipped (verified 2026-06-22: the guest now boots on
`virtio_scsi`, `scsihw: virtio-scsi-single + iothread`). Its post-mortem
explicitly deferred **this** class:

> The recurring *crawl* class (agent storms → swap-thrash; journald
> watchdog-killed 3× on 2026-06-10) is a separate failure mode — ssh/tmux
> sessions remain memory-uncontained by **explicit decision (swap-only,
> 2026-06-10)**.

That explicit decision is the root cause closed here.

## Root cause

Work on the devvm lives in **two independent cgroup-v2 trees per user**, and only
one was capped:

| Tree | cgroup | Cap before today |
|---|---|---|
| t3 web sessions | `system.slice/system-t3\x2dserve.slice/t3-serve@<user>` | `MemoryHigh=12G MemoryMax=16G MemorySwapMax=0 OOMPolicy=continue` ✓ |
| **ssh/tmux sessions** | `user.slice/user-<uid>.slice` | **`MemoryMax=infinity`, swap unlimited** ✗ |

The uncapped `user-<uid>.slice` was the hole. A runaway there (the 10G `ugrep`;
stacked max-effort agents) grew unbounded, spilled into the **14G disk swap**, and
swap-thrashed the **host-mbps-throttled (60/60 MB/s) virtual disk**. That is the
overload chain:

```
uncapped tmux growth → disk-swap thrash on a throttled spindle
   → IO storm (D-state pileup) → load ~60 → box unresponsive → hard kill
```

i.e. **memory pressure becomes the IO storm**. There was also **no global OOM
backstop** (no systemd-oomd / earlyoom) to shed the worst offender before the
kernel OOM or the thrash-wedge. And even the existing t3 caps don't sum safely
(3 users × 16G = 48G > 32G RAM) — nothing reasoned about the *whole box*.

## Fix (`setup-devvm.sh` §10, applied live 2026-06-22)

Design decisions (interviewed with the admin via `/grill-me`): **soft-generous
per-user caps + a hard ceiling + a kill-the-worst backstop**, maximising
single-user utilisation while making a box-wide wedge impossible. (The backstop
was first built on systemd-oomd, then switched to earlyoom mid-rollout when oomd
proved inert with `swap=0` — see Verification + Lessons.)

| Layer | What |
|---|---|
| **Per-user caps, BOTH trees** | `user-.slice.d` drop-in gives every `user-<uid>.slice` the same `MemoryHigh=12G / MemoryMax=16G / MemorySwapMax=0` the t3 tree already had. A user is now bounded in whichever surface they work in. |
| **No disk swap for work** | `MemorySwapMax=0` on every work cgroup → a spike OOMs **locally** at the ceiling instead of thrashing the throttled disk. Kills the IO-storm-via-swap mechanism at the source. The 14G swapfile stays for system cold pages only. |
| **earlyoom backstop (free-RAM threshold)** | New package — used **instead of systemd-oomd** (which is inert with `swap=0`; see Lessons). Watches `MemAvailable%` and SIGTERMs the biggest task at **5%**, SIGKILL at **3%**, swap ignored (`-s 100`). `--avoid` keeps sshd/systemd/dockerd/containerd/t3-dispatch/tmux off the victim list (**the admin's way in always survives**); `--prefer` targets the agent/browser hogs (python3/node/chrome/…). Swap-independent and reliable, where oomd's pressure-kill was not. |
| **Fair-share CPU/IO** | `CPUWeight`/`IOWeight` per slice (system.slice 200, users + docker 100 each). Work-conserving — a lone user still gets all 32 cores + the full IO budget when others idle; weights only bite under contention. No hard CPU/IO caps. |
| **Docker containment** | Containers previously landed in `system.slice` — uncapped. Now `cgroup-parent: docker.slice` in `daemon.json` routes every container into a capped (`MemoryMax=8G`, swap 0) slice, so a runaway container is cgroup-OOM'd locally instead of escaping into the uncapped `system.slice`. |

Durable in `setup-devvm.sh` (survives a VM rebuild); `earlyoom` added to
`packages.txt`. The numbers are tunable — `MemoryHigh=12G` will throttle a *lone*
heavy user between 12–16G even with RAM free; bump to 16/20 if that bites.

## Verification (live, 2026-06-22)

- **Caps live on running cgroups**: all three `user-<uid>.slice` report
  `memory.high=12G memory.max=16G memory.swap.max=0`; `docker.slice` `memory.max=8G`;
  daemon.json kept buildkit/nvidia/insecure-registries; paperless-mcp recovered
  under `docker.slice`.
- **Stress test A (hard cap)** — the PRIMARY guard: a 2G-capped, swap=0 balloon was
  killed at exactly 2G by the cgroup-local OOM (`constraint=CONSTRAINT_MEMCG`) with
  **swap flat at 0MB throughout** — no thrash. Same mechanism protects every user
  slice (16G) and `docker.slice` (8G).
- **Soft cap observed**: a balloon pushed past `MemoryHigh` sat at ~220M / 99%
  memory.pressure, throttled to a crawl, making no progress and harming nothing —
  a runaway is throttled, not just killed.
- **systemd-oomd disproven, then dropped**: a self-policed balloon held
  `memory.pressure full avg10 = 96–99%` (≫ its 20% limit) for >70s but oomd never
  killed it — `Pgscan: 0`. oomd's pressure-kill only acts on cgroups doing active
  reclaim, which a `swap=0` anon workload never does. oomd was purged.
- **earlyoom backstop** — verified via `--dryrun`: at the threshold it logs
  `low memory! … mem 90% swap 100%` (fires on RAM alone, swap ignored) and selects
  `SIGTERM … "chrome"` (a `--prefer` hog), never an `--avoid`'d daemon. Live
  earlyoom v1.7 confirms `SIGTERM mem<=5% / SIGKILL mem<=3%, swap<=100%`.

## Out of scope / follow-ups

- **Alerting** (tracked, fast-follow bead): `DevvmDown` (closes the 90-min
  detection gap the 2026-06-11 PM flagged), sustained-memory-PSI/swap pressure
  early-warning, and an "earlyoom-killed-something" alert (earlyoom logs each kill;
  `-N /script` can push a metric). devvm node-exporter is already scraped
  (`job=devvm`, `10.0.10.10:9100`), so only alert *rules* are new (a
  monitoring-stack Terraform change).
- **zram cushion**: considered, deferred. Could let work cgroups absorb spikes in
  compressed RAM instead of OOMing at the ceiling; not needed for the wedge fix.
- **Per-user docker isolation**: containers share one `docker.slice` budget, not
  per-user. Fine for current usage (krr + short-lived tools).
- **Host-side IO**: the 60/60 mbps cap + the shared `sdc` HDD IO domain are
  host-level (bead `code-oflt`); unchanged here.

## Lessons

- **"Swap as the safety valve" is an IO-storm amplifier on a throttled disk.**
  Leaving ssh/tmux memory-uncontained (the 2026-06-10 decision) traded a clean
  local OOM for a box-wide swap-thrash wedge. `MemorySwapMax=0` + a hard cap turns
  the failure back into a contained, local kill.
- **Cap the box, not one surface.** t3 sessions were capped for months while the
  same user's tmux was unbounded — and the caps that existed didn't sum to < RAM.
  Containment has to reason about every tree and the aggregate.
- **A backstop must protect the operator's way in.** earlyoom `--avoid`s
  sshd/systemd/dockerd/containerd/t3-dispatch/tmux, so the box always stays
  reachable to recover; only the agent/browser hogs are eligible victims.
- **systemd-oomd is the wrong backstop for a no-swap box — verify, don't assume.**
  oomd's memory-pressure killer only fires on cgroups doing active reclaim
  (`pgscan` rising). With `MemorySwapMax=0` + anonymous memory there is nothing to
  reclaim, so a cgroup sat at 99% `memory.pressure` indefinitely and oomd never
  acted (proven with `oomctl` + a balloon). The very `swap=0` that kills the IO
  storm also neuters oomd. earlyoom (free-RAM threshold, swap-independent) is the
  correct pairing. A famous tool that "does OOM" still has to be proven to fire
  under *your* configuration.