workstation: switch devvm OOM backstop from systemd-oomd to earlyoom
All checks were successful
ci/woodpecker/push/postmortem-todos Pipeline was successful
ci/woodpecker/push/default Pipeline was successful

The systemd-oomd backstop added in the previous commit is INERT on this box.
oomd's memory-pressure kill only acts on cgroups doing active reclaim (pgscan
rising); with MemorySwapMax=0 + anonymous agent memory there is nothing to
reclaim, so pgscan stays 0 and oomd never fires. Proven live: a cgroup held at
96-99% memory.pressure for >70s with pgscan=0 was never killed (oomctl + balloon).
The very swap=0 that kills the IO storm also neuters oomd.

Replace it with earlyoom, which watches free RAM (MemAvailable%) and is
swap-independent: SIGTERM the biggest task at 5%, SIGKILL at 3%, swap ignored
(-s 100). It --avoids sshd/systemd/dockerd/containerd/t3-dispatch/tmux (the
admin's way in always survives) and --prefers the agent/browser hogs. Verified
via --dryrun: fires on the RAM threshold and selects a chrome process, not a
protected daemon.

The per-cgroup caps (MemoryHigh=12G/MemoryMax=16G/MemorySwapMax=0 per user,
docker.slice 8G) are unchanged and remain the PRIMARY guard — earlyoom is the
aggregate net for the rare all-users-maxed case. systemd-oomd purged; its config
+ ManagedOOM drop-ins removed. Post-mortem updated with the finding.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-06-22 10:39:16 +00:00
parent 3a59f4a8bf
commit de163aa6af
3 changed files with 88 additions and 68 deletions

View file

@ -24,10 +24,12 @@ rsync
wget
tree
shellcheck
# resource containment — the systemd-oomd backstop (setup-devvm.sh §10, 2026-06-22):
# a PSI-based, cgroup-aware OOM killer that sheds the single worst work cgroup
# before the box swap-thrashes/wedges. Ships SEPARATELY from core systemd on Ubuntu.
systemd-oomd
# resource containment — earlyoom backstop (setup-devvm.sh §10, 2026-06-22): a
# free-RAM-threshold OOM killer used INSTEAD of systemd-oomd, which is inert with
# swap=0 (its pressure-kill needs reclaim/pgscan that no-swap anon workloads never
# produce; verified live — 99% mem.pressure, pgscan=0, no kill). earlyoom watches
# MemAvailable% and is swap-independent.
earlyoom
# --- installed by setup-devvm.sh via NON-apt paths (not apt-installable) ---
# nodejs + npm -> NodeSource repo (claude-code needs node >= 18; distro nodejs is too old)