viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

devvm containment: drop the MemoryHigh throttle band, straight to MemoryMax OOM
All checks were successful
ci/woodpecker/push/postmortem-todos Pipeline was successful
Details
ci/woodpecker/push/default Pipeline was successful
Details

Browse source 
t3.viktorbarzin.me went down 2026-07-02 15:42-16:35 UTC: an agent-spawned
12.3G ugrep plateaued inside t3-serve@wizard's MemoryHigh(12G)..MemoryMax(16G)
band. With MemorySwapMax=0 its anon pages were unreclaimable, so the kernel
throttled every task in the cgroup indefinitely (memory.pressure full ~80%,
oom_kill never fired) - the t3 event loop starved, the accept queue rotted,
and the terminal was dead until the hog was SIGKILLed by hand.

The 2026-06-22 design assumed 'throttle to a crawl, then OOM locally'; a hog
that stabilises between high and max never OOMs, so the throttle band is a
livelock zone, not a safety layer. Viktor asked to close that gap: MemoryHigh
is now explicitly infinity on all three work cgroup definitions (t3-serve@
unit, user-<uid>.slice drop-in, docker.slice) so a runaway is cgroup-OOM-
killed at MemoryMax immediately - OOMPolicy=continue already keeps the t3
server alive when a child dies. MemoryMax/MemorySwapMax=0/earlyoom unchanged.
Applied live to the devvm the same day (daemon-reload + runtime set-property
on running cgroups, no session restarts). Post-mortem addendum + runbook
updated in the same commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-07-02 16:59:38 +00:00
parent 684ca4527c
commit 4c532dbf97
4 changed files with 82 additions and 19 deletions
Download patch file Download diff file Expand all files Collapse all files

19
scripts/t3-serve@.service
View file

@ -21,12 +21,19 @@ WorkingDirectory=/home/%i
ExecStart=/usr/bin/t3 serve --host 0.0.0.0 --port ${T3_PORT} --base-dir /home/%i/.t3
Restart=on-failure
RestartSec=5
# Memory containment (2026-06-10): agent children live in this cgroup; a
# runaway agent (10.8G anon on a 23G host) swap-thrashed the whole devvm —
# every >20s stall fires the t3 client watchdog (visible "disconnects") —
# then global-OOMed. Cap the cgroup so a runaway OOMs early and locally,
# and forbid swap so stalls can't smear into minutes-long freezes.
MemoryHigh=12G
# Memory containment (2026-06-10, amended 2026-07-02): agent children live in
# this cgroup; a runaway agent (10.8G anon on a 23G host) swap-thrashed the
# whole devvm — every >20s stall fires the t3 client watchdog (visible
# "disconnects") — then global-OOMed. Cap the cgroup so a runaway OOMs early
# and locally, and forbid swap so stalls can't smear into minutes-long freezes.
# MemoryHigh is DELIBERATELY infinity — do not add a soft band below MemoryMax:
# with swap=0 a hog that plateaus between high and max is unreclaimable but
# never OOMs, and the kernel's high-throttle stalls EVERY task in the cgroup
# (the t3 event loop included) indefinitely. A 12.3G agent ugrep livelocked
# this unit for ~50min on 2026-07-02 exactly this way. Straight-to-OOM at
# MemoryMax is the containment; OOMPolicy=continue below keeps the server up.
# See docs/post-mortems/2026-06-22-devvm-mem-io-overload-containment.md addendum.
MemoryHigh=infinity
MemoryMax=16G
MemorySwapMax=0
# Default OOMPolicy=stop kills the WHOLE unit (8.5min outage 2026-06-10