4c532dbf97
t3.viktorbarzin.me went down 2026-07-02 15:42-16:35 UTC: an agent-spawned 12.3G ugrep plateaued inside t3-serve@wizard's MemoryHigh(12G)..MemoryMax(16G) band. With MemorySwapMax=0 its anon pages were unreclaimable, so the kernel throttled every task in the cgroup indefinitely (memory.pressure full ~80%, oom_kill never fired) - the t3 event loop starved, the accept queue rotted, and the terminal was dead until the hog was SIGKILLed by hand. The 2026-06-22 design assumed 'throttle to a crawl, then OOM locally'; a hog that stabilises between high and max never OOMs, so the throttle band is a livelock zone, not a safety layer. Viktor asked to close that gap: MemoryHigh is now explicitly infinity on all three work cgroup definitions (t3-serve@ unit, user-<uid>.slice drop-in, docker.slice) so a runaway is cgroup-OOM- killed at MemoryMax immediately - OOMPolicy=continue already keeps the t3 server alive when a child dies. MemoryMax/MemorySwapMax=0/earlyoom unchanged. Applied live to the devvm the same day (daemon-reload + runtime set-property on running cgroups, no session restarts). Post-mortem addendum + runbook updated in the same commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
44 lines
2 KiB
Desktop File
44 lines
2 KiB
Desktop File
[Unit]
|
|
Description=T3 Code server for %i (t3 serve, per-user)
|
|
Documentation=https://github.com/pingdotgg/t3code
|
|
After=network.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=%i
|
|
Group=%i
|
|
Environment=HOME=/home/%i
|
|
Environment=PATH=/usr/local/bin:/usr/bin:/bin:/home/%i/.local/bin
|
|
Environment=NODE_ENV=production
|
|
EnvironmentFile=/etc/t3-serve/%i.env
|
|
# Optional per-user long-lived CLAUDE_CODE_OAUTH_TOKEN, materialized by
|
|
# claude-auth-sync from the user's own Vault path. Non-rotating, so t3's
|
|
# concurrent agent sessions can't race on OAuth refresh-token rotation and wipe
|
|
# the shared ~/.claude/.credentials.json. Leading '-' = optional (absent for
|
|
# users on the normal per-user Enterprise-SSO credential flow).
|
|
EnvironmentFile=-/home/%i/.config/claude-auth-sync/claude-oauth.env
|
|
WorkingDirectory=/home/%i
|
|
ExecStart=/usr/bin/t3 serve --host 0.0.0.0 --port ${T3_PORT} --base-dir /home/%i/.t3
|
|
Restart=on-failure
|
|
RestartSec=5
|
|
# Memory containment (2026-06-10, amended 2026-07-02): agent children live in
|
|
# this cgroup; a runaway agent (10.8G anon on a 23G host) swap-thrashed the
|
|
# whole devvm — every >20s stall fires the t3 client watchdog (visible
|
|
# "disconnects") — then global-OOMed. Cap the cgroup so a runaway OOMs early
|
|
# and locally, and forbid swap so stalls can't smear into minutes-long freezes.
|
|
# MemoryHigh is DELIBERATELY infinity — do not add a soft band below MemoryMax:
|
|
# with swap=0 a hog that plateaus between high and max is unreclaimable but
|
|
# never OOMs, and the kernel's high-throttle stalls EVERY task in the cgroup
|
|
# (the t3 event loop included) indefinitely. A 12.3G agent ugrep livelocked
|
|
# this unit for ~50min on 2026-07-02 exactly this way. Straight-to-OOM at
|
|
# MemoryMax is the containment; OOMPolicy=continue below keeps the server up.
|
|
# See docs/post-mortems/2026-06-22-devvm-mem-io-overload-containment.md addendum.
|
|
MemoryHigh=infinity
|
|
MemoryMax=16G
|
|
MemorySwapMax=0
|
|
# Default OOMPolicy=stop kills the WHOLE unit (8.5min outage 2026-06-10
|
|
# 19:56) when ANY child is OOM-killed; continue = runaway dies, server stays.
|
|
OOMPolicy=continue
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|