# GPU VRAM protection via a scheduler extended-resource budget + a runtime watchdog (HAMi/MPS rejected)

The single Tesla T4 (16 GB, ~15360 MiB usable) on `k8s-node1` is **time-sliced**
(`nvidia.com/gpu` advertised ×100, `migStrategy: none`) and shared by ~9 tenants
(immich-ml, immich-server, frigate, llama-swap, portal-stt, tts,
ebook2audiobook, ytdlp, android-emulator). Time-slicing grants a *scheduling
turn, not memory* — the scheduler is blind to VRAM, so the tenants can
collectively overallocate the card. On 2026-06-02 immich-ml's unbounded
onnxruntime OCR arena grew from ~2 GB to **10.7 GB**, starved llama-swap's
qwen3-8b, and silently broke recruiter-responder triage for ~5 h
(`docs/post-mortems/2026-06-02-immich-ml-ttl-gpu-oom-recruiter.md`). The
post-mortem's #1 follow-up — alert/guard on GPU VRAM — was never built.

## Context

- **MIG is impossible.** The T4 is Turing; hardware memory partitioning (MIG)
  only exists on Ampere+. So per-tenant *hardware* isolation is off the table.
- **The card is busy but not steadily oversubscribed.** Measured steady residents
  (2026-06-17, `gpu_pod_memory_used_bytes`): immich-ml ~2.1 GiB, frigate ~1.9 GiB,
  llama-swap ~4.35 GiB peak (one model at a time — it already swaps), immich-server
  ~1.2 GiB, portal-stt ~1.5 GiB, android-emulator ~0.15 GiB → ~11 GiB used, ~4 GiB
  free. **The failure mode is a single tenant's runtime runaway, not a
  scheduling-time pile-on.**
- **Prior art already exists (soft):** a `gpu-workload` PriorityClass (1,200,000)
  is auto-stamped on every GPU pod by the Kyverno `inject-gpu-workload-priority`
  policy (tts excluded → `tier-2-gpu`, evicted first); tts runs behind a
  free-VRAM demand-gate (`stacks/tts`, scales 0↔1 on `sum(gpu_pod_memory_used_bytes)`
  vs a floor); immich-ml is soft-bounded by `MACHINE_LEARNING_MODEL_TTL=600`. What
  was missing is anything that bounds a tenant's VRAM *during active use*.

### Alternatives considered and rejected

- **NVIDIA MPS** (device-plugin `sharing.mps`, hard `CUDA_MPS_PINNED_DEVICE_MEM_LIMIT`):
  caps are **uniform** — slice = `total ÷ replicas`, tenants get integer multiples.
  Nine heterogeneous tenants spanning 0.15→6 GB do not fit uniform slices without
  large rounding waste on a card that has none to spare. Rejected.
- **HAMi vGPU** (per-container `nvidia.com/gpumem` MiB caps, libvgpu CUDA hook):
  the *correct* hard-cap primitive and T4-supported, but it **replaces the
  operator's device plugin** (the operator owns/reconciles it), enforces via an
  `LD_PRELOAD` CUDA hook that is **unproven for our NVENC transcode path**
  (open codec bug), **cannot cap the android-emulator** (QEMU bypasses the CUDA
  hook — KubeVirt/Kata explicitly unsupported), carries a **restart-triggered
  false-OOM bug** (#1181) directly in our blast radius (kured reboots node1
  regularly), and its reservation-based scheduling would **supersede the working
  demand-gate** and **strand the ~4 GB of steady headroom**. Too much risk and
  behavioral change for the single proven failure mode. Rejected for now; this
  ADR is the record of *why*, so a future "let's just use HAMi" re-opens with the
  trade-offs already on the table.

## Decision

Make the scheduler VRAM-aware and add runtime teeth — entirely with repo-native
pieces, **no device-plugin/driver change, time-slicing untouched**:

1. **Budget (schedule-time).** Advertise a custom node-level **extended resource
   `viktorbarzin.me/gpumem`** on the GPU node (= ~14000 MiB; ~15.4 GB physical
   minus ~1.4 GB driver/CUDA-context/exporter slack), via a reconcile Job +
   CronJob that `kubectl patch node --subresource=status` (dynamic over
   `nvidia.com/gpu.present=true` nodes; re-asserts after node re-register).
   Every GPU tenant declares `resources.limits."viktorbarzin.me/gpumem"` (immich-ml
   3000, llama-swap 5000, frigate 2000, immich-server 1800, portal-stt 1500 — sum
   ≤ advertised). Extended resources are **non-overcommittable** (request==limit,
   integer), so the scheduler refuses to co-schedule past the card → overflow
   `Pending`. On-demand batch tenants (tts/ebook2audiobook/ytdlp) keep the
   free-VRAM demand-gate and fill the real slack rather than holding a reserved seat.
2. **Watchdog (runtime).** A `gpu-vram-watchdog` CronJob (every minute, nvidia ns)
   reads per-pod `gpu_pod_memory_used_bytes` (the host-PID exporter) and each GPU
   pod's *declared* `gpumem`, and **only when actual free VRAM < floor (~1536 MiB)**
   recycles the biggest **over-budget** offender (used > declared). Contract
   enforcement, not priority (immich-ml and llama-swap share `gpu-workload`, so
   priority can't distinguish them). Acting only under pressure lets a tenant burst
   into genuine slack; the recycle clears its arena (exactly what the TTL=600
   Recreate does for immich-ml when idle). This is what would have caught 2026-06-02.
3. **Alerting** (the never-built follow-up): GPU free-VRAM below floor, GPU pod
   `Pending` on `gpumem`, and pod-over-budget → the `#alerts` digest.

This is **soft enforcement**: the scheduler reserves on paper and the watchdog
corrects at runtime with a detection lag (seconds–minute), so a brief physical
overshoot is possible before a recycle. Accepted, given the failure mode is a
slow arena drift, not an instantaneous spike, and the alternative (HAMi) carries
disproportionate risk for this hardware.

## Consequences

- **The 2026-06-02 class is bounded** without touching the pinned driver, the GPU
  operator, or time-slicing. immich-ml can no longer silently grow into
  llama-swap's VRAM: it either schedules within its budget or, on a true runaway
  under pressure, gets recycled (its heavy library job is the intended loser).
- **The card has a seating chart now.** Sum of declared budgets ≤ ~14 GB, so a new
  always-on GPU tenant requires re-budgeting; an over-budget on-demand tenant sits
  `Pending`. This is the intended, legible back-pressure.
- **Small/on-demand tenants (android-emulator, ytdlp, tts, ebook2audiobook) are
  NOT budgeted in v1** — they fill *actual* slack rather than holding a scheduler
  seat (tts via its existing free-VRAM demand-gate), and are covered by the
  ~1.4 GiB physical reserve plus budget headroom (the five residents' budgets sum
  to 13300 ≤ 14000 advertised). Give them budgets later if they grow; until then
  the watchdog protects the budgeted five and counts everyone's usage toward free.
- **New RBAC:** the reconcile SA patches `nodes/status`; the watchdog SA lists pods
  cluster-wide and deletes pods in GPU tenant namespaces. Far less privileged than
  existing cluster-admin tooling (woodpecker-agent).
- **Apply order matters:** advertise `gpumem` (nvidia stack) **before** the
  consumer stacks declare it, or a pod requesting an unadvertised extended
  resource is unschedulable. The reconcile runs as a Job (immediate) for this.
- **Fully reversible:** delete the CronJobs/Job + the `gpumem` stanzas, and
  `kubectl patch node --subresource=status` to remove the capacity key. Nothing
  structural; no driver/operator state to unwind.
- The `gpumem` numbers are first estimates; tune from `gpu_pod_memory_used_bytes`.