infra/stacks/tts/README.md

tts — Chatterbox TTS (tripit narration)

In-cluster text-to-speech for tripit's "Tour guide". Runs the devnen/Chatterbox-TTS-Server (Resemble AI Chatterbox under an OpenAI-compatible HTTP server) as a single Deployment + ClusterIP Service chatterbox-tts.tts.svc.cluster.local:8000, requesting one time-slice of the shared Tesla T4 (nvidia.com/gpu: 1).

Full design + rationale (Option-A off-peak control, OOM analysis, ADR links): docs/plans/2026-06-08-chatterbox-tts-infra.md (in the tripit-tour-guide repo) and infra/docs/post-mortems/2026-06-02-immich-ml-ttl-gpu-oom-recruiter.md.

This stack mirrors infra/stacks/llama-cpp/. The scaffolding files (backend.tf, providers.tf, cloudflare_provider.tf, tiers.tf, .terraform.lock.hcl) are generated by Terragrunt on init and are git-ignored — only main.tf, terragrunt.hcl and this README are tracked.

---

What this stack creates

• kubernetes_namespace.tts — tier 2-gpu, keel-enrolled, istio off.
• module.nfs_models — RWX NFS-SSD PVC at /srv/nfs-ssd/chatterbox, mounted at /data (predefined voices, narrator reference WAVs, and the HuggingFace model cache via HF_HOME=/data/hf_cache, so weights download once and persist across the per-window pod recreation).
• kubernetes_config_map.chatterbox_config — config.yaml: server.port=8004, model.repo_id=chatterbox-multilingual, tts_engine.device=cuda, voices / reference paths under /data.
• kubernetes_deployment.chatterbox — starts at replicas=0; the off-peak CronJobs own the replica count at runtime. TTS_BF16=off (T4 = Turing, no bf16). priority_class_name=tier-2-gpu (the polite-tenant demotion).
• kubernetes_service.chatterbox — ClusterIP, port 8000 → targetPort 8004 so tripit's default TTS_BASE_URL works unchanged. Prometheus scrape annotations.
• Off-peak control (SA + Role + RoleBinding + 3 CronJobs): see below.

Off-peak control (Option A — window + free-VRAM gate)

The T4 is time-sliced with zero VRAM isolation (post-mortem 2026-06-02), so nvidia.com/gpu: 1 buys a scheduling turn, NOT memory. Chatterbox must only allocate VRAM when the card is actually free. Implemented as three CronJobs (all Europe/London), each a bitnami/kubectl pod using the namespace SA:

CronJob	Schedule (default)	Action
chatterbox-window-up	0 2 * * *	Preflight: scrape gpu_pod_memory_used_bytes from gpu-pod-exporter.nvidia.svc:80/metrics, compute free = 16 GiB − Σused; scale to 1 only if free ≥ vram_free_floor_bytes.
chatterbox-vram-guard	*/5 2-5 * * *	Guard: every 5 min in-window, scale to 0 if free < floor (a resident woke; yield the card mid-bake).
chatterbox-window-down	0 6 * * *	Window end: scale to 0 unconditionally.

tripit's bake is best-effort + cached-forever (ADR-0002/0004) — a skipped or aborted window simply backfills on the next one. No latency SLA.

The free-VRAM floor — YOU MUST MEASURE THIS

var.vram_free_floor_bytes defaults to 6 GiB (a conservative guess: ~4 GiB assumed multilingual FP16 peak + ~2 GiB headroom for the read→cudaMalloc race). The real T4 peak of chatterbox-multilingual is not published upstream. Capture it during the first bake:

# while a real synth is running on the freed T4:
kubectl -n monitoring exec deploy/prometheus -- \
  promtool query instant http://localhost:9090 \
  'sum(gpu_pod_memory_used_bytes{namespace="tts"})'
# or read the gauge straight from the exporter:
kubectl -n nvidia exec ds/gpu-pod-exporter -- \
  sh -c 'curl -s localhost:9401/metrics | grep "namespace=\"tts\""'

Then set the floor to measured_peak + ~2 GiB (pass -var or add to the stack tfvars). If the peak is too high to coexist even off-peak, switch model.repo_id in main.tf to chatterbox (English, lighter) or chatterbox-turbo, or escalate to Option B (scale immich-machine-learning to 0 for the window).

---

Build + push the image (do this BEFORE the first apply)

devnen/Chatterbox-TTS-Server ships no published image — build from the repo's cu128 target (matches the cluster's pinned 570.195.03 / CUDA 12.8 driver) and push to the private Forgejo registry. The devvm docker is pre-authed to forgejo.viktorbarzin.me. Run on the devvm (large CUDA image — needs disk + bandwidth):

# 1. Clone the upstream server repo (outside the monorepo).
git clone https://github.com/devnen/Chatterbox-TTS-Server /tmp/chatterbox-tts-server
cd /tmp/chatterbox-tts-server

# 2. Build the cu128 variant (Dockerfile.cu128 — PyTorch 2.9.0+cu128, the target
#    the repo's docker-compose-cu128.yml uses) for linux/amd64.
SHA="$(git rev-parse --short=8 HEAD)"
docker build \
  --platform linux/amd64 \
  --build-arg RUNTIME=nvidia \
  -f Dockerfile.cu128 \
  -t forgejo.viktorbarzin.me/viktor/chatterbox-tts:latest \
  -t "forgejo.viktorbarzin.me/viktor/chatterbox-tts:${SHA}" \
  .

# 3. Push both tags. (If docker isn't authed: log in with the viktor push PAT
#    from Vault — `vault kv get -field=forgejo_push_token secret/ci/global` —
#    `docker login forgejo.viktorbarzin.me -u viktor`.)
docker push forgejo.viktorbarzin.me/viktor/chatterbox-tts:latest
docker push "forgejo.viktorbarzin.me/viktor/chatterbox-tts:${SHA}"

If Dockerfile.cu128 is not a clean docker build target (e.g. it relies on build args defined only in docker-compose-cu128.yml), lift those args onto the docker build line or docker compose -f docker-compose-cu128.yml build then docker tag the resulting chatterbox-tts-server:cu128 image to the Forgejo ref above before pushing.

---

Apply (admin-gated — run in order)

vault login -method=oidc
~/code/scripts/presence claim node:k8s-node1 --purpose "chatterbox-tts first apply (GPU)"
~/code/scripts/presence claim stack:tts      --purpose "chatterbox-tts stack apply"

# 1. The polite-tenant hardening (exclude tts from gpu-workload priority).
~/code/scripts/tg plan  --stack kyverno
~/code/scripts/tg apply --stack kyverno

# 2. This stack.
~/code/scripts/tg plan  --stack tts
~/code/scripts/tg apply --stack tts        # apply does NOT wake the GPU (replicas=0)

# 3. Flip tripit narration on.
~/code/scripts/tg plan  --stack tripit
~/code/scripts/tg apply --stack tripit

See docs/plans/2026-06-08-chatterbox-tts-infra.md §5 for the full go-live checklist (seed voices on NFS-SSD, smoke-test a synth, watch the neighbours).

Rollback (instant, no data loss)

• Narration off: set TTS_MODE=none (or drop the three TTS_* lines) in stacks/tripit/main.tf → tg apply --stack tripit. The bake makes no audio; playback falls back to browser TTS. Cached story_audio rows are harmless.
• Chatterbox off the GPU: kubectl -n tts scale deploy/chatterbox-tts --replicas=0 (transient) and/or tg destroy --stack tts. Best-effort synth means tripit bakes keep running audio-less — no error.
• Neither touches the resident GPU tenants (Option A never modifies them).