infra/docs/post-mortems/2026-05-30-redis-split-brain.md

# Post-mortem: Redis split-brain wedged BullMQ/Celery queues (2026-05-30)

**Severity:** SEV2 (degraded — no data loss in Redis; queue processing stalled
cluster-wide). **Status:** Resolved.

## Summary

The 3-node Sentinel HA Redis (`redis-v2`) split-brained: two pods both held
`role:master`. HAProxy — which routes to any backend reporting `role:master` —
round-robined client connections across **both** masters. Immich enqueued
BullMQ jobs on one master while its workers blocked-popped on the other, so
every queue stalled. User-visible symptom: **newly uploaded Immich photos
returned HTTP 404 for their thumbnails** (the generation job never ran). Celery
apps (real-estate-crawler, trading-bot, paperless) and other queue users were
affected the same way.

## Impact

- Immich: thumbnail/preview/face/ML jobs not processing. `facialRecognition`
  backlog reached ~30k waiting; new uploads showed broken images in the web UI.
- All ~15 shared-Redis consumers had inconsistent reads/writes (connections
  split across two diverging masters).
- No Redis data lost — the larger dataset (`redis-v2-0`, ~30k keys) was
  preserved through the fix.

## Timeline (UTC+1 local)

- **~2026-05-26/27**: `redis-v2` pods recreated (node2 unclean reboot era).
  `redis-v2-0` came up partitioned; its Sentinel saw 0 peers and it declared
  itself master via the init script's deterministic "pod-0 = bootstrap master"
  fallback. Sentinels on `-1`/`-2` independently elected `redis-v2-2`.
  Split-brain formed and persisted (~3-4 days) as the network healed but the
  topology never reconciled.
- **2026-05-30 ~16:58**: investigating "Immich images with no thumbnails."
  Found thumbnail jobs failing on missing/zeroed originals (separate pre-existing
  data-loss issue) AND a stuck job queue.
- **2026-05-30 ~17:00**: user manually restarted immich-server; namespace
  `tier-quota` (24Gi) briefly blocked the replacement pod → ~1 min Immich
  outage. Recovered. (Red herring — not the root cause.)
- **2026-05-30 ~17:1x**: identified two `role:master` redis pods
  (`redis-v2-0` dbsize 30320, isolated, 0 connected slaves; `redis-v2-2` dbsize
  442, quorum master). HAProxy fan-out across both = wedged queues. Ruled out
  IPv6 (cluster is single-stack IPv4) and eviction (`evicted_keys=0`).
- **2026-05-30 ~17:30**: reverted `redis-v2` to a single standalone instance.
  Queues drained immediately; newest Immich assets served HTTP 200.

## Root cause

`redis-v2`'s init container (`generate-sentinel-conf`) falls through to
"Priority 3: pod-0 is always the bootstrap master" when it cannot reach peer
Sentinels/Redis. During a network partition, `redis-v2-0` hit that fallback and
became a second master. HAProxy's health check (`tcp-check expect rstring
role:master`) matches **any** master, so with two masters it placed both in
rotation and round-robined writes/reads across diverging datasets. BullMQ's
enqueue (LPUSH) and worker consume (BRPOPLPUSH) landed on different instances →
jobs never consumed.

This is the **third** Sentinel-class incident (after 2026-04-19 PM quorum drift
and 2026-04-22 flap cascade). The 3-sentinel design was built to *prevent*
split-brain, but the bootstrap fallback re-introduced it.

## Resolution

Reverted `redis-v2` to a **single standalone instance** (`replicas=1`, Sentinel
+ HAProxy removed), collapsing onto `redis-v2-0`'s dataset (preserved Immich's
queued jobs). Eviction policy changed `allkeys-lru` → **`volatile-lru`** so the
shared cache+queue workload is served correctly by one instance (evict only
TTL'd cache keys; never TTL-less queue keys). `redis-master` service name/DNS
unchanged → no consumer edits. Decision rationale: a homelab cache/broker does
not need HA; a few-seconds restart blip beats chasing Sentinel correctness.
Mirrors the 2026-04-16 MySQL InnoDB-Cluster → standalone reversion.

## Follow-ups

- [ ] Re-upload the ~99 Immich images + 12 timeline videos whose **originals**
      are missing/zero-filled on disk (pre-existing data loss, unrelated to the
      split-brain — re-running jobs can't regenerate them). Owner: Viktor.
- [ ] `requirepass` auth on Redis + creds rollout to all consumers (carried over
      from the 2026-04-19 rework; still open).
- [ ] Consider whether any queue user (Immich/Celery) warrants its own dedicated
      Redis if the shared instance's memory ever becomes contended (currently
      ~30MB / 640MB — not a concern).

## Lessons

- HA that re-introduces its own failure class is worse than no HA. For a
  single-node-tolerant homelab, prefer a standalone instance + a small accepted
  downtime window.
- `allkeys-lru` on a shared cache+queue Redis silently drops queue jobs under
  pressure; `volatile-lru` is the correct single-instance policy (Immich even
  logs `IMPORTANT! Eviction policy ... should be "noeviction"`).
- A "bootstrap master" fallback that fires under partition is a split-brain
  generator — avoid deterministic self-promotion when peers are unreachable.