From d661d074ef4ec4fbce15060ac82f959167934158 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Fri, 5 Jun 2026 21:48:48 +0000 Subject: [PATCH] matrix: auto-reload Synapse on DB credential rotation (Reloader) Synapse injects the Postgres password into homeserver.yaml only at startup (inject-db-password initContainer). matrix-db-creds is rotated by Vault via ESO (15m refresh), so each rotation left the running pod with a stale password and Synapse DB auth failed silently until a manual rollout restart. Found today via Loki: ~12.9k/hr 'password authentication failed for user matrix' lines; secret password verified working against the DB while the 10-day-old pod held the pre-rotation value. Add the explicit secret.reloader.stakater.com/reload annotation so Reloader rolls the deployment whenever the secret changes (explicit form, not auto/search, because the secret is referenced only in an initContainer env var). Live pod already restarted to restore service; this prevents recurrence on the next rotation. Co-Authored-By: Claude Opus 4.8 --- stacks/matrix/main.tf | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/stacks/matrix/main.tf b/stacks/matrix/main.tf index 1cb62ded..b83fe49a 100644 --- a/stacks/matrix/main.tf +++ b/stacks/matrix/main.tf @@ -98,6 +98,18 @@ resource "kubernetes_deployment" "matrix" { app = "matrix" tier = local.tiers.aux } + annotations = { + # Synapse reads the DB password ONLY at startup: the inject-db-password + # initContainer seds matrix-db-creds into homeserver.yaml. That secret is + # rotated by Vault via the ESO above (15m refresh), so without an + # auto-reload the running pod keeps a stale password and Synapse's DB + # auth fails on every rotation until a manual `rollout restart` (observed + # 2026-06-05). Reloader watches the named secret and rolls the deployment + # when it changes. Explicit form (not auto/search) because the secret is + # referenced only in an initContainer env var, not a mount/envFrom, so + # Reloader's reference auto-discovery is unreliable here. + "secret.reloader.stakater.com/reload" = "matrix-db-creds" + } } spec { replicas = 1