grafana: env-var datasources + reloader so Vault rotations stop breaking dashboards

Wealth, Payslips, and Job-Hunter Grafana datasources all baked the
rotating PG password into their ConfigMap at TF-apply time, so every
7-day Vault static-role rotation silently broke the panels until a
manual `terragrunt apply`. Same family as the recurring grafana-mysql
backend bug — Grafana caches creds at startup and never picks up the
new ESO-synced password without a restart.

Fix:
- Each source stack now creates an ExternalSecret in `monitoring`
  exposing the rotating password as `<NAME>_PG_PASSWORD` env-var.
- Grafana mounts those via `envFromSecrets` (optional=true so a
  missing source stack doesn't block boot) and the datasource
  ConfigMaps reference `$__env{<NAME>_PG_PASSWORD}` instead of a
  literal password.
- `reloader.stakater.com/auto: "true"` on the Grafana pod restarts
  it whenever any of the four DB-cred Secrets is updated.

Tested end-to-end: forced `vault write -force database/rotate-role/
pg-wealthfolio-sync` → ESO synced (~30s) → reloader fired →
Grafana booted with new env in ~50s total → all three /api/datasources
/uid/*/health endpoints return "Database Connection OK".

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-05-09 17:38:38 +00:00
parent f9f19e4c54
commit 8f0502230b
5 changed files with 147 additions and 28 deletions

View file

@ -22,6 +22,7 @@ topologySpreadConstraints:
app.kubernetes.io/name: grafana
podAnnotations:
dependency.kyverno.io/wait-for: "mysql.dbaas:3306"
reloader.stakater.com/auto: "true"
podDisruptionBudget:
maxUnavailable: 1
persistence:
@ -72,6 +73,19 @@ dashboardProviders:
envFromSecrets:
- name: grafana-db-creds
optional: false
# Cross-namespace passwords for provisioned datasources backed by
# rotating Vault static-roles. Each source stack creates the secret
# via its own ExternalSecret in `monitoring`. `optional: true` lets
# Grafana boot if a stack hasn't applied yet; reloader (podAnnotation
# above) restarts Grafana when any of these secrets is created or
# rotated, so $__env{...} substitution in datasource ConfigMaps stays
# current.
- name: grafana-wealth-pg-creds
optional: true
- name: grafana-payslips-pg-creds
optional: true
- name: grafana-job-hunter-pg-creds
optional: true
env:
GF_SERVER_ROOT_URL: https://grafana.viktorbarzin.me