grafana: env-var datasources + reloader so Vault rotations stop breaking dashboards
Wealth, Payslips, and Job-Hunter Grafana datasources all baked the
rotating PG password into their ConfigMap at TF-apply time, so every
7-day Vault static-role rotation silently broke the panels until a
manual `terragrunt apply`. Same family as the recurring grafana-mysql
backend bug — Grafana caches creds at startup and never picks up the
new ESO-synced password without a restart.
Fix:
- Each source stack now creates an ExternalSecret in `monitoring`
exposing the rotating password as `<NAME>_PG_PASSWORD` env-var.
- Grafana mounts those via `envFromSecrets` (optional=true so a
missing source stack doesn't block boot) and the datasource
ConfigMaps reference `$__env{<NAME>_PG_PASSWORD}` instead of a
literal password.
- `reloader.stakater.com/auto: "true"` on the Grafana pod restarts
it whenever any of the four DB-cred Secrets is updated.
Tested end-to-end: forced `vault write -force database/rotate-role/
pg-wealthfolio-sync` → ESO synced (~30s) → reloader fired →
Grafana booted with new env in ~50s total → all three /api/datasources
/uid/*/health endpoints return "Database Connection OK".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
parent
f9f19e4c54
commit
8f0502230b
5 changed files with 147 additions and 28 deletions
|
|
@ -22,6 +22,7 @@ topologySpreadConstraints:
|
|||
app.kubernetes.io/name: grafana
|
||||
podAnnotations:
|
||||
dependency.kyverno.io/wait-for: "mysql.dbaas:3306"
|
||||
reloader.stakater.com/auto: "true"
|
||||
podDisruptionBudget:
|
||||
maxUnavailable: 1
|
||||
persistence:
|
||||
|
|
@ -72,6 +73,19 @@ dashboardProviders:
|
|||
envFromSecrets:
|
||||
- name: grafana-db-creds
|
||||
optional: false
|
||||
# Cross-namespace passwords for provisioned datasources backed by
|
||||
# rotating Vault static-roles. Each source stack creates the secret
|
||||
# via its own ExternalSecret in `monitoring`. `optional: true` lets
|
||||
# Grafana boot if a stack hasn't applied yet; reloader (podAnnotation
|
||||
# above) restarts Grafana when any of these secrets is created or
|
||||
# rotated, so $__env{...} substitution in datasource ConfigMaps stays
|
||||
# current.
|
||||
- name: grafana-wealth-pg-creds
|
||||
optional: true
|
||||
- name: grafana-payslips-pg-creds
|
||||
optional: true
|
||||
- name: grafana-job-hunter-pg-creds
|
||||
optional: true
|
||||
env:
|
||||
GF_SERVER_ROOT_URL: https://grafana.viktorbarzin.me
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue