grafana: env-var datasources + reloader so Vault rotations stop breaking dashboards

Wealth, Payslips, and Job-Hunter Grafana datasources all baked the
rotating PG password into their ConfigMap at TF-apply time, so every
7-day Vault static-role rotation silently broke the panels until a
manual `terragrunt apply`. Same family as the recurring grafana-mysql
backend bug — Grafana caches creds at startup and never picks up the
new ESO-synced password without a restart.

Fix:
- Each source stack now creates an ExternalSecret in `monitoring`
  exposing the rotating password as `<NAME>_PG_PASSWORD` env-var.
- Grafana mounts those via `envFromSecrets` (optional=true so a
  missing source stack doesn't block boot) and the datasource
  ConfigMaps reference `$__env{<NAME>_PG_PASSWORD}` instead of a
  literal password.
- `reloader.stakater.com/auto: "true"` on the Grafana pod restarts
  it whenever any of the four DB-cred Secrets is updated.

Tested end-to-end: forced `vault write -force database/rotate-role/
pg-wealthfolio-sync` → ESO synced (~30s) → reloader fired →
Grafana booted with new env in ~50s total → all three /api/datasources
/uid/*/health endpoints return "Database Connection OK".

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

stacks/payslip-ingest/main.tf

@@ -404,18 +404,52 @@ resource "kubernetes_cron_job_v1" "actualbudget_payroll_sync" {
   ]
 }
 
-# Plan-time read of the ESO-created K8s Secret for Grafana datasource password.
-# First apply: -target=kubernetes_manifest.db_external_secret first so the Secret exists.
-data "kubernetes_secret" "payslip_ingest_db_creds" {
-  metadata {
-    name      = "payslip-ingest-db-creds"
-    namespace = kubernetes_namespace.payslip_ingest.metadata[0].name
+# ExternalSecret in the monitoring namespace mirroring the rotating
+# payslip-ingest DB password. Grafana mounts this via envFromSecrets in
+# monitoring/grafana_chart_values.yaml; the datasource ConfigMap below
+# references it as $__env{PAYSLIPS_PG_PASSWORD}. Reloader restarts
+# Grafana whenever ESO updates this secret (every 7d on rotation).
+resource "kubernetes_manifest" "grafana_payslips_db_external_secret" {
+  manifest = {
+    apiVersion = "external-secrets.io/v1beta1"
+    kind       = "ExternalSecret"
+    metadata = {
+      name      = "grafana-payslips-pg-creds"
+      namespace = "monitoring"
+    }
+    spec = {
+      refreshInterval = "15m"
+      secretStoreRef = {
+        name = "vault-database"
+        kind = "ClusterSecretStore"
+      }
+      target = {
+        name = "grafana-payslips-pg-creds"
+        template = {
+          metadata = {
+            annotations = {
+              "reloader.stakater.com/match" = "true"
+            }
+          }
+          data = {
+            PAYSLIPS_PG_PASSWORD = "{{ .password }}"
+          }
+        }
+      }
+      data = [{
+        secretKey = "password"
+        remoteRef = {
+          key      = "static-creds/pg-payslip-ingest"
+          property = "password"
+        }
+      }]
+    }
   }
-  depends_on = [kubernetes_manifest.db_external_secret]
 }
 
 # Grafana datasource for payslip_ingest PostgreSQL DB.
 # Lives in the monitoring namespace so the grafana sidecar (label grafana_datasource=1) picks it up.
+# Password is injected via $__env{...} from grafana-payslips-pg-creds (above).
 resource "kubernetes_config_map" "grafana_payslips_datasource" {
   metadata {
     name      = "grafana-payslips-datasource"
@@ -445,10 +479,11 @@ resource "kubernetes_config_map" "grafana_payslips_datasource" {
           timescaledb     = false
         }
         secureJsonData = {
-          password = data.kubernetes_secret.payslip_ingest_db_creds.data["DB_PASSWORD"]
+          password = "$__env{PAYSLIPS_PG_PASSWORD}"
         }
         editable = true
       }]
     })
   }
+  depends_on = [kubernetes_manifest.grafana_payslips_db_external_secret]
 }