Viktor Barzin af2222fce8 backup & DR: add alerting, fix rotation, secure MySQL password, add runbooks

Phase 1: Add 12 PrometheusRules for backup health alerting
- PostgreSQL, MySQL, Vault, Vaultwarden, Redis staleness + never-succeeded alerts
- CSIDriverCrashLoop alert for nfs-csi/iscsi-csi namespaces
- Generic BackupCronJobFailed alert

Phase 2: Fix backup rotation
- etcd: timestamped snapshots instead of overwriting single file
- Redis: timestamped RDB files with 7-day retention purge
- PostgreSQL: retention increased from 7 to 14 days

Phase 3: Fix MySQL password exposure
- Move root password from command line arg to MYSQL_PWD env var via secretKeyRef

Phase 5: Add restore runbooks
- PostgreSQL, MySQL, Vault, etcd, Vaultwarden, full cluster rebuild

2026-03-19 20:34:33 +00:00

2.8 KiB

Raw Blame History

Restore Vault (Raft)

Prerequisites

kubectl access to the cluster
Vault root token (from vault-root-token secret in vault namespace — manually created, independent of automation)
Raft snapshot available on NFS at /mnt/main/vault-backup/
Unseal keys (stored securely — check secret/viktor in Vault or emergency kit)

Backup Location

NFS: /mnt/main/vault-backup/vault-raft-YYYYMMDD-HHMMSS.db
Replicated to Synology NAS (192.168.1.13) via TrueNAS ZFS replication
Retention: 30 days
Schedule: Daily at 02:00

CRITICAL: Vault is a dependency for many services

Vault provides secrets to the entire cluster via ESO (External Secrets Operator). A Vault outage affects:

All ExternalSecrets (43 secrets + 9 DB-creds secrets)
Vault DB engine password rotation
K8s credentials engine
CI/CD secret sync

Priority: Restore Vault before any other service (except etcd).

Restore Procedure

1. Identify the snapshot to restore

# List available snapshots
ls -lt /mnt/main/vault-backup/vault-raft-*.db | head -10

2. Restore Raft snapshot

# Get root token
VAULT_TOKEN=$(kubectl get secret vault-root-token -n vault -o jsonpath='{.data.vault-root-token}' | base64 -d)

# Port-forward to Vault
kubectl port-forward svc/vault-active -n vault 8200:8200 &

# Restore the snapshot (this will overwrite current state)
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN
vault operator raft snapshot restore -force /path/to/vault-raft-YYYYMMDD-HHMMSS.db

3. Unseal Vault (if sealed after restore)

# Check seal status
vault status

# If sealed, unseal with keys (need threshold number of keys)
vault operator unseal <key1>
vault operator unseal <key2>
vault operator unseal <key3>

4. Verify restoration

# Check Vault health
vault status

# Check raft peers
vault operator raft list-peers

# Verify key secrets exist
vault kv get secret/viktor
vault kv list secret/

# Check DB engine
vault list database/roles

# Check K8s engine
vault list kubernetes/roles

5. Trigger ESO refresh

After Vault restore, ExternalSecrets may need a refresh:

# Restart ESO to force re-sync
kubectl rollout restart deployment -n external-secrets

# Check ExternalSecret status
kubectl get externalsecrets -A | grep -v "SecretSynced"

Full Vault Rebuild (from zero)

If Vault needs to be rebuilt from scratch:

Comment out data sources + OIDC config in stacks/vault/main.tf
Apply Helm release: scripts/tg apply -target=helm_release.vault stacks/vault
Initialize: vault operator init
Unseal with generated keys
Restore raft snapshot (step 2 above)
Populate secret/vault with OIDC credentials
Uncomment data sources + OIDC
Re-apply: scripts/tg apply stacks/vault

Estimated Time

Snapshot restore + unseal: ~10 minutes
Full rebuild: ~30-45 minutes

2.8 KiB Raw Blame History