docs: comprehensive audit and update of all architecture docs and runbooks [ci skip]

Audited 14 documentation files against live cluster state and Terraform code.

Architecture docs:
- databases.md: MySQL 8.4.4, proxmox-lvm storage (not iSCSI), anti-affinity
  excludes k8s-node1 (GPU), 2Gi/3Gi resources, 7-day rotation (not 24h),
  CNPG 2 instances, PostGIS 16, postgresql.dbaas has endpoints
- overview.md: 1x CPU, ~160GB RAM, all nodes 32GB, proxmox-lvm storage,
  correct Vault paths (secret/ not kv/)
- compute.md: 272GB physical host RAM, ~160GB allocated to VMs
- secrets.md: 7-day rotation, 7 MySQL + 5 PG roles, correct ESO config
- networking.md: MetalLB pool 10.0.20.200-220
- ci-cd.md: 9 GHA projects, travel_blog 5.7GB

Runbooks:
- restore-mysql/postgresql: backup files are .sql.gz (not .sql)
- restore-vault: weekly backup (not daily), auto-unseal sidecar note
- restore-vaultwarden: PVC is proxmox (not iscsi)
- restore-full-cluster: updated node roles, removed trading

Reference docs:
- CLAUDE.md: 7-day rotation, removed trading from PG list
- AGENTS.md: 100+ stacks, proxmox-lvm, platform empty shell
- service-catalog.md: 6 new stacks, 14 stack column updates

docs/runbooks/restore-full-cluster.md

@@ -23,7 +23,8 @@ cd infra
 scripts/tg apply stacks/infra
 
 # 2. Wait for VMs to boot and be reachable
-# k8s-master, k8s-node3, k8s-node4, k8s-node5 (node1/2 excluded)
+# k8s-master, k8s-node3, k8s-node4, k8s-node5
+# (node1 has GPU workloads, node2 excluded from MySQL anti-affinity only — both are active cluster members)
 ```
 
 ### Phase 2: Kubernetes Control Plane
@@ -84,7 +85,7 @@ kubectl wait --for=condition=Ready cluster/pg-cluster -n dbaas --timeout=600s
 ### Phase 7: Application Services
 ```bash
 # 17. Deploy remaining stacks in any order
-for stack in vaultwarden immich nextcloud linkwarden trading health; do
+for stack in vaultwarden immich nextcloud linkwarden health; do
   scripts/tg apply stacks/$stack
 done
 

docs/runbooks/restore-mysql.md

@@ -6,7 +6,7 @@
 - Backup dump available on NFS at `/mnt/main/mysql-backup/`
 
 ## Backup Location
-- NFS: `/mnt/main/mysql-backup/dump_YYYY_MM_DD_HH_MM.sql`
+- NFS: `/mnt/main/mysql-backup/dump_YYYY_MM_DD_HH_MM.sql.gz`
 - Replicated to Synology NAS (192.168.1.13) via TrueNAS ZFS replication
 - Retention: 14 days
 - Size: ~11MB per dump
@@ -34,8 +34,8 @@ kubectl port-forward svc/mysql -n dbaas 3307:3306 &
 # Get root password
 ROOT_PWD=$(kubectl get secret cluster-secret -n dbaas -o jsonpath='{.data.ROOT_PASSWORD}' | base64 -d)
 
-# Restore (use --host to avoid unix socket, specify non-default port)
-mysql -u root -p"$ROOT_PWD" --host 127.0.0.1 --port 3307 < /path/to/dump_YYYY_MM_DD_HH_MM.sql
+# Restore (decompress and pipe to mysql, use --host to avoid unix socket, specify non-default port)
+zcat /path/to/dump_YYYY_MM_DD_HH_MM.sql.gz | mysql -u root -p"$ROOT_PWD" --host 127.0.0.1 --port 3307
 ```
 
 ### 3. Option B: Restore via in-cluster pod
@@ -43,7 +43,7 @@ mysql -u root -p"$ROOT_PWD" --host 127.0.0.1 --port 3307 < /path/to/dump_YYYY_MM
 ROOT_PWD=$(kubectl get secret cluster-secret -n dbaas -o jsonpath='{.data.ROOT_PASSWORD}' | base64 -d)
 
 kubectl run mysql-restore --rm -it --image=mysql \
-  --overrides='{"spec":{"volumes":[{"name":"backup","persistentVolumeClaim":{"claimName":"dbaas-mysql-backup"}}],"containers":[{"name":"mysql-restore","image":"mysql","env":[{"name":"MYSQL_PWD","value":"'$ROOT_PWD'"}],"volumeMounts":[{"name":"backup","mountPath":"/backup"}],"command":["mysql","-u","root","--host","mysql.dbaas.svc.cluster.local","<","/backup/dump_YYYY_MM_DD_HH_MM.sql"]}]}}' \
+  --overrides='{"spec":{"volumes":[{"name":"backup","persistentVolumeClaim":{"claimName":"dbaas-mysql-backup"}}],"containers":[{"name":"mysql-restore","image":"mysql","env":[{"name":"MYSQL_PWD","value":"'$ROOT_PWD'"}],"volumeMounts":[{"name":"backup","mountPath":"/backup"}],"command":["/bin/sh","-c","zcat /backup/dump_YYYY_MM_DD_HH_MM.sql.gz | mysql -u root --host mysql.dbaas.svc.cluster.local"]}]}}' \
   -n dbaas
 ```
 
@@ -56,7 +56,7 @@ mysql -u root -p"$ROOT_PWD" --host 127.0.0.1 --port 3307 -e "SHOW DATABASES;"
 mysql -u root -p"$ROOT_PWD" --host 127.0.0.1 --port 3307 -e "SELECT * FROM performance_schema.replication_group_members;"
 
 # Check table counts for key databases
-for db in speedtest wrongmove codimd nextcloud shlink grafana; do
+for db in speedtest wrongmove codimd nextcloud shlink grafana technitium; do
   echo "=== $db ==="
   mysql -u root -p"$ROOT_PWD" --host 127.0.0.1 --port 3307 -e "SELECT TABLE_NAME, TABLE_ROWS FROM information_schema.TABLES WHERE TABLE_SCHEMA='$db' ORDER BY TABLE_ROWS DESC LIMIT 5;"
 done
@@ -79,6 +79,8 @@ kubectl exec -n dbaas mysql-cluster-0 -c mysql -- mysql -u root -p"$ROOT_PWD" \
 #
 # For technitium specifically, also run the password sync CronJob:
 # kubectl create job --from=cronjob/technitium-password-sync technitium-pw-resync -n technitium
+#
+# Note: forgejo and uptimekuma may be legacy users not managed by Vault rotation.
 ```
 
 ### 6. InnoDB Cluster Recovery

docs/runbooks/restore-postgresql.md

@@ -7,7 +7,7 @@
 - PostgreSQL superuser password (from `pg-cluster-superuser` secret in `dbaas` namespace)
 
 ## Backup Location
-- NFS: `/mnt/main/postgresql-backup/dump_YYYY_MM_DD_HH_MM.sql`
+- NFS: `/mnt/main/postgresql-backup/dump_YYYY_MM_DD_HH_MM.sql.gz`
 - Replicated to Synology NAS (192.168.1.13) via TrueNAS ZFS replication
 - Retention: 14 days
 
@@ -34,9 +34,9 @@ kubectl get secret pg-cluster-superuser -n dbaas -o jsonpath='{.data.password}'
 # Port-forward to the CNPG primary
 kubectl port-forward svc/pg-cluster-rw -n dbaas 5433:5432 &
 
-# Restore (this will overwrite existing data)
+# Restore (decompress and pipe to psql — this will overwrite existing data)
 PGPASSWORD=$(kubectl get secret pg-cluster-superuser -n dbaas -o jsonpath='{.data.password}' | base64 -d) \
-  psql -h 127.0.0.1 -p 5433 -U postgres -f /path/to/dump_YYYY_MM_DD_HH_MM.sql
+  zcat /path/to/dump_YYYY_MM_DD_HH_MM.sql.gz | psql -h 127.0.0.1 -p 5433 -U postgres
 ```
 
 ### 3. Option B: Rebuild CNPG cluster from scratch
@@ -56,7 +56,7 @@ kubectl wait --for=condition=Ready cluster/pg-cluster -n dbaas --timeout=300s
 # 5. Restore the dump
 PGPASSWORD=$(kubectl get secret pg-cluster-superuser -n dbaas -o jsonpath='{.data.password}' | base64 -d) \
   kubectl run pg-restore --rm -it --image=postgres:16.4-bullseye \
-  --overrides='{"spec":{"volumes":[{"name":"backup","persistentVolumeClaim":{"claimName":"dbaas-postgresql-backup"}}],"containers":[{"name":"pg-restore","image":"postgres:16.4-bullseye","env":[{"name":"PGPASSWORD","value":"'$PGPASSWORD'"}],"volumeMounts":[{"name":"backup","mountPath":"/backup"}],"command":["psql","-h","pg-cluster-rw.dbaas","-U","postgres","-f","/backup/dump_YYYY_MM_DD_HH_MM.sql"]}]}}' \
+  --overrides='{"spec":{"volumes":[{"name":"backup","persistentVolumeClaim":{"claimName":"dbaas-postgresql-backup"}}],"containers":[{"name":"pg-restore","image":"postgres:16.4-bullseye","env":[{"name":"PGPASSWORD","value":"'$PGPASSWORD'"}],"volumeMounts":[{"name":"backup","mountPath":"/backup"}],"command":["/bin/sh","-c","zcat /backup/dump_YYYY_MM_DD_HH_MM.sql.gz | psql -h pg-cluster-rw.dbaas -U postgres"]}]}}' \
   -n dbaas
 ```
 
@@ -66,7 +66,7 @@ PGPASSWORD=$(kubectl get secret pg-cluster-superuser -n dbaas -o jsonpath='{.dat
 PGPASSWORD=$PGPASSWORD psql -h 127.0.0.1 -p 5433 -U postgres -c "\l"
 
 # Check table counts for critical databases
-for db in trading health linkwarden affine woodpecker claude_memory; do
+for db in health linkwarden affine woodpecker claude_memory; do
   echo "=== $db ==="
   PGPASSWORD=$PGPASSWORD psql -h 127.0.0.1 -p 5433 -U postgres -d $db -c \
     "SELECT schemaname, tablename, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC LIMIT 5;"
@@ -76,10 +76,9 @@ done
 ### 5. Restart dependent services
 After restore, restart services that connect to PostgreSQL to pick up fresh connections:
 ```bash
-kubectl rollout restart deployment -n trading
 kubectl rollout restart deployment -n health
 kubectl rollout restart deployment -n linkwarden
-# ... repeat for all 12 PG-dependent services
+# ... repeat for all PG-dependent services (excluding trading — disabled)
 ```
 
 ## Restore from Synology (if TrueNAS is down)

docs/runbooks/restore-vault.md

@@ -10,7 +10,7 @@
 - NFS: `/mnt/main/vault-backup/vault-raft-YYYYMMDD-HHMMSS.db`
 - Replicated to Synology NAS (192.168.1.13) via TrueNAS ZFS replication
 - Retention: 30 days
-- Schedule: Daily at 02:00
+- Schedule: Weekly on Sundays at 02:00 (`0 2 * * 0`)
 
 ## CRITICAL: Vault is a dependency for many services
 Vault provides secrets to the entire cluster via ESO (External Secrets Operator). A Vault outage affects:
@@ -44,6 +44,11 @@ vault operator raft snapshot restore -force /path/to/vault-raft-YYYYMMDD-HHMMSS.
 ```
 
 ### 3. Unseal Vault (if sealed after restore)
+
+> **Note:** Vault now has an auto-unseal sidecar that automatically unseals pods
+> using the `vault-unseal-key` K8s Secret. The manual procedure below is a
+> fallback if auto-unseal fails.
+
 ```bash
 # Check seal status
 vault status

docs/runbooks/restore-vaultwarden.md

@@ -41,7 +41,7 @@ kubectl scale deployment vaultwarden -n vaultwarden --replicas=0
 BACKUP_DIR="YYYY_MM_DD_HH_MM"  # Set to desired backup
 
 kubectl run vw-restore --rm -it --image=alpine \
-  --overrides='{"spec":{"volumes":[{"name":"backup","persistentVolumeClaim":{"claimName":"vaultwarden-backup"}},{"name":"data","persistentVolumeClaim":{"claimName":"vaultwarden-data-iscsi"}}],"containers":[{"name":"vw-restore","image":"alpine","volumeMounts":[{"name":"backup","mountPath":"/backup"},{"name":"data","mountPath":"/data"}],"command":["/bin/sh","-c","cp /backup/'$BACKUP_DIR'/db.sqlite3 /data/db.sqlite3 && cp /backup/'$BACKUP_DIR'/rsa_key.pem /data/ && cp /backup/'$BACKUP_DIR'/rsa_key.pub.pem /data/ && cp -a /backup/'$BACKUP_DIR'/attachments /data/ 2>/dev/null; echo Restore complete"]}]}}' \
+  --overrides='{"spec":{"volumes":[{"name":"backup","persistentVolumeClaim":{"claimName":"vaultwarden-backup"}},{"name":"data","persistentVolumeClaim":{"claimName":"vaultwarden-data-proxmox"}}],"containers":[{"name":"vw-restore","image":"alpine","volumeMounts":[{"name":"backup","mountPath":"/backup"},{"name":"data","mountPath":"/data"}],"command":["/bin/sh","-c","cp /backup/'$BACKUP_DIR'/db.sqlite3 /data/db.sqlite3 && cp /backup/'$BACKUP_DIR'/rsa_key.pem /data/ && cp /backup/'$BACKUP_DIR'/rsa_key.pub.pem /data/ && cp -a /backup/'$BACKUP_DIR'/attachments /data/ 2>/dev/null; echo Restore complete"]}]}}' \
   -n vaultwarden
 ```