fc233bd27f
Audited 14 documentation files against live cluster state and Terraform code. Architecture docs: - databases.md: MySQL 8.4.4, proxmox-lvm storage (not iSCSI), anti-affinity excludes k8s-node1 (GPU), 2Gi/3Gi resources, 7-day rotation (not 24h), CNPG 2 instances, PostGIS 16, postgresql.dbaas has endpoints - overview.md: 1x CPU, ~160GB RAM, all nodes 32GB, proxmox-lvm storage, correct Vault paths (secret/ not kv/) - compute.md: 272GB physical host RAM, ~160GB allocated to VMs - secrets.md: 7-day rotation, 7 MySQL + 5 PG roles, correct ESO config - networking.md: MetalLB pool 10.0.20.200-220 - ci-cd.md: 9 GHA projects, travel_blog 5.7GB Runbooks: - restore-mysql/postgresql: backup files are .sql.gz (not .sql) - restore-vault: weekly backup (not daily), auto-unseal sidecar note - restore-vaultwarden: PVC is proxmox (not iscsi) - restore-full-cluster: updated node roles, removed trading Reference docs: - CLAUDE.md: 7-day rotation, removed trading from PG list - AGENTS.md: 100+ stacks, proxmox-lvm, platform empty shell - service-catalog.md: 6 new stacks, 14 stack column updates
3 KiB
3 KiB
Restore Vaultwarden
Prerequisites
kubectlaccess to the cluster- Backup available on NFS at
/mnt/main/vaultwarden-backup/
Backup Location
- NFS:
/mnt/main/vaultwarden-backup/YYYY_MM_DD_HH_MM/(directory per backup) - Each backup contains:
db.sqlite3,rsa_key.pem,rsa_key.pub.pem,attachments/,sends/,config.json - Replicated to Synology NAS (192.168.1.13) via TrueNAS ZFS replication
- Retention: 30 days
- Schedule: Every 6 hours (00:00, 06:00, 12:00, 18:00)
- Integrity check: Both source and backup are verified before/after each backup
Backup Contents
| File | Purpose | Critical? |
|---|---|---|
db.sqlite3 |
All passwords, TOTP seeds, org data | Yes |
rsa_key.pem / rsa_key.pub.pem |
JWT signing keys | Yes — without these, all sessions invalidate |
attachments/ |
File attachments on vault items | Yes |
sends/ |
Bitwarden Send files | No |
config.json |
Server configuration | No — can be recreated |
Restore Procedure
1. Identify the backup to restore
# List available backups (directories sorted by date)
kubectl run vw-ls --rm -it --image=alpine \
--overrides='{"spec":{"volumes":[{"name":"backup","persistentVolumeClaim":{"claimName":"vaultwarden-backup"}}],"containers":[{"name":"vw-ls","image":"alpine","volumeMounts":[{"name":"backup","mountPath":"/backup"}],"command":["ls","-lt","/backup/"]}]}}' \
-n vaultwarden
2. Scale down Vaultwarden
kubectl scale deployment vaultwarden -n vaultwarden --replicas=0
3. Restore the backup
BACKUP_DIR="YYYY_MM_DD_HH_MM" # Set to desired backup
kubectl run vw-restore --rm -it --image=alpine \
--overrides='{"spec":{"volumes":[{"name":"backup","persistentVolumeClaim":{"claimName":"vaultwarden-backup"}},{"name":"data","persistentVolumeClaim":{"claimName":"vaultwarden-data-proxmox"}}],"containers":[{"name":"vw-restore","image":"alpine","volumeMounts":[{"name":"backup","mountPath":"/backup"},{"name":"data","mountPath":"/data"}],"command":["/bin/sh","-c","cp /backup/'$BACKUP_DIR'/db.sqlite3 /data/db.sqlite3 && cp /backup/'$BACKUP_DIR'/rsa_key.pem /data/ && cp /backup/'$BACKUP_DIR'/rsa_key.pub.pem /data/ && cp -a /backup/'$BACKUP_DIR'/attachments /data/ 2>/dev/null; echo Restore complete"]}]}}' \
-n vaultwarden
4. Scale up Vaultwarden
kubectl scale deployment vaultwarden -n vaultwarden --replicas=1
# Wait for pod to be ready
kubectl wait --for=condition=Ready pod -l app=vaultwarden -n vaultwarden --timeout=120s
5. Verify restoration
# Check pod logs for startup errors
kubectl logs -n vaultwarden -l app=vaultwarden --tail=20
# Test web UI access
curl -s -o /dev/null -w "%{http_code}" https://vaultwarden.viktorbarzin.me/
6. Test login
Log in to the Vaultwarden web UI and verify:
- Can log in with your account
- Vault items are present and readable
- Attachments are accessible
- TOTP codes are generating correctly
Estimated Time
- Restore: ~5 minutes
- Verification: ~5 minutes