deprecate TrueNAS: migrate Immich NFS to Proxmox, remove all 10.0.10.15 references [ci skip]
- Migrate Immich (8 NFS PVs, 1.1TB) from TrueNAS to Proxmox host NFS - Update config.tfvars nfs_server to 192.168.1.127 (Proxmox) - Update nfs-csi StorageClass share to /srv/nfs - Update scripts (weekly-backup, cluster-healthcheck) to Proxmox IP - Delete obsolete TrueNAS scripts (nfs_exports.sh, truenas-status.sh) - Rewrite nfs-health.sh for Proxmox NFS monitoring - Update Freedify nfs_music_server default to Proxmox - Mark CloudSync monitor CronJob as deprecated - Update Prometheus alert summaries - Update all architecture docs, AGENTS.md, and reference docs - Zero PVs remain on TrueNAS — VM ready for decommission Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
69248eaa7b
commit
38d51ab0af
20 changed files with 245 additions and 524 deletions
|
|
@ -1,6 +1,6 @@
|
|||
# Backup & Disaster Recovery Architecture
|
||||
|
||||
Last updated: 2026-04-06
|
||||
Last updated: 2026-04-13
|
||||
|
||||
## Overview
|
||||
|
||||
|
|
@ -9,9 +9,8 @@ The homelab uses a defense-in-depth 3-2-1 backup strategy: **3 copies** (live PV
|
|||
**3-2-1 Breakdown**:
|
||||
- **Copy 1** (live): All PVC data + VM disks on Proxmox sdc thin pool (10.7TB RAID1 HDD)
|
||||
- **Copy 2** (local backup): Weekly file-level backup to sda `/mnt/backup` (1.1TB RAID1 SAS)
|
||||
- **Copy 3** (offsite): Synology NAS at 192.168.1.13 via two paths:
|
||||
- **Copy 3** (offsite): Synology NAS at 192.168.1.13:
|
||||
- `Synology/Backup/Viki/pve-backup/` — structured PVE host backups (rsync --files-from weekly)
|
||||
- `Synology/Backup/Viki/truenas/` — TrueNAS NFS media (Cloud Sync, narrowed to media only)
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
|
|
@ -42,9 +41,8 @@ graph TB
|
|||
PVEConfig --> sda
|
||||
end
|
||||
|
||||
subgraph TrueNAS["TrueNAS (10.0.10.15)"]
|
||||
NFS_Backup["NFS-exported<br/>/mnt/main/*-backup/"]
|
||||
Media["Media (NFS)<br/>Immich ~800GB<br/>audiobookshelf, servarr, navidrome"]
|
||||
subgraph NFS_Storage["Proxmox NFS (/srv/nfs)"]
|
||||
NFS_Backup["NFS dirs<br/>/srv/nfs/*-backup/"]
|
||||
|
||||
subgraph AppBackups["App-Level Backup CronJobs"]
|
||||
CronDaily["Daily 00:00-00:30<br/>PostgreSQL, MySQL<br/>14d retention"]
|
||||
|
|
@ -57,16 +55,13 @@ graph TB
|
|||
|
||||
subgraph Layer3["Layer 3: Offsite Sync"]
|
||||
PVEOffsite["PVE → Synology<br/>Sunday 08:00<br/>rsync --files-from<br/>/Backup/Viki/pve-backup/"]
|
||||
CloudSync["TrueNAS → Synology<br/>Monday 09:00<br/>Cloud Sync (media only)<br/>/Backup/Viki/truenas/"]
|
||||
end
|
||||
|
||||
sda --> PVEOffsite
|
||||
Media --> CloudSync
|
||||
|
||||
Synology["Synology NAS<br/>192.168.1.13<br/>Offsite protection"]
|
||||
|
||||
PVEOffsite --> Synology
|
||||
CloudSync --> Synology
|
||||
|
||||
NFS_Backup -.->|mirrored to sda| NFSMirror
|
||||
|
||||
|
|
@ -100,14 +95,7 @@ graph LR
|
|||
|
||||
S01 --> S02 --> S03a --> S03b --> S05 --> S08
|
||||
|
||||
subgraph Monday["Monday"]
|
||||
M09["09:00 TrueNAS Cloud Sync<br/>Media → Synology"]
|
||||
end
|
||||
|
||||
S08 -.->|next day| M09
|
||||
|
||||
style Sunday fill:#ffe0b2
|
||||
style Monday fill:#e1f5ff
|
||||
```
|
||||
|
||||
### Physical Disk Layout
|
||||
|
|
@ -159,7 +147,7 @@ graph TB
|
|||
|
||||
Type -->|"Database"| AppBackup["Use app-level dump<br/>/mnt/backup/nfs-mirror/<service>-backup/<br/>OR Synology/pve-backup/nfs-mirror/<br/>RTO: <10 min"]
|
||||
Type -->|"PVC files"| Proceed["Proceed with<br/>selected restore method"]
|
||||
Type -->|"Media (NFS)"| CloudSync["Use Synology backup<br/>Synology/truenas/<service>/<br/>RTO: varies by size"]
|
||||
Type -->|"Media (NFS)"| OffsiteMedia["Use Synology backup<br/>Synology/pve-backup/nfs-mirror/<br/>RTO: varies by size"]
|
||||
|
||||
style Start fill:#ffcdd2
|
||||
style LVM fill:#c8e6c9
|
||||
|
|
@ -213,7 +201,7 @@ graph LR
|
|||
| Vault Backup | Weekly Sunday 02:00, 30d | CronJob in `vault` | raft snapshot |
|
||||
| Redis Backup | Weekly Sunday 03:00, 30d | CronJob in `redis` | BGSAVE + copy |
|
||||
| Vaultwarden Integrity Check | Hourly | CronJob in `vaultwarden` | PRAGMA integrity_check → metric |
|
||||
| TrueNAS Cloud Sync | Monday 09:00 (weekly) | TrueNAS Cloud Sync Task 1 | Media → Synology NAS |
|
||||
| ~~TrueNAS Cloud Sync~~ | **DECOMMISSIONED** | Was TrueNAS Cloud Sync Task 1 | Replaced by offsite-sync-backup (Path 1) |
|
||||
|
||||
## How It Works
|
||||
|
||||
|
|
@ -251,7 +239,7 @@ Native LVM thin snapshots provide crash-consistent point-in-time recovery for 62
|
|||
- 4 weekly versions with `--link-dest` hardlink dedup (unchanged files share inodes)
|
||||
|
||||
**2. NFS Backup Mirror** (`/mnt/backup/nfs-mirror/`):
|
||||
- Mount TrueNAS NFS ro → rsync DB dump dirs → unmount
|
||||
- Rsync DB dump dirs from Proxmox NFS (`/srv/nfs/*-backup/`)
|
||||
- Covers: `mysql-backup/`, `postgresql-backup/`, `vault-backup/`, `vaultwarden-backup/`, `redis-backup/`, `etcd-backup/`
|
||||
- Single copy (no rotation) — latest dump only
|
||||
|
||||
|
|
@ -274,11 +262,11 @@ Native LVM thin snapshots provide crash-consistent point-in-time recovery for 62
|
|||
|
||||
### Layer 2b: Application-Level Backups
|
||||
|
||||
K8s CronJobs run inside the cluster, dumping database/state to NFS-exported backup directories. Each service writes to `/mnt/main/<service>-backup/`.
|
||||
K8s CronJobs run inside the cluster, dumping database/state to NFS-exported backup directories. Each service writes to `/srv/nfs/<service>-backup/` (some legacy paths still use `/mnt/main/<service>-backup/`).
|
||||
|
||||
**Why needed**: LVM snapshots capture block-level state, but:
|
||||
- Cannot restore individual databases from a PostgreSQL snapshot
|
||||
- Proxmox CSI LVs are opaque to TrueNAS (raw block devices)
|
||||
- Proxmox CSI LVs are opaque raw block devices
|
||||
- Need point-in-time recovery for specific apps without full LVM rollback
|
||||
|
||||
**Daily backups (00:00-00:30)**:
|
||||
|
|
@ -331,28 +319,9 @@ Two independent paths push backups offsite:
|
|||
|
||||
**Monitoring**: Pushes `offsite_backup_sync_last_success_timestamp` to Pushgateway. Alerts: `OffsiteBackupSyncStale` (>8d), `OffsiteBackupSyncFailing`.
|
||||
|
||||
#### Path 2: TrueNAS Media (Cloud Sync)
|
||||
#### ~~Path 2: TrueNAS Media (Cloud Sync)~~ — DECOMMISSIONED
|
||||
|
||||
**Task**: TrueNAS Cloud Sync Task 1 runs `rclone sync` Monday 09:00
|
||||
**Source**: `/mnt/main/` (NFS pool on TrueNAS)
|
||||
**Destination**: `sftp://192.168.1.13/Backup/Viki/truenas`
|
||||
**Scope**: Media libraries only (Immich ~800GB, audiobookshelf, servarr, navidrome music)
|
||||
|
||||
**Excludes** (Cloud Sync configured to skip):
|
||||
- `clickhouse/**` (2.47M files, regenerable)
|
||||
- `loki/**` (68K files, regenerable)
|
||||
- `prometheus/**` (covered by monthly app backup)
|
||||
- `frigate/**` (ephemeral recordings)
|
||||
- `audiblez/**`, `ebook2audiobook/**` (regenerable)
|
||||
- `ollama/**` (chat history, low value)
|
||||
- `real-estate-crawler/**` (regenerable)
|
||||
- `crowdsec/**` (regenerable)
|
||||
- `servarr/downloads/**` (transient)
|
||||
- `ytldp/**` (replaceable)
|
||||
- `iscsi/**`, `iscsi-snaps/**` (raw zvols, backed at app level)
|
||||
- `*-backup/**` (already mirrored via Path 1)
|
||||
|
||||
**Monitoring**: Existing `CloudSyncStale`, `CloudSyncNeverRun`, `CloudSyncFailing` alerts still apply.
|
||||
> TrueNAS Cloud Sync was decommissioned along with TrueNAS (2026-04). Media offsite backup is now handled by the Proxmox host `offsite-sync-backup` script (Path 1) which includes NFS media directories in its manifest. The `Synology/Backup/Viki/truenas/` directory on the Synology NAS contains the last Cloud Sync snapshot and is no longer updated.
|
||||
|
||||
## Configuration
|
||||
|
||||
|
|
@ -488,12 +457,12 @@ df -h /mnt/backup
|
|||
**Common causes**:
|
||||
- Backup disk full (check `df -h /mnt/backup`, alert: `BackupDiskFull`)
|
||||
- LV mount failed (check `lvs pve`, `dmesg | grep backup`)
|
||||
- NFS mount failed (check `showmount -e 10.0.10.15`)
|
||||
- NFS mount failed (check `showmount -e 192.168.1.127`)
|
||||
|
||||
**Fix**:
|
||||
1. If disk full: Clean up old weekly versions manually, adjust retention
|
||||
2. If LV mount failed: `lvchange -ay backup/data && mount /mnt/backup`
|
||||
3. If NFS failed: Check TrueNAS availability, verify exports
|
||||
3. If NFS failed: Check Proxmox NFS availability (`showmount -e 192.168.1.127`), verify exports
|
||||
4. Manually trigger: `systemctl start weekly-backup.service`
|
||||
|
||||
### Offsite Sync Failing
|
||||
|
|
@ -531,12 +500,12 @@ kubectl logs -n dbaas job/postgresql-backup-<timestamp>
|
|||
|
||||
**Common causes**:
|
||||
- Pod OOMKilled (increase memory limit)
|
||||
- NFS mount unavailable (check TrueNAS)
|
||||
- NFS mount unavailable (check Proxmox NFS)
|
||||
- pg_dumpall command failed (check PostgreSQL connectivity)
|
||||
|
||||
**Fix**:
|
||||
1. If OOM: Increase `resources.limits.memory` in `stacks/dbaas/backup.tf`
|
||||
2. If NFS: Verify mount on worker node, restart NFS server if needed
|
||||
2. If NFS: Verify mount on worker node, restart NFS server on Proxmox host if needed (`systemctl restart nfs-server`)
|
||||
3. Manually trigger: `kubectl create job --from=cronjob/postgresql-backup manual-backup -n dbaas`
|
||||
|
||||
### Vaultwarden Integrity Check Failing
|
||||
|
|
@ -662,7 +631,7 @@ module "nfs_backup" {
|
|||
name = "${var.service_name}-backup"
|
||||
namespace = kubernetes_namespace.service.metadata[0].name
|
||||
nfs_server = var.nfs_server
|
||||
nfs_path = "/mnt/main/${var.service_name}-backup"
|
||||
nfs_path = "/srv/nfs/${var.service_name}-backup"
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -678,9 +647,9 @@ module "nfs_backup" {
|
|||
│ VaultBackupStale > 8d since last success │
|
||||
│ VaultwardenBackupStale > 8d since last success │
|
||||
│ RedisBackupStale > 8d since last success │
|
||||
│ CloudSyncStale > 8d since last success │
|
||||
│ CloudSyncNeverRun task never completed │
|
||||
│ CloudSyncFailing task in error state │
|
||||
│ ~~CloudSyncStale~~ REMOVED (TrueNAS decommissioned) │
|
||||
│ ~~CloudSyncNeverRun~~ REMOVED (TrueNAS decommissioned) │
|
||||
│ ~~CloudSyncFailing~~ REMOVED (TrueNAS decommissioned) │
|
||||
│ VaultwardenIntegrityFail integrity_ok == 0 │
|
||||
│ LVMSnapshotStale > 24h since last snapshot │
|
||||
│ LVMSnapshotFailing snapshot creation failed │
|
||||
|
|
@ -698,7 +667,7 @@ module "nfs_backup" {
|
|||
- LVM snapshot script: Pushes `lvm_snapshot_last_success_timestamp`, `lvm_snapshot_count`, `lvm_thin_pool_free_percent`
|
||||
- Weekly backup script: Pushes `backup_weekly_last_success_timestamp`, `backup_disk_usage_percent`
|
||||
- Offsite sync script: Pushes `offsite_backup_sync_last_success_timestamp`
|
||||
- CloudSync monitor: Queries TrueNAS API every 6h, pushes `cloudsync_last_success_timestamp`
|
||||
- ~~CloudSync monitor~~: Removed (TrueNAS decommissioned)
|
||||
- Vaultwarden integrity: Pushes `vaultwarden_sqlite_integrity_ok` hourly
|
||||
|
||||
**Alert routing**:
|
||||
|
|
@ -738,7 +707,7 @@ module "nfs_backup" {
|
|||
- — = Not needed (other layers cover it, or data is regenerable/disposable)
|
||||
- excluded = Too large/regenerable, not worth offsite bandwidth
|
||||
|
||||
**Note**: All 65 proxmox-lvm PVCs get LVM snapshots (except dbaas+monitoring = 3 PVCs) + file-level backup (except dbaas+monitoring). NFS-backed media relies on TrueNAS Cloud Sync for offsite.
|
||||
**Note**: All 65 proxmox-lvm PVCs get LVM snapshots (except dbaas+monitoring = 3 PVCs) + file-level backup (except dbaas+monitoring). NFS-backed media is included in the Proxmox host weekly-backup offsite sync.
|
||||
|
||||
## Recovery Procedures
|
||||
|
||||
|
|
@ -761,7 +730,7 @@ Detailed runbooks in `docs/runbooks/`:
|
|||
- Vault: <10 min
|
||||
- Vaultwarden: <5 min
|
||||
- etcd: <20 min (requires cluster rebuild)
|
||||
- Full cluster from offsite: <4 hours (TrueNAS restore + K8s bootstrap + app deploys)
|
||||
- Full cluster from offsite: <4 hours (NFS restore + K8s bootstrap + app deploys)
|
||||
|
||||
## Related
|
||||
|
||||
|
|
|
|||
|
|
@ -23,7 +23,6 @@ graph TB
|
|||
NODE3["k8s-node3<br/>203"]
|
||||
NODE4["k8s-node4<br/>204"]
|
||||
REG["docker-registry<br/>220"]
|
||||
TN["TrueNAS<br/>9000"]
|
||||
end
|
||||
|
||||
subgraph Network["Network Bridges"]
|
||||
|
|
@ -48,7 +47,6 @@ graph TB
|
|||
PF --> VMBR1_20
|
||||
HA --> VMBR0
|
||||
DEV --> VMBR1_10
|
||||
TN --> VMBR1_10
|
||||
|
||||
MASTER --> VMBR1_20
|
||||
NODE1 --> VMBR1_20
|
||||
|
|
@ -78,7 +76,7 @@ graph TB
|
|||
| Network | VLAN | CIDR | Purpose |
|
||||
|---------|------|------|---------|
|
||||
| Physical | - | 192.168.1.0/24 | Physical devices, Proxmox host (192.168.1.127) |
|
||||
| Management | 10 | 10.0.10.0/24 | Infrastructure VMs, TrueNAS, devvm |
|
||||
| Management | 10 | 10.0.10.0/24 | Infrastructure VMs, devvm |
|
||||
| Kubernetes | 20 | 10.0.20.0/24 | K8s cluster nodes and services |
|
||||
|
||||
### Virtual Machine Inventory
|
||||
|
|
@ -94,7 +92,7 @@ graph TB
|
|||
| 203 | k8s-node3 | 8 | 32GB | vmbr1:vlan20 | - | Worker node |
|
||||
| 204 | k8s-node4 | 8 | 32GB | vmbr1:vlan20 | - | Worker node |
|
||||
| 220 | docker-registry | 4 | 4GB | vmbr1:vlan20 | 10.0.20.10 | Private Docker registry |
|
||||
| 9000 | truenas | 16 | 16GB | vmbr1:vlan10 | 10.0.10.15 | NFS storage server |
|
||||
| ~~9000~~ | ~~truenas~~ | — | — | — | ~~10.0.10.15~~ | **DECOMMISSIONED** — NFS now served by Proxmox host (192.168.1.127) |
|
||||
|
||||
### Kubernetes Cluster
|
||||
|
||||
|
|
@ -103,7 +101,7 @@ graph TB
|
|||
| Version | v1.34.2 |
|
||||
| Nodes | 5 (1 control plane, 4 workers) |
|
||||
| CNI | Calico |
|
||||
| Storage | NFS (democratic-csi) + Proxmox-LVM (Proxmox CSI) |
|
||||
| Storage | NFS (Proxmox host, nfs-csi) + Proxmox-LVM (Proxmox CSI) |
|
||||
| Ingress | Traefik v3 |
|
||||
| Total Services | 70+ services across 5 tiers |
|
||||
|
||||
|
|
@ -164,8 +162,8 @@ Kyverno policies automatically inject namespace labels, LimitRange, ResourceQuot
|
|||
- **Headscale**: Tailscale-compatible mesh VPN control plane
|
||||
|
||||
**Storage & Security**:
|
||||
- **TrueNAS**: NFS storage backend (10.0.10.15)
|
||||
- **democratic-csi**: Dynamic PV provisioning from TrueNAS
|
||||
- **Proxmox NFS**: NFS storage served directly from Proxmox host (192.168.1.127) at `/srv/nfs` (HDD) and `/srv/nfs-ssd` (SSD)
|
||||
- **Proxmox CSI**: Block storage via LVM-thin hotplug for databases
|
||||
- **Vaultwarden**: Password manager
|
||||
- **Immich**: Photo management
|
||||
- **CrowdSec**: IPS/IDS with community threat intelligence
|
||||
|
|
|
|||
|
|
@ -1,18 +1,24 @@
|
|||
# Storage Architecture
|
||||
|
||||
Last updated: 2026-04-06
|
||||
Last updated: 2026-04-13
|
||||
|
||||
## Overview
|
||||
|
||||
The cluster uses two storage backends: **Proxmox CSI** for database block storage and **TrueNAS NFS** for application data.
|
||||
The cluster uses two storage backends: **Proxmox CSI** for database block storage and **Proxmox NFS** for application data.
|
||||
|
||||
**Block storage (Proxmox CSI)**: 65 PVCs for databases and stateful apps (CNPG PostgreSQL, MySQL InnoDB, Redis, Vaultwarden, Prometheus, Nextcloud, Calibre-Web, Forgejo, FreshRSS, ActualBudget, NovelApp, Headscale, Uptime Kuma, etc.) use `StorageClass: proxmox-lvm`, which provisions thin LVs directly from the Proxmox host's `local-lvm` storage (sdc, 10.7TB RAID1 HDD thin pool). This eliminates the previous double-CoW (ZFS + LVM-thin) path that caused 56 ZFS checksum errors.
|
||||
|
||||
**NFS storage (TrueNAS)**: ~100 NFS shares for media libraries (Immich, audiobookshelf, servarr, navidrome), backup targets (`*-backup/` directories), and legacy app data continue to use TrueNAS ZFS at `10.0.10.15` via `StorageClass: nfs-truenas`.
|
||||
**NFS storage (Proxmox host)**: ~100 NFS shares for media libraries (Immich, audiobookshelf, servarr, navidrome), backup targets (`*-backup/` directories), and app data are served directly from the Proxmox host at `192.168.1.127`. Two NFS export roots exist:
|
||||
- **HDD NFS**: `/srv/nfs` on ext4 LV `pve/nfs-data` (2TB) — bulk media and backup targets
|
||||
- **SSD NFS**: `/srv/nfs-ssd` on ext4 LV `ssd/nfs-ssd-data` (100GB) — high-performance data (Immich ML)
|
||||
|
||||
Both `StorageClass: nfs-truenas` (name kept for compatibility) and `StorageClass: nfs-proxmox` (identical) point to the Proxmox host. Migrated from TrueNAS (10.0.10.15) which has been fully decommissioned.
|
||||
|
||||
**Backup storage (sda)**: 1.1TB RAID1 SAS disk, VG `backup`, LV `data` (ext4), mounted at `/mnt/backup` on PVE host. Dedicated backup disk for weekly PVC file backups, NFS mirrors, pfSense backups, and PVE config. Independent of live storage (sdc).
|
||||
|
||||
**Migration (2026-04-02)**: All iSCSI block volumes were migrated from democratic-csi (TrueNAS iSCSI → ZFS → LVM-thin) to Proxmox CSI (direct LVM-thin hotplug). democratic-csi iSCSI driver is deprecated and pending removal.
|
||||
**Migration (2026-04-02)**: All iSCSI block volumes were migrated from democratic-csi (TrueNAS iSCSI → ZFS → LVM-thin) to Proxmox CSI (direct LVM-thin hotplug). democratic-csi iSCSI driver has been removed.
|
||||
|
||||
**Migration (2026-04)**: TrueNAS (10.0.10.15) fully decommissioned. All NFS storage migrated to the Proxmox host (192.168.1.127). ZFS datasets under `/mnt/main/` and `/mnt/ssd/` moved to ext4 LVs at `/srv/nfs/` and `/srv/nfs-ssd/`. Legacy PVs referencing `/mnt/main/` paths still work (bind-mounted or symlinked on the Proxmox host); new PVs use `/srv/nfs/` and `/srv/nfs-ssd/`.
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
|
|
@ -21,43 +27,37 @@ graph TB
|
|||
subgraph Proxmox["Proxmox Host (192.168.1.127)"]
|
||||
sdc["sdc: 10.7TB RAID1 HDD<br/>VG pve, LV data (thin pool)<br/>65 proxmox-lvm PVCs"]
|
||||
sda["sda: 1.1TB RAID1 SAS<br/>VG backup, LV data (ext4)<br/>/mnt/backup"]
|
||||
end
|
||||
|
||||
subgraph TrueNAS["TrueNAS (10.0.10.15)<br/>VMID 9000, 16c/16GB"]
|
||||
ZFS_Main["ZFS Pool: main<br/>1.64 TiB<br/>32G + 7x256G + 1T disks"]
|
||||
ZFS_SSD["ZFS Pool: ssd<br/>~256GB SSD<br/>Immich ML, PostgreSQL hot data"]
|
||||
|
||||
ZFS_Main --> NFS_Datasets["NFS Datasets<br/>~100 shares<br/>main/<service><br/>Media + backup targets"]
|
||||
|
||||
NFS_Datasets --> NFS_Exports["NFS Exports<br/>managed by secrets/nfs_exports.sh"]
|
||||
|
||||
ZFS_SSD --> SSD_Data["Immich ML models"]
|
||||
NFS_HDD["LV pve/nfs-data (2TB ext4)<br/>/srv/nfs<br/>~100 NFS shares<br/>Media + backup targets"]
|
||||
NFS_SSD["LV ssd/nfs-ssd-data (100GB ext4)<br/>/srv/nfs-ssd<br/>High-performance data<br/>(Immich ML)"]
|
||||
NFS_Exports["NFS Exports<br/>managed by /etc/exports"]
|
||||
NFS_HDD --> NFS_Exports
|
||||
NFS_SSD --> NFS_Exports
|
||||
end
|
||||
|
||||
subgraph K8s["Kubernetes Cluster"]
|
||||
CSI_NFS["democratic-csi-nfs<br/>StorageClass: nfs-truenas<br/>soft,timeo=30,retrans=3"]
|
||||
CSI_iSCSI["democratic-csi-iscsi<br/>StorageClass: iscsi-truenas<br/>SSH driver"]
|
||||
CSI_NFS["nfs-csi driver<br/>StorageClass: nfs-truenas / nfs-proxmox<br/>soft,timeo=30,retrans=3"]
|
||||
CSI_PVE["Proxmox CSI plugin<br/>StorageClass: proxmox-lvm"]
|
||||
|
||||
NFS_PV["NFS PersistentVolumes<br/>RWX, ~100 volumes"]
|
||||
iSCSI_PV["iSCSI PersistentVolumes<br/>RWO, ~19 volumes"]
|
||||
Block_PV["Block PersistentVolumes<br/>RWO, 65 PVCs"]
|
||||
|
||||
Pods["Application Pods"]
|
||||
DBPods["Database Pods<br/>PostgreSQL CNPG<br/>MySQL InnoDB"]
|
||||
end
|
||||
|
||||
NFS_Exports -->|CSI driver| CSI_NFS
|
||||
iSCSI_Targets -->|CSI driver| CSI_iSCSI
|
||||
NFS_Exports -->|NFS mount| CSI_NFS
|
||||
sdc -->|LVM-thin hotplug| CSI_PVE
|
||||
|
||||
CSI_NFS --> NFS_PV
|
||||
CSI_iSCSI --> iSCSI_PV
|
||||
CSI_PVE --> Block_PV
|
||||
|
||||
NFS_PV --> Pods
|
||||
iSCSI_PV --> DBPods
|
||||
Block_PV --> DBPods
|
||||
|
||||
style TrueNAS fill:#e1f5ff
|
||||
style Proxmox fill:#e1f5ff
|
||||
style K8s fill:#fff4e1
|
||||
style ZFS_Main fill:#c8e6c9
|
||||
style ZFS_SSD fill:#ffe0b2
|
||||
style NFS_HDD fill:#c8e6c9
|
||||
style NFS_SSD fill:#ffe0b2
|
||||
```
|
||||
|
||||
## Components
|
||||
|
|
@ -66,36 +66,38 @@ graph TB
|
|||
|-----------|---------------|----------|---------|
|
||||
| **Proxmox CSI plugin** | Helm chart | Namespace: proxmox-csi | Block storage via LVM-thin hotplug |
|
||||
| **StorageClass `proxmox-lvm`** | RWO, WaitForFirstConsumer | Cluster-wide | Databases and stateful apps |
|
||||
| TrueNAS VM | VMID 9000, 16c/16GB | Proxmox host (10.0.10.15) | ZFS NFS storage server |
|
||||
| ZFS pool `main` | 1.64 TiB usable | 32G + 7x256G + 1T disks | NFS data for all services |
|
||||
| ZFS pool `ssd` | ~256GB SSD | Dedicated SSD | High-performance data (Immich ML) |
|
||||
| Proxmox NFS (HDD) | LV `pve/nfs-data`, 2TB ext4 | 192.168.1.127:/srv/nfs | Bulk NFS data for all services |
|
||||
| Proxmox NFS (SSD) | LV `ssd/nfs-ssd-data`, 100GB ext4 | 192.168.1.127:/srv/nfs-ssd | High-performance data (Immich ML) |
|
||||
| nfs-csi | Helm chart | Namespace: nfs-csi | NFS CSI driver |
|
||||
| StorageClass `nfs-truenas` | RWX, soft mount | Cluster-wide | Default storage for apps |
|
||||
| ~~democratic-csi-iscsi~~ | **DEPRECATED** | Namespace: iscsi-csi | Replaced by Proxmox CSI (2026-04-02) |
|
||||
| ~~StorageClass `iscsi-truenas`~~ | **DEPRECATED** | Cluster-wide | Replaced by `proxmox-lvm` |
|
||||
| TF module `nfs_volume` | `modules/kubernetes/nfs_volume/` | Infra repo | NFS PV/PVC factory |
|
||||
| StorageClass `nfs-truenas` | RWX, soft mount | Cluster-wide | NFS storage (name kept for compatibility, points to Proxmox) |
|
||||
| StorageClass `nfs-proxmox` | RWX, soft mount | Cluster-wide | NFS storage (identical to nfs-truenas) |
|
||||
| TF module `nfs_volume` | `modules/kubernetes/nfs_volume/` | Infra repo | Static NFS PV/PVC factory |
|
||||
| ~~TrueNAS VM~~ | **DECOMMISSIONED** | Was VMID 9000 at 10.0.10.15 | Replaced by Proxmox NFS (2026-04) |
|
||||
| ~~democratic-csi-iscsi~~ | **REMOVED** | Was namespace: iscsi-csi | Replaced by Proxmox CSI (2026-04-02) |
|
||||
| ~~StorageClass `iscsi-truenas`~~ | **REMOVED** | Was cluster-wide | Replaced by `proxmox-lvm` |
|
||||
|
||||
## How It Works
|
||||
|
||||
### NFS Storage Flow
|
||||
|
||||
1. **Dataset creation**: NFS shares are created as ZFS datasets under `main/<service>` (e.g., `main/immich`, `main/nextcloud`)
|
||||
2. **Export configuration**: `/root/secrets/nfs_exports.sh` on TrueNAS generates `/etc/exports` with per-dataset exports (`/mnt/main/<service>`)
|
||||
3. **CSI provisioning**: democratic-csi-nfs mounts NFS shares and creates K8s PersistentVolumes
|
||||
4. **Terraform module**: Stacks use `modules/kubernetes/nfs_volume/` to declaratively create PV + PVC pairs:
|
||||
1. **Directory creation**: NFS share directories are created under `/srv/nfs/<service>` (HDD) or `/srv/nfs-ssd/<service>` (SSD) on the Proxmox host
|
||||
2. **Export configuration**: `/etc/exports` on the Proxmox host lists per-directory NFS exports
|
||||
3. **Terraform module**: Stacks use `modules/kubernetes/nfs_volume/` to declaratively create static PV + PVC pairs:
|
||||
```hcl
|
||||
module "nfs_data" {
|
||||
source = "../../modules/kubernetes/nfs_volume"
|
||||
name = "immich-data"
|
||||
namespace = kubernetes_namespace.immich.metadata[0].name
|
||||
nfs_server = var.nfs_server # 10.0.10.15
|
||||
nfs_path = "/mnt/main/immich"
|
||||
nfs_server = var.nfs_server # 192.168.1.127
|
||||
nfs_path = "/srv/nfs/immich"
|
||||
}
|
||||
```
|
||||
5. **Pod mount**: Applications reference PVCs in their deployment specs
|
||||
6. **Mount options**: All NFS mounts use `soft,timeo=30,retrans=3` (set in StorageClass) to prevent indefinite hangs
|
||||
4. **Pod mount**: Applications reference PVCs in their deployment specs
|
||||
5. **Mount options**: All NFS mounts use `soft,timeo=30,retrans=3` (set in StorageClass) to prevent indefinite hangs
|
||||
|
||||
**CRITICAL**: Never use inline `nfs {}` blocks in pod specs — they default to `hard,timeo=600` which causes 10-minute hangs on network issues. Always use the `nfs-truenas` StorageClass via PVCs.
|
||||
**Note**: Some legacy PVs still reference `/mnt/main/<service>` paths. These work via compatibility symlinks/bind-mounts on the Proxmox host. New PVs should use `/srv/nfs/<service>` or `/srv/nfs-ssd/<service>`.
|
||||
|
||||
**CRITICAL**: Never use inline `nfs {}` blocks in pod specs — they default to `hard,timeo=600` which causes 10-minute hangs on network issues. Always use the `nfs-truenas` or `nfs-proxmox` StorageClass via PVCs.
|
||||
|
||||
### Block Storage Flow (Proxmox CSI) — NEW
|
||||
|
||||
|
|
@ -127,28 +129,9 @@ SQLite uses `fsync()` to guarantee durability. NFS's soft mount + async semantic
|
|||
|
||||
**Solution**: Use Proxmox CSI (`proxmox-lvm`) for any SQLite database (Vaultwarden, plotting-book) or local disk (ephemeral).
|
||||
|
||||
### Democratic-CSI Sidecar Resources
|
||||
### ~~Democratic-CSI Sidecar Resources~~ (HISTORICAL — democratic-csi removed)
|
||||
|
||||
The Helm chart spawns 17 sidecar containers (driver-registrar, external-provisioner, etc.) across controller + node DaemonSet pods. Each sidecar defaults to `resources: {}`, which gets LimitRange defaults of 256Mi.
|
||||
|
||||
**Fix**: Set explicit resources in `values.yaml`:
|
||||
```yaml
|
||||
csiProxy: # TOP-LEVEL key, not nested
|
||||
resources:
|
||||
requests:
|
||||
memory: "32Mi"
|
||||
limits:
|
||||
memory: "32Mi"
|
||||
|
||||
controller:
|
||||
externalProvisioner:
|
||||
resources:
|
||||
requests: {memory: "64Mi"}
|
||||
limits: {memory: "64Mi"}
|
||||
# ... repeat for all sidecars
|
||||
```
|
||||
|
||||
Total footprint: ~1.5Gi → ~400Mi.
|
||||
> Democratic-csi has been removed along with TrueNAS decommissioning (2026-04). This section is kept for historical reference only.
|
||||
|
||||
## Configuration
|
||||
|
||||
|
|
@ -156,25 +139,23 @@ Total footprint: ~1.5Gi → ~400Mi.
|
|||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `/root/secrets/nfs_exports.sh` | TrueNAS: generates `/etc/exports` with all service shares |
|
||||
| `/etc/exports` (on Proxmox host) | NFS export configuration for all service shares |
|
||||
| `stacks/proxmox-csi/` | Terraform stack for Proxmox CSI plugin + StorageClass |
|
||||
| `stacks/iscsi-csi/` | **DEPRECATED** — democratic-csi iSCSI driver (pending removal) |
|
||||
| `stacks/nfs-csi/` | NFS CSI driver |
|
||||
| `modules/kubernetes/nfs_volume/` | Reusable module for NFS PV/PVC creation |
|
||||
| `config.tfvars` | Variable `nfs_server = "10.0.10.15"` shared by all stacks |
|
||||
| `stacks/nfs-csi/` | NFS CSI driver + StorageClasses (`nfs-truenas`, `nfs-proxmox`) |
|
||||
| `modules/kubernetes/nfs_volume/` | Reusable module for static NFS PV/PVC creation |
|
||||
| `config.tfvars` | Variable `nfs_server = "192.168.1.127"` shared by all stacks |
|
||||
|
||||
### Vault Paths
|
||||
|
||||
| Path | Contents |
|
||||
|------|----------|
|
||||
| `secret/viktor/truenas_ssh_key` | SSH private key for democratic-csi SSH driver |
|
||||
| `secret/viktor/truenas_root_password` | TrueNAS root password (web UI access) |
|
||||
| ~~`secret/viktor/truenas_ssh_key`~~ | **LEGACY** — was SSH key for democratic-csi SSH driver (TrueNAS decommissioned) |
|
||||
| ~~`secret/viktor/truenas_root_password`~~ | **LEGACY** — was TrueNAS root password (TrueNAS decommissioned) |
|
||||
|
||||
### Terraform Stacks
|
||||
|
||||
- **`stacks/proxmox-csi/`**: Deploys Proxmox CSI plugin + `proxmox-lvm` StorageClass + node topology labels
|
||||
- **`stacks/nfs-csi/`**: Deploys NFS CSI driver for TrueNAS
|
||||
- **`stacks/iscsi-csi/`**: ~~Deploys democratic-csi iSCSI driver~~ — **DEPRECATED**, pending removal
|
||||
- **`stacks/nfs-csi/`**: Deploys NFS CSI driver + StorageClasses for Proxmox NFS
|
||||
- All application stacks reference NFS volumes via `module "nfs_<name>"` calls
|
||||
- Database PVCs use `storageClass: proxmox-lvm` (CNPG, MySQL Helm VCT, Redis Helm, standalone PVCs)
|
||||
|
||||
|
|
@ -182,14 +163,11 @@ Total footprint: ~1.5Gi → ~400Mi.
|
|||
|
||||
NFS exports are NOT managed by Terraform. To add a new service:
|
||||
|
||||
1. SSH to TrueNAS: `ssh root@10.0.10.15`
|
||||
2. Edit `/root/secrets/nfs_exports.sh`
|
||||
3. Add dataset + export entry:
|
||||
```bash
|
||||
create_nfs_export "main/<service>" "/mnt/main/<service>"
|
||||
```
|
||||
4. Run the script: `/root/secrets/nfs_exports.sh`
|
||||
5. Verify: `showmount -e 10.0.10.15`
|
||||
1. SSH to Proxmox host: `ssh root@192.168.1.127`
|
||||
2. Create the directory: `mkdir -p /srv/nfs/<service> && chmod 777 /srv/nfs/<service>`
|
||||
3. Edit `/etc/exports` — add the export entry
|
||||
4. Reload exports: `exportfs -ra`
|
||||
5. Verify: `showmount -e 192.168.1.127`
|
||||
|
||||
## Decisions & Rationale
|
||||
|
||||
|
|
@ -197,26 +175,14 @@ NFS exports are NOT managed by Terraform. To add a new service:
|
|||
|
||||
- **Simplicity**: No volume provisioning delays, instant mounts
|
||||
- **RWX support**: Multiple pods can share one volume (Nextcloud, Immich)
|
||||
- **ZFS benefits**: Snapshots, compression, dedup all work at dataset level
|
||||
- **Good enough**: For SQLite on NFS specifically, we accept the risk for low-value data (logs, caches) but mandate iSCSI for critical DBs
|
||||
- **Good enough**: For SQLite on NFS specifically, we accept the risk for low-value data (logs, caches) but mandate proxmox-lvm for critical DBs
|
||||
|
||||
### Why iSCSI for Databases?
|
||||
### Why Proxmox CSI for Databases? (formerly iSCSI)
|
||||
|
||||
- **ACID guarantees**: Block device + local filesystem = real fsync
|
||||
- **Performance**: No NFS protocol overhead for random I/O
|
||||
- **Tested**: PostgreSQL CNPG and MySQL InnoDB Cluster both run on iSCSI, zero corruption in 2+ years
|
||||
|
||||
### Why SSH Driver Over API?
|
||||
|
||||
The democratic-csi API driver (`driver: freenas-api-iscsi`) has these issues:
|
||||
- Requires TrueNAS API credentials in plaintext ConfigMap
|
||||
- Fails silently when API schema changes between TrueNAS versions
|
||||
- No retry logic on transient API errors
|
||||
|
||||
SSH driver (`driver: freenas-ssh`) is simpler:
|
||||
- Direct `zfs` commands, no API translation layer
|
||||
- SSH key auth (Vault-managed)
|
||||
- Deterministic error messages
|
||||
- **Performance**: No NFS protocol overhead for random I/O, no network hop (LVM-thin hotplug direct to VM)
|
||||
- **Tested**: PostgreSQL CNPG and MySQL InnoDB Cluster both run on proxmox-lvm, zero corruption
|
||||
- **Single CoW layer**: LVM-thin only, no ZFS double-CoW issues
|
||||
|
||||
### Why Soft Mount for NFS?
|
||||
|
||||
|
|
@ -230,7 +196,7 @@ Soft mount (`soft,timeo=30,retrans=3`) trades availability for responsiveness:
|
|||
- Operations return EIO after timeout → app can handle error
|
||||
- Acceptable for non-critical data paths
|
||||
|
||||
**Critical paths**: Databases use iSCSI (not NFS), so soft mount never affects data integrity.
|
||||
**Critical paths**: Databases use proxmox-lvm (not NFS), so soft mount never affects data integrity.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
|
@ -242,55 +208,23 @@ Soft mount (`soft,timeo=30,retrans=3`) trades availability for responsiveness:
|
|||
```bash
|
||||
# On K8s node
|
||||
mount | grep nfs
|
||||
showmount -e 10.0.10.15
|
||||
showmount -e 192.168.1.127
|
||||
|
||||
# Check NFS server
|
||||
ssh root@10.0.10.15
|
||||
zfs list | grep main/<service>
|
||||
# Check NFS server (Proxmox host)
|
||||
ssh root@192.168.1.127
|
||||
ls -la /srv/nfs/<service>
|
||||
cat /etc/exports | grep <service>
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
1. Verify dataset exists: `zfs list main/<service>`
|
||||
1. Verify directory exists: `ls /srv/nfs/<service>` (or `/srv/nfs-ssd/<service>`)
|
||||
2. Verify export: `grep <service> /etc/exports`
|
||||
3. If missing: re-run `/root/secrets/nfs_exports.sh`
|
||||
4. Restart NFS server: `service nfs-server restart`
|
||||
3. If missing: add to `/etc/exports` and run `exportfs -ra`
|
||||
4. Restart NFS server: `systemctl restart nfs-server`
|
||||
|
||||
### iSCSI Session Drops
|
||||
### ~~iSCSI Session Drops~~ (HISTORICAL — iSCSI removed)
|
||||
|
||||
**Symptom**: PostgreSQL/MySQL pod restarts, iSCSI reconnection loops
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# On K8s node
|
||||
iscsiadm -m session
|
||||
dmesg | grep iscsi
|
||||
journalctl -u iscsid -f
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
1. Check TrueNAS iSCSI service: WebUI → Sharing → iSCSI → Targets
|
||||
2. Verify hardened timeouts: `iscsiadm -m node -o show | grep timeout`
|
||||
3. If defaults: re-apply cloud-init or manually update `/etc/iscsi/iscsid.conf`
|
||||
4. Restart session:
|
||||
```bash
|
||||
iscsiadm -m node -u
|
||||
iscsiadm -m node -l
|
||||
```
|
||||
|
||||
### Democratic-CSI Sidecar OOMKill
|
||||
|
||||
**Symptom**: `kubectl describe pod` shows sidecar containers OOMKilled
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
kubectl get events -n democratic-csi | grep OOM
|
||||
kubectl top pod -n democratic-csi
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
1. Set explicit resources in Helm values (see "Democratic-CSI Sidecar Resources" above)
|
||||
2. Apply: `terragrunt apply` in `stacks/democratic-csi/`
|
||||
> iSCSI was replaced by Proxmox CSI (2026-04-02) and TrueNAS has been decommissioned. This section is kept for historical reference only.
|
||||
|
||||
### SQLite Corruption on NFS
|
||||
|
||||
|
|
@ -302,8 +236,8 @@ kubectl top pod -n democratic-csi
|
|||
sqlite3 /data/db.sqlite "PRAGMA integrity_check;"
|
||||
```
|
||||
|
||||
**Fix**: Migrate to iSCSI
|
||||
1. Create iSCSI PVC in Terraform stack
|
||||
**Fix**: Migrate to proxmox-lvm
|
||||
1. Create proxmox-lvm PVC in Terraform stack
|
||||
2. Restore from backup to new volume
|
||||
3. Update deployment to use new PVC
|
||||
4. Delete old NFS PVC
|
||||
|
|
@ -314,18 +248,18 @@ sqlite3 /data/db.sqlite "PRAGMA integrity_check;"
|
|||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# On TrueNAS
|
||||
zpool iostat -v 5
|
||||
arc_summary | grep "Hit Rate"
|
||||
# On Proxmox host
|
||||
ssh root@192.168.1.127
|
||||
iostat -x 5
|
||||
lvs --reportformat json pve/nfs-data ssd/nfs-ssd-data
|
||||
|
||||
# On K8s node
|
||||
nfsiostat 5
|
||||
```
|
||||
|
||||
**Optimization**:
|
||||
1. Check ZFS ARC hit rate (should be >90%)
|
||||
2. Move hot datasets to SSD pool: `zfs send main/<dataset> | zfs recv ssd/<dataset>`
|
||||
3. Tune NFS mount: add `rsize=1048576,wsize=1048576` to StorageClass `mountOptions`
|
||||
1. Move hot data to SSD NFS: relocate from `/srv/nfs/<service>` to `/srv/nfs-ssd/<service>` and update PV path
|
||||
2. Tune NFS mount: add `rsize=1048576,wsize=1048576` to StorageClass `mountOptions`
|
||||
|
||||
## Related
|
||||
|
||||
|
|
@ -333,5 +267,5 @@ nfsiostat 5
|
|||
- `docs/runbooks/restore-postgresql.md`
|
||||
- `docs/runbooks/restore-mysql.md`
|
||||
- `docs/runbooks/recover-nfs-mount.md`
|
||||
- **Architecture**: `docs/architecture/backup-dr.md` (backup strategy using ZFS snapshots)
|
||||
- **Reference**: `.claude/reference/service-catalog.md` (which services use NFS vs iSCSI)
|
||||
- **Architecture**: `docs/architecture/backup-dr.md` (backup strategy using LVM snapshots and Proxmox host scripts)
|
||||
- **Reference**: `.claude/reference/service-catalog.md` (which services use NFS vs proxmox-lvm)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue