infra/docs/architecture/storage.md

Storage Architecture

Last updated: 2026-04-13

Overview

The cluster uses two storage backends: Proxmox CSI for database block storage and Proxmox NFS for application data.

Block storage (Proxmox CSI): 65 PVCs for databases and stateful apps (CNPG PostgreSQL, MySQL InnoDB, Redis, Vaultwarden, Prometheus, Nextcloud, Calibre-Web, Forgejo, FreshRSS, ActualBudget, NovelApp, Headscale, Uptime Kuma, etc.) use StorageClass: proxmox-lvm, which provisions thin LVs directly from the Proxmox host's local-lvm storage (sdc, 10.7TB RAID1 HDD thin pool). This eliminates the previous double-CoW (ZFS + LVM-thin) path that caused 56 ZFS checksum errors.

NFS storage (Proxmox host): ~100 NFS shares for media libraries (Immich, audiobookshelf, servarr, navidrome), backup targets (*-backup/ directories), and app data are served directly from the Proxmox host at 192.168.1.127. Two NFS export roots exist:

• HDD NFS: /srv/nfs on ext4 LV pve/nfs-data (2TB) — bulk media and backup targets
• SSD NFS: /srv/nfs-ssd on ext4 LV ssd/nfs-ssd-data (100GB) — high-performance data (Immich ML)

Both StorageClass: nfs-truenas (name kept for compatibility) and StorageClass: nfs-proxmox (identical) point to the Proxmox host. Migrated from TrueNAS (10.0.10.15) which has been fully decommissioned.

Backup storage (sda): 1.1TB RAID1 SAS disk, VG backup, LV data (ext4), mounted at /mnt/backup on PVE host. Dedicated backup disk for weekly PVC file backups, NFS mirrors, pfSense backups, and PVE config. Independent of live storage (sdc).

Migration (2026-04-02): All iSCSI block volumes were migrated from democratic-csi (TrueNAS iSCSI → ZFS → LVM-thin) to Proxmox CSI (direct LVM-thin hotplug). democratic-csi iSCSI driver has been removed.

Migration (2026-04): TrueNAS (10.0.10.15) fully decommissioned. All NFS storage migrated to the Proxmox host (192.168.1.127). ZFS datasets under /mnt/main/ and /mnt/ssd/ moved to ext4 LVs at /srv/nfs/ and /srv/nfs-ssd/. Legacy PVs referencing /mnt/main/ paths still work (bind-mounted or symlinked on the Proxmox host); new PVs use /srv/nfs/ and /srv/nfs-ssd/.

Architecture Diagram

graph TB
    subgraph Proxmox["Proxmox Host (192.168.1.127)"]
        sdc["sdc: 10.7TB RAID1 HDD<br/>VG pve, LV data (thin pool)<br/>65 proxmox-lvm PVCs"]
        sda["sda: 1.1TB RAID1 SAS<br/>VG backup, LV data (ext4)<br/>/mnt/backup"]
        NFS_HDD["LV pve/nfs-data (2TB ext4)<br/>/srv/nfs<br/>~100 NFS shares<br/>Media + backup targets"]
        NFS_SSD["LV ssd/nfs-ssd-data (100GB ext4)<br/>/srv/nfs-ssd<br/>High-performance data<br/>(Immich ML)"]
        NFS_Exports["NFS Exports<br/>managed by /etc/exports"]
        NFS_HDD --> NFS_Exports
        NFS_SSD --> NFS_Exports
    end

    subgraph K8s["Kubernetes Cluster"]
        CSI_NFS["nfs-csi driver<br/>StorageClass: nfs-truenas / nfs-proxmox<br/>soft,timeo=30,retrans=3"]
        CSI_PVE["Proxmox CSI plugin<br/>StorageClass: proxmox-lvm"]

        NFS_PV["NFS PersistentVolumes<br/>RWX, ~100 volumes"]
        Block_PV["Block PersistentVolumes<br/>RWO, 65 PVCs"]

        Pods["Application Pods"]
        DBPods["Database Pods<br/>PostgreSQL CNPG<br/>MySQL InnoDB"]
    end

    NFS_Exports -->|NFS mount| CSI_NFS
    sdc -->|LVM-thin hotplug| CSI_PVE

    CSI_NFS --> NFS_PV
    CSI_PVE --> Block_PV

    NFS_PV --> Pods
    Block_PV --> DBPods

    style Proxmox fill:#e1f5ff
    style K8s fill:#fff4e1
    style NFS_HDD fill:#c8e6c9
    style NFS_SSD fill:#ffe0b2

Components

Component	Version/Config	Location	Purpose
Proxmox CSI plugin	Helm chart	Namespace: proxmox-csi	Block storage via LVM-thin hotplug
StorageClass proxmox-lvm	RWO, WaitForFirstConsumer	Cluster-wide	Databases and stateful apps
Proxmox NFS (HDD)	LV pve/nfs-data, 2TB ext4	192.168.1.127:/srv/nfs	Bulk NFS data for all services
Proxmox NFS (SSD)	LV ssd/nfs-ssd-data, 100GB ext4	192.168.1.127:/srv/nfs-ssd	High-performance data (Immich ML)
nfs-csi	Helm chart	Namespace: nfs-csi	NFS CSI driver
StorageClass nfs-truenas	RWX, soft mount	Cluster-wide	NFS storage (name kept for compatibility, points to Proxmox)
StorageClass nfs-proxmox	RWX, soft mount	Cluster-wide	NFS storage (identical to nfs-truenas)
TF module nfs_volume	modules/kubernetes/nfs_volume/	Infra repo	Static NFS PV/PVC factory
TrueNAS VM	DECOMMISSIONED	Was VMID 9000 at 10.0.10.15	Replaced by Proxmox NFS (2026-04)
democratic-csi-iscsi	REMOVED	Was namespace: iscsi-csi	Replaced by Proxmox CSI (2026-04-02)
StorageClass iscsi-truenas	REMOVED	Was cluster-wide	Replaced by proxmox-lvm

How It Works

NFS Storage Flow

1. Directory creation: NFS share directories are created under /srv/nfs/<service> (HDD) or /srv/nfs-ssd/<service> (SSD) on the Proxmox host
2. Export configuration: /etc/exports on the Proxmox host lists per-directory NFS exports
3. Terraform module: Stacks use modules/kubernetes/nfs_volume/ to declaratively create static PV + PVC pairs:

   module "nfs_data" {
     source     = "../../modules/kubernetes/nfs_volume"
     name       = "immich-data"
     namespace  = kubernetes_namespace.immich.metadata[0].name
     nfs_server = var.nfs_server  # 192.168.1.127
     nfs_path   = "/srv/nfs/immich"
   }
   

4. Pod mount: Applications reference PVCs in their deployment specs
5. Mount options: All NFS mounts use soft,timeo=30,retrans=3 (set in StorageClass) to prevent indefinite hangs

Note: Some legacy PVs still reference /mnt/main/<service> paths. These work via compatibility symlinks/bind-mounts on the Proxmox host. New PVs should use /srv/nfs/<service> or /srv/nfs-ssd/<service>.

CRITICAL: Never use inline nfs {} blocks in pod specs — they default to hard,timeo=600 which causes 10-minute hangs on network issues. Always use the nfs-truenas or nfs-proxmox StorageClass via PVCs.

Block Storage Flow (Proxmox CSI) — NEW

1. PVC creation: Pod requests a PVC with storageClass: proxmox-lvm
2. CSI provisioning: Proxmox CSI plugin calls the Proxmox API to create a thin LV in the local-lvm storage
3. SCSI hotplug: The thin LV is hotplugged as a VirtIO-SCSI disk directly into the K8s node VM
4. Filesystem: CSI formats the disk as ext4 and mounts it into the pod
5. Exclusive access: RWO only — disk is attached to one VM at a time
6. Topology: Nodes are labeled with topology.kubernetes.io/region=pve and zone=pve for scheduling

Key advantage: Single CoW layer (LVM-thin only). No ZFS, no iSCSI network hop, no double-CoW corruption.

Proxmox API token: csi@pve!csi-token with CSI role (VM.Audit VM.Config.Disk Datastore.Allocate Datastore.AllocateSpace Datastore.Audit). Stored in Vault at secret/viktor.

iSCSI Storage Flow (DEPRECATED — replaced 2026-04-02)

This section is historical. All iSCSI PVCs have been migrated to Proxmox CSI (proxmox-lvm). The democratic-csi iSCSI driver is pending removal.

1. Zvol creation: democratic-csi creates ZFS zvols under main/iscsi/<pvc-name> via SSH commands
2. Target setup: TrueNAS iSCSI service exposes zvols as iSCSI LUNs
3. Initiator connection: K8s nodes connect via open-iscsi

SQLite on NFS — Why It Fails

SQLite uses fsync() to guarantee durability. NFS's soft mount + async semantics break this:

• Soft mount returns success even if data is still in client cache
• Network blips during fsync → incomplete writes → corruption
• WAL mode helps but doesn't eliminate the race

Solution: Use Proxmox CSI (proxmox-lvm) for any SQLite database (Vaultwarden, plotting-book) or local disk (ephemeral).

Democratic-CSI Sidecar Resources (HISTORICAL — democratic-csi removed)

Democratic-csi has been removed along with TrueNAS decommissioning (2026-04). This section is kept for historical reference only.

Configuration

Key Files

Path	Purpose
/etc/exports (on Proxmox host)	NFS export configuration for all service shares
stacks/proxmox-csi/	Terraform stack for Proxmox CSI plugin + StorageClass
stacks/nfs-csi/	NFS CSI driver + StorageClasses (nfs-truenas, nfs-proxmox)
modules/kubernetes/nfs_volume/	Reusable module for static NFS PV/PVC creation
config.tfvars	Variable nfs_server = "192.168.1.127" shared by all stacks

Vault Paths

Path	Contents
secret/viktor/truenas_ssh_key	LEGACY — was SSH key for democratic-csi SSH driver (TrueNAS decommissioned)
secret/viktor/truenas_root_password	LEGACY — was TrueNAS root password (TrueNAS decommissioned)

Terraform Stacks

• stacks/proxmox-csi/: Deploys Proxmox CSI plugin + proxmox-lvm StorageClass + node topology labels
• stacks/nfs-csi/: Deploys NFS CSI driver + StorageClasses for Proxmox NFS
• All application stacks reference NFS volumes via module "nfs_<name>" calls
• Database PVCs use storageClass: proxmox-lvm (CNPG, MySQL Helm VCT, Redis Helm, standalone PVCs)

NFS Export Management

NFS exports are NOT managed by Terraform. To add a new service:

1. SSH to Proxmox host: ssh root@192.168.1.127
2. Create the directory: mkdir -p /srv/nfs/<service> && chmod 777 /srv/nfs/<service>
3. Edit /etc/exports — add the export entry
4. Reload exports: exportfs -ra
5. Verify: showmount -e 192.168.1.127

Decisions & Rationale

Why NFS for Most Workloads?

• Simplicity: No volume provisioning delays, instant mounts
• RWX support: Multiple pods can share one volume (Nextcloud, Immich)
• Good enough: For SQLite on NFS specifically, we accept the risk for low-value data (logs, caches) but mandate proxmox-lvm for critical DBs

Why Proxmox CSI for Databases? (formerly iSCSI)

• ACID guarantees: Block device + local filesystem = real fsync
• Performance: No NFS protocol overhead for random I/O, no network hop (LVM-thin hotplug direct to VM)
• Tested: PostgreSQL CNPG and MySQL InnoDB Cluster both run on proxmox-lvm, zero corruption
• Single CoW layer: LVM-thin only, no ZFS double-CoW issues

Why Soft Mount for NFS?

Hard mounts with default timeo=600 (10 minutes) cause:

• 10-minute pod startup delays if NFS server is unreachable
• kubectl delete pod hangs for 10 minutes
• Kernel task hangs blocking node operations

Soft mount (soft,timeo=30,retrans=3) trades availability for responsiveness:

• Max 90s hang (30s × 3 retries)
• Operations return EIO after timeout → app can handle error
• Acceptable for non-critical data paths

Critical paths: Databases use proxmox-lvm (not NFS), so soft mount never affects data integrity.

Troubleshooting

NFS Mount Hangs

Symptom: Pod stuck in ContainerCreating, df -h hangs on NFS mount

Diagnosis:

# On K8s node
mount | grep nfs
showmount -e 192.168.1.127

# Check NFS server (Proxmox host)
ssh root@192.168.1.127
ls -la /srv/nfs/<service>
cat /etc/exports | grep <service>

Fix:

1. Verify directory exists: ls /srv/nfs/<service> (or /srv/nfs-ssd/<service>)
2. Verify export: grep <service> /etc/exports
3. If missing: add to /etc/exports and run exportfs -ra
4. Restart NFS server: systemctl restart nfs-server

iSCSI Session Drops (HISTORICAL — iSCSI removed)

iSCSI was replaced by Proxmox CSI (2026-04-02) and TrueNAS has been decommissioned. This section is kept for historical reference only.

SQLite Corruption on NFS

Symptom: database disk image is malformed, checksum errors

Diagnosis:

# In pod
sqlite3 /data/db.sqlite "PRAGMA integrity_check;"

Fix: Migrate to proxmox-lvm

1. Create proxmox-lvm PVC in Terraform stack
2. Restore from backup to new volume
3. Update deployment to use new PVC
4. Delete old NFS PVC

Slow NFS Performance

Symptom: High latency on file operations, iostat shows NFS wait times

Diagnosis:

# On Proxmox host
ssh root@192.168.1.127
iostat -x 5
lvs --reportformat json pve/nfs-data ssd/nfs-ssd-data

# On K8s node
nfsiostat 5

Optimization:

1. Move hot data to SSD NFS: relocate from /srv/nfs/<service> to /srv/nfs-ssd/<service> and update PV path
2. Tune NFS mount: add rsize=1048576,wsize=1048576 to StorageClass mountOptions

Related

• Runbooks:
  • docs/runbooks/restore-postgresql.md
  • docs/runbooks/restore-mysql.md
  • docs/runbooks/recover-nfs-mount.md
• Architecture: docs/architecture/backup-dr.md (backup strategy using LVM snapshots and Proxmox host scripts)
• Reference: .claude/reference/service-catalog.md (which services use NFS vs proxmox-lvm)