infra/docs/plans/2026-06-05-block-storage-harden-nfs-design.md

# Block-Storage Scaling — Harden proxmox-csi + NFS (Decision + Design)

**Date**: 2026-06-05
**Status**: Decided — supersedes the recommendation in `2026-06-01-topolvm-evaluation.md`
**Decision owner**: Viktor

## TL;DR

We keep the **proxmox-csi** block-storage model (which already gives cross-node
PVC mobility) and **harden** it, rather than re-architecting to TopoLVM or
Longhorn. The 29-PVC/node cap is made *unreachable* (not removed) by shrinking
the block footprint via NFS migration of non-DB workloads; the ghost-disk
doom-loop is *prevented* (not just detected); and node placement is rebalanced.
**£0, no new hardware, mobility preserved.**

## Why this, not TopoLVM / Longhorn

Hard constraints set by Viktor (2026-06-05): **(a)** must keep the ability to
move pods across VM nodes if one goes down (mobility), **(b)** no new hardware,
**(c)** sdc IO contention is acceptable / not worth spending on.

Key architectural insight that drove the decision:

- **Mobility and the LUN cap are two sides of the same mechanism.** proxmox-csi
  gives mobility *because* it hot-plugs each PVC as a Proxmox virtio-scsi disk
  that re-attaches wherever the pod lands — and that hot-plug is exactly what
  imposes the `lun < 30` cap and spawns the query-pci ghost-disk loop.
- **TopoLVM** removes the cap by killing the hot-plug — which is *why* it pins a
  PVC to one node. Rejected: violates constraint (a).
- **Longhorn** keeps mobility via replication, but mobility-via-replication
  costs **≥2× writes** (1 replica = no failover). On a single PVE host both
  replicas land on the same sdc HDD — you pay double the write IO for redundancy
  that dies with the host anyway (host = SPOF). Longhorn's own docs say "use a
  dedicated disk, not the root disk." Rejected: wasteful on a single host;
  reconsider only if a 2nd physical host is added.
- proxmox-csi already provides mobility at **1× write** (centralized LV
  re-attaches) — strictly more IO-efficient than replication on one host. The
  cap and ghost-loop are *warts on a good model*, not reasons to replace it.

| Option | 29-cap | Ghost loop | Mobility | sdc IO | Hardware | Verdict |
|---|---|---|---|---|---|---|
| **① Harden proxmox-csi + NFS** | managed (far off) | prevented | ✅ kept (1×) | same/better | £0 | **CHOSEN** |
| TopoLVM (A/C) | removed | eliminated | ❌ pinned | A: same / C: better | £0 / £200 | rejected — loses mobility |
| Longhorn | removed | eliminated | ✅ (2×) | worse | £0 | rejected — replication wasted on 1 host |

## Live state at decision time (2026-06-05)

- 6 workers (VMID 201–206), proxmox-csi `CSINode.allocatable.count = 28`/node →
  **168 slots**; **69 used (41%)**; **0 PVCs Pending**.
- **Imbalance is the live risk, not aggregate capacity**: node6 **21/28** (hot),
  node5 **3/28**. node1=9, node2/3/4=12.
- **Ghost-disk drift = 0** (the 2026-06-04 cleanup held; `qm config` scsi counts
  match tracked VolumeAttachments). Prevention still open (beads `code-dfjn`).
  Retained `unusedN` LVs: node1=6, node2=9, node3=6 (harmless to the cap).
- Block PVCs: **74** (44 encrypted + 30 plain). NFS: 64. local-path: 9.
- PVE host RAM **222/267 GiB used, swap in use** → adding more worker VMs is
  memory-bound (the May 4→6 escape hatch is mostly spent).
- sdc thin pool `data`: 69.67% data / 15.89% meta. `nfs-data` LV 74% of 4 TiB.
  VG `pve` raw free <16 GiB; VG `ssd` free 475 GiB.

## NFS-migration candidates (embedded-DB preflight is mandatory)

Rule: embedded transactional stores (SQLite/LevelDB/RocksDB/H2/LMDB/ClickHouse)
corrupt on NFS; sensitive `-encrypted` PVCs lose LUKS-at-rest on NFS. Only
non-DB, non-sensitive (or app-encrypted) workloads qualify.

**Verified NFS-safe (preflighted 2026-06-05, no embedded DB):**

| PVC | Node | SC | Evidence |
|---|---|---|---|
| `tandoor/tandoor-data-proxmox` | node6 | proxmox-lvm | `/opt/recipes/mediafiles` = media + bundled static; PG-backed |
| `speedtest/speedtest-config-proxmox` | node6 | proxmox-lvm | `/config` = logs (383 MB `laravel.log`) + config; MySQL-backed |
| `hackmd/hackmd-data-encrypted` | node6 | encrypted | `/…/public/uploads` = PNG uploads (4.5 MB); MySQL-backed |
| `changedetection/changedetection-data-proxmox` | node6 | proxmox-lvm | `/datastore` = JSON + brotli snapshots; no DB |
| `send/send-data-proxmox` | node2 | proxmox-lvm | `/uploads` = encrypted blobs; Redis metadata |

**Phase-1 candidates (preflight before migrating):** instagram-poster,
insta2spotify, novelapp, openclaw/openlobster, servarr/qbittorrent, postiz
(scaled-0), priority-pass-uploads*, tripit-personal-documents* (*app-encrypted /
sensitive — keep app-layer crypto, confirm before moving).

**Must stay on block** (embedded DB or fsync-critical): vaultwarden, ntfy,
uptime-kuma, navidrome, actualbudget×3, openclaw×2, servarr arr-apps, freshrss
(SQLite); stirling-pdf (H2); rybbit (ClickHouse); beads/dolt; all CNPG
pg-cluster, mysql-standalone, immich-pg, redis; prometheus, alertmanager, loki,
vault×3, technitium×3, mailserver, paperless, forgejo, matrix, n8n.

## Plan

### Phase 0 — Tactical relief (now): migrate the 5 verified-safe PVCs
Per service, following the proven 2026-05-26 Wave-1 pattern (reversible — source
block PVC kept until the NFS copy is verified):
1. `presence claim service:<svc>`.
2. Create NFS export dir on PVE host + add to git-managed
   `infra/scripts/pve-nfs-exports`; `exportfs -ra`.
3. Add `module "nfs_<svc>"` (`modules/kubernetes/nfs_volume`) to the stack;
   `scripts/tg apply` to create the static NFS PV/PVC.
4. Scale the workload to 0 (RWO → must release the block PVC).
5. rsync block→NFS with `--checksum` (exclude cruft: changedetection
   `test-direct`/`test-seq`/`lost+found`; speedtest can drop `log/`).
6. Swap the workload's `claim_name` to the NFS PVC; `tg apply`; scale up.
7. Verify app health + data intact.
8. Delete the old block PVC → frees the LUN slot; confirm with check #47.
9. Commit + push per service; wait for CI/Woodpecker.

Result: node6 **21 → 17**, node2 **12 → 11**.
hackmd note: confirm the LUKS→NFS downgrade is acceptable (low-sensitivity doc
images) or leave hackmd on encrypted block and accept 21→18.

### Phase 1 — Broader NFS sweep (this session if smooth, else tracked)
Preflight + migrate the Phase-1 candidates above. Goal: leave **only true DBs +
fsync-critical** services on block, so per-node block counts stay well under the
cap with years of runway.

### Phase 2 — Ghost-loop prevention (beads `code-dfjn`; design separately)
The structural half of "harden". Substantial — propose to design + plan on its
own rather than rush:
- Soft-cap block PVCs/node below the query-pci failure threshold (observed safe
  ≤24, fails ≥25) — alert + scheduler hint.
- Raise the proxmox-csi controller's QMP/query-pci timeout (and/or QEMU side).
- Auto-reconcile CronJob: detect drift (check #47 logic) → safe
  `qm set <vmid> --delete scsiN` (detach-only, retains LV).
- Rebalance residual node6 block PVCs → node5, one at a time, check #47 watching.

### Phase 3 — Docs
Update `storage.md` (Wave-2 NFS migration + the ① decision), `scale-k8s-cluster.md`,
`.claude/reference/service-catalog.md`; add a "Decided: ①" banner to
`2026-06-01-topolvm-evaluation.md` pointing here.

## Risks
- Data loss during migration → mitigated by rsync `--checksum` + keep source
  until verified + the workload is scaled-0 during copy.
- LUKS-at-rest dropped for `-encrypted` PVCs moved to NFS → only migrate
  low-sensitivity or app-encrypted ones; flag each.
- NFS soft-mount semantics → only non-DB workloads (preflighted); `nfsvers=4`,
  `soft,timeo=30,retrans=3` per the `nfs_volume` module defaults.
- Block rebalance (Phase 2) re-introduces detach/reattach ghost risk → one at a
  time with check #47.

## Related
- `2026-06-01-topolvm-evaluation.md` (superseded recommendation)
- `docs/architecture/storage.md` § "Per-VM SCSI-LUN cap"
- `docs/runbooks/scale-k8s-cluster.md`
- beads `code-dfjn` (ghost prevention), `code-oflt` (IO isolation — not pursued here)