diff --git a/docs/plans/2026-06-05-block-storage-harden-nfs-design.md b/docs/plans/2026-06-05-block-storage-harden-nfs-design.md new file mode 100644 index 00000000..d936d881 --- /dev/null +++ b/docs/plans/2026-06-05-block-storage-harden-nfs-design.md @@ -0,0 +1,144 @@ +# Block-Storage Scaling — Harden proxmox-csi + NFS (Decision + Design) + +**Date**: 2026-06-05 +**Status**: Decided — supersedes the recommendation in `2026-06-01-topolvm-evaluation.md` +**Decision owner**: Viktor + +## TL;DR + +We keep the **proxmox-csi** block-storage model (which already gives cross-node +PVC mobility) and **harden** it, rather than re-architecting to TopoLVM or +Longhorn. The 29-PVC/node cap is made *unreachable* (not removed) by shrinking +the block footprint via NFS migration of non-DB workloads; the ghost-disk +doom-loop is *prevented* (not just detected); and node placement is rebalanced. +**£0, no new hardware, mobility preserved.** + +## Why this, not TopoLVM / Longhorn + +Hard constraints set by Viktor (2026-06-05): **(a)** must keep the ability to +move pods across VM nodes if one goes down (mobility), **(b)** no new hardware, +**(c)** sdc IO contention is acceptable / not worth spending on. + +Key architectural insight that drove the decision: + +- **Mobility and the LUN cap are two sides of the same mechanism.** proxmox-csi + gives mobility *because* it hot-plugs each PVC as a Proxmox virtio-scsi disk + that re-attaches wherever the pod lands — and that hot-plug is exactly what + imposes the `lun < 30` cap and spawns the query-pci ghost-disk loop. +- **TopoLVM** removes the cap by killing the hot-plug — which is *why* it pins a + PVC to one node. Rejected: violates constraint (a). +- **Longhorn** keeps mobility via replication, but mobility-via-replication + costs **≥2× writes** (1 replica = no failover). On a single PVE host both + replicas land on the same sdc HDD — you pay double the write IO for redundancy + that dies with the host anyway (host = SPOF). Longhorn's own docs say "use a + dedicated disk, not the root disk." Rejected: wasteful on a single host; + reconsider only if a 2nd physical host is added. +- proxmox-csi already provides mobility at **1× write** (centralized LV + re-attaches) — strictly more IO-efficient than replication on one host. The + cap and ghost-loop are *warts on a good model*, not reasons to replace it. + +| Option | 29-cap | Ghost loop | Mobility | sdc IO | Hardware | Verdict | +|---|---|---|---|---|---|---| +| **① Harden proxmox-csi + NFS** | managed (far off) | prevented | ✅ kept (1×) | same/better | £0 | **CHOSEN** | +| TopoLVM (A/C) | removed | eliminated | ❌ pinned | A: same / C: better | £0 / £200 | rejected — loses mobility | +| Longhorn | removed | eliminated | ✅ (2×) | worse | £0 | rejected — replication wasted on 1 host | + +## Live state at decision time (2026-06-05) + +- 6 workers (VMID 201–206), proxmox-csi `CSINode.allocatable.count = 28`/node → + **168 slots**; **69 used (41%)**; **0 PVCs Pending**. +- **Imbalance is the live risk, not aggregate capacity**: node6 **21/28** (hot), + node5 **3/28**. node1=9, node2/3/4=12. +- **Ghost-disk drift = 0** (the 2026-06-04 cleanup held; `qm config` scsi counts + match tracked VolumeAttachments). Prevention still open (beads `code-dfjn`). + Retained `unusedN` LVs: node1=6, node2=9, node3=6 (harmless to the cap). +- Block PVCs: **74** (44 encrypted + 30 plain). NFS: 64. local-path: 9. +- PVE host RAM **222/267 GiB used, swap in use** → adding more worker VMs is + memory-bound (the May 4→6 escape hatch is mostly spent). +- sdc thin pool `data`: 69.67% data / 15.89% meta. `nfs-data` LV 74% of 4 TiB. + VG `pve` raw free <16 GiB; VG `ssd` free 475 GiB. + +## NFS-migration candidates (embedded-DB preflight is mandatory) + +Rule: embedded transactional stores (SQLite/LevelDB/RocksDB/H2/LMDB/ClickHouse) +corrupt on NFS; sensitive `-encrypted` PVCs lose LUKS-at-rest on NFS. Only +non-DB, non-sensitive (or app-encrypted) workloads qualify. + +**Verified NFS-safe (preflighted 2026-06-05, no embedded DB):** + +| PVC | Node | SC | Evidence | +|---|---|---|---| +| `tandoor/tandoor-data-proxmox` | node6 | proxmox-lvm | `/opt/recipes/mediafiles` = media + bundled static; PG-backed | +| `speedtest/speedtest-config-proxmox` | node6 | proxmox-lvm | `/config` = logs (383 MB `laravel.log`) + config; MySQL-backed | +| `hackmd/hackmd-data-encrypted` | node6 | encrypted | `/…/public/uploads` = PNG uploads (4.5 MB); MySQL-backed | +| `changedetection/changedetection-data-proxmox` | node6 | proxmox-lvm | `/datastore` = JSON + brotli snapshots; no DB | +| `send/send-data-proxmox` | node2 | proxmox-lvm | `/uploads` = encrypted blobs; Redis metadata | + +**Phase-1 candidates (preflight before migrating):** instagram-poster, +insta2spotify, novelapp, openclaw/openlobster, servarr/qbittorrent, postiz +(scaled-0), priority-pass-uploads*, tripit-personal-documents* (*app-encrypted / +sensitive — keep app-layer crypto, confirm before moving). + +**Must stay on block** (embedded DB or fsync-critical): vaultwarden, ntfy, +uptime-kuma, navidrome, actualbudget×3, openclaw×2, servarr arr-apps, freshrss +(SQLite); stirling-pdf (H2); rybbit (ClickHouse); beads/dolt; all CNPG +pg-cluster, mysql-standalone, immich-pg, redis; prometheus, alertmanager, loki, +vault×3, technitium×3, mailserver, paperless, forgejo, matrix, n8n. + +## Plan + +### Phase 0 — Tactical relief (now): migrate the 5 verified-safe PVCs +Per service, following the proven 2026-05-26 Wave-1 pattern (reversible — source +block PVC kept until the NFS copy is verified): +1. `presence claim service:`. +2. Create NFS export dir on PVE host + add to git-managed + `infra/scripts/pve-nfs-exports`; `exportfs -ra`. +3. Add `module "nfs_"` (`modules/kubernetes/nfs_volume`) to the stack; + `scripts/tg apply` to create the static NFS PV/PVC. +4. Scale the workload to 0 (RWO → must release the block PVC). +5. rsync block→NFS with `--checksum` (exclude cruft: changedetection + `test-direct`/`test-seq`/`lost+found`; speedtest can drop `log/`). +6. Swap the workload's `claim_name` to the NFS PVC; `tg apply`; scale up. +7. Verify app health + data intact. +8. Delete the old block PVC → frees the LUN slot; confirm with check #47. +9. Commit + push per service; wait for CI/Woodpecker. + +Result: node6 **21 → 17**, node2 **12 → 11**. +hackmd note: confirm the LUKS→NFS downgrade is acceptable (low-sensitivity doc +images) or leave hackmd on encrypted block and accept 21→18. + +### Phase 1 — Broader NFS sweep (this session if smooth, else tracked) +Preflight + migrate the Phase-1 candidates above. Goal: leave **only true DBs + +fsync-critical** services on block, so per-node block counts stay well under the +cap with years of runway. + +### Phase 2 — Ghost-loop prevention (beads `code-dfjn`; design separately) +The structural half of "harden". Substantial — propose to design + plan on its +own rather than rush: +- Soft-cap block PVCs/node below the query-pci failure threshold (observed safe + ≤24, fails ≥25) — alert + scheduler hint. +- Raise the proxmox-csi controller's QMP/query-pci timeout (and/or QEMU side). +- Auto-reconcile CronJob: detect drift (check #47 logic) → safe + `qm set --delete scsiN` (detach-only, retains LV). +- Rebalance residual node6 block PVCs → node5, one at a time, check #47 watching. + +### Phase 3 — Docs +Update `storage.md` (Wave-2 NFS migration + the ① decision), `scale-k8s-cluster.md`, +`.claude/reference/service-catalog.md`; add a "Decided: ①" banner to +`2026-06-01-topolvm-evaluation.md` pointing here. + +## Risks +- Data loss during migration → mitigated by rsync `--checksum` + keep source + until verified + the workload is scaled-0 during copy. +- LUKS-at-rest dropped for `-encrypted` PVCs moved to NFS → only migrate + low-sensitivity or app-encrypted ones; flag each. +- NFS soft-mount semantics → only non-DB workloads (preflighted); `nfsvers=4`, + `soft,timeo=30,retrans=3` per the `nfs_volume` module defaults. +- Block rebalance (Phase 2) re-introduces detach/reattach ghost risk → one at a + time with check #47. + +## Related +- `2026-06-01-topolvm-evaluation.md` (superseded recommendation) +- `docs/architecture/storage.md` § "Per-VM SCSI-LUN cap" +- `docs/runbooks/scale-k8s-cluster.md` +- beads `code-dfjn` (ghost prevention), `code-oflt` (IO isolation — not pursued here) diff --git a/stacks/tandoor/main.tf b/stacks/tandoor/main.tf index e5fe20ba..822576ab 100644 --- a/stacks/tandoor/main.tf +++ b/stacks/tandoor/main.tf @@ -59,33 +59,16 @@ module "tls_secret" { tls_secret_name = var.tls_secret_name } -resource "kubernetes_persistent_volume_claim" "data_proxmox" { - wait_until_bound = false - metadata { - name = "tandoor-data-proxmox" - namespace = kubernetes_namespace.tandoor.metadata[0].name - annotations = { - "resize.topolvm.io/threshold" = "10%" - "resize.topolvm.io/increase" = "100%" - "resize.topolvm.io/storage_limit" = "5Gi" - } - } - spec { - access_modes = ["ReadWriteOnce"] - storage_class_name = "proxmox-lvm" - resources { - requests = { - storage = "1Gi" - } - } - } - lifecycle { - # The autoresizer expands requests.storage up to storage_limit and - # PVCs can't shrink. Without this, every TF apply tries to revert - # to the spec value, K8s rejects the shrink, and the PVC ends up - # in Terminating-but-in-use limbo. - ignore_changes = [spec[0].resources[0].requests] - } +# Media + staticfiles on NFS. Migrated off proxmox-lvm 2026-06-05 for LUN-cap +# relief — tandoor is PostgreSQL-backed with no embedded DB, so NFS is safe. +# See docs/plans/2026-06-05-block-storage-harden-nfs-design.md +module "nfs_tandoor" { + source = "../../modules/kubernetes/nfs_volume" + name = "tandoor-data-nfs" + namespace = kubernetes_namespace.tandoor.metadata[0].name + nfs_server = var.nfs_server + nfs_path = "/srv/nfs/tandoor" + storage = "5Gi" } resource "kubernetes_deployment" "tandoor" { @@ -226,7 +209,7 @@ resource "kubernetes_deployment" "tandoor" { volume { name = "data" persistent_volume_claim { - claim_name = kubernetes_persistent_volume_claim.data_proxmox.metadata[0].name + claim_name = module.nfs_tandoor.claim_name } } }