viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

immich: migrate PostgreSQL off NFS to proxmox-lvm-encrypted

Browse source 
Live PG data moves to a 10Gi LUKS-encrypted RWO PVC. WAL fsync per
commit on NFS contributed to the 2026-04-22 NFS writeback storm
(2h43m recovery, 3 of 4 nodes hard-reset). Backups remain on NFS
(append-only, NFS-tolerant).

The init container that writes postgresql.override.conf is now gated
on PG_VERSION presence — on a fresh PVC the file would otherwise make
initdb refuse the non-empty PGDATA. First boot skips the override and
initdb's cleanly; second boot (after a forced restart) writes the
override so vchord/vectors/pg_prewarm load before the dump restore.
Idempotent on initialised PVCs.

Migration executed: pg_dumpall (1.9GB) → restore on encrypted PVC →
REINDEX clip_index/face_index → 111,843 assets verified, external
HTTP 200, all 10 extensions present (vector minor 0.8.0→0.8.1 only).
LV created on PVE host, picked up by lvm-pvc-snapshot.

See docs/plans/2026-04-25-nfs-hostile-migration-{design,plan}.md.
Phase 2 (Vault Raft) follows under code-gy7h.

Closes: code-ahr7

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-04-25 15:47:30 +00:00
parent 0d5f53f337
commit 43e4f3f68e
3 changed files with 267 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

33
stacks/immich/main.tf
View file

@ -85,6 +85,30 @@ module "nfs_postgresql_host" {
nfs_path = "/srv/nfs/immich/postgresql"
}
# Migrated 2026-04-25: PG live data moved off NFS to LUKS-encrypted block.
# WAL fsync per commit on NFS contributed to the 2026-04-22 NFS writeback storm
# (see post-mortems/2026-04-22-vault-raft-leader-deadlock.md).
# Backup CronJob still writes to module.nfs_postgresql_host (NFS append-only).
resource "kubernetes_persistent_volume_claim" "immich_postgresql_encrypted" {
wait_until_bound = false
metadata {
name = "immich-postgresql-data-encrypted"
namespace = kubernetes_namespace.immich.metadata[0].name
annotations = {
"resize.topolvm.io/threshold" = "80%"
"resize.topolvm.io/increase" = "100%"
"resize.topolvm.io/storage_limit" = "20Gi"
}
}
spec {
access_modes = ["ReadWriteOnce"]
storage_class_name = "proxmox-lvm-encrypted"
resources {
requests = { storage = "10Gi" }
}
}
}
module "nfs_ml_cache_host" {
source = "../../modules/kubernetes/nfs_volume"
name = "immich-ml-cache-host"
@ -462,6 +486,13 @@ resource "kubernetes_deployment" "immich-postgres" {
name = "write-pg-override-conf"
image = "busybox:1.36"
command = ["sh", "-c", <<-EOT
# Skip write on uninitialised PGDATA — initdb refuses non-empty dirs.
# On first boot the override is absent; trigger a pod restart after
# initdb completes so the override is applied before extension load.
if [ ! -f /data/PG_VERSION ]; then
echo "PGDATA uninitialised, skipping override conf (will write on next pod start)"
exit 0
fi
cat > /data/postgresql.override.conf <<'PGCONF'
# Immich vector search performance tuning
shared_buffers = 2048MB
@ -481,7 +512,7 @@ resource "kubernetes_deployment" "immich-postgres" {
volume {
name = "postgresql-persistent-storage"
persistent_volume_claim {
claim_name = module.nfs_postgresql_host.claim_name
claim_name = kubernetes_persistent_volume_claim.immich_postgresql_encrypted.metadata[0].name
}
}
}