infra/docs/plans/2026-02-28-storage-reliability-design.md

Storage Reliability: Database Replication + SQLite Consolidation

Date: 2026-02-28 Status: Revised (v2) — incorporates research agent findings Goal: Eliminate data corruption risk from NFS outages by moving databases off NFS

Problem

All 70+ services store data on a single TrueNAS VM (10.0.10.15) via NFS. When this VM crashes or hangs:

• 22 services risk data corruption (databases with WAL/fsync requirements on NFS)
• 12 services experience downtime but no corruption (media, configs)
• The shared PostgreSQL alone backs 12 services — a single NFS hiccup can corrupt data for all of them

SQLite-over-NFS is fundamentally broken (advisory locking unreliable, WAL mode unsafe).

Constraints

• Zero cost — all self-hosted, OSS
• Must preserve backup workflow (consolidate to TrueNAS → rsync to backup NAS)
• Stop-and-verify after each service migration
• No data loss tolerance

Single-Host Limitation (Explicit Acknowledgment)

All K8s nodes are VMs on a single Proxmox host (192.168.1.127). This means:

Replication PROTECTS against: individual VM crash/restart, NFS outage, individual node rebuild, pod OOM/eviction, software-level failures.

Replication does NOT protect against: Proxmox host failure, physical disk failure, power loss — all replicas die simultaneously.

Given this, the plan uses minimal replication (1 primary + 1 replica for PostgreSQL, single instance for Redis) rather than full 3-instance clusters. The primary reliability gain comes from moving off NFS to local disk with proper fsync semantics, not from replication count.

Design

Strategy Overview

BEFORE:  All services → NFS (TrueNAS VM) → single point of failure

AFTER:   Databases → local disk (proper fsync, no NFS SPOF)
         SQLite apps → migrated to shared PostgreSQL where supported
         Media/configs → NFS (TrueNAS, non-critical path)
         Backups → all consolidate to NFS → rsync to backup NAS

Component 1: PostgreSQL via CloudNativePG

Current: Single PostgreSQL 16 pod on NFS (/mnt/main/postgresql/data) using custom image viktorbarzin/postgres:16-master (postgis + pgvector + pgvecto-rs).

Target: CloudNativePG operator with 2-instance cluster on local disk.

CloudNativePG (CNCF project, v1.28+, supports K8s 1.34 and PG 14-18):

• Automatic primary/replica failover
• Streaming replication
• Declarative CRD-based management (Terraform/Terragrunt compatible)
• Built-in monolith import mode (better than manual pg_dumpall)
• Built-in PgBouncer pooler CRD

Architecture:

CloudNativePG Cluster (namespace: dbaas)
├── Primary (worker node A) — local PVC via local-path-provisioner
├── Replica (worker node B) — local PVC, streaming replication
└── Services: <cluster>-rw (read-write), <cluster>-ro (read-only)

Migration approach: Use CNPG's native monolith import mode, which connects to the running old PostgreSQL and imports databases + roles using pg_dump -Fd per database. Superior to manual pg_dumpall.

Service endpoint strategy: Create an ExternalName Service called postgresql in namespace dbaas pointing to the CNPG -rw service. This preserves var.postgresql_host = postgresql.dbaas.svc.cluster.local with zero changes to dependent services.

Special cases:

• Authentik: Replace manual PgBouncer deployment with CNPG's built-in Pooler CRD, or update PgBouncer to point to CNPG's -rw service
• Init containers (woodpecker, trading-bot): Enable enableSuperuserAccess: true in CNPG Cluster spec — CNPG strips SUPERUSER from imported roles by default
• Custom image: Test viktorbarzin/postgres:16-master with CNPG first. Move shared_preload_libraries=vectors.so to CNPG postgresql.parameters (CNPG overrides container CMD). Tag format may need adjusting.

Backup: Keep existing pg_dumpall CronJob, pointed at new CNPG endpoint. CNPG's native WAL archiving requires S3-compatible backend (not NFS) — adding MinIO is a future enhancement, not a blocker.

Dependent services (12): authentik, n8n, dawarich, tandoor, linkwarden, netbox, woodpecker, rybbit, affine, health, resume, trading-bot

Resource overhead: ~2GB RAM total (2 instances), ~50GB local disk per node

Component 2: Redis — Single Instance on Local Disk

Current: Single redis-stack pod on NFS (/mnt/main/redis). RDB background save takes 39 seconds on NFS (should be <1s on local disk).

Finding: redis-stack modules (RedisJSON, RediSearch, RedisTimeSeries, RedisBloom, RedisGears) are completely unused. Zero module commands in INFO commandstats. All 11 services use plain Redis commands only (GET, SET, BullMQ queues, Celery broker, caching).

Finding: No service stores critical primary data in Redis. All use it for job queues and caching. Losing Redis data means: users re-login, jobs retry, caches rebuild. Inconvenient but never catastrophic.

Finding: None of the 11 services support Sentinel-aware connections. Redis Sentinel would require a proxy layer with no reliability gain on a single physical host.

Target: Single redis:7-alpine (or valkey:9) on local PVC. Drop redis-stack — modules are unused overhead (~100MB RAM saved).

Architecture:

Redis 7 (single instance)
├── Local PVC via local-path-provisioner (fast RDB saves)
├── K8s Service: redis.redis.svc.cluster.local (unchanged)
└── Hourly CronJob: cp dump.rdb → NFS:/mnt/main/redis-backup/

No client changes needed. Same service endpoint. Same Redis commands.

Resource overhead: ~650MB RAM (same as today minus module overhead), ~1GB local disk

Component 3: MySQL — Single Instance on Local Disk

Current: Single MySQL pod on NFS (/mnt/main/mysql) Target: Single MySQL on local PVC

Services on MySQL (8): hackmd, speedtest, onlyoffice, crowdsec, paperless-ngx, real-estate-crawler, url-shortener, grafana

Evaluate per-service whether migration to PostgreSQL is feasible (reduces operational complexity to one DB engine). Do during implementation research phase.

Backup: Keep existing mysqldump CronJob.

Component 4: Immich PostgreSQL

Current: Dedicated PostgreSQL + pgvector on NFS (ghcr.io/immich-app/postgres:15-vectorchord0.3.0-pgvectors0.2.0)

Target: Move to local PVC (same image, same single instance). Immich's PG has specialized extensions (VectorChord, pgvectors) that may not be compatible with CNPG operand images. Simpler to keep as standalone PG on local disk.

Component 5: ClickHouse (Rybbit)

Current: Single ClickHouse on NFS (/mnt/main/clickhouse) Target: Move to local PVC (single instance). Analytics data is rebuildable. ClickHouse replication is not justified for a homelab.

Component 6: SQLite App Consolidation to PostgreSQL

REVISED based on per-app research:

Apps confirmed safe to migrate:

App	Config mechanism	Migration tool	Risk	Notes
Forgejo	[database] in app.ini	forgejo dump --database postgres	Moderate	Git repos stay on NFS
FreshRSS	DB_HOST env vars	OPML export/import (fresh install)	Low	PG is the recommended backend
Open WebUI	DATABASE_URL env var	None (start fresh)	Low	Chat history is disposable

Apps REMOVED from migration plan:

App	Reason
Headscale	Project EXPLICITLY DISCOURAGES PostgreSQL: "highly discouraged, only supported for legacy reasons. All new development and testing are SQLite." Migrating risks VPN stability.
MeshCentral	Uses NeDB (document store), not SQLite. NeDB→PG migration path is poorly documented and risky.

Apps confirmed SQLite/BoltDB-only (stay on NFS):

App	Storage engine	Mitigation
Headscale	SQLite (recommended by project)	Accept (project-recommended config)
Vaultwarden	SQLite	Defer (migration too risky for password vault)
Uptime Kuma	SQLite (v2 adds MariaDB, not PG)	Accept or Litestream
Navidrome	SQLite only	Accept or Litestream
Audiobookshelf	SQLite only	Accept or Litestream
Calibre-Web	SQLite (Calibre format)	Accept (format constraint)
Wealthfolio	SQLite only	Accept or Litestream
MeshCentral	NeDB (document store)	Accept
Diun	bbolt (BoltDB fork)	Accept (rebuildable state)

Component 7: Monitoring Stack

Prometheus, Loki, Alertmanager use specialized storage (TSDB, BoltDB). Cannot migrate to PostgreSQL. Prometheus WAL is already on tmpfs (good).

Recommendation: Move to local PVCs. Losing metrics history on node failure is acceptable for a homelab.

Component 8: What Stays on NFS (unchanged)

All ~35 LOW risk services: media files, configs, caches, static content. Immich photos, Jellyfin media, Audiobookshelf audiobooks, Calibre ebooks, Frigate recordings, downloads, backups, model caches, etc.

NFS failure for these = temporary unavailability, not corruption.

Backup Strategy

CNPG PostgreSQL  → pg_dumpall CronJob (daily) → NFS:/mnt/main/postgresql-backup/
MySQL            → mysqldump CronJob (daily)  → NFS:/mnt/main/mysql-backup/
Redis            → RDB copy CronJob (hourly)  → NFS:/mnt/main/redis-backup/
Immich PG        → pg_dump CronJob (daily)    → NFS:/mnt/main/immich-pg-backup/
Litestream       → continuous SQLite backup   → NFS:/mnt/main/litestream/ (optional)
Media/configs    → already on NFS

NFS (TrueNAS) → rsync → Backup NAS  (unchanged)

All backups still consolidate to TrueNAS. Rsync-to-backup-NAS workflow is completely unchanged.

Note: CNPG's native WAL archiving requires S3-compatible storage (not NFS). Adding MinIO for PITR capability is a future enhancement. The pg_dumpall CronJob provides adequate backup for a homelab.

Migration Order (Safety-First)

Each phase: research → backup → migrate → verify → user confirms → next.

Before each service migration, a research subagent will:

1. Confirm current setup and configuration
2. Research online best practices and documentation
3. Scrutinize the migration plan for that specific service
4. Present findings for review before execution

Phase 0: Infrastructure Prerequisites

• Verify RAM headroom (current overcommit must be addressed first)
• Add dedicated local virtual disks to K8s worker nodes (via Proxmox)
• Verify local-path-provisioner is configured for new disks
• Install CloudNativePG operator (Helm)
• Test CNPG with custom PostgreSQL image (throwaway cluster)

Phase 1: PostgreSQL Migration (highest impact, most preparation)

1. Deploy throwaway CNPG cluster to test image compatibility and import
2. Full pg_dumpall backup to NFS
3. Deploy production CNPG cluster with monolith import from running PG
4. Create ExternalName Service for backwards compatibility
5. Migrate ONE low-risk service first (e.g., resume or health)
6. Verify for 24-48 hours
7. Migrate remaining services one at a time, verify each
8. Migrate authentik LAST (identity provider — highest blast radius)
9. Keep old PG pod scaled to 0 for one week as rollback safety net
10. Decommission old PG only after stability confirmed

Phase 2: Redis Migration

1. RDB snapshot backup to NFS
2. Deploy single redis:7-alpine on local PVC (same namespace, new pod)
3. Restore RDB snapshot
4. Update redis Service to point to new pod
5. Verify all 11 dependent services
6. Add hourly RDB backup CronJob to NFS
7. Decommission old redis-stack pod

Phase 3: MySQL Migration

1. mysqldump backup
2. Deploy single MySQL on local PVC
3. Restore dump
4. Verify all 8 dependent services
5. Research per-service PostgreSQL migration feasibility (future work)

Phase 4: Immich PostgreSQL

1. pg_dump backup
2. Move Immich PG to local PVC (same image, same config)
3. Verify Immich functionality (upload, search, face recognition)

Phase 5: SQLite Apps → PostgreSQL

Migrate one at a time, safest first: 5a. FreshRSS (lowest risk — fresh install with OPML import) 5b. Open WebUI (low risk — start fresh, chat history disposable) 5c. Forgejo (moderate risk — use forgejo dump, verify git operations)

Phase 6: ClickHouse + Monitoring

6a. ClickHouse → local PVC 6b. Prometheus → local PVC 6c. Loki → local PVC 6d. Alertmanager → local PVC

Phase 7: Cleanup + Optional Enhancements

• Remove old NFS directories from nfs_directories.txt
• Update nfs_exports.sh
• Optional: Add Litestream for SQLite-only apps
• Optional: Add MinIO for CNPG WAL archiving (PITR capability)
• Optional: Evaluate MySQL→PostgreSQL consolidation

Rollback Plan (per component)

PostgreSQL: Old pod kept scaled to 0 with NFS data intact. Rollback = scale old pod back up, revert ExternalName Service. Pre-migration pg_dumpall available if NFS data is stale.

Redis: Old redis-stack pod kept scaled to 0. Rollback = scale up, revert Service. Pre-migration RDB snapshot on NFS.

MySQL: Same pattern — old pod scaled to 0, mysqldump on NFS.

SQLite apps: Original SQLite databases remain on NFS untouched. Rollback = remove DATABASE_URL env var, restart pod.

Resource Budget

Component	RAM	Local Disk
CloudNativePG (2 instances)	~2GB	~50GB/node (2 nodes)
Redis 7 (single instance)	~550MB	~1GB
MySQL (single instance)	~1GB	~20GB
Immich PG (single instance)	~500MB	~10GB
CNPG Operator	~200MB	None
Total new overhead	~4.25GB	~81GB across 2 nodes

RAM WARNING: Proxmox host has 142GB physical RAM with ~156GB allocated to running VMs (already ~10% overcommitted). This plan adds ~4.25GB but also frees ~1.5GB by dropping redis-stack modules and removing old DB pods. Net increase: ~2.75GB. The old DB pods (postgresql, mysql, redis-stack on NFS) will be decommissioned, partially offsetting the new resource usage. Monitor swap usage closely.

Consider stopping unused VMs (PBS is already stopped, Windows10 uses 8GB and may not need to run continuously).

Monitoring Additions

After migration, add alerts for:

• CNPG replication lag
• CNPG instance count (< 2 = degraded)
• Local disk space on /opt/local-path-provisioner per node
• Redis RDB save failures
• Backup CronJob failures (pg_dumpall, mysqldump, RDB copy)

Success Criteria

• PostgreSQL, MySQL, Redis, Immich PG, ClickHouse all on local disk
• TrueNAS VM restart causes zero data corruption
• TrueNAS VM restart only affects media/config services (temporary unavailability)
• All backups still consolidate to TrueNAS for rsync to backup NAS
• Each migrated service verified working before proceeding to next
• Rollback tested for PostgreSQL before decommissioning old pod