Anca's photos are being ingested into Immich (started 2026-05-24
afternoon), so /srv/nfs/immich/library/ becomes the canonical copy
for those photos. The separate /srv/nfs/anca-elements/ archive tree
+ its sda mirror at /mnt/backup/anca-elements/ are now redundant.
Going forward:
- nfs-mirror EXCLUDES /anca-elements/ so future weekly runs don't
re-touch the 771G subtree (also no longer required since Immich
has the data via its NFS library).
- offsite-sync Step 1 also excludes /anca-elements/ — the historical
771G under /mnt/backup/anca-elements/ stays on sda for now but is
NOT shipped to Synology pve-backup/ (Immich's library reaches
Synology via Step 2 bypass leg anyway).
The 771G on /mnt/backup/anca-elements/ will be cleaned up manually
once Immich ingest completes and we verify all photos are in the
Immich library. Same for /srv/nfs/anca-elements/ on sdc thin pool —
freeing both would reclaim ~1.5 TB across sdc + sda.
In-flight context: today's nfs-mirror first run was killed mid-flight
at ~70% (was at /srv/nfs/postgresql/). The killed run wrote ~200G of
service NFS subtrees to /mnt/backup/<svc>/, then sda hit 95% used,
prompting this change. Next nfs-mirror run will not touch
anca-elements and will fit comfortably (~250G total for the keep-list
minus anca-elements).
Three more audit fixes from the 2026-05-24 backup-pipeline review:
#5 (S1 race) — manifest flock
daily-backup and nfs-mirror both append to /mnt/backup/.changed-files.
If they overlap (nfs-mirror Mon 04:11 running long, daily-backup
starting Mon 05:00), concurrent appends from `find | tee` and
`find | sed >>` could interleave mid-line — partial paths would slip
past rsync's --files-from. Both scripts now share a manifest_append()
helper using `flock -x` on /mnt/backup/.changed-files.lock. The 4
daily-backup call sites + the 1 nfs-mirror call site all pipe through
it instead of redirecting directly.
#7 (S2 unbounded manifest)
daily-backup gains check_manifest_size() invoked after the PVE-config
append (the last manifest writer of the run). Above MANIFEST_MAX_LINES
(500k) it touches /mnt/backup/.force-full-sync — offsite-sync's Step 1
now treats that flag the same as day-of-month ≤ 7 (full sync with
--delete) and clears it on success. Catches the "Synology unreachable
for many days" edge case where the manifest would grow unbounded.
#9 (wear — drop -z on LAN hops)
offsite-sync rsync calls to Synology over the same 192.168.1.0/24
gigabit LAN had `-rltz`. Compression burns CPU on the PVE host (already
IO-busy) and gives nothing on a saturated GigE link. Dropped to `-rlt`
on all 5 offsite rsync invocations (Step 1 full + Step 1 incremental +
Step 2 full nfs + Step 2 full nfs-ssd + Step 2 incremental).
Other adjustments:
- nfs-mirror's find-after-rsync now also excludes the new state files
(.changed-files.lock, .force-full-sync) when populating the manifest.
- offsite-sync Step 1 full-sync excludes the same .force-full-sync flag
so it doesn't ship to Synology.
Deployed to PVE host (/usr/local/bin/{daily-backup,nfs-mirror,
offsite-sync-backup}). Currently in-flight nfs-mirror run is unaffected
(bash loaded the old script into memory at start). Next runs use the
new behaviour.
Refs: 2026-05-24 audit Section 2 items #1 (manifest race), #4 (unbounded
manifest), #6 (LAN -z wear).
Step 1 of offsite-sync-backup is incremental on non-monthly days,
driven by /mnt/backup/.changed-files which only daily-backup wrote
to. nfs-mirror's writes were therefore invisible to Step 1 until the
next monthly --delete pass — which would *also* wipe data
pre-positioned on Synology pve-backup/ (e.g. the in-place btrfs
rename we just did to relocate ~160G of NFS subtrees from
/Backup/Viki/nfs/<svc>/ to /Backup/Viki/pve-backup/<svc>/).
Fix: snapshot a timestamp before rsync, then after rsync use
`find -newer $STAMP -type f -printf '%P\n'` to enumerate every file
nfs-mirror created/modified and append to the manifest. Paths are
relative to /mnt/backup/ (matches Step 1 --files-from expectation).
State files are excluded.
The current in-flight first run started before this patch was
deployed, so its writes won't auto-populate the manifest — a one-off
manual backfill will be done after it completes.
Before this commit, the in-flight design split anca-elements (its own
mirror script + timer) from the rest of /srv/nfs (still going to
Synology via inotify-tracked offsite-sync). It also meant Synology
received some bytes via both paths (sda → Synology AND direct NFS →
Synology), which doubled consumption.
This commit collapses both into a clean 3-2-1:
Copy 1 (sdc): live /srv/nfs/* + cluster block PVCs
Copy 2 (sda): /mnt/backup/{pvc-data,sqlite-backup,pfsense,
pve-config,<critical-nfs>/}
← daily-backup + nfs-mirror (one script each)
Copy 3 (Synology): /Backup/Viki/{pve-backup,nfs,nfs-ssd}
← offsite-sync-backup Step 1 (sda → Synology)
+ Step 2 (sda-BYPASS paths only → Synology direct)
scripts/nfs-mirror.{sh,service,timer}:
New consolidated weekly mirror. Replaces anca-elements-mirror (to be
removed in a follow-up after the current in-flight rsync completes,
parity-verified, and Synology source-of-truth is deleted). Single
rsync /srv/nfs/ → /mnt/backup/ with an explicit EXCLUDES list that
drops paths not worth a local 2nd copy: immich (1.2T — too big),
frigate (14d ring), prometheus/loki (rebuildable), ollama/llamacpp/
audiblez/ebook2audiobook (re-fetchable), *-backup (already backups),
temp/alertmanager (transient). Nice=10, IOSchedulingClass=idle.
scripts/offsite-sync-backup.sh:
Step 2 (NFS → Synology) filter inverted: instead of `--exclude=
anca-elements/`, it now `--include`s only the sda-BYPASS paths
(immich, frigate, prometheus, *-backup, …). The bypass-include
regex MUST stay in lockstep with nfs-mirror's EXCLUDES — they are
complementary and any drift creates either gaps or duplication on
Synology. Comment in the script flags this.
monitoring alerts: renamed AncaElementsMirror{Stale,Failing} to
NfsMirror{Stale,Failing} matching the new metric job name
`nfs-mirror`. Thresholds unchanged.
docs/architecture/backup-dr.md: rewritten Step 1/Step 2 sections and
added the bypass-list rationale + cross-reference between scripts.
NOT YET DEPLOYED — gated on the in-flight anca-elements-mirror rsync
finishing + parity verification + Synology /volume1/Backup/Anca/
Elements deletion. The old scripts (anca-elements-{mirror,sync.sh})
remain on the PVE host until then, and will be removed in a cleanup
commit.
