backup: consolidate to one local-mirror script + invert offsite filter

Before this commit, the in-flight design split anca-elements (its own
mirror script + timer) from the rest of /srv/nfs (still going to
Synology via inotify-tracked offsite-sync). It also meant Synology
received some bytes via both paths (sda → Synology AND direct NFS →
Synology), which doubled consumption.

This commit collapses both into a clean 3-2-1:

  Copy 1 (sdc):       live /srv/nfs/* + cluster block PVCs
  Copy 2 (sda):       /mnt/backup/{pvc-data,sqlite-backup,pfsense,
                                   pve-config,<critical-nfs>/}
                      ← daily-backup + nfs-mirror (one script each)
  Copy 3 (Synology):  /Backup/Viki/{pve-backup,nfs,nfs-ssd}
                      ← offsite-sync-backup Step 1 (sda → Synology)
                        + Step 2 (sda-BYPASS paths only → Synology direct)

scripts/nfs-mirror.{sh,service,timer}:
  New consolidated weekly mirror. Replaces anca-elements-mirror (to be
  removed in a follow-up after the current in-flight rsync completes,
  parity-verified, and Synology source-of-truth is deleted). Single
  rsync /srv/nfs/ → /mnt/backup/ with an explicit EXCLUDES list that
  drops paths not worth a local 2nd copy: immich (1.2T — too big),
  frigate (14d ring), prometheus/loki (rebuildable), ollama/llamacpp/
  audiblez/ebook2audiobook (re-fetchable), *-backup (already backups),
  temp/alertmanager (transient). Nice=10, IOSchedulingClass=idle.

scripts/offsite-sync-backup.sh:
  Step 2 (NFS → Synology) filter inverted: instead of `--exclude=
  anca-elements/`, it now `--include`s only the sda-BYPASS paths
  (immich, frigate, prometheus, *-backup, …). The bypass-include
  regex MUST stay in lockstep with nfs-mirror's EXCLUDES — they are
  complementary and any drift creates either gaps or duplication on
  Synology. Comment in the script flags this.

monitoring alerts: renamed AncaElementsMirror{Stale,Failing} to
NfsMirror{Stale,Failing} matching the new metric job name
`nfs-mirror`. Thresholds unchanged.

docs/architecture/backup-dr.md: rewritten Step 1/Step 2 sections and
added the bypass-list rationale + cross-reference between scripts.

NOT YET DEPLOYED — gated on the in-flight anca-elements-mirror rsync
finishing + parity verification + Synology /volume1/Backup/Anca/
Elements deletion. The old scripts (anca-elements-{mirror,sync.sh})
remain on the PVE host until then, and will be removed in a cleanup
commit.

scripts/nfs-mirror.sh

@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+# nfs-mirror — local 2nd copy of /srv/nfs (selective) → /mnt/backup
+#
+# Deploy to PVE host at /usr/local/bin/nfs-mirror.
+# Schedule: weekly Mon 04:00 via nfs-mirror.timer.
+#
+# ROLE in the 3-2-1 strategy:
+#   Copy 1 (sdc):       /srv/nfs/* (live PVE NFS)
+#   Copy 2 (sda, this): /mnt/backup/<svc>/  ← this script
+#   Copy 3 (Synology):  /Backup/Viki/nfs/  (via offsite-sync-backup + inotify)
+#
+# Replaces the dedicated anca-elements-mirror script; same disk, same
+# destination layout (anca-elements lives at /mnt/backup/anca-elements/),
+# but now covers every other critical NFS subtree in one pass.
+#
+# SKIP-LIST rationale (paths NOT mirrored — Synology offsite still covers them):
+#   immich       — 1.2T, doesn't fit on sda; Synology only by design
+#   frigate      — 14d camera ring, auto-rotates
+#   prometheus   — TSDB, rebuildable from cluster state
+#   loki         — log retention is a policy choice, not durable data
+#   temp         — scratch
+#   alertmanager — transient state
+#   ollama       — LLM model weights, re-downloadable
+#   audiblez     — re-fetchable from Audible
+#   ebook2audiobook — regenerable from book sources
+#   *-backup     — CronJob output (these ARE backups; backing them up is meta)
+#
+# Note: /srv/nfs-ssd is intentionally NOT mirrored — after skipping immich
+# (47G), ollama (59G), and llamacpp (26G) there's effectively zero residual.
+
+set -euo pipefail
+
+SRC=/srv/nfs/
+DST=/mnt/backup/
+LOG=/var/log/nfs-mirror.log
+LOCKFILE=/run/nfs-mirror.lock
+PUSHGATEWAY="${NFS_MIRROR_PUSHGATEWAY:-http://10.0.20.100:30091}"
+PUSHGATEWAY_JOB=nfs-mirror
+
+EXCLUDES=(
+    # ---- /mnt/backup subtrees owned by daily-backup — leave alone ----
+    --exclude='/pvc-data/'
+    --exclude='/sqlite-backup/'
+    --exclude='/pfsense/'
+    --exclude='/pve-config/'
+    --exclude='/lost+found/'
+
+    # ---- state files used by other backup jobs ----
+    --exclude='/.changed-files'
+    --exclude='/.last-offsite-sync'
+    --exclude='/.lv-pvc-mapping.json'
+    --exclude='/.nfs-changes.log'
+
+    # ---- NFS paths: too big / transient / re-fetchable ----
+    --exclude='/immich/'
+    --exclude='/frigate/'
+    --exclude='/prometheus/'
+    --exclude='/loki/'
+    --exclude='/temp/'
+    --exclude='/alertmanager/'
+    --exclude='/ollama/'
+    --exclude='/audiblez/'
+    --exclude='/ebook2audiobook/'
+
+    # ---- *-backup CronJob outputs (don't back up backups) ----
+    --exclude='/*-backup/'
+
+    # ---- Synology / Windows / macOS cruft ----
+    --exclude='/@eaDir/'
+    --exclude='*@synoeastream'
+    --exclude='/.DS_Store'
+    --exclude='/Thumbs.db'
+)
+
+log()  { echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] $*" | tee -a "$LOG"; }
+warn() { log "WARN: $*"; }
+
+push_metrics() {
+    local status="${1:-0}" bytes="${2:-0}"
+    cat <<EOF | curl -s --connect-timeout 5 --max-time 10 --data-binary @- "${PUSHGATEWAY}/metrics/job/${PUSHGATEWAY_JOB}" 2>/dev/null || true
+nfs_mirror_last_run_timestamp $(date +%s)
+nfs_mirror_last_status ${status}
+nfs_mirror_bytes ${bytes}
+EOF
+}
+
+KILLED=""
+cleanup() {
+    rm -f "$LOCKFILE"
+    if [ -n "$KILLED" ]; then
+        push_metrics 2 0  # status=2 = aborted
+    fi
+}
+trap cleanup EXIT
+trap 'KILLED=1; exit 143' TERM INT
+
+if ! ( set -o noclobber; echo $$ > "$LOCKFILE" ) 2>/dev/null; then
+    log "FATAL: another instance running (pid $(cat "$LOCKFILE" 2>/dev/null || echo unknown))"
+    exit 1
+fi
+
+mountpoint -q /mnt/backup || { log "FATAL: /mnt/backup not mounted"; push_metrics 1 0; exit 1; }
+[ -d "$SRC" ]              || { log "FATAL: source $SRC missing"; push_metrics 1 0; exit 1; }
+
+log "=== mirror starting: $SRC → $DST ==="
+log "skip: immich, frigate, prometheus, loki, ollama, audiblez, *-backup, temp"
+
+RSYNC_RC=0
+rsync \
+    -rlt --delete -H \
+    --no-perms --no-owner --no-group \
+    --info=stats2 \
+    "${EXCLUDES[@]}" \
+    "$SRC" "$DST" 2>&1 | tee -a "$LOG" || RSYNC_RC=${PIPESTATUS[0]}
+
+DST_BYTES=$(df -B1 --output=used /mnt/backup | tail -1)
+
+if [ "$RSYNC_RC" -eq 0 ]; then
+    log "=== mirror complete; /mnt/backup used: $(df -h --output=used /mnt/backup | tail -1 | tr -d ' ') ==="
+    push_metrics 0 "$DST_BYTES"
+else
+    log "=== mirror failed: rsync exited $RSYNC_RC ==="
+    push_metrics 1 "$DST_BYTES"
+    exit "$RSYNC_RC"
+fi