diff --git a/docs/architecture/backup-dr.md b/docs/architecture/backup-dr.md index 84ae792c..fa955df4 100644 --- a/docs/architecture/backup-dr.md +++ b/docs/architecture/backup-dr.md @@ -319,23 +319,42 @@ Two-step offsite sync: #### Step 1: sda to Synology pve-backup/ **Method**: `rsync` from `/mnt/backup/` to `synology.viktorbarzin.lan:/Backup/Viki/pve-backup/` -**Content**: PVC snapshots (`pvc-data/`), pfSense backups, PVE config, SQLite backups only. NFS data is no longer on sda. +**Content**: PVC snapshots (`pvc-data/`), pfSense backups, PVE config, SQLite backups, **plus the nfs-mirror output** (anca-elements + ~30 critical NFS subtrees) — see Layer 3a. After consolidation, sda is the single source for the bulk of Synology's payload. **Destination**: `Synology/Backup/Viki/pve-backup/`: - `pvc-data//` — 4 weekly PVC file backups - `sqlite-backup/` — auto SQLite backups - `pfsense//` — 4 weekly pfSense backups - `pve-config/` — latest PVE config +- `anca-elements/`, `mysql/`, `postgresql/`, `nextcloud/`, `health/`, `/` — from nfs-mirror (Layer 3a) -#### Step 2: NFS to Synology nfs/ + nfs-ssd/ (inotify change-tracked) +#### Step 2: sda-bypass NFS to Synology nfs/ + nfs-ssd/ (inotify change-tracked, FILTERED) -**Method**: `rsync --files-from /mnt/backup/.nfs-changes.log` — two calls, one for `/srv/nfs` to `nfs/`, one for `/srv/nfs-ssd` to `nfs-ssd/` -**Change tracking**: `nfs-change-tracker.service` (systemd, inotifywait) on PVE host watches `/srv/nfs` and `/srv/nfs-ssd` continuously. Changed file paths are logged to `/mnt/backup/.nfs-changes.log`. The offsite sync reads this log and transfers only changed files. Incremental syncs complete in seconds instead of 30+ minutes. -**Monthly full sync**: On 1st Sunday of month, runs `rsync --delete` for cleanup (removes orphaned files on Synology). +**Role**: Only carries paths that **bypass sda** — i.e., paths the nfs-mirror script explicitly skips (immich, frigate, prometheus, *-backup, …). Paths that ARE on sda reach Synology via Step 1 and are explicitly excluded from Step 2 to prevent double-syncing. The Step 2 INCLUDE list MUST stay in sync with nfs-mirror's `EXCLUDES` — they are complementary. -**Path exclusions**: `/srv/nfs/anca-elements/` (~770G) is excluded from both layers. From 2026-05-24 onward `/srv/nfs/anca-elements` is the source of truth for this archive — the Synology copy at `/volume1/Backup/Anca/Elements` was deleted (it had been the upstream source, but anca-elements-sync.sh's role inverted: PVE now writes, Synology no longer holds it). Single-disk-failure protection is provided by a SEPARATE local mirror on sda (`anca-elements-mirror.{service,timer}`, weekly Mon 04:00) — not by Synology. The Synology exclusion lives in `nfs-change-tracker.service` (inotify `--exclude` regex) and `offsite-sync-backup` (rsync `--exclude` on full sync + `grep -v` on the incremental files-from list). +**Method**: `rsync --files-from /mnt/backup/.nfs-changes.log` with regex filter `^/srv/nfs/(immich|frigate|prometheus|loki|temp|alertmanager|ollama|audiblez|ebook2audiobook|[^/]+-backup)/`. The monthly full sync uses `--include='//***' … --exclude='*'` to limit to the same set. `nfs-ssd/` (all of immich-ML / ollama / llamacpp) is entirely bypass-list, so a plain `--delete` still applies. -**Layer 3a: anca-elements local mirror (sda)**: `/usr/local/bin/anca-elements-mirror` rsyncs `/srv/nfs/anca-elements/` → `/mnt/backup/anca-elements/` weekly. `rsync -rlt --delete -H --no-perms --no-owner --no-group`. Idempotent; subsequent runs only transfer changes. Pushes `anca_elements_mirror_last_run_timestamp` + `anca_elements_mirror_last_status` to Pushgateway. No offsite copy — by design; the archive is single-disk-failure tolerant only. +**Change tracking**: `nfs-change-tracker.service` (systemd, inotifywait) on PVE host watches `/srv/nfs` and `/srv/nfs-ssd` continuously. Changed file paths are logged to `/mnt/backup/.nfs-changes.log`. Step 2 reads this log and transfers only changed files matching the bypass regex. Incremental syncs complete in seconds. + +**Monthly full sync**: On 1st Sunday of month, runs `rsync --delete` with the bypass-only include list for cleanup. + +**`/srv/nfs/anca-elements/` history**: had its own dedicated Synology exclusion line earlier in 2026-05-24 because the original Synology source (`/volume1/Backup/Anca/Elements`) was being preserved while we moved canonical to PVE. After the original was deleted (same day), anca-elements joined the broader "NOT bypassing sda" category and is covered by Step 1 via `nfs-mirror`. + +**Layer 3a: NFS local mirror on sda (3-2-1 second copy)**: `/usr/local/bin/nfs-mirror` rsyncs the *critical* subset of `/srv/nfs/` → `/mnt/backup//` weekly (Mon 04:00). Single rsync invocation, single destination. The skip-list (in `nfs-mirror.sh` `EXCLUDES`) drops paths that don't justify a second local copy: + +- **immich** (1.2T) — too big for sda; Synology offsite is the only 2nd copy by design +- **frigate** (camera recordings, 14d auto-rotate) +- **prometheus**, **loki** (TSDB + logs — rebuildable / policy-driven retention) +- **ollama**, **llamacpp**, **audiblez**, **ebook2audiobook** (re-downloadable / regenerable) +- **temp**, **alertmanager** (transient state) +- **`*-backup`** (CronJob outputs — these ARE backups; backing up the backup is meta) +- **/srv/nfs-ssd** entirely (after the SSD skips above, residual is ~0) + +Everything else under `/srv/nfs/` (anca-elements + ~30 critical service NFS subtrees: mysql, postgresql, nextcloud, health, real-estate-crawler, audiobookshelf, servarr, technitium, openclaw, ...) lands at `/mnt/backup//`. Total mirror size ≈ 900 GB (mostly anca-elements at 770G). + +Pushes `nfs_mirror_last_run_timestamp` + `nfs_mirror_last_status` + `nfs_mirror_bytes` to Pushgateway. Alerts: `NfsMirrorStale` (>16d), `NfsMirrorFailing` (status != 0). `rsync -rlt --delete -H --no-perms --no-owner --no-group`; idempotent. Nice=10, IOSchedulingClass=idle (won't compete with foreground IO). + +> History: `anca-elements-mirror.{sh,service,timer}` was a precursor (2026-05-24 morning) dedicated to /srv/nfs/anca-elements only. Subsumed by `nfs-mirror` later the same day to consolidate ad-hoc copy scripts into one. **Destination**: - `Synology/Backup/Viki/nfs/` — mirrors `/srv/nfs` @@ -362,8 +381,8 @@ Two-step offsite sync: | `/etc/systemd/system/lvm-pvc-snapshot.timer` | Daily 03:00 (LVM snapshots) | | `/etc/systemd/system/daily-backup.timer` | Daily 05:00 (file backup) | | `/etc/systemd/system/offsite-sync-backup.timer` | Daily 06:00 (offsite sync) | -| `/usr/local/bin/anca-elements-mirror` | PVE host: weekly mirror of /srv/nfs/anca-elements → sda /mnt/backup/anca-elements | -| `/etc/systemd/system/anca-elements-mirror.timer` | Weekly Mon 04:00 (anca-elements mirror) | +| `/usr/local/bin/nfs-mirror` | PVE host: weekly selective mirror of /srv/nfs/* → sda /mnt/backup// (Layer 3a) | +| `/etc/systemd/system/nfs-mirror.timer` | Weekly Mon 04:00 (NFS local mirror to sda) | | `stacks/dbaas/` | Terraform: PostgreSQL/MySQL backup CronJobs | | `stacks/vault/` | Terraform: Vault backup CronJob | | `stacks/vaultwarden/` | Terraform: Vaultwarden backup + integrity CronJobs | diff --git a/scripts/nfs-mirror.service b/scripts/nfs-mirror.service new file mode 100644 index 00000000..df86005c --- /dev/null +++ b/scripts/nfs-mirror.service @@ -0,0 +1,15 @@ +[Unit] +Description=Mirror /srv/nfs (selective) to /mnt/backup (local 2nd copy of critical NFS) +After=network-online.target local-fs.target +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart=/usr/local/bin/nfs-mirror +StandardOutput=journal +StandardError=journal +SyslogIdentifier=nfs-mirror +# Heavy sustained IO — don't compete with foreground services. +Nice=10 +IOSchedulingClass=idle +TimeoutStartSec=18000 diff --git a/scripts/nfs-mirror.sh b/scripts/nfs-mirror.sh new file mode 100644 index 00000000..9bbd0496 --- /dev/null +++ b/scripts/nfs-mirror.sh @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# nfs-mirror — local 2nd copy of /srv/nfs (selective) → /mnt/backup +# +# Deploy to PVE host at /usr/local/bin/nfs-mirror. +# Schedule: weekly Mon 04:00 via nfs-mirror.timer. +# +# ROLE in the 3-2-1 strategy: +# Copy 1 (sdc): /srv/nfs/* (live PVE NFS) +# Copy 2 (sda, this): /mnt/backup// ← this script +# Copy 3 (Synology): /Backup/Viki/nfs/ (via offsite-sync-backup + inotify) +# +# Replaces the dedicated anca-elements-mirror script; same disk, same +# destination layout (anca-elements lives at /mnt/backup/anca-elements/), +# but now covers every other critical NFS subtree in one pass. +# +# SKIP-LIST rationale (paths NOT mirrored — Synology offsite still covers them): +# immich — 1.2T, doesn't fit on sda; Synology only by design +# frigate — 14d camera ring, auto-rotates +# prometheus — TSDB, rebuildable from cluster state +# loki — log retention is a policy choice, not durable data +# temp — scratch +# alertmanager — transient state +# ollama — LLM model weights, re-downloadable +# audiblez — re-fetchable from Audible +# ebook2audiobook — regenerable from book sources +# *-backup — CronJob output (these ARE backups; backing them up is meta) +# +# Note: /srv/nfs-ssd is intentionally NOT mirrored — after skipping immich +# (47G), ollama (59G), and llamacpp (26G) there's effectively zero residual. + +set -euo pipefail + +SRC=/srv/nfs/ +DST=/mnt/backup/ +LOG=/var/log/nfs-mirror.log +LOCKFILE=/run/nfs-mirror.lock +PUSHGATEWAY="${NFS_MIRROR_PUSHGATEWAY:-http://10.0.20.100:30091}" +PUSHGATEWAY_JOB=nfs-mirror + +EXCLUDES=( + # ---- /mnt/backup subtrees owned by daily-backup — leave alone ---- + --exclude='/pvc-data/' + --exclude='/sqlite-backup/' + --exclude='/pfsense/' + --exclude='/pve-config/' + --exclude='/lost+found/' + + # ---- state files used by other backup jobs ---- + --exclude='/.changed-files' + --exclude='/.last-offsite-sync' + --exclude='/.lv-pvc-mapping.json' + --exclude='/.nfs-changes.log' + + # ---- NFS paths: too big / transient / re-fetchable ---- + --exclude='/immich/' + --exclude='/frigate/' + --exclude='/prometheus/' + --exclude='/loki/' + --exclude='/temp/' + --exclude='/alertmanager/' + --exclude='/ollama/' + --exclude='/audiblez/' + --exclude='/ebook2audiobook/' + + # ---- *-backup CronJob outputs (don't back up backups) ---- + --exclude='/*-backup/' + + # ---- Synology / Windows / macOS cruft ---- + --exclude='/@eaDir/' + --exclude='*@synoeastream' + --exclude='/.DS_Store' + --exclude='/Thumbs.db' +) + +log() { echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] $*" | tee -a "$LOG"; } +warn() { log "WARN: $*"; } + +push_metrics() { + local status="${1:-0}" bytes="${2:-0}" + cat </dev/null || true +nfs_mirror_last_run_timestamp $(date +%s) +nfs_mirror_last_status ${status} +nfs_mirror_bytes ${bytes} +EOF +} + +KILLED="" +cleanup() { + rm -f "$LOCKFILE" + if [ -n "$KILLED" ]; then + push_metrics 2 0 # status=2 = aborted + fi +} +trap cleanup EXIT +trap 'KILLED=1; exit 143' TERM INT + +if ! ( set -o noclobber; echo $$ > "$LOCKFILE" ) 2>/dev/null; then + log "FATAL: another instance running (pid $(cat "$LOCKFILE" 2>/dev/null || echo unknown))" + exit 1 +fi + +mountpoint -q /mnt/backup || { log "FATAL: /mnt/backup not mounted"; push_metrics 1 0; exit 1; } +[ -d "$SRC" ] || { log "FATAL: source $SRC missing"; push_metrics 1 0; exit 1; } + +log "=== mirror starting: $SRC → $DST ===" +log "skip: immich, frigate, prometheus, loki, ollama, audiblez, *-backup, temp" + +RSYNC_RC=0 +rsync \ + -rlt --delete -H \ + --no-perms --no-owner --no-group \ + --info=stats2 \ + "${EXCLUDES[@]}" \ + "$SRC" "$DST" 2>&1 | tee -a "$LOG" || RSYNC_RC=${PIPESTATUS[0]} + +DST_BYTES=$(df -B1 --output=used /mnt/backup | tail -1) + +if [ "$RSYNC_RC" -eq 0 ]; then + log "=== mirror complete; /mnt/backup used: $(df -h --output=used /mnt/backup | tail -1 | tr -d ' ') ===" + push_metrics 0 "$DST_BYTES" +else + log "=== mirror failed: rsync exited $RSYNC_RC ===" + push_metrics 1 "$DST_BYTES" + exit "$RSYNC_RC" +fi diff --git a/scripts/nfs-mirror.timer b/scripts/nfs-mirror.timer new file mode 100644 index 00000000..8061dffc --- /dev/null +++ b/scripts/nfs-mirror.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Weekly local NFS mirror to /mnt/backup + +[Timer] +OnCalendar=Mon *-*-* 04:00:00 +Persistent=true +RandomizedDelaySec=15min + +[Install] +WantedBy=timers.target diff --git a/scripts/offsite-sync-backup.sh b/scripts/offsite-sync-backup.sh index 5f7a4106..c286cf58 100644 --- a/scripts/offsite-sync-backup.sh +++ b/scripts/offsite-sync-backup.sh @@ -72,37 +72,67 @@ else fi # ============================================================ -# STEP 2: NFS → Synology nfs/ + nfs-ssd/ (inotify change-tracked) +# STEP 2: NFS → Synology nfs/ + nfs-ssd/ (inotify change-tracked, FILTERED) # ============================================================ -log "--- Step 2: NFS → Synology (change-tracked) ---" +# +# DESIGN: Step 2 only carries paths that BYPASS the sda mirror. Paths that ARE +# mirrored to sda by nfs-mirror reach Synology via Step 1 (sda → Synology +# pve-backup/) and must NOT also flow through Step 2 — that would duplicate +# every byte and double Synology consumption. +# +# The skip-list below MUST stay in sync with EXCLUDES in +# /usr/local/bin/nfs-mirror (which defines what nfs-mirror does NOT copy to +# sda). The two are complementary: nfs-mirror EXCLUDES = offsite-sync Step 2 +# INCLUDES. Failing to keep them aligned creates either gaps (data missing +# from Synology) or duplication (data on Synology via both paths). +log "--- Step 2: NFS → Synology (skip-list paths only — sda-bypass leg) ---" + +# Regex matching paths NOT on sda (must reach Synology directly). +# Top-level dirs under /srv/nfs/ — anchored, no nesting allowed. +NFS_SDA_BYPASS_RE='^/srv/nfs/(immich|frigate|prometheus|loki|temp|alertmanager|ollama|audiblez|ebook2audiobook|[^/]+-backup)/' + +# rsync include/exclude args for the monthly full sync (HDD). +# Order matters: --include patterns first, --exclude '*' last. +NFS_FULL_INCLUDES=( + --include='/immich/' --include='/immich/***' + --include='/frigate/' --include='/frigate/***' + --include='/prometheus/' --include='/prometheus/***' + --include='/loki/' --include='/loki/***' + --include='/temp/' --include='/temp/***' + --include='/alertmanager/' --include='/alertmanager/***' + --include='/ollama/' --include='/ollama/***' + --include='/audiblez/' --include='/audiblez/***' + --include='/ebook2audiobook/' --include='/ebook2audiobook/***' + --include='/*-backup/' --include='/*-backup/***' + --exclude='*' +) if [ "${DAY_OF_MONTH}" -le 7 ]; then - # Monthly: full sync with --delete for cleanup - # anca-elements/ is excluded — source of truth is Synology /volume1/Backup/Anca/Elements; - # the PVE copy is a downstream replica, syncing it back would just duplicate ~770G. - log "Monthly full NFS sync..." - rsync -rltz --delete --exclude='anca-elements/' /srv/nfs/ "${NFS_DEST}/" 2>&1 \ - && log " OK: nfs/ full sync" || { warn "nfs/ full sync failed"; STATUS=1; } + # Monthly: full sync with --delete for cleanup, restricted to bypass-list. + log "Monthly full NFS sync (sda-bypass paths only)..." + rsync -rltz --delete "${NFS_FULL_INCLUDES[@]}" /srv/nfs/ "${NFS_DEST}/" 2>&1 \ + && log " OK: nfs/ full sync (bypass-list)" || { warn "nfs/ full sync failed"; STATUS=1; } + # nfs-ssd: every dir under it (immich/ollama/llamacpp) is in the bypass list, + # so a plain --delete still applies cleanly. rsync -rltz --delete /srv/nfs-ssd/ "${NFS_SSD_DEST}/" 2>&1 \ && log " OK: nfs-ssd/ full sync" || { warn "nfs-ssd/ full sync failed"; STATUS=1; } > "${NFS_CHANGE_LOG}" elif [ -s "${NFS_CHANGE_LOG}" ]; then - # Incremental: only sync files logged by inotifywait + # Incremental: only sync changed files in bypass-list paths. sort -u "${NFS_CHANGE_LOG}" > /tmp/nfs-changes-deduped - # HDD NFS — drop anca-elements/* paths (excluded from offsite; see Monthly block) - grep '^/srv/nfs/' /tmp/nfs-changes-deduped | \ - grep -v '^/srv/nfs/anca-elements/' | \ + # HDD NFS — include only sda-bypass paths. + grep -E "${NFS_SDA_BYPASS_RE}" /tmp/nfs-changes-deduped | \ while IFS= read -r f; do [ -f "$f" ] && echo "${f#/srv/nfs/}"; done \ > /tmp/sync-nfs.list 2>/dev/null NFS_COUNT=$(wc -l < /tmp/sync-nfs.list 2>/dev/null || echo 0) if [ "${NFS_COUNT:-0}" -gt 0 ]; then rsync -rltz --files-from=/tmp/sync-nfs.list /srv/nfs/ "${NFS_DEST}/" 2>&1 \ - && log " OK: nfs/ (${NFS_COUNT} files)" \ + && log " OK: nfs/ (${NFS_COUNT} bypass files)" \ || { warn "nfs/ incremental failed"; STATUS=1; } fi - # SSD NFS + # SSD NFS — every nfs-ssd path (immich/ollama/llamacpp) is in the bypass list. grep '^/srv/nfs-ssd/' /tmp/nfs-changes-deduped | \ while IFS= read -r f; do [ -f "$f" ] && echo "${f#/srv/nfs-ssd/}"; done \ > /tmp/sync-nfs-ssd.list 2>/dev/null || true @@ -114,7 +144,7 @@ elif [ -s "${NFS_CHANGE_LOG}" ]; then fi TOTAL=$(wc -l < /tmp/nfs-changes-deduped) - log " Processed ${TOTAL} change events (${NFS_COUNT} nfs + ${SSD_COUNT} nfs-ssd files synced)" + log " Processed ${TOTAL} change events (${NFS_COUNT} nfs + ${SSD_COUNT} nfs-ssd bypass-list files synced)" > "${NFS_CHANGE_LOG}" rm -f /tmp/nfs-changes-deduped /tmp/sync-nfs.list /tmp/sync-nfs-ssd.list else diff --git a/stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl b/stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl index d590b435..51aabca4 100755 --- a/stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl +++ b/stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl @@ -1562,20 +1562,20 @@ serverFiles: severity: warning annotations: summary: "Offsite backup sync is {{ $value | humanizeDuration }} old (threshold: 9d)" - - alert: AncaElementsMirrorStale - expr: (time() - anca_elements_mirror_last_run_timestamp{job="anca-elements-mirror"}) > 1382400 + - alert: NfsMirrorStale + expr: (time() - nfs_mirror_last_run_timestamp{job="nfs-mirror"}) > 1382400 for: 30m labels: severity: warning annotations: - summary: "anca-elements mirror is {{ $value | humanizeDuration }} old (threshold: 16d / 2 weekly cycles)" - - alert: AncaElementsMirrorFailing - expr: anca_elements_mirror_last_status{job="anca-elements-mirror"} != 0 + summary: "NFS local mirror to sda is {{ $value | humanizeDuration }} old (threshold: 16d / 2 weekly cycles)" + - alert: NfsMirrorFailing + expr: nfs_mirror_last_status{job="nfs-mirror"} != 0 for: 0m labels: severity: warning annotations: - summary: "anca-elements mirror last run failed (status={{ $value }})" + summary: "NFS local mirror last run failed (status={{ $value }})" - alert: BackupDiskFull expr: (1 - node_filesystem_avail_bytes{job="proxmox-host", mountpoint="/mnt/backup"} / node_filesystem_size_bytes{job="proxmox-host", mountpoint="/mnt/backup"}) > 0.85 for: 15m