backup: consolidate to one local-mirror script + invert offsite filter
Before this commit, the in-flight design split anca-elements (its own
mirror script + timer) from the rest of /srv/nfs (still going to
Synology via inotify-tracked offsite-sync). It also meant Synology
received some bytes via both paths (sda → Synology AND direct NFS →
Synology), which doubled consumption.
This commit collapses both into a clean 3-2-1:
Copy 1 (sdc): live /srv/nfs/* + cluster block PVCs
Copy 2 (sda): /mnt/backup/{pvc-data,sqlite-backup,pfsense,
pve-config,<critical-nfs>/}
← daily-backup + nfs-mirror (one script each)
Copy 3 (Synology): /Backup/Viki/{pve-backup,nfs,nfs-ssd}
← offsite-sync-backup Step 1 (sda → Synology)
+ Step 2 (sda-BYPASS paths only → Synology direct)
scripts/nfs-mirror.{sh,service,timer}:
New consolidated weekly mirror. Replaces anca-elements-mirror (to be
removed in a follow-up after the current in-flight rsync completes,
parity-verified, and Synology source-of-truth is deleted). Single
rsync /srv/nfs/ → /mnt/backup/ with an explicit EXCLUDES list that
drops paths not worth a local 2nd copy: immich (1.2T — too big),
frigate (14d ring), prometheus/loki (rebuildable), ollama/llamacpp/
audiblez/ebook2audiobook (re-fetchable), *-backup (already backups),
temp/alertmanager (transient). Nice=10, IOSchedulingClass=idle.
scripts/offsite-sync-backup.sh:
Step 2 (NFS → Synology) filter inverted: instead of `--exclude=
anca-elements/`, it now `--include`s only the sda-BYPASS paths
(immich, frigate, prometheus, *-backup, …). The bypass-include
regex MUST stay in lockstep with nfs-mirror's EXCLUDES — they are
complementary and any drift creates either gaps or duplication on
Synology. Comment in the script flags this.
monitoring alerts: renamed AncaElementsMirror{Stale,Failing} to
NfsMirror{Stale,Failing} matching the new metric job name
`nfs-mirror`. Thresholds unchanged.
docs/architecture/backup-dr.md: rewritten Step 1/Step 2 sections and
added the bypass-list rationale + cross-reference between scripts.
NOT YET DEPLOYED — gated on the in-flight anca-elements-mirror rsync
finishing + parity verification + Synology /volume1/Backup/Anca/
Elements deletion. The old scripts (anca-elements-{mirror,sync.sh})
remain on the PVE host until then, and will be removed in a cleanup
commit.
This commit is contained in:
parent
416c2a0468
commit
4d756be4f5
6 changed files with 229 additions and 30 deletions
|
|
@ -319,23 +319,42 @@ Two-step offsite sync:
|
|||
#### Step 1: sda to Synology pve-backup/
|
||||
|
||||
**Method**: `rsync` from `/mnt/backup/` to `synology.viktorbarzin.lan:/Backup/Viki/pve-backup/`
|
||||
**Content**: PVC snapshots (`pvc-data/`), pfSense backups, PVE config, SQLite backups only. NFS data is no longer on sda.
|
||||
**Content**: PVC snapshots (`pvc-data/`), pfSense backups, PVE config, SQLite backups, **plus the nfs-mirror output** (anca-elements + ~30 critical NFS subtrees) — see Layer 3a. After consolidation, sda is the single source for the bulk of Synology's payload.
|
||||
|
||||
**Destination**: `Synology/Backup/Viki/pve-backup/`:
|
||||
- `pvc-data/<YYYY-WW>/` — 4 weekly PVC file backups
|
||||
- `sqlite-backup/` — auto SQLite backups
|
||||
- `pfsense/<YYYY-WW>/` — 4 weekly pfSense backups
|
||||
- `pve-config/` — latest PVE config
|
||||
- `anca-elements/`, `mysql/`, `postgresql/`, `nextcloud/`, `health/`, `<other critical NFS dirs>/` — from nfs-mirror (Layer 3a)
|
||||
|
||||
#### Step 2: NFS to Synology nfs/ + nfs-ssd/ (inotify change-tracked)
|
||||
#### Step 2: sda-bypass NFS to Synology nfs/ + nfs-ssd/ (inotify change-tracked, FILTERED)
|
||||
|
||||
**Method**: `rsync --files-from /mnt/backup/.nfs-changes.log` — two calls, one for `/srv/nfs` to `nfs/`, one for `/srv/nfs-ssd` to `nfs-ssd/`
|
||||
**Change tracking**: `nfs-change-tracker.service` (systemd, inotifywait) on PVE host watches `/srv/nfs` and `/srv/nfs-ssd` continuously. Changed file paths are logged to `/mnt/backup/.nfs-changes.log`. The offsite sync reads this log and transfers only changed files. Incremental syncs complete in seconds instead of 30+ minutes.
|
||||
**Monthly full sync**: On 1st Sunday of month, runs `rsync --delete` for cleanup (removes orphaned files on Synology).
|
||||
**Role**: Only carries paths that **bypass sda** — i.e., paths the nfs-mirror script explicitly skips (immich, frigate, prometheus, *-backup, …). Paths that ARE on sda reach Synology via Step 1 and are explicitly excluded from Step 2 to prevent double-syncing. The Step 2 INCLUDE list MUST stay in sync with nfs-mirror's `EXCLUDES` — they are complementary.
|
||||
|
||||
**Path exclusions**: `/srv/nfs/anca-elements/` (~770G) is excluded from both layers. From 2026-05-24 onward `/srv/nfs/anca-elements` is the source of truth for this archive — the Synology copy at `/volume1/Backup/Anca/Elements` was deleted (it had been the upstream source, but anca-elements-sync.sh's role inverted: PVE now writes, Synology no longer holds it). Single-disk-failure protection is provided by a SEPARATE local mirror on sda (`anca-elements-mirror.{service,timer}`, weekly Mon 04:00) — not by Synology. The Synology exclusion lives in `nfs-change-tracker.service` (inotify `--exclude` regex) and `offsite-sync-backup` (rsync `--exclude` on full sync + `grep -v` on the incremental files-from list).
|
||||
**Method**: `rsync --files-from /mnt/backup/.nfs-changes.log` with regex filter `^/srv/nfs/(immich|frigate|prometheus|loki|temp|alertmanager|ollama|audiblez|ebook2audiobook|[^/]+-backup)/`. The monthly full sync uses `--include='/<bypass-path>/***' … --exclude='*'` to limit to the same set. `nfs-ssd/` (all of immich-ML / ollama / llamacpp) is entirely bypass-list, so a plain `--delete` still applies.
|
||||
|
||||
**Layer 3a: anca-elements local mirror (sda)**: `/usr/local/bin/anca-elements-mirror` rsyncs `/srv/nfs/anca-elements/` → `/mnt/backup/anca-elements/` weekly. `rsync -rlt --delete -H --no-perms --no-owner --no-group`. Idempotent; subsequent runs only transfer changes. Pushes `anca_elements_mirror_last_run_timestamp` + `anca_elements_mirror_last_status` to Pushgateway. No offsite copy — by design; the archive is single-disk-failure tolerant only.
|
||||
**Change tracking**: `nfs-change-tracker.service` (systemd, inotifywait) on PVE host watches `/srv/nfs` and `/srv/nfs-ssd` continuously. Changed file paths are logged to `/mnt/backup/.nfs-changes.log`. Step 2 reads this log and transfers only changed files matching the bypass regex. Incremental syncs complete in seconds.
|
||||
|
||||
**Monthly full sync**: On 1st Sunday of month, runs `rsync --delete` with the bypass-only include list for cleanup.
|
||||
|
||||
**`/srv/nfs/anca-elements/` history**: had its own dedicated Synology exclusion line earlier in 2026-05-24 because the original Synology source (`/volume1/Backup/Anca/Elements`) was being preserved while we moved canonical to PVE. After the original was deleted (same day), anca-elements joined the broader "NOT bypassing sda" category and is covered by Step 1 via `nfs-mirror`.
|
||||
|
||||
**Layer 3a: NFS local mirror on sda (3-2-1 second copy)**: `/usr/local/bin/nfs-mirror` rsyncs the *critical* subset of `/srv/nfs/` → `/mnt/backup/<service>/` weekly (Mon 04:00). Single rsync invocation, single destination. The skip-list (in `nfs-mirror.sh` `EXCLUDES`) drops paths that don't justify a second local copy:
|
||||
|
||||
- **immich** (1.2T) — too big for sda; Synology offsite is the only 2nd copy by design
|
||||
- **frigate** (camera recordings, 14d auto-rotate)
|
||||
- **prometheus**, **loki** (TSDB + logs — rebuildable / policy-driven retention)
|
||||
- **ollama**, **llamacpp**, **audiblez**, **ebook2audiobook** (re-downloadable / regenerable)
|
||||
- **temp**, **alertmanager** (transient state)
|
||||
- **`*-backup`** (CronJob outputs — these ARE backups; backing up the backup is meta)
|
||||
- **/srv/nfs-ssd** entirely (after the SSD skips above, residual is ~0)
|
||||
|
||||
Everything else under `/srv/nfs/` (anca-elements + ~30 critical service NFS subtrees: mysql, postgresql, nextcloud, health, real-estate-crawler, audiobookshelf, servarr, technitium, openclaw, ...) lands at `/mnt/backup/<svc>/`. Total mirror size ≈ 900 GB (mostly anca-elements at 770G).
|
||||
|
||||
Pushes `nfs_mirror_last_run_timestamp` + `nfs_mirror_last_status` + `nfs_mirror_bytes` to Pushgateway. Alerts: `NfsMirrorStale` (>16d), `NfsMirrorFailing` (status != 0). `rsync -rlt --delete -H --no-perms --no-owner --no-group`; idempotent. Nice=10, IOSchedulingClass=idle (won't compete with foreground IO).
|
||||
|
||||
> History: `anca-elements-mirror.{sh,service,timer}` was a precursor (2026-05-24 morning) dedicated to /srv/nfs/anca-elements only. Subsumed by `nfs-mirror` later the same day to consolidate ad-hoc copy scripts into one.
|
||||
|
||||
**Destination**:
|
||||
- `Synology/Backup/Viki/nfs/` — mirrors `/srv/nfs`
|
||||
|
|
@ -362,8 +381,8 @@ Two-step offsite sync:
|
|||
| `/etc/systemd/system/lvm-pvc-snapshot.timer` | Daily 03:00 (LVM snapshots) |
|
||||
| `/etc/systemd/system/daily-backup.timer` | Daily 05:00 (file backup) |
|
||||
| `/etc/systemd/system/offsite-sync-backup.timer` | Daily 06:00 (offsite sync) |
|
||||
| `/usr/local/bin/anca-elements-mirror` | PVE host: weekly mirror of /srv/nfs/anca-elements → sda /mnt/backup/anca-elements |
|
||||
| `/etc/systemd/system/anca-elements-mirror.timer` | Weekly Mon 04:00 (anca-elements mirror) |
|
||||
| `/usr/local/bin/nfs-mirror` | PVE host: weekly selective mirror of /srv/nfs/* → sda /mnt/backup/<svc>/ (Layer 3a) |
|
||||
| `/etc/systemd/system/nfs-mirror.timer` | Weekly Mon 04:00 (NFS local mirror to sda) |
|
||||
| `stacks/dbaas/` | Terraform: PostgreSQL/MySQL backup CronJobs |
|
||||
| `stacks/vault/` | Terraform: Vault backup CronJob |
|
||||
| `stacks/vaultwarden/` | Terraform: Vaultwarden backup + integrity CronJobs |
|
||||
|
|
|
|||
15
scripts/nfs-mirror.service
Normal file
15
scripts/nfs-mirror.service
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
[Unit]
|
||||
Description=Mirror /srv/nfs (selective) to /mnt/backup (local 2nd copy of critical NFS)
|
||||
After=network-online.target local-fs.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/bin/nfs-mirror
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=nfs-mirror
|
||||
# Heavy sustained IO — don't compete with foreground services.
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
TimeoutStartSec=18000
|
||||
125
scripts/nfs-mirror.sh
Normal file
125
scripts/nfs-mirror.sh
Normal file
|
|
@ -0,0 +1,125 @@
|
|||
#!/usr/bin/env bash
|
||||
# nfs-mirror — local 2nd copy of /srv/nfs (selective) → /mnt/backup
|
||||
#
|
||||
# Deploy to PVE host at /usr/local/bin/nfs-mirror.
|
||||
# Schedule: weekly Mon 04:00 via nfs-mirror.timer.
|
||||
#
|
||||
# ROLE in the 3-2-1 strategy:
|
||||
# Copy 1 (sdc): /srv/nfs/* (live PVE NFS)
|
||||
# Copy 2 (sda, this): /mnt/backup/<svc>/ ← this script
|
||||
# Copy 3 (Synology): /Backup/Viki/nfs/ (via offsite-sync-backup + inotify)
|
||||
#
|
||||
# Replaces the dedicated anca-elements-mirror script; same disk, same
|
||||
# destination layout (anca-elements lives at /mnt/backup/anca-elements/),
|
||||
# but now covers every other critical NFS subtree in one pass.
|
||||
#
|
||||
# SKIP-LIST rationale (paths NOT mirrored — Synology offsite still covers them):
|
||||
# immich — 1.2T, doesn't fit on sda; Synology only by design
|
||||
# frigate — 14d camera ring, auto-rotates
|
||||
# prometheus — TSDB, rebuildable from cluster state
|
||||
# loki — log retention is a policy choice, not durable data
|
||||
# temp — scratch
|
||||
# alertmanager — transient state
|
||||
# ollama — LLM model weights, re-downloadable
|
||||
# audiblez — re-fetchable from Audible
|
||||
# ebook2audiobook — regenerable from book sources
|
||||
# *-backup — CronJob output (these ARE backups; backing them up is meta)
|
||||
#
|
||||
# Note: /srv/nfs-ssd is intentionally NOT mirrored — after skipping immich
|
||||
# (47G), ollama (59G), and llamacpp (26G) there's effectively zero residual.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SRC=/srv/nfs/
|
||||
DST=/mnt/backup/
|
||||
LOG=/var/log/nfs-mirror.log
|
||||
LOCKFILE=/run/nfs-mirror.lock
|
||||
PUSHGATEWAY="${NFS_MIRROR_PUSHGATEWAY:-http://10.0.20.100:30091}"
|
||||
PUSHGATEWAY_JOB=nfs-mirror
|
||||
|
||||
EXCLUDES=(
|
||||
# ---- /mnt/backup subtrees owned by daily-backup — leave alone ----
|
||||
--exclude='/pvc-data/'
|
||||
--exclude='/sqlite-backup/'
|
||||
--exclude='/pfsense/'
|
||||
--exclude='/pve-config/'
|
||||
--exclude='/lost+found/'
|
||||
|
||||
# ---- state files used by other backup jobs ----
|
||||
--exclude='/.changed-files'
|
||||
--exclude='/.last-offsite-sync'
|
||||
--exclude='/.lv-pvc-mapping.json'
|
||||
--exclude='/.nfs-changes.log'
|
||||
|
||||
# ---- NFS paths: too big / transient / re-fetchable ----
|
||||
--exclude='/immich/'
|
||||
--exclude='/frigate/'
|
||||
--exclude='/prometheus/'
|
||||
--exclude='/loki/'
|
||||
--exclude='/temp/'
|
||||
--exclude='/alertmanager/'
|
||||
--exclude='/ollama/'
|
||||
--exclude='/audiblez/'
|
||||
--exclude='/ebook2audiobook/'
|
||||
|
||||
# ---- *-backup CronJob outputs (don't back up backups) ----
|
||||
--exclude='/*-backup/'
|
||||
|
||||
# ---- Synology / Windows / macOS cruft ----
|
||||
--exclude='/@eaDir/'
|
||||
--exclude='*@synoeastream'
|
||||
--exclude='/.DS_Store'
|
||||
--exclude='/Thumbs.db'
|
||||
)
|
||||
|
||||
log() { echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] $*" | tee -a "$LOG"; }
|
||||
warn() { log "WARN: $*"; }
|
||||
|
||||
push_metrics() {
|
||||
local status="${1:-0}" bytes="${2:-0}"
|
||||
cat <<EOF | curl -s --connect-timeout 5 --max-time 10 --data-binary @- "${PUSHGATEWAY}/metrics/job/${PUSHGATEWAY_JOB}" 2>/dev/null || true
|
||||
nfs_mirror_last_run_timestamp $(date +%s)
|
||||
nfs_mirror_last_status ${status}
|
||||
nfs_mirror_bytes ${bytes}
|
||||
EOF
|
||||
}
|
||||
|
||||
KILLED=""
|
||||
cleanup() {
|
||||
rm -f "$LOCKFILE"
|
||||
if [ -n "$KILLED" ]; then
|
||||
push_metrics 2 0 # status=2 = aborted
|
||||
fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
trap 'KILLED=1; exit 143' TERM INT
|
||||
|
||||
if ! ( set -o noclobber; echo $$ > "$LOCKFILE" ) 2>/dev/null; then
|
||||
log "FATAL: another instance running (pid $(cat "$LOCKFILE" 2>/dev/null || echo unknown))"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mountpoint -q /mnt/backup || { log "FATAL: /mnt/backup not mounted"; push_metrics 1 0; exit 1; }
|
||||
[ -d "$SRC" ] || { log "FATAL: source $SRC missing"; push_metrics 1 0; exit 1; }
|
||||
|
||||
log "=== mirror starting: $SRC → $DST ==="
|
||||
log "skip: immich, frigate, prometheus, loki, ollama, audiblez, *-backup, temp"
|
||||
|
||||
RSYNC_RC=0
|
||||
rsync \
|
||||
-rlt --delete -H \
|
||||
--no-perms --no-owner --no-group \
|
||||
--info=stats2 \
|
||||
"${EXCLUDES[@]}" \
|
||||
"$SRC" "$DST" 2>&1 | tee -a "$LOG" || RSYNC_RC=${PIPESTATUS[0]}
|
||||
|
||||
DST_BYTES=$(df -B1 --output=used /mnt/backup | tail -1)
|
||||
|
||||
if [ "$RSYNC_RC" -eq 0 ]; then
|
||||
log "=== mirror complete; /mnt/backup used: $(df -h --output=used /mnt/backup | tail -1 | tr -d ' ') ==="
|
||||
push_metrics 0 "$DST_BYTES"
|
||||
else
|
||||
log "=== mirror failed: rsync exited $RSYNC_RC ==="
|
||||
push_metrics 1 "$DST_BYTES"
|
||||
exit "$RSYNC_RC"
|
||||
fi
|
||||
10
scripts/nfs-mirror.timer
Normal file
10
scripts/nfs-mirror.timer
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
[Unit]
|
||||
Description=Weekly local NFS mirror to /mnt/backup
|
||||
|
||||
[Timer]
|
||||
OnCalendar=Mon *-*-* 04:00:00
|
||||
Persistent=true
|
||||
RandomizedDelaySec=15min
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
|
|
@ -72,37 +72,67 @@ else
|
|||
fi
|
||||
|
||||
# ============================================================
|
||||
# STEP 2: NFS → Synology nfs/ + nfs-ssd/ (inotify change-tracked)
|
||||
# STEP 2: NFS → Synology nfs/ + nfs-ssd/ (inotify change-tracked, FILTERED)
|
||||
# ============================================================
|
||||
log "--- Step 2: NFS → Synology (change-tracked) ---"
|
||||
#
|
||||
# DESIGN: Step 2 only carries paths that BYPASS the sda mirror. Paths that ARE
|
||||
# mirrored to sda by nfs-mirror reach Synology via Step 1 (sda → Synology
|
||||
# pve-backup/) and must NOT also flow through Step 2 — that would duplicate
|
||||
# every byte and double Synology consumption.
|
||||
#
|
||||
# The skip-list below MUST stay in sync with EXCLUDES in
|
||||
# /usr/local/bin/nfs-mirror (which defines what nfs-mirror does NOT copy to
|
||||
# sda). The two are complementary: nfs-mirror EXCLUDES = offsite-sync Step 2
|
||||
# INCLUDES. Failing to keep them aligned creates either gaps (data missing
|
||||
# from Synology) or duplication (data on Synology via both paths).
|
||||
log "--- Step 2: NFS → Synology (skip-list paths only — sda-bypass leg) ---"
|
||||
|
||||
# Regex matching paths NOT on sda (must reach Synology directly).
|
||||
# Top-level dirs under /srv/nfs/ — anchored, no nesting allowed.
|
||||
NFS_SDA_BYPASS_RE='^/srv/nfs/(immich|frigate|prometheus|loki|temp|alertmanager|ollama|audiblez|ebook2audiobook|[^/]+-backup)/'
|
||||
|
||||
# rsync include/exclude args for the monthly full sync (HDD).
|
||||
# Order matters: --include patterns first, --exclude '*' last.
|
||||
NFS_FULL_INCLUDES=(
|
||||
--include='/immich/' --include='/immich/***'
|
||||
--include='/frigate/' --include='/frigate/***'
|
||||
--include='/prometheus/' --include='/prometheus/***'
|
||||
--include='/loki/' --include='/loki/***'
|
||||
--include='/temp/' --include='/temp/***'
|
||||
--include='/alertmanager/' --include='/alertmanager/***'
|
||||
--include='/ollama/' --include='/ollama/***'
|
||||
--include='/audiblez/' --include='/audiblez/***'
|
||||
--include='/ebook2audiobook/' --include='/ebook2audiobook/***'
|
||||
--include='/*-backup/' --include='/*-backup/***'
|
||||
--exclude='*'
|
||||
)
|
||||
|
||||
if [ "${DAY_OF_MONTH}" -le 7 ]; then
|
||||
# Monthly: full sync with --delete for cleanup
|
||||
# anca-elements/ is excluded — source of truth is Synology /volume1/Backup/Anca/Elements;
|
||||
# the PVE copy is a downstream replica, syncing it back would just duplicate ~770G.
|
||||
log "Monthly full NFS sync..."
|
||||
rsync -rltz --delete --exclude='anca-elements/' /srv/nfs/ "${NFS_DEST}/" 2>&1 \
|
||||
&& log " OK: nfs/ full sync" || { warn "nfs/ full sync failed"; STATUS=1; }
|
||||
# Monthly: full sync with --delete for cleanup, restricted to bypass-list.
|
||||
log "Monthly full NFS sync (sda-bypass paths only)..."
|
||||
rsync -rltz --delete "${NFS_FULL_INCLUDES[@]}" /srv/nfs/ "${NFS_DEST}/" 2>&1 \
|
||||
&& log " OK: nfs/ full sync (bypass-list)" || { warn "nfs/ full sync failed"; STATUS=1; }
|
||||
# nfs-ssd: every dir under it (immich/ollama/llamacpp) is in the bypass list,
|
||||
# so a plain --delete still applies cleanly.
|
||||
rsync -rltz --delete /srv/nfs-ssd/ "${NFS_SSD_DEST}/" 2>&1 \
|
||||
&& log " OK: nfs-ssd/ full sync" || { warn "nfs-ssd/ full sync failed"; STATUS=1; }
|
||||
> "${NFS_CHANGE_LOG}"
|
||||
elif [ -s "${NFS_CHANGE_LOG}" ]; then
|
||||
# Incremental: only sync files logged by inotifywait
|
||||
# Incremental: only sync changed files in bypass-list paths.
|
||||
sort -u "${NFS_CHANGE_LOG}" > /tmp/nfs-changes-deduped
|
||||
|
||||
# HDD NFS — drop anca-elements/* paths (excluded from offsite; see Monthly block)
|
||||
grep '^/srv/nfs/' /tmp/nfs-changes-deduped | \
|
||||
grep -v '^/srv/nfs/anca-elements/' | \
|
||||
# HDD NFS — include only sda-bypass paths.
|
||||
grep -E "${NFS_SDA_BYPASS_RE}" /tmp/nfs-changes-deduped | \
|
||||
while IFS= read -r f; do [ -f "$f" ] && echo "${f#/srv/nfs/}"; done \
|
||||
> /tmp/sync-nfs.list 2>/dev/null
|
||||
NFS_COUNT=$(wc -l < /tmp/sync-nfs.list 2>/dev/null || echo 0)
|
||||
if [ "${NFS_COUNT:-0}" -gt 0 ]; then
|
||||
rsync -rltz --files-from=/tmp/sync-nfs.list /srv/nfs/ "${NFS_DEST}/" 2>&1 \
|
||||
&& log " OK: nfs/ (${NFS_COUNT} files)" \
|
||||
&& log " OK: nfs/ (${NFS_COUNT} bypass files)" \
|
||||
|| { warn "nfs/ incremental failed"; STATUS=1; }
|
||||
fi
|
||||
|
||||
# SSD NFS
|
||||
# SSD NFS — every nfs-ssd path (immich/ollama/llamacpp) is in the bypass list.
|
||||
grep '^/srv/nfs-ssd/' /tmp/nfs-changes-deduped | \
|
||||
while IFS= read -r f; do [ -f "$f" ] && echo "${f#/srv/nfs-ssd/}"; done \
|
||||
> /tmp/sync-nfs-ssd.list 2>/dev/null || true
|
||||
|
|
@ -114,7 +144,7 @@ elif [ -s "${NFS_CHANGE_LOG}" ]; then
|
|||
fi
|
||||
|
||||
TOTAL=$(wc -l < /tmp/nfs-changes-deduped)
|
||||
log " Processed ${TOTAL} change events (${NFS_COUNT} nfs + ${SSD_COUNT} nfs-ssd files synced)"
|
||||
log " Processed ${TOTAL} change events (${NFS_COUNT} nfs + ${SSD_COUNT} nfs-ssd bypass-list files synced)"
|
||||
> "${NFS_CHANGE_LOG}"
|
||||
rm -f /tmp/nfs-changes-deduped /tmp/sync-nfs.list /tmp/sync-nfs-ssd.list
|
||||
else
|
||||
|
|
|
|||
|
|
@ -1562,20 +1562,20 @@ serverFiles:
|
|||
severity: warning
|
||||
annotations:
|
||||
summary: "Offsite backup sync is {{ $value | humanizeDuration }} old (threshold: 9d)"
|
||||
- alert: AncaElementsMirrorStale
|
||||
expr: (time() - anca_elements_mirror_last_run_timestamp{job="anca-elements-mirror"}) > 1382400
|
||||
- alert: NfsMirrorStale
|
||||
expr: (time() - nfs_mirror_last_run_timestamp{job="nfs-mirror"}) > 1382400
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "anca-elements mirror is {{ $value | humanizeDuration }} old (threshold: 16d / 2 weekly cycles)"
|
||||
- alert: AncaElementsMirrorFailing
|
||||
expr: anca_elements_mirror_last_status{job="anca-elements-mirror"} != 0
|
||||
summary: "NFS local mirror to sda is {{ $value | humanizeDuration }} old (threshold: 16d / 2 weekly cycles)"
|
||||
- alert: NfsMirrorFailing
|
||||
expr: nfs_mirror_last_status{job="nfs-mirror"} != 0
|
||||
for: 0m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "anca-elements mirror last run failed (status={{ $value }})"
|
||||
summary: "NFS local mirror last run failed (status={{ $value }})"
|
||||
- alert: BackupDiskFull
|
||||
expr: (1 - node_filesystem_avail_bytes{job="proxmox-host", mountpoint="/mnt/backup"} / node_filesystem_size_bytes{job="proxmox-host", mountpoint="/mnt/backup"}) > 0.85
|
||||
for: 15m
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue