backup: stop offsite-copying regenerable data; shrink nextcloud backup; pin nextcloud image
The offsite Synology hit 97% — the Backup share grew +670G in a week, traced to the 2026-05-26 change that began mirroring large regenerable services offsite, plus an unbounded nextcloud.log bloating its backups to 87G. - nfs-mirror: re-exclude ollama, prometheus-backup, audiblez, ebook2audiobook (regenerable; live-only on sdc). Keep *-backup DB dumps (real safety copies). - offsite-sync Step 2: nfs-ssd leg is now immich-only; ollama/llamacpp on the SSD no longer ship offsite (re-pullable models). - daily-backup: skip nextcloud/nextcloud-data-proxmox (orphaned pre-encryption PV, still backed up weekly). - nextcloud: cap+rotate the log (log_rotate_size=10MB); the dedicated backup now excludes html/ (app code, from image), logs, and preview cache and keeps only the latest copy (pvc-data holds version history) → <5G (was 87G). - nextcloud: pin image to 32.0.9 in chart_values. A 2026-05-26 Keel bump moved the live pod to 32.0.9 (data migrated to 32.0.9.2) but TF still defaulted to 32.0.3; reconciling that drift this session rolled a 32.0.3 pod that CrashLooped on the downgrade. Pinning eliminates the drift. Docs: backup-dr.md + infra CLAUDE.md updated (add nfs-mirror, new exclusions). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
0dd4a31eff
commit
ddd582a28c
7 changed files with 122 additions and 35 deletions
|
|
@ -215,6 +215,17 @@ else
|
|||
continue
|
||||
fi
|
||||
|
||||
# Skip-list: PVCs we deliberately don't keep offsite copies of.
|
||||
# nextcloud-data-proxmox — orphaned pre-encryption PV (Released,
|
||||
# Retain). Nextcloud moved to nextcloud-data-encrypted on 2026-04-13;
|
||||
# this old unencrypted PV lingers (Retain) and was still being backed
|
||||
# up weekly, filling the offsite Synology. Stop copying it (2026-06-01).
|
||||
case "${ns_pvc}" in
|
||||
nextcloud/nextcloud-data-proxmox)
|
||||
log " skip ${ns_pvc} (orphaned pre-encryption PVC)"
|
||||
continue ;;
|
||||
esac
|
||||
|
||||
# Detect LUKS-encrypted volumes and set up mount device
|
||||
LUKS_NAME=""
|
||||
MOUNT_DEV="/dev/pve/${snap}"
|
||||
|
|
|
|||
|
|
@ -13,21 +13,30 @@
|
|||
# destination layout (anca-elements lives at /mnt/backup/anca-elements/),
|
||||
# but now covers every other critical NFS subtree in one pass.
|
||||
#
|
||||
# SKIP-LIST rationale (2026-05-26 simplification — see commit notes):
|
||||
# SKIP-LIST rationale (2026-05-26 simplification; REGENERABLE-SERVICE
|
||||
# CARVE-OUT added 2026-06-01 — see below):
|
||||
# immich — 1.5T, doesn't fit on sda; offsite-sync ships it direct to Synology
|
||||
# frigate — camera ring buffer; intentionally NOT backed up anywhere
|
||||
# temp — scratch; intentionally NOT backed up
|
||||
#
|
||||
# Everything else (ollama, audiblez, ebook2audiobook, *-backup, …) now
|
||||
# flows sdc → sda (this script) → Synology pve-backup/ via offsite-sync
|
||||
# Step 1. Previously they went sdc → Synology DIRECT via Step 2; the
|
||||
# bypass list got pruned to just `immich` so we have a single canonical
|
||||
# mirror at sda. Prometheus/loki/alertmanager were live-orphan entries
|
||||
# that no longer exist on /srv/nfs (cleaned 2026-05-26) — dropped from
|
||||
# the exclude list as a no-op.
|
||||
# 2026-06-01 carve-out: the offsite Synology (5.3T) hit 97% and the
|
||||
# `Backup` share had grown +670G in a week — traced to the 2026-05-26
|
||||
# change that started mirroring large *regenerable* services to sda and
|
||||
# thence to Synology pve-backup/. These are now re-excluded because they
|
||||
# cost offsite capacity for data we can rebuild on demand:
|
||||
# ollama (20G) — LLM model blobs, re-pullable
|
||||
# prometheus-backup (64G) — metrics TSDB snapshots; was offsite-excluded
|
||||
# pre-2026-05-26 by original intent
|
||||
# audiblez (24G) — generated audiobooks, re-derivable from ebooks
|
||||
# ebook2audiobook (11G) — same, generation output
|
||||
# Their live copy stays on sdc (/srv/nfs); only the sda + Synology copies
|
||||
# are dropped. `*-backup` DB dumps (sqlite-backup et al.) are intentionally
|
||||
# KEPT — they are real database safety copies, not regenerable.
|
||||
#
|
||||
# Note: /srv/nfs-ssd is intentionally NOT mirrored — its three dirs
|
||||
# (immich, ollama, llamacpp) all go direct to Synology nfs-ssd/.
|
||||
# Note: /srv/nfs-ssd is intentionally NOT mirrored — its dirs (immich,
|
||||
# ollama, llamacpp) go direct to Synology nfs-ssd/ via offsite-sync
|
||||
# Step 2, which (also 2026-06-01) was narrowed to immich-only so ollama
|
||||
# + llamacpp on the SSD stop reaching Synology too.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
|
|
@ -68,6 +77,14 @@ EXCLUDES=(
|
|||
--exclude='/frigate/' # ring buffer — no backup anywhere
|
||||
--exclude='/temp/' # scratch — no backup anywhere
|
||||
|
||||
# ---- regenerable services: live-only on sdc, no offsite (2026-06-01) ----
|
||||
# See header carve-out. --delete reaps any existing copies from sda on
|
||||
# the next run; a one-off direct delete already cleared them from Synology.
|
||||
--exclude='/ollama/' # LLM models — re-pullable
|
||||
--exclude='/prometheus-backup/' # metrics TSDB snapshots
|
||||
--exclude='/audiblez/' # generated audiobooks
|
||||
--exclude='/ebook2audiobook/' # generated audiobooks
|
||||
|
||||
# ---- Synology / Windows / macOS cruft ----
|
||||
--exclude='/@eaDir/'
|
||||
--exclude='*@synoeastream'
|
||||
|
|
@ -119,7 +136,7 @@ mountpoint -q /mnt/backup || { log "FATAL: /mnt/backup not mounted"; push_metric
|
|||
[ -d "$SRC" ] || { log "FATAL: source $SRC missing"; push_metrics 1 0; exit 1; }
|
||||
|
||||
log "=== mirror starting: $SRC → $DST ==="
|
||||
log "skip: immich (Synology direct), frigate (no backup), temp (no backup), anca-elements"
|
||||
log "skip: immich (Synology direct), frigate/temp (no backup), anca-elements, ollama/prometheus-backup/audiblez/ebook2audiobook (regenerable, live-only)"
|
||||
|
||||
# Marker file used to identify files written by this rsync run, so we can append
|
||||
# their paths to the offsite-sync manifest. Touch BEFORE rsync; `find -newer` AFTER.
|
||||
|
|
|
|||
|
|
@ -95,16 +95,17 @@ fi
|
|||
# reaching Synology via Step 1 (sda → pve-backup/). frigate and temp are
|
||||
# excluded from both legs — intentionally NOT backed up.
|
||||
#
|
||||
# nfs-ssd is handled separately below: its three dirs (immich, ollama,
|
||||
# llamacpp) all go direct to Synology since /srv/nfs-ssd is not mirrored
|
||||
# to sda. ollama+llamacpp are small enough (~85G total) that the direct
|
||||
# leg is fine and we don't need to extend nfs-mirror to cover the SSD.
|
||||
# nfs-ssd: as of 2026-06-01 this leg is ALSO immich-only. ollama (59G) and
|
||||
# llamacpp (26G) on the SSD were filling the offsite Synology (5.3T hit 97%)
|
||||
# for re-pullable model blobs, so they're dropped — live copy stays on the
|
||||
# SSD, no offsite. The monthly --delete pass below reaps them from Synology
|
||||
# nfs-ssd/; a one-off direct delete cleared the bulk on 2026-06-01.
|
||||
#
|
||||
# Keep this aligned with /usr/local/bin/nfs-mirror's EXCLUDES — the
|
||||
# excludes there are { immich (this leg), frigate (no backup), temp
|
||||
# (no backup), anca-elements (deleted), pvc-data and friends (owned by
|
||||
# daily-backup) }. Only the bypass-leg subset matters here: { immich }.
|
||||
log "--- Step 2: NFS → Synology (immich-only direct leg + nfs-ssd) ---"
|
||||
# Keep this aligned with /usr/local/bin/nfs-mirror's EXCLUDES. Both legs now
|
||||
# carry immich only; everything else is either curated through sda (Step 1)
|
||||
# or intentionally live-only (frigate, temp, ollama, llamacpp, audiblez,
|
||||
# ebook2audiobook, prometheus-backup).
|
||||
log "--- Step 2: NFS → Synology (immich-only on both nfs/ and nfs-ssd/) ---"
|
||||
|
||||
# Regex matching paths NOT on sda (must reach Synology directly).
|
||||
NFS_SDA_BYPASS_RE='^/srv/nfs/immich/'
|
||||
|
|
@ -123,9 +124,9 @@ if [ "${DAY_OF_MONTH}" -le 7 ]; then
|
|||
log "Monthly full NFS sync (immich-only — reaps legacy bypass dirs)..."
|
||||
rsync -rlt --delete "${NFS_FULL_INCLUDES[@]}" /srv/nfs/ "${NFS_DEST}/" 2>&1 \
|
||||
&& log " OK: nfs/ full sync (immich-only)" || { warn "nfs/ full sync failed"; STATUS=1; }
|
||||
# nfs-ssd: full sync of all three dirs (immich, ollama, llamacpp).
|
||||
rsync -rlt --delete /srv/nfs-ssd/ "${NFS_SSD_DEST}/" 2>&1 \
|
||||
&& log " OK: nfs-ssd/ full sync" || { warn "nfs-ssd/ full sync failed"; STATUS=1; }
|
||||
# nfs-ssd: immich-only (2026-06-01) — --delete reaps legacy ollama/llamacpp.
|
||||
rsync -rlt --delete "${NFS_FULL_INCLUDES[@]}" /srv/nfs-ssd/ "${NFS_SSD_DEST}/" 2>&1 \
|
||||
&& log " OK: nfs-ssd/ full sync (immich-only)" || { warn "nfs-ssd/ full sync failed"; STATUS=1; }
|
||||
> "${NFS_CHANGE_LOG}"
|
||||
elif [ -s "${NFS_CHANGE_LOG}" ]; then
|
||||
# Incremental: only sync changed files matching the bypass leg (immich).
|
||||
|
|
@ -147,8 +148,8 @@ elif [ -s "${NFS_CHANGE_LOG}" ]; then
|
|||
|| { warn "nfs/ incremental failed"; STATUS=1; }
|
||||
fi
|
||||
|
||||
# SSD NFS — every nfs-ssd path (immich/ollama/llamacpp) ships direct.
|
||||
grep '^/srv/nfs-ssd/' /tmp/nfs-changes-deduped | \
|
||||
# SSD NFS — immich-only (2026-06-01); ollama/llamacpp are live-only, no offsite.
|
||||
grep '^/srv/nfs-ssd/immich/' /tmp/nfs-changes-deduped | \
|
||||
while IFS= read -r f; do [ -f "$f" ] && echo "${f#/srv/nfs-ssd/}"; done \
|
||||
> /tmp/sync-nfs-ssd.list 2>/dev/null || true
|
||||
SSD_COUNT=$(wc -l < /tmp/sync-nfs-ssd.list 2>/dev/null || echo 0)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue