backup: stop offsite-copying regenerable data; shrink nextcloud backup; pin nextcloud image

The offsite Synology hit 97% — the Backup share grew +670G in a week, traced
to the 2026-05-26 change that began mirroring large regenerable services
offsite, plus an unbounded nextcloud.log bloating its backups to 87G.

- nfs-mirror: re-exclude ollama, prometheus-backup, audiblez, ebook2audiobook
  (regenerable; live-only on sdc). Keep *-backup DB dumps (real safety copies).
- offsite-sync Step 2: nfs-ssd leg is now immich-only; ollama/llamacpp on the
  SSD no longer ship offsite (re-pullable models).
- daily-backup: skip nextcloud/nextcloud-data-proxmox (orphaned pre-encryption
  PV, still backed up weekly).
- nextcloud: cap+rotate the log (log_rotate_size=10MB); the dedicated backup
  now excludes html/ (app code, from image), logs, and preview cache and keeps
  only the latest copy (pvc-data holds version history) → <5G (was 87G).
- nextcloud: pin image to 32.0.9 in chart_values. A 2026-05-26 Keel bump moved
  the live pod to 32.0.9 (data migrated to 32.0.9.2) but TF still defaulted to
  32.0.3; reconciling that drift this session rolled a 32.0.3 pod that
  CrashLooped on the downgrade. Pinning eliminates the drift.

Docs: backup-dr.md + infra CLAUDE.md updated (add nfs-mirror, new exclusions).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-06-01 14:01:21 +00:00
parent 0dd4a31eff
commit ddd582a28c
7 changed files with 122 additions and 35 deletions

View file

@ -1,3 +1,14 @@
# Pin the image to 32.0.9 (apache). On 2026-05-26 Keel bumped the live
# Deployment 32.0.3 → 32.0.9-apache and the DATA migrated to 32.0.9.2; Keel
# was then disabled but chart_values was never pinned, so it kept defaulting
# to the chart's appVersion (32.0.3). A 2026-06-01 `terragrunt apply`
# reconciled that drift, rolled a 32.0.3 pod, and Nextcloud refused to
# downgrade (data 32.0.9.2 > image 32.0.3.2) → CrashLoopBackOff. Pinning here
# keeps TF the source of truth and matches the on-disk data version.
image:
flavor: apache
tag: "32.0.9"
nextcloud:
host: nextcloud.viktorbarzin.me
trustedDomains:
@ -51,6 +62,10 @@ nextcloud:
<?php
$CONFIG = array(
'loglevel' => 2,
// Cap + rotate nextcloud.log. Without this it grew unbounded to
// 10GB+ and bloated every backup (2026-06-01 space incident).
// At 10MB the log rotates to nextcloud.log.1 (1 kept) → ~20MB max.
'log_rotate_size' => 10485760,
'mail_smtpdebug' => false,
);
zzz-mysql.config.php: |

View file

@ -382,14 +382,31 @@ resource "kubernetes_config_map" "backup-script" {
# Create backup directory
mkdir -p "$BACKUP_PATH"
# Backup everything (config, data, custom_apps, themes, etc.)
# Backup config/data/custom_apps. Exclusions (2026-06-01 space fix):
# - nextcloud.log* rotated at source via log_rotate_size; previously
# grew to 10GB+ and bloated every dated copy (backups hit 20G each).
# - preview cache regenerable thumbnails, no need to back up.
# Backs up config/, data/, custom_apps/ (the irreplaceable bits). Skips:
# - html/ the Nextcloud app code, reproducible from the pinned image
# (real config is at config/config.php; html/config/config.php is empty).
# - nextcloud.log* capped at source via log_rotate_size; was 10GB+.
# - preview cache regenerable thumbnails.
echo "Backing up Nextcloud installation..."
rsync -a "$DATA_DIR/" "$BACKUP_PATH/"
rsync -a \
--exclude='/html/' \
--exclude='nextcloud.log' \
--exclude='nextcloud.log.*' \
--exclude='data/appdata_*/preview/' \
"$DATA_DIR/" "$BACKUP_PATH/"
# Keep only last 7 backups
# Keep only the latest backup. The version history lives in daily-backup's
# pvc-data (4 weekly snapshot-consistent copies of this same encrypted PVC),
# so this browsable app-level copy only needs the most recent. Keeping the
# whole installation (incl. logs) x7 here was the bulk of the 87G that
# filled the offsite Synology.
echo "Cleaning old backups..."
cd "$BACKUP_DIR"
ls -dt */ | tail -n +8 | xargs -r rm -rf
ls -dt */ | tail -n +2 | xargs -r rm -rf
echo "Backup completed at $(date)"
echo "Backup stored at: $BACKUP_PATH"