viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

nfs-mirror: weekly Mon 04:00 → daily 02:00

Browse source 
Steady-state delta runs in 10-20 min and the weekly cadence left a
real RPO gap: app data under /srv/nfs/<svc>/ that isn't a PVC
(captured by daily-backup) or a *-backup CronJob (captured daily by
the CronJob writing to /srv/nfs/<svc>-backup/) was on a 7-day worst
case for off-disk durability. Affected paths include nextcloud shared
files, audiobookshelf library, mailserver Maildir, calibre, servarr
metadata, real-estate-crawler scraped data, openclaw agent state.
Daily cadence drops their RPO to ~24h at negligible cost.

Slot: 02:00, 3h ahead of daily-backup (05:00) so the manifest is
populated before offsite-sync reads it at 06:00.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-05-26 19:00:10 +00:00
parent 1eee56d0ba
commit 37d88ce50e
2 changed files with 17 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

13
docs/architecture/backup-dr.md
View file

@ -16,6 +16,11 @@ Last updated: 2026-05-26
> exist on `/srv/nfs`. Dropped from the exclude/include lists as no-ops.
> - `/mnt/backup/anca-elements` (423 G) deleted — canonical copy lives in
> Immich since the 2026-05-24 ingest.
> - **`nfs-mirror.timer`: weekly Mon 04:00 → daily 02:00.** Steady-state
> delta is 10-20 min of mostly-metadata rsync, so the IO cost is
> negligible. RPO for non-CronJob app data (nextcloud shared files,
> audiobookshelf library, mailserver Maildir, real-estate-crawler scraped
> data, etc.) drops from 7 days to ~24h.
> - Aftermath: sda 87% → 46% used; Synology `/Viki/nfs/` shrinks to
> immich-only on next monthly `--delete` pass (or manual cleanup —
> see runbook).
@ -32,7 +37,7 @@ Last updated: 2026-05-26
The homelab runs a 3-2-1 strategy with a **two-leg** path to Synology so every NFS byte takes exactly one route to offsite (no duplication, no gaps):
```
sdc /srv/nfs/<svc>/ ──nfs-mirror weekly──→ sda /mnt/backup/<svc>/ ──offsite-sync Step 1──→ Synology /Backup/Viki/pve-backup/<svc>/ [leg 1]
sdc /srv/nfs/<svc>/ ──nfs-mirror daily 02:00──→ sda /mnt/backup/<svc>/ ──offsite-sync Step 1──→ Synology /Backup/Viki/pve-backup/<svc>/ [leg 1]
sdc /srv/nfs/immich/ ──inotify (nfs-change-tracker)──→ offsite-sync Step 2 ──→ Synology /Backup/Viki/nfs/immich/ [leg 2]
sdc PVCs (LVM thin) ──daily-backup~snapshot~rsync──→ sda /mnt/backup/{pvc-data,sqlite-backup,pfsense,pve-config}/ ──Step 1──→ Synology /Backup/Viki/pve-backup/
```
@ -373,7 +378,7 @@ Two-step offsite sync:
**`/srv/nfs/anca-elements/` history**: had its own dedicated Synology exclusion line earlier in 2026-05-24 because the original Synology source (`/volume1/Backup/Anca/Elements`) was being preserved while we moved canonical to PVE. After the original was deleted (same day), anca-elements joined the broader "NOT bypassing sda" category and is covered by Step 1 via `nfs-mirror`.
**Layer 3a: NFS local mirror on sda (3-2-1 second copy)**: `/usr/local/bin/nfs-mirror` rsyncs `/srv/nfs/` → `/mnt/backup/<service>/` weekly (Mon 04:00). Single rsync invocation, single destination. As of 2026-05-26 the skip-list (in `nfs-mirror.sh` `EXCLUDES`) is intentionally minimal:
**Layer 3a: NFS local mirror on sda (3-2-1 second copy)**: `/usr/local/bin/nfs-mirror` rsyncs `/srv/nfs/` → `/mnt/backup/<service>/` daily at 02:00 (switched from weekly Mon 04:00 on 2026-05-26 — steady-state delta is 10-20 min of mostly-metadata rsync, cuts non-CronJob app-data RPO from 7d to ~24h). Single rsync invocation, single destination. As of 2026-05-26 the skip-list (in `nfs-mirror.sh` `EXCLUDES`) is intentionally minimal:
- **immich** (1.5 T) — too big for sda; ships sdc → Synology direct (leg 2)
- **frigate** (camera ring buffer) — intentionally NOT backed up
@ -444,8 +449,8 @@ The btrfs cleaner thread reclaims async — `df` may lag the snapshot-delete by
| `/etc/systemd/system/lvm-pvc-snapshot.timer` | Daily 03:00 (LVM snapshots) |
| `/etc/systemd/system/daily-backup.timer` | Daily 05:00 (file backup) |
| `/etc/systemd/system/offsite-sync-backup.timer` | Daily 06:00 (offsite sync) |
| `/usr/local/bin/nfs-mirror` | PVE host: weekly selective mirror of /srv/nfs/* → sda /mnt/backup/<svc>/ (Layer 3a) |
| `/etc/systemd/system/nfs-mirror.timer` | Weekly Mon 04:00 (NFS local mirror to sda) |
| `/usr/local/bin/nfs-mirror` | PVE host: daily 02:00 mirror of /srv/nfs/* → sda /mnt/backup/<svc>/ (Layer 3a) |
| `/etc/systemd/system/nfs-mirror.timer` | Daily 02:00 (NFS local mirror to sda) |
| `stacks/dbaas/` | Terraform: PostgreSQL/MySQL backup CronJobs |
| `stacks/vault/` | Terraform: Vault backup CronJob |
| `stacks/vaultwarden/` | Terraform: Vaultwarden backup + integrity CronJobs |

10
scripts/nfs-mirror.timer
View file

@ -1,8 +1,14 @@
[Unit]
Description=Weekly local NFS mirror to /mnt/backup
Description=Daily local NFS mirror to /mnt/backup
[Timer]
OnCalendar=Mon *-*-* 04:00:00
# Daily 02:00 — runs 3h before daily-backup (05:00) so the .changed-files
# manifest is populated and offsite-sync (06:00) ships both legs' deltas.
# Switched from weekly Mon 04:00 → daily 2026-05-26: steady-state delta is
# 10-20 min of mostly-metadata rsync, so the IO cost is negligible and it
# cuts non-CronJob app-data RPO from 7d to ~24h (matters for nextcloud
# shared files, audiobookshelf library, mailserver Maildir, etc.).
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=15min