infra/scripts
Viktor Barzin 4d756be4f5
Some checks failed
ci/woodpecker/push/build-cli Pipeline failed
ci/woodpecker/push/default Pipeline failed
backup: consolidate to one local-mirror script + invert offsite filter
Before this commit, the in-flight design split anca-elements (its own
mirror script + timer) from the rest of /srv/nfs (still going to
Synology via inotify-tracked offsite-sync). It also meant Synology
received some bytes via both paths (sda → Synology AND direct NFS →
Synology), which doubled consumption.

This commit collapses both into a clean 3-2-1:

  Copy 1 (sdc):       live /srv/nfs/* + cluster block PVCs
  Copy 2 (sda):       /mnt/backup/{pvc-data,sqlite-backup,pfsense,
                                   pve-config,<critical-nfs>/}
                      ← daily-backup + nfs-mirror (one script each)
  Copy 3 (Synology):  /Backup/Viki/{pve-backup,nfs,nfs-ssd}
                      ← offsite-sync-backup Step 1 (sda → Synology)
                        + Step 2 (sda-BYPASS paths only → Synology direct)

scripts/nfs-mirror.{sh,service,timer}:
  New consolidated weekly mirror. Replaces anca-elements-mirror (to be
  removed in a follow-up after the current in-flight rsync completes,
  parity-verified, and Synology source-of-truth is deleted). Single
  rsync /srv/nfs/ → /mnt/backup/ with an explicit EXCLUDES list that
  drops paths not worth a local 2nd copy: immich (1.2T — too big),
  frigate (14d ring), prometheus/loki (rebuildable), ollama/llamacpp/
  audiblez/ebook2audiobook (re-fetchable), *-backup (already backups),
  temp/alertmanager (transient). Nice=10, IOSchedulingClass=idle.

scripts/offsite-sync-backup.sh:
  Step 2 (NFS → Synology) filter inverted: instead of `--exclude=
  anca-elements/`, it now `--include`s only the sda-BYPASS paths
  (immich, frigate, prometheus, *-backup, …). The bypass-include
  regex MUST stay in lockstep with nfs-mirror's EXCLUDES — they are
  complementary and any drift creates either gaps or duplication on
  Synology. Comment in the script flags this.

monitoring alerts: renamed AncaElementsMirror{Stale,Failing} to
NfsMirror{Stale,Failing} matching the new metric job name
`nfs-mirror`. Thresholds unchanged.

docs/architecture/backup-dr.md: rewritten Step 1/Step 2 sections and
added the bypass-list rationale + cross-reference between scripts.

NOT YET DEPLOYED — gated on the in-flight anca-elements-mirror rsync
finishing + parity verification + Synology /volume1/Backup/Anca/
Elements deletion. The old scripts (anca-elements-{mirror,sync.sh})
remain on the PVE host until then, and will be removed in a cleanup
commit.
2026-05-24 12:49:20 +00:00
..
server_safe_poweroff move helper scripts in scripts dir [ci skip] 2025-10-11 17:14:59 +00:00
anca-elements-mirror.service anca-elements: weekly local mirror sdc → sda (replaces Synology as 2nd copy) 2026-05-24 11:51:52 +00:00
anca-elements-mirror.sh anca-elements: weekly local mirror sdc → sda (replaces Synology as 2nd copy) 2026-05-24 11:51:52 +00:00
anca-elements-mirror.timer anca-elements: weekly local mirror sdc → sda (replaces Synology as 2nd copy) 2026-05-24 11:51:52 +00:00
anca-elements-sync.sh docs+scripts: lock in nextcloud-as-PVE-NFS-browser surface 2026-05-24 11:45:01 +00:00
check-ingress-auth-comments.py infra/scripts/tg: enforce ingress_factory auth-comment convention 2026-05-11 19:18:27 +00:00
cluster_healthcheck.sh cluster-health #43: tighten PVE thermal threshold to 65 C 2026-05-22 14:09:08 +00:00
cluster_manager.py chore: add untracked stacks, scripts, and agent configs 2026-04-15 09:33:06 +00:00
daily-backup.service backup: fix daily-backup silent failures, postiz pg_dump CronJob, doc reconcile 2026-05-09 17:41:04 +00:00
daily-backup.sh scripts: timeout rsync + sqlite calls in daily-backup 2026-05-10 18:39:07 +00:00
daily-backup.timer rename weekly-backup → daily-backup across scripts, timers, services, and docs [ci skip] 2026-04-13 18:37:04 +00:00
extend_vm_storage.sh [ci skip] expand k8s worker nodes to 256G, update inventory and extend script 2026-02-28 16:00:16 +00:00
forgejo-migrate-orphan-images.sh [forgejo] Migration script: exclude empty repos, all-images full mode 2026-05-07 17:21:39 +00:00
frigate-bulk-classify.js [ci skip] sync tfstate and add frigate helper scripts 2026-02-12 23:11:23 +00:00
frigate-inspect.mjs [ci skip] sync tfstate and add frigate helper scripts 2026-02-12 23:11:23 +00:00
gen_service_stacks.py cleanup: remove calibre and audiobookshelf stacks after ebooks migration [ci skip] 2026-03-25 23:56:07 +02:00
graceful-db-maintenance.sh add pod dependency management via Kyverno init container injection 2026-03-15 19:17:57 +00:00
image_pull.sh chore: add untracked stacks, scripts, and agent configs 2026-04-15 09:33:06 +00:00
image_pull_remote.sh chore: add untracked stacks, scripts, and agent configs 2026-04-15 09:33:06 +00:00
kill_ns.sh move helper scripts in scripts dir [ci skip] 2025-10-11 17:14:59 +00:00
lvm-pvc-snapshot.sh [backup] Fix lvm-pvc-snapshot Pushgateway push (stdout pollution in cmd_prune_count) 2026-04-25 14:30:58 +00:00
lvm-pvc-snapshot.timer add 3-2-1 backup pipeline: weekly PVC file copy, NFS mirror, pfsense, offsite sync 2026-04-06 14:53:28 +03:00
migrate-state-to-pg [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
migrate_service_state.sh cleanup: remove calibre and audiobookshelf stacks after ebooks migration [ci skip] 2026-03-25 23:56:07 +02:00
nfs-change-tracker.service offsite-sync-backup + nfs-change-tracker: exclude /srv/nfs/anca-elements 2026-05-24 11:03:09 +00:00
nfs-mirror.service backup: consolidate to one local-mirror script + invert offsite filter 2026-05-24 12:49:20 +00:00
nfs-mirror.sh backup: consolidate to one local-mirror script + invert offsite filter 2026-05-24 12:49:20 +00:00
nfs-mirror.timer backup: consolidate to one local-mirror script + invert offsite filter 2026-05-24 12:49:20 +00:00
node_registry_manager.sh some nits on the registry manager script - note it is still not working correctly [ci skip] 2025-10-17 19:23:43 +00:00
offsite-sync-backup.service rename weekly-backup → daily-backup across scripts, timers, services, and docs [ci skip] 2026-04-13 18:37:04 +00:00
offsite-sync-backup.sh backup: consolidate to one local-mirror script + invert offsite filter 2026-05-24 12:49:20 +00:00
offsite-sync-backup.timer switch backup + offsite sync from weekly to daily — RPO 7d → 1d [ci skip] 2026-04-13 18:24:38 +00:00
parse-postmortem-todos.sh fix: use sh instead of bash in pipeline (Alpine compat) 2026-04-14 17:29:14 +00:00
pfsense-haproxy-bootstrap.php mailserver: split healthcheck path off PROXY-aware listeners + book-search uses ClusterIP 2026-05-05 19:45:33 +00:00
pfsense-nat-mailserver-haproxy-flip.php [mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] 2026-04-19 12:24:50 +00:00
pfsense-nat-mailserver-haproxy-unflip.php [mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] 2026-04-19 12:24:50 +00:00
postmortem-pipeline.sh [claude-agent-service] Migrate all pipelines from DevVM SSH to K8s HTTP 2026-04-18 10:12:02 +00:00
pve-nfs-exports fix(post-mortem): add /etc/exports to git, NFS health check in daily-backup, document CSI requirements [PM-2026-04-14] 2026-04-14 18:08:24 +00:00
renew_worker_certs.sh move helper scripts in scripts dir [ci skip] 2025-10-11 17:14:59 +00:00
setup-containerd-pullthrough.sh chore: add untracked stacks, scripts, and agent configs 2026-04-15 09:33:06 +00:00
setup-forgejo-containerd-mirror.sh [forgejo] Phase 0 of registry consolidation: prepare Forgejo OCI registry 2026-05-07 15:51:34 +00:00
setup-task-pipeline.sh [ci skip] add Forgejo task pipeline for OpenClaw AI agent 2026-03-07 21:11:07 +00:00
setup_containerd_mirrors.sh add upstream fallback to containerd registry mirrors 2026-04-02 11:05:30 +03:00
state-sync [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
stop_storage_services.sh cleanup: remove calibre and audiobookshelf stacks after ebooks migration [ci skip] 2026-03-25 23:56:07 +02:00
task-processor.sh [ci skip] add Forgejo task pipeline for OpenClaw AI agent 2026-03-07 21:11:07 +00:00
tg infra/scripts/tg: enforce ingress_factory auth-comment convention 2026-05-11 19:18:27 +00:00
update-istio-injection.sh move helper scripts in scripts dir [ci skip] 2025-10-11 17:14:59 +00:00
update_k8s.sh k8s-version-upgrade: tigera quiesce + etcd-skip retry + IO-wait alert ignore 2026-05-23 08:40:11 +00:00
update_node.sh k8s-version-upgrade: automated kubeadm/kubelet/kubectl upgrade pipeline 2026-05-10 19:07:42 +00:00
upgrade_state.sh upgrade-state: filter transient registry digest-check errors 2026-05-19 22:06:21 +00:00
vault-kubeconfig remove SOPS pipeline, deploy ESO + Vault DB/K8s engines 2026-03-15 16:37:38 +00:00
woodpecker-register-forgejo-repo.sh [woodpecker] Programmatic Forgejo repo registration 2026-05-07 23:33:26 +00:00