infra/scripts
Viktor Barzin 4315ed5c2a [backup] Fix lvm-pvc-snapshot Pushgateway push (stdout pollution in cmd_prune_count)
cmd_prune_count's `log "  Pruned: ..."` wrote to stdout, which the
caller captures via `pruned=$(cmd_prune_count)`. From 2026-04-16 onward
(7d retention kicked in), pruned snapshots polluted the captured value
with multi-line log text, breaking the Prometheus exposition format
on the metric push (`lvm_snapshot_pruned_total ${pruned}` → 400 from
Pushgateway). Snapshots themselves were always fine; only the metric
push silently failed for ~9 nights, eventually triggering
LVMSnapshotNeverRun (alert has 48h `for:`).

Fix: redirect the inner log call to stderr so cmd_prune_count's stdout
contains only the count. Also adopts `infra/scripts/lvm-pvc-snapshot.sh`
as the source-of-truth (was edited only on the PVE host) and updates
backup-dr.md to point at the .sh and document the scp deploy.

Deploy: scp infra/scripts/lvm-pvc-snapshot.sh root@192.168.1.127:/usr/local/bin/lvm-pvc-snapshot

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 14:30:58 +00:00
..
server_safe_poweroff
cluster_healthcheck.sh
cluster_manager.py
daily-backup.service
daily-backup.sh
daily-backup.timer
extend_vm_storage.sh
frigate-bulk-classify.js
frigate-inspect.mjs
gen_service_stacks.py
graceful-db-maintenance.sh
image_pull.sh
image_pull_remote.sh
kill_ns.sh
lvm-pvc-snapshot.sh [backup] Fix lvm-pvc-snapshot Pushgateway push (stdout pollution in cmd_prune_count) 2026-04-25 14:30:58 +00:00
lvm-pvc-snapshot.timer
migrate-state-to-pg
migrate_service_state.sh
nfs-change-tracker.service
node_registry_manager.sh
offsite-sync-backup.service
offsite-sync-backup.sh
offsite-sync-backup.timer
parse-postmortem-todos.sh
pfsense-haproxy-bootstrap.php
pfsense-nat-mailserver-haproxy-flip.php
pfsense-nat-mailserver-haproxy-unflip.php
postmortem-pipeline.sh
pve-nfs-exports
renew_worker_certs.sh
setup-containerd-pullthrough.sh
setup-task-pipeline.sh
setup_containerd_mirrors.sh
state-sync
stop_storage_services.sh
task-processor.sh
tg
update-istio-injection.sh
update_k8s.sh
update_node.sh
vault-kubeconfig