viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
infra/scripts
History
Viktor Barzin 1979c2b213 cluster-health: add checks 43 + 44 (PVE host thermals + load) 
Both new checks SSH read-only to the PVE host and emit PASS/WARN/FAIL
via the standard healthcheck output + JSON. They run alongside the
existing 42 checks and surface the same alerts the 2026-05-20/21
optimization session had to gather by hand.

#43 PVE Host Thermals — Xeon E5-2699v4 package + per-core temps
  Reads every /sys/class/hwmon/hwmon0/temp*_input in one SSH round-trip.
  Thresholds tuned to the live TjMax=83 / Tcrit=93:
    PASS  < 75 °C package
    WARN  75-82 °C  (approaching max, action time)
    FAIL  >= 83 °C  (at/above TjMax, throttling imminent)
  Reports hottest core label too so a single hot core doesn't hide in
  the package average.

#44 PVE Host Load — load avg vs 44-thread capacity
  Reads /proc/loadavg, compares 5-min to thread count (44):
    PASS  load_5 < 30   (< 70% threads busy)
    WARN  30-37         (oversubscribed but not saturating)
    FAIL  >= 38         (~85%+ threads busy — scheduler saturation)
  Uses 5-min so brief work spikes don't false-fail.

Both gracefully WARN-degrade if SSH BatchMode fails, matching the
existing check 36 (LVM PVC snapshots) pattern. TOTAL_CHECKS bumped
42 -> 44 and the dispatcher updated.
2026-05-22 14:17:01 +00:00
..
server_safe_poweroff move helper scripts in scripts dir [ci skip] 2025-10-11 17:14:59 +00:00
check-ingress-auth-comments.py infra/scripts/tg: enforce ingress_factory auth-comment convention 2026-05-22 14:16:44 +00:00
cluster_healthcheck.sh cluster-health: add checks 43 + 44 (PVE host thermals + load) 2026-05-22 14:17:01 +00:00
cluster_manager.py chore: add untracked stacks, scripts, and agent configs 2026-04-15 09:33:06 +00:00
daily-backup.service backup: fix daily-backup silent failures, postiz pg_dump CronJob, doc reconcile 2026-05-10 11:12:39 +00:00
daily-backup.sh scripts: timeout rsync + sqlite calls in daily-backup 2026-05-22 14:16:42 +00:00
daily-backup.timer rename weekly-backup → daily-backup across scripts, timers, services, and docs [ci skip] 2026-04-13 18:37:04 +00:00
extend_vm_storage.sh [ci skip] expand k8s worker nodes to 256G, update inventory and extend script 2026-02-28 16:00:16 +00:00
forgejo-migrate-orphan-images.sh [forgejo] Migration script: exclude empty repos, all-images full mode 2026-05-07 23:29:34 +00:00
frigate-bulk-classify.js [ci skip] sync tfstate and add frigate helper scripts 2026-02-12 23:11:23 +00:00
frigate-inspect.mjs [ci skip] sync tfstate and add frigate helper scripts 2026-02-12 23:11:23 +00:00
gen_service_stacks.py cleanup: remove calibre and audiobookshelf stacks after ebooks migration [ci skip] 2026-03-25 23:56:07 +02:00
graceful-db-maintenance.sh add pod dependency management via Kyverno init container injection 2026-03-15 19:17:57 +00:00
image_pull.sh chore: add untracked stacks, scripts, and agent configs 2026-04-15 09:33:06 +00:00
image_pull_remote.sh chore: add untracked stacks, scripts, and agent configs 2026-04-15 09:33:06 +00:00
kill_ns.sh move helper scripts in scripts dir [ci skip] 2025-10-11 17:14:59 +00:00
lvm-pvc-snapshot.sh [backup] Fix lvm-pvc-snapshot Pushgateway push (stdout pollution in cmd_prune_count) 2026-04-25 14:30:58 +00:00
lvm-pvc-snapshot.timer add 3-2-1 backup pipeline: weekly PVC file copy, NFS mirror, pfsense, offsite sync 2026-04-06 14:53:28 +03:00
migrate-state-to-pg [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
migrate_service_state.sh cleanup: remove calibre and audiobookshelf stacks after ebooks migration [ci skip] 2026-03-25 23:56:07 +02:00
nfs-change-tracker.service consolidate offsite backup: inotify change tracking, deduplicate Synology paths [ci skip] 2026-04-13 18:06:20 +00:00
node_registry_manager.sh some nits on the registry manager script - note it is still not working correctly [ci skip] 2025-10-17 19:23:43 +00:00
offsite-sync-backup.service rename weekly-backup → daily-backup across scripts, timers, services, and docs [ci skip] 2026-04-13 18:37:04 +00:00
offsite-sync-backup.sh rename weekly-backup → daily-backup across scripts, timers, services, and docs [ci skip] 2026-04-13 18:37:04 +00:00
offsite-sync-backup.timer switch backup + offsite sync from weekly to daily — RPO 7d → 1d [ci skip] 2026-04-13 18:24:38 +00:00
parse-postmortem-todos.sh fix: use sh instead of bash in pipeline (Alpine compat) 2026-04-14 17:29:14 +00:00
pfsense-haproxy-bootstrap.php mailserver: split healthcheck path off PROXY-aware listeners + book-search uses ClusterIP 2026-05-05 19:45:33 +00:00
pfsense-nat-mailserver-haproxy-flip.php [mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] 2026-04-19 12:24:50 +00:00
pfsense-nat-mailserver-haproxy-unflip.php [mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] 2026-04-19 12:24:50 +00:00
postmortem-pipeline.sh [claude-agent-service] Migrate all pipelines from DevVM SSH to K8s HTTP 2026-04-18 10:12:02 +00:00
pve-nfs-exports fix(post-mortem): add /etc/exports to git, NFS health check in daily-backup, document CSI requirements [PM-2026-04-14] 2026-04-14 18:08:24 +00:00
renew_worker_certs.sh move helper scripts in scripts dir [ci skip] 2025-10-11 17:14:59 +00:00
setup-containerd-pullthrough.sh chore: add untracked stacks, scripts, and agent configs 2026-04-15 09:33:06 +00:00
setup-forgejo-containerd-mirror.sh [forgejo] Phase 0 of registry consolidation: prepare Forgejo OCI registry 2026-05-07 23:29:33 +00:00
setup-task-pipeline.sh [ci skip] add Forgejo task pipeline for OpenClaw AI agent 2026-03-07 21:11:07 +00:00
setup_containerd_mirrors.sh add upstream fallback to containerd registry mirrors 2026-04-02 11:05:30 +03:00
state-sync [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
stop_storage_services.sh cleanup: remove calibre and audiobookshelf stacks after ebooks migration [ci skip] 2026-03-25 23:56:07 +02:00
task-processor.sh [ci skip] add Forgejo task pipeline for OpenClaw AI agent 2026-03-07 21:11:07 +00:00
tg infra/scripts/tg: enforce ingress_factory auth-comment convention 2026-05-22 14:16:44 +00:00
update-istio-injection.sh move helper scripts in scripts dir [ci skip] 2025-10-11 17:14:59 +00:00
update_k8s.sh k8s-version-upgrade: retry kubeadm apply on static-pod-hash timeout 2026-05-22 14:17:00 +00:00
update_node.sh k8s-version-upgrade: automated kubeadm/kubelet/kubectl upgrade pipeline 2026-05-22 14:16:42 +00:00
upgrade_state.sh upgrade-state: filter transient registry digest-check errors 2026-05-22 14:17:00 +00:00
vault-kubeconfig remove SOPS pipeline, deploy ESO + Vault DB/K8s engines 2026-03-15 16:37:38 +00:00
woodpecker-register-forgejo-repo.sh [woodpecker] Programmatic Forgejo repo registration 2026-05-10 11:12:36 +00:00