ead876ec65
All checks were successful
ci/woodpecker/push/default Pipeline was successful
Adds a daily visibility layer so every night's autonomous-upgrade outcome is reviewable at a glance during the upgrade-cleanup window (Viktor: "track every night's upgrade for the next 7 days; clean up all bugs and blockers"). Last night (2026-06-20) confirmed BOTH prior fixes work in production: the detector resolved target 1.35.6 (k8s_upgrade_available) and the compat gate correctly REFUSED it (k8s_upgrade_blocked=1 -> K8sUpgradeBlocked) because ESO v0.12 (<=1.31) and kyverno v1.16 (<=1.34) don't support 1.35. What's here: - CronJob k8s-upgrade-nightly-report (06:07 UTC) -> one Slack summary/morning: running version, detector freshness, detected target, outcome (no-op / blocked+live reasons / upgraded / in-progress / detector-stale), recent jobs. Read-only: reads Pushgateway gauges + live nodes/jobs, re-runs compat-gate.py for fresh blockers; reuses the chain SA + slack_webhook + scripts ConfigMap. Pure helpers unit-tested (test_nightly_report.py, 8 cases incl. a real v-prefix bug TDD caught). Verified end-to-end in-cluster (posted to Slack). - K8sUpgradeChainJobFailed regex scoped from `k8s-upgrade-.*` to `k8s-upgrade-(preflight|master|worker|postflight)-.*` so the new report job (or any future helper) can't false-trip the chain-wedged alarm. Manual state repair (no git artifact): imported the orphaned `alert-digest` CronJob into the monitoring stack state (`tg import module.monitoring.kubernetes_cron_job_v1.alert_digest monitoring/alert-digest`). Root cause: when alert_digest was added (2026-06-12) the apply recorded its ConfigMap + Secret but not the CronJob, so every full monitoring apply since has failed with `cronjobs.batch "alert-digest" already exists` (Woodpecker pipeline 298 today) — surviving only via targeted prometheus applies. Now in state, so monitoring CI applies cleanly again. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| apiserver-audit-logging.md | ||
| beads-auto-dispatch.md | ||
| breakglass-ssh.md | ||
| breakglass-ui.md | ||
| chrome-service-snapshot.md | ||
| claude-auth-renew-workstation.md | ||
| fan-control.md | ||
| forgejo-open-signups.md | ||
| forgejo-registry-breakglass.md | ||
| forgejo-registry-rebuild-image.md | ||
| forgejo-registry-setup.md | ||
| grow-pve-nfs-lv.md | ||
| immich-transcode-bitrate.md | ||
| job-hunter.md | ||
| k8s-node-auto-upgrades.md | ||
| k8s-version-upgrade.md | ||
| kms-public-exposure.md | ||
| mailserver-pfsense-haproxy.md | ||
| mailserver-proxy-protocol.md | ||
| nextcloud-add-archive.md | ||
| nfs-prerequisites.md | ||
| offboard-user.md | ||
| pfsense-unbound.md | ||
| proxmox-host.md | ||
| r730-ram-upgrade-272gb.md | ||
| registry-rebuild-image.md | ||
| registry-vm.md | ||
| restore-etcd.md | ||
| restore-full-cluster.md | ||
| restore-lvm-snapshot.md | ||
| restore-mysql.md | ||
| restore-postgresql.md | ||
| restore-pvc-from-backup.md | ||
| restore-vault.md | ||
| restore-vaultwarden.md | ||
| scale-k8s-cluster.md | ||
| security-incident.md | ||
| synology-storage.md | ||
| t3-drop-attribution.md | ||
| t3-version-bump.md | ||
| technitium-apply.md | ||
| vault-raft-leader-deadlock.md | ||
| vault-token-renew-devvm.md | ||
| woodpecker-onboard-forgejo-repo.md | ||