diff --git a/docs/runbooks/k8s-version-upgrade.md b/docs/runbooks/k8s-version-upgrade.md index 4b4b42b0..ef42b9ff 100644 --- a/docs/runbooks/k8s-version-upgrade.md +++ b/docs/runbooks/k8s-version-upgrade.md @@ -3,11 +3,11 @@ ## Overview Kubernetes component versions (`kubeadm`/`kubelet`/`kubectl`) on the 7 K8s -nodes (k8s-master + k8s-node1..6) are upgraded automatically by a nightly +nodes (k8s-master + k8s-node1..6) are upgraded automatically by a weekly detection CronJob that seeds a chain of small phase Jobs. Each Job is **pinned to a node that is NOT its drain target** — so no pod in the chain can preempt itself. -The chain (23:00 UTC nightly): +The chain (weekly Sunday 23:00 UTC): ``` detection CronJob → preflight Job → master Job → one worker Job per worker (enumerated live) → postflight Job @@ -16,7 +16,7 @@ detection CronJob → preflight Job → master Job → one worker Job per worker This is **independent** of the OS-side `unattended-upgrades + kured` pipeline (see `k8s-node-auto-upgrades.md`). They do not share rollouts. Schedules can overlap (kured runs daily 02:00-06:00 London; detection -here runs 23:00 UTC nightly) — when a kured reboot lands within 24h of the +here runs weekly Sunday 23:00 UTC) — when a kured reboot lands within 24h of the Sunday detection, the `RecentNodeReboot` alert in the Upgrade Gates group blocks the version-upgrade preflight, so the chain self-defers to the next Sunday rather than rolling on top of a half-fresh node. @@ -24,7 +24,7 @@ to the next Sunday rather than rolling on top of a half-fresh node. ## Architecture ``` -k8s-version-check CronJob (23:00 UTC nightly, k8s-upgrade ns, SA: k8s-upgrade-job) +k8s-version-check CronJob (weekly Sunday 23:00 UTC, k8s-upgrade ns, SA: k8s-upgrade-job) │ kubectl get nodes → running version │ ssh master 'apt-cache madison kubeadm' → latest patch (within current minor) │ HEAD pkgs.k8s.io/.../v/deb/Release → next minor available? @@ -134,11 +134,11 @@ for actionable, `k8s_upgrade_held=1` for held), sets `HALT_CHAIN` so the chain doesn't advance, and **exits 0 — the Job Completes cleanly** (a refusal is a decision, not a failure: no Failed Job, no `K8sUpgradeChainJobFailed`). It's before any mutation, so no rollback. Reasons (grouped by class) appear in the -**morning nightly report**, not a per-run Slack. +**morning weekly report**, not a per-run Slack. - **Actionable** → `K8sUpgradeBlocked` fires (once, via alert-on-change). Clear - it by doing the named upgrade/migration; the next nightly run proceeds. -- **Held** → **deliberately NO alert** — only the nightly report's `⏸️ HELD` + it by doing the named upgrade/migration; the next weekly run proceeds. +- **Held** → **deliberately NO alert** — only the weekly report's `⏸️ HELD` line, because it can't be actioned now (a nightly alert would cry wolf). It clears itself once upstream ships support (refresh `addon-compat.json`) or the pin is lifted (delete `pinned`+`pin_reason`). The detector re-evaluates every @@ -171,7 +171,7 @@ it current**; the gate reads it on every run. Gate logic: | **ConfigMap `k8s-upgrade-job-template`** | Mounts `/template/job-template.yaml` — universal Job manifest with envsubst placeholders. Rendered by upgrade-step.sh and the detection CronJob via `envsubst | kubectl apply`. | | **ServiceAccount `k8s-upgrade-job`** | Used by both the detection CronJob and every chain Job. ClusterRole binding grants: nodes get/list/patch, pods/eviction create, pods delete, batch/jobs CRUD, PDB list (for predrain_unstick), CronJob get (snapshot trigger), namespaces patch on `k8s-upgrade` only. Namespace-scoped Role binding grants secrets:get on `k8s-upgrade-creds`. | | **ExternalSecret `k8s-upgrade-creds`** | Syncs `secret/k8s-upgrade/{ssh_key, slack_webhook}` from Vault. Mounted into every Job at `/secrets/k8s-upgrade`. | -| **CronJob `k8s-version-check`** | 23:00 UTC nightly. Probes apt + pkgs.k8s.io for target. If found, renders Job 0 from `job-template.yaml` and applies it. | +| **CronJob `k8s-version-check`** | weekly Sunday 23:00 UTC. Probes apt + pkgs.k8s.io for target. If found, renders Job 0 from `job-template.yaml` and applies it. | ### Pushgateway metrics @@ -194,14 +194,15 @@ Pushed by upgrade-step.sh during phase execution; observed by the - **`EtcdPreUpgradeSnapshotMissing`** — `k8s_upgrade_in_flight==1 && k8s_upgrade_snapshot_taken==0` for 10m. Catches preflight Stage 2 failing silently. - **`K8sUpgradeStalled`** — `k8s_upgrade_in_flight==1 && time()-k8s_upgrade_started_timestamp > 5400` for 5m. Catches a Job in the chain dying without spawning its successor. - **`K8sUpgradeChainJobFailed`** — `kube_job_status_failed{namespace="k8s-upgrade",job_name=~"k8s-upgrade-(preflight|master|worker|postflight)-.*",reason=~"BackoffLimitExceeded|DeadlineExceeded"} > 0` for 15m (warning). Catches a phase Job that **terminally failed before `k8s_upgrade_in_flight` was set** — the preflight gates exit pre-metric, so the two `in_flight`-based alerts above are blind to a failed preflight (this is what hid the 5-day 1.34.9 wedge on 2026-06-12). Reason-scoped to terminal job conditions so a retry-success doesn't false-positive (a bare failed-pod-count would otherwise also block kured for the Job's 7d TTL). The old `unless on() (k8s_upgrade_blocked == 1)` clause was **dropped 2026-06-28**: compat-gate refusals now Complete cleanly (exit 0) instead of Failing, so a terminally-Failed chain Job again means a genuine wedge with nothing to exclude. -- **`K8sUpgradeBlocked`** — `k8s_upgrade_blocked == 1` (warning). An **ACTIONABLE** compat-gate refusal — a newer version of the lagging addon exists and upgrading it would clear the block (or an in-use deprecated API must be migrated / a node's containerd bumped). Reasons (grouped by class) are in the **morning nightly report**; clear it by doing the named upgrade/migration, after which the next nightly run proceeds (see "Auto-upgrade compat gate"). No upgrade was attempted, so this is not a half-done-rollout alert. **There is deliberately NO companion alert for the held verdict** (`k8s_upgrade_held=1` — waiting-on-upstream / pinned): nothing can be actioned now, so it is surfaced only by the nightly report's `⏸️ HELD` line. +- **`K8sUpgradeBlocked`** — `k8s_upgrade_blocked == 1` (warning). An **ACTIONABLE** compat-gate refusal — a newer version of the lagging addon exists and upgrading it would clear the block (or an in-use deprecated API must be migrated / a node's containerd bumped). Reasons (grouped by class) are in the **morning weekly report**; clear it by doing the named upgrade/migration, after which the next weekly run proceeds (see "Auto-upgrade compat gate"). No upgrade was attempted, so this is not a half-done-rollout alert. **There is deliberately NO companion alert for the held verdict** (`k8s_upgrade_held=1` — waiting-on-upstream / pinned): nothing can be actioned now, so it is surfaced only by the weekly report's `⏸️ HELD` line. - The first four alerts ALSO block kured (same `--prometheus-url` halt-on-alert mechanism) so the OS-reboot pipeline can't run on top of a half-done version upgrade. -### Nightly upgrade report (Slack) +### Weekly upgrade report (Slack) CronJob `k8s-upgrade-nightly-report` (k8s-upgrade ns, `var.report_schedule`, -default `7 6 * * *` = 06:07 UTC — after the 23:00 chain, before the 08:00 London -alert-digest) posts ONE Slack summary each morning of the previous night's run: +default `7 6 * * 1` = Monday 06:07 UTC — after the Sunday-night chain, before the +08:00 London alert-digest; historical CronJob name kept) posts ONE Slack summary +each Monday of the past week's run: running version, detector freshness, detected target + kind, the outcome (⚪ no upgrade needed / 🔴 blocked-actionable + reasons / ⏸️ held = waiting-upstream/pinned / 🟢 upgraded / 🟡 in progress / ⚠️ detector stale), and recent chain jobs. Read-only — it reads @@ -209,7 +210,7 @@ the Pushgateway gauges + live nodes/jobs and re-runs `compat-gate.py` for fresh blocker reasons; reuses the chain's SA + `slack_webhook` + scripts ConfigMap. Logic + unit tests: `scripts/nightly-report.py`, `scripts/test_nightly_report.py`. This is the day-to-day visibility layer (it does NOT replace the alerts above — -those fire on problems; this reports the outcome every night). Manual run: +those fire on problems; this reports the outcome every week). Manual run: `kubectl -n k8s-upgrade create job --from=cronjob/k8s-upgrade-nightly-report nightly-report-test` (name it WITHOUT a `k8s-upgrade-{phase}-` prefix so a failure can't trip `K8sUpgradeChainJobFailed`). @@ -356,7 +357,7 @@ which `K8sUpgradeStalled` cannot see). `spawn_next` delete + re-spawn a terminally-Failed Job instead of skipping it on name-existence (retry-on-failure), so a transient preflight gate — e.g. a spurious critical alert like the ttyd web-terminal probe that wedged 1.34.9 for 5 days — -clears on the next daily cycle. A mid-chain phase that keeps failing still needs +clears on the next weekly cycle. A mid-chain phase that keeps failing still needs manual recovery: fix the root cause, then: ```bash diff --git a/stacks/k8s-version-upgrade/main.tf b/stacks/k8s-version-upgrade/main.tf index c812afd9..6c2ef060 100644 --- a/stacks/k8s-version-upgrade/main.tf +++ b/stacks/k8s-version-upgrade/main.tf @@ -27,13 +27,15 @@ variable "schedule" { type = string - # Nightly 23:00 UTC (00:00 London) — overnight / low cluster usage, and clear + # Weekly Sunday 23:00 UTC (00:00 London) — overnight / low cluster usage, clear # of the kured OS-reboot window (01:00-05:00 UTC = 02:00-06:00 London) so the - # two drain-pipelines never overlap. Moved from 12:00 UTC noon on 2026-06-17 - # (Viktor: disruptive node drains should run overnight). Was weekly Sunday - # until 2026-05-18. Concurrency bounded by the CronJob's Forbid policy + - # retry-on-failure Job-name idempotency. - default = "0 23 * * *" + # two drain-pipelines never overlap. Cadence history: weekly Sunday until + # 2026-05-18 → daily noon → daily 23:00 (2026-06-17) → back to weekly Sunday + # (2026-06-28). Rationale for weekly: the actionable-vs-held gate now quiets the + # routine "held" churn (e.g. 1.36), so a daily check/attempt buys little; weekly + # is enough and patch uptake lags ≤7d (an accepted trade-off). Concurrency + # bounded by the CronJob's Forbid policy + retry-on-failure Job-name idempotency. + default = "0 23 * * 0" } variable "enabled" { @@ -41,13 +43,15 @@ variable "enabled" { default = true } -# Nightly upgrade-report CronJob schedule. 06:07 UTC (07:07 London) — safely -# after the 23:00 chain has finished (worst case ~02:00) and before the 08:00 -# London alert-digest, so the morning Slack skim shows last night's upgrade -# outcome + any live blocker. Posts once/day; read-only. +# Weekly upgrade-report CronJob schedule. Monday 06:07 UTC (07:07 London) — the +# morning AFTER the Sunday-night check (~7h later, so nightly-report.py's ~25h +# staleness threshold stays valid AND still flags a missed weekly run), and before +# the 08:00 London alert-digest so the Monday skim shows the weekly upgrade +# outcome + any live blocker. Posts once/week; read-only. (The CronJob keeps the +# historical name k8s-upgrade-nightly-report — not renamed to avoid churn.) variable "report_schedule" { type = string - default = "7 6 * * *" + default = "7 6 * * 1" } # Mirrors `local.image_tag` in stacks/claude-agent-service/main.tf — bump @@ -494,16 +498,16 @@ resource "kubernetes_cron_job_v1" "k8s_version_check" { # Idempotency: deterministic name reconciles via `apply`. JOB_NAME="k8s-upgrade-preflight-$${TARGET//./-}" MASTER_JOB="k8s-upgrade-master-$${TARGET//./-}" - ANNOUNCE=yes # Slack the spawn? Suppressed for silent nightly re-evaluations of a standing gate refusal. + ANNOUNCE=yes # Slack the spawn? Suppressed for silent per-run re-evaluations of a standing gate refusal. - # Idempotency + nightly re-evaluation: + # Idempotency + per-run re-evaluation: # - FAILED preflight (transient gate abort, e.g. a spurious # critical alert / unhealthy node) -> delete + re-spawn, announced. # - COMPLETE preflight but NO master Job spawned -> the compat # gate REFUSED the target (blocked/held now Complete cleanly # rather than Failing). Re-spawn SILENTLY so the gate re-checks - # nightly (the refusal may have cleared: addon upgraded / matrix - # updated / upstream shipped) WITHOUT nightly Slack noise for a + # each scheduled run (the refusal may have cleared: addon + # upgraded / matrix updated / upstream shipped) WITHOUT per-run Slack noise for a # standing refusal — the morning report (+ K8sUpgradeBlocked for # actionable) is the signal. # - Otherwise (Active, or Complete with the chain advanced) -> skip. @@ -521,7 +525,7 @@ resource "kubernetes_cron_job_v1" "k8s_version_check" { slack "Preflight Job $JOB_NAME exists but FAILED — deleting and re-spawning" /usr/local/bin/kubectl -n k8s-upgrade delete job "$JOB_NAME" --wait=true >/dev/null 2>&1 || true elif [ "$JOB_COMPLETE" = "True" ] && ! /usr/local/bin/kubectl -n k8s-upgrade get job "$MASTER_JOB" >/dev/null 2>&1; then - echo "Preflight $JOB_NAME Complete + no master Job (gate refused) — silent nightly re-evaluate" + echo "Preflight $JOB_NAME Complete + no master Job (gate refused) — silent per-run re-evaluate" /usr/local/bin/kubectl -n k8s-upgrade delete job "$JOB_NAME" --wait=true >/dev/null 2>&1 || true ANNOUNCE=no else @@ -591,7 +595,7 @@ resource "kubernetes_cron_job_v1" "k8s_version_check" { # # Each morning, after the 23:00 chain has finished, posts ONE concise Slack # report of last night's upgrade outcome (no-op / blocked+reasons / upgraded / -# in-progress) so the autonomous upgrader's nightly result — and any live +# in-progress) so the autonomous upgrader's weekly result — and any live # blocker — is visible at a glance. Read-only: reads the chain's Pushgateway # gauges + live nodes/jobs and re-runs compat-gate.py for fresh blocker reasons. # Reuses the same SA, creds secret (slack_webhook), and scripts ConfigMap as the diff --git a/stacks/k8s-version-upgrade/scripts/nightly-report.py b/stacks/k8s-version-upgrade/scripts/nightly-report.py index 53eb79ea..531a2979 100644 --- a/stacks/k8s-version-upgrade/scripts/nightly-report.py +++ b/stacks/k8s-version-upgrade/scripts/nightly-report.py @@ -1,18 +1,18 @@ #!/usr/bin/env python3 -"""Nightly k8s-upgrade report -> Slack. +"""Weekly k8s-upgrade report -> Slack. -Runs each morning (CronJob k8s-upgrade-nightly-report) after the 23:00 UTC -version-check chain has finished. Reads the chain's Pushgateway gauges + live -cluster state and posts ONE concise, actionable report to Slack so the -autonomous upgrader's nightly outcome — and any blocker holding it back — is -visible at a glance during the upgrade-cleanup window. +Runs Monday morning (CronJob k8s-upgrade-nightly-report — historical name kept) +after the Sunday-night version-check chain has finished. Reads the chain's +Pushgateway gauges + live cluster state and posts ONE concise, actionable report +to Slack so the autonomous upgrader's weekly outcome — and any blocker holding it +back — is visible at a glance during the upgrade-cleanup window. Outcomes it distinguishes: ⚪ no upgrade needed — cluster already at the latest supported patch 🔴 BLOCKED — compat gate refused the target; lists live reasons 🟢 UPGRADED — all nodes now on the detected target 🟡 in progress / passed — gate passed, chain mid-flight (or partial) - ⚠️ detector STALE — the 23:00 detector did not run last night + ⚠️ detector STALE — the weekly (Sun 23:00) detector did not run Read-only. The pure helpers (parse_metrics / select / fmt_age / compose_report) are unit-tested in test_nightly_report.py; all I/O (kubectl, Pushgateway, the @@ -28,7 +28,7 @@ import urllib.request PUSHGW = os.environ.get("PUSHGW", "http://prometheus-prometheus-pushgateway.monitoring:9091/metrics") SLACK_FILE = os.environ.get("SLACK_FILE", "/secrets/k8s-upgrade/slack_webhook") SCRIPTS_DIR = os.environ.get("SCRIPTS_DIR", "/scripts") -STALE_SECONDS = 90000 # ~25h: a nightly detector older than this didn't run last night +STALE_SECONDS = 90000 # ~25h: report runs Mon ~7h after the Sun-night check, so >25h ⇒ the weekly check didn't fire _METRIC_RE = re.compile(r"^(?P\w+)(?:\{(?P[^}]*)\})?\s+(?P[-+0-9.eE]+)\s*$") _LABEL_RE = re.compile(r'(\w+)="([^"]*)"')