k8s-upgrade: nightly Slack report monitor + scope chain-failed alert to phases

Adds a daily visibility layer so every night's autonomous-upgrade outcome is reviewable at a glance during the upgrade-cleanup window (Viktor: "track every night's upgrade for the next 7 days; clean up all bugs and blockers"). Last night (2026-06-20) confirmed BOTH prior fixes work in production: the detector resolved target 1.35.6 (k8s_upgrade_available) and the compat gate correctly REFUSED it (k8s_upgrade_blocked=1 -> K8sUpgradeBlocked) because ESO v0.12 (<=1.31) and kyverno v1.16 (<=1.34) don't support 1.35. What's here: - CronJob k8s-upgrade-nightly-report (06:07 UTC) -> one Slack summary/morning: running version, detector freshness, detected target, outcome (no-op / blocked+live reasons / upgraded / in-progress / detector-stale), recent jobs. Read-only: reads Pushgateway gauges + live nodes/jobs, re-runs compat-gate.py for fresh blockers; reuses the chain SA + slack_webhook + scripts ConfigMap. Pure helpers unit-tested (test_nightly_report.py, 8 cases incl. a real v-prefix bug TDD caught). Verified end-to-end in-cluster (posted to Slack). - K8sUpgradeChainJobFailed regex scoped from `k8s-upgrade-.*` to `k8s-upgrade-(preflight|master|worker|postflight)-.*` so the new report job (or any future helper) can't false-trip the chain-wedged alarm. Manual state repair (no git artifact): imported the orphaned `alert-digest` CronJob into the monitoring stack state (`tg import module.monitoring.kubernetes_cron_job_v1.alert_digest monitoring/alert-digest`). Root cause: when alert_digest was added (2026-06-12) the apply recorded its ConfigMap + Secret but not the CronJob, so every full monitoring apply since has failed with `cronjobs.batch "alert-digest" already exists` (Woodpecker pipeline 298 today) — surviving only via targeted prometheus applies. Now in state, so monitoring CI applies cleanly again. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 16:57:44 +00:00 · 2026-06-21 16:57:44 +00:00 · ead876ec65
commit ead876ec65
parent 7270e2be3b
6 changed files with 431 additions and 3 deletions
--- a/stacks/k8s-version-upgrade/scripts/nightly-report.py
+++ b/stacks/k8s-version-upgrade/scripts/nightly-report.py
@ -0,0 +1,224 @@
+#!/usr/bin/env python3
+"""Nightly k8s-upgrade report -> Slack.
+
+Runs each morning (CronJob k8s-upgrade-nightly-report) after the 23:00 UTC
+version-check chain has finished. Reads the chain's Pushgateway gauges + live
+cluster state and posts ONE concise, actionable report to Slack so the
+autonomous upgrader's nightly outcome — and any blocker holding it back — is
+visible at a glance during the upgrade-cleanup window.
+
+Outcomes it distinguishes:
+  ⚪ no upgrade needed     — cluster already at the latest supported patch
+  🔴 BLOCKED              — compat gate refused the target; lists live reasons
+  🟢 UPGRADED             — all nodes now on the detected target
+  🟡 in progress / passed — gate passed, chain mid-flight (or partial)
+  ⚠️  detector STALE       — the 23:00 detector did not run last night
+
+Read-only. The pure helpers (parse_metrics / select / fmt_age / compose_report)
+are unit-tested in test_nightly_report.py; all I/O (kubectl, Pushgateway, the
+compat-gate subprocess, Slack) lives in thin wrappers below them.
+"""
+import json
+import os
+import re
+import subprocess
+import sys
+import urllib.request
+
+PUSHGW = os.environ.get("PUSHGW", "http://prometheus-prometheus-pushgateway.monitoring:9091/metrics")
+SLACK_FILE = os.environ.get("SLACK_FILE", "/secrets/k8s-upgrade/slack_webhook")
+SCRIPTS_DIR = os.environ.get("SCRIPTS_DIR", "/scripts")
+STALE_SECONDS = 90000  # ~25h: a nightly detector older than this didn't run last night
+
+_METRIC_RE = re.compile(r"^(?P<name>\w+)(?:\{(?P<labels>[^}]*)\})?\s+(?P<value>[-+0-9.eE]+)\s*$")
+_LABEL_RE = re.compile(r'(\w+)="([^"]*)"')
+
+
+# ---------------------------------------------------------------- pure helpers
+def parse_metrics(text):
+    """Prometheus text exposition -> list of (name, {labels}, float value)."""
+    out = []
+    for line in (text or "").splitlines():
+        line = line.strip()
+        if not line or line.startswith("#"):
+            continue
+        m = _METRIC_RE.match(line)
+        if not m:
+            continue
+        labels = dict(_LABEL_RE.findall(m.group("labels") or ""))
+        try:
+            val = float(m.group("value"))
+        except ValueError:
+            continue
+        out.append((m.group("name"), labels, val))
+    return out
+
+
+def select(metrics, name):
+    """All (labels, value) tuples for a given metric name."""
+    return [(lbl, val) for (n, lbl, val) in metrics if n == name]
+
+
+def fmt_age(seconds):
+    if seconds < 0:
+        return "in the future?!"
+    if seconds < 3600:
+        return f"{int(seconds // 60)}m ago"
+    if seconds < 86400:
+        return f"{seconds / 3600:.1f}h ago"
+    return f"{seconds / 86400:.1f}d ago"
+
+
+def compose_report(now_ts, nodes, metrics, blocker_reasons, jobs):
+    """Build the Slack message text from gathered facts. PURE.
+
+    nodes: list of (name, kubeletVersion). metrics: parse_metrics() output.
+    blocker_reasons: multi-line str (compat-gate output) or None.
+    jobs: list of {name, status, age_s}.
+    """
+    # kubelet reports "v1.35.6" but the gauges carry "1.35.6" — normalise so the
+    # UPGRADED comparison against the target actually matches.
+    versions = sorted({v.lstrip("v") for _, v in nodes})
+    if len(versions) == 1:
+        node_line = f"Running: *{versions[0]}* (all {len(nodes)} nodes uniform)"
+    elif versions:
+        node_line = f"Running: *MIXED* {', '.join(versions)} across {len(nodes)} nodes"
+    else:
+        node_line = "Running: *unknown* (could not read nodes)"
+
+    lr = select(metrics, "k8s_version_check_last_run_timestamp")
+    stale = False
+    if lr:
+        age = now_ts - lr[0][1]
+        stale = age > STALE_SECONDS
+        run_line = f"Last detector run: {fmt_age(age)} ({'STALE ⚠️' if stale else 'fresh ✓'})"
+    else:
+        run_line = "Last detector run: *unknown* (no metric)"
+        stale = True
+
+    avail = [(lbl, val) for lbl, val in select(metrics, "k8s_upgrade_available") if val == 1]
+    blocked = any(val == 1 for _, val in select(metrics, "k8s_upgrade_blocked"))
+
+    if avail:
+        lbl = avail[0][0]
+        target = lbl.get("target", "?")
+        kind = lbl.get("kind", "?")
+        tgt_line = f"Detected target: *{target}* ({kind})"
+        if blocked:
+            headline = f"🔴 BLOCKED — compat gate refused {target}"
+        elif len(versions) == 1 and target == versions[0]:
+            headline = f"🟢 UPGRADED — all nodes now on {target}"
+        else:
+            headline = f"🟡 IN PROGRESS / gate passed for {target}"
+    else:
+        target = None
+        tgt_line = "Detected target: none"
+        headline = "⚪ No upgrade needed (cluster at latest supported patch)"
+
+    if stale:
+        headline = "⚠️ Detector did not run last night — " + headline
+
+    msg = [f"*[k8s-upgrade nightly]* {headline}", node_line, run_line, tgt_line]
+
+    if blocked and blocker_reasons:
+        msg.append("Blockers (live):")
+        for r in blocker_reasons.splitlines():
+            r = r.strip()
+            if r:
+                msg.append(f"  • {r}")
+
+    if jobs:
+        msg.append("Chain jobs (recent):")
+        for j in jobs:
+            msg.append(f"  • {j['name']}: {j['status']} ({fmt_age(j['age_s'])})")
+
+    return "\n".join(msg)
+
+
+# ----------------------------------------------------------------------- I/O
+def _kubectl_json(args):
+    try:
+        r = subprocess.run(["kubectl", *args], capture_output=True, text=True, timeout=30)
+        return json.loads(r.stdout) if r.stdout.strip() else {}
+    except Exception:
+        return {}
+
+
+def get_nodes():
+    d = _kubectl_json(["get", "nodes", "-o", "json"])
+    return [(it["metadata"]["name"],
+             it.get("status", {}).get("nodeInfo", {}).get("kubeletVersion", "?"))
+            for it in d.get("items", [])]
+
+
+def _job_status(it):
+    st = it.get("status", {})
+    for c in st.get("conditions", []):
+        if c.get("type") == "Failed" and c.get("status") == "True":
+            return "Failed"
+        if c.get("type") == "Complete" and c.get("status") == "True":
+            return "Complete"
+    if st.get("active"):
+        return "Active"
+    return "Pending"
+
+
+def get_jobs(now_ts):
+    import datetime
+    d = _kubectl_json(["-n", "k8s-upgrade", "get", "jobs", "-o", "json"])
+    out = []
+    for it in d.get("items", []):
+        ct = it["metadata"].get("creationTimestamp")
+        try:
+            age = now_ts - datetime.datetime.strptime(
+                ct, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=datetime.timezone.utc).timestamp()
+        except Exception:
+            age = 0
+        if age <= 93600:  # last 26h only
+            out.append({"name": it["metadata"]["name"], "status": _job_status(it), "age_s": age})
+    return sorted(out, key=lambda j: j["age_s"])
+
+
+def get_blocker_reasons(target):
+    try:
+        with open(f"{SCRIPTS_DIR}/addon-compat.json") as f:
+            matrix = f.read()
+        r = subprocess.run(["python3", f"{SCRIPTS_DIR}/compat-gate.py", target],
+                           input=matrix, capture_output=True, text=True, timeout=60)
+        return r.stdout.strip()
+    except Exception as e:
+        return f"(could not run compat-gate: {e})"
+
+
+def post_slack(text):
+    if os.environ.get("DRY_RUN"):
+        return  # main() always prints the message; DRY_RUN just skips the POST
+    with open(SLACK_FILE) as f:
+        url = f.read().strip()
+    data = json.dumps({"text": text}).encode()
+    req = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json"})
+    urllib.request.urlopen(req, timeout=20)
+
+
+def main():
+    import time
+    now_ts = float(os.environ.get("NOW_TS", "")) if os.environ.get("NOW_TS") else time.time()
+    try:
+        metrics_txt = urllib.request.urlopen(PUSHGW, timeout=20).read().decode()
+    except Exception:
+        metrics_txt = ""
+    metrics = parse_metrics(metrics_txt)
+    nodes = get_nodes()
+    jobs = get_jobs(now_ts)
+
+    avail = [(lbl, val) for lbl, val in select(metrics, "k8s_upgrade_available") if val == 1]
+    blocked = any(val == 1 for _, val in select(metrics, "k8s_upgrade_blocked"))
+    reasons = get_blocker_reasons(avail[0][0].get("target", "")) if (avail and blocked) else None
+
+    msg = compose_report(now_ts, nodes, metrics, reasons, jobs)
+    post_slack(msg)
+    print(msg)
+
+
+if __name__ == "__main__":
+    main()
--- a/stacks/k8s-version-upgrade/scripts/test_nightly_report.py
+++ b/stacks/k8s-version-upgrade/scripts/test_nightly_report.py
@ -0,0 +1,81 @@
+"""Unit tests for nightly-report.py (pure helpers only).
+
+Run: pytest stacks/k8s-version-upgrade/scripts/test_nightly_report.py
+Loaded via importlib because the filename has a hyphen.
+"""
+import importlib.util
+import pathlib
+
+HERE = pathlib.Path(__file__).parent
+_spec = importlib.util.spec_from_file_location("nightly_report", HERE / "nightly-report.py")
+nr = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(nr)
+
+LAST_RUN = 1781996424.0  # 2026-06-20T23:00:24Z — matches last night's gauge
+METRICS_BLOCKED = f"""# TYPE k8s_upgrade_available gauge
+k8s_upgrade_available{{instance="",job="k8s-version-check",kind="minor",running="1.34.9",target="1.35.6"}} 1
+k8s_upgrade_blocked{{instance="",job="k8s-version-upgrade"}} 1
+k8s_version_check_last_run_timestamp{{instance="",job="k8s-version-check"}} {LAST_RUN}
+"""
+NODES_UNIFORM = [(f"k8s-node{i}", "v1.34.9") for i in range(7)]
+
+
+def test_parse_metrics_basic():
+    m = nr.parse_metrics(METRICS_BLOCKED)
+    names = {n for n, _, _ in m}
+    assert names == {"k8s_upgrade_available", "k8s_upgrade_blocked", "k8s_version_check_last_run_timestamp"}
+    avail = nr.select(m, "k8s_upgrade_available")
+    assert avail[0][0]["target"] == "1.35.6"
+    assert avail[0][0]["kind"] == "minor"
+    assert avail[0][1] == 1.0
+
+
+def test_parse_metrics_ignores_comments_and_junk():
+    assert nr.parse_metrics("# HELP foo\n\ngarbage line\n") == []
+
+
+def test_fmt_age():
+    assert nr.fmt_age(120) == "2m ago"
+    assert nr.fmt_age(7200) == "2.0h ago"
+    assert nr.fmt_age(172800) == "2.0d ago"
+
+
+def test_compose_blocked_lists_reasons():
+    m = nr.parse_metrics(METRICS_BLOCKED)
+    reasons = ("addon external-secrets v0.12 supports k8s <= 1.31; target 1.35 exceeds it\n"
+               "addon kyverno v1.16 supports k8s <= 1.34; target 1.35 exceeds it")
+    out = nr.compose_report(LAST_RUN + 30000, NODES_UNIFORM, m, reasons, [])
+    assert "🔴 BLOCKED" in out and "1.35.6" in out
+    assert "external-secrets" in out and "kyverno" in out
+    assert "all 7 nodes uniform" in out
+    assert "fresh ✓" in out
+
+
+def test_compose_noop_when_no_target():
+    m = nr.parse_metrics(f'k8s_version_check_last_run_timestamp{{}} {LAST_RUN}\n')
+    out = nr.compose_report(LAST_RUN + 30000, NODES_UNIFORM, m, None, [])
+    assert "⚪ No upgrade needed" in out
+
+
+def test_compose_upgraded_when_nodes_match_target():
+    m = nr.parse_metrics(f"""k8s_upgrade_available{{kind="minor",target="1.35.6"}} 1
+k8s_upgrade_blocked{{}} 0
+k8s_version_check_last_run_timestamp{{}} {LAST_RUN}
+""")
+    nodes = [(f"k8s-node{i}", "v1.35.6") for i in range(7)]
+    out = nr.compose_report(LAST_RUN + 30000, nodes, m, None, [])
+    assert "🟢 UPGRADED" in out and "1.35.6" in out
+
+
+def test_compose_stale_detector_flagged():
+    m = nr.parse_metrics(METRICS_BLOCKED)
+    out = nr.compose_report(LAST_RUN + 200000, NODES_UNIFORM, m, "x", [])  # ~55h later
+    assert "Detector did not run last night" in out
+    assert "STALE" in out
+
+
+def test_compose_includes_recent_jobs():
+    m = nr.parse_metrics(METRICS_BLOCKED)
+    jobs = [{"name": "k8s-upgrade-preflight-1-35-6", "status": "Failed", "age_s": 3600}]
+    out = nr.compose_report(LAST_RUN + 30000, NODES_UNIFORM, m, "x", jobs)
+    assert "k8s-upgrade-preflight-1-35-6: Failed" in out