Three changes from today's autonomous-pipeline validation session:
1. **Kill-switch ConfigMap** — chain checks for `k8s-upgrade-killswitch`
ConfigMap in `k8s-upgrade` namespace at the top of every phase + at the
start of version-check. Existence halts the chain (exit 0) with a Slack
message. Single-command emergency stop:
kubectl -n k8s-upgrade create configmap k8s-upgrade-killswitch \
--from-literal=reason="storm response"
Resume: kubectl -n k8s-upgrade delete cm k8s-upgrade-killswitch
Role rule for `configmaps` get/list/watch added (resourceName-scoped).
2. **Ignore RecentNodeReboot in halt_on_alert_query everywhere** — the
chain itself causes reboots. The pre-drain master check, post-upgrade
worker check, postflight check, and preflight halt-on-alert all now
pass `RecentNodeReboot` as the extra-ignore. Previously only worker
phase's post-upgrade gate did this. Master Failed silently this morning
on the pre-drain check after my own master reboot.
3. **Preflight quiet-baseline 3600s → 600s** — the 1h cooldown after any
Ready transition meant the chain refused to run for an hour after
every kured reboot. 10 min is enough for kubelet/control-plane to
settle; the 24h-between-cluster-reboots invariant lives in
kured-sentinel-gate, not here.
Validated by running the chain end-to-end: preflight passed in 5s,
master phase now in drain. Today's storm post-mortem (snapshot CoW
amplification + tigera-operator crashloop feedback loop) drove the
kill-switch design.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Same `grep -v` / `set -o pipefail` interaction as commit 10b261d2,
in two more callsites the previous fix didn't cover:
Line 354 (phase_master): control-plane Running check —
`grep -v Running | wc -l` returns 1 when all pods are Running
(the happy path), aborting the chain right after master upgrades.
Line 419 (phase_postflight): on-target node check —
`grep -v ":v$TARGET_VERSION$" | wc -l` returns 1 when all nodes
are on the target version (the happy path, exactly when postflight
should succeed). Aborts at the moment of victory.
Forensics on yesterday's master Job failure (see commit message of
10b261d2 for context): the master Job spawned 16s after the previous
fix's TF apply, before configmap propagation completed on the kubelet.
With those two latent bugs also looming, the chain would have died
post-master-upgrade and again at postflight even if propagation had
been timely.
Wrapping each grep in `{ ... || true; }` so a no-matches result
returns success.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
halt_on_alert_query() ends with `grep -vE "$regex" | sort -u`. When
zero alerts are firing (the desired healthy state), grep matches
nothing and exits 1. Under `set -o pipefail`, the whole pipeline
returns 1; under `set -e`, the caller's `alerts=$(...)` assignment
fails and aborts the script in ~1s with no diagnostic output.
The chain effectively required at least one non-meta alert to be
firing to make any forward progress. Today (2026-05-19) the cluster
is fully clean post-MySQL recovery, the daily 12:00 UTC detection
spawned the preflight Job, and it died instantly — blocking the
1.34.7 → 1.34.8 patch chain.
Fix: wrap the grep in `{ ... || true; }` so a no-matches result
returns success. Preflight verified end-to-end after the fix — the
chain is now in flight (preflight ✓, master phase running).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 24h kubelet-uptime threshold (process_start_time_seconds < 86400)
was a defense-in-depth duplicate of the 24h-since-Ready-transition
check in kured-sentinel-gate Check 4 — but they used different
signals (kubelet process start vs node Ready transition). Whenever
the cluster cycled through reboots, the alert kept firing for a full
day even after sentinel-gate's check passed, and blocked anything
querying halt-on-alert (kured, K8s version-upgrade preflight).
Tightened to 1h (3600s) for "node just rebooted, give it a settle
window". The cluster-wide 24h-between-reboots invariant lives
exclusively in kured-sentinel-gate Check 4 from now on (independent,
uses lastTransitionTime).
Matched the preflight's own 24h-quiet check in upgrade-step.sh
(86400 → 3600) so it doesn't act as a second blocker.
Empirically verified: all 5 kubelets are >10h up, alert cleared on
next eval after the rule reload.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two latent bugs in the K8s-version-upgrade pipeline surfaced when a
real detection run ran post-26.04 upgrade today:
1. **DNS**: pod's CoreDNS search path is `<ns>.svc.cluster.local
svc.cluster.local cluster.local` (+ ndots=2 via Kyverno mutation).
Unqualified `k8s-master` falls through all of those and then queries
upstream Technitium for the bare name → NXDOMAIN. The FQDN
`k8s-master.viktorbarzin.lan` is what Technitium actually serves.
Suffix every node SSH target with `$NODE_DOMAIN`.
2. **envsubst missing**: claude-agent-service image doesn't ship
`gettext-base`. Replace `envsubst <template | apply` with
`python3 -c 'import os,sys; sys.stdout.write(os.path.expandvars(
sys.stdin.read()))' <template | apply`. Same semantics, image
already has python3. Multi-line $SCHEDULING_BLOCK is preserved
correctly through expandvars.
Verified by manually triggering `k8s-version-check` post-fix:
detection now reads `Latest patch: v1.34.8` (currently running 1.34.7)
and spawns `k8s-upgrade-preflight-1-34-8`. The Job pod scheduled and
started; killed before it touched the cluster (will land on Sunday
2026-05-24 12:00 UTC like the schedule says).
Root cause of why these bugs lay dormant: yesterday's first
manual-test detection found "no upgrade needed" so neither code path
exercised SSH or envsubst. Today's apt-source restore (do-release-
upgrade had mangled them) unmasked the v1.34.8 candidate, which made
detection finally proceed past the SSH step.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The agent-based v1 ran inside claude-agent-service (replicas=1, no
nodeSelector) and self-evicted when it tried to drain its host (k8s-node4
on 2026-05-11). Cluster ended half-upgraded (master v1.34.7, workers
v1.34.2) until manual recovery.
Rewrite the pipeline as a chain of nodeSelector-pinned Jobs:
preflight (k8s-node1)
→ master (k8s-node1) drains k8s-master
→ worker × 4 (k8s-node1) drains k8s-node{4,3,2}
→ worker (k8s-master + control-plane toleration) drains k8s-node1
→ postflight (no pinning)
Each Job runs scripts/upgrade-step.sh (case-on-$PHASE) and ends by
envsubst-ing job-template.yaml into the next Job. Deterministic names
(k8s-upgrade-<phase>-<target_version>[-<node>]) make `kubectl apply`
idempotent — a failed Job can be re-created without duplicating
downstream.
Also lands `predrain_unstick`: deletes pods on the target node whose PDB
has 0 disruptionsAllowed. Without this, drain loops indefinitely on
single-replica deployments (e.g. every Anubis instance — discovered the
hard way during 2026-05-11 manual recovery of k8s-node3).
Adds K8sUpgradeStalled alert (in_flight + started_timestamp > 90 min).
Deprecates the agent prompt (renamed to *.deprecated.md with a header
pointer to the new code).
Apply order: k8s-version-upgrade first (consumes new SA + ConfigMaps),
then monitoring (loads the new alert). Both applied 2026-05-11.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
