k8s-version-upgrade: dynamic worker enumeration + IP-based SSH (auto-cover all/new nodes)
All checks were successful
ci/woodpecker/push/default Pipeline was successful

The chain hardcoded master→node4→node3→node2→node1→postflight and SSHed by
FQDN. It silently SKIPPED node5/node6 (added 2026-05-26) — postflight would
have failed even if reachable — and node5/node6 had no .viktorbarzin.lan DNS
records, so the chain couldn't SSH to them at all.

Refactor (upgrade-step.sh):
  - Worker set + order derived live from `kubectl get nodes` (worker_nodes /
    next_pending_worker), so EVERY worker still off-target is upgraded and a
    newly-joined node is covered with zero script change.
  - SSH targets are node InternalIPs (ssh_target), removing the dependency on
    node DNS records entirely — a new node is reachable the moment it joins.
  - The two remaining hardcoded loops (containerd skew, apt-repo rewrite) now
    enumerate workers/all-nodes dynamically too.
  - Topology preserved: master-drain Job runs on the first worker; every
    worker-drain Job runs on the already-upgraded k8s-master (self-preemption
    invariant intact).
  - next_pending_worker returns 0 explicitly on the no-match path — the
    `while read … done < <(…)` loop exits 1 at EOF, which under set -e would
    abort the LAST worker's Job before it spawns postflight (cluster upgraded
    but no cleanup / in_flight reset). Caught in review.

Docs (runbook + architecture + headers) updated to the dynamic topology.

NOTE: nodes still need the k8s-upgrade SSH public key in authorized_keys; it was
deployed to node4/5/6 by hand this session. Baking it into node provisioning
(so new nodes get it automatically) is the remaining follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-06-17 16:56:02 +00:00
parent 0c5a9b5f44
commit ed53b34bf4
4 changed files with 112 additions and 79 deletions

View file

@ -10,7 +10,7 @@ drain target** — so no pod in the chain can preempt itself.
The chain (Sun 12:00 UTC weekly):
```
detection CronJob → preflight Job → master Job → worker × 4 Jobs → postflight Job
detection CronJob → preflight Job → master Job → one worker Job per worker (enumerated live) → postflight Job
```
This is **independent** of the OS-side `unattended-upgrades + kured`
@ -76,14 +76,16 @@ Job 6 — postflight (no pinning)
└── Slack: ✅ K8s upgrade complete
```
**Pin choices summarised:**
- k8s-node1 hosts every Job that drains master or another worker. k8s-node1
itself is upgraded **last**.
- k8s-master hosts Job 5 (which drains k8s-node1). Job 5's spec includes a
toleration for `node-role.kubernetes.io/control-plane:NoSchedule`.
- If anyone reorders the worker sequence, the pin for Job 5 needs to track
whatever worker is upgraded last. The mapping is in `scripts/upgrade-step.sh`
→ the `case "${PHASE}:${TARGET_NODE:-}"` block.
**Pin choices summarised** (dynamic since 2026-06-17 — no hardcoded node list):
- The **master-drain Job** runs on the first worker (the "runner"); since it
drains the control-plane node, it must not run there.
- **Every worker-drain Job** runs on **k8s-master** (already upgraded by then),
with a `node-role.kubernetes.io/control-plane:NoSchedule` toleration — so a
Job never runs on the node it drains (self-preemption invariant).
- The worker set + order come from `kubectl get nodes` at runtime
(`worker_nodes` / `next_pending_worker` in `scripts/upgrade-step.sh`), so
**adding a node needs no change** — the chain upgrades every worker still
off-target, then runs postflight. SSH uses node InternalIPs (no DNS needed).
## Components