k8s-version-upgrade: dynamic worker enumeration + IP-based SSH (auto-cover all/new nodes)
All checks were successful
ci/woodpecker/push/default Pipeline was successful
All checks were successful
ci/woodpecker/push/default Pipeline was successful
The chain hardcoded master→node4→node3→node2→node1→postflight and SSHed by
FQDN. It silently SKIPPED node5/node6 (added 2026-05-26) — postflight would
have failed even if reachable — and node5/node6 had no .viktorbarzin.lan DNS
records, so the chain couldn't SSH to them at all.
Refactor (upgrade-step.sh):
- Worker set + order derived live from `kubectl get nodes` (worker_nodes /
next_pending_worker), so EVERY worker still off-target is upgraded and a
newly-joined node is covered with zero script change.
- SSH targets are node InternalIPs (ssh_target), removing the dependency on
node DNS records entirely — a new node is reachable the moment it joins.
- The two remaining hardcoded loops (containerd skew, apt-repo rewrite) now
enumerate workers/all-nodes dynamically too.
- Topology preserved: master-drain Job runs on the first worker; every
worker-drain Job runs on the already-upgraded k8s-master (self-preemption
invariant intact).
- next_pending_worker returns 0 explicitly on the no-match path — the
`while read … done < <(…)` loop exits 1 at EOF, which under set -e would
abort the LAST worker's Job before it spawns postflight (cluster upgraded
but no cleanup / in_flight reset). Caught in review.
Docs (runbook + architecture + headers) updated to the dynamic topology.
NOTE: nodes still need the k8s-upgrade SSH public key in authorized_keys; it was
deployed to node4/5/6 by hand this session. Baking it into node provisioning
(so new nodes get it automatically) is the remaining follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
0c5a9b5f44
commit
ed53b34bf4
4 changed files with 112 additions and 79 deletions
|
|
@ -10,7 +10,7 @@ drain target** — so no pod in the chain can preempt itself.
|
|||
The chain (Sun 12:00 UTC weekly):
|
||||
|
||||
```
|
||||
detection CronJob → preflight Job → master Job → worker × 4 Jobs → postflight Job
|
||||
detection CronJob → preflight Job → master Job → one worker Job per worker (enumerated live) → postflight Job
|
||||
```
|
||||
|
||||
This is **independent** of the OS-side `unattended-upgrades + kured`
|
||||
|
|
@ -76,14 +76,16 @@ Job 6 — postflight (no pinning)
|
|||
└── Slack: ✅ K8s upgrade complete
|
||||
```
|
||||
|
||||
**Pin choices summarised:**
|
||||
- k8s-node1 hosts every Job that drains master or another worker. k8s-node1
|
||||
itself is upgraded **last**.
|
||||
- k8s-master hosts Job 5 (which drains k8s-node1). Job 5's spec includes a
|
||||
toleration for `node-role.kubernetes.io/control-plane:NoSchedule`.
|
||||
- If anyone reorders the worker sequence, the pin for Job 5 needs to track
|
||||
whatever worker is upgraded last. The mapping is in `scripts/upgrade-step.sh`
|
||||
→ the `case "${PHASE}:${TARGET_NODE:-}"` block.
|
||||
**Pin choices summarised** (dynamic since 2026-06-17 — no hardcoded node list):
|
||||
- The **master-drain Job** runs on the first worker (the "runner"); since it
|
||||
drains the control-plane node, it must not run there.
|
||||
- **Every worker-drain Job** runs on **k8s-master** (already upgraded by then),
|
||||
with a `node-role.kubernetes.io/control-plane:NoSchedule` toleration — so a
|
||||
Job never runs on the node it drains (self-preemption invariant).
|
||||
- The worker set + order come from `kubectl get nodes` at runtime
|
||||
(`worker_nodes` / `next_pending_worker` in `scripts/upgrade-step.sh`), so
|
||||
**adding a node needs no change** — the chain upgrades every worker still
|
||||
off-target, then runs postflight. SSH uses node InternalIPs (no DNS needed).
|
||||
|
||||
## Components
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue