viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

k8s-version-upgrade: retry failed phases + surface wedged chain (fix 5-day silent stall)

Browse source 
The 1.34.9 patch auto-upgrade sat stuck for 5 days without anyone knowing.
On 2026-06-12 a transient critical alert (the ttyd web-terminal probe on the
devvm) was firing when the daily detection ran; the preflight's "halt on any
critical alert" gate aborted it, so the preflight Job Failed (backoffLimit=1).
Two design gaps then turned that blip into a multi-day wedge:

  * the detection guard and spawn_next only checked whether the phase Job
    EXISTED, not whether it succeeded — and the Failed Job lingers 7 days via
    ttlSecondsAfterFinished, so every daily run skipped re-spawning it;
  * the abort happens before the in-flight metric is pushed, so neither
    K8sUpgradeStalled nor upgrade_state.sh could see it — the pipeline reported
    "never ran" while actually being stuck.

Fixes:
  D1 retry-on-failure: detection CronJob (main.tf) and spawn_next
     (upgrade-step.sh) now delete + re-spawn a terminally-Failed phase Job
     instead of skipping it, so a transient gate self-corrects next cycle
     rather than wedging the pipeline for a week.
  D2 WebterminalTtydUnreachable critical -> warning: a devvm developer
     web-terminal is not cluster infrastructure and must not block upgrades.
  D3 observability: new K8sUpgradeChainJobFailed alert
     (kube_job_status_failed in k8s-upgrade ns) and upgrade_state.sh now flags
     a Failed chain Job as "chain failed" — closing the pre-in-flight blind
     spot so a wedge is visible immediately.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-06-17 13:07:36 +00:00
parent 7e7e41cbef
commit dfa1a12a86
4 changed files with 75 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

18
stacks/k8s-version-upgrade/scripts/upgrade-step.sh
View file

@ -222,9 +222,23 @@ spawn_next() {
local job_name="k8s-upgrade-${NEXT_PHASE}-${TARGET_VERSION//./-}"
[ -n "${NEXT_TARGET_NODE:-}" ] && job_name="${job_name}-${NEXT_TARGET_NODE}"
# Retry-on-failure idempotency: skip an existing next-Job ONLY if it is
# Active or Complete. A *Failed* Job (a phase that aborted on a transient
# gate) is deleted and re-created — otherwise its deterministic name plus
# ttlSecondsAfterFinished (7d) would block the whole chain from re-running
# that phase until the dead Job aged out. (Stuck-pipeline fix 2026-06-17:
# a transient critical alert wedged the 1.34.9 preflight for 5 days.)
if $KUBECTL -n "$NS" get job "$job_name" >/dev/null 2>&1; then
echo "Next Job $job_name already exists; idempotent skip."
return 0
local job_failed
job_failed=$($KUBECTL -n "$NS" get job "$job_name" \
-o jsonpath='{.status.conditions[?(@.type=="Failed")].status}' 2>/dev/null || true)
if [ "$job_failed" = "True" ]; then
echo "Next Job $job_name exists but FAILED — deleting and re-spawning."
$KUBECTL -n "$NS" delete job "$job_name" --wait=true >/dev/null 2>&1 || true
else
echo "Next Job $job_name already exists (active/complete); idempotent skip."
return 0
fi
fi
local scheduling_block=""