01bc16d592
The agent-based v1 ran inside claude-agent-service (replicas=1, no
nodeSelector) and self-evicted when it tried to drain its host (k8s-node4
on 2026-05-11). Cluster ended half-upgraded (master v1.34.7, workers
v1.34.2) until manual recovery.
Rewrite the pipeline as a chain of nodeSelector-pinned Jobs:
preflight (k8s-node1)
→ master (k8s-node1) drains k8s-master
→ worker × 4 (k8s-node1) drains k8s-node{4,3,2}
→ worker (k8s-master + control-plane toleration) drains k8s-node1
→ postflight (no pinning)
Each Job runs scripts/upgrade-step.sh (case-on-$PHASE) and ends by
envsubst-ing job-template.yaml into the next Job. Deterministic names
(k8s-upgrade-<phase>-<target_version>[-<node>]) make `kubectl apply`
idempotent — a failed Job can be re-created without duplicating
downstream.
Also lands `predrain_unstick`: deletes pods on the target node whose PDB
has 0 disruptionsAllowed. Without this, drain loops indefinitely on
single-replica deployments (e.g. every Anubis instance — discovered the
hard way during 2026-05-11 manual recovery of k8s-node3).
Adds K8sUpgradeStalled alert (in_flight + started_timestamp > 90 min).
Deprecates the agent prompt (renamed to *.deprecated.md with a header
pointer to the new code).
Apply order: k8s-version-upgrade first (consumes new SA + ConfigMaps),
then monitoring (loads the new alert). Both applied 2026-05-11.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
88 lines
2.8 KiB
YAML
88 lines
2.8 KiB
YAML
# k8s-upgrade-chain Job template.
|
|
#
|
|
# Rendered by `envsubst` inside upgrade-step.sh (and the detection CronJob)
|
|
# before `kubectl apply`. All ${VAR} placeholders are envsubst-side; this file
|
|
# is NOT processed by Terraform.
|
|
#
|
|
# Required environment for envsubst:
|
|
# JOB_NAME unique-per-(phase, target_version[, target_node])
|
|
# PHASE_NEXT phase the Job runs (preflight|master|worker|postflight)
|
|
# TARGET_NODE_NEXT node the Job operates on (empty for preflight/postflight)
|
|
# TARGET_VERSION X.Y.Z
|
|
# TARGET_VERSION_LABEL X-Y-Z (label-safe)
|
|
# KIND patch | minor
|
|
# IMAGE container image to run upgrade-step.sh
|
|
# SCHEDULING_BLOCK YAML fragment with nodeSelector/tolerations (may be empty)
|
|
#
|
|
# Idempotency: name is deterministic per (phase, target_version[, target_node])
|
|
# so `kubectl apply` reconciles to a single Job per run.
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: ${JOB_NAME}
|
|
namespace: k8s-upgrade
|
|
labels:
|
|
app: k8s-upgrade-chain
|
|
phase: ${PHASE_NEXT}
|
|
target-version: "${TARGET_VERSION_LABEL}"
|
|
spec:
|
|
ttlSecondsAfterFinished: 604800 # 7 days for postmortem review
|
|
backoffLimit: 1
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: k8s-upgrade-chain
|
|
phase: ${PHASE_NEXT}
|
|
spec:
|
|
serviceAccountName: k8s-upgrade-job
|
|
restartPolicy: Never
|
|
${SCHEDULING_BLOCK}
|
|
imagePullSecrets:
|
|
- name: registry-credentials
|
|
containers:
|
|
- name: upgrade-step
|
|
image: ${IMAGE}
|
|
env:
|
|
- name: PHASE
|
|
value: "${PHASE_NEXT}"
|
|
- name: TARGET_NODE
|
|
value: "${TARGET_NODE_NEXT}"
|
|
- name: TARGET_VERSION
|
|
value: "${TARGET_VERSION}"
|
|
- name: KIND
|
|
value: "${KIND}"
|
|
- name: IMAGE
|
|
value: "${IMAGE}"
|
|
- name: HOME
|
|
value: "/tmp"
|
|
command: ["/bin/bash", "/scripts/upgrade-step.sh"]
|
|
volumeMounts:
|
|
- name: creds
|
|
mountPath: /secrets/k8s-upgrade
|
|
readOnly: true
|
|
- name: scripts
|
|
mountPath: /scripts
|
|
readOnly: true
|
|
- name: template
|
|
mountPath: /template
|
|
readOnly: true
|
|
resources:
|
|
requests:
|
|
cpu: "100m"
|
|
memory: "256Mi"
|
|
limits:
|
|
memory: "512Mi"
|
|
volumes:
|
|
- name: creds
|
|
secret:
|
|
secretName: k8s-upgrade-creds
|
|
# 0444 so the non-root container can read; upgrade-step.sh copies
|
|
# the SSH key to /tmp/ssh_key with mode 0400 for openssh.
|
|
defaultMode: 0444
|
|
- name: scripts
|
|
configMap:
|
|
name: k8s-upgrade-scripts
|
|
defaultMode: 0755
|
|
- name: template
|
|
configMap:
|
|
name: k8s-upgrade-job-template
|