a58d777059
Adds a weekly detection CronJob (Sun 12:00 UTC) that probes apt-cache madison
on master for new patches + HEAD pkgs.k8s.io for next-minor availability,
then POSTs to claude-agent-service to dispatch the k8s-version-upgrade agent.
The agent (.claude/agents/k8s-version-upgrade.md) orchestrates:
pre-flight (5 nodes Ready + halt-on-alert + 24h-quiet + plan target match)
-> etcd snapshot save
-> optional master containerd skew fix
-> apt repo URL rewrite (minor bumps only)
-> drain/upgrade/uncordon master via ssh < update_k8s.sh
-> sequential workers k8s-node4 -> 3 -> 2 -> 1 with 10-min soak each
-> post-flight verification
Two new Upgrade Gates alerts catch failure modes:
- K8sVersionSkew (kubelet/apiserver gitVersion mismatch >30m)
- EtcdPreUpgradeSnapshotMissing (in_flight without snapshot_taken >10m)
update_k8s.sh refactored to take --role / --release args; the agent shells
it into each node via SSH pipe. update_node.sh annotated as OS-major path.
Operator-facing docs: docs/runbooks/k8s-version-upgrade.md and a new section
in docs/architecture/automated-upgrades.md.
Secrets: secret/k8s-upgrade/{ssh_key,ssh_key_pub,slack_webhook} (ed25519
keypair distributed to all 5 nodes via authorized_keys; slack_webhook
reuses kured webhook URL on initial deploy).
23 lines
831 B
HCL
23 lines
831 B
HCL
include "root" {
|
|
path = find_in_parent_folders()
|
|
}
|
|
|
|
# ExternalSecret hits ESO which needs to be alive when the manifest applies.
|
|
dependency "external_secrets" {
|
|
config_path = "../external-secrets"
|
|
skip_outputs = true
|
|
}
|
|
|
|
# Upgrade Gates rules (incl. K8sVersionSkew + EtcdPreUpgradeSnapshotMissing)
|
|
# live in the monitoring stack — make the relationship visible so reapplies
|
|
# don't race the alerts being available.
|
|
dependency "monitoring" {
|
|
config_path = "../monitoring"
|
|
skip_outputs = true
|
|
}
|
|
|
|
# Note: stacks/claude-agent-service has no terragrunt.hcl yet (manual apply
|
|
# pattern) — its ServiceAccount + Namespace are referenced by name from this
|
|
# stack's RoleBindings, which is fine because RoleBindings allow forward
|
|
# references. Apply order: claude-agent-service first (or already deployed),
|
|
# then this stack.
|