cloud-init: hands-off k8s worker provisioning + 5 bug fixes
Goal: re-clone the worker template, boot, and have it appear as `kubectl get nodes …Ready` with no manual steps. Adds `scripts/provision-k8s-worker NAME VMID IP` and rebuilds the cloud-init pipeline that was failing five distinct ways on a clean boot. Bugs fixed (all hit during the k8s-node5 + k8s-node6 builds today): 1. `indent(6, containerd_config_update_command)` indented the bodies of `cat >> /etc/containerd/config.toml <<'CONTAINERD_GC'` heredocs, so [plugins.*] TOML sections landed in /etc/containerd/config.toml at col 6 — containerd refused to parse them. Source is now a normal .sh file (`modules/create-template-vm/k8s-node-containerd-setup.sh`) base64-embedded into `write_files`; YAML whitespace never touches the heredoc bodies. 2. The same script tried to `cat >> /etc/containerd/config.toml` `[plugins."io.containerd.gc.v1.scheduler"]` etc., which containerd v2.2.4's `config default` ALREADY emits. Result: `toml: table … already exists`. Patched with sed-in-place overrides instead. 3. Kubelet tuning (sed against /var/lib/kubelet/config.yaml) ran from the containerd setup script — BEFORE `kubeadm join` writes that file. Sed aborted with "No such file or directory", `set -e` killed the script, post-script cloud-init steps kept going (cloud-init doesn't stop on runcmd failure). Split into a dedicated `k8s-node-post-join-tune.sh` invoked AFTER kubeadm join. 4. cloud_init.yaml fallocate'd a 4G swapfile and `swapon`'d it BEFORE kubeadm join. kubelet defaults to failSwapOn=true → exited 1 immediately. Replaced the swap setup with `swapoff -a` (node4 already runs this way and the cluster is fine). 5. Without `hostname:` in the shared user-data snippet, Proxmox's auto-generated meta-data does NOT include local-hostname when `cicustom user=…` is set — so cloud-init falls back to the cloud image's default `ubuntu` and `kubeadm join` registers the wrong node name. `provision-k8s-worker` now writes a per-node `<NAME>-meta.yaml` snippet and passes both via `cicustom user=…,meta=…`. Other improvements rolled in while fixing the above: - `ssh_public_key` read from Vault (`secret/viktor.ssh_public_key`, added today) instead of `var.ssh_public_key`. The last `terragrunt apply` was run with that var empty, leaving the snippet's `ssh_authorized_keys` with a single blank entry; the wizard user was effectively locked out of every fresh node. - `cloud_init.yaml` adds `/etc/systemd/resolved.conf.d/global-dns.conf` with `DNS=8.8.8.8 1.1.1.1, FallbackDNS=10.0.20.201`. Without it, systemd-resolved only consulted Technitium (link-level), which returns NXDOMAIN for `forgejo.viktorbarzin.me` — kubelet pulls from the Forgejo registry then failed DNS until I patched it manually on node5. - k8s apt repo bumped v1.32 → v1.34 (matches cluster). - The containerd setup script now creates hosts.toml for forgejo, quay, registry.k8s.io in addition to docker.io + ghcr.io. node3/4 had these added by hand post-bootstrap; now they're baked in. - `config_path` sed matches both `""` (containerd v1) and `''` (containerd v2.x). Without the v2 match, the certs.d mirror dir was silently ignored. - `proxmox-csi` node map adds k8s-node5 + k8s-node6 entries so CSI topology labels (region/zone, max-volume-attachments=28) apply on next `tg apply`. - `stacks/infra/main.tf` shed the 160-line inline containerd setup heredoc — that whole thing now lives in the module as a .sh file. Known unsolved gaps (deferred): - iscsid restart hangs ~90s on first boot before SIGKILL releases it (systemd-resolved restart kicks iscsid via dependency). Adds wall- clock time but doesn't block the join. - `provision-k8s-worker` doesn't run `tg apply` on `proxmox-csi` afterward, so the CSI topology labels need a manual apply after the node joins. Solving cleanly needs the CSI map to derive from `kubectl get nodes` instead of a static local — separate work. - `var.containerd_config_update_command` is now ignored when is_k8s_template=true (replaced by the bundled .sh file). Variable kept with a deprecation note to avoid breaking other call sites. E2E proof: k8s-node6 (VMID 206) boots hands-off from `provision-k8s-worker k8s-node6 206 10.0.20.106` and appears as `kubectl get nodes …Ready` ~7 min later (most of which is the apt package_upgrade — separate optimization). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
e4c0cbc3d0
commit
8ed427a7e4
7 changed files with 408 additions and 201 deletions
|
|
@ -1,5 +1,8 @@
|
|||
#cloud-config
|
||||
hostname: terraform-vm
|
||||
#cloud-config
|
||||
# Hostname intentionally NOT set here — cloud-init reads it from
|
||||
# Proxmox's auto-generated meta-data (which uses `qm set --name <X>`),
|
||||
# so a single shared snippet works for every node.
|
||||
manage_etc_hosts: true
|
||||
users:
|
||||
- name: wizard
|
||||
sudo: ALL=(ALL) NOPASSWD:ALL
|
||||
|
|
@ -46,7 +49,7 @@ apt:
|
|||
sources:
|
||||
%{if is_k8s_template}
|
||||
kubernetes:
|
||||
source: "deb https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /"
|
||||
source: "deb https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /"
|
||||
keyid: "DE15B14486CD377B9E876E1A234654DA9A296436"
|
||||
filename: kubernetes.list
|
||||
%{endif}
|
||||
|
|
@ -56,23 +59,23 @@ apt:
|
|||
filename: docker.list
|
||||
|
||||
%{if is_k8s_template}
|
||||
# write_files delivers the multi-line containerd/kubelet setup script to a
|
||||
# file BEFORE runcmd executes. This pattern avoids the YAML interpolation bug
|
||||
# where multi-line $${containerd_config_update_command} (from
|
||||
# stacks/infra/main.tf — has mixed-indent inner shell heredocs) inserted into
|
||||
# a single `runcmd: - $${var}` item produces invalid YAML and silently fails
|
||||
# cloud-init parsing (observed 2026-05-26 during node4 rebuild). With write_files,
|
||||
# the multi-line content lives in a YAML literal block where ANY indent >= the
|
||||
# block's content indent is valid — so col-0 heredoc lines like
|
||||
# `[plugins."io.containerd.gc.v1.scheduler"]` survive cleanly.
|
||||
# Setup script is base64-encoded by the module so YAML whitespace
|
||||
# handling never touches the heredoc bodies inside it. Replaces an
|
||||
# earlier `indent(6, …)` approach that put `[plugins.*]` TOML
|
||||
# sections at col 6 inside `cat >> /etc/containerd/config.toml`
|
||||
# heredocs — containerd refused to parse the result and the node5 v1
|
||||
# boot failed there (2026-05-26). Source: modules/create-template-vm/k8s-node-containerd-setup.sh
|
||||
write_files:
|
||||
- path: /usr/local/bin/k8s-node-containerd-setup.sh
|
||||
permissions: '0755'
|
||||
owner: root:root
|
||||
content: |
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
${indent(6, containerd_config_update_command)}
|
||||
encoding: b64
|
||||
content: ${k8s_node_setup_script_b64}
|
||||
- path: /usr/local/bin/k8s-node-post-join-tune.sh
|
||||
permissions: '0755'
|
||||
owner: root:root
|
||||
encoding: b64
|
||||
content: ${k8s_node_post_join_script_b64}
|
||||
%{endif}
|
||||
|
||||
runcmd:
|
||||
|
|
@ -87,6 +90,20 @@ runcmd:
|
|||
- sed -i 's/#Compress=yes/Compress=yes/' /etc/systemd/journald.conf
|
||||
- systemctl restart systemd-journald
|
||||
%{if is_k8s_template}
|
||||
# systemd-resolved global DNS fallback. Without this, only the
|
||||
# link-level DNS from Proxmox's `qm set --nameserver` (Technitium,
|
||||
# 10.0.20.201) is consulted — and Technitium returns NXDOMAIN for
|
||||
# forgejo.viktorbarzin.me, so kubelet image pulls from the Forgejo
|
||||
# registry break. Public DNS upstream + Technitium fallback matches
|
||||
# the pre-existing manual setup on k8s-node1..4.
|
||||
- mkdir -p /etc/systemd/resolved.conf.d
|
||||
- |
|
||||
cat > /etc/systemd/resolved.conf.d/global-dns.conf <<'EOF'
|
||||
[Resolve]
|
||||
DNS=8.8.8.8 1.1.1.1
|
||||
FallbackDNS=10.0.20.201
|
||||
EOF
|
||||
- systemctl restart systemd-resolved
|
||||
# Re-enabled 2026-05-10: unattended-upgrades is back on, but with a tight
|
||||
# Allowed-Origins list, a Package-Blacklist for k8s/containerd/runc/calico,
|
||||
# and Automatic-Reboot disabled (kured + sentinel-gate handles reboots in a
|
||||
|
|
@ -149,17 +166,19 @@ runcmd:
|
|||
- systemctl restart iscsid
|
||||
# Create /sentinel directory for kured reboot gating (sentinel gate DaemonSet)
|
||||
- mkdir -p /sentinel
|
||||
# Create 4Gi swap file for worker node memory pressure relief (NOT for master — etcd is latency-critical)
|
||||
- fallocate -l 4G /swapfile
|
||||
- chmod 600 /swapfile
|
||||
- mkswap /swapfile
|
||||
- swapon /swapfile
|
||||
- echo '/swapfile none swap sw 0 0' >> /etc/fstab
|
||||
- sysctl -w vm.swappiness=10
|
||||
- echo 'vm.swappiness=10' >> /etc/sysctl.d/99-swap.conf
|
||||
# Disable swap — kubelet defaults to failSwapOn=true and won't start otherwise.
|
||||
# (Previously this snippet created a 4G swapfile for "memory pressure relief"
|
||||
# but never set failSwapOn=false / memorySwap.swapBehavior together, so the
|
||||
# join consistently bricked kubelet — observed on node6 boot v3 2026-05-26.)
|
||||
- swapoff -a
|
||||
- sed -i '/ swap / s/^/#/' /etc/fstab
|
||||
- ${k8s_join_command}
|
||||
- systemctl enable kubelet
|
||||
- systemctl start kubelet
|
||||
# Kubelet tuning runs AFTER kubeadm join — that's when
|
||||
# /var/lib/kubelet/config.yaml gets written. Restarts kubelet at the
|
||||
# end to pick up the patched config.
|
||||
- bash /usr/local/bin/k8s-node-post-join-tune.sh
|
||||
%{ endif }
|
||||
%{ for provision_cmd in provision_cmds ~}
|
||||
- ${provision_cmd}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue