8ed427a7e4
Goal: re-clone the worker template, boot, and have it appear as `kubectl get nodes …Ready` with no manual steps. Adds `scripts/provision-k8s-worker NAME VMID IP` and rebuilds the cloud-init pipeline that was failing five distinct ways on a clean boot. Bugs fixed (all hit during the k8s-node5 + k8s-node6 builds today): 1. `indent(6, containerd_config_update_command)` indented the bodies of `cat >> /etc/containerd/config.toml <<'CONTAINERD_GC'` heredocs, so [plugins.*] TOML sections landed in /etc/containerd/config.toml at col 6 — containerd refused to parse them. Source is now a normal .sh file (`modules/create-template-vm/k8s-node-containerd-setup.sh`) base64-embedded into `write_files`; YAML whitespace never touches the heredoc bodies. 2. The same script tried to `cat >> /etc/containerd/config.toml` `[plugins."io.containerd.gc.v1.scheduler"]` etc., which containerd v2.2.4's `config default` ALREADY emits. Result: `toml: table … already exists`. Patched with sed-in-place overrides instead. 3. Kubelet tuning (sed against /var/lib/kubelet/config.yaml) ran from the containerd setup script — BEFORE `kubeadm join` writes that file. Sed aborted with "No such file or directory", `set -e` killed the script, post-script cloud-init steps kept going (cloud-init doesn't stop on runcmd failure). Split into a dedicated `k8s-node-post-join-tune.sh` invoked AFTER kubeadm join. 4. cloud_init.yaml fallocate'd a 4G swapfile and `swapon`'d it BEFORE kubeadm join. kubelet defaults to failSwapOn=true → exited 1 immediately. Replaced the swap setup with `swapoff -a` (node4 already runs this way and the cluster is fine). 5. Without `hostname:` in the shared user-data snippet, Proxmox's auto-generated meta-data does NOT include local-hostname when `cicustom user=…` is set — so cloud-init falls back to the cloud image's default `ubuntu` and `kubeadm join` registers the wrong node name. `provision-k8s-worker` now writes a per-node `<NAME>-meta.yaml` snippet and passes both via `cicustom user=…,meta=…`. Other improvements rolled in while fixing the above: - `ssh_public_key` read from Vault (`secret/viktor.ssh_public_key`, added today) instead of `var.ssh_public_key`. The last `terragrunt apply` was run with that var empty, leaving the snippet's `ssh_authorized_keys` with a single blank entry; the wizard user was effectively locked out of every fresh node. - `cloud_init.yaml` adds `/etc/systemd/resolved.conf.d/global-dns.conf` with `DNS=8.8.8.8 1.1.1.1, FallbackDNS=10.0.20.201`. Without it, systemd-resolved only consulted Technitium (link-level), which returns NXDOMAIN for `forgejo.viktorbarzin.me` — kubelet pulls from the Forgejo registry then failed DNS until I patched it manually on node5. - k8s apt repo bumped v1.32 → v1.34 (matches cluster). - The containerd setup script now creates hosts.toml for forgejo, quay, registry.k8s.io in addition to docker.io + ghcr.io. node3/4 had these added by hand post-bootstrap; now they're baked in. - `config_path` sed matches both `""` (containerd v1) and `''` (containerd v2.x). Without the v2 match, the certs.d mirror dir was silently ignored. - `proxmox-csi` node map adds k8s-node5 + k8s-node6 entries so CSI topology labels (region/zone, max-volume-attachments=28) apply on next `tg apply`. - `stacks/infra/main.tf` shed the 160-line inline containerd setup heredoc — that whole thing now lives in the module as a .sh file. Known unsolved gaps (deferred): - iscsid restart hangs ~90s on first boot before SIGKILL releases it (systemd-resolved restart kicks iscsid via dependency). Adds wall- clock time but doesn't block the join. - `provision-k8s-worker` doesn't run `tg apply` on `proxmox-csi` afterward, so the CSI topology labels need a manual apply after the node joins. Solving cleanly needs the CSI map to derive from `kubectl get nodes` instead of a static local — separate work. - `var.containerd_config_update_command` is now ignored when is_k8s_template=true (replaced by the bundled .sh file). Variable kept with a deprecation note to avoid breaking other call sites. E2E proof: k8s-node6 (VMID 206) boots hands-off from `provision-k8s-worker k8s-node6 206 10.0.20.106` and appears as `kubectl get nodes …Ready` ~7 min later (most of which is the apt package_upgrade — separate optimization). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
109 lines
3.6 KiB
Bash
Executable file
109 lines
3.6 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# provision-k8s-worker NAME VMID IP[/CIDR]
|
|
#
|
|
# Clone PVE template 2000 (ubuntu-2404-cloudinit-k8s-template) into a new
|
|
# VM, configure resources to match k8s-node3/4 (32G RAM, 8 vCPU, host CPU,
|
|
# 256G disk, VLAN 20 on vmbr1), attach the shared cicustom snippet
|
|
# (/var/lib/vz/snippets/k8s_cloud_init.yaml), and start it. Cloud-init
|
|
# inside the VM installs containerd + kubelet, applies the bundled
|
|
# setup script, and runs the kubeadm join. No manual steps after this.
|
|
#
|
|
# Hostname is derived from `qm set --name $NAME` and read by cloud-init
|
|
# from Proxmox metadata — DO NOT hard-code in the snippet.
|
|
#
|
|
# Idempotent: aborts if VMID already exists or IP is already in use.
|
|
#
|
|
# Usage:
|
|
# ssh root@192.168.1.127 bash -s -- k8s-node6 206 10.0.20.106 < provision-k8s-worker
|
|
# or, if the script lives on the PVE host:
|
|
# provision-k8s-worker k8s-node6 206 10.0.20.106
|
|
#
|
|
# Run on the PVE host (needs qm + /var/lib/vz/snippets access).
|
|
set -euo pipefail
|
|
|
|
if [ $# -ne 3 ]; then
|
|
echo "usage: $0 NAME VMID IP" >&2
|
|
echo " e.g. $0 k8s-node6 206 10.0.20.106" >&2
|
|
exit 2
|
|
fi
|
|
|
|
NAME=$1
|
|
VMID=$2
|
|
IP=$3
|
|
CIDR_IP="${IP}/22"
|
|
GW="10.0.20.1"
|
|
DNS="10.0.20.201"
|
|
SEARCH="viktorbarzin.lan"
|
|
TEMPLATE_ID=2000
|
|
STORAGE="local-lvm"
|
|
USER_SNIPPET="local:snippets/k8s_cloud_init.yaml"
|
|
# Per-node meta-data snippet — written below — supplies local-hostname.
|
|
# Proxmox's auto-generated metadata DOESN'T include hostname when
|
|
# cicustom user=… is set, so the shared user-data snippet alone leaves
|
|
# nodes joining as "ubuntu" (image default). Per-node meta-data is the
|
|
# clean fix.
|
|
META_SNIPPET_FILE="/var/lib/vz/snippets/${NAME}-meta.yaml"
|
|
META_SNIPPET="local:snippets/${NAME}-meta.yaml"
|
|
BRIDGE="vmbr1"
|
|
VLAN=20
|
|
|
|
# Sanity: VMID must be free
|
|
if qm status "$VMID" >/dev/null 2>&1; then
|
|
echo "ERROR: VM $VMID already exists. Refusing to clobber." >&2
|
|
qm status "$VMID" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Sanity: IP must not be pingable
|
|
if ping -c 1 -W 1 "$IP" >/dev/null 2>&1; then
|
|
echo "ERROR: $IP is already responding to ping. Refusing to assign." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Sanity: snippet must exist
|
|
if [ ! -f "/var/lib/vz/snippets/k8s_cloud_init.yaml" ]; then
|
|
echo "ERROR: /var/lib/vz/snippets/k8s_cloud_init.yaml missing." >&2
|
|
echo " Run `tg apply` in infra/stacks/infra/ to regenerate it." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Sanity: template must be a template
|
|
if ! qm config "$TEMPLATE_ID" | grep -q '^template: 1'; then
|
|
echo "ERROR: VMID $TEMPLATE_ID is not a template." >&2
|
|
exit 1
|
|
fi
|
|
|
|
echo "[1/6] write per-node meta-data snippet ($META_SNIPPET_FILE)"
|
|
cat > "$META_SNIPPET_FILE" <<META
|
|
local-hostname: $NAME
|
|
instance-id: $NAME-$(date +%s)
|
|
META
|
|
|
|
echo "[2/6] qm clone $TEMPLATE_ID -> $VMID ($NAME)"
|
|
qm clone "$TEMPLATE_ID" "$VMID" --name "$NAME" --full true --storage "$STORAGE"
|
|
|
|
echo "[3/6] qm set $VMID — VM resources + network + cicustom"
|
|
qm set "$VMID" \
|
|
--agent 1 \
|
|
--balloon 32768 \
|
|
--cores 8 \
|
|
--cpu host \
|
|
--memory 32768 \
|
|
--net0 "virtio,bridge=$BRIDGE,tag=$VLAN" \
|
|
--ipconfig0 "ip=$CIDR_IP,gw=$GW" \
|
|
--nameserver "$DNS" \
|
|
--searchdomain "$SEARCH" \
|
|
--onboot 1 \
|
|
--startup 'order=5,up=45,down=420' \
|
|
--cicustom "user=$USER_SNIPPET,meta=$META_SNIPPET"
|
|
|
|
echo "[4/6] qm resize $VMID scsi0 256G"
|
|
qm resize "$VMID" scsi0 256G
|
|
|
|
echo "[5/6] qm start $VMID"
|
|
qm start "$VMID"
|
|
|
|
echo "[6/6] Done. Cloud-init runs now; node should appear in 'kubectl get nodes' within ~6-10 min."
|
|
echo " Tail cloud-init: socat -u UNIX-CONNECT:/var/run/qemu-server/$VMID.serial0 STDOUT | strings"
|
|
echo " Final config:"
|
|
qm config "$VMID" | grep -E '^(name|cores|memory|net0|ipconfig0|cicustom|scsi0|onboot):'
|