viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

cloud-init: hands-off k8s worker provisioning + 5 bug fixes

Browse source 
Goal: re-clone the worker template, boot, and have it appear as `kubectl
get nodes …Ready` with no manual steps. Adds `scripts/provision-k8s-worker
NAME VMID IP` and rebuilds the cloud-init pipeline that was failing five
distinct ways on a clean boot.

Bugs fixed (all hit during the k8s-node5 + k8s-node6 builds today):

1. `indent(6, containerd_config_update_command)` indented the bodies of
   `cat >> /etc/containerd/config.toml <<'CONTAINERD_GC'` heredocs, so
   [plugins.*] TOML sections landed in /etc/containerd/config.toml at
   col 6 — containerd refused to parse them. Source is now a normal
   .sh file (`modules/create-template-vm/k8s-node-containerd-setup.sh`)
   base64-embedded into `write_files`; YAML whitespace never touches
   the heredoc bodies.

2. The same script tried to `cat >> /etc/containerd/config.toml`
   `[plugins."io.containerd.gc.v1.scheduler"]` etc., which containerd
   v2.2.4's `config default` ALREADY emits. Result: `toml: table …
   already exists`. Patched with sed-in-place overrides instead.

3. Kubelet tuning (sed against /var/lib/kubelet/config.yaml) ran from
   the containerd setup script — BEFORE `kubeadm join` writes that
   file. Sed aborted with "No such file or directory", `set -e` killed
   the script, post-script cloud-init steps kept going (cloud-init
   doesn't stop on runcmd failure). Split into a dedicated
   `k8s-node-post-join-tune.sh` invoked AFTER kubeadm join.

4. cloud_init.yaml fallocate'd a 4G swapfile and `swapon`'d it BEFORE
   kubeadm join. kubelet defaults to failSwapOn=true → exited 1
   immediately. Replaced the swap setup with `swapoff -a` (node4
   already runs this way and the cluster is fine).

5. Without `hostname:` in the shared user-data snippet, Proxmox's
   auto-generated meta-data does NOT include local-hostname when
   `cicustom user=…` is set — so cloud-init falls back to the cloud
   image's default `ubuntu` and `kubeadm join` registers the wrong
   node name. `provision-k8s-worker` now writes a per-node
   `<NAME>-meta.yaml` snippet and passes both via
   `cicustom user=…,meta=…`.

Other improvements rolled in while fixing the above:

- `ssh_public_key` read from Vault (`secret/viktor.ssh_public_key`,
  added today) instead of `var.ssh_public_key`. The last
  `terragrunt apply` was run with that var empty, leaving the snippet's
  `ssh_authorized_keys` with a single blank entry; the wizard user
  was effectively locked out of every fresh node.
- `cloud_init.yaml` adds `/etc/systemd/resolved.conf.d/global-dns.conf`
  with `DNS=8.8.8.8 1.1.1.1, FallbackDNS=10.0.20.201`. Without it,
  systemd-resolved only consulted Technitium (link-level), which
  returns NXDOMAIN for `forgejo.viktorbarzin.me` — kubelet pulls from
  the Forgejo registry then failed DNS until I patched it manually
  on node5.
- k8s apt repo bumped v1.32 → v1.34 (matches cluster).
- The containerd setup script now creates hosts.toml for forgejo,
  quay, registry.k8s.io in addition to docker.io + ghcr.io. node3/4
  had these added by hand post-bootstrap; now they're baked in.
- `config_path` sed matches both `""` (containerd v1) and `''`
  (containerd v2.x). Without the v2 match, the certs.d mirror dir was
  silently ignored.
- `proxmox-csi` node map adds k8s-node5 + k8s-node6 entries so CSI
  topology labels (region/zone, max-volume-attachments=28) apply on
  next `tg apply`.
- `stacks/infra/main.tf` shed the 160-line inline containerd setup
  heredoc — that whole thing now lives in the module as a .sh file.

Known unsolved gaps (deferred):

- iscsid restart hangs ~90s on first boot before SIGKILL releases it
  (systemd-resolved restart kicks iscsid via dependency). Adds wall-
  clock time but doesn't block the join.
- `provision-k8s-worker` doesn't run `tg apply` on `proxmox-csi`
  afterward, so the CSI topology labels need a manual apply after
  the node joins. Solving cleanly needs the CSI map to derive from
  `kubectl get nodes` instead of a static local — separate work.
- `var.containerd_config_update_command` is now ignored when
  is_k8s_template=true (replaced by the bundled .sh file). Variable
  kept with a deprecation note to avoid breaking other call sites.

E2E proof: k8s-node6 (VMID 206) boots hands-off from
`provision-k8s-worker k8s-node6 206 10.0.20.106` and appears as
`kubectl get nodes …Ready` ~7 min later (most of which is the apt
package_upgrade — separate optimization).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-05-26 11:52:00 +00:00
parent e4c0cbc3d0
commit 8ed427a7e4
7 changed files with 408 additions and 201 deletions
Download patch file Download diff file Expand all files Collapse all files

67
modules/create-template-vm/cloud_init.yaml
View file

@ -1,5 +1,8 @@
#cloud-config
hostname: terraform-vm
#cloud-config
# Hostname intentionally NOT set here — cloud-init reads it from
# Proxmox's auto-generated meta-data (which uses `qm set --name <X>`),
# so a single shared snippet works for every node.
manage_etc_hosts: true
users:
- name: wizard
sudo: ALL=(ALL) NOPASSWD:ALL
@ -46,7 +49,7 @@ apt:
sources:
%{if is_k8s_template}
kubernetes:
source: "deb https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /"
source: "deb https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /"
keyid: "DE15B14486CD377B9E876E1A234654DA9A296436"
filename: kubernetes.list
%{endif}
@ -56,23 +59,23 @@ apt:
filename: docker.list
%{if is_k8s_template}
# write_files delivers the multi-line containerd/kubelet setup script to a
# file BEFORE runcmd executes. This pattern avoids the YAML interpolation bug
# where multi-line $${containerd_config_update_command} (from
# stacks/infra/main.tf — has mixed-indent inner shell heredocs) inserted into
# a single `runcmd: - $${var}` item produces invalid YAML and silently fails
# cloud-init parsing (observed 2026-05-26 during node4 rebuild). With write_files,
# the multi-line content lives in a YAML literal block where ANY indent >= the
# block's content indent is valid — so col-0 heredoc lines like
# `[plugins."io.containerd.gc.v1.scheduler"]` survive cleanly.
# Setup script is base64-encoded by the module so YAML whitespace
# handling never touches the heredoc bodies inside it. Replaces an
# earlier `indent(6, …)` approach that put `[plugins.*]` TOML
# sections at col 6 inside `cat >> /etc/containerd/config.toml`
# heredocs — containerd refused to parse the result and the node5 v1
# boot failed there (2026-05-26). Source: modules/create-template-vm/k8s-node-containerd-setup.sh
write_files:
- path: /usr/local/bin/k8s-node-containerd-setup.sh
permissions: '0755'
owner: root:root
content: |
#!/usr/bin/env bash
set -euo pipefail
${indent(6, containerd_config_update_command)}
encoding: b64
content: ${k8s_node_setup_script_b64}
- path: /usr/local/bin/k8s-node-post-join-tune.sh
permissions: '0755'
owner: root:root
encoding: b64
content: ${k8s_node_post_join_script_b64}
%{endif}
runcmd:
@ -87,6 +90,20 @@ runcmd:
- sed -i 's/#Compress=yes/Compress=yes/' /etc/systemd/journald.conf
- systemctl restart systemd-journald
%{if is_k8s_template}
# systemd-resolved global DNS fallback. Without this, only the
# link-level DNS from Proxmox's `qm set --nameserver` (Technitium,
# 10.0.20.201) is consulted — and Technitium returns NXDOMAIN for
# forgejo.viktorbarzin.me, so kubelet image pulls from the Forgejo
# registry break. Public DNS upstream + Technitium fallback matches
# the pre-existing manual setup on k8s-node1..4.
- mkdir -p /etc/systemd/resolved.conf.d
- |
cat > /etc/systemd/resolved.conf.d/global-dns.conf <<'EOF'
[Resolve]
DNS=8.8.8.8 1.1.1.1
FallbackDNS=10.0.20.201
EOF
- systemctl restart systemd-resolved
# Re-enabled 2026-05-10: unattended-upgrades is back on, but with a tight
# Allowed-Origins list, a Package-Blacklist for k8s/containerd/runc/calico,
# and Automatic-Reboot disabled (kured + sentinel-gate handles reboots in a
@ -149,17 +166,19 @@ runcmd:
- systemctl restart iscsid
# Create /sentinel directory for kured reboot gating (sentinel gate DaemonSet)
- mkdir -p /sentinel
# Create 4Gi swap file for worker node memory pressure relief (NOT for master — etcd is latency-critical)
- fallocate -l 4G /swapfile
- chmod 600 /swapfile
- mkswap /swapfile
- swapon /swapfile
- echo '/swapfile none swap sw 0 0' >> /etc/fstab
- sysctl -w vm.swappiness=10
- echo 'vm.swappiness=10' >> /etc/sysctl.d/99-swap.conf
# Disable swap — kubelet defaults to failSwapOn=true and won't start otherwise.
# (Previously this snippet created a 4G swapfile for "memory pressure relief"
# but never set failSwapOn=false / memorySwap.swapBehavior together, so the
# join consistently bricked kubelet — observed on node6 boot v3 2026-05-26.)
- swapoff -a
- sed -i '/ swap / s/^/#/' /etc/fstab
- ${k8s_join_command}
- systemctl enable kubelet
- systemctl start kubelet
# Kubelet tuning runs AFTER kubeadm join — that's when
# /var/lib/kubelet/config.yaml gets written. Restarts kubelet at the
# end to pick up the patched config.
- bash /usr/local/bin/k8s-node-post-join-tune.sh
%{ endif }
%{ for provision_cmd in provision_cmds ~}
- ${provision_cmd}

146
modules/create-template-vm/k8s-node-containerd-setup.sh Executable file
View file

@ -0,0 +1,146 @@
#!/usr/bin/env bash
#
# K8s node containerd + kubelet bootstrap. Runs once via cloud-init runcmd.
# Embedded into the cloud-init snippet base64-encoded by main.tf so YAML
# whitespace handling never touches the heredoc bodies — TOML / Python
# blocks below land in /etc/containerd/config.toml etc. with their leading
# whitespace intact.
#
# Layout:
# 1. /etc/containerd/config.toml — config_path + mirror dirs + GC tuning
# 2. /etc/containerd/certs.d/*/hosts.toml — per-registry mirror configs
# 3. /var/lib/kubelet/config.yaml — eviction + shutdown grace + log rotation
# 4. /etc/systemd/logind.conf.d + kubelet.service.d — graceful shutdown
# 5. (master-only) /etc/kubernetes/manifests — apiserver + controller flags
set -euo pipefail
# 1. config_path — match BOTH quote styles. containerd v1 writes `""`,
# containerd v2.x writes `''`. Without the v2 match, hosts.toml mirror
# config is silently ignored — observed 2026-05-26 on k8s-node4
# (containerd v2.2.4) and reproduced on k8s-node5 v1 boot.
sed -i "s|config_path = \"\"|config_path = \"/etc/containerd/certs.d\"|g" /etc/containerd/config.toml
sed -i "s|config_path = ''|config_path = \"/etc/containerd/certs.d\"|g" /etc/containerd/config.toml
# 2. Per-registry hosts.toml — pull-through caches on docker-registry VM
# (10.0.20.10) for high-traffic registries, Traefik LB (10.0.20.200) for
# forgejo. Low-traffic registries (registry.k8s.io, reg.kyverno.io) skip
# the cache and pull direct because past pull-through cache attempts
# truncated downloads and broke VPA certgen + Kyverno image pulls.
mkdir -p /etc/containerd/certs.d/docker.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml <<'DOCKERIO'
server = "https://registry-1.docker.io"
[host."http://10.0.20.10:5000"]
capabilities = ["pull", "resolve"]
[host."https://registry-1.docker.io"]
capabilities = ["pull", "resolve"]
DOCKERIO
mkdir -p /etc/containerd/certs.d/ghcr.io
cat > /etc/containerd/certs.d/ghcr.io/hosts.toml <<'GHCR'
server = "https://ghcr.io"
[host."http://10.0.20.10:5010"]
capabilities = ["pull", "resolve"]
[host."https://ghcr.io"]
capabilities = ["pull", "resolve"]
GHCR
# Forgejo OCI registry: prefer in-cluster Traefik LB (10.0.20.200) to
# avoid hairpin NAT. Traefik serves the *.viktorbarzin.me wildcard so
# SNI verification succeeds. If the mirror is unreachable, fall back to
# public DNS resolution (needs the global DNS fallback set up below).
mkdir -p /etc/containerd/certs.d/forgejo.viktorbarzin.me
cat > /etc/containerd/certs.d/forgejo.viktorbarzin.me/hosts.toml <<'FORGEJO'
server = "https://forgejo.viktorbarzin.me"
[host."https://10.0.20.200"]
capabilities = ["pull", "resolve"]
FORGEJO
# quay.io + registry.k8s.io: include mirror configs that match node4's
# layout (no real pull-through cache today, server line is the direct
# upstream). Keeping these present makes the per-node config uniform and
# lets us flip a cache on later by editing only the [host."..."] block.
mkdir -p /etc/containerd/certs.d/quay.io
cat > /etc/containerd/certs.d/quay.io/hosts.toml <<'QUAY'
server = "https://quay.io"
[host."http://10.0.20.10:5020"]
capabilities = ["pull", "resolve"]
QUAY
mkdir -p /etc/containerd/certs.d/registry.k8s.io
cat > /etc/containerd/certs.d/registry.k8s.io/hosts.toml <<'K8SREG'
server = "https://registry.k8s.io"
[host."http://10.0.20.10:5030"]
capabilities = ["pull", "resolve"]
K8SREG
# 3. containerd tuning: parallel pulls + selective GC overrides.
# containerd v2's `config default` ALREADY emits `[plugins.'io.containerd.gc.v1.scheduler']`,
# `[plugins.'io.containerd.runtime.v2.task']`, and `[plugins.'io.containerd.metadata.v1.bolt']`
# sections — declaring them again fails with `toml: table … already exists`
# (observed on node6 boot 2026-05-26). Patch values in place instead.
sed -i 's/.*max_concurrent_downloads = 3/max_concurrent_downloads = 20/g' /etc/containerd/config.toml
# pause_threshold: 0.5 → 0.02 (run GC more aggressively when images dirty %)
sed -i "s/^[[:space:]]*pause_threshold = .*/ pause_threshold = 0.02/" /etc/containerd/config.toml
# schedule_delay: 0s/1ms → 30 min (longer cool-down between GC runs)
sed -i "s/^[[:space:]]*schedule_delay = .*/ schedule_delay = '1800s'/" /etc/containerd/config.toml
# exit_timeout: 0s → 5m (more aggressive container cleanup)
sed -i "s/^[[:space:]]*exit_timeout = .*/ exit_timeout = '5m'/" /etc/containerd/config.toml
# 4. (kubelet tuning intentionally NOT here — /var/lib/kubelet/config.yaml
# only exists AFTER kubeadm join. That work runs in
# k8s-node-post-join-tune.sh, invoked as a separate cloud-init runcmd
# step after the join completes.)
# 5. logind + kubelet systemd unit — total kubelet shutdown 310s, so
# logind InhibitDelay > that and kubelet TimeoutStopSec > that.
mkdir -p /etc/systemd/logind.conf.d
cat > /etc/systemd/logind.conf.d/kubelet-shutdown.conf <<'LOGIND_CONF'
[Login]
InhibitDelayMaxSec=480
LOGIND_CONF
systemctl restart systemd-logind
mkdir -p /etc/systemd/system/kubelet.service.d
cat > /etc/systemd/system/kubelet.service.d/20-shutdown.conf <<'KUBELET_SHUTDOWN'
[Service]
TimeoutStopSec=420s
KUBELET_SHUTDOWN
systemctl daemon-reload
# 6. (master-only) faster pod eviction + attach-detach reconcile.
if [ -f /etc/kubernetes/manifests/kube-controller-manager.yaml ]; then
python3 - <<'CM_PATCH'
import yaml
with open('/etc/kubernetes/manifests/kube-controller-manager.yaml') as f:
m = yaml.safe_load(f)
args = m['spec']['containers'][0]['command']
for flag in ['--attach-detach-reconcile-sync-period=15s']:
key = flag.split('=')[0]
args = [a for a in args if not a.startswith(key)]
args.append(flag)
m['spec']['containers'][0]['command'] = args
with open('/etc/kubernetes/manifests/kube-controller-manager.yaml', 'w') as f:
yaml.dump(m, f, default_flow_style=False)
CM_PATCH
python3 - <<'AS_PATCH'
import yaml
with open('/etc/kubernetes/manifests/kube-apiserver.yaml') as f:
m = yaml.safe_load(f)
args = m['spec']['containers'][0]['command']
for flag in ['--default-unreachable-toleration-seconds=60', '--default-not-ready-toleration-seconds=60']:
key = flag.split('=')[0]
args = [a for a in args if not a.startswith(key)]
args.append(flag)
m['spec']['containers'][0]['command'] = args
with open('/etc/kubernetes/manifests/kube-apiserver.yaml', 'w') as f:
yaml.dump(m, f, default_flow_style=False)
AS_PATCH
fi

78
modules/create-template-vm/k8s-node-post-join-tune.sh Executable file
View file

@ -0,0 +1,78 @@
#!/usr/bin/env bash
#
# Runs AFTER `kubeadm join` has written /var/lib/kubelet/config.yaml.
# Patches kubelet config in place (parallel image pulls, eviction
# thresholds, priority-based shutdown grace, container log rotation)
# and (on master) tightens controller-manager / apiserver flags.
#
# Embedded into the cloud-init snippet base64-encoded by main.tf so
# YAML whitespace doesn't touch the heredoc bodies inside.
set -euo pipefail
if [ ! -f /var/lib/kubelet/config.yaml ]; then
echo "post-join-tune: /var/lib/kubelet/config.yaml not found — was kubeadm join run?" >&2
exit 1
fi
# Parallel image pulls.
sed -i '/serializeImagePulls:/d' /var/lib/kubelet/config.yaml
sed -i '/maxParallelImagePulls:/d' /var/lib/kubelet/config.yaml
printf 'serializeImagePulls: false\nmaxParallelImagePulls: 50\n' >> /var/lib/kubelet/config.yaml
# Memory / disk eviction. Aggressive disk thresholds (15%/20%)
# prevent the 2026-03-13 containerd image-store corruption that took
# down k8s-node2.
sed -i '/systemReserved:/d; /kubeReserved:/d; /evictionHard:/,/^[^ ]/{ /evictionHard:/d; /^ /d }; /evictionSoft:/,/^[^ ]/{ /evictionSoft:/d; /^ /d }; /evictionSoftGracePeriod:/,/^[^ ]/{ /evictionSoftGracePeriod:/d; /^ /d }' /var/lib/kubelet/config.yaml
cat >> /var/lib/kubelet/config.yaml <<'KUBELET_PATCH'
systemReserved:
memory: "512Mi"
cpu: "200m"
kubeReserved:
memory: "512Mi"
cpu: "200m"
evictionHard:
memory.available: "500Mi"
nodefs.available: "15%"
imagefs.available: "20%"
evictionSoft:
memory.available: "1Gi"
nodefs.available: "20%"
imagefs.available: "25%"
evictionSoftGracePeriod:
memory.available: "30s"
nodefs.available: "60s"
imagefs.available: "30s"
memorySwap:
swapBehavior: "LimitedSwap"
KUBELET_PATCH
# Container log rotation + priority-based shutdown grace.
sed -i '/^shutdownGracePeriod:/d; /^shutdownGracePeriodCriticalPods:/d' /var/lib/kubelet/config.yaml
python3 - <<'KUBELET_FINAL'
import yaml
with open('/var/lib/kubelet/config.yaml') as f:
cfg = yaml.safe_load(f)
cfg.pop('shutdownGracePeriod', None)
cfg.pop('shutdownGracePeriodCriticalPods', None)
cfg.pop('shutdownGracePeriodByPodPriority', None)
cfg['containerLogMaxSize'] = '10Mi'
cfg['containerLogMaxFiles'] = 3
cfg['shutdownGracePeriodByPodPriority'] = [
{'priority': 0, 'shutdownGracePeriodSeconds': 20},
{'priority': 200000, 'shutdownGracePeriodSeconds': 20},
{'priority': 400000, 'shutdownGracePeriodSeconds': 30},
{'priority': 600000, 'shutdownGracePeriodSeconds': 30},
{'priority': 800000, 'shutdownGracePeriodSeconds': 90},
{'priority': 1000000, 'shutdownGracePeriodSeconds': 30},
{'priority': 1200000, 'shutdownGracePeriodSeconds': 30},
{'priority': 2000000000, 'shutdownGracePeriodSeconds': 30},
{'priority': 2000001000, 'shutdownGracePeriodSeconds': 30},
]
with open('/var/lib/kubelet/config.yaml', 'w') as f:
yaml.dump(cfg, f, default_flow_style=False)
KUBELET_FINAL
# Reload kubelet to pick up new config (it's already started by the
# preceding cloud-init runcmd line — restart, not start).
systemctl restart kubelet

29
modules/create-template-vm/main.tf
View file

@ -16,7 +16,7 @@ variable "k8s_join_command" {
variable "containerd_config_update_command" {
type = string
default = ""
description = "Command to execute to update containerd config.toml; e.g add mirror"
description = "DEPRECATED: was inlined into write_files via indent(); the heredoc-TOML interaction broke containerd config parsing on node5 v1 boot 2026-05-26. The k8s setup script is now bundled inside the module at k8s-node-containerd-setup.sh — pass nothing here. Kept to avoid breaking stacks that still reference it; ignored when is_k8s_template=true."
}
variable "is_k8s_template" { type = bool }
variable "ssh_private_key" {
@ -79,23 +79,26 @@ resource "null_resource" "upload_cloud_init" {
provisioner "file" {
destination = "/var/lib/vz/snippets/${var.snippet_name}"
content = templatefile("${path.module}/cloud_init.yaml", {
is_k8s_template = var.is_k8s_template,
authorized_ssh_key = var.ssh_public_key,
passwd = var.user_passwd,
provision_cmds = var.provision_cmds,
k8s_join_command = var.k8s_join_command,
containerd_config_update_command = var.containerd_config_update_command
is_k8s_template = var.is_k8s_template,
authorized_ssh_key = var.ssh_public_key,
passwd = var.user_passwd,
provision_cmds = var.provision_cmds,
k8s_join_command = var.k8s_join_command,
k8s_node_setup_script_b64 = var.is_k8s_template ? base64encode(file("${path.module}/k8s-node-containerd-setup.sh")) : ""
k8s_node_post_join_script_b64 = var.is_k8s_template ? base64encode(file("${path.module}/k8s-node-post-join-tune.sh")) : ""
}
)
}
# Force recreate when the below changes
triggers = {
file_hash = filesha256("${path.module}/cloud_init.yaml")
provision_cmds = join(", ", var.provision_cmds)
is_k8s_template = var.is_k8s_template,
passwd = var.user_passwd,
k8s_join_command = var.k8s_join_command,
containerd_config_update_command = var.containerd_config_update_command
file_hash = filesha256("${path.module}/cloud_init.yaml")
setup_script_hash = var.is_k8s_template ? filesha256("${path.module}/k8s-node-containerd-setup.sh") : ""
post_join_script_hash = var.is_k8s_template ? filesha256("${path.module}/k8s-node-post-join-tune.sh") : ""
provision_cmds = join(", ", var.provision_cmds)
is_k8s_template = var.is_k8s_template,
passwd = var.user_passwd,
k8s_join_command = var.k8s_join_command,
ssh_public_key = var.ssh_public_key,
}
}