From 71501be40881e15f6aff9f4318f14045e9077fe6 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 30 Jun 2026 08:15:38 +0000 Subject: [PATCH] nodes: journald -> volatile (RAM) to cut sdc write-IOPS Node "container churn" investigation (code-oflt): container logs (~30 KB/s) and overlayfs (~17 KB/s) are negligible; the node OS-disk churn is ext4 journal (jbd2) metadata writes driven mostly by journald's continuous appends. node4 + node5 had drifted to uncapped persistent journald (4 GB each, ~100 KB/s); master/node1-3 were correctly capped at 500M. Node + pod journals already ship to Loki (alloy loki.source.journal), so on-disk journald is pure write-IOPS overhead on the IOPS-bound sdc. Switch journald to Storage=volatile (RAM, RuntimeMaxUse=200M) fleet-wide: - cloud_init.yaml: drop-in 90-oflt-volatile.conf for new nodes (replaces the old persistent seds). - running nodes (master + node1-5): pushed the same drop-in via qm guest exec + journald restart + cleared /var/log/journal. Verified node5: OS-disk writers jbd2/sda1-8 931->46 KB/s, systemd-journal gone (~94% drop); ~4 GB freed each on node4/node5. Logs stay queryable in Loki. Trade-off: a hard crash loses the last unshipped journal. Co-Authored-By: Claude Opus 4.8 --- .claude/CLAUDE.md | 2 +- modules/create-template-vm/cloud_init.yaml | 13 ++++++------- 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 8df9fd8d..42d72a35 100755 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -42,7 +42,7 @@ Violations cause state drift, which causes future applies to break or silently r - **Image registry**: **Owned images now live on `ghcr.io/viktorbarzin/`** (ADR-0002, built by GHA — see the CI/CD Architecture section). The **Forgejo container registry is FROZEN + emptied** (break-glass only — `docs/runbooks/forgejo-registry-breakglass.md`); nothing pushes to it. The rest of this bullet documents the **still-live forgejo-pull DNS/mirror machinery** (it remains in place for the break-glass path + because `registry-credentials` is still Kyverno-synced; the hairpin lessons apply to any internal-registry pull). Historical usage was `image: forgejo.viktorbarzin.me/viktor/:` + `imagePullSecrets: [{name: registry-credentials}]`. **Kubelet pulls** are kept off the hairpin **at the resolver, with zero node-side DNS config**: pfSense Unbound carries a domain override forwarding the whole `viktorbarzin.me` zone to Technitium (added 2026-06-10, `docs/runbooks/pfsense-unbound.md`), whose split-horizon zone CNAMEs every ingress host (auto-synced hourly by `technitium-ingress-dns-sync`) to the zone apex whose A record tracks the **live** Traefik LB IP (canary: `viktorbarzin-apex-probe`, alerts ViktorBarzinApexDrift). Nodes are stock — link DNS `10.0.20.1 94.140.14.14` via `qm set --nameserver`, no `/etc/hosts` pins, no resolved drop-ins (two same-day interim approaches on 2026-06-10 were removed the same day). The containerd `hosts.toml` mirror (`[host."https://10.0.20.203"]`, `skip_verify = true`) still exists but is **vestigial** — it can NOT keep pulls internal on its own: Traefik routes by Host/SNI and 404s the mirror's bare-IP requests, and the registry's Bearer auth realm is the absolute `https://forgejo.viktorbarzin.me/v2/token` URL fetched outside the mirror — without internal DNS every fresh pull degrades to public DNS → hairpin → intermittent `dial tcp 176.12.22.76:443: i/o timeout` ImagePullBackOff (tuya-bridge 7.5h outage 2026-06-10, tripit 2026-06-09; see `docs/post-mortems/2026-06-10-tuya-bridge-forgejo-pull-hairpin.md`). **In-cluster pods are ordinary internal clients too** (since 2026-06-10 evening) — CoreDNS's dedicated `viktorbarzin.me:53` block (Corefile in `stacks/technitium/modules/technitium/main.tf`) forwards to the Technitium ClusterIP `10.96.0.53`, so pods get the same split-horizon answers as everyone else; forgejo stays pinned to Traefik's **ClusterIP** in that block (TF-interpolated from the live Service) so CI pushes survive a Technitium outage. This relies on a k8s-1.34 behavior verified 2026-06-10: **pods CAN reach the ETP=Local Traefik LB IP** (kube-proxy short-circuits in-cluster traffic to LB IPs via the cluster path) — re-verify after major k8s upgrades; canary = the uptime-kuma `[External]` fleet going red. (The block briefly forwarded to `8.8.8.8/1.1.1.1` earlier that day, which kept pods on the WAN IP and the broken TP-Link NAT loopback — 27 non-proxied `[External]` monitors dark; beads code-yh33.) **Was `.200` until 2026-06-01** — Traefik's 2026-05-30 move to its dedicated `.203` left the mirror pointing at the now-dead `.200:443`, silently breaking every *fresh* forgejo pull; a future LB renumber is now handled by DNS (apex record + drift probe) — only the vestigial hosts.toml literal would go stale. Mirror source lives in `modules/create-template-vm/k8s-node-containerd-setup.sh` (new nodes) and `scripts/setup-forgejo-containerd-mirror.sh` (existing nodes; also cleans up the legacy 2026-06-10 node-DNS customization). Push-side: viktor PAT in Vault `secret/ci/global/forgejo_push_token` (Forgejo container packages are scoped per-user; only the package owner can push, ci-pusher cannot write to viktor/*). Pull-side: cluster-puller PAT in Vault `secret/viktor/forgejo_pull_token`. Retention CronJob (`forgejo-cleanup` in `forgejo` ns, daily 04:00) keeps newest 10 versions + always `:latest` + any buildkit `*cache*` tag — **REVERTED to DRY_RUN 2026-06-10 after its first live run orphaned OCI index children** (multi-arch/attestation children are separate *untagged* sha256 versions that sort outside the newest-10 window while their parent index is kept; broke `kms-website:latest`+`:dfc83fb`, caught by the integrity probe, healed by re-tagging latest→a794d1a + deleting the corrupt version; see `docs/post-mortems/2026-06-10-forgejo-retention-orphaned-indexes.md`). Do NOT re-enable deletes until the keep-set resolves kept indexes' child digests (or skips untagged versions, or moves to Forgejo's native container-aware cleanup rules). The registry PVC remains at its 50Gi autoresize ceiling on the HDD (we did NOT move it to SSD, see beads code-oflt), so a container-aware retention is still needed. Integrity probed every 15min by `forgejo-integrity-probe` in `monitoring` ns (catalog walk + manifest HEAD on every blob). See `docs/plans/2026-05-07-forgejo-registry-consolidation-{design,plan}.md` for the migration history. Pull-through caches for upstream registries (DockerHub, GHCR, Quay, k8s.gcr, Kyverno) stay on the registry VM at `10.0.20.10` ports 5000/5010/5020/5030/5040 — the old port-5050 R/W private registry was decommissioned 2026-05-07. - **LinuxServer.io containers**: `DOCKER_MODS` runs apt-get on every start — bake slow mods into a custom image (`RUN /docker-mods || true` then `ENV DOCKER_MODS=`). Set `NO_CHOWN=true` to skip recursive chown that hangs on NFS mounts. - **Node memory changes**: When changing VM memory on any k8s node, update kubelet `systemReserved`, `kubeReserved`, and eviction thresholds accordingly. Config: `/var/lib/kubelet/config.yaml`. Template: `stacks/infra/main.tf`. Current values: systemReserved=512Mi, kubeReserved=512Mi, evictionHard=500Mi, evictionSoft=1Gi. -- **Node OS disk tuning** (in `stacks/infra/main.tf`): kubelet `imageGCHighThresholdPercent=70` (was 85), `imageGCLowThresholdPercent=60` (was 80), ext4 `commit=60` in fstab (was default 5s), journald `SystemMaxUse=200M` + `MaxRetentionSec=3day`. +- **Node OS disk tuning** (cloud-init in `modules/create-template-vm/cloud_init.yaml`; kubelet/ext4 also in `stacks/infra/main.tf`): kubelet `imageGCHighThresholdPercent=70` (was 85), `imageGCLowThresholdPercent=60` (was 80), ext4 `commit=60` in fstab (was default 5s). **journald `Storage=volatile` (RAM, `RuntimeMaxUse=200M`) since 2026-06-30 (code-oflt)** — node+pod journals ship to Loki via alloy (`loki.source.journal`), so on-disk journald was pure write-IOPS overhead on the IOPS-bound `sdc`; moving it to RAM cut node5's OS-disk writes ~94% (`jbd2` 931→46 KB/s) and freed ~4 GB each on node4/node5 (which had drifted to uncapped persistent). Applied to running nodes via `qm guest exec` drop-in `/etc/systemd/journald.conf.d/90-oflt-volatile.conf` + journald restart. Trade-off: a hard crash loses the last unshipped journal (Loki has the rest). - **Sealed Secrets**: User-managed secrets go in `sealed-*.yaml` files in the stack directory. Stacks pick them up via `kubernetes_manifest` + `fileset(path.module, "sealed-*.yaml")`. See AGENTS.md for full workflow. - **CRITICAL — Update docs with every change**: When modifying infrastructure (Terraform, Vault, networking, storage, CI/CD, monitoring), you MUST update all affected documentation in the same commit. Check and update: `docs/architecture/*.md`, `docs/runbooks/*.md`, `.claude/CLAUDE.md`, `AGENTS.md`, `.claude/reference/service-catalog.md`. Stale docs cause incident response failures and onboarding confusion. If unsure which docs are affected, grep for the service/resource name across all doc files. diff --git a/modules/create-template-vm/cloud_init.yaml b/modules/create-template-vm/cloud_init.yaml index 11a86b6e..1ade11c5 100644 --- a/modules/create-template-vm/cloud_init.yaml +++ b/modules/create-template-vm/cloud_init.yaml @@ -88,13 +88,12 @@ write_files: runcmd: # Enable weekly TRIM/discard to reclaim freed blocks in LVM thin pool - systemctl enable --now fstrim.timer - # Enable persistent journald logging for crash forensics, with size limits to reduce disk wear - - mkdir -p /var/log/journal - - sed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf - - sed -i 's/#SystemMaxUse=/SystemMaxUse=500M/' /etc/systemd/journald.conf - - sed -i 's/#MaxRetentionSec=/MaxRetentionSec=7day/' /etc/systemd/journald.conf - - sed -i 's/#MaxFileSec=/MaxFileSec=1day/' /etc/systemd/journald.conf - - sed -i 's/#Compress=yes/Compress=yes/' /etc/systemd/journald.conf + # journald in RAM (volatile, capped) — node + pod journals already ship to + # Loki via alloy (loki.source.journal), so on-disk journald is pure sdc + # write-IOPS overhead on the IOPS-bound HDD. code-oflt 2026-06-30. + - mkdir -p /etc/systemd/journald.conf.d + - printf '[Journal]\nStorage=volatile\nRuntimeMaxUse=200M\nCompress=yes\n' > /etc/systemd/journald.conf.d/90-oflt-volatile.conf + - rm -rf /var/log/journal - systemctl restart systemd-journald %{if is_k8s_template} # Node DNS is intentionally STOCK — no resolved drop-ins, no /etc/hosts