71501be408
Some checks failed
ci/woodpecker/push/default Pipeline failed
Node "container churn" investigation (code-oflt): container logs (~30 KB/s) and overlayfs (~17 KB/s) are negligible; the node OS-disk churn is ext4 journal (jbd2) metadata writes driven mostly by journald's continuous appends. node4 + node5 had drifted to uncapped persistent journald (4 GB each, ~100 KB/s); master/node1-3 were correctly capped at 500M. Node + pod journals already ship to Loki (alloy loki.source.journal), so on-disk journald is pure write-IOPS overhead on the IOPS-bound sdc. Switch journald to Storage=volatile (RAM, RuntimeMaxUse=200M) fleet-wide: - cloud_init.yaml: drop-in 90-oflt-volatile.conf for new nodes (replaces the old persistent seds). - running nodes (master + node1-5): pushed the same drop-in via qm guest exec + journald restart + cleared /var/log/journal. Verified node5: OS-disk writers jbd2/sda1-8 931->46 KB/s, systemd-journal gone (~94% drop); ~4 GB freed each on node4/node5. Logs stay queryable in Loki. Trade-off: a hard crash loses the last unshipped journal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| cloud_init.yaml | ||
| k8s-node-containerd-setup.sh | ||
| k8s-node-post-join-tune.sh | ||
| main.tf | ||