alloy: switch pod log shipping from apiserver to file-tail

Replaced 'loki.source.kubernetes' with 'loki.source.file' in alloy DS
config. discovery.relabel.pod_logs already sets __path__ to the kubelet
log path (/var/log/pods/*<uid>/<container>/*.log) and varlog host-mount
was already present, so this is a one-line swap.

Why: apiserver was burning ~700m sustained on 'CONNECT pods/log' streams
(13 req/s, ~2200 sec/s of long-lived TCP connections). Streaming pod
logs through the apiserver instead of tailing kubelet's log files was
the dominant residual cost after the recent Loki/Alloy onboarding.

Measured before/after:
- Alloy DS: ~620m total (5 x ~125m) -> ~92m total (5 x ~18m)
- kube-apiserver: peak 1959m midnight burst, settled 632m

(Stuck-pod recovery: alloy-7zg7t on k8s-master needed --force delete
during rollout — FailedKillPod 'unable to signal init: permission denied'
on runc, transient runtime issue, unrelated to this change.)
This commit is contained in:
Viktor Barzin 2026-05-21 08:27:34 +00:00
parent b6724a5d48
commit aba061cf2e

View file

@ -81,8 +81,11 @@ alloy:
}
}
// loki.source.kubernetes tails logs from Kubernetes containers using the Kubernetes API.
loki.source.kubernetes "pod_logs" {
// loki.source.file tails pod logs from /var/log/pods/* on the host filesystem.
// Previously used loki.source.kubernetes (apiserver streaming) which drove
// kube-apiserver `CONNECT pods/log` to ~13 req/s + ~2200 sec/s of streams.
// discovery.relabel.pod_logs already sets __path__ to the kubelet log path.
loki.source.file "pod_logs" {
targets = discovery.relabel.pod_logs.output
forward_to = [loki.process.pod_logs.receiver]
}