alloy: switch pod log shipping from apiserver to file-tail
Replaced 'loki.source.kubernetes' with 'loki.source.file' in alloy DS config. discovery.relabel.pod_logs already sets __path__ to the kubelet log path (/var/log/pods/*<uid>/<container>/*.log) and varlog host-mount was already present, so this is a one-line swap. Why: apiserver was burning ~700m sustained on 'CONNECT pods/log' streams (13 req/s, ~2200 sec/s of long-lived TCP connections). Streaming pod logs through the apiserver instead of tailing kubelet's log files was the dominant residual cost after the recent Loki/Alloy onboarding. Measured before/after: - Alloy DS: ~620m total (5 x ~125m) -> ~92m total (5 x ~18m) - kube-apiserver: peak 1959m midnight burst, settled 632m (Stuck-pod recovery: alloy-7zg7t on k8s-master needed --force delete during rollout — FailedKillPod 'unable to signal init: permission denied' on runc, transient runtime issue, unrelated to this change.)
This commit is contained in:
parent
b6724a5d48
commit
aba061cf2e
1 changed files with 5 additions and 2 deletions
|
|
@ -81,8 +81,11 @@ alloy:
|
|||
}
|
||||
}
|
||||
|
||||
// loki.source.kubernetes tails logs from Kubernetes containers using the Kubernetes API.
|
||||
loki.source.kubernetes "pod_logs" {
|
||||
// loki.source.file tails pod logs from /var/log/pods/* on the host filesystem.
|
||||
// Previously used loki.source.kubernetes (apiserver streaming) which drove
|
||||
// kube-apiserver `CONNECT pods/log` to ~13 req/s + ~2200 sec/s of streams.
|
||||
// discovery.relabel.pod_logs already sets __path__ to the kubelet log path.
|
||||
loki.source.file "pod_logs" {
|
||||
targets = discovery.relabel.pod_logs.output
|
||||
forward_to = [loki.process.pod_logs.receiver]
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue