infra/docs/plans/2026-02-13-centralized-log-collection-design.md

Centralized Log Collection Design

Date: 2026-02-13

Goal

Centrally collect logs from all Kubernetes pods for monitoring and alerting. Minimize disk I/O by holding logs in memory for extended periods, flushing to NFS once daily. Alert on log patterns via existing Alertmanager pipeline.

Requirements

• Primary use case: Monitoring and alerting (log-based alert rules evaluated in real-time)
• Retention: 7 days on disk after flush
• Memory budget: 4-8GB total (~6.6GB used)
• Disk strategy: 24h in-memory chunks, WAL on tmpfs, single daily flush to NFS
• Crash policy: Accept up to 24h log loss on pod/node crash (alerts still fire in real-time before flush)
• Alert delivery: Loki Ruler -> existing Alertmanager -> Slack/email

Architecture

┌──────────────────┐     ┌──────────────────────┐     ┌──────────────┐
│ Alloy DaemonSet  │     │ Loki SingleBinary     │     │ Grafana       │
│ 5 pods, 128Mi ea │────>│ 1 pod, 6Gi RAM        │<────│ (existing)    │
│ tails /var/log/  │     │                        │     │ + Loki        │
│ pods on each node│     │ Ingester: 24h chunks   │     │   datasource  │
└──────────────────┘     │ WAL: tmpfs (in-memory) │     └──────────────┘
                         │ Storage: NFS 15Gi      │
┌──────────────────┐     │ Ruler ──> Alertmanager │
│ Sysctl DaemonSet │     └──────────────────────┘
│ 5 pods (pause)   │
│ sets inotify     │
│ limits on nodes  │
└──────────────────┘

Components

1. Sysctl DaemonSet

Solves the too many open files / fsnotify watcher exhaustion problem that previously blocked Alloy.

• Privileged init container runs sysctl -w on each node
• Settings: fs.inotify.max_user_watches=1048576, fs.inotify.max_user_instances=512, fs.inotify.max_queued_events=1048576
• Main container: pause image (near-zero resources)
• Survives node reboots (DaemonSet recreates pod)
• Namespace: monitoring

2. Loki (Helm Release)

Single-binary deployment. Existing Helm chart config in loki.yaml, updated with:

Ingester tuning (disk-friendly):

• chunk_idle_period: 12h — don't flush idle streams quickly
• max_chunk_age: 24h — hold chunks in memory for full day
• chunk_retain_period: 1m — brief retain after flush
• chunk_target_size: 1572864 (1.5MB) — larger chunks = fewer writes
• WAL: tmpfs emptyDir (medium: Memory, 2Gi limit)

Retention:

• retention_period: 168h (7 days)
• Compactor enabled for retention enforcement

Ruler:

• Evaluates LogQL alert rules in real-time (before chunk flush)
• Fires to http://alertmanager.monitoring.svc.cluster.local:9093

Storage:

• NFS PV/PVC at /mnt/main/loki/loki (15Gi, existing)
• TSDB index with 24h period

Resources:

• Memory: 6Gi limit
• CPU: 500m limit

3. Alloy (Helm Release)

DaemonSet log collector. Existing config in alloy.yaml is complete:

• Discovers pods via discovery.kubernetes
• Labels: namespace, pod, container, app, job, container_runtime, cluster
• Tails /var/log/pods/ on each node
• Forwards to http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push

Resources per pod:

• Memory: 128Mi limit
• CPU: 100m limit

4. Grafana Datasource

ConfigMap with label grafana_datasource: "1" for sidecar auto-discovery:

• Name: Loki
• Type: loki
• URL: http://loki.monitoring.svc.cluster.local:3100
• Existing loki.json dashboard already in dashboards directory

5. Starter Alert Rules

Configured in Loki Ruler (evaluated in real-time, before disk flush):

Alert	LogQL Expression	Severity
HighErrorRate	sum(rate({namespace=~".+"} |= "error" [5m])) by (namespace) > 10	warning
PodCrashLoopBackOff	count_over_time({namespace=~".+"} |= "CrashLoopBackOff" [5m]) > 0	critical
OOMKilled	count_over_time({namespace=~".+"} |= "OOMKilled" [5m]) > 0	critical

Memory Budget

Component	Per-pod	Pods	Total
Alloy	128Mi	5	640Mi
Loki	6Gi	1	6Gi
Sysctl DS	~0 (pause)	5	~0
Total			~6.6 GB

Files to Change

File	Action
modules/kubernetes/monitoring/loki.tf	Uncomment Loki + Alloy helm releases, add sysctl DaemonSet, add Grafana Loki datasource ConfigMap
modules/kubernetes/monitoring/loki.yaml	Update with ingester tuning, ruler config, retention, resource limits
modules/kubernetes/monitoring/alloy.yaml	Add resource limits in Helm values wrapper
secrets/nfs_directories.txt	Ensure /mnt/main/loki entries exist

Implementation Steps

1. Add sysctl DaemonSet to loki.tf
2. Update loki.yaml with disk-friendly tuning, ruler, retention, resources
3. Update alloy.yaml with resource limits
4. Uncomment Loki Helm release in loki.tf, wire up NFS PV/PVC
5. Uncomment Alloy Helm release in loki.tf
6. Add Grafana Loki datasource ConfigMap to loki.tf
7. Add alert rules to Loki config
8. Ensure NFS exports exist in secrets/nfs_directories.txt
9. terraform apply -target=module.kubernetes_cluster.module.monitoring
10. Verify: Grafana Explore -> Loki datasource -> query {namespace="monitoring"}

Risks

• 24h data loss on crash: Accepted trade-off. Alerts fire in real-time before flush, so alert coverage is not affected — only historical log browsing is at risk.
• Memory pressure: 6Gi for Loki on a 16GB node is significant. Monitor with existing Prometheus memory alerts.
• Log volume spikes: A chatty pod could cause Loki to OOM. Alloy can be configured with rate limiting if needed (future enhancement).