viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

[monitoring] Exclude websocket protocol from HighServiceLatency alert

Browse source 
Traefik records websocket connection lifetimes (minutes to hours) as
"request duration." When websockets close, the full lifetime pollutes
the average latency metric — Authentik showed 6.7s avg (201s websocket
avg) vs 0.065s actual HTTP avg. This caused ~90 false alerts/day across
12 services (Authentik, Vaultwarden, Terminal, HA, etc.).

Changes:
- Add protocol!="websocket" filter to HighServiceLatency alert expr
- Raise minimum traffic threshold from 0.01 to 0.05 rps to filter
  statistical noise from services with <3 req/min
- Remove .githooks/pre-commit file-size hook (blocked state commits)

Validated against 7-day historical data: 637 breaches → ~2 with both
filters applied (99.7% reduction).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-04-15 21:51:19 +00:00
parent 3e273399c1
commit 375a3d91d5
3 changed files with 2998 additions and 3035 deletions
Download patch file Download diff file Expand all files Collapse all files

6
stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl
View file

@ -1594,10 +1594,10 @@ serverFiles:
- alert: HighServiceLatency
expr: |
(
sum(rate(traefik_service_request_duration_seconds_sum{service!~".*idrac.*|.*headscale.*"}[5m])) by (service)
/ sum(rate(traefik_service_request_duration_seconds_count{service!~".*idrac.*|.*headscale.*"}[5m])) by (service)
sum(rate(traefik_service_request_duration_seconds_sum{service!~".*idrac.*|.*headscale.*",protocol!="websocket"}[5m])) by (service)
/ sum(rate(traefik_service_request_duration_seconds_count{service!~".*idrac.*|.*headscale.*",protocol!="websocket"}[5m])) by (service)
) > 10
and sum(rate(traefik_service_request_duration_seconds_count{service!~".*idrac.*|.*headscale.*"}[5m])) by (service) > 0.01
and sum(rate(traefik_service_request_duration_seconds_count{service!~".*idrac.*|.*headscale.*",protocol!="websocket"}[5m])) by (service) > 0.05
and on() (time() - process_start_time_seconds{job="prometheus"}) > 900
for: 5m
labels: