Every internal *.viktorbarzin.me hostname (~80 services) chains through the
split-horizon `viktorbarzin.me` apex A record. If the apex drifts (ISP
rollover, accidental edit), every internal service breaks at once — the
2026-05-22 ha-sofia incident was exactly this.
This adds a backstop probe so the next drift surfaces in <10 min instead
of via user-reported outage:
- CronJob `viktorbarzin-apex-probe` in `technitium` namespace, every 5 min,
resolves `viktorbarzin.me A` against the Technitium LB IP (10.0.20.201)
and pushes `viktorbarzin_apex_correct` + `_last_correct_timestamp` to
Pushgateway. Python+dnspython, ~30 LOC.
- 3 Prometheus alerts:
- `ViktorBarzinApexDrift` (critical, 10m) — apex resolved to anything
other than 10.0.20.200.
- `ViktorBarzinApexProbeStale` (warning, 5m on 15m gap) — probe stopped
succeeding.
- `ViktorBarzinApexProbeNeverRun` (warning, 30m absent) — probe never
reported.
- Added the new alert names to the Slack receiver matcher in both routes
alongside EmailRoundtrip*.
Verified: rules loaded as inactive (apex is correct), metric flowing, manual
probe job pass observed.
Viktor Barzin
000d306542