viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: correct cloudflared-502 post-mortem + fix stale .200 Traefik ref [ci skip]

Browse source 
Real root cause of the 2026-06-01 full-site 502 was not a missed
reference but an out-of-band fix that Terraform reverted: the 2026-05-30
Traefik .200->.203 migration repointed the Cloudflare tunnel to the
Traefik service DNS via the CF Global API Key, but never landed that
change in cloudflare.tf (left at .200). A terragrunt apply on 2026-06-01
reconciled live back to the stale .200, breaking all external ingress.
Rewrite the post-mortem around the "codify out-of-band fixes or TF
reverts them" lesson (a Terraform-Only-rule violation).

Also fix docs/runbooks/kms-public-exposure.md, which still claimed
Traefik served on 10.0.20.200:443 (now .203) — same migration fallout.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-06-01 21:25:33 +00:00
parent f807050eb5
commit 9fb3e6e851
2 changed files with 40 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

73
docs/post-mortems/2026-06-01-cloudflared-stale-traefik-origin.md
View file

@ -1,73 +1,78 @@
# Post-Mortem: Cloudflare Tunnel Pointed at Traefik's Old LB IP → Full External 502
# Post-Mortem: Out-of-Band Tunnel Repoint Reverted by Terraform → Full External 502
| Field | Value |
|-------|-------|
| **Date** | 2026-06-01 |
| **Duration** | Misconfiguration latent since 2026-05-30 08:09Z (Traefik LB-IP move). Confirmed external outage in cloudflared logs from ~20:58Z; root-caused and fixed at 21:15Z; all pods converged by 21:16Z. Detection→fix window ~17 min. |
| **Severity** | SEV1 — *every* Cloudflare-proxied hostname (`viktorbarzin.me` + all `*.viktorbarzin.me`) returned HTTP 502 to external clients. Internal/LAN access was unaffected (split-horizon → Traefik direct), which is why it stayed hidden. |
| **Duration** | Drift present 2026-05-30 → 2026-06-01. Actual external outage began when a `terragrunt apply` reverted the tunnel origin on 2026-06-01 (cloudflared errors visible from ≥20:58Z); root-caused and fixed at 21:15Z; pods converged 21:16Z. |
| **Severity** | SEV1 — *every* Cloudflare-proxied hostname (`viktorbarzin.me` + all `*.viktorbarzin.me`) returned HTTP 502 to external clients. Internal/LAN access (split-horizon → Traefik direct) was unaffected, which is why it stayed hidden. |
| **Affected Services** | All external ingress: viktorbarzin.me, nextcloud, vault, authentik, vaultwarden, immich, linkwarden, nas, technitium, terminal, speedtest, and every other proxied app. |
| **Issue** | None filed (diagnosed and fixed in-session). |
| **Status** | Resolved. |
| **Recurrence count** | 1st of this kind. Same *class* as the 2026-06-01 forgejo-registry `.200→.203` redirect breakage (containerd mirror) — both are fallout from the 2026-05-30 Traefik LB-IP move leaving a hard-coded `10.0.20.200` reference behind. |
| **Recurrence count** | 1st of this exact kind. Same `.200→.203` migration family as the 2026-06-01 forgejo-registry containerd-redirect fix (`a382683c`). |
## Summary
On 2026-05-30 (commit `0c01adac`) Traefik was moved off the shared MetalLB IP `10.0.20.200` onto its own dedicated IP `10.0.20.203` (with `externalTrafficPolicy: Local`). The Cloudflare tunnel's ingress rules — Terraform-managed in `stacks/cloudflared/modules/cloudflared/cloudflare.tf` — still routed `*.viktorbarzin.me` and `viktorbarzin.me` to `https://10.0.20.200:443`. After the move, nothing serves HTTPS on `.200:443` (the shared IP keeps only the non-HTTP LB services: postgresql-lb, headscale, wireguard, coturn, xray). cloudflared therefore could not reach its origin (`connect: no route to host` / `i/o timeout`), and Cloudflare returned 502 for the entire public surface.
On 2026-05-30 Traefik was moved off the shared MetalLB IP `10.0.20.200` onto a dedicated `10.0.20.203`. The migration plan correctly identified that the Cloudflare tunnel had to be repointed away from `10.0.20.200:443` **first**, and it was — but the repoint was done **out-of-band via the Cloudflare Global API Key**, not in Terraform. The Terraform source (`stacks/cloudflared/modules/cloudflared/cloudflare.tf`) was left pointing at `https://10.0.20.200:443`, creating silent drift between live (correct: service DNS) and code (stale: `.200`).
The fix: repoint both ingress rules at the in-cluster Traefik **Service DNS** `https://traefik.traefik.svc.cluster.local:443` — the design the docs already *described* (CLAUDE.md "Networking" §) but which the code never actually implemented. Service DNS decouples the tunnel from the LB IP, so a future Traefik IP change cannot reproduce this.
External ingress kept working for ~2 days on the manual config. Then on 2026-06-01 a `terragrunt apply` of the cloudflared stack reconciled live back to the stale Terraform value `https://10.0.20.200:443`. Nothing serves HTTPS on `.200:443` after the Traefik move, so cloudflared could not reach its origin (`connect: no route to host` / `i/o timeout`) and Cloudflare returned 502 across the entire public surface.
Fix: codify the correct origin in Terraform — both ingress rules now point at `https://traefik.traefik.svc.cluster.local:443` (in-cluster Traefik Service DNS). This both restores ingress and makes it permanent (TF and live agree; future applies can't revert it; the origin is decoupled from the Traefik LB IP entirely).
## Impact
- **User-facing**: 100% of externally-reachable services returned 502 via Cloudflare. LAN/internal access (which resolves `*.viktorbarzin.me` → `10.0.20.203` via Technitium split-horizon, bypassing Cloudflare) kept working — this masked the outage.
- **Blast radius**: every proxied hostname. Origin (Traefik) was healthy the entire time — purely a tunnel-origin routing fault.
- **Blast radius**: every proxied hostname. Origin (Traefik) was healthy the whole time — purely a tunnel-origin routing fault.
- **Data loss**: none.
- **Collateral**: Vault's own public hostname (`vault.viktorbarzin.me`) was also 502, creating a bootstrap problem for the fix — `terragrunt apply` needs Vault for the PG state-backend creds, but Vault was only reachable via the broken tunnel from the dev box. Worked around with a temporary `/etc/hosts` entry pointing `vault.viktorbarzin.me` → `10.0.20.203` (internal Traefik), removed after the apply.
- **Collateral**: Vault's own public hostname (`vault.viktorbarzin.me`) was also 502, creating a bootstrap problem — `terragrunt apply` needs Vault for the PG state-backend creds, but Vault was only reachable from the dev box via the broken tunnel. Worked around with a temporary `/etc/hosts` entry pointing `vault.viktorbarzin.me` → `10.0.20.203` (internal Traefik), removed after the apply.
## Root Cause
A hard-coded LB IP (`10.0.20.200`) in the tunnel origin survived the Traefik dedicated-IP migration. The 2026-05-30 migration updated Traefik's Service and the split-horizon DNS but did not grep for every consumer of the old `.200` HTTPS endpoint. The cloudflared tunnel origin (and, separately, the containerd forgejo-registry redirect — fixed earlier the same day in `42db69a2`) were missed.
**A manual (out-of-band) fix was never codified in Terraform, and a later Terraform apply reverted it.** This is a direct violation of the repo's "Terraform Only — all infra changes go through Terraform" rule. The 2026-05-30 plan applied the tunnel repoint via the Cloudflare API for speed/safety during the migration but did not land the equivalent change in `cloudflare.tf`. Terraform's authority over the resource guaranteed the manual change would eventually be reverted; it was, on the next apply.
Contributing factors:
- **Docs described intent as reality.** CLAUDE.md stated cloudflared targets `traefik.traefik.svc.cluster.local:443` "so proxied apps are decoupled from the LB IP." The code used a raw IP. The doc gave false confidence that the decoupling existed.
- **No guard** tied the tunnel origin to Traefik's actual address; a stale value plans/applies cleanly.
- **Detection gap (masking).** Split-horizon means LAN users never see external-only breakage. The `[External]` Uptime-Kuma monitors + `ExternalAccessDivergence` alert are the only signal for this failure mode.
- **No drift alarm tied this to user impact.** The TF/live divergence on `cloudflare_zero_trust_tunnel_cloudflared_config` existed for ~2 days; drift-detection (if it ran) didn't escalate it as outage-risk.
- **Detection gap (masking).** Split-horizon means LAN users never see external-only breakage. The `[External]` Uptime-Kuma monitors + `ExternalAccessDivergence` alert are the only signal for this failure mode, and they did not prompt action.
- **Docs vs code.** CLAUDE.md described cloudflared as targeting the service DNS — true of live (post-manual-fix) but not of the TF source. The doc masked the drift.
## Timeline (UTC)
| Time | Event |
|------|-------|
| **2026-05-30 08:09** | Commit `0c01adac` — Traefik moves to dedicated LB IP `10.0.20.203`. `.200:443` stops serving HTTPS. Tunnel origin still `.200`. Outage latent from here. |
| **2026-06-01 ~20:51** | Keel auto-patches the cloudflared image; all 3 pods roll (coincidental — not the cause; the misconfig predates it). |
| **2026-06-01 ~20:58** | cloudflared logs show every proxied hostname failing: `originService=https://10.0.20.200:443 … no route to host / i/o timeout`. |
| **2026-06-01 ~21:08** | User reports "no ingress coming in." Investigation starts. |
| **21:09** | Isolated: origin healthy (direct to `.203` → 200/302), public path → 502. cloudflared logs pin origin to dead `.200:443`. |
| **21:10** | Confirmed tunnel config is Terraform-managed (`cloudflare_zero_trust_tunnel_cloudflared_config.sof`), origin = `.200` on both ingress rules. |
| **21:13** | Vault unreachable via public name (circular dep); worked around with temp `/etc/hosts` → `.203`. `tg init -reconfigure` (rotated PG backend creds). |
| **21:15:25** | Targeted apply: both ingress origins → `https://traefik.traefik.svc.cluster.local:443`. `Apply complete! 1 changed`. |
| **21:15:34–50** | cloudflared pushes config `version=253`; pods converge. |
| **21:16** | 10/10 curls to `viktorbarzin.me` → 200; 0 `.200` errors across all pods; `vault.viktorbarzin.me` via real Cloudflare path → 200. Temp hosts entry removed. Resolved. |
| **2026-05-30 ~08:09** | Traefik Service moved to `10.0.20.203` (`ETP=Local`). Plan step 1 repoints the tunnel `https://10.0.20.200:443` → `https://traefik.traefik.svc.cluster.local:443` **via the CF Global API Key (out-of-band)**. Ingress works. `cloudflare.tf` still says `.200` → **drift**. |
| **2026-05-30 → 06-01** | External ingress healthy on the manual config. Drift sits unnoticed. |
| **2026-06-01 (during the day)** | A `terragrunt apply` of the cloudflared stack reconciles the tunnel origin back to the stale TF value `https://10.0.20.200:443`. External ingress breaks → 502. |
| **2026-06-01 ~20:51** | Keel auto-patches the cloudflared image; pods roll (coincidental, not causal). |
| **~20:58** | cloudflared logs show every proxied hostname failing against `https://10.0.20.200:443` (`no route to host` / `i/o timeout`). |
| **21:08** | User reports "no ingress coming in." Investigation starts. |
| **21:09** | Isolated: origin healthy (direct to `.203` → 200/302), public path → 502; logs pin origin to dead `.200:443`. |
| **21:13** | Vault unreachable via public name (circular dep); temp `/etc/hosts` → `.203`. `tg init -reconfigure` (rotated PG backend creds). |
| **21:15:25** | Targeted apply: both ingress origins → service DNS. `Apply complete! 1 changed`. |
| **21:16** | 10/10 curls to `viktorbarzin.me` → 200; 0 `.200` errors across all pods; `vault.viktorbarzin.me` via real Cloudflare → 200. Temp hosts entry removed. Resolved + committed (`f807050e`). |
## Resolution
Changed both `ingress_rule` blocks in `cloudflare.tf` from `https://10.0.20.200:443` to `https://traefik.traefik.svc.cluster.local:443` (`no_tls_verify = true` retained). Applied surgically with `-target` on the tunnel config resource only, to avoid touching two pre-existing, unrelated drift items the full plan surfaced (see below).
Changed both `ingress_rule` blocks in `cloudflare.tf` from `https://10.0.20.200:443` to `https://traefik.traefik.svc.cluster.local:443` (`no_tls_verify` retained), making the Terraform source match the intended (and previously-manual) live config. Applied surgically with `-target` on the tunnel config resource only, to avoid touching two pre-existing, unrelated drift items the full plan surfaced (below). Committed `[ci skip]` since live already matched after the targeted apply.
## Pre-existing drift (NOT part of this incident, left untouched)
The full `cloudflared` stack plan showed two extra in-place changes, deliberately **not** applied:
1. `kubernetes_deployment.cloudflared` — TF would strip Keel's runtime annotations (`keel.sh/policy|pollSchedule|trigger|update-time`). The deployment ignores `dns_config` but not `metadata.annotations`, so Keel's enrollment annotations look like drift. Self-healing (Keel re-adds within its 1h poll), but a clean fix is to add `metadata[0].annotations` (and the template equivalent) to `ignore_changes`, or codify the policy annotation in TF.
2. `cloudflare_record.mail_domainkey_rspamd` — cosmetic re-chunking of the DKIM TXT record (identical key, different 255-char split). Benign.
1. `kubernetes_deployment.cloudflared` — TF would strip Keel's runtime annotations (`keel.sh/policy|pollSchedule|trigger|update-time`). The deployment ignores `dns_config` but not `metadata.annotations`. Self-healing (Keel re-adds within its 1h poll); clean fix is to add `metadata[0].annotations` (+ template equivalent) to `ignore_changes`.
2. `cloudflare_record.mail_domainkey_rspamd` — cosmetic re-chunking of the DKIM TXT record (identical key). Benign.
## Action Items
- [x] Repoint tunnel origin to Traefik Service DNS (this fix).
- [x] Post-mortem written; CLAUDE.md networking claim is now actually true.
- [ ] **Pin exact outage-start** via Uptime-Kuma `[External]` monitor history / `ExternalAccessDivergence` firing time (confirm whether it began at the 05-30 move and went unnoticed, or at a later tunnel re-apply).
- [ ] **Verify `ExternalAccessDivergence` is wired to a channel that gets seen** — this is the only alert that catches external-only breakage; it apparently did not prompt action for ≤2.5 days.
- [ ] **Migration checklist**: when an LB IP changes, grep the whole repo for the old IP before declaring done (this and the forgejo redirect were both missed `.200` references on 2026-05-30).
- [ ] (Optional) Address the cloudflared Keel-annotation drift so the stack plans clean.
- [x] Codify tunnel origin (service DNS) in `cloudflare.tf` (this fix) — drift eliminated.
- [x] Fix stale `10.0.20.200:443` Traefik reference in `docs/runbooks/kms-public-exposure.md` (→ `.203`).
- [x] Post-mortem written.
- [ ] **Audit for other out-of-band changes from the 2026-05-30 migration** that were applied via CF API / kubectl / pfSense but not landed in code — they will all revert on the next apply.
- [ ] **Make `ExternalAccessDivergence` trustworthy and seen** — it is the only signal for external-only outages and did not prompt action here.
- [ ] **Drift detection should flag tunnel-origin divergence as outage-risk**, not just generic drift.
- [ ] (Optional) Pin the exact reverting-apply time via Woodpecker pipeline history for the cloudflared stack on 2026-06-01.
- [ ] (Optional) Fix the cloudflared Keel-annotation drift so the stack plans clean.
## Lessons
- Reference shared infra (Traefik) by **stable Service DNS, not LB IP**, from anything that can use cluster DNS. IPs are migration landmines.
- Keep docs honest: a doc that describes intended design as current reality hides exactly this class of bug.
- External-only outages are invisible from the LAN (split-horizon). The `[External]` divergence signal is load-bearing — it must be trustworthy and seen.
- **Codify out-of-band fixes immediately.** A manual change to a Terraform-managed resource is a time bomb — Terraform *will* revert it on the next apply. The "Terraform Only" rule exists for exactly this; the 05-30 manual tunnel repoint should have been mirrored into `cloudflare.tf` the same day.
- **Reference shared infra (Traefik) by stable Service DNS, not LB IP**, from anything that can use cluster DNS. The service-DNS origin also happens to survive LB-IP moves.
- **External-only outages are invisible from the LAN** (split-horizon). The `[External]` divergence signal is load-bearing — it must be trustworthy and actually seen.
- **Keep docs honest about source-of-truth.** "Live is correct" is not the same as "code is correct"; a doc that conflates them hides drift.

2
docs/runbooks/kms-public-exposure.md
View file

@ -216,7 +216,7 @@ If the activation surface needs to come down (abuse, legal, audit):
The k8s service stays reachable on the LAN
(`10.0.20.202:1688` directly, and the website at `kms.viktorbarzin.lan`
via Traefik on `10.0.20.200:443`) — only the WAN port-forward is removed.
via Traefik on `10.0.20.203:443`) — only the WAN port-forward is removed.
To put it back, recreate the NAT rule (target alias `k8s_kms_lb`,
port `1688`) and the filter rule with the same per-source caps. The alias