infra/docs/post-mortems/2026-06-10-tuya-bridge-forgejo-pull-hairpin.md

2026-06-10 — tuya-bridge down 7.5h: forgejo image pulls ride the public-IP hairpin

Impact

• tuya-bridge (Flask/tinytuya bridge feeding HA-Sofia's ATS, fuse-main, fuse-garage and 4 thermostat REST sensors) unavailable ~02:15–09:50 EEST. HA REST sensors 503'd; the official-tuya integration devices were unaffected (hybrid architecture limited the blast radius to the 3 power devices' advanced telemetry + thermostats extras).
• Third incident from the same root cause class: Woodpecker buildkit pushes (2026-06-04, code-yh33), tripit ImagePullBackOff on node2/node3 + devvm git timeouts (2026-06-09), tuya-bridge (this one).

Timeline (EEST)

• 02:15 — tuya-bridge pod rescheduled onto k8s-node3 (its previous node5/6-era home was rebuilt 14d ago; the forgejo-path image was never cached on node3 — only stale docker.io/* copies). Kubelet must pull forgejo.viktorbarzin.me/viktor/tuya_bridge:3216c87a.
• 02:15→09:30 — 51 consecutive pull failures: dial tcp 176.12.22.76:443: i/o timeout → ImagePullBackOff. HA shows 503s (emo observed at 02:20).
• 09:40 — investigation: forgejo healthy via internal Traefik (10.0.20.203), manifest exists; node3's hosts.toml mirror present and correct; bare-IP request to the mirror returns 404 from Traefik; registry auth realm is the absolute public URL.
• 09:48 — /etc/hosts pin 10.0.20.203 forgejo.viktorbarzin.me added on node3; crictl pull succeeds immediately; pod replaced → Running; /health ok; all 27 device getstatus() calls succeed; all 7 *_tuya_cloud_up Prometheus gauges = 1.
• 10:05 — pin rolled to all 7 nodes; provisioning scripts + docs updated.

Root cause

Fresh kubelet pulls of forgejo.viktorbarzin.me images depend on pfSense NAT reflection of the public IP 176.12.22.76, which is intermittently broken from the 10.0.20.0/24 network. The containerd certs.d/.../hosts.toml mirror that was believed to keep pulls internal cannot do so, for two independent reasons:

1. Traefik routes by Host/SNI. The mirror entry [host."https://10.0.20.203"] makes containerd dial the bare IP (no SNI, Host: 10.0.20.203) — no Traefik router matches → 404 → con- tainerd treats the mirror as a miss and falls back to server = "https://forgejo.viktorbarzin.me" → public DNS → hairpin.
2. The Bearer auth realm is absolute. /v2/ challenges with realm="https://forgejo.viktorbarzin.me/v2/token"; containerd fetches that URL verbatim — this leg never goes through the mirror at all.

So every fresh pull silently depended on hairpin luck. Cached images masked the problem; it only fired when a pod landed on a node without the image (node rebuilds, new nodes, evictions, new tags).

Why DNS-side fixes don't reach this path: nodes resolve via systemd-resolved → pfSense (10.0.20.1) + public fallback (94.140.14.14), so Technitium split-horizon (scoped to 192.168.1.0/24 clients) never applies; the CoreDNS forgejo rewrite (2026-06-04) covers pods only, not kubelet.

Fix

Initial mitigation (same morning): /etc/hosts pin 10.0.20.203 forgejo.viktorbarzin.me on every node — restored service immediately (resolve + token + blob legs all internal with correct SNI).

Superseded same day (Viktor: "no hardcoded IPs in nodes") by a DNS-based fix. Discovery: Technitium's split-horizon zone already resolves forgejo.viktorbarzin.me → CNAME viktorbarzin.me → A <live Traefik IP> — the technitium-ingress-dns-sync CronJob auto-CNAMEs every ingress host hourly, the apex A record tracks the live Traefik LB IP, and the viktorbarzin-apex-probe canary alerts on drift. The nodes simply never queried Technitium (resolv chain: pfSense + public AdGuard fallback). The devvm already solved this with a systemd-resolved routing domain drop-in; the same was rolled to all 7 nodes:

# /etc/systemd/resolved.conf.d/viktorbarzin.conf
[Resolve]
DNS=10.0.20.201
Domains=~viktorbarzin.me

The /etc/hosts pins were then removed (verified getent still returns the Traefik IP via DNS, and crictl pull succeeds). On node5/6 the cloud-init global-dns.conf (DNS=8.8.8.8 1.1.1.1) was demoted to FallbackDNS= only — public servers in the global set merge with and race the routing domain. That file's original justification ("Technitium NXDOMAINs forgejo.viktorbarzin.me") was obsolete: the ingress-dns-sync has since added forgejo to the zone — a stale comment that actively pointed new nodes at the hairpin.

Final architecture (same day, round 3 — Viktor: "no customization, everything handled by the DNS infra"): the routing-domain drop-ins were ALSO removed; nodes are now completely stock. Two resolver-side changes replaced them:

1. pfSense Unbound domain override viktorbarzin.me → 10.0.20.201 (forward-zone to Technitium). Every Unbound client on every VLAN gets the internal split-horizon answers with zero per-host config. No DNSSEC complications (zone unsigned), private-IP answers pass, mail's non-Traefik record (→ 10.0.20.1) verified working. Runbook: docs/runbooks/pfsense-unbound.md; on-box backup config.xml.bak-2026-06-10-pre-me-forward.

2. CoreDNS pod carve-out (TF, stacks/technitium): a dedicated viktorbarzin.me:53 server block pins forgejo to Traefik's ClusterIP (interpolated from the live Service — pods cannot reach the ETP=Local LB IP that pfSense now returns) and forwards all other .me names to 8.8.8.8/1.1.1.1, preserving pods' pre-existing public-IP behavior. Replaces the old forgejo rewrite in .:53.

   Addendum (same day, evening): the "pods cannot reach the ETP=Local LB IP" premise was re-tested and is FALSE on k8s 1.34 (kube-proxy short-circuits in-cluster traffic to LB IPs via the cluster path; verified from pods on three non-Traefik nodes). The public-answer carve-out had meanwhile left pods as the only client class still riding the TP-Link NAT loopback, which hard-died 2026-06-09 — 27 non-proxied [External] uptime-kuma monitors dark. Fix: the block now forwards to the Technitium ClusterIP (10.96.0.53) — pods are ordinary internal clients; forgejo pin kept for Technitium-outage resilience. In-cluster [External] monitors now test the internal path for all names; genuine edge-path fidelity belongs to a true external vantage (ha-london).

node5/6 were also re-pointed from link-DNS=Technitium to 10.0.20.1 94.140.14.14 (netplan + qm set --nameserver on PVE VMs 205/206) for fleet parity, and their global-dns.conf was deleted.

Renumber hazard: resolved. A future Traefik LB renumber propagates via the apex A record automatically (drift probe alerts if it doesn't); only the vestigial hosts.toml literal goes stale. Trade-offs: viktorbarzin.me resolution via pfSense depends on in-cluster Technitium (3 replicas) — SERVFAIL during a full cluster outage (services down anyway; bootstrap images pull via the IP-addressed 10.0.20.10 mirrors). Nodes keep 94.140.14.14 as secondary DNS: a resolved failover during a pfSense blip briefly re-exposes public answers — rare, self-healing, accepted.

Verification (final architecture)

• All 7 nodes stock (no pins, no drop-ins); getent hosts forgejo.viktorbarzin.me → 10.0.20.203 via pfSense → Technitium; general resolution intact; crictl pull succeeds end-to-end.
• pfSense: forgejo/immich/vault → apex CNAME → .203; mail → 10.0.20.1 (:993 verified); google.com public; .lan auth-zone unaffected.
• Pods: forgejo → 10.111.111.95 (Traefik ClusterIP), immich → 176.12.22.76 (public, status quo) — verified in-pod after CoreDNS reload.
• tuya-bridge pod Running; /health ok=true; 27/27 devices success=true; 7/7 *_tuya_cloud_up gauges = 1; no tuya-related alerts.

Lessons

• A mirror that can fall back to a broken path is not a fix — it's a latency bomb with the blast delayed until the cache misses.
• Registry token realms are absolute URLs: any "redirect the registry" scheme must also redirect the name, not just the endpoint.
• Before inventing a redirect mechanism, check what the DNS authority already serves: the Technitium split-horizon zone had the correct, auto-maintained answer all along — the clients just weren't asking it.
• Stale config comments are load-bearing: the obsolete "Technitium NXDOMAINs forgejo" comment in cloud-init steered new nodes onto public DNS, recreating the hairpin exposure on every node added after it.
• All 10.0.x legs are now DNS-routed (nodes + devvm via routing domain, pods via CoreDNS rewrite). pfSense Unbound host overrides remain an option for other LAN segments if a non-Technitium client ever needs internal answers (live network device — deliberate, separate change).

Related

• Beads code-2or8 (Tuya Cloud subscription) — verified resolved during this incident: subscription is active again, all gauges green; closed.
• 2026-06-09 tripit ImagePullBackOff — same cause, self-recovered when the hairpin flapped back; the two ScrapeTargetDown[tripit] alerts firing during this investigation were scrapes of Completed cronjob pod endpoints (separate monitoring wart, not this outage).