viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

forgejo pulls: pin registry name to internal Traefik in node /etc/hosts [ci skip]

Browse source 
tuya-bridge was down 7.5h (ImagePullBackOff on k8s-node3): fresh kubelet
pulls of forgejo.viktorbarzin.me images depended on the intermittently
broken public-IP hairpin. The containerd hosts.toml mirror cannot keep
pulls internal on its own — Traefik 404s its bare-IP requests (no
Host/SNI match) and the registry Bearer realm is an absolute public URL
fetched outside the mirror. Third incident of this class (buildkit
06-04, tripit/devvm 06-09).

Fix: /etc/hosts pin 10.0.20.203 forgejo.viktorbarzin.me on every node —
covers resolve + token + blob legs with correct SNI and valid cert.
Applied live to all 7 nodes; persisted in the cloud-init bootstrap and
the existing-node rollout script. Docs updated (registry bullet, dns.md
hairpin scope + stale .200 literals, runbook) + post-mortem.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-06-10 07:15:24 +00:00
parent eb8695743b
commit b6976ce014
6 changed files with 150 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

16
modules/create-template-vm/k8s-node-containerd-setup.sh
View file

@ -49,7 +49,7 @@ server = "https://ghcr.io"
capabilities = ["pull", "resolve"]
GHCR
# Forgejo OCI registry: prefer in-cluster Traefik LB (10.0.20.200) to
# Forgejo OCI registry: prefer in-cluster Traefik LB (10.0.20.203) to
# avoid hairpin NAT. Traefik serves the *.viktorbarzin.me wildcard so
# SNI verification succeeds. If the mirror is unreachable, fall back to
# public DNS resolution (needs the global DNS fallback set up below).
@ -62,6 +62,20 @@ server = "https://forgejo.viktorbarzin.me"
skip_verify = true
FORGEJO
# /etc/hosts pin — REQUIRED in addition to the hosts.toml mirror. The
# mirror alone cannot make forgejo pulls hairpin-proof for two reasons
# (2026-06-10 tuya-bridge outage, third incident of this class):
# a) Traefik routes by Host/SNI and 404s the mirror's bare-IP requests,
# so containerd always falls back to `server` (public DNS → hairpin).
# b) The registry's Bearer auth realm is the absolute URL
# https://forgejo.viktorbarzin.me/v2/token, which containerd fetches
# verbatim — that leg never goes through the mirror at all.
# Pinning the name to Traefik's LB fixes resolve + token + blob legs with
# correct SNI and a valid cert. If Traefik's LB IP ever changes, update
# this pin together with the hosts.toml IP above.
grep -q forgejo-internal-pin /etc/hosts || \
echo '10.0.20.203 forgejo.viktorbarzin.me # forgejo-internal-pin (managed: setup-forgejo-containerd-mirror.sh)' >> /etc/hosts
# quay.io + registry.k8s.io: include mirror configs that match node4's
# layout (no real pull-through cache today, server line is the direct
# upstream). Keeping these present makes the per-node config uniform and