forgejo pulls: pin registry name to internal Traefik in node /etc/hosts [ci skip]
tuya-bridge was down 7.5h (ImagePullBackOff on k8s-node3): fresh kubelet pulls of forgejo.viktorbarzin.me images depended on the intermittently broken public-IP hairpin. The containerd hosts.toml mirror cannot keep pulls internal on its own — Traefik 404s its bare-IP requests (no Host/SNI match) and the registry Bearer realm is an absolute public URL fetched outside the mirror. Third incident of this class (buildkit 06-04, tripit/devvm 06-09). Fix: /etc/hosts pin 10.0.20.203 forgejo.viktorbarzin.me on every node — covers resolve + token + blob legs with correct SNI and valid cert. Applied live to all 7 nodes; persisted in the cloud-init bootstrap and the existing-node rollout script. Docs updated (registry bullet, dns.md hairpin scope + stale .200 literals, runbook) + post-mortem. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
eb8695743b
commit
b6976ce014
6 changed files with 150 additions and 11 deletions
|
|
@ -38,7 +38,7 @@ Violations cause state drift, which causes future applies to break or silently r
|
|||
- **DNS**: `dns_type = "proxied"` (Cloudflare CDN) or `"non-proxied"` (direct A/AAAA). DNS records are auto-created — no need to edit `config.tfvars`. Smoke-test target: `echo.viktorbarzin.me` (auth=public, header-reflecting backend).
|
||||
- **Anubis PoW challenge** (`modules/kubernetes/anubis_instance/`): per-site reverse proxy that issues a 30-day JWT cookie after a tiny PoW solve. Use for **public, content-bearing sites without app-level auth** (blog, docs, wikis, static landing pages). Pattern: declare `module "anubis" { source = "../../modules/kubernetes/anubis_instance"; name = "X"; namespace = ...; target_url = "http://<backend>.<ns>.svc.cluster.local" }`, then in `ingress_factory` set `service_name = module.anubis.service_name`, `port = module.anubis.service_port`, `anti_ai_scraping = false`. Shared ed25519 key in Vault `secret/viktor` -> `anubis_ed25519_key`; cookie scoped to `viktorbarzin.me` so one solve covers all Anubis-fronted subdomains. **DO NOT put Anubis in front of Git/API/WebDAV/CLI endpoints** — clients without JS can't solve PoW. **Replicas default to 1** because Anubis stores in-flight challenges in process memory; a challenge issued by pod A and solved against pod B errors with `store: key not found` (HTTP 500). Bumping replicas requires wiring a shared Redis store (TODO). For path-level carve-outs (e.g. wrongmove has `/` behind Anubis but `/api` direct, blog has `/net-diag.sh` direct), declare a second `ingress_factory` with `ingress_path = ["/<path>"]` pointing at the bare backend service. Active on: blog (except `/net-diag.sh`), www, kms, travel, f1, cc, json, pb (privatebin), home (homepage), wrongmove (UI only). See `.claude/reference/patterns.md` "Anti-AI Scraping" for full layering.
|
||||
- **Docker images**: Always build for `linux/amd64`. SHA-tag rule is being phased out — see `docs/plans/2026-05-16-auto-upgrade-apps-{design,plan}.md`. New model: CI pushes `:latest` (optionally also `:<8-char-sha>` for traceability), Keel polls and triggers rollouts. Cache-staleness concern from the old rule is resolved at the nginx layer (URL-split — manifests pass through, blobs cached). Until Phase 1 of the migration completes (per the plan), follow the SHA-tag rule for new services to match existing pattern.
|
||||
- **Private registry**: `forgejo.viktorbarzin.me/viktor/<name>` (Forgejo packages, OAuth-style PAT auth). Use `image: forgejo.viktorbarzin.me/viktor/<name>:<tag>` + `imagePullSecrets: [{name: registry-credentials}]`. Kyverno auto-syncs the Secret to all namespaces. Containerd `hosts.toml` on every node redirects to in-cluster Traefik LB `10.0.20.203` (with `skip_verify = true`, since the node dials Traefik by IP but the cert is for `forgejo.viktorbarzin.me`) to avoid hairpin NAT. That redirect covers **kubelet pulls** only — in-cluster pods (notably Woodpecker buildkit build pods pushing images) resolve `forgejo.viktorbarzin.me` via a CoreDNS `rewrite name exact ... traefik.traefik.svc.cluster.local` (Corefile in `stacks/technitium/modules/technitium/main.tf`), since they do NOT use the node containerd mirror; without it, buildkit pushes intermittently timed out on the public-IP hairpin (added 2026-06-04, beads code-yh33). **Was `.200` until 2026-06-01** — Traefik's 2026-05-30 move to its dedicated `.203` left this redirect pointing at the now-dead `.200:443`, silently breaking every *fresh* forgejo pull (cached images kept running, so it stayed hidden until a new image tag was pulled). Redirect source lives in `modules/create-template-vm/k8s-node-containerd-setup.sh` (new nodes) and `scripts/setup-forgejo-containerd-mirror.sh` (existing nodes). Push-side: viktor PAT in Vault `secret/ci/global/forgejo_push_token` (Forgejo container packages are scoped per-user; only the package owner can push, ci-pusher cannot write to viktor/*). Pull-side: cluster-puller PAT in Vault `secret/viktor/forgejo_pull_token`. Retention CronJob (`forgejo-cleanup` in `forgejo` ns, daily 04:00) keeps newest 10 versions + always `:latest` + any buildkit `*cache*` tag (so `--cache-from`/`--cache-to` refs survive retention — added 2026-06-09); **went live (DRY_RUN=false) 2026-06-09** after verifying 0 running images on the delete set — the registry PVC is at its 50Gi autoresize ceiling on the HDD (we did NOT move it to SSD, see beads code-oflt), so live retention is what keeps it from filling. Integrity probed every 15min by `forgejo-integrity-probe` in `monitoring` ns (catalog walk + manifest HEAD on every blob). See `docs/plans/2026-05-07-forgejo-registry-consolidation-{design,plan}.md` for the migration history. Pull-through caches for upstream registries (DockerHub, GHCR, Quay, k8s.gcr, Kyverno) stay on the registry VM at `10.0.20.10` ports 5000/5010/5020/5030/5040 — the old port-5050 R/W private registry was decommissioned 2026-05-07.
|
||||
- **Private registry**: `forgejo.viktorbarzin.me/viktor/<name>` (Forgejo packages, OAuth-style PAT auth). Use `image: forgejo.viktorbarzin.me/viktor/<name>:<tag>` + `imagePullSecrets: [{name: registry-credentials}]`. Kyverno auto-syncs the Secret to all namespaces. **Kubelet pulls** are kept off the hairpin by an `/etc/hosts` pin `10.0.20.203 forgejo.viktorbarzin.me` on every node (marker comment `forgejo-internal-pin`) — the older containerd `hosts.toml` mirror (`[host."https://10.0.20.203"]`, `skip_verify = true`) still exists but is NOT sufficient on its own: Traefik routes by Host/SNI and 404s the mirror's bare-IP requests, and the registry's Bearer auth realm is the absolute `https://forgejo.viktorbarzin.me/v2/token` URL which containerd fetches outside the mirror — so without the pin every fresh pull degrades to public DNS → hairpin → intermittent `dial tcp 176.12.22.76:443: i/o timeout` ImagePullBackOff (tuya-bridge 7.5h outage 2026-06-10, tripit 2026-06-09; see `docs/post-mortems/2026-06-10-tuya-bridge-forgejo-pull-hairpin.md`). In-cluster pods (notably Woodpecker buildkit build pods pushing images) resolve `forgejo.viktorbarzin.me` via a CoreDNS `rewrite name exact ... traefik.traefik.svc.cluster.local` (Corefile in `stacks/technitium/modules/technitium/main.tf`), since they use neither the node pin nor the containerd mirror; without it, buildkit pushes intermittently timed out on the public-IP hairpin (added 2026-06-04, beads code-yh33). **Was `.200` until 2026-06-01** — Traefik's 2026-05-30 move to its dedicated `.203` left the mirror pointing at the now-dead `.200:443`, silently breaking every *fresh* forgejo pull (cached images kept running, so it stayed hidden until a new image tag was pulled); if the Traefik LB IP ever moves again, update BOTH the hosts.toml mirror and the `/etc/hosts` pin. Pin + mirror source lives in `modules/create-template-vm/k8s-node-containerd-setup.sh` (new nodes) and `scripts/setup-forgejo-containerd-mirror.sh` (existing nodes). Push-side: viktor PAT in Vault `secret/ci/global/forgejo_push_token` (Forgejo container packages are scoped per-user; only the package owner can push, ci-pusher cannot write to viktor/*). Pull-side: cluster-puller PAT in Vault `secret/viktor/forgejo_pull_token`. Retention CronJob (`forgejo-cleanup` in `forgejo` ns, daily 04:00) keeps newest 10 versions + always `:latest` + any buildkit `*cache*` tag (so `--cache-from`/`--cache-to` refs survive retention — added 2026-06-09); **went live (DRY_RUN=false) 2026-06-09** after verifying 0 running images on the delete set — the registry PVC is at its 50Gi autoresize ceiling on the HDD (we did NOT move it to SSD, see beads code-oflt), so live retention is what keeps it from filling. Integrity probed every 15min by `forgejo-integrity-probe` in `monitoring` ns (catalog walk + manifest HEAD on every blob). See `docs/plans/2026-05-07-forgejo-registry-consolidation-{design,plan}.md` for the migration history. Pull-through caches for upstream registries (DockerHub, GHCR, Quay, k8s.gcr, Kyverno) stay on the registry VM at `10.0.20.10` ports 5000/5010/5020/5030/5040 — the old port-5050 R/W private registry was decommissioned 2026-05-07.
|
||||
- **LinuxServer.io containers**: `DOCKER_MODS` runs apt-get on every start — bake slow mods into a custom image (`RUN /docker-mods || true` then `ENV DOCKER_MODS=`). Set `NO_CHOWN=true` to skip recursive chown that hangs on NFS mounts.
|
||||
- **Node memory changes**: When changing VM memory on any k8s node, update kubelet `systemReserved`, `kubeReserved`, and eviction thresholds accordingly. Config: `/var/lib/kubelet/config.yaml`. Template: `stacks/infra/main.tf`. Current values: systemReserved=512Mi, kubeReserved=512Mi, evictionHard=500Mi, evictionSoft=1Gi.
|
||||
- **Node OS disk tuning** (in `stacks/infra/main.tf`): kubelet `imageGCHighThresholdPercent=70` (was 85), `imageGCLowThresholdPercent=60` (was 80), ext4 `commit=60` in fstab (was default 5s), journald `SystemMaxUse=200M` + `MaxRetentionSec=3day`.
|
||||
|
|
|
|||
|
|
@ -258,16 +258,21 @@ The TP-Link AP (dumb AP on 192.168.1.x) does not support hairpin NAT. LAN client
|
|||
Technitium's **Split Horizon AddressTranslation** app post-processes DNS responses for 192.168.1.0/24 clients, translating the public IP to the internal Traefik LB IP:
|
||||
|
||||
```
|
||||
176.12.22.76 → 10.0.20.200
|
||||
176.12.22.76 → 10.0.20.203
|
||||
```
|
||||
|
||||
(Was `10.0.20.200` until Traefik's 2026-05-30 move to its dedicated `.203` LB IP.)
|
||||
|
||||
**DNS Rebinding Protection** has `viktorbarzin.me` in `privateDomains` to allow the translated private IP without being stripped as a rebinding attack.
|
||||
|
||||
### Scope
|
||||
|
||||
- **Affected**: Non-proxied domains (ha-sofia, immich, headscale, calibre, vaultwarden, etc.) for 192.168.1.x clients
|
||||
- **Not affected**: Cloudflare-proxied domains (resolve to Cloudflare edge IPs, no translation needed)
|
||||
- **Not affected**: 10.0.x.x and K8s clients (reach public IP via pfSense outbound NAT normally)
|
||||
- **Not affected**: 10.0.x.x and K8s clients — these resolve non-proxied domains to the public IP and rely on pfSense NAT reflection, which is **intermittently broken** (observed i/o timeouts to `176.12.22.76:443` from k8s nodes and the devvm, 2026-06-04 → 2026-06-10). Hairpin-sensitive paths on this network get explicit per-leg fixes instead:
|
||||
- **kubelet image pulls of `forgejo.viktorbarzin.me`**: `/etc/hosts` pin `10.0.20.203 forgejo.viktorbarzin.me` on every k8s node (marker `forgejo-internal-pin`; deployed via `modules/create-template-vm/k8s-node-containerd-setup.sh` for new nodes, `scripts/setup-forgejo-containerd-mirror.sh` rollout for existing ones). The containerd hosts.toml mirror alone is insufficient — Traefik 404s its bare-IP requests (no Host/SNI match) and the registry's Bearer auth realm is an absolute public URL fetched outside the mirror. Root cause of the 2026-06-10 tuya-bridge outage (`docs/post-mortems/2026-06-10-tuya-bridge-forgejo-pull-hairpin.md`).
|
||||
- **in-cluster pods → forgejo**: CoreDNS `rewrite name exact forgejo.viktorbarzin.me traefik.traefik.svc.cluster.local` (2026-06-04, beads code-yh33).
|
||||
- **devvm git → forgejo**: still exposed to the hairpin (manual `/etc/hosts` pin workaround when it flares).
|
||||
|
||||
Config is synced to all 3 Technitium instances by CronJob `technitium-split-horizon-sync` (every 6h).
|
||||
|
||||
|
|
@ -462,7 +467,7 @@ post-processing does NOT run for 192.168.1.x clients anymore. Non-proxied
|
|||
services break hairpin on LAN clients again. Options:
|
||||
|
||||
1. **Switch service to proxied Cloudflare** (preferred) — set `dns_type = "proxied"` in the `ingress_factory` module call; DNS now resolves to Cloudflare edge, hairpin-independent.
|
||||
2. **Add a local-data override on pfSense Unbound** — under `Services → DNS Resolver → Host Overrides`, set `<service>.viktorbarzin.me → 10.0.20.200` (Traefik LB IP). This is equivalent to what Split Horizon did, applied at the resolver.
|
||||
2. **Add a local-data override on pfSense Unbound** — under `Services → DNS Resolver → Host Overrides`, set `<service>.viktorbarzin.me → 10.0.20.203` (Traefik LB IP). This is equivalent to what Split Horizon did, applied at the resolver.
|
||||
3. **Revert to prior NAT rdr + Technitium Split Horizon** — documented in `docs/runbooks/pfsense-unbound.md` rollback section.
|
||||
|
||||
K8s-side Split Horizon is still configured and applies when `*.viktorbarzin.me` queries DO reach Technitium (e.g., from pods that query via CoreDNS → Technitium forwarding for `.viktorbarzin.me` via pfSense). Verify Technitium split-horizon app:
|
||||
|
|
@ -470,7 +475,7 @@ K8s-side Split Horizon is still configured and applies when `*.viktorbarzin.me`
|
|||
1. Verify Split Horizon app is installed on all instances
|
||||
2. Check CronJob status: `kubectl get cronjob -n technitium technitium-split-horizon-sync`
|
||||
3. Run the job manually: `kubectl create job --from=cronjob/technitium-split-horizon-sync test-sh -n technitium`
|
||||
4. Test: `dig @10.0.20.201 immich.viktorbarzin.me` — should return 10.0.20.200 for 192.168.1.x source
|
||||
4. Test: `dig @10.0.20.201 immich.viktorbarzin.me` — should return 10.0.20.203 for 192.168.1.x source
|
||||
|
||||
### Zone Not Replicating to Secondary/Tertiary
|
||||
|
||||
|
|
|
|||
105
docs/post-mortems/2026-06-10-tuya-bridge-forgejo-pull-hairpin.md
Normal file
105
docs/post-mortems/2026-06-10-tuya-bridge-forgejo-pull-hairpin.md
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
# 2026-06-10 — tuya-bridge down 7.5h: forgejo image pulls ride the public-IP hairpin
|
||||
|
||||
## Impact
|
||||
|
||||
- `tuya-bridge` (Flask/tinytuya bridge feeding HA-Sofia's ATS, fuse-main,
|
||||
fuse-garage and 4 thermostat REST sensors) unavailable ~02:15–09:50 EEST.
|
||||
HA REST sensors 503'd; the official-tuya integration devices were
|
||||
unaffected (hybrid architecture limited the blast radius to the 3 power
|
||||
devices' advanced telemetry + thermostats extras).
|
||||
- Third incident from the same root cause class:
|
||||
Woodpecker buildkit pushes (2026-06-04, code-yh33), tripit
|
||||
ImagePullBackOff on node2/node3 + devvm git timeouts (2026-06-09),
|
||||
tuya-bridge (this one).
|
||||
|
||||
## Timeline (EEST)
|
||||
|
||||
- **02:15** — tuya-bridge pod rescheduled onto `k8s-node3` (its previous
|
||||
node5/6-era home was rebuilt 14d ago; the forgejo-path image was never
|
||||
cached on node3 — only stale `docker.io/*` copies). Kubelet must pull
|
||||
`forgejo.viktorbarzin.me/viktor/tuya_bridge:3216c87a`.
|
||||
- **02:15→09:30** — 51 consecutive pull failures:
|
||||
`dial tcp 176.12.22.76:443: i/o timeout` → ImagePullBackOff. HA shows
|
||||
503s (emo observed at 02:20).
|
||||
- **09:40** — investigation: forgejo healthy via internal Traefik
|
||||
(`10.0.20.203`), manifest exists; node3's hosts.toml mirror present and
|
||||
correct; bare-IP request to the mirror returns **404 from Traefik**;
|
||||
registry auth realm is the **absolute** public URL.
|
||||
- **09:48** — `/etc/hosts` pin `10.0.20.203 forgejo.viktorbarzin.me` added
|
||||
on node3; `crictl pull` succeeds immediately; pod replaced → Running;
|
||||
`/health` ok; all 27 device `getstatus()` calls succeed; all 7
|
||||
`*_tuya_cloud_up` Prometheus gauges = 1.
|
||||
- **10:05** — pin rolled to all 7 nodes; provisioning scripts + docs updated.
|
||||
|
||||
## Root cause
|
||||
|
||||
Fresh kubelet pulls of `forgejo.viktorbarzin.me` images depend on pfSense
|
||||
NAT reflection of the public IP `176.12.22.76`, which is intermittently
|
||||
broken from the `10.0.20.0/24` network. The containerd
|
||||
`certs.d/.../hosts.toml` mirror that was *believed* to keep pulls internal
|
||||
cannot do so, for two independent reasons:
|
||||
|
||||
1. **Traefik routes by Host/SNI.** The mirror entry
|
||||
`[host."https://10.0.20.203"]` makes containerd dial the bare IP (no
|
||||
SNI, `Host: 10.0.20.203`) — no Traefik router matches → **404** → con-
|
||||
tainerd treats the mirror as a miss and falls back to
|
||||
`server = "https://forgejo.viktorbarzin.me"` → public DNS → hairpin.
|
||||
2. **The Bearer auth realm is absolute.** `/v2/` challenges with
|
||||
`realm="https://forgejo.viktorbarzin.me/v2/token"`; containerd fetches
|
||||
that URL verbatim — this leg never goes through the mirror at all.
|
||||
|
||||
So every fresh pull silently depended on hairpin luck. Cached images masked
|
||||
the problem; it only fired when a pod landed on a node without the image
|
||||
(node rebuilds, new nodes, evictions, new tags).
|
||||
|
||||
Why DNS-side fixes don't reach this path: nodes resolve via systemd-resolved
|
||||
→ pfSense (10.0.20.1) + public fallback (94.140.14.14), so Technitium
|
||||
split-horizon (scoped to `192.168.1.0/24` clients) never applies; the
|
||||
CoreDNS forgejo rewrite (2026-06-04) covers pods only, not kubelet.
|
||||
|
||||
## Fix
|
||||
|
||||
`/etc/hosts` pin on every k8s node (hot, no drain, no containerd restart):
|
||||
|
||||
```
|
||||
10.0.20.203 forgejo.viktorbarzin.me # forgejo-internal-pin (managed: setup-forgejo-containerd-mirror.sh)
|
||||
```
|
||||
|
||||
Go's resolver (containerd) consults `/etc/hosts` first, so resolve + token
|
||||
+ blob legs all go to internal Traefik with correct SNI and a valid
|
||||
wildcard cert (no `skip_verify` needed on this path). Applied live to all
|
||||
7 nodes; persisted in `modules/create-template-vm/k8s-node-containerd-setup.sh`
|
||||
(new nodes) and `scripts/setup-forgejo-containerd-mirror.sh` (existing-node
|
||||
rollout). hosts.toml mirror left in place (harmless, uniform config).
|
||||
|
||||
**Renumber hazard:** the pin hardcodes Traefik's LB IP, same as the
|
||||
hosts.toml mirror and the 5 literals broken by the 2026-05-30 `.200→.203`
|
||||
move. Any future Traefik LB renumber must update both (grep nodes for
|
||||
`forgejo-internal-pin`).
|
||||
|
||||
## Verification
|
||||
|
||||
- `getent hosts forgejo.viktorbarzin.me` → `10.0.20.203` on all 7 nodes;
|
||||
`curl https://forgejo.viktorbarzin.me/v2/` → 401 (internal route, valid TLS).
|
||||
- tuya-bridge pod Running; `/health` `ok=true`; 27/27 devices
|
||||
`success=true`; 7/7 `*_tuya_cloud_up` gauges = 1; no tuya-related alerts.
|
||||
|
||||
## Lessons
|
||||
|
||||
- A mirror that *can* fall back to a broken path is not a fix — it's a
|
||||
latency bomb with the blast delayed until the cache misses.
|
||||
- Registry token realms are absolute URLs: any "redirect the registry"
|
||||
scheme must also redirect the *name*, not just the endpoint.
|
||||
- The remaining hairpin-exposed leg is **devvm git** (manual `/etc/hosts`
|
||||
workaround documented in memory); a durable LAN-wide fix would need
|
||||
pfSense Unbound host overrides (live network device — deliberate,
|
||||
separate change).
|
||||
|
||||
## Related
|
||||
|
||||
- Beads `code-2or8` (Tuya Cloud subscription) — verified resolved during
|
||||
this incident: subscription is active again, all gauges green; closed.
|
||||
- 2026-06-09 tripit ImagePullBackOff — same cause, self-recovered when the
|
||||
hairpin flapped back; the two `ScrapeTargetDown[tripit]` alerts firing
|
||||
during this investigation were scrapes of *Completed* cronjob pod
|
||||
endpoints (separate monitoring wart, not this outage).
|
||||
|
|
@ -119,8 +119,11 @@ cd infra/stacks/kyverno && scripts/tg apply
|
|||
cd infra/stacks/monitoring && scripts/tg apply
|
||||
cd infra/stacks/forgejo && scripts/tg apply
|
||||
|
||||
# Containerd hosts.toml on each existing k8s node — VM cloud-init
|
||||
# only fires on first boot.
|
||||
# Containerd hosts.toml + /etc/hosts pin on each existing k8s node — VM
|
||||
# cloud-init only fires on first boot. The /etc/hosts pin
|
||||
# (10.0.20.203 forgejo.viktorbarzin.me) is what makes pulls hairpin-proof:
|
||||
# the hosts.toml mirror alone falls back to public DNS (Traefik 404s its
|
||||
# bare-IP requests, and the registry auth realm is an absolute public URL).
|
||||
infra/scripts/setup-forgejo-containerd-mirror.sh
|
||||
```
|
||||
|
||||
|
|
@ -135,7 +138,9 @@ docker pull alpine:3.20
|
|||
docker tag alpine:3.20 forgejo.viktorbarzin.me/viktor/smoketest:1
|
||||
docker push forgejo.viktorbarzin.me/viktor/smoketest:1
|
||||
|
||||
# Pull from a k8s node.
|
||||
# Per-node pull path: pin present + name resolves internally + pull works.
|
||||
ssh wizard@<node> 'grep forgejo-internal-pin /etc/hosts && getent hosts forgejo.viktorbarzin.me'
|
||||
# Expect: 10.0.20.203 forgejo.viktorbarzin.me
|
||||
ssh wizard@<node> sudo crictl pull forgejo.viktorbarzin.me/viktor/smoketest:1
|
||||
|
||||
# Confirm the cluster-wide Secret was synced into a fresh namespace.
|
||||
|
|
|
|||
|
|
@ -49,7 +49,7 @@ server = "https://ghcr.io"
|
|||
capabilities = ["pull", "resolve"]
|
||||
GHCR
|
||||
|
||||
# Forgejo OCI registry: prefer in-cluster Traefik LB (10.0.20.200) to
|
||||
# Forgejo OCI registry: prefer in-cluster Traefik LB (10.0.20.203) to
|
||||
# avoid hairpin NAT. Traefik serves the *.viktorbarzin.me wildcard so
|
||||
# SNI verification succeeds. If the mirror is unreachable, fall back to
|
||||
# public DNS resolution (needs the global DNS fallback set up below).
|
||||
|
|
@ -62,6 +62,20 @@ server = "https://forgejo.viktorbarzin.me"
|
|||
skip_verify = true
|
||||
FORGEJO
|
||||
|
||||
# /etc/hosts pin — REQUIRED in addition to the hosts.toml mirror. The
|
||||
# mirror alone cannot make forgejo pulls hairpin-proof for two reasons
|
||||
# (2026-06-10 tuya-bridge outage, third incident of this class):
|
||||
# a) Traefik routes by Host/SNI and 404s the mirror's bare-IP requests,
|
||||
# so containerd always falls back to `server` (public DNS → hairpin).
|
||||
# b) The registry's Bearer auth realm is the absolute URL
|
||||
# https://forgejo.viktorbarzin.me/v2/token, which containerd fetches
|
||||
# verbatim — that leg never goes through the mirror at all.
|
||||
# Pinning the name to Traefik's LB fixes resolve + token + blob legs with
|
||||
# correct SNI and a valid cert. If Traefik's LB IP ever changes, update
|
||||
# this pin together with the hosts.toml IP above.
|
||||
grep -q forgejo-internal-pin /etc/hosts || \
|
||||
echo '10.0.20.203 forgejo.viktorbarzin.me # forgejo-internal-pin (managed: setup-forgejo-containerd-mirror.sh)' >> /etc/hosts
|
||||
|
||||
# quay.io + registry.k8s.io: include mirror configs that match node4's
|
||||
# layout (no real pull-through cache today, server line is the direct
|
||||
# upstream). Keeping these present makes the per-node config uniform and
|
||||
|
|
|
|||
|
|
@ -1,11 +1,19 @@
|
|||
#!/usr/bin/env bash
|
||||
# One-shot deployment of the forgejo.viktorbarzin.me containerd hosts.toml
|
||||
# entry across every k8s node. Cloud-init only fires on VM provision, so
|
||||
# existing nodes need this manual rollout.
|
||||
# entry + /etc/hosts pin across every k8s node. Cloud-init only fires on VM
|
||||
# provision, so existing nodes need this manual rollout.
|
||||
#
|
||||
# The /etc/hosts pin (forgejo.viktorbarzin.me -> Traefik LB) is what actually
|
||||
# makes pulls hairpin-proof: Traefik 404s the mirror's bare-IP requests (no
|
||||
# Host/SNI match) and the registry's Bearer auth realm is the absolute public
|
||||
# URL, so the hosts.toml mirror alone always degrades to the flaky public-IP
|
||||
# hairpin (2026-06-10 tuya-bridge outage; see
|
||||
# docs/post-mortems/2026-06-10-tuya-bridge-forgejo-pull-hairpin.md).
|
||||
#
|
||||
# What it does, per node:
|
||||
# 1. drain (ignore-daemonsets, delete-emptydir-data)
|
||||
# 2. ssh in: mkdir + write /etc/containerd/certs.d/forgejo.viktorbarzin.me/hosts.toml
|
||||
# + append the forgejo /etc/hosts pin
|
||||
# 3. systemctl restart containerd
|
||||
# 4. uncordon
|
||||
#
|
||||
|
|
@ -42,6 +50,8 @@ mkdir -p "$CERTS_DIR"
|
|||
cat > "$CERTS_DIR/hosts.toml" <<'TOML'
|
||||
$HOSTS_TOML
|
||||
TOML
|
||||
grep -q forgejo-internal-pin /etc/hosts || \
|
||||
echo '10.0.20.203 forgejo.viktorbarzin.me # forgejo-internal-pin (managed: setup-forgejo-containerd-mirror.sh)' >> /etc/hosts
|
||||
systemctl restart containerd
|
||||
EOF
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue