fd0f4a0365
6d224861 came from a --no-checkout worktree whose empty index made the
commit drop every file except two. This restores 05b50d2b's full tree and
correctly adds stacks/stem95su/gdrive-sync.tf + the service-catalog stem95su
entry. Forward-only (parent=6d224861, no force-push); [ci skip] since the
live infra was never applied from the broken commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
103 lines
6.3 KiB
Markdown
103 lines
6.3 KiB
Markdown
# L4 LoadBalancer IP review & pfSense hygiene — design + decisions
|
|
|
|
**Date:** 2026-06-03
|
|
**Status:** repo changes implemented; pfSense DHCP shrink pending live-change approval
|
|
**Trigger:** "Review the L4 LB IPs we give away, consolidate, and use pfSense Virtual IPs instead of hardcoding IPs in rules."
|
|
|
|
## TL;DR
|
|
|
|
The headline ask — **consolidate to fewer MetalLB IPs** — is a verified dead end. The
|
|
real, worthwhile outcome is a **single source of truth (this doc + the renumber
|
|
checklist in `architecture/networking.md`) plus two stale-reference fixes**. We
|
|
deliberately did **not** reduce the IP count and did **not** do the high-risk pfSense
|
|
mail-VIP surgery.
|
|
|
|
## Current state (verified live, 2026-06-03)
|
|
|
|
MetalLB L2, pool `10.0.20.200-220` (21 IPs, **17 free**). Four in use:
|
|
|
|
| IP | ETP | What | Why dedicated |
|
|
|----|-----|------|---------------|
|
|
| `.200` | Cluster (shared) | ~9 svcs: postgresql-lb (TF state), dolt, coturn, headscale, wireguard, qbittorrent, shadowsocks, torrserver, xray | already maximally consolidated (the 2026-03 "5→1" merge) |
|
|
| `.201` | Local | technitium-dns | real client IP → network-scoped split-horizon |
|
|
| `.202` | Local | windows-kms | real client IP → notifier source labeling |
|
|
| `.203` | Local | traefik | real client IP (CrowdSec) + QUIC/HTTP3 (UDP) |
|
|
|
|
## Why consolidation fails (the core finding)
|
|
|
|
MetalLB L2 only lets multiple `ETP=Local` services **share** an IP if they have
|
|
**identical pod selectors** (so traffic to the single announcing node lands on the
|
|
right pods). Traefik / KMS / Technitium have disjoint selectors and disjoint pods,
|
|
and a shared `ETP=Local` IP announces from one node (stateless `node+VIP` hash) —
|
|
blackholing any service whose pods aren't there. Refs:
|
|
[MetalLB L2 concepts](https://metallb.universe.tf/concepts/layer2/),
|
|
[Usage / IP sharing](https://metallb.universe.tf/usage/),
|
|
[issue #271](https://github.com/metallb/metallb/issues/271).
|
|
|
|
Consequences:
|
|
- **Traefik can never leave a dedicated `ETP=Local` IP** — QUIC's UDP listener needs it; QUIC can't traverse pfSense HAProxy either.
|
|
- The trio is **fewer-IPs XOR client-IP preservation** — not both. The only "both" is making all three DaemonSets (breaks Technitium's primary/secondary AXFR design; burns resources to save 2 of 17 free IPs). Not worth it.
|
|
|
|
**Decision (user, 2026-06-03):** keep all 4 dedicated, preserve client IPs everywhere. No MetalLB changes.
|
|
|
|
## Why a doc registry instead of a `config.tfvars`/Terraform IP variable
|
|
|
|
The cascade risk is **consumers that hardcode another service's IP and get forgotten**
|
|
(the 2026-05-30 Traefik `.200→.203` move broke cloudflared, woodpecker, containerd,
|
|
and the `.lan`+`.me` zones). A Terraform-var single-source was considered and rejected:
|
|
|
|
1. Editing `terragrunt.hcl`/`config.tfvars` triggers the CI "global change → apply ALL
|
|
~37 platform stacks" path (`.woodpecker/default.yml`) — a 37-stack apply for what
|
|
are no-op refactors (rendered IPs unchanged), risking unrelated drift surfacing.
|
|
2. It can't cover the **out-of-band** consumers (cloudflared via CF-API, containerd
|
|
`hosts.toml` on each node) — which were half the 2026-05-30 breakage.
|
|
3. Bootstrap-critical literals (PG state in `scripts/tg`, node DNS) must stay literals
|
|
(DNS chicken-and-egg) regardless.
|
|
|
|
A **documentation registry** (the "LB-IP renumber checklist" in
|
|
`architecture/networking.md`) covers *all* consumers — in-band and OOB — at zero
|
|
apply-risk, and is the complete pre-move checklist. That is the single source of truth.
|
|
|
|
## Changes made (minimal-hygiene scope)
|
|
|
|
1. **`architecture/networking.md`** — rewrote the stale MetalLB section into an accurate
|
|
registry (it had KMS on `.200`, mailserver on a LB IP, "two dedicated" — all wrong)
|
|
+ added the **renumber checklist**.
|
|
2. **woodpecker** (`stacks/woodpecker/main.tf`) — the `forgejo.viktorbarzin.me`
|
|
hostAlias hardcoded the **dead** `10.0.20.200` (Traefik moved to `.203`; `.200:443`
|
|
refuses TLS). Now reads the Traefik **ClusterIP dynamically** (`data
|
|
"kubernetes_service" "traefik"`) so it can't rot on a future renumber and avoids the
|
|
ETP=Local hairpin trap. (Real fix — the next woodpecker apply would otherwise
|
|
re-pin the dead IP and break pipeline creation.)
|
|
3. **monitoring** (`prometheus_chart_values.tpl`) — `ViktorBarzinApexDrift` alert
|
|
summary said "expected 10.0.20.200" (stale post-Traefik-move) → `.203`. Cosmetic
|
|
(alert logic was already correct) but prevents a misleading incident message.
|
|
4. **`backend.tf`** — 72 stale generated copies were tracked in git with a plaintext
|
|
(Vault-rotated, ~expired) PG password + `.200` literal, despite already being in
|
|
`.gitignore`. `git rm --cached` (they regenerate from `PG_CONN_STR`). History scrub
|
|
deferred (creds rotate weekly → low urgency).
|
|
5. **pfSense DHCP range** (`opt1`/K8s VLAN) — `.200-.254` overlaps the MetalLB pool
|
|
`.200-.220` (latent IP-conflict: DHCP could hand out a live LB IP). Plan: shrink to
|
|
start at `.221`. Verified zero leases/statics in the band. **PENDING** — live
|
|
pfSense change, applied separately after explicit approval (live network device).
|
|
|
|
## Explicitly NOT done (rationale)
|
|
|
|
- **No MetalLB IP merging** — infeasible without losing client-IP/QUIC/HA (above).
|
|
- **No mail Virtual IP** — mail binds pfSense's own `10.0.20.1`, the most stable IP in
|
|
the system; the 2026-06-02 incident was a *DNS split-horizon* bug, not an IP move.
|
|
A mail VIP is 4 NAT + 5 filter + HAProxy cutover on the live mail path for marginal
|
|
"identity" benefit. Skipped.
|
|
- **No `nginx`-alias delete / NAT literal→alias** — pfSense rule cosmetics; left for a
|
|
later pfSense-focused pass (would also need the web filter F#2/F#3 `nginx`→`traefik_lb`
|
|
repoint to avoid breaking 80/443).
|
|
- **No Terraform IP variable** — see registry rationale above.
|
|
|
|
## Known latent items (documented, not fixed here)
|
|
|
|
- pfSense web filter rules F#2/F#3 reference `nginx` (.200) while their NAT targets
|
|
`traefik_lb` (.203) — inconsistent but currently passing; fix in a pfSense pass.
|
|
- pfSense NAT 53 hardcodes literal `10.0.20.201` instead of the `technitium_dns` alias.
|
|
- In-cluster `*.viktorbarzin.me` split-horizon still resolves some hosts to the dead
|
|
`.200` (beads `code-yh33`) — the woodpecker hostAlias is the per-app workaround.
|
|
- CrowdSec syslog `remoteserver` doc/config drift (`.200` vs comment `.202`).
|