fix: restore tree dropped by 6d224861; land stem95su gdrive-sync (10m) [ci skip]
6d224861 came from a --no-checkout worktree whose empty index made the
commit drop every file except two. This restores 05b50d2b's full tree and
correctly adds stacks/stem95su/gdrive-sync.tf + the service-catalog stem95su
entry. Forward-only (parent=6d224861, no force-push); [ci skip] since the
live infra was never applied from the broken commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>This commit is contained in:
Viktor Barzin
parent
6d224861c4
commit
fd0f4a0365
1166 changed files with 358546 additions and 0 deletions
222
docs/runbooks/mailserver-pfsense-haproxy.md
Normal file
222
docs/runbooks/mailserver-pfsense-haproxy.md
Normal file
|
|
@ -0,0 +1,222 @@
|
|||
# pfSense HAProxy for Mailserver — Runbook
|
||||
|
||||
Last updated: 2026-04-19 (Phase 6 complete)
|
||||
|
||||
## What & why
|
||||
|
||||
External mail traffic (SMTP/IMAP) requires **real client IP visibility** for
|
||||
CrowdSec + Postfix rate-limiting. MetalLB cannot inject PROXY-protocol
|
||||
headers (see [`mailserver-proxy-protocol.md`](./mailserver-proxy-protocol.md)),
|
||||
so pfSense runs a small HAProxy that:
|
||||
|
||||
1. Listens on the pfSense VLAN20 IP (`10.0.20.1`) on all 4 mail ports,
|
||||
2. Forwards each connection to a k8s node's NodePort with `send-proxy-v2`,
|
||||
3. Injects PROXY v2 framing so Postfix/Dovecot see the original client IP,
|
||||
4. TCP-checks every k8s worker via dedicated **non-PROXY healthcheck NodePorts**
|
||||
(30145/30146/30147 → pod stock 25/465/587 listeners, no PROXY required).
|
||||
This split path avoids the `smtpd_peer_hostaddr_to_sockaddr` fatal that
|
||||
used to fire on every PROXY-aware health probe and throttled real client
|
||||
connections.
|
||||
|
||||
Corresponding k8s-side setup (`stacks/mailserver/modules/mailserver/`):
|
||||
|
||||
- ConfigMap `mailserver-user-patches` → `user-patches.sh` appends 3 alt
|
||||
`master.cf` services to Postfix:
|
||||
- `:2525` postscreen (alt :25) with `postscreen_upstream_proxy_protocol=haproxy`
|
||||
- `:4465` smtpd (alt :465 SMTPS) with `smtpd_upstream_proxy_protocol=haproxy`
|
||||
- `:5587` smtpd (alt :587 submission) with `smtpd_upstream_proxy_protocol=haproxy`
|
||||
- ConfigMap `mailserver.config` adds Dovecot `inet_listener imaps_proxy` on
|
||||
port 10993 with `haproxy = yes` and `haproxy_trusted_networks = 10.0.20.0/24`.
|
||||
- Service `mailserver-proxy` (NodePort, ETP:Cluster) — 4 PROXY data ports +
|
||||
3 non-PROXY healthcheck ports:
|
||||
- Data (PROXY v2):
|
||||
- `port 25 → targetPort 2525 → nodePort 30125`
|
||||
- `port 465 → targetPort 4465 → nodePort 30126`
|
||||
- `port 587 → targetPort 5587 → nodePort 30127`
|
||||
- `port 993 → targetPort 10993 → nodePort 30128`
|
||||
- Healthcheck (no PROXY, stock SMTP/SMTPS/Submission listeners):
|
||||
- `port 2500 → targetPort 25 → nodePort 30145` (smtp-check)
|
||||
- `port 4650 → targetPort 465 → nodePort 30146` (smtps-check)
|
||||
- `port 5870 → targetPort 587 → nodePort 30147` (sub-check)
|
||||
- Service `mailserver` (ClusterIP) — unchanged stock ports 25/465/587/993
|
||||
for intra-cluster clients (Roundcube pod, `email-roundtrip-monitor`
|
||||
CronJob, book-search). These listeners are PROXY-free.
|
||||
|
||||
bd: `code-yiu`.
|
||||
|
||||
## Steady-state architecture
|
||||
|
||||
```
|
||||
External mail (WAN) path — PROXY v2
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Client (real IP) │
|
||||
│ │ SMTP/SMTPS/Sub/IMAPS │
|
||||
│ ▼ │
|
||||
│ pfSense WAN:{25,465,587,993} │
|
||||
│ │ NAT rdr → 10.0.20.1:{same} │
|
||||
│ ▼ │
|
||||
│ pfSense HAProxy (mode tcp, 4 frontends, 4 backend pools) │
|
||||
│ │ data: send-proxy-v2 → :{30125..30128} (PROXY-aware pod) │
|
||||
│ │ health: TCP-check → :{30145..30147} (no-PROXY pod) │
|
||||
│ │ inter 5000 │
|
||||
│ ▼ │
|
||||
│ k8s-node<1-4>:{30125..30128} ← any node (ETP:Cluster) │
|
||||
│ │ kube-proxy SNAT (source IP lost on the wire) │
|
||||
│ ▼ │
|
||||
│ mailserver pod :{2525,4465,5587,10993} │
|
||||
│ │ postscreen / smtpd / Dovecot parse PROXY v2 header │
|
||||
│ │ → real client IP recovered despite kube-proxy SNAT │
|
||||
│ ▼ │
|
||||
│ CrowdSec + Postfix / Dovecot see the true source IP ✓ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Intra-cluster path — no PROXY
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Roundcube pod / email-roundtrip-monitor CronJob │
|
||||
│ │ SMTP/IMAP │
|
||||
│ ▼ │
|
||||
│ mailserver.mailserver.svc.cluster.local:{25,465,587,993} │
|
||||
│ │ ClusterIP — bypasses LoadBalancer/NodePort layer entirely │
|
||||
│ ▼ │
|
||||
│ mailserver pod stock :{25,465,587,993} (PROXY-free) │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Validation
|
||||
|
||||
```sh
|
||||
# All HAProxy frontends listening
|
||||
ssh admin@10.0.20.1 'sockstat -l | grep haproxy'
|
||||
# Expect: *:25, *:465, *:587, *:993, *:2525 (test port)
|
||||
|
||||
# All backend pools healthy
|
||||
ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \
|
||||
| awk 'NR>1 {print $3, $4, $6}'
|
||||
# srv_op_state 2 = UP, 0 = DOWN
|
||||
|
||||
# Container listens on all 8 ports
|
||||
kubectl exec -n mailserver -c docker-mailserver deployment/mailserver -- \
|
||||
ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b'
|
||||
|
||||
# pf rdr points at pfSense (10.0.20.1), not <mailserver> alias
|
||||
ssh admin@10.0.20.1 'pfctl -sn' | grep -E 'port = (25|submission|imaps|smtps)'
|
||||
|
||||
# E2E probe — Brevo → external MX :25 → IMAP fetch
|
||||
kubectl create job --from=cronjob/email-roundtrip-monitor probe-test -n mailserver
|
||||
kubectl wait --for=condition=complete --timeout=90s job/probe-test -n mailserver
|
||||
kubectl logs job/probe-test -n mailserver | grep SUCCESS
|
||||
kubectl delete job probe-test -n mailserver
|
||||
|
||||
# Real client IP in maillog post-delivery
|
||||
kubectl logs -c docker-mailserver deployment/mailserver -n mailserver \
|
||||
| grep 'smtpd-proxy25.*CONNECT from' | tail -5
|
||||
# Expect external source IPs (e.g., Brevo 77.32.148.x), NOT 10.0.20.x
|
||||
```
|
||||
|
||||
## Bootstrap / restore from scratch
|
||||
|
||||
pfSense HAProxy config lives in `/cf/conf/config.xml` under
|
||||
`<installedpackages><haproxy>`. That file is scp'd nightly to
|
||||
`/mnt/backup/pfsense/config-YYYYMMDD.xml` by `scripts/daily-backup.sh`, then
|
||||
synced to Synology. To rebuild from source of truth (git):
|
||||
|
||||
```sh
|
||||
scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/
|
||||
ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'
|
||||
```
|
||||
|
||||
The script is idempotent — re-runs reset the mailserver frontends + backends
|
||||
to the declared state.
|
||||
|
||||
Expected output:
|
||||
```
|
||||
haproxy_check_and_run rc=OK
|
||||
```
|
||||
|
||||
## Operations
|
||||
|
||||
### Change backend k8s node IPs / NodePorts
|
||||
|
||||
Edit `infra/scripts/pfsense-haproxy-bootstrap.php` — `$NODES` array + the
|
||||
`build_pool()` port arguments. Re-run the bootstrap command above. Don't
|
||||
hand-edit `/var/etc/haproxy/haproxy.cfg` — it is regenerated from XML on
|
||||
every apply.
|
||||
|
||||
### Check health of backends
|
||||
|
||||
```sh
|
||||
ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio"
|
||||
```
|
||||
`srv_op_state=2` means UP, `0` means DOWN.
|
||||
|
||||
### View live HAProxy stats (WebUI)
|
||||
|
||||
`https://pfsense.viktorbarzin.me` → Services → HAProxy → Stats.
|
||||
|
||||
### Reload after config.xml edit
|
||||
|
||||
```sh
|
||||
ssh admin@10.0.20.1 'pfSsh.php playback svc restart haproxy'
|
||||
```
|
||||
|
||||
### Rollback (flip NAT back to MetalLB, post-Phase-6 only partial)
|
||||
|
||||
There is no Phase-6 rollback one-liner. Phase 6 removed the MetalLB
|
||||
LoadBalancer 10.0.20.202 entirely, so un-flipping NAT now would send
|
||||
traffic to a dead alias. To regress:
|
||||
|
||||
1. Re-add `metallb.io/loadBalancerIPs = "10.0.20.202"` + `type = "LoadBalancer"`
|
||||
+ `external_traffic_policy = "Local"` to `kubernetes_service.mailserver`,
|
||||
apply.
|
||||
2. Re-add the `mailserver` host alias in pfSense pointing at 10.0.20.202
|
||||
(Firewall → Aliases → Hosts).
|
||||
3. Run `infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php` on pfSense.
|
||||
|
||||
For rollback of just the NAT (Phase 4) without touching the Service, only
|
||||
the third step is needed — but only meaningful BEFORE Phase 6.
|
||||
|
||||
### Restore from backup
|
||||
|
||||
pfSense config backup is a plain XML file:
|
||||
```
|
||||
/mnt/backup/pfsense/config-YYYYMMDD.xml # sda host copy (1.1TB RAID1)
|
||||
/volume1/Backup/Viki/pve-backup/pfsense/... # Synology offsite
|
||||
```
|
||||
|
||||
Full restore: pfSense WebUI → Diagnostics → Backup & Restore → Upload that
|
||||
`config.xml`. The `<installedpackages><haproxy>` section is included.
|
||||
|
||||
## Phase history (bd code-yiu)
|
||||
|
||||
| Phase | Status | Description |
|
||||
|---|---|---|
|
||||
| 1a | ✅ commit `ef75c02f` | k8s alt :2525 listener + NodePort Service |
|
||||
| 2 | ✅ 2026-04-19 | pfSense HAProxy pkg installed (`pfSense-pkg-haproxy-devel-0.63_2`, HAProxy 2.9-dev6) |
|
||||
| 3 | ✅ commit `ba697b02` | HAProxy config persisted in pfSense XML (bootstrap script + this runbook) |
|
||||
| 4+5| ✅ commit `9806d515` | 4-port alt listeners + HAProxy frontends for 25/465/587/993 + NAT flip |
|
||||
| 6 | ✅ this commit | Mailserver Service downgraded LoadBalancer → ClusterIP; `10.0.20.202` released back to MetalLB pool; orphan `mailserver` pfSense alias removed; monitors retargeted |
|
||||
|
||||
## Known warts
|
||||
|
||||
- ~~HAProxy TCP health-check with `send-proxy-v2` generates `getpeername:
|
||||
Transport endpoint not connected` warnings on postscreen every check cycle.~~
|
||||
**Resolved 2026-05-05**: dedicated non-PROXY healthcheck NodePorts
|
||||
(30145/30146/30147 → stock pod 25/465/587) added; HAProxy now checks
|
||||
those, eliminating both the `getpeername` postscreen warnings and the
|
||||
`smtpd_peer_hostaddr_to_sockaddr: ... Servname not supported` fatals
|
||||
that were throttling smtpd respawns and causing ~50% client timeouts on
|
||||
the public 587 path. `inter` dropped 120000 → 5000 (fast failover, no
|
||||
log-spam concern). `option smtpchk` was tried but flapped against
|
||||
postscreen (multi-line greet + DNSBL silence + anti-pre-greet detection
|
||||
trip HAProxy's parser → L7RSP). Plain TCP check on the no-PROXY ports
|
||||
is sufficient.
|
||||
- Frontend binds on all pfSense interfaces (`bind :25` instead of
|
||||
`10.0.20.1:25`). `<extaddr>` is set in XML but pfSense templates it
|
||||
port-only. Low concern in practice because WAN firewall rules plus the
|
||||
NAT rdr gate external access; internal VLAN clients SHOULD be able to
|
||||
reach HAProxy on any pfSense-local IP.
|
||||
- k8s-node5 doesn't exist — cluster has master + 4 workers. Backend pool
|
||||
capped at 4 servers.
|
||||
- Postscreen still logs `improper command pipelining` for legitimate
|
||||
clients that send `EHLO\r\nQUIT\r\n` as a single TCP write. This is
|
||||
unchanged pre/post-migration — postscreen's anti-bot heuristic.
|
||||
Loading…
Add table
Add a link
Reference in a new issue