9806d515dd
## Context (bd code-yiu)
Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local,
pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real
client IP now preserved end-to-end on ports 25/465/587/993, both for
postscreen anti-spam scoring and CrowdSec auth-failure bans.
## This change
### k8s (stacks/mailserver/modules/mailserver/main.tf)
- `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3
alt PROXY-speaking services to master.cf:
- `:2525` postscreen (alt :25)
- `:4465` smtpd (alt :465 SMTPS, wrappermode TLS)
- `:5587` smtpd (alt :587 submission)
All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`.
Mirror stock submission/submissions options (SASL via Dovecot, TLS,
client restrictions, mua_sender_restrictions). chroot=n so the SASL
socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot.
- `dovecot.cf` ConfigMap adds:
```
haproxy_trusted_networks = 10.0.20.0/24
service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } }
```
Stock :993 stays PROXY-free for internal Roundcube/probe clients.
- Container ports: 4 new (4465, 5587, 10993, 2525 already there).
- `mailserver-proxy` NodePort Service now exposes all 4 ports:
25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128
(ETP:Cluster).
### pfSense (scripts/pfsense-haproxy-bootstrap.php)
Rebuilt to declare 4 backend pools (one per NodePort) and 4 production
frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy
`:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`.
Idempotent — re-runs converge on declared state.
### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php)
Flip script: updates `<nat><rule>` entries for mail ports from target
`<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense
HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the
rollback. Both scripts are idempotent.
## What is NOT in this change
- Phase 6 (decommission MetalLB LB path, downgrade mailserver Service
from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do
NOT run until explicit approval.
- Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock
ETP:Local ports — functional backup path + consumed by internal
clients that hit `mailserver.mailserver.svc.cluster.local` (routes
via ClusterIP layer of the LB Service, bypassing ETP).
- Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via
unchanged NAT rule.
## Test Plan
### Automated (verified pre-commit 2026-04-19)
```
# k8s container listens on all 8 ports
$ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \
-- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b'
... all 8 listening ...
# pfSense HAProxy listens on all 5 (production + legacy test)
$ ssh admin@10.0.20.1 'sockstat -l | grep haproxy'
www haproxy 49418 5 tcp4 *:25
www haproxy 49418 6 tcp4 *:2525
www haproxy 49418 10 tcp4 *:465
www haproxy 49418 11 tcp4 *:587
www haproxy 49418 12 tcp4 *:993
# Post-flip: pf rdr rules point at pfSense, not <mailserver>
$ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25'
rdr on vtnet0 ... port = submission -> 10.0.20.1
rdr on vtnet0 ... port = imaps -> 10.0.20.1
rdr on vtnet0 ... port = smtps -> 10.0.20.1
rdr on vtnet0 ... port = 25 -> 10.0.20.1
# 4 HAProxy frontends reachable + SMTP/IMAP banners
$ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly
# Real client IP in maillog for external delivery via Brevo → MX
postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25
postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334
# E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds
$ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver
... Round-trip SUCCESS in 20.3s ...
# Internal Roundcube path unchanged
$ curl -sI https://mail.viktorbarzin.me/ → 302 (Authentik gate intact)
# No mail alerts firing
$ kubectl exec prometheus-server ... /api/v1/alerts | grep Email → (empty)
```
### Rollback
```
scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/
ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php'
```
Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias.
Pre-flip config snapshot also saved at
`/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense.
## Phase roadmap (bd code-yiu)
| Phase | Status |
|---|---|
| 1a | ✅ commit ef75c02f — alt :2525 listener + NodePort |
| 2 | ✅ 2026-04-19 — HAProxy pkg installed on pfSense |
| 3 | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML |
| 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip |
| 6 | ⏸ USER-GATED — MetalLB LB decommission after 48h observation |
68 lines
2.5 KiB
PHP
68 lines
2.5 KiB
PHP
<?php
|
|
// pfSense NAT redirect flip — mail ports 25/465/587/993 from
|
|
// <mailserver> alias (10.0.20.202 MetalLB LB) to pfSense's own HAProxy
|
|
// listener (10.0.20.1). bd code-yiu.
|
|
//
|
|
// THIS IS THE CUTOVER. After this script:
|
|
// Internet → pfSense WAN:{25,465,587,993} → rdr → 10.0.20.1:{...}
|
|
// (pfSense HAProxy) → send-proxy-v2 → k8s-node:{30125..30128} NodePort
|
|
// → kube-proxy → mailserver pod alt listeners (2525/4465/5587/10993)
|
|
// → Postfix/Dovecot parse PROXY v2 → real client IP recovered.
|
|
//
|
|
// Internal clients (Roundcube, email-roundtrip-monitor CronJob) continue
|
|
// using the existing mailserver ClusterIP Service on the stock ports
|
|
// (25/465/587/993) which hit container stock listeners WITHOUT PROXY.
|
|
// No change to internal traffic paths.
|
|
//
|
|
// USAGE
|
|
// scp infra/scripts/pfsense-nat-mailserver-haproxy-flip.php admin@10.0.20.1:/tmp/
|
|
// ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-flip.php'
|
|
//
|
|
// REVERT — run pfsense-nat-mailserver-haproxy-unflip.php (companion script).
|
|
//
|
|
// IDEMPOTENT — re-runs converge. Flips nothing if already pointed at 10.0.20.1.
|
|
|
|
require_once('/etc/inc/config.inc');
|
|
require_once('/etc/inc/filter.inc');
|
|
|
|
global $config;
|
|
parse_config(true);
|
|
|
|
$PORTS_TO_FLIP = ['25', '465', '587', '993'];
|
|
$OLD_TARGET = 'mailserver';
|
|
$NEW_TARGET = '10.0.20.1';
|
|
|
|
$changed = 0;
|
|
foreach ($config['nat']['rule'] as $i => &$r) {
|
|
$iface = $r['interface'] ?? '';
|
|
$lport = $r['local-port'] ?? '';
|
|
$tgt = $r['target'] ?? '';
|
|
|
|
if ($iface !== 'wan') continue;
|
|
if (!in_array($lport, $PORTS_TO_FLIP, true)) continue;
|
|
if ($tgt !== $OLD_TARGET) {
|
|
printf("rule %d (dport=%s) target=%s — not flipping (already %s or unexpected)\n",
|
|
$i, $lport, $tgt, $NEW_TARGET);
|
|
continue;
|
|
}
|
|
|
|
$r['target'] = $NEW_TARGET;
|
|
// Also unset the 'associated-rule-id' linked filter rule target if any —
|
|
// actually pfSense regenerates the associated rule from NAT rule on apply,
|
|
// so leaving associated-rule-id intact is fine.
|
|
$changed++;
|
|
printf("rule %d (dport=%s): target %s → %s\n", $i, $lport, $OLD_TARGET, $NEW_TARGET);
|
|
}
|
|
unset($r);
|
|
|
|
if ($changed === 0) {
|
|
echo "No changes. (Already flipped? Run unflip script to revert.)\n";
|
|
exit(0);
|
|
}
|
|
|
|
write_config("code-yiu: NAT rdr — mail ports {$changed} flipped to HAProxy (10.0.20.1)");
|
|
|
|
// Rebuild pf rules & reload.
|
|
$rc = filter_configure();
|
|
printf("filter_configure rc=%s\n", var_export($rc, true));
|
|
echo "done.\n";
|