viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip]

Browse source 
## Context (bd code-yiu)

Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local,
pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real
client IP now preserved end-to-end on ports 25/465/587/993, both for
postscreen anti-spam scoring and CrowdSec auth-failure bans.

## This change

### k8s (stacks/mailserver/modules/mailserver/main.tf)

- `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3
  alt PROXY-speaking services to master.cf:
  - `:2525` postscreen (alt :25)
  - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS)
  - `:5587` smtpd (alt :587 submission)
  All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`.
  Mirror stock submission/submissions options (SASL via Dovecot, TLS,
  client restrictions, mua_sender_restrictions). chroot=n so the SASL
  socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot.
- `dovecot.cf` ConfigMap adds:
  ```
  haproxy_trusted_networks = 10.0.20.0/24
  service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } }
  ```
  Stock :993 stays PROXY-free for internal Roundcube/probe clients.
- Container ports: 4 new (4465, 5587, 10993, 2525 already there).
- `mailserver-proxy` NodePort Service now exposes all 4 ports:
  25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128
  (ETP:Cluster).

### pfSense (scripts/pfsense-haproxy-bootstrap.php)

Rebuilt to declare 4 backend pools (one per NodePort) and 4 production
frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy
`:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`.
Idempotent — re-runs converge on declared state.

### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php)

Flip script: updates `<nat><rule>` entries for mail ports from target
`<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense
HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the
rollback. Both scripts are idempotent.

## What is NOT in this change

- Phase 6 (decommission MetalLB LB path, downgrade mailserver Service
  from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do
  NOT run until explicit approval.
- Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock
  ETP:Local ports — functional backup path + consumed by internal
  clients that hit `mailserver.mailserver.svc.cluster.local` (routes
  via ClusterIP layer of the LB Service, bypassing ETP).
- Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via
  unchanged NAT rule.

## Test Plan

### Automated (verified pre-commit 2026-04-19)
```
# k8s container listens on all 8 ports
$ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \
    -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b'
... all 8 listening ...

# pfSense HAProxy listens on all 5 (production + legacy test)
$ ssh admin@10.0.20.1 'sockstat -l | grep haproxy'
www  haproxy  49418  5   tcp4  *:25
www  haproxy  49418  6   tcp4  *:2525
www  haproxy  49418  10  tcp4  *:465
www  haproxy  49418  11  tcp4  *:587
www  haproxy  49418  12  tcp4  *:993

# Post-flip: pf rdr rules point at pfSense, not <mailserver>
$ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25'
rdr on vtnet0 ... port = submission -> 10.0.20.1
rdr on vtnet0 ... port = imaps -> 10.0.20.1
rdr on vtnet0 ... port = smtps -> 10.0.20.1
rdr on vtnet0 ... port = 25 -> 10.0.20.1

# 4 HAProxy frontends reachable + SMTP/IMAP banners
$ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly

# Real client IP in maillog for external delivery via Brevo → MX
postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25
postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334

# E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds
$ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver
... Round-trip SUCCESS in 20.3s ...

# Internal Roundcube path unchanged
$ curl -sI https://mail.viktorbarzin.me/  →  302 (Authentik gate intact)

# No mail alerts firing
$ kubectl exec prometheus-server ... /api/v1/alerts | grep Email  →  (empty)
```

### Rollback
```
scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/
ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php'
```
Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias.
Pre-flip config snapshot also saved at
`/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense.

## Phase roadmap (bd code-yiu)

| Phase | Status |
|---|---|
| 1a | ✅ commit ef75c02f  — alt :2525 listener + NodePort |
| 2  | ✅ 2026-04-19      — HAProxy pkg installed on pfSense |
| 3  | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML |
| 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip |
| 6  | ⏸ USER-GATED      — MetalLB LB decommission after 48h observation |
This commit is contained in:
Viktor Barzin 2026-04-19 12:24:50 +00:00
parent 702db75f84
commit 9806d515dd
4 changed files with 355 additions and 74 deletions
Download patch file Download diff file Expand all files Collapse all files

114
stacks/mailserver/modules/mailserver/main.tf
View file

@ -139,6 +139,24 @@ resource "kubernetes_config_map" "mailserver_config" {
# attempt waits 5s before responding, stretching a 1000-password
# dictionary attack from <1s to ~85min. Addresses code-9mi.
auth_failure_delay = 5s
# code-yiu Phase 5: alt IMAPS listener on :10993 that REQUIRES the
# HAProxy PROXY v2 wire format. pfSense HAProxy injects the header
# on backend connects via k8s-node:30128 → kube-proxy → pod :10993.
# Real client IP recovered from header despite kube-proxy SNAT.
# The stock :993 listener stays PROXY-free for internal clients
# (Roundcube, email-roundtrip-monitor) on the mailserver ClusterIP.
# haproxy_trusted_networks = source IPs allowed to *send* PROXY v2.
# Post kube-proxy SNAT the source is the k8s node IP (10.0.20.101-104);
# allow-list the whole VLAN 20 node subnet.
haproxy_trusted_networks = 10.0.20.0/24
service imap-login {
inet_listener imaps_proxy {
port = 10993
ssl = yes
haproxy = yes
}
}
EOF
fail2ban_conf = <<-EOF
[DEFAULT]
@ -192,22 +210,60 @@ resource "kubernetes_config_map" "mailserver_user_patches" {
data = {
"user-patches.sh" = <<-EOT
#!/bin/bash
# code-yiu: append PROXY-speaking alt SMTP listener on :2525 to master.cf.
# Runs in parallel to stock :25 postscreen (which stays PROXY-free for
# internal clients). pfSense HAProxy injects PROXY v2 on connections to
# k8s-node:NodePort → kube-proxy → pod :2525. Real client IP recovered
# from PROXY header despite kube-proxy SNAT.
# code-yiu Phase 5: append PROXY-speaking alt listeners to Postfix master.cf:
# :2525 postscreen (alt :25) — injected with PROXY v2 by pfSense HAProxy
# :4465 smtpd (alt :465 SMTPS) — ditto, wrappermode TLS
# :5587 smtpd (alt :587 submission) — ditto
# Stock :25/:465/:587 stay in parallel (no PROXY required) so internal
# Roundcube/probe traffic on mailserver.svc ClusterIP keeps working.
# Dovecot alt IMAPS listener on :10993 is configured via dovecot.cf
# (not here) because that's a Dovecot config, not a Postfix master.cf.
set -euxo pipefail
MASTER_CF=/etc/postfix/master.cf
SENTINEL='# code-yiu:2525'
SENTINEL='# code-yiu:alt-proxy'
if ! grep -qF "$SENTINEL" "$MASTER_CF"; then
cat >> "$MASTER_CF" <<'PFXEOF'
# code-yiu:2525 — PROXY-speaking postscreen listener for pfSense HAProxy backend.
2525 inet n - y - 1 postscreen
-o syslog_name=postfix/smtpd-proxy
# code-yiu:alt-proxy — PROXY-speaking alt listeners for pfSense HAProxy backend pool.
# Mirrors stock docker-mailserver submission/submissions options (incl. SASL via
# Dovecot's /dev/shm/sasl-auth.sock) but with PROXY v2 upstream. chroot=n so the
# SASL path is readable from the smtpd process (sockets live outside /var/spool).
2525 inet n - n - 1 postscreen
-o syslog_name=postfix/smtpd-proxy25
-o postscreen_upstream_proxy_protocol=haproxy
-o postscreen_upstream_proxy_timeout=5s
4465 inet n - n - - smtpd
-o syslog_name=postfix/smtpd-proxy465
-o smtpd_tls_wrappermode=yes
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_type=dovecot
-o smtpd_tls_auth_only=yes
-o smtpd_reject_unlisted_recipient=no
-o smtpd_sasl_authenticated_header=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
-o smtpd_sender_restrictions=$mua_sender_restrictions
-o smtpd_discard_ehlo_keywords=
-o milter_macro_daemon_name=ORIGINATING
-o cleanup_service_name=sender-cleanup
-o smtpd_upstream_proxy_protocol=haproxy
-o smtpd_upstream_proxy_timeout=5s
5587 inet n - n - - smtpd
-o syslog_name=postfix/smtpd-proxy587
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_type=dovecot
-o smtpd_tls_auth_only=yes
-o smtpd_reject_unlisted_recipient=no
-o smtpd_sasl_authenticated_header=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
-o smtpd_sender_restrictions=$mua_sender_restrictions
-o smtpd_discard_ehlo_keywords=
-o milter_macro_daemon_name=ORIGINATING
-o cleanup_service_name=sender-cleanup
-o smtpd_upstream_proxy_protocol=haproxy
-o smtpd_upstream_proxy_timeout=5s
PFXEOF
fi
EOT
@ -455,12 +511,29 @@ resource "kubernetes_deployment" "mailserver" {
container_port = 993
protocol = "TCP"
}
# code-yiu Phase 1a: alt PROXY-speaking SMTP listener.
# code-yiu Phase 5: alt PROXY-speaking listeners.
# Postfix: 2525 (postscreen), 4465 (smtps), 5587 (submission).
# Dovecot: 10993 (imaps). All require PROXY v2 from pfSense HAProxy.
port {
name = "smtp-proxy"
container_port = 2525
protocol = "TCP"
}
port {
name = "smtps-proxy"
container_port = 4465
protocol = "TCP"
}
port {
name = "sub-proxy"
container_port = 5587
protocol = "TCP"
}
port {
name = "imaps-proxy"
container_port = 10993
protocol = "TCP"
}
env_from {
config_map_ref {
name = "mailserver.env.config"
@ -637,6 +710,27 @@ resource "kubernetes_service" "mailserver_proxy" {
target_port = 2525
node_port = 30125
}
port {
name = "smtps-proxy"
protocol = "TCP"
port = 465
target_port = 4465
node_port = 30126
}
port {
name = "sub-proxy"
protocol = "TCP"
port = 587
target_port = 5587
node_port = 30127
}
port {
name = "imaps-proxy"
protocol = "TCP"
port = 993
target_port = 10993
node_port = 30128
}
}
}