viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip]

Browse source 
## Context (bd code-yiu)

Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local,
pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real
client IP now preserved end-to-end on ports 25/465/587/993, both for
postscreen anti-spam scoring and CrowdSec auth-failure bans.

## This change

### k8s (stacks/mailserver/modules/mailserver/main.tf)

- `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3
  alt PROXY-speaking services to master.cf:
  - `:2525` postscreen (alt :25)
  - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS)
  - `:5587` smtpd (alt :587 submission)
  All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`.
  Mirror stock submission/submissions options (SASL via Dovecot, TLS,
  client restrictions, mua_sender_restrictions). chroot=n so the SASL
  socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot.
- `dovecot.cf` ConfigMap adds:
  ```
  haproxy_trusted_networks = 10.0.20.0/24
  service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } }
  ```
  Stock :993 stays PROXY-free for internal Roundcube/probe clients.
- Container ports: 4 new (4465, 5587, 10993, 2525 already there).
- `mailserver-proxy` NodePort Service now exposes all 4 ports:
  25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128
  (ETP:Cluster).

### pfSense (scripts/pfsense-haproxy-bootstrap.php)

Rebuilt to declare 4 backend pools (one per NodePort) and 4 production
frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy
`:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`.
Idempotent — re-runs converge on declared state.

### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php)

Flip script: updates `<nat><rule>` entries for mail ports from target
`<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense
HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the
rollback. Both scripts are idempotent.

## What is NOT in this change

- Phase 6 (decommission MetalLB LB path, downgrade mailserver Service
  from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do
  NOT run until explicit approval.
- Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock
  ETP:Local ports — functional backup path + consumed by internal
  clients that hit `mailserver.mailserver.svc.cluster.local` (routes
  via ClusterIP layer of the LB Service, bypassing ETP).
- Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via
  unchanged NAT rule.

## Test Plan

### Automated (verified pre-commit 2026-04-19)
```
# k8s container listens on all 8 ports
$ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \
    -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b'
... all 8 listening ...

# pfSense HAProxy listens on all 5 (production + legacy test)
$ ssh admin@10.0.20.1 'sockstat -l | grep haproxy'
www  haproxy  49418  5   tcp4  *:25
www  haproxy  49418  6   tcp4  *:2525
www  haproxy  49418  10  tcp4  *:465
www  haproxy  49418  11  tcp4  *:587
www  haproxy  49418  12  tcp4  *:993

# Post-flip: pf rdr rules point at pfSense, not <mailserver>
$ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25'
rdr on vtnet0 ... port = submission -> 10.0.20.1
rdr on vtnet0 ... port = imaps -> 10.0.20.1
rdr on vtnet0 ... port = smtps -> 10.0.20.1
rdr on vtnet0 ... port = 25 -> 10.0.20.1

# 4 HAProxy frontends reachable + SMTP/IMAP banners
$ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly

# Real client IP in maillog for external delivery via Brevo → MX
postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25
postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334

# E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds
$ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver
... Round-trip SUCCESS in 20.3s ...

# Internal Roundcube path unchanged
$ curl -sI https://mail.viktorbarzin.me/  →  302 (Authentik gate intact)

# No mail alerts firing
$ kubectl exec prometheus-server ... /api/v1/alerts | grep Email  →  (empty)
```

### Rollback
```
scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/
ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php'
```
Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias.
Pre-flip config snapshot also saved at
`/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense.

## Phase roadmap (bd code-yiu)

| Phase | Status |
|---|---|
| 1a | ✅ commit ef75c02f  — alt :2525 listener + NodePort |
| 2  | ✅ 2026-04-19      — HAProxy pkg installed on pfSense |
| 3  | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML |
| 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip |
| 6  | ⏸ USER-GATED      — MetalLB LB decommission after 48h observation |
This commit is contained in:
Viktor Barzin 2026-04-19 12:24:50 +00:00
parent 702db75f84
commit 9806d515dd
4 changed files with 355 additions and 74 deletions
Download patch file Download diff file Expand all files Collapse all files

199
scripts/pfsense-haproxy-bootstrap.php
View file

@ -1,6 +1,6 @@
<?php
// pfSense HAProxy bootstrap — adds/refreshes the mailserver_proxy_test
// frontend + mailserver_nodes backend for code-yiu (PROXY-v2 SMTP path).
// pfSense HAProxy bootstrap — configures the mailserver PROXY-v2 path
// (bd code-yiu, Phases 2/3 + 5).
//
// WHY THIS EXISTS
// pfSense HAProxy config is stored XML-in-`/cf/conf/config.xml` under
@ -11,18 +11,22 @@
// from scratch (DR restore, fresh pfSense install, etc.).
//
// WHAT IT BUILDS
// Backend pool `mailserver_nodes`: 4 k8s workers on NodePort 30125 with
// `send-proxy-v2` + TCP health-check every 120s.
// Frontend `mailserver_proxy_test`: listens on 10.0.20.1:2525, TCP mode,
// forwards to the pool above.
// 4 backend pools — one per mail port:
// mailserver_nodes_smtp → k8s-node1..4:30125 (container :2525 postscreen)
// mailserver_nodes_smtps → k8s-node1..4:30126 (container :4465 smtps)
// mailserver_nodes_sub → k8s-node1..4:30127 (container :5587 submission)
// mailserver_nodes_imaps → k8s-node1..4:30128 (container :10993 IMAPS)
// Each server uses `send-proxy-v2` and TCP health-check every 120s.
// 4 frontends on pfSense 10.0.20.1:{25,465,587,993} TCP mode.
// + 1 legacy test frontend on :2525 (kept for validation; safe to remove later).
//
// USAGE (on pfSense host, via SSH as admin)
// scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/
// ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'
//
// IDEMPOTENCY
// Removes any existing entries named `mailserver_nodes` / `mailserver_proxy_test`
// before re-adding, so repeat runs are safe and behave as reset-to-declared.
// Removes any existing entries named mailserver_* before re-adding, so
// repeat runs are safe and behave as reset-to-declared.
require_once('/etc/inc/config.inc');
require_once('/usr/local/pkg/haproxy/haproxy.inc');
@ -39,77 +43,144 @@ $h = &$config['installedpackages']['haproxy'];
$h['enable'] = 'yes';
$h['maxconn'] = '1000';
// ── Backend pool ────────────────────────────────────────────────────────
if (!is_array($h['ha_pools'])) $h['ha_pools'] = ['item' => []];
if (!is_array($h['ha_pools']['item'])) $h['ha_pools']['item'] = [];
$h['ha_pools']['item'] = array_values(array_filter(
$h['ha_pools']['item'],
fn($p) => ($p['name'] ?? '') !== 'mailserver_nodes'
));
// Our declared object names (anything starting with mailserver_ is ours)
$POOL_NAMES = [
'mailserver_nodes', // legacy (Phase 2/3 test)
'mailserver_nodes_smtp',
'mailserver_nodes_smtps',
'mailserver_nodes_sub',
'mailserver_nodes_imaps',
];
$FRONTEND_NAMES = [
'mailserver_proxy_test', // legacy (Phase 2/3 test, :2525)
'mailserver_proxy_25',
'mailserver_proxy_465',
'mailserver_proxy_587',
'mailserver_proxy_993',
];
$servers = [];
foreach ([
// k8s workers. Not in the cluster: master (control-plane) and node5
// (doesn't exist in this topology).
$NODES = [
['k8s-node1', '10.0.20.101'],
['k8s-node2', '10.0.20.102'],
['k8s-node3', '10.0.20.103'],
['k8s-node4', '10.0.20.104'],
] as $n) {
$servers[] = [
'name' => $n[0],
'address' => $n[1],
'port' => '30125',
'weight' => '10',
'ssl' => '',
// check every 2 minutes to avoid flooding postscreen with
// send-proxy-v2 + immediate close connections (see bd code-yiu notes).
'checkinter' => '120000',
'advanced' => 'send-proxy-v2',
'status' => 'active',
];
function build_pool(string $name, string $nodeport, array $nodes): array {
$servers = [];
foreach ($nodes as $n) {
$servers[] = [
'name' => $n[0],
'address' => $n[1],
'port' => $nodeport,
'weight' => '10',
'ssl' => '',
// check every 2 min — send-proxy-v2 check + close generates
// noise on postscreen, not worth doing more often.
'checkinter' => '120000',
'advanced' => 'send-proxy-v2',
'status' => 'active',
];
}
return [
'name' => $name,
'balance' => 'roundrobin',
'check_type' => 'TCP',
'checkinter' => '120000',
'retries' => '3',
'ha_servers' => ['item' => $servers],
'advanced_bind' => '',
'persist_cookie_enabled' => '',
'transparent_clientip' => '',
'advanced' => '',
];
}
$h['ha_pools']['item'][] = [
'name' => 'mailserver_nodes',
'balance' => 'roundrobin',
'check_type' => 'TCP',
'checkinter' => '120000',
'retries' => '3',
'ha_servers' => ['item' => $servers],
'advanced_bind' => '',
'persist_cookie_enabled' => '',
'transparent_clientip' => '',
'advanced' => '',
];
function build_frontend(string $name, string $descr, string $extaddr, string $port, string $pool): array {
return [
'name' => $name,
'descr' => $descr,
'status' => 'active',
'secondary' => '',
'type' => 'tcp',
'a_extaddr' => ['item' => [[
'extaddr' => $extaddr,
'extaddr_port' => $port,
'extaddr_ssl' => '',
'extaddr_advanced' => '',
]]],
'backend_serverpool' => $pool,
'ha_acls' => '',
'dontlognull'=> '',
'httpclose' => '',
'forwardfor' => '',
'advanced' => '',
];
}
// ── Frontend (pfSense "ha_backends") ────────────────────────────────────
if (!is_array($h['ha_backends'])) $h['ha_backends'] = ['item' => []];
// ── Backend pools ───────────────────────────────────────────────────────
if (!is_array($h['ha_pools'])) $h['ha_pools'] = ['item' => []];
if (!is_array($h['ha_pools']['item'])) $h['ha_pools']['item'] = [];
$h['ha_pools']['item'] = array_values(array_filter(
$h['ha_pools']['item'],
fn($p) => !in_array($p['name'] ?? '', $POOL_NAMES, true)
));
// Legacy test pool (still used by the :2525 test frontend for manual SMTP roundtrip).
$h['ha_pools']['item'][] = build_pool('mailserver_nodes', '30125', $NODES);
// Production pools — one per mail port.
$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtp', '30125', $NODES);
$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtps', '30126', $NODES);
$h['ha_pools']['item'][] = build_pool('mailserver_nodes_sub', '30127', $NODES);
$h['ha_pools']['item'][] = build_pool('mailserver_nodes_imaps', '30128', $NODES);
// ── Frontends ───────────────────────────────────────────────────────────
if (!is_array($h['ha_backends'])) $h['ha_backends'] = ['item' => []];
if (!is_array($h['ha_backends']['item'])) $h['ha_backends']['item'] = [];
$h['ha_backends']['item'] = array_values(array_filter(
$h['ha_backends']['item'],
fn($f) => ($f['name'] ?? '') !== 'mailserver_proxy_test'
fn($f) => !in_array($f['name'] ?? '', $FRONTEND_NAMES, true)
));
$h['ha_backends']['item'][] = [
'name' => 'mailserver_proxy_test',
'descr' => 'code-yiu Phase 3 test — PROXY v2 to k8s mailserver NodePort 30125',
'status' => 'active',
'secondary' => '',
'type' => 'tcp',
'a_extaddr' => ['item' => [[
'extaddr' => '10.0.20.1',
'extaddr_port' => '2525',
'extaddr_ssl' => '',
'extaddr_advanced' => '',
]]],
'backend_serverpool' => 'mailserver_nodes',
'ha_acls' => '',
'dontlognull'=> '',
'httpclose' => '',
'forwardfor' => '',
'advanced' => '',
];
// Legacy test frontend — :2525 — retained so SMTP roundtrip tests keep working
// without touching the real :25. Safe to remove once fully validated.
$h['ha_backends']['item'][] = build_frontend(
'mailserver_proxy_test',
'code-yiu Phase 2/3 test — PROXY v2 to k8s mailserver NodePort 30125 (alt port :2525)',
'10.0.20.1', '2525',
'mailserver_nodes'
);
write_config('code-yiu: mailserver_proxy HAProxy frontend + backend (bootstrap)');
// Production frontends — 4 ports listening on pfSense VLAN20 IP 10.0.20.1.
$h['ha_backends']['item'][] = build_frontend(
'mailserver_proxy_25',
'code-yiu Phase 4/5 — external SMTP (:25) via PROXY v2 → pod :2525 postscreen',
'10.0.20.1', '25',
'mailserver_nodes_smtp'
);
$h['ha_backends']['item'][] = build_frontend(
'mailserver_proxy_465',
'code-yiu Phase 4/5 — external SMTPS (:465) via PROXY v2 → pod :4465 smtpd',
'10.0.20.1', '465',
'mailserver_nodes_smtps'
);
$h['ha_backends']['item'][] = build_frontend(
'mailserver_proxy_587',
'code-yiu Phase 4/5 — external submission (:587) via PROXY v2 → pod :5587 smtpd',
'10.0.20.1', '587',
'mailserver_nodes_sub'
);
$h['ha_backends']['item'][] = build_frontend(
'mailserver_proxy_993',
'code-yiu Phase 4/5 — external IMAPS (:993) via PROXY v2 → pod :10993 Dovecot',
'10.0.20.1', '993',
'mailserver_nodes_imaps'
);
write_config('code-yiu: mailserver HAProxy — 4 production frontends + legacy :2525 test');
$messages = '';
$rc = haproxy_check_and_run($messages, true);