infra/scripts/pfsense-haproxy-bootstrap.php

189 lines
7.3 KiB
PHP
Raw Normal View History

[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip] ## Context (bd code-yiu) Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so it lives in the nightly backup) of the PROXY-v2 migration. Test path only — listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525 postscreen. Real client IP verified in maillog (`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a container plumbing is already live (commit ef75c02f). pfSense HAProxy config lives in `/cf/conf/config.xml` under `<installedpackages><haproxy>`. That file is captured daily by `scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`) and synced offsite to Synology. No new backup wiring needed — this commit documents the fact + adds the reproducer script. ## This change Two files, both additive: 1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that edits pfSense config.xml to add: - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125, `send-proxy-v2`, TCP health-check every 120000 ms (2 min). - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525 in TCP mode, forwarding to the pool. Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg` and reload HAProxy. Removes existing items with the same name before adding, so repeat runs converge on declared state. 2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering current state, validation, bootstrap/restore, health checks, phase roadmap, and known warts (health-check noise + bind-address templating). ## What is NOT in this change - Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred. - Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual- inet_listener) — deferred. - Terraform for pfSense HAProxy pkg install — not possible (no Terraform provider for pfSense pkg management). Runbook documents the manual `pkg install` command. ## Test Plan ### Automated ``` $ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525' 64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D www haproxy 64009 5 tcp4 *:2525 *:* $ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \ | awk 'NR>1 {print $4, $6}' node1 2 node2 2 node3 2 node4 2 # all UP $ python3 -c " import socket; s=socket.socket(); s.settimeout(10) s.connect(('10.0.20.1', 2525)) print(s.recv(200).decode()) s.send(b'EHLO persist-test.example.com\r\n') print(s.recv(500).decode()) s.send(b'QUIT\r\n'); s.close()" 220-mail.viktorbarzin.me ESMTP ... 250-mail.viktorbarzin.me 250-SIZE 209715200 ... 221 2.0.0 Bye $ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \ | grep smtpd-proxy.*CONNECT postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525 ``` Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy SNAT) → PROXY-v2 roundtrip confirmed. ### Manual Verification Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the now-persisted config (`<enable>yes</enable>` in XML). Connection test above should still work. ## Reproduce locally 1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/` 2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK 3. `python3 -c '...' ` SMTP roundtrip test above.
2026-04-19 12:07:47 +00:00
<?php
[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] ## Context (bd code-yiu) Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local, pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real client IP now preserved end-to-end on ports 25/465/587/993, both for postscreen anti-spam scoring and CrowdSec auth-failure bans. ## This change ### k8s (stacks/mailserver/modules/mailserver/main.tf) - `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3 alt PROXY-speaking services to master.cf: - `:2525` postscreen (alt :25) - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS) - `:5587` smtpd (alt :587 submission) All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`. Mirror stock submission/submissions options (SASL via Dovecot, TLS, client restrictions, mua_sender_restrictions). chroot=n so the SASL socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot. - `dovecot.cf` ConfigMap adds: ``` haproxy_trusted_networks = 10.0.20.0/24 service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } } ``` Stock :993 stays PROXY-free for internal Roundcube/probe clients. - Container ports: 4 new (4465, 5587, 10993, 2525 already there). - `mailserver-proxy` NodePort Service now exposes all 4 ports: 25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128 (ETP:Cluster). ### pfSense (scripts/pfsense-haproxy-bootstrap.php) Rebuilt to declare 4 backend pools (one per NodePort) and 4 production frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy `:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`. Idempotent — re-runs converge on declared state. ### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php) Flip script: updates `<nat><rule>` entries for mail ports from target `<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the rollback. Both scripts are idempotent. ## What is NOT in this change - Phase 6 (decommission MetalLB LB path, downgrade mailserver Service from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do NOT run until explicit approval. - Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock ETP:Local ports — functional backup path + consumed by internal clients that hit `mailserver.mailserver.svc.cluster.local` (routes via ClusterIP layer of the LB Service, bypassing ETP). - Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via unchanged NAT rule. ## Test Plan ### Automated (verified pre-commit 2026-04-19) ``` # k8s container listens on all 8 ports $ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \ -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b' ... all 8 listening ... # pfSense HAProxy listens on all 5 (production + legacy test) $ ssh admin@10.0.20.1 'sockstat -l | grep haproxy' www haproxy 49418 5 tcp4 *:25 www haproxy 49418 6 tcp4 *:2525 www haproxy 49418 10 tcp4 *:465 www haproxy 49418 11 tcp4 *:587 www haproxy 49418 12 tcp4 *:993 # Post-flip: pf rdr rules point at pfSense, not <mailserver> $ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25' rdr on vtnet0 ... port = submission -> 10.0.20.1 rdr on vtnet0 ... port = imaps -> 10.0.20.1 rdr on vtnet0 ... port = smtps -> 10.0.20.1 rdr on vtnet0 ... port = 25 -> 10.0.20.1 # 4 HAProxy frontends reachable + SMTP/IMAP banners $ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly # Real client IP in maillog for external delivery via Brevo → MX postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25 postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334 # E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds $ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver ... Round-trip SUCCESS in 20.3s ... # Internal Roundcube path unchanged $ curl -sI https://mail.viktorbarzin.me/ → 302 (Authentik gate intact) # No mail alerts firing $ kubectl exec prometheus-server ... /api/v1/alerts | grep Email → (empty) ``` ### Rollback ``` scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/ ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php' ``` Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias. Pre-flip config snapshot also saved at `/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense. ## Phase roadmap (bd code-yiu) | Phase | Status | |---|---| | 1a | ✅ commit ef75c02f — alt :2525 listener + NodePort | | 2 | ✅ 2026-04-19 — HAProxy pkg installed on pfSense | | 3 | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML | | 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip | | 6 | ⏸ USER-GATED — MetalLB LB decommission after 48h observation |
2026-04-19 12:24:50 +00:00
// pfSense HAProxy bootstrap — configures the mailserver PROXY-v2 path
// (bd code-yiu, Phases 2/3 + 5).
[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip] ## Context (bd code-yiu) Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so it lives in the nightly backup) of the PROXY-v2 migration. Test path only — listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525 postscreen. Real client IP verified in maillog (`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a container plumbing is already live (commit ef75c02f). pfSense HAProxy config lives in `/cf/conf/config.xml` under `<installedpackages><haproxy>`. That file is captured daily by `scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`) and synced offsite to Synology. No new backup wiring needed — this commit documents the fact + adds the reproducer script. ## This change Two files, both additive: 1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that edits pfSense config.xml to add: - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125, `send-proxy-v2`, TCP health-check every 120000 ms (2 min). - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525 in TCP mode, forwarding to the pool. Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg` and reload HAProxy. Removes existing items with the same name before adding, so repeat runs converge on declared state. 2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering current state, validation, bootstrap/restore, health checks, phase roadmap, and known warts (health-check noise + bind-address templating). ## What is NOT in this change - Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred. - Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual- inet_listener) — deferred. - Terraform for pfSense HAProxy pkg install — not possible (no Terraform provider for pfSense pkg management). Runbook documents the manual `pkg install` command. ## Test Plan ### Automated ``` $ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525' 64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D www haproxy 64009 5 tcp4 *:2525 *:* $ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \ | awk 'NR>1 {print $4, $6}' node1 2 node2 2 node3 2 node4 2 # all UP $ python3 -c " import socket; s=socket.socket(); s.settimeout(10) s.connect(('10.0.20.1', 2525)) print(s.recv(200).decode()) s.send(b'EHLO persist-test.example.com\r\n') print(s.recv(500).decode()) s.send(b'QUIT\r\n'); s.close()" 220-mail.viktorbarzin.me ESMTP ... 250-mail.viktorbarzin.me 250-SIZE 209715200 ... 221 2.0.0 Bye $ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \ | grep smtpd-proxy.*CONNECT postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525 ``` Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy SNAT) → PROXY-v2 roundtrip confirmed. ### Manual Verification Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the now-persisted config (`<enable>yes</enable>` in XML). Connection test above should still work. ## Reproduce locally 1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/` 2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK 3. `python3 -c '...' ` SMTP roundtrip test above.
2026-04-19 12:07:47 +00:00
//
// WHY THIS EXISTS
// pfSense HAProxy config is stored XML-in-`/cf/conf/config.xml` under
// `<installedpackages><haproxy>`. That file IS picked up by the nightly
// `daily-backup` on the PVE host (see `scripts/daily-backup.sh` → `scp
// root@10.0.20.1:/cf/conf/config.xml`) and synced to Synology. This script
// is the canonical reproducer: run it to rebuild the pfSense HAProxy config
// from scratch (DR restore, fresh pfSense install, etc.).
//
// WHAT IT BUILDS
[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] ## Context (bd code-yiu) Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local, pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real client IP now preserved end-to-end on ports 25/465/587/993, both for postscreen anti-spam scoring and CrowdSec auth-failure bans. ## This change ### k8s (stacks/mailserver/modules/mailserver/main.tf) - `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3 alt PROXY-speaking services to master.cf: - `:2525` postscreen (alt :25) - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS) - `:5587` smtpd (alt :587 submission) All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`. Mirror stock submission/submissions options (SASL via Dovecot, TLS, client restrictions, mua_sender_restrictions). chroot=n so the SASL socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot. - `dovecot.cf` ConfigMap adds: ``` haproxy_trusted_networks = 10.0.20.0/24 service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } } ``` Stock :993 stays PROXY-free for internal Roundcube/probe clients. - Container ports: 4 new (4465, 5587, 10993, 2525 already there). - `mailserver-proxy` NodePort Service now exposes all 4 ports: 25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128 (ETP:Cluster). ### pfSense (scripts/pfsense-haproxy-bootstrap.php) Rebuilt to declare 4 backend pools (one per NodePort) and 4 production frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy `:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`. Idempotent — re-runs converge on declared state. ### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php) Flip script: updates `<nat><rule>` entries for mail ports from target `<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the rollback. Both scripts are idempotent. ## What is NOT in this change - Phase 6 (decommission MetalLB LB path, downgrade mailserver Service from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do NOT run until explicit approval. - Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock ETP:Local ports — functional backup path + consumed by internal clients that hit `mailserver.mailserver.svc.cluster.local` (routes via ClusterIP layer of the LB Service, bypassing ETP). - Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via unchanged NAT rule. ## Test Plan ### Automated (verified pre-commit 2026-04-19) ``` # k8s container listens on all 8 ports $ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \ -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b' ... all 8 listening ... # pfSense HAProxy listens on all 5 (production + legacy test) $ ssh admin@10.0.20.1 'sockstat -l | grep haproxy' www haproxy 49418 5 tcp4 *:25 www haproxy 49418 6 tcp4 *:2525 www haproxy 49418 10 tcp4 *:465 www haproxy 49418 11 tcp4 *:587 www haproxy 49418 12 tcp4 *:993 # Post-flip: pf rdr rules point at pfSense, not <mailserver> $ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25' rdr on vtnet0 ... port = submission -> 10.0.20.1 rdr on vtnet0 ... port = imaps -> 10.0.20.1 rdr on vtnet0 ... port = smtps -> 10.0.20.1 rdr on vtnet0 ... port = 25 -> 10.0.20.1 # 4 HAProxy frontends reachable + SMTP/IMAP banners $ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly # Real client IP in maillog for external delivery via Brevo → MX postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25 postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334 # E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds $ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver ... Round-trip SUCCESS in 20.3s ... # Internal Roundcube path unchanged $ curl -sI https://mail.viktorbarzin.me/ → 302 (Authentik gate intact) # No mail alerts firing $ kubectl exec prometheus-server ... /api/v1/alerts | grep Email → (empty) ``` ### Rollback ``` scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/ ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php' ``` Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias. Pre-flip config snapshot also saved at `/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense. ## Phase roadmap (bd code-yiu) | Phase | Status | |---|---| | 1a | ✅ commit ef75c02f — alt :2525 listener + NodePort | | 2 | ✅ 2026-04-19 — HAProxy pkg installed on pfSense | | 3 | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML | | 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip | | 6 | ⏸ USER-GATED — MetalLB LB decommission after 48h observation |
2026-04-19 12:24:50 +00:00
// 4 backend pools — one per mail port:
// mailserver_nodes_smtp → k8s-node1..4:30125 (container :2525 postscreen)
// mailserver_nodes_smtps → k8s-node1..4:30126 (container :4465 smtps)
// mailserver_nodes_sub → k8s-node1..4:30127 (container :5587 submission)
// mailserver_nodes_imaps → k8s-node1..4:30128 (container :10993 IMAPS)
// Each server uses `send-proxy-v2` and TCP health-check every 120s.
// 4 frontends on pfSense 10.0.20.1:{25,465,587,993} TCP mode.
// + 1 legacy test frontend on :2525 (kept for validation; safe to remove later).
[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip] ## Context (bd code-yiu) Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so it lives in the nightly backup) of the PROXY-v2 migration. Test path only — listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525 postscreen. Real client IP verified in maillog (`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a container plumbing is already live (commit ef75c02f). pfSense HAProxy config lives in `/cf/conf/config.xml` under `<installedpackages><haproxy>`. That file is captured daily by `scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`) and synced offsite to Synology. No new backup wiring needed — this commit documents the fact + adds the reproducer script. ## This change Two files, both additive: 1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that edits pfSense config.xml to add: - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125, `send-proxy-v2`, TCP health-check every 120000 ms (2 min). - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525 in TCP mode, forwarding to the pool. Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg` and reload HAProxy. Removes existing items with the same name before adding, so repeat runs converge on declared state. 2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering current state, validation, bootstrap/restore, health checks, phase roadmap, and known warts (health-check noise + bind-address templating). ## What is NOT in this change - Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred. - Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual- inet_listener) — deferred. - Terraform for pfSense HAProxy pkg install — not possible (no Terraform provider for pfSense pkg management). Runbook documents the manual `pkg install` command. ## Test Plan ### Automated ``` $ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525' 64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D www haproxy 64009 5 tcp4 *:2525 *:* $ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \ | awk 'NR>1 {print $4, $6}' node1 2 node2 2 node3 2 node4 2 # all UP $ python3 -c " import socket; s=socket.socket(); s.settimeout(10) s.connect(('10.0.20.1', 2525)) print(s.recv(200).decode()) s.send(b'EHLO persist-test.example.com\r\n') print(s.recv(500).decode()) s.send(b'QUIT\r\n'); s.close()" 220-mail.viktorbarzin.me ESMTP ... 250-mail.viktorbarzin.me 250-SIZE 209715200 ... 221 2.0.0 Bye $ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \ | grep smtpd-proxy.*CONNECT postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525 ``` Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy SNAT) → PROXY-v2 roundtrip confirmed. ### Manual Verification Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the now-persisted config (`<enable>yes</enable>` in XML). Connection test above should still work. ## Reproduce locally 1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/` 2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK 3. `python3 -c '...' ` SMTP roundtrip test above.
2026-04-19 12:07:47 +00:00
//
// USAGE (on pfSense host, via SSH as admin)
// scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/
// ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'
//
// IDEMPOTENCY
[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] ## Context (bd code-yiu) Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local, pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real client IP now preserved end-to-end on ports 25/465/587/993, both for postscreen anti-spam scoring and CrowdSec auth-failure bans. ## This change ### k8s (stacks/mailserver/modules/mailserver/main.tf) - `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3 alt PROXY-speaking services to master.cf: - `:2525` postscreen (alt :25) - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS) - `:5587` smtpd (alt :587 submission) All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`. Mirror stock submission/submissions options (SASL via Dovecot, TLS, client restrictions, mua_sender_restrictions). chroot=n so the SASL socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot. - `dovecot.cf` ConfigMap adds: ``` haproxy_trusted_networks = 10.0.20.0/24 service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } } ``` Stock :993 stays PROXY-free for internal Roundcube/probe clients. - Container ports: 4 new (4465, 5587, 10993, 2525 already there). - `mailserver-proxy` NodePort Service now exposes all 4 ports: 25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128 (ETP:Cluster). ### pfSense (scripts/pfsense-haproxy-bootstrap.php) Rebuilt to declare 4 backend pools (one per NodePort) and 4 production frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy `:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`. Idempotent — re-runs converge on declared state. ### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php) Flip script: updates `<nat><rule>` entries for mail ports from target `<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the rollback. Both scripts are idempotent. ## What is NOT in this change - Phase 6 (decommission MetalLB LB path, downgrade mailserver Service from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do NOT run until explicit approval. - Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock ETP:Local ports — functional backup path + consumed by internal clients that hit `mailserver.mailserver.svc.cluster.local` (routes via ClusterIP layer of the LB Service, bypassing ETP). - Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via unchanged NAT rule. ## Test Plan ### Automated (verified pre-commit 2026-04-19) ``` # k8s container listens on all 8 ports $ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \ -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b' ... all 8 listening ... # pfSense HAProxy listens on all 5 (production + legacy test) $ ssh admin@10.0.20.1 'sockstat -l | grep haproxy' www haproxy 49418 5 tcp4 *:25 www haproxy 49418 6 tcp4 *:2525 www haproxy 49418 10 tcp4 *:465 www haproxy 49418 11 tcp4 *:587 www haproxy 49418 12 tcp4 *:993 # Post-flip: pf rdr rules point at pfSense, not <mailserver> $ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25' rdr on vtnet0 ... port = submission -> 10.0.20.1 rdr on vtnet0 ... port = imaps -> 10.0.20.1 rdr on vtnet0 ... port = smtps -> 10.0.20.1 rdr on vtnet0 ... port = 25 -> 10.0.20.1 # 4 HAProxy frontends reachable + SMTP/IMAP banners $ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly # Real client IP in maillog for external delivery via Brevo → MX postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25 postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334 # E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds $ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver ... Round-trip SUCCESS in 20.3s ... # Internal Roundcube path unchanged $ curl -sI https://mail.viktorbarzin.me/ → 302 (Authentik gate intact) # No mail alerts firing $ kubectl exec prometheus-server ... /api/v1/alerts | grep Email → (empty) ``` ### Rollback ``` scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/ ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php' ``` Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias. Pre-flip config snapshot also saved at `/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense. ## Phase roadmap (bd code-yiu) | Phase | Status | |---|---| | 1a | ✅ commit ef75c02f — alt :2525 listener + NodePort | | 2 | ✅ 2026-04-19 — HAProxy pkg installed on pfSense | | 3 | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML | | 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip | | 6 | ⏸ USER-GATED — MetalLB LB decommission after 48h observation |
2026-04-19 12:24:50 +00:00
// Removes any existing entries named mailserver_* before re-adding, so
// repeat runs are safe and behave as reset-to-declared.
[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip] ## Context (bd code-yiu) Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so it lives in the nightly backup) of the PROXY-v2 migration. Test path only — listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525 postscreen. Real client IP verified in maillog (`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a container plumbing is already live (commit ef75c02f). pfSense HAProxy config lives in `/cf/conf/config.xml` under `<installedpackages><haproxy>`. That file is captured daily by `scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`) and synced offsite to Synology. No new backup wiring needed — this commit documents the fact + adds the reproducer script. ## This change Two files, both additive: 1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that edits pfSense config.xml to add: - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125, `send-proxy-v2`, TCP health-check every 120000 ms (2 min). - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525 in TCP mode, forwarding to the pool. Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg` and reload HAProxy. Removes existing items with the same name before adding, so repeat runs converge on declared state. 2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering current state, validation, bootstrap/restore, health checks, phase roadmap, and known warts (health-check noise + bind-address templating). ## What is NOT in this change - Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred. - Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual- inet_listener) — deferred. - Terraform for pfSense HAProxy pkg install — not possible (no Terraform provider for pfSense pkg management). Runbook documents the manual `pkg install` command. ## Test Plan ### Automated ``` $ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525' 64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D www haproxy 64009 5 tcp4 *:2525 *:* $ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \ | awk 'NR>1 {print $4, $6}' node1 2 node2 2 node3 2 node4 2 # all UP $ python3 -c " import socket; s=socket.socket(); s.settimeout(10) s.connect(('10.0.20.1', 2525)) print(s.recv(200).decode()) s.send(b'EHLO persist-test.example.com\r\n') print(s.recv(500).decode()) s.send(b'QUIT\r\n'); s.close()" 220-mail.viktorbarzin.me ESMTP ... 250-mail.viktorbarzin.me 250-SIZE 209715200 ... 221 2.0.0 Bye $ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \ | grep smtpd-proxy.*CONNECT postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525 ``` Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy SNAT) → PROXY-v2 roundtrip confirmed. ### Manual Verification Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the now-persisted config (`<enable>yes</enable>` in XML). Connection test above should still work. ## Reproduce locally 1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/` 2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK 3. `python3 -c '...' ` SMTP roundtrip test above.
2026-04-19 12:07:47 +00:00
require_once('/etc/inc/config.inc');
require_once('/usr/local/pkg/haproxy/haproxy.inc');
require_once('/usr/local/pkg/haproxy/haproxy_utils.inc');
global $config;
parse_config(true);
if (!is_array($config['installedpackages']['haproxy'])) {
$config['installedpackages']['haproxy'] = [];
}
$h = &$config['installedpackages']['haproxy'];
$h['enable'] = 'yes';
$h['maxconn'] = '1000';
[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] ## Context (bd code-yiu) Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local, pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real client IP now preserved end-to-end on ports 25/465/587/993, both for postscreen anti-spam scoring and CrowdSec auth-failure bans. ## This change ### k8s (stacks/mailserver/modules/mailserver/main.tf) - `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3 alt PROXY-speaking services to master.cf: - `:2525` postscreen (alt :25) - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS) - `:5587` smtpd (alt :587 submission) All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`. Mirror stock submission/submissions options (SASL via Dovecot, TLS, client restrictions, mua_sender_restrictions). chroot=n so the SASL socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot. - `dovecot.cf` ConfigMap adds: ``` haproxy_trusted_networks = 10.0.20.0/24 service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } } ``` Stock :993 stays PROXY-free for internal Roundcube/probe clients. - Container ports: 4 new (4465, 5587, 10993, 2525 already there). - `mailserver-proxy` NodePort Service now exposes all 4 ports: 25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128 (ETP:Cluster). ### pfSense (scripts/pfsense-haproxy-bootstrap.php) Rebuilt to declare 4 backend pools (one per NodePort) and 4 production frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy `:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`. Idempotent — re-runs converge on declared state. ### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php) Flip script: updates `<nat><rule>` entries for mail ports from target `<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the rollback. Both scripts are idempotent. ## What is NOT in this change - Phase 6 (decommission MetalLB LB path, downgrade mailserver Service from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do NOT run until explicit approval. - Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock ETP:Local ports — functional backup path + consumed by internal clients that hit `mailserver.mailserver.svc.cluster.local` (routes via ClusterIP layer of the LB Service, bypassing ETP). - Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via unchanged NAT rule. ## Test Plan ### Automated (verified pre-commit 2026-04-19) ``` # k8s container listens on all 8 ports $ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \ -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b' ... all 8 listening ... # pfSense HAProxy listens on all 5 (production + legacy test) $ ssh admin@10.0.20.1 'sockstat -l | grep haproxy' www haproxy 49418 5 tcp4 *:25 www haproxy 49418 6 tcp4 *:2525 www haproxy 49418 10 tcp4 *:465 www haproxy 49418 11 tcp4 *:587 www haproxy 49418 12 tcp4 *:993 # Post-flip: pf rdr rules point at pfSense, not <mailserver> $ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25' rdr on vtnet0 ... port = submission -> 10.0.20.1 rdr on vtnet0 ... port = imaps -> 10.0.20.1 rdr on vtnet0 ... port = smtps -> 10.0.20.1 rdr on vtnet0 ... port = 25 -> 10.0.20.1 # 4 HAProxy frontends reachable + SMTP/IMAP banners $ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly # Real client IP in maillog for external delivery via Brevo → MX postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25 postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334 # E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds $ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver ... Round-trip SUCCESS in 20.3s ... # Internal Roundcube path unchanged $ curl -sI https://mail.viktorbarzin.me/ → 302 (Authentik gate intact) # No mail alerts firing $ kubectl exec prometheus-server ... /api/v1/alerts | grep Email → (empty) ``` ### Rollback ``` scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/ ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php' ``` Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias. Pre-flip config snapshot also saved at `/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense. ## Phase roadmap (bd code-yiu) | Phase | Status | |---|---| | 1a | ✅ commit ef75c02f — alt :2525 listener + NodePort | | 2 | ✅ 2026-04-19 — HAProxy pkg installed on pfSense | | 3 | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML | | 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip | | 6 | ⏸ USER-GATED — MetalLB LB decommission after 48h observation |
2026-04-19 12:24:50 +00:00
// Our declared object names (anything starting with mailserver_ is ours)
$POOL_NAMES = [
'mailserver_nodes', // legacy (Phase 2/3 test)
'mailserver_nodes_smtp',
'mailserver_nodes_smtps',
'mailserver_nodes_sub',
'mailserver_nodes_imaps',
];
$FRONTEND_NAMES = [
'mailserver_proxy_test', // legacy (Phase 2/3 test, :2525)
'mailserver_proxy_25',
'mailserver_proxy_465',
'mailserver_proxy_587',
'mailserver_proxy_993',
];
[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip] ## Context (bd code-yiu) Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so it lives in the nightly backup) of the PROXY-v2 migration. Test path only — listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525 postscreen. Real client IP verified in maillog (`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a container plumbing is already live (commit ef75c02f). pfSense HAProxy config lives in `/cf/conf/config.xml` under `<installedpackages><haproxy>`. That file is captured daily by `scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`) and synced offsite to Synology. No new backup wiring needed — this commit documents the fact + adds the reproducer script. ## This change Two files, both additive: 1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that edits pfSense config.xml to add: - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125, `send-proxy-v2`, TCP health-check every 120000 ms (2 min). - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525 in TCP mode, forwarding to the pool. Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg` and reload HAProxy. Removes existing items with the same name before adding, so repeat runs converge on declared state. 2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering current state, validation, bootstrap/restore, health checks, phase roadmap, and known warts (health-check noise + bind-address templating). ## What is NOT in this change - Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred. - Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual- inet_listener) — deferred. - Terraform for pfSense HAProxy pkg install — not possible (no Terraform provider for pfSense pkg management). Runbook documents the manual `pkg install` command. ## Test Plan ### Automated ``` $ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525' 64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D www haproxy 64009 5 tcp4 *:2525 *:* $ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \ | awk 'NR>1 {print $4, $6}' node1 2 node2 2 node3 2 node4 2 # all UP $ python3 -c " import socket; s=socket.socket(); s.settimeout(10) s.connect(('10.0.20.1', 2525)) print(s.recv(200).decode()) s.send(b'EHLO persist-test.example.com\r\n') print(s.recv(500).decode()) s.send(b'QUIT\r\n'); s.close()" 220-mail.viktorbarzin.me ESMTP ... 250-mail.viktorbarzin.me 250-SIZE 209715200 ... 221 2.0.0 Bye $ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \ | grep smtpd-proxy.*CONNECT postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525 ``` Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy SNAT) → PROXY-v2 roundtrip confirmed. ### Manual Verification Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the now-persisted config (`<enable>yes</enable>` in XML). Connection test above should still work. ## Reproduce locally 1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/` 2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK 3. `python3 -c '...' ` SMTP roundtrip test above.
2026-04-19 12:07:47 +00:00
[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] ## Context (bd code-yiu) Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local, pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real client IP now preserved end-to-end on ports 25/465/587/993, both for postscreen anti-spam scoring and CrowdSec auth-failure bans. ## This change ### k8s (stacks/mailserver/modules/mailserver/main.tf) - `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3 alt PROXY-speaking services to master.cf: - `:2525` postscreen (alt :25) - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS) - `:5587` smtpd (alt :587 submission) All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`. Mirror stock submission/submissions options (SASL via Dovecot, TLS, client restrictions, mua_sender_restrictions). chroot=n so the SASL socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot. - `dovecot.cf` ConfigMap adds: ``` haproxy_trusted_networks = 10.0.20.0/24 service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } } ``` Stock :993 stays PROXY-free for internal Roundcube/probe clients. - Container ports: 4 new (4465, 5587, 10993, 2525 already there). - `mailserver-proxy` NodePort Service now exposes all 4 ports: 25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128 (ETP:Cluster). ### pfSense (scripts/pfsense-haproxy-bootstrap.php) Rebuilt to declare 4 backend pools (one per NodePort) and 4 production frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy `:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`. Idempotent — re-runs converge on declared state. ### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php) Flip script: updates `<nat><rule>` entries for mail ports from target `<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the rollback. Both scripts are idempotent. ## What is NOT in this change - Phase 6 (decommission MetalLB LB path, downgrade mailserver Service from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do NOT run until explicit approval. - Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock ETP:Local ports — functional backup path + consumed by internal clients that hit `mailserver.mailserver.svc.cluster.local` (routes via ClusterIP layer of the LB Service, bypassing ETP). - Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via unchanged NAT rule. ## Test Plan ### Automated (verified pre-commit 2026-04-19) ``` # k8s container listens on all 8 ports $ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \ -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b' ... all 8 listening ... # pfSense HAProxy listens on all 5 (production + legacy test) $ ssh admin@10.0.20.1 'sockstat -l | grep haproxy' www haproxy 49418 5 tcp4 *:25 www haproxy 49418 6 tcp4 *:2525 www haproxy 49418 10 tcp4 *:465 www haproxy 49418 11 tcp4 *:587 www haproxy 49418 12 tcp4 *:993 # Post-flip: pf rdr rules point at pfSense, not <mailserver> $ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25' rdr on vtnet0 ... port = submission -> 10.0.20.1 rdr on vtnet0 ... port = imaps -> 10.0.20.1 rdr on vtnet0 ... port = smtps -> 10.0.20.1 rdr on vtnet0 ... port = 25 -> 10.0.20.1 # 4 HAProxy frontends reachable + SMTP/IMAP banners $ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly # Real client IP in maillog for external delivery via Brevo → MX postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25 postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334 # E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds $ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver ... Round-trip SUCCESS in 20.3s ... # Internal Roundcube path unchanged $ curl -sI https://mail.viktorbarzin.me/ → 302 (Authentik gate intact) # No mail alerts firing $ kubectl exec prometheus-server ... /api/v1/alerts | grep Email → (empty) ``` ### Rollback ``` scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/ ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php' ``` Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias. Pre-flip config snapshot also saved at `/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense. ## Phase roadmap (bd code-yiu) | Phase | Status | |---|---| | 1a | ✅ commit ef75c02f — alt :2525 listener + NodePort | | 2 | ✅ 2026-04-19 — HAProxy pkg installed on pfSense | | 3 | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML | | 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip | | 6 | ⏸ USER-GATED — MetalLB LB decommission after 48h observation |
2026-04-19 12:24:50 +00:00
// k8s workers. Not in the cluster: master (control-plane) and node5
// (doesn't exist in this topology).
$NODES = [
[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip] ## Context (bd code-yiu) Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so it lives in the nightly backup) of the PROXY-v2 migration. Test path only — listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525 postscreen. Real client IP verified in maillog (`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a container plumbing is already live (commit ef75c02f). pfSense HAProxy config lives in `/cf/conf/config.xml` under `<installedpackages><haproxy>`. That file is captured daily by `scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`) and synced offsite to Synology. No new backup wiring needed — this commit documents the fact + adds the reproducer script. ## This change Two files, both additive: 1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that edits pfSense config.xml to add: - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125, `send-proxy-v2`, TCP health-check every 120000 ms (2 min). - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525 in TCP mode, forwarding to the pool. Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg` and reload HAProxy. Removes existing items with the same name before adding, so repeat runs converge on declared state. 2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering current state, validation, bootstrap/restore, health checks, phase roadmap, and known warts (health-check noise + bind-address templating). ## What is NOT in this change - Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred. - Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual- inet_listener) — deferred. - Terraform for pfSense HAProxy pkg install — not possible (no Terraform provider for pfSense pkg management). Runbook documents the manual `pkg install` command. ## Test Plan ### Automated ``` $ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525' 64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D www haproxy 64009 5 tcp4 *:2525 *:* $ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \ | awk 'NR>1 {print $4, $6}' node1 2 node2 2 node3 2 node4 2 # all UP $ python3 -c " import socket; s=socket.socket(); s.settimeout(10) s.connect(('10.0.20.1', 2525)) print(s.recv(200).decode()) s.send(b'EHLO persist-test.example.com\r\n') print(s.recv(500).decode()) s.send(b'QUIT\r\n'); s.close()" 220-mail.viktorbarzin.me ESMTP ... 250-mail.viktorbarzin.me 250-SIZE 209715200 ... 221 2.0.0 Bye $ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \ | grep smtpd-proxy.*CONNECT postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525 ``` Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy SNAT) → PROXY-v2 roundtrip confirmed. ### Manual Verification Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the now-persisted config (`<enable>yes</enable>` in XML). Connection test above should still work. ## Reproduce locally 1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/` 2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK 3. `python3 -c '...' ` SMTP roundtrip test above.
2026-04-19 12:07:47 +00:00
['k8s-node1', '10.0.20.101'],
['k8s-node2', '10.0.20.102'],
['k8s-node3', '10.0.20.103'],
['k8s-node4', '10.0.20.104'],
[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] ## Context (bd code-yiu) Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local, pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real client IP now preserved end-to-end on ports 25/465/587/993, both for postscreen anti-spam scoring and CrowdSec auth-failure bans. ## This change ### k8s (stacks/mailserver/modules/mailserver/main.tf) - `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3 alt PROXY-speaking services to master.cf: - `:2525` postscreen (alt :25) - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS) - `:5587` smtpd (alt :587 submission) All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`. Mirror stock submission/submissions options (SASL via Dovecot, TLS, client restrictions, mua_sender_restrictions). chroot=n so the SASL socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot. - `dovecot.cf` ConfigMap adds: ``` haproxy_trusted_networks = 10.0.20.0/24 service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } } ``` Stock :993 stays PROXY-free for internal Roundcube/probe clients. - Container ports: 4 new (4465, 5587, 10993, 2525 already there). - `mailserver-proxy` NodePort Service now exposes all 4 ports: 25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128 (ETP:Cluster). ### pfSense (scripts/pfsense-haproxy-bootstrap.php) Rebuilt to declare 4 backend pools (one per NodePort) and 4 production frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy `:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`. Idempotent — re-runs converge on declared state. ### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php) Flip script: updates `<nat><rule>` entries for mail ports from target `<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the rollback. Both scripts are idempotent. ## What is NOT in this change - Phase 6 (decommission MetalLB LB path, downgrade mailserver Service from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do NOT run until explicit approval. - Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock ETP:Local ports — functional backup path + consumed by internal clients that hit `mailserver.mailserver.svc.cluster.local` (routes via ClusterIP layer of the LB Service, bypassing ETP). - Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via unchanged NAT rule. ## Test Plan ### Automated (verified pre-commit 2026-04-19) ``` # k8s container listens on all 8 ports $ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \ -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b' ... all 8 listening ... # pfSense HAProxy listens on all 5 (production + legacy test) $ ssh admin@10.0.20.1 'sockstat -l | grep haproxy' www haproxy 49418 5 tcp4 *:25 www haproxy 49418 6 tcp4 *:2525 www haproxy 49418 10 tcp4 *:465 www haproxy 49418 11 tcp4 *:587 www haproxy 49418 12 tcp4 *:993 # Post-flip: pf rdr rules point at pfSense, not <mailserver> $ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25' rdr on vtnet0 ... port = submission -> 10.0.20.1 rdr on vtnet0 ... port = imaps -> 10.0.20.1 rdr on vtnet0 ... port = smtps -> 10.0.20.1 rdr on vtnet0 ... port = 25 -> 10.0.20.1 # 4 HAProxy frontends reachable + SMTP/IMAP banners $ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly # Real client IP in maillog for external delivery via Brevo → MX postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25 postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334 # E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds $ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver ... Round-trip SUCCESS in 20.3s ... # Internal Roundcube path unchanged $ curl -sI https://mail.viktorbarzin.me/ → 302 (Authentik gate intact) # No mail alerts firing $ kubectl exec prometheus-server ... /api/v1/alerts | grep Email → (empty) ``` ### Rollback ``` scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/ ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php' ``` Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias. Pre-flip config snapshot also saved at `/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense. ## Phase roadmap (bd code-yiu) | Phase | Status | |---|---| | 1a | ✅ commit ef75c02f — alt :2525 listener + NodePort | | 2 | ✅ 2026-04-19 — HAProxy pkg installed on pfSense | | 3 | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML | | 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip | | 6 | ⏸ USER-GATED — MetalLB LB decommission after 48h observation |
2026-04-19 12:24:50 +00:00
];
function build_pool(string $name, string $nodeport, array $nodes): array {
$servers = [];
foreach ($nodes as $n) {
$servers[] = [
'name' => $n[0],
'address' => $n[1],
'port' => $nodeport,
'weight' => '10',
'ssl' => '',
// check every 2 min — send-proxy-v2 check + close generates
// noise on postscreen, not worth doing more often.
'checkinter' => '120000',
'advanced' => 'send-proxy-v2',
'status' => 'active',
];
}
return [
'name' => $name,
'balance' => 'roundrobin',
'check_type' => 'TCP',
'checkinter' => '120000',
'retries' => '3',
'ha_servers' => ['item' => $servers],
'advanced_bind' => '',
'persist_cookie_enabled' => '',
'transparent_clientip' => '',
'advanced' => '',
[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip] ## Context (bd code-yiu) Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so it lives in the nightly backup) of the PROXY-v2 migration. Test path only — listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525 postscreen. Real client IP verified in maillog (`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a container plumbing is already live (commit ef75c02f). pfSense HAProxy config lives in `/cf/conf/config.xml` under `<installedpackages><haproxy>`. That file is captured daily by `scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`) and synced offsite to Synology. No new backup wiring needed — this commit documents the fact + adds the reproducer script. ## This change Two files, both additive: 1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that edits pfSense config.xml to add: - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125, `send-proxy-v2`, TCP health-check every 120000 ms (2 min). - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525 in TCP mode, forwarding to the pool. Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg` and reload HAProxy. Removes existing items with the same name before adding, so repeat runs converge on declared state. 2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering current state, validation, bootstrap/restore, health checks, phase roadmap, and known warts (health-check noise + bind-address templating). ## What is NOT in this change - Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred. - Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual- inet_listener) — deferred. - Terraform for pfSense HAProxy pkg install — not possible (no Terraform provider for pfSense pkg management). Runbook documents the manual `pkg install` command. ## Test Plan ### Automated ``` $ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525' 64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D www haproxy 64009 5 tcp4 *:2525 *:* $ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \ | awk 'NR>1 {print $4, $6}' node1 2 node2 2 node3 2 node4 2 # all UP $ python3 -c " import socket; s=socket.socket(); s.settimeout(10) s.connect(('10.0.20.1', 2525)) print(s.recv(200).decode()) s.send(b'EHLO persist-test.example.com\r\n') print(s.recv(500).decode()) s.send(b'QUIT\r\n'); s.close()" 220-mail.viktorbarzin.me ESMTP ... 250-mail.viktorbarzin.me 250-SIZE 209715200 ... 221 2.0.0 Bye $ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \ | grep smtpd-proxy.*CONNECT postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525 ``` Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy SNAT) → PROXY-v2 roundtrip confirmed. ### Manual Verification Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the now-persisted config (`<enable>yes</enable>` in XML). Connection test above should still work. ## Reproduce locally 1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/` 2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK 3. `python3 -c '...' ` SMTP roundtrip test above.
2026-04-19 12:07:47 +00:00
];
}
[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] ## Context (bd code-yiu) Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local, pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real client IP now preserved end-to-end on ports 25/465/587/993, both for postscreen anti-spam scoring and CrowdSec auth-failure bans. ## This change ### k8s (stacks/mailserver/modules/mailserver/main.tf) - `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3 alt PROXY-speaking services to master.cf: - `:2525` postscreen (alt :25) - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS) - `:5587` smtpd (alt :587 submission) All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`. Mirror stock submission/submissions options (SASL via Dovecot, TLS, client restrictions, mua_sender_restrictions). chroot=n so the SASL socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot. - `dovecot.cf` ConfigMap adds: ``` haproxy_trusted_networks = 10.0.20.0/24 service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } } ``` Stock :993 stays PROXY-free for internal Roundcube/probe clients. - Container ports: 4 new (4465, 5587, 10993, 2525 already there). - `mailserver-proxy` NodePort Service now exposes all 4 ports: 25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128 (ETP:Cluster). ### pfSense (scripts/pfsense-haproxy-bootstrap.php) Rebuilt to declare 4 backend pools (one per NodePort) and 4 production frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy `:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`. Idempotent — re-runs converge on declared state. ### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php) Flip script: updates `<nat><rule>` entries for mail ports from target `<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the rollback. Both scripts are idempotent. ## What is NOT in this change - Phase 6 (decommission MetalLB LB path, downgrade mailserver Service from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do NOT run until explicit approval. - Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock ETP:Local ports — functional backup path + consumed by internal clients that hit `mailserver.mailserver.svc.cluster.local` (routes via ClusterIP layer of the LB Service, bypassing ETP). - Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via unchanged NAT rule. ## Test Plan ### Automated (verified pre-commit 2026-04-19) ``` # k8s container listens on all 8 ports $ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \ -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b' ... all 8 listening ... # pfSense HAProxy listens on all 5 (production + legacy test) $ ssh admin@10.0.20.1 'sockstat -l | grep haproxy' www haproxy 49418 5 tcp4 *:25 www haproxy 49418 6 tcp4 *:2525 www haproxy 49418 10 tcp4 *:465 www haproxy 49418 11 tcp4 *:587 www haproxy 49418 12 tcp4 *:993 # Post-flip: pf rdr rules point at pfSense, not <mailserver> $ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25' rdr on vtnet0 ... port = submission -> 10.0.20.1 rdr on vtnet0 ... port = imaps -> 10.0.20.1 rdr on vtnet0 ... port = smtps -> 10.0.20.1 rdr on vtnet0 ... port = 25 -> 10.0.20.1 # 4 HAProxy frontends reachable + SMTP/IMAP banners $ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly # Real client IP in maillog for external delivery via Brevo → MX postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25 postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334 # E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds $ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver ... Round-trip SUCCESS in 20.3s ... # Internal Roundcube path unchanged $ curl -sI https://mail.viktorbarzin.me/ → 302 (Authentik gate intact) # No mail alerts firing $ kubectl exec prometheus-server ... /api/v1/alerts | grep Email → (empty) ``` ### Rollback ``` scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/ ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php' ``` Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias. Pre-flip config snapshot also saved at `/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense. ## Phase roadmap (bd code-yiu) | Phase | Status | |---|---| | 1a | ✅ commit ef75c02f — alt :2525 listener + NodePort | | 2 | ✅ 2026-04-19 — HAProxy pkg installed on pfSense | | 3 | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML | | 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip | | 6 | ⏸ USER-GATED — MetalLB LB decommission after 48h observation |
2026-04-19 12:24:50 +00:00
function build_frontend(string $name, string $descr, string $extaddr, string $port, string $pool): array {
return [
'name' => $name,
'descr' => $descr,
'status' => 'active',
'secondary' => '',
'type' => 'tcp',
'a_extaddr' => ['item' => [[
'extaddr' => $extaddr,
'extaddr_port' => $port,
'extaddr_ssl' => '',
'extaddr_advanced' => '',
]]],
'backend_serverpool' => $pool,
'ha_acls' => '',
'dontlognull'=> '',
'httpclose' => '',
'forwardfor' => '',
'advanced' => '',
];
}
// ── Backend pools ───────────────────────────────────────────────────────
if (!is_array($h['ha_pools'])) $h['ha_pools'] = ['item' => []];
if (!is_array($h['ha_pools']['item'])) $h['ha_pools']['item'] = [];
$h['ha_pools']['item'] = array_values(array_filter(
$h['ha_pools']['item'],
fn($p) => !in_array($p['name'] ?? '', $POOL_NAMES, true)
));
// Legacy test pool (still used by the :2525 test frontend for manual SMTP roundtrip).
$h['ha_pools']['item'][] = build_pool('mailserver_nodes', '30125', $NODES);
// Production pools — one per mail port.
$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtp', '30125', $NODES);
$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtps', '30126', $NODES);
$h['ha_pools']['item'][] = build_pool('mailserver_nodes_sub', '30127', $NODES);
$h['ha_pools']['item'][] = build_pool('mailserver_nodes_imaps', '30128', $NODES);
[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip] ## Context (bd code-yiu) Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so it lives in the nightly backup) of the PROXY-v2 migration. Test path only — listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525 postscreen. Real client IP verified in maillog (`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a container plumbing is already live (commit ef75c02f). pfSense HAProxy config lives in `/cf/conf/config.xml` under `<installedpackages><haproxy>`. That file is captured daily by `scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`) and synced offsite to Synology. No new backup wiring needed — this commit documents the fact + adds the reproducer script. ## This change Two files, both additive: 1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that edits pfSense config.xml to add: - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125, `send-proxy-v2`, TCP health-check every 120000 ms (2 min). - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525 in TCP mode, forwarding to the pool. Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg` and reload HAProxy. Removes existing items with the same name before adding, so repeat runs converge on declared state. 2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering current state, validation, bootstrap/restore, health checks, phase roadmap, and known warts (health-check noise + bind-address templating). ## What is NOT in this change - Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred. - Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual- inet_listener) — deferred. - Terraform for pfSense HAProxy pkg install — not possible (no Terraform provider for pfSense pkg management). Runbook documents the manual `pkg install` command. ## Test Plan ### Automated ``` $ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525' 64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D www haproxy 64009 5 tcp4 *:2525 *:* $ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \ | awk 'NR>1 {print $4, $6}' node1 2 node2 2 node3 2 node4 2 # all UP $ python3 -c " import socket; s=socket.socket(); s.settimeout(10) s.connect(('10.0.20.1', 2525)) print(s.recv(200).decode()) s.send(b'EHLO persist-test.example.com\r\n') print(s.recv(500).decode()) s.send(b'QUIT\r\n'); s.close()" 220-mail.viktorbarzin.me ESMTP ... 250-mail.viktorbarzin.me 250-SIZE 209715200 ... 221 2.0.0 Bye $ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \ | grep smtpd-proxy.*CONNECT postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525 ``` Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy SNAT) → PROXY-v2 roundtrip confirmed. ### Manual Verification Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the now-persisted config (`<enable>yes</enable>` in XML). Connection test above should still work. ## Reproduce locally 1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/` 2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK 3. `python3 -c '...' ` SMTP roundtrip test above.
2026-04-19 12:07:47 +00:00
[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] ## Context (bd code-yiu) Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local, pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real client IP now preserved end-to-end on ports 25/465/587/993, both for postscreen anti-spam scoring and CrowdSec auth-failure bans. ## This change ### k8s (stacks/mailserver/modules/mailserver/main.tf) - `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3 alt PROXY-speaking services to master.cf: - `:2525` postscreen (alt :25) - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS) - `:5587` smtpd (alt :587 submission) All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`. Mirror stock submission/submissions options (SASL via Dovecot, TLS, client restrictions, mua_sender_restrictions). chroot=n so the SASL socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot. - `dovecot.cf` ConfigMap adds: ``` haproxy_trusted_networks = 10.0.20.0/24 service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } } ``` Stock :993 stays PROXY-free for internal Roundcube/probe clients. - Container ports: 4 new (4465, 5587, 10993, 2525 already there). - `mailserver-proxy` NodePort Service now exposes all 4 ports: 25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128 (ETP:Cluster). ### pfSense (scripts/pfsense-haproxy-bootstrap.php) Rebuilt to declare 4 backend pools (one per NodePort) and 4 production frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy `:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`. Idempotent — re-runs converge on declared state. ### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php) Flip script: updates `<nat><rule>` entries for mail ports from target `<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the rollback. Both scripts are idempotent. ## What is NOT in this change - Phase 6 (decommission MetalLB LB path, downgrade mailserver Service from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do NOT run until explicit approval. - Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock ETP:Local ports — functional backup path + consumed by internal clients that hit `mailserver.mailserver.svc.cluster.local` (routes via ClusterIP layer of the LB Service, bypassing ETP). - Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via unchanged NAT rule. ## Test Plan ### Automated (verified pre-commit 2026-04-19) ``` # k8s container listens on all 8 ports $ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \ -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b' ... all 8 listening ... # pfSense HAProxy listens on all 5 (production + legacy test) $ ssh admin@10.0.20.1 'sockstat -l | grep haproxy' www haproxy 49418 5 tcp4 *:25 www haproxy 49418 6 tcp4 *:2525 www haproxy 49418 10 tcp4 *:465 www haproxy 49418 11 tcp4 *:587 www haproxy 49418 12 tcp4 *:993 # Post-flip: pf rdr rules point at pfSense, not <mailserver> $ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25' rdr on vtnet0 ... port = submission -> 10.0.20.1 rdr on vtnet0 ... port = imaps -> 10.0.20.1 rdr on vtnet0 ... port = smtps -> 10.0.20.1 rdr on vtnet0 ... port = 25 -> 10.0.20.1 # 4 HAProxy frontends reachable + SMTP/IMAP banners $ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly # Real client IP in maillog for external delivery via Brevo → MX postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25 postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334 # E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds $ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver ... Round-trip SUCCESS in 20.3s ... # Internal Roundcube path unchanged $ curl -sI https://mail.viktorbarzin.me/ → 302 (Authentik gate intact) # No mail alerts firing $ kubectl exec prometheus-server ... /api/v1/alerts | grep Email → (empty) ``` ### Rollback ``` scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/ ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php' ``` Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias. Pre-flip config snapshot also saved at `/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense. ## Phase roadmap (bd code-yiu) | Phase | Status | |---|---| | 1a | ✅ commit ef75c02f — alt :2525 listener + NodePort | | 2 | ✅ 2026-04-19 — HAProxy pkg installed on pfSense | | 3 | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML | | 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip | | 6 | ⏸ USER-GATED — MetalLB LB decommission after 48h observation |
2026-04-19 12:24:50 +00:00
// ── Frontends ───────────────────────────────────────────────────────────
if (!is_array($h['ha_backends'])) $h['ha_backends'] = ['item' => []];
[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip] ## Context (bd code-yiu) Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so it lives in the nightly backup) of the PROXY-v2 migration. Test path only — listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525 postscreen. Real client IP verified in maillog (`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a container plumbing is already live (commit ef75c02f). pfSense HAProxy config lives in `/cf/conf/config.xml` under `<installedpackages><haproxy>`. That file is captured daily by `scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`) and synced offsite to Synology. No new backup wiring needed — this commit documents the fact + adds the reproducer script. ## This change Two files, both additive: 1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that edits pfSense config.xml to add: - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125, `send-proxy-v2`, TCP health-check every 120000 ms (2 min). - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525 in TCP mode, forwarding to the pool. Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg` and reload HAProxy. Removes existing items with the same name before adding, so repeat runs converge on declared state. 2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering current state, validation, bootstrap/restore, health checks, phase roadmap, and known warts (health-check noise + bind-address templating). ## What is NOT in this change - Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred. - Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual- inet_listener) — deferred. - Terraform for pfSense HAProxy pkg install — not possible (no Terraform provider for pfSense pkg management). Runbook documents the manual `pkg install` command. ## Test Plan ### Automated ``` $ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525' 64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D www haproxy 64009 5 tcp4 *:2525 *:* $ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \ | awk 'NR>1 {print $4, $6}' node1 2 node2 2 node3 2 node4 2 # all UP $ python3 -c " import socket; s=socket.socket(); s.settimeout(10) s.connect(('10.0.20.1', 2525)) print(s.recv(200).decode()) s.send(b'EHLO persist-test.example.com\r\n') print(s.recv(500).decode()) s.send(b'QUIT\r\n'); s.close()" 220-mail.viktorbarzin.me ESMTP ... 250-mail.viktorbarzin.me 250-SIZE 209715200 ... 221 2.0.0 Bye $ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \ | grep smtpd-proxy.*CONNECT postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525 ``` Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy SNAT) → PROXY-v2 roundtrip confirmed. ### Manual Verification Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the now-persisted config (`<enable>yes</enable>` in XML). Connection test above should still work. ## Reproduce locally 1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/` 2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK 3. `python3 -c '...' ` SMTP roundtrip test above.
2026-04-19 12:07:47 +00:00
if (!is_array($h['ha_backends']['item'])) $h['ha_backends']['item'] = [];
$h['ha_backends']['item'] = array_values(array_filter(
$h['ha_backends']['item'],
[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] ## Context (bd code-yiu) Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local, pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real client IP now preserved end-to-end on ports 25/465/587/993, both for postscreen anti-spam scoring and CrowdSec auth-failure bans. ## This change ### k8s (stacks/mailserver/modules/mailserver/main.tf) - `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3 alt PROXY-speaking services to master.cf: - `:2525` postscreen (alt :25) - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS) - `:5587` smtpd (alt :587 submission) All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`. Mirror stock submission/submissions options (SASL via Dovecot, TLS, client restrictions, mua_sender_restrictions). chroot=n so the SASL socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot. - `dovecot.cf` ConfigMap adds: ``` haproxy_trusted_networks = 10.0.20.0/24 service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } } ``` Stock :993 stays PROXY-free for internal Roundcube/probe clients. - Container ports: 4 new (4465, 5587, 10993, 2525 already there). - `mailserver-proxy` NodePort Service now exposes all 4 ports: 25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128 (ETP:Cluster). ### pfSense (scripts/pfsense-haproxy-bootstrap.php) Rebuilt to declare 4 backend pools (one per NodePort) and 4 production frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy `:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`. Idempotent — re-runs converge on declared state. ### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php) Flip script: updates `<nat><rule>` entries for mail ports from target `<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the rollback. Both scripts are idempotent. ## What is NOT in this change - Phase 6 (decommission MetalLB LB path, downgrade mailserver Service from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do NOT run until explicit approval. - Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock ETP:Local ports — functional backup path + consumed by internal clients that hit `mailserver.mailserver.svc.cluster.local` (routes via ClusterIP layer of the LB Service, bypassing ETP). - Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via unchanged NAT rule. ## Test Plan ### Automated (verified pre-commit 2026-04-19) ``` # k8s container listens on all 8 ports $ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \ -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b' ... all 8 listening ... # pfSense HAProxy listens on all 5 (production + legacy test) $ ssh admin@10.0.20.1 'sockstat -l | grep haproxy' www haproxy 49418 5 tcp4 *:25 www haproxy 49418 6 tcp4 *:2525 www haproxy 49418 10 tcp4 *:465 www haproxy 49418 11 tcp4 *:587 www haproxy 49418 12 tcp4 *:993 # Post-flip: pf rdr rules point at pfSense, not <mailserver> $ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25' rdr on vtnet0 ... port = submission -> 10.0.20.1 rdr on vtnet0 ... port = imaps -> 10.0.20.1 rdr on vtnet0 ... port = smtps -> 10.0.20.1 rdr on vtnet0 ... port = 25 -> 10.0.20.1 # 4 HAProxy frontends reachable + SMTP/IMAP banners $ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly # Real client IP in maillog for external delivery via Brevo → MX postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25 postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334 # E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds $ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver ... Round-trip SUCCESS in 20.3s ... # Internal Roundcube path unchanged $ curl -sI https://mail.viktorbarzin.me/ → 302 (Authentik gate intact) # No mail alerts firing $ kubectl exec prometheus-server ... /api/v1/alerts | grep Email → (empty) ``` ### Rollback ``` scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/ ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php' ``` Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias. Pre-flip config snapshot also saved at `/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense. ## Phase roadmap (bd code-yiu) | Phase | Status | |---|---| | 1a | ✅ commit ef75c02f — alt :2525 listener + NodePort | | 2 | ✅ 2026-04-19 — HAProxy pkg installed on pfSense | | 3 | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML | | 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip | | 6 | ⏸ USER-GATED — MetalLB LB decommission after 48h observation |
2026-04-19 12:24:50 +00:00
fn($f) => !in_array($f['name'] ?? '', $FRONTEND_NAMES, true)
[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip] ## Context (bd code-yiu) Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so it lives in the nightly backup) of the PROXY-v2 migration. Test path only — listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525 postscreen. Real client IP verified in maillog (`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a container plumbing is already live (commit ef75c02f). pfSense HAProxy config lives in `/cf/conf/config.xml` under `<installedpackages><haproxy>`. That file is captured daily by `scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`) and synced offsite to Synology. No new backup wiring needed — this commit documents the fact + adds the reproducer script. ## This change Two files, both additive: 1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that edits pfSense config.xml to add: - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125, `send-proxy-v2`, TCP health-check every 120000 ms (2 min). - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525 in TCP mode, forwarding to the pool. Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg` and reload HAProxy. Removes existing items with the same name before adding, so repeat runs converge on declared state. 2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering current state, validation, bootstrap/restore, health checks, phase roadmap, and known warts (health-check noise + bind-address templating). ## What is NOT in this change - Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred. - Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual- inet_listener) — deferred. - Terraform for pfSense HAProxy pkg install — not possible (no Terraform provider for pfSense pkg management). Runbook documents the manual `pkg install` command. ## Test Plan ### Automated ``` $ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525' 64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D www haproxy 64009 5 tcp4 *:2525 *:* $ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \ | awk 'NR>1 {print $4, $6}' node1 2 node2 2 node3 2 node4 2 # all UP $ python3 -c " import socket; s=socket.socket(); s.settimeout(10) s.connect(('10.0.20.1', 2525)) print(s.recv(200).decode()) s.send(b'EHLO persist-test.example.com\r\n') print(s.recv(500).decode()) s.send(b'QUIT\r\n'); s.close()" 220-mail.viktorbarzin.me ESMTP ... 250-mail.viktorbarzin.me 250-SIZE 209715200 ... 221 2.0.0 Bye $ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \ | grep smtpd-proxy.*CONNECT postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525 ``` Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy SNAT) → PROXY-v2 roundtrip confirmed. ### Manual Verification Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the now-persisted config (`<enable>yes</enable>` in XML). Connection test above should still work. ## Reproduce locally 1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/` 2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK 3. `python3 -c '...' ` SMTP roundtrip test above.
2026-04-19 12:07:47 +00:00
));
[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] ## Context (bd code-yiu) Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local, pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real client IP now preserved end-to-end on ports 25/465/587/993, both for postscreen anti-spam scoring and CrowdSec auth-failure bans. ## This change ### k8s (stacks/mailserver/modules/mailserver/main.tf) - `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3 alt PROXY-speaking services to master.cf: - `:2525` postscreen (alt :25) - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS) - `:5587` smtpd (alt :587 submission) All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`. Mirror stock submission/submissions options (SASL via Dovecot, TLS, client restrictions, mua_sender_restrictions). chroot=n so the SASL socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot. - `dovecot.cf` ConfigMap adds: ``` haproxy_trusted_networks = 10.0.20.0/24 service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } } ``` Stock :993 stays PROXY-free for internal Roundcube/probe clients. - Container ports: 4 new (4465, 5587, 10993, 2525 already there). - `mailserver-proxy` NodePort Service now exposes all 4 ports: 25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128 (ETP:Cluster). ### pfSense (scripts/pfsense-haproxy-bootstrap.php) Rebuilt to declare 4 backend pools (one per NodePort) and 4 production frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy `:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`. Idempotent — re-runs converge on declared state. ### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php) Flip script: updates `<nat><rule>` entries for mail ports from target `<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the rollback. Both scripts are idempotent. ## What is NOT in this change - Phase 6 (decommission MetalLB LB path, downgrade mailserver Service from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do NOT run until explicit approval. - Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock ETP:Local ports — functional backup path + consumed by internal clients that hit `mailserver.mailserver.svc.cluster.local` (routes via ClusterIP layer of the LB Service, bypassing ETP). - Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via unchanged NAT rule. ## Test Plan ### Automated (verified pre-commit 2026-04-19) ``` # k8s container listens on all 8 ports $ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \ -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b' ... all 8 listening ... # pfSense HAProxy listens on all 5 (production + legacy test) $ ssh admin@10.0.20.1 'sockstat -l | grep haproxy' www haproxy 49418 5 tcp4 *:25 www haproxy 49418 6 tcp4 *:2525 www haproxy 49418 10 tcp4 *:465 www haproxy 49418 11 tcp4 *:587 www haproxy 49418 12 tcp4 *:993 # Post-flip: pf rdr rules point at pfSense, not <mailserver> $ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25' rdr on vtnet0 ... port = submission -> 10.0.20.1 rdr on vtnet0 ... port = imaps -> 10.0.20.1 rdr on vtnet0 ... port = smtps -> 10.0.20.1 rdr on vtnet0 ... port = 25 -> 10.0.20.1 # 4 HAProxy frontends reachable + SMTP/IMAP banners $ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly # Real client IP in maillog for external delivery via Brevo → MX postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25 postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334 # E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds $ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver ... Round-trip SUCCESS in 20.3s ... # Internal Roundcube path unchanged $ curl -sI https://mail.viktorbarzin.me/ → 302 (Authentik gate intact) # No mail alerts firing $ kubectl exec prometheus-server ... /api/v1/alerts | grep Email → (empty) ``` ### Rollback ``` scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/ ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php' ``` Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias. Pre-flip config snapshot also saved at `/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense. ## Phase roadmap (bd code-yiu) | Phase | Status | |---|---| | 1a | ✅ commit ef75c02f — alt :2525 listener + NodePort | | 2 | ✅ 2026-04-19 — HAProxy pkg installed on pfSense | | 3 | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML | | 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip | | 6 | ⏸ USER-GATED — MetalLB LB decommission after 48h observation |
2026-04-19 12:24:50 +00:00
// Legacy test frontend — :2525 — retained so SMTP roundtrip tests keep working
// without touching the real :25. Safe to remove once fully validated.
$h['ha_backends']['item'][] = build_frontend(
'mailserver_proxy_test',
'code-yiu Phase 2/3 test — PROXY v2 to k8s mailserver NodePort 30125 (alt port :2525)',
'10.0.20.1', '2525',
'mailserver_nodes'
);
// Production frontends — 4 ports listening on pfSense VLAN20 IP 10.0.20.1.
$h['ha_backends']['item'][] = build_frontend(
'mailserver_proxy_25',
'code-yiu Phase 4/5 — external SMTP (:25) via PROXY v2 → pod :2525 postscreen',
'10.0.20.1', '25',
'mailserver_nodes_smtp'
);
$h['ha_backends']['item'][] = build_frontend(
'mailserver_proxy_465',
'code-yiu Phase 4/5 — external SMTPS (:465) via PROXY v2 → pod :4465 smtpd',
'10.0.20.1', '465',
'mailserver_nodes_smtps'
);
$h['ha_backends']['item'][] = build_frontend(
'mailserver_proxy_587',
'code-yiu Phase 4/5 — external submission (:587) via PROXY v2 → pod :5587 smtpd',
'10.0.20.1', '587',
'mailserver_nodes_sub'
);
$h['ha_backends']['item'][] = build_frontend(
'mailserver_proxy_993',
'code-yiu Phase 4/5 — external IMAPS (:993) via PROXY v2 → pod :10993 Dovecot',
'10.0.20.1', '993',
'mailserver_nodes_imaps'
);
[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip] ## Context (bd code-yiu) Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so it lives in the nightly backup) of the PROXY-v2 migration. Test path only — listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525 postscreen. Real client IP verified in maillog (`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a container plumbing is already live (commit ef75c02f). pfSense HAProxy config lives in `/cf/conf/config.xml` under `<installedpackages><haproxy>`. That file is captured daily by `scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`) and synced offsite to Synology. No new backup wiring needed — this commit documents the fact + adds the reproducer script. ## This change Two files, both additive: 1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that edits pfSense config.xml to add: - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125, `send-proxy-v2`, TCP health-check every 120000 ms (2 min). - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525 in TCP mode, forwarding to the pool. Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg` and reload HAProxy. Removes existing items with the same name before adding, so repeat runs converge on declared state. 2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering current state, validation, bootstrap/restore, health checks, phase roadmap, and known warts (health-check noise + bind-address templating). ## What is NOT in this change - Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred. - Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual- inet_listener) — deferred. - Terraform for pfSense HAProxy pkg install — not possible (no Terraform provider for pfSense pkg management). Runbook documents the manual `pkg install` command. ## Test Plan ### Automated ``` $ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525' 64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D www haproxy 64009 5 tcp4 *:2525 *:* $ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \ | awk 'NR>1 {print $4, $6}' node1 2 node2 2 node3 2 node4 2 # all UP $ python3 -c " import socket; s=socket.socket(); s.settimeout(10) s.connect(('10.0.20.1', 2525)) print(s.recv(200).decode()) s.send(b'EHLO persist-test.example.com\r\n') print(s.recv(500).decode()) s.send(b'QUIT\r\n'); s.close()" 220-mail.viktorbarzin.me ESMTP ... 250-mail.viktorbarzin.me 250-SIZE 209715200 ... 221 2.0.0 Bye $ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \ | grep smtpd-proxy.*CONNECT postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525 ``` Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy SNAT) → PROXY-v2 roundtrip confirmed. ### Manual Verification Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the now-persisted config (`<enable>yes</enable>` in XML). Connection test above should still work. ## Reproduce locally 1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/` 2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK 3. `python3 -c '...' ` SMTP roundtrip test above.
2026-04-19 12:07:47 +00:00
[mailserver] Phase 4+5 — pfSense HAProxy cutover for all 4 mail ports [ci skip] ## Context (bd code-yiu) Cutover of external mail traffic from the MetalLB LB IP path (ETP:Local, pod-speaker colocation) to pfSense HAProxy + PROXY v2 (ETP:Cluster). Real client IP now preserved end-to-end on ports 25/465/587/993, both for postscreen anti-spam scoring and CrowdSec auth-failure bans. ## This change ### k8s (stacks/mailserver/modules/mailserver/main.tf) - `mailserver-user-patches` ConfigMap's `user-patches.sh` now appends 3 alt PROXY-speaking services to master.cf: - `:2525` postscreen (alt :25) - `:4465` smtpd (alt :465 SMTPS, wrappermode TLS) - `:5587` smtpd (alt :587 submission) All with `postscreen_upstream_proxy_protocol=haproxy` / `smtpd_upstream_proxy_protocol=haproxy`. Mirror stock submission/submissions options (SASL via Dovecot, TLS, client restrictions, mua_sender_restrictions). chroot=n so the SASL socket path `/dev/shm/sasl-auth.sock` resolves outside the chroot. - `dovecot.cf` ConfigMap adds: ``` haproxy_trusted_networks = 10.0.20.0/24 service imap-login { inet_listener imaps_proxy { port=10993; ssl=yes; haproxy=yes } } ``` Stock :993 stays PROXY-free for internal Roundcube/probe clients. - Container ports: 4 new (4465, 5587, 10993, 2525 already there). - `mailserver-proxy` NodePort Service now exposes all 4 ports: 25→2525→30125, 465→4465→30126, 587→5587→30127, 993→10993→30128 (ETP:Cluster). ### pfSense (scripts/pfsense-haproxy-bootstrap.php) Rebuilt to declare 4 backend pools (one per NodePort) and 4 production frontends on `10.0.20.1:{25,465,587,993}` TCP mode, plus the legacy `:2525` test frontend. All pools: `send-proxy-v2 check inter 120000`. Idempotent — re-runs converge on declared state. ### pfSense (scripts/pfsense-nat-mailserver-haproxy-{flip,unflip}.php) Flip script: updates `<nat><rule>` entries for mail ports from target `<mailserver>` alias (10.0.20.202 MetalLB) → `10.0.20.1` (pfSense HAProxy). Runs `filter_configure()` to rebuild pf rules. Unflip is the rollback. Both scripts are idempotent. ## What is NOT in this change - Phase 6 (decommission MetalLB LB path, downgrade mailserver Service from LoadBalancer to ClusterIP, free 10.0.20.202) — USER-GATED. Do NOT run until explicit approval. - Legacy MetalLB `mailserver` LB still live on 10.0.20.202 with stock ETP:Local ports — functional backup path + consumed by internal clients that hit `mailserver.mailserver.svc.cluster.local` (routes via ClusterIP layer of the LB Service, bypassing ETP). - Port :143 (plain IMAP) — no HAProxy frontend; stays on MetalLB via unchanged NAT rule. ## Test Plan ### Automated (verified pre-commit 2026-04-19) ``` # k8s container listens on all 8 ports $ kubectl exec -c docker-mailserver deployment/mailserver -n mailserver \ -- ss -ltn | grep -E ':(25|2525|465|4465|587|5587|993|10993)\b' ... all 8 listening ... # pfSense HAProxy listens on all 5 (production + legacy test) $ ssh admin@10.0.20.1 'sockstat -l | grep haproxy' www haproxy 49418 5 tcp4 *:25 www haproxy 49418 6 tcp4 *:2525 www haproxy 49418 10 tcp4 *:465 www haproxy 49418 11 tcp4 *:587 www haproxy 49418 12 tcp4 *:993 # Post-flip: pf rdr rules point at pfSense, not <mailserver> $ ssh admin@10.0.20.1 'pfctl -sn' | grep 'smtp\|sub\|imap\|:25' rdr on vtnet0 ... port = submission -> 10.0.20.1 rdr on vtnet0 ... port = imaps -> 10.0.20.1 rdr on vtnet0 ... port = smtps -> 10.0.20.1 rdr on vtnet0 ... port = 25 -> 10.0.20.1 # 4 HAProxy frontends reachable + SMTP/IMAP banners $ python3 <test script> → SMTP/SMTPS/Sub/IMAPS all respond correctly # Real client IP in maillog for external delivery via Brevo → MX postfix/smtpd-proxy25/postscreen: CONNECT from [77.32.148.26]:36334 to [10.0.20.1]:25 postfix/smtpd-proxy25/postscreen: PASS NEW [77.32.148.26]:36334 # E2E probe (Brevo HTTP → external SMTP delivery → IMAP fetch) succeeds $ kubectl create job --from=cronjob/email-roundtrip-monitor probe-yiu-flip -n mailserver ... Round-trip SUCCESS in 20.3s ... # Internal Roundcube path unchanged $ curl -sI https://mail.viktorbarzin.me/ → 302 (Authentik gate intact) # No mail alerts firing $ kubectl exec prometheus-server ... /api/v1/alerts | grep Email → (empty) ``` ### Rollback ``` scp infra/scripts/pfsense-nat-mailserver-haproxy-unflip.php admin@10.0.20.1:/tmp/ ssh admin@10.0.20.1 'php /tmp/pfsense-nat-mailserver-haproxy-unflip.php' ``` Immediate (<2s). Flips all 4 NAT rdrs back to `<mailserver>` alias. Pre-flip config snapshot also saved at `/tmp/config.xml.pre-yiu-flip.20260419-1222` on pfSense. ## Phase roadmap (bd code-yiu) | Phase | Status | |---|---| | 1a | ✅ commit ef75c02f — alt :2525 listener + NodePort | | 2 | ✅ 2026-04-19 — HAProxy pkg installed on pfSense | | 3 | ✅ commit ba697b02 — HAProxy config persisted in pfSense XML | | 4+5| ✅ **this commit** — 4-port alt listeners + HAProxy frontends + NAT flip | | 6 | ⏸ USER-GATED — MetalLB LB decommission after 48h observation |
2026-04-19 12:24:50 +00:00
write_config('code-yiu: mailserver HAProxy — 4 production frontends + legacy :2525 test');
[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip] ## Context (bd code-yiu) Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so it lives in the nightly backup) of the PROXY-v2 migration. Test path only — listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525 postscreen. Real client IP verified in maillog (`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a container plumbing is already live (commit ef75c02f). pfSense HAProxy config lives in `/cf/conf/config.xml` under `<installedpackages><haproxy>`. That file is captured daily by `scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`) and synced offsite to Synology. No new backup wiring needed — this commit documents the fact + adds the reproducer script. ## This change Two files, both additive: 1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that edits pfSense config.xml to add: - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125, `send-proxy-v2`, TCP health-check every 120000 ms (2 min). - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525 in TCP mode, forwarding to the pool. Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg` and reload HAProxy. Removes existing items with the same name before adding, so repeat runs converge on declared state. 2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering current state, validation, bootstrap/restore, health checks, phase roadmap, and known warts (health-check noise + bind-address templating). ## What is NOT in this change - Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred. - Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual- inet_listener) — deferred. - Terraform for pfSense HAProxy pkg install — not possible (no Terraform provider for pfSense pkg management). Runbook documents the manual `pkg install` command. ## Test Plan ### Automated ``` $ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525' 64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D www haproxy 64009 5 tcp4 *:2525 *:* $ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \ | awk 'NR>1 {print $4, $6}' node1 2 node2 2 node3 2 node4 2 # all UP $ python3 -c " import socket; s=socket.socket(); s.settimeout(10) s.connect(('10.0.20.1', 2525)) print(s.recv(200).decode()) s.send(b'EHLO persist-test.example.com\r\n') print(s.recv(500).decode()) s.send(b'QUIT\r\n'); s.close()" 220-mail.viktorbarzin.me ESMTP ... 250-mail.viktorbarzin.me 250-SIZE 209715200 ... 221 2.0.0 Bye $ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \ | grep smtpd-proxy.*CONNECT postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525 ``` Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy SNAT) → PROXY-v2 roundtrip confirmed. ### Manual Verification Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the now-persisted config (`<enable>yes</enable>` in XML). Connection test above should still work. ## Reproduce locally 1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/` 2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK 3. `python3 -c '...' ` SMTP roundtrip test above.
2026-04-19 12:07:47 +00:00
$messages = '';
$rc = haproxy_check_and_run($messages, true);
echo 'haproxy_check_and_run rc=' . ($rc ? 'OK' : 'FAIL') . "\n";
echo "messages: $messages\n";