4c8d12229f
Two coordinated fixes for the same root cause: Postfix's smtpd_upstream_proxy_protocol
listener fatals on every HAProxy health probe with `smtpd_peer_hostaddr_to_sockaddr:
... Servname not supported for ai_socktype` — the daemon respawns get throttled by
postfix master, and real client connections that land mid-respawn time out. We saw
this as ~50% timeout rate on public 587 from inside the cluster.
Layer 1 (book-search) — stacks/ebooks/main.tf:
SMTP_HOST mail.viktorbarzin.me → mailserver.mailserver.svc.cluster.local
Internal services should use ClusterIP, not hairpin through pfSense+HAProxy.
12/12 OK in <28ms vs ~6/12 timeouts on the public path.
Layer 2 (pfSense HAProxy) — stacks/mailserver + scripts/pfsense-haproxy-bootstrap.php:
Add 3 non-PROXY healthcheck NodePorts to mailserver-proxy svc:
30145 → pod 25 (stock postscreen)
30146 → pod 465 (stock smtps)
30147 → pod 587 (stock submission)
HAProxy uses `port <healthcheck-nodeport>` (per-server in advanced field) to
redirect L4 health probes to those ports while real client traffic keeps
going to 30125-30128 with PROXY v2.
Result: 0 fatals/min (was 96), 30/30 probes OK on 587, e2e roundtrip 20.4s.
Inter dropped 120000 → 5000 since log-spam concern is gone.
`option smtpchk EHLO` was tried first but flapped against postscreen (multi-line
greet + DNSBL silence + anti-pre-greet detection trip HAProxy's parser → L7RSP).
Plain TCP accept-on-port check is sufficient for both submission and postscreen.
Updated docs/runbooks/mailserver-pfsense-haproxy.md to reflect the new healthcheck
path and mark the "Known warts" entry as resolved.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
236 lines
9.7 KiB
PHP
236 lines
9.7 KiB
PHP
<?php
|
|
// pfSense HAProxy bootstrap — configures the mailserver PROXY-v2 path
|
|
// (bd code-yiu, Phases 2/3 + 5).
|
|
//
|
|
// WHY THIS EXISTS
|
|
// pfSense HAProxy config is stored XML-in-`/cf/conf/config.xml` under
|
|
// `<installedpackages><haproxy>`. That file IS picked up by the nightly
|
|
// `daily-backup` on the PVE host (see `scripts/daily-backup.sh` → `scp
|
|
// root@10.0.20.1:/cf/conf/config.xml`) and synced to Synology. This script
|
|
// is the canonical reproducer: run it to rebuild the pfSense HAProxy config
|
|
// from scratch (DR restore, fresh pfSense install, etc.).
|
|
//
|
|
// WHAT IT BUILDS
|
|
// 4 backend pools — one per mail port:
|
|
// mailserver_nodes_smtp → k8s-node1..4:30125 (container :2525 postscreen)
|
|
// mailserver_nodes_smtps → k8s-node1..4:30126 (container :4465 smtps)
|
|
// mailserver_nodes_sub → k8s-node1..4:30127 (container :5587 submission)
|
|
// mailserver_nodes_imaps → k8s-node1..4:30128 (container :10993 IMAPS)
|
|
// Each server uses `send-proxy-v2` and TCP health-check every 120s.
|
|
// 4 frontends on pfSense 10.0.20.1:{25,465,587,993} TCP mode.
|
|
// + 1 legacy test frontend on :2525 (kept for validation; safe to remove later).
|
|
//
|
|
// USAGE (on pfSense host, via SSH as admin)
|
|
// scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/
|
|
// ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'
|
|
//
|
|
// IDEMPOTENCY
|
|
// Removes any existing entries named mailserver_* before re-adding, so
|
|
// repeat runs are safe and behave as reset-to-declared.
|
|
|
|
require_once('/etc/inc/config.inc');
|
|
require_once('/usr/local/pkg/haproxy/haproxy.inc');
|
|
require_once('/usr/local/pkg/haproxy/haproxy_utils.inc');
|
|
|
|
global $config;
|
|
parse_config(true);
|
|
|
|
if (!is_array($config['installedpackages']['haproxy'])) {
|
|
$config['installedpackages']['haproxy'] = [];
|
|
}
|
|
$h = &$config['installedpackages']['haproxy'];
|
|
|
|
$h['enable'] = 'yes';
|
|
$h['maxconn'] = '1000';
|
|
|
|
// Our declared object names (anything starting with mailserver_ is ours)
|
|
$POOL_NAMES = [
|
|
'mailserver_nodes', // legacy (Phase 2/3 test)
|
|
'mailserver_nodes_smtp',
|
|
'mailserver_nodes_smtps',
|
|
'mailserver_nodes_sub',
|
|
'mailserver_nodes_imaps',
|
|
];
|
|
$FRONTEND_NAMES = [
|
|
'mailserver_proxy_test', // legacy (Phase 2/3 test, :2525)
|
|
'mailserver_proxy_25',
|
|
'mailserver_proxy_465',
|
|
'mailserver_proxy_587',
|
|
'mailserver_proxy_993',
|
|
];
|
|
|
|
// k8s workers. Not in the cluster: master (control-plane) and node5
|
|
// (doesn't exist in this topology).
|
|
$NODES = [
|
|
['k8s-node1', '10.0.20.101'],
|
|
['k8s-node2', '10.0.20.102'],
|
|
['k8s-node3', '10.0.20.103'],
|
|
['k8s-node4', '10.0.20.104'],
|
|
];
|
|
|
|
// Build a pool with optional split healthcheck path.
|
|
//
|
|
// $check_port: if non-null, HAProxy sends health probes to that NodePort
|
|
// (which Service `mailserver-proxy` maps to the pod's stock no-PROXY
|
|
// listener — see infra/stacks/mailserver/.../mailserver_proxy ports
|
|
// 30145/30146/30147). Real client traffic still goes to $nodeport with
|
|
// PROXY v2 framing.
|
|
// $check_type: 'TCP' for plain accept-on-port checks, 'ESMTP' for
|
|
// `option smtpchk EHLO <monitor_domain>` (real SMTP banner+EHLO+250).
|
|
//
|
|
// Why split: smtpd-proxy587/4465 fatal on every PROXY-v2-aware health
|
|
// probe with `smtpd_peer_hostaddr_to_sockaddr: ... Servname not supported`
|
|
// — the daemon respawns get throttled by Postfix master and real clients
|
|
// land mid-respawn → 6s TCP timeout. Routing health probes to the stock
|
|
// no-PROXY port sidesteps the bug entirely while data path still gets
|
|
// PROXY v2 for CrowdSec/Postfix client-IP visibility. The HAProxy package
|
|
// has no `checkport` field, so `port N` is appended via the server's
|
|
// `advanced` string (HAProxy parses server keywords in any order).
|
|
function build_pool(
|
|
string $name,
|
|
string $nodeport,
|
|
array $nodes,
|
|
string $check_type = 'TCP',
|
|
?string $check_port = null,
|
|
string $monitor_domain = ''
|
|
): array {
|
|
$advanced_check = $check_port !== null
|
|
? "send-proxy-v2 port {$check_port}"
|
|
: 'send-proxy-v2';
|
|
$servers = [];
|
|
foreach ($nodes as $n) {
|
|
$servers[] = [
|
|
'name' => $n[0],
|
|
'address' => $n[1],
|
|
'port' => $nodeport,
|
|
'weight' => '10',
|
|
'ssl' => '',
|
|
// 5s = sub-block-window failover when a NodePort goes sour.
|
|
// Safe to be aggressive once health probes don't fatal smtpd.
|
|
'checkinter' => '5000',
|
|
'advanced' => $advanced_check,
|
|
'status' => 'active',
|
|
];
|
|
}
|
|
return [
|
|
'name' => $name,
|
|
'balance' => 'roundrobin',
|
|
'check_type' => $check_type,
|
|
'monitor_domain' => $monitor_domain,
|
|
'checkinter' => '5000',
|
|
'retries' => '3',
|
|
'ha_servers' => ['item' => $servers],
|
|
'advanced_bind' => '',
|
|
'persist_cookie_enabled' => '',
|
|
'transparent_clientip' => '',
|
|
'advanced' => '',
|
|
];
|
|
}
|
|
|
|
function build_frontend(string $name, string $descr, string $extaddr, string $port, string $pool): array {
|
|
return [
|
|
'name' => $name,
|
|
'descr' => $descr,
|
|
'status' => 'active',
|
|
'secondary' => '',
|
|
'type' => 'tcp',
|
|
'a_extaddr' => ['item' => [[
|
|
'extaddr' => $extaddr,
|
|
'extaddr_port' => $port,
|
|
'extaddr_ssl' => '',
|
|
'extaddr_advanced' => '',
|
|
]]],
|
|
'backend_serverpool' => $pool,
|
|
'ha_acls' => '',
|
|
'dontlognull'=> '',
|
|
'httpclose' => '',
|
|
'forwardfor' => '',
|
|
'advanced' => '',
|
|
];
|
|
}
|
|
|
|
// ── Backend pools ───────────────────────────────────────────────────────
|
|
if (!is_array($h['ha_pools'])) $h['ha_pools'] = ['item' => []];
|
|
if (!is_array($h['ha_pools']['item'])) $h['ha_pools']['item'] = [];
|
|
$h['ha_pools']['item'] = array_values(array_filter(
|
|
$h['ha_pools']['item'],
|
|
fn($p) => !in_array($p['name'] ?? '', $POOL_NAMES, true)
|
|
));
|
|
|
|
// Legacy test pool (still used by the :2525 test frontend for manual SMTP roundtrip).
|
|
$h['ha_pools']['item'][] = build_pool('mailserver_nodes', '30125', $NODES);
|
|
|
|
// Production pools — one per mail port.
|
|
//
|
|
// All SMTP/SMTPS/Submission backends use plain TCP checks against
|
|
// dedicated non-PROXY healthcheck NodePorts (30145/30146/30147 → pod
|
|
// stock 25/465/587) so probes hit the no-PROXY listeners and avoid
|
|
// the smtpd_peer_hostaddr_to_sockaddr fatal that fires on PROXY-v2
|
|
// LOCAL frames. Real client traffic still goes to 30125-30128 with
|
|
// PROXY v2 for client-IP visibility.
|
|
//
|
|
// We tried `option smtpchk EHLO` initially — it works on the plain
|
|
// `submission` daemon (587) but flaps the `postscreen` listener on
|
|
// port 25 (multi-line greet + DNSBL silence + anti-pre-greet
|
|
// detection makes HAProxy's simple smtpchk parser hit L7RSP). A
|
|
// plain TCP accept-on-port check is enough for both: HAProxy still
|
|
// gets fast failover when the listener actually goes away, and we
|
|
// stop triggering the Postfix fatal entirely.
|
|
//
|
|
// IMAPS stays on its existing TCP-check-with-PROXY-frame for now —
|
|
// Dovecot's PROXY parser doesn't show the same fatal pattern; adding
|
|
// a separate IMAP healthcheck path would require another svc port.
|
|
$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtp', '30125', $NODES, 'TCP', '30145');
|
|
$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtps', '30126', $NODES, 'TCP', '30146');
|
|
$h['ha_pools']['item'][] = build_pool('mailserver_nodes_sub', '30127', $NODES, 'TCP', '30147');
|
|
$h['ha_pools']['item'][] = build_pool('mailserver_nodes_imaps', '30128', $NODES);
|
|
|
|
// ── Frontends ───────────────────────────────────────────────────────────
|
|
if (!is_array($h['ha_backends'])) $h['ha_backends'] = ['item' => []];
|
|
if (!is_array($h['ha_backends']['item'])) $h['ha_backends']['item'] = [];
|
|
$h['ha_backends']['item'] = array_values(array_filter(
|
|
$h['ha_backends']['item'],
|
|
fn($f) => !in_array($f['name'] ?? '', $FRONTEND_NAMES, true)
|
|
));
|
|
|
|
// Legacy test frontend — :2525 — retained so SMTP roundtrip tests keep working
|
|
// without touching the real :25. Safe to remove once fully validated.
|
|
$h['ha_backends']['item'][] = build_frontend(
|
|
'mailserver_proxy_test',
|
|
'code-yiu Phase 2/3 test — PROXY v2 to k8s mailserver NodePort 30125 (alt port :2525)',
|
|
'10.0.20.1', '2525',
|
|
'mailserver_nodes'
|
|
);
|
|
|
|
// Production frontends — 4 ports listening on pfSense VLAN20 IP 10.0.20.1.
|
|
$h['ha_backends']['item'][] = build_frontend(
|
|
'mailserver_proxy_25',
|
|
'code-yiu Phase 4/5 — external SMTP (:25) via PROXY v2 → pod :2525 postscreen',
|
|
'10.0.20.1', '25',
|
|
'mailserver_nodes_smtp'
|
|
);
|
|
$h['ha_backends']['item'][] = build_frontend(
|
|
'mailserver_proxy_465',
|
|
'code-yiu Phase 4/5 — external SMTPS (:465) via PROXY v2 → pod :4465 smtpd',
|
|
'10.0.20.1', '465',
|
|
'mailserver_nodes_smtps'
|
|
);
|
|
$h['ha_backends']['item'][] = build_frontend(
|
|
'mailserver_proxy_587',
|
|
'code-yiu Phase 4/5 — external submission (:587) via PROXY v2 → pod :5587 smtpd',
|
|
'10.0.20.1', '587',
|
|
'mailserver_nodes_sub'
|
|
);
|
|
$h['ha_backends']['item'][] = build_frontend(
|
|
'mailserver_proxy_993',
|
|
'code-yiu Phase 4/5 — external IMAPS (:993) via PROXY v2 → pod :10993 Dovecot',
|
|
'10.0.20.1', '993',
|
|
'mailserver_nodes_imaps'
|
|
);
|
|
|
|
write_config('code-yiu: mailserver HAProxy — 4 production frontends + legacy :2525 test');
|
|
|
|
$messages = '';
|
|
$rc = haproxy_check_and_run($messages, true);
|
|
echo 'haproxy_check_and_run rc=' . ($rc ? 'OK' : 'FAIL') . "\n";
|
|
echo "messages: $messages\n";
|