[mailserver] Phase 2-3 — pfSense HAProxy bootstrap + runbook [ci skip]

## Context (bd code-yiu)

Phase 2 (HAProxy on pfSense) and Phase 3 (persist config in pfSense XML so
it lives in the nightly backup) of the PROXY-v2 migration. Test path only —
listens on pfSense 10.0.20.1:2525 → k8s node NodePort :30125 → pod :2525
postscreen. Real client IP verified in maillog
(`postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:...`), Phase 1a
container plumbing is already live (commit ef75c02f).

pfSense HAProxy config lives in `/cf/conf/config.xml` under
`<installedpackages><haproxy>`. That file is captured daily by
`scripts/daily-backup.sh` (scp → `/mnt/backup/pfsense/config-YYYYMMDD.xml`)
and synced offsite to Synology. No new backup wiring needed — this commit
documents the fact + adds the reproducer script.

## This change

Two files, both additive:

1. `scripts/pfsense-haproxy-bootstrap.php` — idempotent PHP script that
   edits pfSense config.xml to add:
   - Backend pool `mailserver_nodes` with 4 k8s workers on NodePort 30125,
     `send-proxy-v2`, TCP health-check every 120000 ms (2 min).
   - Frontend `mailserver_proxy_test` listening on pfSense 10.0.20.1:2525
     in TCP mode, forwarding to the pool.
   Uses `haproxy_check_and_run()` to regenerate `/var/etc/haproxy/haproxy.cfg`
   and reload HAProxy. Removes existing items with the same name before
   adding, so repeat runs converge on declared state.

2. `docs/runbooks/mailserver-pfsense-haproxy.md` — ops runbook covering
   current state, validation, bootstrap/restore, health checks, phase
   roadmap, and known warts (health-check noise + bind-address templating).

## What is NOT in this change

- Phase 4 (NAT rdr flip for :25 from `<mailserver>` → HAProxy) — deferred.
- Phase 5 (extend to 465/587/993 with alt listeners + Dovecot dual-
  inet_listener) — deferred.
- Terraform for pfSense HAProxy pkg install — not possible (no Terraform
  provider for pfSense pkg management). Runbook documents the manual
  `pkg install` command.

## Test Plan

### Automated
```
$ ssh admin@10.0.20.1 'pgrep -lf haproxy; sockstat -l | grep :2525'
64009 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D
www  haproxy  64009 5 tcp4  *:2525  *:*

$ ssh admin@10.0.20.1 "echo 'show servers state' | socat /tmp/haproxy.socket stdio" \
    | awk 'NR>1 {print $4, $6}'
node1 2
node2 2
node3 2
node4 2        # all UP

$ python3 -c "
import socket; s=socket.socket(); s.settimeout(10)
s.connect(('10.0.20.1', 2525))
print(s.recv(200).decode())
s.send(b'EHLO persist-test.example.com\r\n')
print(s.recv(500).decode())
s.send(b'QUIT\r\n'); s.close()"
220-mail.viktorbarzin.me ESMTP
...
250-mail.viktorbarzin.me
250-SIZE 209715200
...
221 2.0.0 Bye

$ kubectl logs -c docker-mailserver deployment/mailserver -n mailserver --tail=50 \
    | grep smtpd-proxy.*CONNECT
postfix/smtpd-proxy/postscreen: CONNECT from [10.0.10.10]:33010 to [10.0.20.1]:2525
```

Real client IP `[10.0.10.10]` visible (not the k8s-node IP after kube-proxy
SNAT) → PROXY-v2 roundtrip confirmed.

### Manual Verification
Trigger a pfSense reboot; after boot, HAProxy should auto-restart from the
now-persisted config (`<enable>yes</enable>` in XML). Connection test above
should still work.

## Reproduce locally
1. `scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/`
2. `ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'` → rc=OK
3. `python3 -c '...' ` SMTP roundtrip test above.

scripts/pfsense-haproxy-bootstrap.php

@@ -0,0 +1,117 @@
+<?php
+// pfSense HAProxy bootstrap — adds/refreshes the mailserver_proxy_test
+// frontend + mailserver_nodes backend for code-yiu (PROXY-v2 SMTP path).
+//
+// WHY THIS EXISTS
+//   pfSense HAProxy config is stored XML-in-`/cf/conf/config.xml` under
+//   `<installedpackages><haproxy>`. That file IS picked up by the nightly
+//   `daily-backup` on the PVE host (see `scripts/daily-backup.sh` → `scp
+//   root@10.0.20.1:/cf/conf/config.xml`) and synced to Synology. This script
+//   is the canonical reproducer: run it to rebuild the pfSense HAProxy config
+//   from scratch (DR restore, fresh pfSense install, etc.).
+//
+// WHAT IT BUILDS
+//   Backend pool `mailserver_nodes`: 4 k8s workers on NodePort 30125 with
+//   `send-proxy-v2` + TCP health-check every 120s.
+//   Frontend `mailserver_proxy_test`: listens on 10.0.20.1:2525, TCP mode,
+//   forwards to the pool above.
+//
+// USAGE (on pfSense host, via SSH as admin)
+//   scp infra/scripts/pfsense-haproxy-bootstrap.php admin@10.0.20.1:/tmp/
+//   ssh admin@10.0.20.1 'php /tmp/pfsense-haproxy-bootstrap.php'
+//
+// IDEMPOTENCY
+//   Removes any existing entries named `mailserver_nodes` / `mailserver_proxy_test`
+//   before re-adding, so repeat runs are safe and behave as reset-to-declared.
+
+require_once('/etc/inc/config.inc');
+require_once('/usr/local/pkg/haproxy/haproxy.inc');
+require_once('/usr/local/pkg/haproxy/haproxy_utils.inc');
+
+global $config;
+parse_config(true);
+
+if (!is_array($config['installedpackages']['haproxy'])) {
+    $config['installedpackages']['haproxy'] = [];
+}
+$h = &$config['installedpackages']['haproxy'];
+
+$h['enable']  = 'yes';
+$h['maxconn'] = '1000';
+
+// ── Backend pool ────────────────────────────────────────────────────────
+if (!is_array($h['ha_pools'])) $h['ha_pools'] = ['item' => []];
+if (!is_array($h['ha_pools']['item'])) $h['ha_pools']['item'] = [];
+$h['ha_pools']['item'] = array_values(array_filter(
+    $h['ha_pools']['item'],
+    fn($p) => ($p['name'] ?? '') !== 'mailserver_nodes'
+));
+
+$servers = [];
+foreach ([
+    ['k8s-node1', '10.0.20.101'],
+    ['k8s-node2', '10.0.20.102'],
+    ['k8s-node3', '10.0.20.103'],
+    ['k8s-node4', '10.0.20.104'],
+] as $n) {
+    $servers[] = [
+        'name'       => $n[0],
+        'address'    => $n[1],
+        'port'       => '30125',
+        'weight'     => '10',
+        'ssl'        => '',
+        // check every 2 minutes to avoid flooding postscreen with
+        // send-proxy-v2 + immediate close connections (see bd code-yiu notes).
+        'checkinter' => '120000',
+        'advanced'   => 'send-proxy-v2',
+        'status'     => 'active',
+    ];
+}
+
+$h['ha_pools']['item'][] = [
+    'name'                   => 'mailserver_nodes',
+    'balance'                => 'roundrobin',
+    'check_type'             => 'TCP',
+    'checkinter'             => '120000',
+    'retries'                => '3',
+    'ha_servers'             => ['item' => $servers],
+    'advanced_bind'          => '',
+    'persist_cookie_enabled' => '',
+    'transparent_clientip'   => '',
+    'advanced'               => '',
+];
+
+// ── Frontend (pfSense "ha_backends") ────────────────────────────────────
+if (!is_array($h['ha_backends'])) $h['ha_backends'] = ['item' => []];
+if (!is_array($h['ha_backends']['item'])) $h['ha_backends']['item'] = [];
+$h['ha_backends']['item'] = array_values(array_filter(
+    $h['ha_backends']['item'],
+    fn($f) => ($f['name'] ?? '') !== 'mailserver_proxy_test'
+));
+
+$h['ha_backends']['item'][] = [
+    'name'   => 'mailserver_proxy_test',
+    'descr'  => 'code-yiu Phase 3 test — PROXY v2 to k8s mailserver NodePort 30125',
+    'status' => 'active',
+    'secondary' => '',
+    'type'   => 'tcp',
+    'a_extaddr' => ['item' => [[
+        'extaddr'          => '10.0.20.1',
+        'extaddr_port'     => '2525',
+        'extaddr_ssl'      => '',
+        'extaddr_advanced' => '',
+    ]]],
+    'backend_serverpool' => 'mailserver_nodes',
+    'ha_acls'    => '',
+    'dontlognull'=> '',
+    'httpclose'  => '',
+    'forwardfor' => '',
+    'advanced'   => '',
+];
+
+write_config('code-yiu: mailserver_proxy HAProxy frontend + backend (bootstrap)');
+
+$messages = '';
+$rc = haproxy_check_and_run($messages, true);
+echo 'haproxy_check_and_run rc=' . ($rc ? 'OK' : 'FAIL') . "\n";
+echo "messages: $messages\n";