diff --git a/docs/adr/0019-backup-mx-roller-network-free-tier.md b/docs/adr/0019-backup-mx-roller-network-free-tier.md new file mode 100644 index 00000000..0cf40523 --- /dev/null +++ b/docs/adr/0019-backup-mx-roller-network-free-tier.md @@ -0,0 +1,65 @@ +# Inbound mail gets a free store-and-forward backup MX (Roller Network) + +`viktorbarzin.me` has run a single direct MX to the home IP since the 2026-04-12 +inbound overhaul, with sender-MTA retry (1–5 days, sender-dependent) as the only +outage protection — a documented "No Backup MX" decision made after ForwardEmail's +forced anti-spoofing rejected legitimate forwarded mail and Cloudflare Email +Routing proved pass-through-only (no queue). Viktor now wants inbound mail to +survive homelab outages **without loss** (2026-07-04): delayed delivery is fine, +mid-outage reading is not required, and the budget is **$0** — which rules out +the doc-flagged Dynu fallback ($9.99/yr). + +We adopt **Roller Network's free-tier Secondary MX** (`mail.rollernet.us` + +`mail2.rollernet.us` at equal MX preference 20, primary untouched): a +purpose-built store-and-forward relay with a **3-week queue** (sliding retries, +15 min doubling to 1-week max), **no forced spam filtering** on the secondary +path, and a valid-user table with a default-*allow-any* mode that preserves our +catch-all's infinite ad-hoc aliases. Our side whitelists their relay CIDRs in +postscreen (skip DNSBL/pregreet for queue drains) and exempts them from +SPF/DMARC *scoring* in rspamd — the ForwardEmail lesson applied at the right +layer; DKIM verification and content/AV scanning stay fully active. Go-live is +gated: confirm the free tier still includes Secondary MX, confirm the 10 MB/day +overage lock answers 4xx (defer) rather than 5xx (bounce), capture their +authoritative relay CIDRs, apply the whitelist **before** the MX records, and +finish with a live failover test (mailserver scaled to 0, probes from Gmail + +Brevo, verified queue-and-drain). Design: +[`plans/2026-07-04-backup-mx-rollernet-design.md`](../plans/2026-07-04-backup-mx-rollernet-design.md). + +## Considered options + +- **Dynu Email Backup ($9.99/yr)** — the previously doc-flagged option; simple, + but queue lifetime is undocumented (FAQ hints at 12–24 h retry ceilings), + filtering behaviour is a black box, and it costs money the free requirement + excludes. +- **Self-hosted VPS relay** (Hetzner ~€50/yr, or Oracle Always-Free at $0) — + full control (30-day queue, own TLS/MTA-STS story), but a second + internet-facing pet to patch and monitor; Oracle hard-blocks egress port 25, + forcing delivery to the primary on a custom port, and idles risk free-tier + reclamation. +- **Cloudflare Email Routing / mailflare** — no store-and-forward (pass-through + only) / a terminal inbox on Cloudflare respectively; both previously + evaluated and rejected (2026-04-12; 2026-07-04, memory #7148). +- **Harden-only** (guard hard-5xx misconfig modes, add paging) — cheaper but + does not address multi-day outages or short-retry senders; deferred as a + complementary track, not an alternative. + +## Consequences + +- Outage mail queues **in plaintext at a third party** for up to 3 weeks — + accepted; same trust class as Brevo holding our outbound relay traffic. +- The backup path bypasses postscreen DNSBL and SPF/DMARC scoring for + Rollernet's CIDRs; content/AV/Bayes and DKIM verification still apply. A + slight spam uptick during outages is possible (catch-all absorbs to `spam@`). +- The free tier's **10 MB/day cap** locks the domain until midnight Pacific + when exceeded; the G2 gate decides whether that lock defers (harmless) or + bounces (revisit: paid tier or accept). Overage never affects the primary + path — only mail arriving via the backup while locked. +- Two more records in a Cloudflare zone already near the Free-plan 200-record + cap (headroom must be verified at apply time). +- **MTA-STS was found dangling** during design: the `_mta-sts` TXT is published + but no policy host exists, so MTA-STS is inert today. Any future fix must + list the Rollernet MX hosts in the policy or enforcing senders will skip the + backup path. +- `architecture/mailserver.md` §"No Backup MX" is superseded at implementation + time; a new runbook covers ACC queue inspection, post-outage drain checks, + and Accept-and-Hold for planned maintenance. diff --git a/docs/plans/2026-07-04-backup-mx-rollernet-design.md b/docs/plans/2026-07-04-backup-mx-rollernet-design.md new file mode 100644 index 00000000..2cb737e9 --- /dev/null +++ b/docs/plans/2026-07-04-backup-mx-rollernet-design.md @@ -0,0 +1,179 @@ +# Backup MX via Roller Network free Secondary MX — design + +Date: 2026-07-04 · Status: design approved pending user review, pre-implementation · ADR: [0019](../adr/0019-backup-mx-roller-network-free-tier.md) + +## Goal + +Inbound mail for `viktorbarzin.me` must survive homelab outages without loss. +Requirement level (Viktor, 2026-07-04): **never lose mail; delayed delivery is +acceptable; budget is $0**. A store-and-forward backup MX queues mail while the +homelab is down and re-delivers when it returns. + +Out of scope, explicitly: + +- Reading new mail *during* an outage (would need a deliver-to-mailbox backup — + rejected in favour of queue-only). +- Outbound mail during outages. +- The "primary up but hard-bouncing 5xx" misconfig class (e.g. broken alias map + → `550 user unknown`): a backup MX is never consulted when the primary + answers. That is a separate hardening/alerting track. + +## Current state and gap + +- Single MX: `mail.viktorbarzin.me` (pri 1) → `176.12.22.76` → pfSense HAProxy + (PROXY v2) → mailserver pod. No backup MX — documented decision in + [`architecture/mailserver.md`](../architecture/mailserver.md) §"No Backup MX" + (2026-04-12), which this design supersedes (ADR-0019). +- Only protection today: sender MTAs queue and retry, typically 1–5 days. + Loss vectors: outages longer than a sender's retry window, and senders with + unusually short retry policies. +- Prior art: **ForwardEmail** relay abandoned 2026-04-12 (its forced + anti-spoofing rejected legitimate forwarded mail); **Cloudflare Email + Routing** rejected (pass-through only, no queue); **Dynu** ($9.99/yr) was the + doc-flagged fallback; **mailflare** (hieunc229) evaluated and rejected + 2026-07-04 (memory #7148). + +## Decision + +Adopt **Roller Network free-tier Secondary MX** (`mail.rollernet.us` + +`mail2.rollernet.us`) as a store-and-forward backup MX. Rationale (full +alternatives in ADR-0019): + +- Purpose-built queue relay: **3-week queue**, sliding retry (15 min doubling + to a max 1-week interval), queue storage not counted against the account. +- Spam filtering on secondary MX is **optional and off by default** ("little to + no spam filtering" per their FAQ) — avoids the ForwardEmail failure class. +- **Catch-all compatible**: their valid-user table supports a default *allow + any* ("catch-all/dropbox") action, preserving the `@viktorbarzin.me → spam@` + infinite-alias pattern. New domains default to *deny* — must be flipped + explicitly at setup. +- Free; unlimited domains; config API; "Accept and Hold" mode usable for + planned maintenance windows. + +## Architecture + +Normal operation (unchanged): senders resolve MX, prefer pri 1 +`mail.viktorbarzin.me`, deliver directly. Rollernet sits idle. (Spammers +deliberately targeting the backup MX get relayed to the primary immediately — +see failure modes.) + +Outage: senders fail to connect to pri 1 → fall back to pri 20 +`mail{,2}.rollernet.us` → Rollernet accepts (allow-any user table), queues up +to 3 weeks, retries the primary on a sliding schedule → queue drains +automatically after recovery, entering via the standard external path (pfSense +HAProxy → `:2525` postscreen, PROXY v2), then rspamd → Dovecot as usual. + +``` + ┌── pri 1 mail.viktorbarzin.me ──► pfSense HAProxy ──► mailserver pod +sender MTA ──► MX lookup ┤ ▲ + └── pri 20 mail.rollernet.us ─┐ │ retry ≤ 3 weeks + pri 20 mail2.rollernet.us ┴─► Rollernet queue ───────┘ + (only used when pri 1 unreachable) +``` + +## Rollernet account & configuration (out-of-band SaaS, like Brevo) + +- Account email: **`rollernet@viktorbarzin.me`** (Viktor, 2026-07-04; resolves + via catch-all → `spam@`). Known circularity: during an outage their + notifications to this address are themselves queued (at their side) until + recovery. Accepted — credentials and config live in Vault and the runbook + documents ACC access; nothing operational depends on receiving their mail + mid-outage. +- Credentials → Vault `secret/viktor` (`rollernet_password`, plus API key if + minted). +- Domain `viktorbarzin.me` in **Secondary MX** mode; valid-user table default + action = **allow any** (catch-all). +- `abuse@` / `postmaster@` must be deliverable (their RFC requirement) — the + catch-all already satisfies this. +- Record their **relay source CIDRs** from the post-signup Resource Access page + (feeds the whitelist below). Their published mail ranges as of 2026 include + `162.216.242.0/24` and `72.51.58.0/24` — confirm the authoritative list in + the ACC. + +## Our-side changes (all Terraform; worktree → master → CI apply) + +1. **DNS** — `stacks/cloudflared/modules/cloudflared/cloudflare.tf`: add two MX + records for the zone apex, `mail.rollernet.us` and `mail2.rollernet.us` at + equal preference **20** (primary record untouched at pri 1). Implementation + checks: (a) their MX-setup help page has a loop-avoidance rule about + priority layout — confirm 1/20/20 matches their prescription post-signup; + (b) **the zone sits near Cloudflare's Free-plan 200-record cap** (commit + `1a63fee4` dropped 6 names for headroom) — verify ≥2 free slots before + apply. +2. **Postscreen whitelist** — `stacks/mailserver/modules/mailserver/main.tf`: + mount a `postscreen_access.cidr` (permit Rollernet CIDRs) via the existing + config ConfigMap and set `postscreen_access_list = + permit_mynetworks, cidr:/tmp/docker-mailserver/postscreen_access.cidr` on + the `:2525` alt listener (in `user-patches.sh`, where the listener is + defined). Rationale: their relays must not be DNSBL-scored or + pregreet-tested — queue drains would tempfail/deferred otherwise. +3. **rspamd SPF/DMARC exemption** — same stack, via the established + `override.d`/`local.d` ConfigMap-mount pattern (as `dkim_signing.conf` + today): exempt the Rollernet CIDRs from **SPF and DMARC scoring only** + (relayed mail arrives from their IPs, so envelope SPF legitimately fails — + the exact ForwardEmail lesson applied on our side). Content, AV, Bayes and + DKIM verification stay fully active; DKIM-signed senders still validate + end-to-end through the relay. +4. **Monitoring** — blackbox DNS assertion that the MX set contains all three + hosts (drift guard, same pattern as `viktorbarzin-apex-probe`); alert on + drift. Optional informational probe: TCP:25 reachability of + `mail.rollernet.us` (their uptime, weekly cadence, no paging). +5. **Docs (same commit as implementation)** — rewrite `mailserver.md` §"No + Backup MX" (decision superseded by ADR-0019, new inbound flow + DNS table + + monitoring rows), add `docs/runbooks/backup-mx-rollernet.md` (ACC queue + inspection, post-outage drain verification, Accept-and-Hold for planned + maintenance, overage semantics, whitelist upkeep if their CIDRs change). + +### MTA-STS finding (no action in this change) + +`_mta-sts.viktorbarzin.me TXT "v=STSv1; id=20260412"` is published, but +`mta-sts.viktorbarzin.me` has **no public DNS record and nothing serves the +policy file** → per RFC 8461 senders that see the TXT fail the HTTPS policy +fetch and proceed as if no policy exists. MTA-STS is inert today (docs-vs-live +mismatch vs the mailserver.md DNS table). Whenever it is fixed properly, the +policy's `mx:` list MUST include `mail.rollernet.us` and `mail2.rollernet.us`, +or MTA-STS-enforcing senders will refuse the backup path. Tracked as a +follow-up, not part of this design. + +## Validation gates (in order; any failure → stop and report) + +| # | Gate | Method | Failure handling | +|---|------|--------|------------------| +| G1 | Free tier still includes Secondary MX (2026) | Signup + ACC | Decision returns to Viktor: Dynu $9.99/yr vs Rollernet Basic $30/yr vs Oracle-VM self-host | +| G2 | 10 MB/day overage semantics: locked domain answers **4xx (defer)** not 5xx (bounce) | Their docs/support ticket before DNS golive | If 5xx: decision returns to Viktor (paid tier lifts cap, or accept the risk window) | +| G3 | STARTTLS on their MX hosts (cert quality) | `openssl s_client -starttls smtp -connect mail.rollernet.us:25` | Informational now (blocks only the future MTA-STS fix) | +| G4 | Authoritative relay CIDRs published | ACC Resource Access page | Whitelist (changes 2–3) MUST be applied **before** the MX records go live — ordering guard | +| G5 | Live failover test | See below | Debug or roll back (remove MX records) | + +**G5 live failover test**: `presence claim service:mailserver` → scale +mailserver deployment to 0 (≈30 min window) → send probes from Gmail and via +Brevo API → confirm both queue in Rollernet ACC → scale back to 1 → verify +clean drain: delivered to `spam@`/target mailbox, headers show no SPF/DMARC +penalty and no postscreen interference, DKIM still validates. Also verify the +E2E roundtrip probe recovers on its own. + +## Failure modes + +Covered: cluster/pod outages, pfSense/power/ISP outages, WAN IP changes (queue +holds while DNS is fixed), multi-day outages ≤ 3 weeks, short-retry senders. + +Not covered (out of scope, above): primary-up-but-5xx misconfigs; outbound; +mid-outage mailbox access. + +Newly introduced, accepted: + +- **Plaintext queue at a third party** during outages (same trust class as + Brevo holding outbound today). +- **Spam-path bypass**: mail via the backup skips postscreen DNSBL (their IPs + are whitelisted) and SPF/DMARC scoring; rspamd content/AV/Bayes still apply. + Slight spam uptick possible during outages; catch-all absorbs to `spam@`. +- **10 MB/day cap** mid-outage → domain locks until midnight PT (severity + depends on G2: defer = harmless delay at sender; bounce = loss → gate). +- Rollernet outage while primary is also down = status quo ante (sender + retries), never worse than today. + +## Rollback + +Remove the two MX records (TTL is automatic/low) and disable the domain in the +ACC. Whitelist + rspamd exemption are inert without the MX records and may be +reverted in the same commit or left for a retry.