infra/docs/plans/2026-06-08-matrix-synapse-to-tuwunel-plan.md

# Matrix: Synapse → tuwunel migration — Plan (executed)

**Date:** 2026-06-08 · **Companion:** `2026-06-08-matrix-synapse-to-tuwunel-design.md`

## Executed steps

1. **Vault** — generated a 32-byte `registration_token`, stored at
   `secret/matrix`.
2. **`stacks/matrix` rewrite** — replaced Synapse with tuwunel: removed the
   `matrix-db-creds` ExternalSecret, both init-containers (`install-psycopg2`,
   `inject-db-password`), the `extra-packages` volume, and the Reloader
   annotation; added the `matrix-secrets` ExternalSecret (vault-kv `dataFrom`),
   the `TUWUNEL_*` env, `securityContext` 1000, and the tuwunel image. Encrypted
   PVC, Service (`80→8008`), and ingress (`auth="none"`, proxied) unchanged.
   - The image is in the deployment's `ignore_changes` (KEEL_IGNORE_IMAGE); it
     was **temporarily un-ignored** for this base-image swap, then re-added at
     step 4 so Keel resumes tag management.
   - `tg init -reconfigure` was required first (Tier-1 PG-backend creds rotate
     weekly → "Backend configuration block has changed").
3. **Apply** — `Plan: 1 to add, 2 to change, 1 to destroy`. tuwunel 1.7.1 came up
   1/1, created a fresh RocksDB on the encrypted PVC (no permission errors —
   fsGroup worked).
4. **Verify** — all `200`: `/_tuwunel/server_version`, `.well-known/matrix/
   {client,server}`, `/_matrix/client/versions`, `/_matrix/federation/v1/version`.
   Registered `@viktor:matrix.viktorbarzin.me` (first user → admin) via the token
   flow; `whoami` confirmed. Creds stored at `secret/matrix`
   (`admin_user`, `admin_password`).
5. **Lock down** — `TUWUNEL_ALLOW_REGISTRATION=false` + re-added image
   `ignore_changes`; applied. Registration now returns `403 M_FORBIDDEN`.
6. **Cleanup** —
   - `stacks/vault`: removed the `pg_matrix` static role + its `allowed_roles`
     entry (targeted apply — the full plan also wanted an **unrelated** OIDC
     `tune`-TTL change, deliberately NOT applied; see residual items).
   - Dropped the orphaned `matrix` Postgres DB (16 MB) + `matrix` role on the
     CNPG primary (`pg-cluster-2`).
   - Docs updated: `.claude/CLAUDE.md` (PG-rotation list), `service-catalog.md`,
     `upgrade-config.json` (removed synapse image-rename + matrix PG entry),
     `authentication.md` + `authentik-state.md` (Matrix OIDC → orphaned).

## Rollback

Fresh start was confirmed, so there is no Synapse data to preserve. To revert the
*service*: restore the Synapse `main.tf` from git, re-add the `pg_matrix` Vault
role, and restore the `matrix` Postgres DB from the daily per-db dump
(`/backup/per-db/matrix/`). The reused encrypted PVC still holds Synapse's old
`homeserver.yaml` / signing key / media at the volume root alongside the new
RocksDB dir.

## Residual / follow-up items (flagged to user)

- **Authentik Matrix OAuth2 app — REMOVED 2026-06-08** (user-confirmed). It was
  UI-managed (NOT in the authentik TF stack), so it was deleted via the Authentik
  API: application `matrix` + OAuth2 provider `pk=6`. tuwunel uses native password
  auth, so nothing consumed it.
- **Pre-existing drift in `stacks/vault`**: `vault_jwt_auth_backend.oidc` shows a
  `tune` diff (explicit `768h` default/max lease TTLs being dropped). This
  predates this migration and was **not** applied. Resolve separately.
- **Synapse leftover files** remain on the encrypted PVC volume root (unused by
  tuwunel). Can be `rm`'d after confidence in the new server.

## Follow-up: open registration + bot mitigations (2026-06-08, user-chosen)

Registration was opened **fully (tokenless)** — `TUWUNEL_ALLOW_REGISTRATION=true`
+ `TUWUNEL_YES_I_AM_VERY_VERY_SURE_I_WANT_AN_OPEN_REGISTRATION_SERVER_PRONE_TO_ABUSE=true`,
dropped the `TUWUNEL_REGISTRATION_TOKEN` env (the Vault `secret/matrix` token +
`matrix-secrets` ESO are kept for one-env-change revert to token-gated). tuwunel
has **no CAPTCHA** (only Synapse does) and a browser challenge would break native
clients, so bot defense is layered instead:

- **Traefik rate-limit on `/register`** — a `register-ratelimit` Middleware
  (`stacks/matrix`) on a path-scoped `ingress_register` carve-out (longer prefix
  wins over the catch-all). Keyed on the **request Host (global `/register` cap),
  not source IP** — because the host is reachable both via Cloudflare-IPv4
  (`CF-Connecting-IP`) and **IPv6-direct (HE tunnel → pfSense HAProxy → Traefik,
  no CF header)**; a per-source key let IPv6 bots bypass entirely (found during
  testing). 10/min, burst 20, **per Traefik replica (×3)**.
- **CrowdSec** (already on the ingress chain) is the hard backstop — bans abusive
  IPs on both paths; covers the per-replica looseness of the soft rate-limit.
- **Notification:** Loki ruler rule `MatrixNewUserRegistered` (`stacks/monitoring`,
  matches `... registered on this server`, never the rejection line) → `lane=security`
  → existing `#security` Slack receiver. Also note tuwunel's admin bot
  (`@conduit:matrix.viktorbarzin.me`) **natively posts every registration to the
  server admin room**, so there's an in-Matrix notice too.
- **Verification:** open signup returns 200 (`@regtest1`, since deactivated via
  `!admin users deactivate` in the admin room); Traefik access logs confirm
  `/register` routes through the rate-limited carve-out router. A live 429 was not
  force-tested (per-replica burst ~60 across 3 replicas; avoided hammering so as
  not to trip CrowdSec on the test source IP).

**Add a user:** anyone can self-register now. To provision manually instead:
`!admin users create-user <name>` in the admin room (first user `@viktor` is admin).
**Revert to token-gated:** drop the YES_I_AM... flag, re-add `TUWUNEL_REGISTRATION_TOKEN`.