Commit graph

4 commits

Author SHA1 Message Date
Viktor Barzin
0a2d8b2138 [mailserver] Move probe secrets to ExternalSecret via ESO [ci skip]
## Context
The email-roundtrip-monitor CronJob injected `BREVO_API_KEY` and
`EMAIL_MONITOR_IMAP_PASSWORD` as inline `env { value = var.xxx }` —
Terraform read them from Vault at plan time and embedded them in the
generated CronJob spec. Anyone with `kubectl describe cronjob` (or
pod-event read) in the `mailserver` namespace could read both secrets
verbatim.

The two upstream Vault entries are not flat strings:
- `secret/viktor`  → `brevo_api_key`      = base64(JSON({"api_key": "..."}))
- `secret/platform` → `mailserver_accounts` = JSON({"spam@viktorbarzin.me": "<pw>", ...})

A plain ESO `remoteRef.property` can traverse one level of JSON but
cannot base64-decode the wrapper or index a map key that contains `@`.
So the ExternalSecret pulls the raw Vault values and the rendered K8s
Secret is produced via ESO's `target.template` (engineVersion v2, sprig
pipeline `b64dec | fromJson | dig`). `mergePolicy` defaults to Replace,
so only the transformed `BREVO_API_KEY` / `EMAIL_MONITOR_IMAP_PASSWORD`
keys land in the K8s Secret — the raw wrapped inputs never reach it.

## This change
1. New `kubernetes_manifest.email_roundtrip_monitor_secrets` rendering
   an `external-secrets.io/v1beta1` ExternalSecret into a K8s Secret
   named `mailserver-probe-secrets` via the `vault-kv` ClusterSecretStore.
2. CronJob's two `env { name=... value=var.xxx }` blocks replaced with
   a single `env_from { secret_ref { name = "mailserver-probe-secrets" } }`.
3. Unused `brevo_api_key` / `email_monitor_imap_password` module
   variables + their wiring in `stacks/mailserver/main.tf` removed.
   `data "vault_kv_secret_v2" "viktor"` dropped (last consumer gone).

```
Before:                                      After:
┌────────────┐                                ┌────────────┐
│ Vault KV   │                                │ Vault KV   │
└────┬───────┘                                └────┬───────┘
     │ (plan-time read)                            │ (runtime pull)
     ▼                                             ▼
┌────────────┐                                ┌────────────┐
│ Terraform  │                                │ ESO ctrl   │
│ state      │                                │ +template  │
└────┬───────┘                                └────┬───────┘
     │ inline value=                               │ sprig b64dec | fromJson
     ▼                                             ▼
┌────────────┐                                ┌────────────┐
│ CronJob    │ <-- kubectl describe leaks!    │ K8s Secret │
│ env[].value│                                │ probe-sec  │
└────────────┘                                └────┬───────┘
                                                   │ env_from.secret_ref
                                                   ▼
                                              ┌────────────┐
                                              │ CronJob    │
                                              │ (no values │
                                              │  in spec)  │
                                              └────────────┘
```

## Test Plan

### Automated
`terragrunt plan -target=...ExternalSecret -target=...CronJob`:
```
Plan: 1 to add, 1 to change, 0 to destroy.
  + kubernetes_manifest.email_roundtrip_monitor_secrets (ExternalSecret)
  ~ kubernetes_cron_job_v1.email_roundtrip_monitor
      - env { name = "BREVO_API_KEY" ... }
      - env { name = "EMAIL_MONITOR_IMAP_PASSWORD" ... }
      + env_from { secret_ref { name = "mailserver-probe-secrets" } }
```
`terragrunt apply --non-interactive` same targets:
```
Apply complete! Resources: 1 added, 1 changed, 0 destroyed.
```
`kubectl get externalsecret -n mailserver mailserver-probe-secrets`:
```
NAME                       STORE      REFRESH INTERVAL   STATUS         READY
mailserver-probe-secrets   vault-kv   15m                SecretSynced   True
```
`kubectl get secret -n mailserver mailserver-probe-secrets -o yaml`
exposes exactly two data keys (`BREVO_API_KEY`, `EMAIL_MONITOR_IMAP_PASSWORD`) —
both populated, 120 / 32 base64 chars, no raw `brevo_api_key_wrapped` /
`mailserver_accounts` keys.

`kubectl describe cronjob -n mailserver email-roundtrip-monitor`:
```
    Environment Variables from:
      mailserver-probe-secrets  Secret  Optional: false
    Environment:                <none>
```
(Previously the `Environment:` block listed both secrets with their raw
values.)

### Manual Verification
1. `kubectl create job --from=cronjob/email-roundtrip-monitor \
     probe-test-$RANDOM -n mailserver`
2. `kubectl logs -n mailserver -l job-name=probe-test-... --tail=30`
   expected:
   ```
   Sent test email via Brevo: 201 marker=e2e-probe-...
   Found test email after 1 attempts
   Deleted 1 e2e probe email(s)
   Round-trip SUCCESS in 20.3s
   Pushed metrics to Pushgateway
   Pushed to Uptime Kuma
   ```
3. `kubectl exec -n monitoring deploy/prometheus-prometheus-pushgateway \
     -- wget -q -O- http://localhost:9091/metrics | grep email_roundtrip`
   shows `email_roundtrip_success=1`, fresh timestamp, duration in range.
4. `kubectl delete job -n mailserver probe-test-...` to clean up.

Closes: code-39v

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 23:39:06 +00:00
Viktor Barzin
bd41bb9230 fix: cluster healthcheck fixes + Authentik upgrade to 2026.2.2
- Authentik: upgrade 2025.10.3 → 2025.12.4 → 2026.2.2 with DB restore
  and stepped migration. Switch to existingSecret, PgBouncer session mode.
- Mailserver: migrate email roundtrip probe from Mailgun to Brevo API
- Redis: fix HAProxy tcp-check regex (rstring), faster health intervals
- Nextcloud: fix Redis fallback to HAProxy service, update dependency
- MeshCentral: fix TLSOffload + certUrl init container for first-run
- Monitoring: remove authentik from latency alert exclusion
- Diun: simplify to webhook notifier, remove git auto-update

[ci skip]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 06:41:56 +00:00
Viktor Barzin
78dec8f0ad add e2e email roundtrip monitoring
CronJob (every 30 min) sends test email via Mailgun API to
smoke-test@viktorbarzin.me, verifies IMAP delivery in spam@ catch-all,
deletes test email, pushes metrics to Pushgateway + Uptime Kuma.

Prometheus alerts: EmailRoundtripFailing, EmailRoundtripStale,
EmailRoundtripNeverRun. Uptime Kuma: SMTP/IMAP port checks + E2E push.
2026-03-25 22:50:22 +02:00
Viktor Barzin
ae36dc253b extract monitoring, nvidia, mailserver, cloudflared, kyverno from platform [ci skip]
Phase 2 of platform stack split. 5 more modules extracted into
independent stacks. All applied successfully with zero destroys.
Cloudflared now reads k8s_users from Vault directly to compute
user_domains. Woodpecker pipeline runs all 8 extracted stacks
in parallel. Memory bumped to 6Gi for 9 concurrent TF processes.
Platform reduced from 27 to 19 modules.
2026-03-17 21:34:11 +00:00