[mailserver] Move probe secrets to ExternalSecret via ESO [ci skip]
## Context
The email-roundtrip-monitor CronJob injected `BREVO_API_KEY` and
`EMAIL_MONITOR_IMAP_PASSWORD` as inline `env { value = var.xxx }` —
Terraform read them from Vault at plan time and embedded them in the
generated CronJob spec. Anyone with `kubectl describe cronjob` (or
pod-event read) in the `mailserver` namespace could read both secrets
verbatim.
The two upstream Vault entries are not flat strings:
- `secret/viktor` → `brevo_api_key` = base64(JSON({"api_key": "..."}))
- `secret/platform` → `mailserver_accounts` = JSON({"spam@viktorbarzin.me": "<pw>", ...})
A plain ESO `remoteRef.property` can traverse one level of JSON but
cannot base64-decode the wrapper or index a map key that contains `@`.
So the ExternalSecret pulls the raw Vault values and the rendered K8s
Secret is produced via ESO's `target.template` (engineVersion v2, sprig
pipeline `b64dec | fromJson | dig`). `mergePolicy` defaults to Replace,
so only the transformed `BREVO_API_KEY` / `EMAIL_MONITOR_IMAP_PASSWORD`
keys land in the K8s Secret — the raw wrapped inputs never reach it.
## This change
1. New `kubernetes_manifest.email_roundtrip_monitor_secrets` rendering
an `external-secrets.io/v1beta1` ExternalSecret into a K8s Secret
named `mailserver-probe-secrets` via the `vault-kv` ClusterSecretStore.
2. CronJob's two `env { name=... value=var.xxx }` blocks replaced with
a single `env_from { secret_ref { name = "mailserver-probe-secrets" } }`.
3. Unused `brevo_api_key` / `email_monitor_imap_password` module
variables + their wiring in `stacks/mailserver/main.tf` removed.
`data "vault_kv_secret_v2" "viktor"` dropped (last consumer gone).
```
Before: After:
┌────────────┐ ┌────────────┐
│ Vault KV │ │ Vault KV │
└────┬───────┘ └────┬───────┘
│ (plan-time read) │ (runtime pull)
▼ ▼
┌────────────┐ ┌────────────┐
│ Terraform │ │ ESO ctrl │
│ state │ │ +template │
└────┬───────┘ └────┬───────┘
│ inline value= │ sprig b64dec | fromJson
▼ ▼
┌────────────┐ ┌────────────┐
│ CronJob │ <-- kubectl describe leaks! │ K8s Secret │
│ env[].value│ │ probe-sec │
└────────────┘ └────┬───────┘
│ env_from.secret_ref
▼
┌────────────┐
│ CronJob │
│ (no values │
│ in spec) │
└────────────┘
```
## Test Plan
### Automated
`terragrunt plan -target=...ExternalSecret -target=...CronJob`:
```
Plan: 1 to add, 1 to change, 0 to destroy.
+ kubernetes_manifest.email_roundtrip_monitor_secrets (ExternalSecret)
~ kubernetes_cron_job_v1.email_roundtrip_monitor
- env { name = "BREVO_API_KEY" ... }
- env { name = "EMAIL_MONITOR_IMAP_PASSWORD" ... }
+ env_from { secret_ref { name = "mailserver-probe-secrets" } }
```
`terragrunt apply --non-interactive` same targets:
```
Apply complete! Resources: 1 added, 1 changed, 0 destroyed.
```
`kubectl get externalsecret -n mailserver mailserver-probe-secrets`:
```
NAME STORE REFRESH INTERVAL STATUS READY
mailserver-probe-secrets vault-kv 15m SecretSynced True
```
`kubectl get secret -n mailserver mailserver-probe-secrets -o yaml`
exposes exactly two data keys (`BREVO_API_KEY`, `EMAIL_MONITOR_IMAP_PASSWORD`) —
both populated, 120 / 32 base64 chars, no raw `brevo_api_key_wrapped` /
`mailserver_accounts` keys.
`kubectl describe cronjob -n mailserver email-roundtrip-monitor`:
```
Environment Variables from:
mailserver-probe-secrets Secret Optional: false
Environment: <none>
```
(Previously the `Environment:` block listed both secrets with their raw
values.)
### Manual Verification
1. `kubectl create job --from=cronjob/email-roundtrip-monitor \
probe-test-$RANDOM -n mailserver`
2. `kubectl logs -n mailserver -l job-name=probe-test-... --tail=30`
expected:
```
Sent test email via Brevo: 201 marker=e2e-probe-...
Found test email after 1 attempts
Deleted 1 e2e probe email(s)
Round-trip SUCCESS in 20.3s
Pushed metrics to Pushgateway
Pushed to Uptime Kuma
```
3. `kubectl exec -n monitoring deploy/prometheus-prometheus-pushgateway \
-- wget -q -O- http://localhost:9091/metrics | grep email_roundtrip`
shows `email_roundtrip_success=1`, fresh timestamp, duration in range.
4. `kubectl delete job -n mailserver probe-test-...` to clean up.
Closes: code-39v
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
238a3f14c9
commit
0a2d8b2138
2 changed files with 69 additions and 33 deletions
|
|
@ -11,11 +11,6 @@ data "vault_kv_secret_v2" "secrets" {
|
|||
name = "platform"
|
||||
}
|
||||
|
||||
data "vault_kv_secret_v2" "viktor" {
|
||||
mount = "secret"
|
||||
name = "viktor"
|
||||
}
|
||||
|
||||
locals {
|
||||
mailserver_accounts = jsondecode(data.vault_kv_secret_v2.secrets.data["mailserver_accounts"])
|
||||
mailserver_aliases = jsondecode(data.vault_kv_secret_v2.secrets.data["mailserver_aliases"])
|
||||
|
|
@ -24,16 +19,14 @@ locals {
|
|||
}
|
||||
|
||||
module "mailserver" {
|
||||
source = "./modules/mailserver"
|
||||
tls_secret_name = var.tls_secret_name
|
||||
nfs_server = var.nfs_server
|
||||
mysql_host = var.mysql_host
|
||||
mailserver_accounts = local.mailserver_accounts
|
||||
postfix_account_aliases = local.mailserver_aliases
|
||||
opendkim_key = local.mailserver_opendkim_key
|
||||
sasl_passwd = local.mailserver_sasl_passwd
|
||||
roundcube_db_password = data.vault_kv_secret_v2.secrets.data["mailserver_roundcubemail_db_password"]
|
||||
tier = local.tiers.edge
|
||||
brevo_api_key = jsondecode(base64decode(data.vault_kv_secret_v2.viktor.data["brevo_api_key"]))["api_key"]
|
||||
email_monitor_imap_password = local.mailserver_accounts["spam@viktorbarzin.me"]
|
||||
source = "./modules/mailserver"
|
||||
tls_secret_name = var.tls_secret_name
|
||||
nfs_server = var.nfs_server
|
||||
mysql_host = var.mysql_host
|
||||
mailserver_accounts = local.mailserver_accounts
|
||||
postfix_account_aliases = local.mailserver_aliases
|
||||
opendkim_key = local.mailserver_opendkim_key
|
||||
sasl_passwd = local.mailserver_sasl_passwd
|
||||
roundcube_db_password = data.vault_kv_secret_v2.secrets.data["mailserver_roundcubemail_db_password"]
|
||||
tier = local.tiers.edge
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,15 +5,6 @@ variable "postfix_account_aliases" {}
|
|||
variable "opendkim_key" {}
|
||||
variable "sasl_passwd" {} # For sendgrid i.e relayhost
|
||||
variable "nfs_server" { type = string }
|
||||
variable "brevo_api_key" {
|
||||
type = string
|
||||
sensitive = true
|
||||
}
|
||||
variable "email_monitor_imap_password" {
|
||||
type = string
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
# Build the virtual-alias map, dropping aliases where BOTH the source and
|
||||
# target are real mailboxes in var.mailserver_accounts (and are different).
|
||||
# Without this filter, docker-mailserver emits two passwd-file userdb lines
|
||||
|
|
@ -564,6 +555,61 @@ resource "kubernetes_service" "mailserver" {
|
|||
# E2E Email Roundtrip Monitor
|
||||
# Sends test email via Brevo API, verifies delivery via IMAP, pushes metrics
|
||||
# =============================================================================
|
||||
# ExternalSecret syncing the probe's Vault inputs into a K8s Secret, so
|
||||
# `kubectl describe cronjob email-roundtrip-monitor` no longer leaks the
|
||||
# Brevo API key and IMAP password via `env[].value`. The two upstream Vault
|
||||
# entries both wrap the effective secret:
|
||||
# - secret/viktor → brevo_api_key = base64(JSON({"api_key": "..."}))
|
||||
# - secret/platform → mailserver_accounts = JSON({"spam@viktorbarzin.me": "<pw>", ...})
|
||||
# ESO's `target.template` (engineVersion v2) runs sprig on the raw remote
|
||||
# values so the rendered K8s Secret contains ONLY the two env vars the probe
|
||||
# actually needs, under the exact keys `BREVO_API_KEY` and
|
||||
# `EMAIL_MONITOR_IMAP_PASSWORD` so the CronJob can consume them via a single
|
||||
# `env_from { secret_ref {} }` block.
|
||||
resource "kubernetes_manifest" "email_roundtrip_monitor_secrets" {
|
||||
manifest = {
|
||||
apiVersion = "external-secrets.io/v1beta1"
|
||||
kind = "ExternalSecret"
|
||||
metadata = {
|
||||
name = "mailserver-probe-secrets"
|
||||
namespace = kubernetes_namespace.mailserver.metadata[0].name
|
||||
}
|
||||
spec = {
|
||||
refreshInterval = "15m"
|
||||
secretStoreRef = {
|
||||
name = "vault-kv"
|
||||
kind = "ClusterSecretStore"
|
||||
}
|
||||
target = {
|
||||
name = "mailserver-probe-secrets"
|
||||
template = {
|
||||
engineVersion = "v2"
|
||||
data = {
|
||||
BREVO_API_KEY = "{{ .brevo_api_key_wrapped | b64dec | fromJson | dig \"api_key\" \"\" }}"
|
||||
EMAIL_MONITOR_IMAP_PASSWORD = "{{ .mailserver_accounts | fromJson | dig \"spam@viktorbarzin.me\" \"\" }}"
|
||||
}
|
||||
}
|
||||
}
|
||||
data = [
|
||||
{
|
||||
secretKey = "brevo_api_key_wrapped"
|
||||
remoteRef = {
|
||||
key = "viktor"
|
||||
property = "brevo_api_key"
|
||||
}
|
||||
},
|
||||
{
|
||||
secretKey = "mailserver_accounts"
|
||||
remoteRef = {
|
||||
key = "platform"
|
||||
property = "mailserver_accounts"
|
||||
}
|
||||
},
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_cron_job_v1" "email_roundtrip_monitor" {
|
||||
metadata {
|
||||
name = "email-roundtrip-monitor"
|
||||
|
|
@ -695,13 +741,10 @@ sys.exit(0 if success else 1)
|
|||
'
|
||||
EOT
|
||||
]
|
||||
env {
|
||||
name = "BREVO_API_KEY"
|
||||
value = var.brevo_api_key
|
||||
}
|
||||
env {
|
||||
name = "EMAIL_MONITOR_IMAP_PASSWORD"
|
||||
value = var.email_monitor_imap_password
|
||||
env_from {
|
||||
secret_ref {
|
||||
name = "mailserver-probe-secrets"
|
||||
}
|
||||
}
|
||||
resources {
|
||||
requests = {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue