From 2033e767981aaad6ae5a074c7b0592c2e8932605 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Fri, 17 Apr 2026 22:05:21 +0000 Subject: [PATCH] [dbaas] Declare forgejo + roundcubemail MySQL users in Terraform MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Context The 2026-04-16 MySQL InnoDB Cluster → standalone migration recreated the MySQL user table but scripted fresh passwords for every app user. Two apps (forgejo, roundcubemail) store their DB password inside their own application config — forgejo in `/data/gitea/conf/app.ini` (baked into the PVC), roundcubemail in the ROUNDCUBEMAIL_DB_PASSWORD env from the mailserver stack (sourced from Vault `secret/platform`). Neither app could be restarted with a new password without rewriting its own config. Both apps silently broke with `Access denied for user 'X'@'%'` after the migration. Remediation on 2026-04-17 was a manual `ALTER USER ... IDENTIFIED BY ''` to re-sync MySQL with what each app already has. With nothing in Terraform managing those users, the next migration would break them again — that's the gap this change closes. ## What this change does Codifies both MySQL users in `stacks/dbaas/modules/dbaas/` using the same `null_resource` + `local-exec` + `kubectl exec` pattern already used for `pg_terraform_state_db` (line 1373 of the same file). Rejected alternatives: - `petoju/mysql` Terraform provider — no existing usage in the repo; would be a net-new dependency. Module-level `for_each` over `mysql_user` + `mysql_grant` is cleaner, but the added machinery (new provider block, extra auth path via `MYSQL_HOST`/`MYSQL_USERNAME`/`MYSQL_PASSWORD` TF env vars, state-dependent password reads) outweighs the benefit for two static users. - K8s Job — adds lifecycle management for a one-shot resource; needs secret mounts and is harder to retry. `local-exec` is exactly what the existing PG bootstrap uses. Idempotency contract: CREATE DATABASE IF NOT EXISTS ; CREATE USER IF NOT EXISTS ''@'%' IDENTIFIED WITH caching_sha2_password BY ''; ALTER USER ''@'%' IDENTIFIED WITH caching_sha2_password BY ''; GRANT ALL PRIVILEGES ON .* TO ''@'%'; FLUSH PRIVILEGES; The `ALTER USER` on every re-run re-syncs the password if Vault was rotated out-of-band (healing drift). The `sha256(password)` trigger also re-runs the provisioner when the Vault password legitimately changes, so the resource is responsive to both new and rotated passwords. `caching_sha2_password` matches the live plugin returned by `SHOW CREATE USER`; forcing it prevents silent drift to `mysql_native_password`. Flow (apply-time): scripts/tg apply │ ├── data.vault_kv_secret_v2.viktor ── reads mysql_{forgejo,roundcubemail}_password │ ▼ module.dbaas │ ├── mysql-standalone-0 (StatefulSet, already running) │ ├── null_resource.mysql_static_user["forgejo"] │ └── kubectl exec ... mysql -uroot -p$ROOT_PASSWORD ... CREATE/ALTER/GRANT │ └── null_resource.mysql_static_user["roundcubemail"] └── (same, for roundcubemail) ## Secrets Two new keys added to Vault `secret/viktor`: mysql_forgejo_password # bound to forgejo `[database]` in app.ini mysql_roundcubemail_password # duplicates secret/platform # mailserver_roundcubemail_db_password; # secret/viktor is the personal vault of # record per .claude/CLAUDE.md Passwords are never written to the repo — both come from Vault via `data "vault_kv_secret_v2" "viktor"` in the dbaas root module. ## What is NOT in this change - PG-side users (managed by Vault DB engine static-roles already — see MEMORY.md "Database rotation") - Other MySQL users (speedtest, wrongmove, codimd, nextcloud, shlink, grafana, phpipam are all rotated by Vault DB engine; root users excluded by design) - Removing the old mysql-operator / InnoDB Cluster helm releases (Phase 4 cleanup tracked under the MySQL standalone migration work — still pending) ## Test plan ### Automated `terraform fmt -check -recursive stacks/dbaas` → exit 0 `scripts/tg plan` in stacks/dbaas → Plan: 2 to add, 7 to change, 0 to destroy. # module.dbaas.null_resource.mysql_static_user["forgejo"] will be created # module.dbaas.null_resource.mysql_static_user["roundcubemail"] will be created The 7 "update in-place" entries are pre-existing drift (Kyverno labels on LimitRange, MetalLB ip-allocated-from-pool annotation on postgresql_lb, Kyverno-injected `dns_config` on 4 CronJobs lacking the `ignore_changes` workaround, `resize.topolvm.io/storage_limit` bump 30Gi→50Gi on mysql-standalone PVC). None of those are introduced by this commit and all are benign (no data loss, no pod restart). ### Manual Verification # 1. Sanity check pre-apply — users are in their current (manually-fixed) state. kubectl exec -n dbaas mysql-standalone-0 -c mysql -- bash -c \ 'mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -N -e \ "SELECT user,host,plugin FROM mysql.user WHERE user IN (\"forgejo\",\"roundcubemail\");"' # Expected: # forgejo % caching_sha2_password # roundcubemail % caching_sha2_password # 2. Apply and confirm the provisioner exits 0. cd stacks/dbaas && ../../scripts/tg apply # Expect: null_resource.mysql_static_user["forgejo"]: Creation complete # null_resource.mysql_static_user["roundcubemail"]: Creation complete # 3. App-level smoke: log in to forgejo.viktorbarzin.me (any git push) # and load https://mail.viktorbarzin.me/roundcube (IMAP login). Both # must succeed. # 4. Destructive test (run ONCE, off-hours): kubectl exec -n dbaas mysql-standalone-0 -c mysql -- bash -c \ 'mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "DROP USER '\''forgejo'\''@'\''%'\''"' cd stacks/dbaas && ../../scripts/tg apply # Expected: apply recreates the user with the Vault password, forgejo UI # recovers without touching /data/gitea/conf/app.ini. ### Reproduce locally 1. vault login -method=oidc 2. cd infra/stacks/dbaas 3. ../../scripts/tg plan 4. Expected: "Plan: 2 to add, 7 to change, 0 to destroy." with the two null_resource.mysql_static_user additions. 7 changes are pre-existing drift unrelated to this commit. Closes: code-6th Closes: code-96w Co-Authored-By: Claude Opus 4.7 (1M context) --- stacks/dbaas/main.tf | 30 +++++++++----- stacks/dbaas/modules/dbaas/main.tf | 65 ++++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+), 10 deletions(-) diff --git a/stacks/dbaas/main.tf b/stacks/dbaas/main.tf index 9825da78..2902a48c 100644 --- a/stacks/dbaas/main.tf +++ b/stacks/dbaas/main.tf @@ -14,14 +14,24 @@ data "vault_kv_secret_v2" "secrets" { name = "platform" } -module "dbaas" { - source = "./modules/dbaas" - prod = var.prod - tls_secret_name = var.tls_secret_name - nfs_server = var.nfs_server - dbaas_root_password = data.vault_kv_secret_v2.secrets.data["dbaas_root_password"] - postgresql_root_password = data.vault_kv_secret_v2.secrets.data["dbaas_postgresql_root_password"] - pgadmin_password = data.vault_kv_secret_v2.secrets.data["dbaas_pgadmin_password"] - kube_config_path = var.kube_config_path - tier = local.tiers.cluster +# Personal/app-user secrets (forgejo + roundcubemail MySQL passwords live here, +# not under secret/platform, to match the "secret/viktor as the go-to personal +# vault" convention documented in .claude/CLAUDE.md). +data "vault_kv_secret_v2" "viktor" { + mount = "secret" + name = "viktor" +} + +module "dbaas" { + source = "./modules/dbaas" + prod = var.prod + tls_secret_name = var.tls_secret_name + nfs_server = var.nfs_server + dbaas_root_password = data.vault_kv_secret_v2.secrets.data["dbaas_root_password"] + postgresql_root_password = data.vault_kv_secret_v2.secrets.data["dbaas_postgresql_root_password"] + pgadmin_password = data.vault_kv_secret_v2.secrets.data["dbaas_pgadmin_password"] + mysql_forgejo_password = data.vault_kv_secret_v2.viktor.data["mysql_forgejo_password"] + mysql_roundcubemail_password = data.vault_kv_secret_v2.viktor.data["mysql_roundcubemail_password"] + kube_config_path = var.kube_config_path + tier = local.tiers.cluster } diff --git a/stacks/dbaas/modules/dbaas/main.tf b/stacks/dbaas/modules/dbaas/main.tf index df68afbd..7b69a01c 100644 --- a/stacks/dbaas/modules/dbaas/main.tf +++ b/stacks/dbaas/modules/dbaas/main.tf @@ -17,6 +17,18 @@ variable "kube_config_path" { sensitive = true } +# MySQL static application users (not rotated by Vault DB engine; baked into +# each app's config). Codified here so future MySQL rebuilds cannot silently +# drop them. +variable "mysql_forgejo_password" { + type = string + sensitive = true +} +variable "mysql_roundcubemail_password" { + type = string + sensitive = true +} + resource "kubernetes_namespace" "dbaas" { metadata { name = "dbaas" @@ -562,6 +574,59 @@ resource "kubernetes_service" "mysql" { depends_on = [kubernetes_stateful_set_v1.mysql_standalone] } +# MySQL static application users — not rotated by Vault DB engine. +# Each app stores its password in its own config (forgejo app.ini, roundcube +# ROUNDCUBEMAIL_DB_PASSWORD env). During the 2026-04-16 InnoDB Cluster → +# standalone migration these users were accidentally dropped and recreated with +# mismatched passwords; this block codifies them so a future rebuild cannot +# silently break the apps. +# +# Pattern matches `null_resource.pg_terraform_state_db` below (local-exec into +# the DB pod). We CREATE IF NOT EXISTS + ALTER USER on every apply so a +# password rotation in Vault is re-synced on the next `scripts/tg apply`. The +# `password_hash` trigger re-runs the provisioner when the Vault password +# changes; the namespace/user triggers re-run if identifiers change. +locals { + mysql_static_users = { + forgejo = { + database = "forgejo" + password = var.mysql_forgejo_password + } + roundcubemail = { + database = "roundcubemail" + password = var.mysql_roundcubemail_password + } + } +} + +resource "null_resource" "mysql_static_user" { + for_each = local.mysql_static_users + + depends_on = [kubernetes_stateful_set_v1.mysql_standalone] + + triggers = { + username = each.key + database = each.value.database + password_hash = sha256(each.value.password) + } + + provisioner "local-exec" { + command = <<-EOT + kubectl --kubeconfig ${var.kube_config_path} exec -n dbaas mysql-standalone-0 -c mysql -- \ + env USER='${each.key}' DB='${each.value.database}' PW='${each.value.password}' \ + bash -c ' + mysql -uroot -p"$MYSQL_ROOT_PASSWORD" <