[dbaas] Declare forgejo + roundcubemail MySQL users in Terraform

## Context

The 2026-04-16 MySQL InnoDB Cluster → standalone migration recreated the
MySQL user table but scripted fresh passwords for every app user. Two apps
(forgejo, roundcubemail) store their DB password inside their own
application config — forgejo in `/data/gitea/conf/app.ini` (baked into the
PVC), roundcubemail in the ROUNDCUBEMAIL_DB_PASSWORD env from the
mailserver stack (sourced from Vault `secret/platform`). Neither app
could be restarted with a new password without rewriting its own config.

Both apps silently broke with `Access denied for user 'X'@'%'` after the
migration. Remediation on 2026-04-17 was a manual `ALTER USER ... IDENTIFIED
BY '<app_password>'` to re-sync MySQL with what each app already has. With
nothing in Terraform managing those users, the next migration would break
them again — that's the gap this change closes.

## What this change does

Codifies both MySQL users in `stacks/dbaas/modules/dbaas/` using the same
`null_resource` + `local-exec` + `kubectl exec` pattern already used for
`pg_terraform_state_db` (line 1373 of the same file). Rejected alternatives:

- `petoju/mysql` Terraform provider — no existing usage in the repo; would
  be a net-new dependency. Module-level `for_each` over `mysql_user` +
  `mysql_grant` is cleaner, but the added machinery (new provider block,
  extra auth path via `MYSQL_HOST`/`MYSQL_USERNAME`/`MYSQL_PASSWORD` TF
  env vars, state-dependent password reads) outweighs the benefit for two
  static users.
- K8s Job — adds lifecycle management for a one-shot resource; needs
  secret mounts and is harder to retry. `local-exec` is exactly what the
  existing PG bootstrap uses.

Idempotency contract:

    CREATE DATABASE IF NOT EXISTS <db>;
    CREATE USER IF NOT EXISTS '<user>'@'%' IDENTIFIED WITH caching_sha2_password BY '<pw>';
    ALTER USER '<user>'@'%' IDENTIFIED WITH caching_sha2_password BY '<pw>';
    GRANT ALL PRIVILEGES ON <db>.* TO '<user>'@'%';
    FLUSH PRIVILEGES;

The `ALTER USER` on every re-run re-syncs the password if Vault was rotated
out-of-band (healing drift). The `sha256(password)` trigger also re-runs
the provisioner when the Vault password legitimately changes, so the
resource is responsive to both new and rotated passwords. `caching_sha2_password`
matches the live plugin returned by `SHOW CREATE USER`; forcing it prevents
silent drift to `mysql_native_password`.

Flow (apply-time):

    scripts/tg apply
        │
        ├── data.vault_kv_secret_v2.viktor  ── reads mysql_{forgejo,roundcubemail}_password
        │
        ▼
    module.dbaas
        │
        ├── mysql-standalone-0 (StatefulSet, already running)
        │
        ├── null_resource.mysql_static_user["forgejo"]
        │     └── kubectl exec ... mysql -uroot -p$ROOT_PASSWORD ... CREATE/ALTER/GRANT
        │
        └── null_resource.mysql_static_user["roundcubemail"]
              └── (same, for roundcubemail)

## Secrets

Two new keys added to Vault `secret/viktor`:

    mysql_forgejo_password        # bound to forgejo `[database]` in app.ini
    mysql_roundcubemail_password  # duplicates secret/platform
                                  # mailserver_roundcubemail_db_password;
                                  # secret/viktor is the personal vault of
                                  # record per .claude/CLAUDE.md

Passwords are never written to the repo — both come from Vault via
`data "vault_kv_secret_v2" "viktor"` in the dbaas root module.

## What is NOT in this change

- PG-side users (managed by Vault DB engine static-roles already — see
  MEMORY.md "Database rotation")
- Other MySQL users (speedtest, wrongmove, codimd, nextcloud, shlink,
  grafana, phpipam are all rotated by Vault DB engine; root users
  excluded by design)
- Removing the old mysql-operator / InnoDB Cluster helm releases (Phase 4
  cleanup tracked under the MySQL standalone migration work — still
  pending)

## Test plan

### Automated

`terraform fmt -check -recursive stacks/dbaas` → exit 0
`scripts/tg plan` in stacks/dbaas →

    Plan: 2 to add, 7 to change, 0 to destroy.
    # module.dbaas.null_resource.mysql_static_user["forgejo"]     will be created
    # module.dbaas.null_resource.mysql_static_user["roundcubemail"] will be created

The 7 "update in-place" entries are pre-existing drift (Kyverno labels on
LimitRange, MetalLB ip-allocated-from-pool annotation on postgresql_lb,
Kyverno-injected `dns_config` on 4 CronJobs lacking the
`ignore_changes` workaround, `resize.topolvm.io/storage_limit` bump
30Gi→50Gi on mysql-standalone PVC). None of those are introduced by this
commit and all are benign (no data loss, no pod restart).

### Manual Verification

    # 1. Sanity check pre-apply — users are in their current (manually-fixed) state.
    kubectl exec -n dbaas mysql-standalone-0 -c mysql -- bash -c \
      'mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -N -e \
       "SELECT user,host,plugin FROM mysql.user WHERE user IN (\"forgejo\",\"roundcubemail\");"'
    # Expected:
    #   forgejo       %   caching_sha2_password
    #   roundcubemail %   caching_sha2_password

    # 2. Apply and confirm the provisioner exits 0.
    cd stacks/dbaas && ../../scripts/tg apply
    # Expect: null_resource.mysql_static_user["forgejo"]: Creation complete
    #         null_resource.mysql_static_user["roundcubemail"]: Creation complete

    # 3. App-level smoke: log in to forgejo.viktorbarzin.me (any git push)
    #    and load https://mail.viktorbarzin.me/roundcube (IMAP login). Both
    #    must succeed.

    # 4. Destructive test (run ONCE, off-hours):
    kubectl exec -n dbaas mysql-standalone-0 -c mysql -- bash -c \
      'mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "DROP USER '\''forgejo'\''@'\''%'\''"'
    cd stacks/dbaas && ../../scripts/tg apply
    # Expected: apply recreates the user with the Vault password, forgejo UI
    # recovers without touching /data/gitea/conf/app.ini.

### Reproduce locally

    1. vault login -method=oidc
    2. cd infra/stacks/dbaas
    3. ../../scripts/tg plan
    4. Expected: "Plan: 2 to add, 7 to change, 0 to destroy." with the two
       null_resource.mysql_static_user additions. 7 changes are pre-existing
       drift unrelated to this commit.

Closes: code-6th
Closes: code-96w

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-04-17 22:05:21 +00:00
parent b326c572a6
commit 2033e76798
2 changed files with 85 additions and 10 deletions

View file

@ -14,14 +14,24 @@ data "vault_kv_secret_v2" "secrets" {
name = "platform" name = "platform"
} }
module "dbaas" { # Personal/app-user secrets (forgejo + roundcubemail MySQL passwords live here,
source = "./modules/dbaas" # not under secret/platform, to match the "secret/viktor as the go-to personal
prod = var.prod # vault" convention documented in .claude/CLAUDE.md).
tls_secret_name = var.tls_secret_name data "vault_kv_secret_v2" "viktor" {
nfs_server = var.nfs_server mount = "secret"
dbaas_root_password = data.vault_kv_secret_v2.secrets.data["dbaas_root_password"] name = "viktor"
postgresql_root_password = data.vault_kv_secret_v2.secrets.data["dbaas_postgresql_root_password"] }
pgadmin_password = data.vault_kv_secret_v2.secrets.data["dbaas_pgadmin_password"]
kube_config_path = var.kube_config_path module "dbaas" {
tier = local.tiers.cluster source = "./modules/dbaas"
prod = var.prod
tls_secret_name = var.tls_secret_name
nfs_server = var.nfs_server
dbaas_root_password = data.vault_kv_secret_v2.secrets.data["dbaas_root_password"]
postgresql_root_password = data.vault_kv_secret_v2.secrets.data["dbaas_postgresql_root_password"]
pgadmin_password = data.vault_kv_secret_v2.secrets.data["dbaas_pgadmin_password"]
mysql_forgejo_password = data.vault_kv_secret_v2.viktor.data["mysql_forgejo_password"]
mysql_roundcubemail_password = data.vault_kv_secret_v2.viktor.data["mysql_roundcubemail_password"]
kube_config_path = var.kube_config_path
tier = local.tiers.cluster
} }

View file

@ -17,6 +17,18 @@ variable "kube_config_path" {
sensitive = true sensitive = true
} }
# MySQL static application users (not rotated by Vault DB engine; baked into
# each app's config). Codified here so future MySQL rebuilds cannot silently
# drop them.
variable "mysql_forgejo_password" {
type = string
sensitive = true
}
variable "mysql_roundcubemail_password" {
type = string
sensitive = true
}
resource "kubernetes_namespace" "dbaas" { resource "kubernetes_namespace" "dbaas" {
metadata { metadata {
name = "dbaas" name = "dbaas"
@ -562,6 +574,59 @@ resource "kubernetes_service" "mysql" {
depends_on = [kubernetes_stateful_set_v1.mysql_standalone] depends_on = [kubernetes_stateful_set_v1.mysql_standalone]
} }
# MySQL static application users not rotated by Vault DB engine.
# Each app stores its password in its own config (forgejo app.ini, roundcube
# ROUNDCUBEMAIL_DB_PASSWORD env). During the 2026-04-16 InnoDB Cluster
# standalone migration these users were accidentally dropped and recreated with
# mismatched passwords; this block codifies them so a future rebuild cannot
# silently break the apps.
#
# Pattern matches `null_resource.pg_terraform_state_db` below (local-exec into
# the DB pod). We CREATE IF NOT EXISTS + ALTER USER on every apply so a
# password rotation in Vault is re-synced on the next `scripts/tg apply`. The
# `password_hash` trigger re-runs the provisioner when the Vault password
# changes; the namespace/user triggers re-run if identifiers change.
locals {
mysql_static_users = {
forgejo = {
database = "forgejo"
password = var.mysql_forgejo_password
}
roundcubemail = {
database = "roundcubemail"
password = var.mysql_roundcubemail_password
}
}
}
resource "null_resource" "mysql_static_user" {
for_each = local.mysql_static_users
depends_on = [kubernetes_stateful_set_v1.mysql_standalone]
triggers = {
username = each.key
database = each.value.database
password_hash = sha256(each.value.password)
}
provisioner "local-exec" {
command = <<-EOT
kubectl --kubeconfig ${var.kube_config_path} exec -n dbaas mysql-standalone-0 -c mysql -- \
env USER='${each.key}' DB='${each.value.database}' PW='${each.value.password}' \
bash -c '
mysql -uroot -p"$MYSQL_ROOT_PASSWORD" <<SQL
CREATE DATABASE IF NOT EXISTS \`'"$DB"'\`;
CREATE USER IF NOT EXISTS '"'$USER'"'@'"'%'"' IDENTIFIED WITH caching_sha2_password BY '"'$PW'"';
ALTER USER '"'$USER'"'@'"'%'"' IDENTIFIED WITH caching_sha2_password BY '"'$PW'"';
GRANT ALL PRIVILEGES ON \`'"$DB"'\`.* TO '"'$USER'"'@'"'%'"';
FLUSH PRIVILEGES;
SQL
'
EOT
}
}
module "nfs_mysql_backup_host" { module "nfs_mysql_backup_host" {
source = "../../../../modules/kubernetes/nfs_volume" source = "../../../../modules/kubernetes/nfs_volume"
name = "dbaas-mysql-backup-host" name = "dbaas-mysql-backup-host"