infra/stacks
Viktor Barzin f79e3c563e [infra] Remove mysql InnoDB Cluster + Operator HCL (Phase 4 cleanup) [ci skip]
## Context

On 2026-04-16 (memory #711) MySQL was migrated from InnoDB Cluster (3-member
Group Replication + MySQL Operator) to a raw `kubernetes_stateful_set_v1.mysql_standalone`
on `mysql:8.4`. The migration preserved the `mysql.dbaas` Service name
(selector switched to the standalone pod), all 20 databases/688 tables/14
users were dump-restored, and Vault rotated credentials against the new
instance. The InnoDB Cluster has been dark since — Phase 4 was to remove
the dead code and decommission its cluster-side Helm state.

Memory #711 explicitly notes Phase 4 as: "Remove helm_release.mysql_cluster
+ mysql_operator + namespace + RBAC + Delete PVC datadir-mysql-cluster-0
(30Gi) + Delete mysql-operator namespace + CRDs + stale Vault roles."

## This change

Phase 4 scope executed in this session (beads code-qai):

1. `terragrunt destroy -target` against 6 resources in the dbaas Tier 0 stack:
   - `module.dbaas.helm_release.mysql_cluster` — uninstalled InnoDBCluster CR
     + MySQL Router Deployment + 8 Services (mysql-cluster, -instances,
     ports 6446/6448/6447/6449/6450/8443, etc.)
   - `module.dbaas.helm_release.mysql_operator` — uninstalled MySQL Operator
     Deployment, InnoDBCluster CRD + webhook, operator ClusterRoles
   - `module.dbaas.kubernetes_namespace.mysql_operator` — deleted the ns
   - `module.dbaas.kubernetes_cluster_role.mysql_sidecar_extra` — leftover
     permissions patch that existed to work around the sidecar's kopf
     permissions bug; unused without the operator
   - `module.dbaas.kubernetes_cluster_role_binding.mysql_sidecar_extra`
   - `module.dbaas.kubernetes_config_map.mysql_extra_cnf` — used to override
     `innodb_doublewrite=OFF` via subPath mount; standalone does not need it
2. `kubectl delete pvc datadir-mysql-cluster-0 -n dbaas` — Helm does not
   garbage-collect PVCs; 30Gi reclaimed.
3. Removed 295 lines (lines 86–380) from `stacks/dbaas/modules/dbaas/main.tf`
   covering the `#### MYSQL — InnoDB Cluster via MySQL Operator` section
   and all six resources above.

The first destroy hit a Helm timeout on `mysql-cluster` uninstall ("context
deadline exceeded"). Uninstallation had in fact completed cluster-side by
that point but TF rolled back the state delta. A second `terragrunt destroy
-target` call with the same args resolved cleanly — destroyed the remaining
2 tracked resources (the first pass cleared 4) and encrypted+committed the
Tier 0 state.

## What is NOT in this change

- CRDs (`innodbclusters.mysql.oracle.com`, etc.) — Helm does delete these
  on uninstall. Verified clean: `kubectl get crd | grep mysql.oracle.com`
  returns nothing.
- Orphan PVC `datadir-mysql-cluster-0` — already deleted via kubectl; not
  a TF-managed resource.
- Stale Vault DB roles (health, linkwarden, affine, woodpecker,
  claude_memory, crowdsec, technitium) for services migrated MySQL→PG —
  sandbox denies `vault list database/roles` as credential scouting, so
  the user handles this manually.
- 2 state-commits preceding this one (`30fa411b`, `6cf3575e`) are automatic
  SOPS-encrypted-state commits produced by `scripts/tg` after each
  `terragrunt destroy` pass. Standard Tier 0 workflow.

## Verification

```
$ helm list -A | grep -E 'mysql-cluster|mysql-operator'
(no output)

$ kubectl get ns mysql-operator
Error from server (NotFound): namespaces "mysql-operator" not found

$ kubectl get pvc -n dbaas datadir-mysql-cluster-0
Error from server (NotFound): persistentvolumeclaims "datadir-mysql-cluster-0" not found

$ kubectl get pod -n dbaas -l app.kubernetes.io/instance=mysql-standalone
NAME                 READY   STATUS    RESTARTS       AGE
mysql-standalone-0   1/1     Running   1 (118m ago)   2d

$ ../../scripts/tg state list | grep -i 'mysql_operator\|mysql_cluster\|mysql_sidecar\|mysql_extra_cnf'
(no output)

$ ../../scripts/tg plan | grep -E 'mysql_cluster|mysql_operator|mysql_sidecar|mysql_extra_cnf'
(no output — Wave 2 drift is gone; remaining plan items are pre-existing
drift unrelated to this change, see Wave 3 + in-flight payslip work)
```

## Reproduce locally
1. `git pull`
2. `cd stacks/dbaas && ../../scripts/tg state list | grep mysql_cluster` → no output
3. `helm list -A | grep mysql-cluster` → no output

Closes: code-qai

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 19:19:48 +00:00
..
_template [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
actualbudget [actualbudget] Upgrade 26.3.0 → 26.4.0 for native Sankey report 2026-04-18 13:19:27 +00:00
affine [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
authentik [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
beads-server [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
blog [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
broker-sync broker-sync: add Fidelity PlanViewer CronJob (suspended) 2026-04-18 18:51:20 +00:00
changedetection [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
city-guesser [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
claude-agent-service [infra/claude-agent-service] Seed beads metadata + scratch dir at runtime 2026-04-18 14:23:19 +00:00
claude-memory [infra] Document intended ignore_changes drift-workarounds [ci skip] 2026-04-18 14:08:10 +00:00
cloudflared [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
cnpg [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
coturn [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
crowdsec [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
cyberchef [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
dashy [cleanup] Remove ollama from dashy + docs + nfs_directories 2026-04-18 11:17:59 +00:00
dawarich [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
dbaas [infra] Remove mysql InnoDB Cluster + Operator HCL (Phase 4 cleanup) [ci skip] 2026-04-18 19:19:48 +00:00
descheduler [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
diun [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
ebook2audiobook [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
ebooks [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
echo [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
excalidraw [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
external-secrets [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
f1-stream [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
foolery Add broker-sync Terraform stack (#7) 2026-04-17 21:17:45 +01:00
forgejo [forgejo] Probe /api/healthz for external monitor 2026-04-17 22:06:23 +00:00
freedify [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
freshrss [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
frigate [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
grampsweb [grampsweb] Align PVC resource to encrypted storage; imported state 2026-04-18 11:37:45 +00:00
hackmd [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
headscale [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
health [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
hermes-agent [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
homepage [cleanup] Remove ollama from dashy + docs + nfs_directories 2026-04-18 11:17:59 +00:00
immich [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
infra [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
infra-maintenance [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
insta2spotify [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
isponsorblocktv [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
jsoncrack [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
k8s-dashboard [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
k8s-portal [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
kms [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
kyverno kyverno: strip resources.limits.cpu cluster-wide via ClusterPolicy 2026-04-18 11:34:39 +00:00
linkwarden [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
mailserver [infra] Document intended ignore_changes drift-workarounds [ci skip] 2026-04-18 14:08:10 +00:00
matrix [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
meshcentral [meshcentral] Remove accidentally-committed Terragrunt-generated files 2026-04-18 12:35:44 +00:00
metallb [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
metrics-server [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
monitoring [payslip-ingest] Deploy stack + Grafana dashboard + Vault DB role 2026-04-18 19:07:05 +00:00
n8n [n8n] Fix broken DIUN auto-upgrade pipeline — missing auth token to claude-agent-service 2026-04-18 10:41:09 +00:00
navidrome [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
netbox [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
networking-toolbox [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
nextcloud [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
nfs-csi [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
novelapp [infra] Document intended ignore_changes drift-workarounds [ci skip] 2026-04-18 14:08:10 +00:00
ntfy [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
nvidia [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
onlyoffice [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
openclaw [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
osm_routing [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
owntracks [infra] Document intended ignore_changes drift-workarounds [ci skip] 2026-04-18 14:08:10 +00:00
paperless-ngx [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
payslip-ingest [payslip-ingest] Deploy stack + Grafana dashboard + Vault DB role 2026-04-18 19:07:05 +00:00
phpipam [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
platform [infra] Add Cloudflare provider to all stack lock files and generated providers 2026-04-16 16:31:36 +00:00
plotting-book [infra] Document intended ignore_changes drift-workarounds [ci skip] 2026-04-18 14:08:10 +00:00
poison-fountain [infra] Scale down unused services + remove DoH ingress 2026-04-17 18:55:52 +00:00
priority-pass [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
privatebin [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
proxmox-csi feat(storage): migrate all sensitive services to proxmox-lvm-encrypted 2026-04-15 20:15:30 +00:00
pvc-autoresizer fix: disable cert-manager webhook for pvc-autoresizer, use self-signed cert [ci skip] 2026-04-03 23:44:49 +03:00
rbac [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
real-estate-crawler [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
redis [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
reloader [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
resume [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
reverse-proxy [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
rybbit [rybbit] Narrow CF Worker routes to SITE_IDS hosts — fix free-tier quota breach 2026-04-18 13:23:15 +00:00
sealed-secrets [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
send [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
servarr [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
shadowsocks [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
speedtest [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
status-page [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
stirling-pdf [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
tandoor [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
technitium [infra] Scale down unused services + remove DoH ingress 2026-04-17 18:55:52 +00:00
terminal Add broker-sync Terraform stack (#7) 2026-04-17 21:17:45 +01:00
tor-proxy [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
trading-bot [infra] Document intended ignore_changes drift-workarounds [ci skip] 2026-04-18 14:08:10 +00:00
traefik [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
travel_blog [infra] Scale down unused services + remove DoH ingress 2026-04-17 18:55:52 +00:00
tuya-bridge [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
uptime-kuma [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
url [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
vault [payslip-ingest] Deploy stack + Grafana dashboard + Vault DB role 2026-04-18 19:07:05 +00:00
vaultwarden [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
vpa [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
wealthfolio wealthfolio: bump memory 64Mi → 1Gi (limit) / 256Mi (request) 2026-04-18 19:13:05 +00:00
webhook_handler [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
whisper [whisper] Remove ollama_tcp IngressRouteTCP (ollama decom) 2026-04-18 11:11:21 +00:00
wireguard [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
woodpecker [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
xray [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
ytdlp [ytdlp] Remove ollama_host variable and fallback env vars 2026-04-18 11:13:42 +00:00