fix: restore technitium MySQL query logging with Vault auto-rotation [ci skip]

Query logs stopped syncing on 2026-03-16 due to password mismatch after
MySQL cluster rebuild and Technitium app config reset.

- Add Vault static role mysql-technitium (7-day rotation)
- Add ExternalSecret for technitium-db-creds in technitium namespace
- Add password-sync CronJob (6h) to push rotated password to Technitium API
- Update Grafana datasource to use ESO-managed password
- Remove stale technitium_db_password variable (replaced by ESO)
- Update databases.md and restore-mysql.md runbook
This commit is contained in:
Viktor Barzin 2026-04-06 13:00:49 +03:00
parent 1d7244e47a
commit 9492874c43
6 changed files with 149 additions and 19 deletions

View file

@ -117,6 +117,7 @@ graph TB
- nextcloud
- shlink
- grafana
- technitium (DNS query logs via QueryLogsMySqlApp plugin)
### Redis
@ -157,17 +158,18 @@ graph TB
- nextcloud
- shlink
- grafana
- technitium (password synced to Technitium DNS app via CronJob every 6h)
**Excluded from Rotation**:
- authentik (uses PgBouncer, incompatible)
- technitium, crowdsec (Helm-baked credentials)
- crowdsec (Helm-baked credentials)
- Root users (manual management)
**How Rotation Works**:
1. Vault creates new user with same permissions
2. App fetches new credentials on next Vault lease renewal
3. Old credentials revoked after grace period
4. Zero-downtime rotation
1. Vault rotates the MySQL user's password (static role, 7-day period)
2. ExternalSecrets Operator syncs new password to K8s Secret (15-min refresh)
3. Apps read from K8s Secret via `secret_key_ref` env vars
4. Special case: Technitium stores its MySQL connection in internal app config, so a CronJob pushes the rotated password to the Technitium API every 6 hours
## Configuration

View file

@ -62,7 +62,26 @@ for db in speedtest wrongmove codimd nextcloud shlink grafana; do
done
```
### 5. InnoDB Cluster Recovery
### 5. Verify application MySQL users exist
After any cluster rebuild or PVC recreation, the MySQL operator only recreates its own system users. Application users may be lost.
```bash
ROOT_PWD=$(kubectl get secret cluster-secret -n dbaas -o jsonpath='{.data.ROOT_PASSWORD}' | base64 -d)
# Check all expected application users exist
kubectl exec -n dbaas mysql-cluster-0 -c mysql -- mysql -u root -p"$ROOT_PWD" \
-e "SELECT user, host FROM mysql.user WHERE user IN ('nextcloud','forgejo','crowdsec','grafana','speedtest','wrongmove','codimd','shlink','technitium','uptimekuma');"
# If users are missing, force Vault to re-rotate their credentials:
# vault write -f database/rotate-role/mysql-<app>
# This will recreate the user with the correct password.
#
# For technitium specifically, also run the password sync CronJob:
# kubectl create job --from=cronjob/technitium-password-sync technitium-pw-resync -n technitium
```
### 6. InnoDB Cluster Recovery
If the InnoDB Cluster itself is broken (not just data loss):
```bash
# Check cluster status via MySQL Shell