fix: restore technitium MySQL query logging with Vault auto-rotation [ci skip]
Query logs stopped syncing on 2026-03-16 due to password mismatch after MySQL cluster rebuild and Technitium app config reset. - Add Vault static role mysql-technitium (7-day rotation) - Add ExternalSecret for technitium-db-creds in technitium namespace - Add password-sync CronJob (6h) to push rotated password to Technitium API - Update Grafana datasource to use ESO-managed password - Remove stale technitium_db_password variable (replaced by ESO) - Update databases.md and restore-mysql.md runbook
This commit is contained in:
parent
1d7244e47a
commit
9492874c43
6 changed files with 149 additions and 19 deletions
|
|
@ -117,6 +117,7 @@ graph TB
|
|||
- nextcloud
|
||||
- shlink
|
||||
- grafana
|
||||
- technitium (DNS query logs via QueryLogsMySqlApp plugin)
|
||||
|
||||
### Redis
|
||||
|
||||
|
|
@ -157,17 +158,18 @@ graph TB
|
|||
- nextcloud
|
||||
- shlink
|
||||
- grafana
|
||||
- technitium (password synced to Technitium DNS app via CronJob every 6h)
|
||||
|
||||
**Excluded from Rotation**:
|
||||
- authentik (uses PgBouncer, incompatible)
|
||||
- technitium, crowdsec (Helm-baked credentials)
|
||||
- crowdsec (Helm-baked credentials)
|
||||
- Root users (manual management)
|
||||
|
||||
**How Rotation Works**:
|
||||
1. Vault creates new user with same permissions
|
||||
2. App fetches new credentials on next Vault lease renewal
|
||||
3. Old credentials revoked after grace period
|
||||
4. Zero-downtime rotation
|
||||
1. Vault rotates the MySQL user's password (static role, 7-day period)
|
||||
2. ExternalSecrets Operator syncs new password to K8s Secret (15-min refresh)
|
||||
3. Apps read from K8s Secret via `secret_key_ref` env vars
|
||||
4. Special case: Technitium stores its MySQL connection in internal app config, so a CronJob pushes the rotated password to the Technitium API every 6 hours
|
||||
|
||||
## Configuration
|
||||
|
||||
|
|
|
|||
|
|
@ -62,7 +62,26 @@ for db in speedtest wrongmove codimd nextcloud shlink grafana; do
|
|||
done
|
||||
```
|
||||
|
||||
### 5. InnoDB Cluster Recovery
|
||||
### 5. Verify application MySQL users exist
|
||||
|
||||
After any cluster rebuild or PVC recreation, the MySQL operator only recreates its own system users. Application users may be lost.
|
||||
|
||||
```bash
|
||||
ROOT_PWD=$(kubectl get secret cluster-secret -n dbaas -o jsonpath='{.data.ROOT_PASSWORD}' | base64 -d)
|
||||
|
||||
# Check all expected application users exist
|
||||
kubectl exec -n dbaas mysql-cluster-0 -c mysql -- mysql -u root -p"$ROOT_PWD" \
|
||||
-e "SELECT user, host FROM mysql.user WHERE user IN ('nextcloud','forgejo','crowdsec','grafana','speedtest','wrongmove','codimd','shlink','technitium','uptimekuma');"
|
||||
|
||||
# If users are missing, force Vault to re-rotate their credentials:
|
||||
# vault write -f database/rotate-role/mysql-<app>
|
||||
# This will recreate the user with the correct password.
|
||||
#
|
||||
# For technitium specifically, also run the password sync CronJob:
|
||||
# kubectl create job --from=cronjob/technitium-password-sync technitium-pw-resync -n technitium
|
||||
```
|
||||
|
||||
### 6. InnoDB Cluster Recovery
|
||||
If the InnoDB Cluster itself is broken (not just data loss):
|
||||
```bash
|
||||
# Check cluster status via MySQL Shell
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue