57250cfda2
mysql-standalone OOMKilled May 8 18:05 (anon-rss 2 GB at the 2 Gi limit). innodb_buffer_pool_size=1Gi plus connection buffers and InnoDB internals don't fit in 2 Gi. Bumping limit to 4 Gi (request 3 Gi) leaves headroom without changing the buffer pool config. /srv/nfs was at 90% (1.7T / 2T); grew the underlying pve/nfs-data LV 1 TiB online and ran resize2fs (now 60% used). Triggered by surfacing during the 2026-05-09 IO-pressure post-mortem; thinpool had ~4.6 TiB free. The post-mortem also covers the stale-NFS-client trigger (legacy /usr/local/bin/weekly-backup pointing at the decommissioned TrueNAS IP) and the resulting wedged kthread on the PVE host. Script removed and node_exporter restarted out-of-band; kthread will clear at next PVE reboot. See docs/post-mortems/2026-05-09-io-pressure-stale-nfs.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| 2026-03-16-kured-containerd-cascade-outage.html | ||
| 2026-03-16-nfs-csi-cascade-failure.md | ||
| 2026-04-14-nfs-fsid0-dns-vault-outage.md | ||
| 2026-04-14-postmortem-pipeline-test.md | ||
| 2026-04-18-authentik-outpost-shm-full.md | ||
| 2026-04-19-registry-orphan-index.md | ||
| 2026-04-22-vault-raft-leader-deadlock.md | ||
| 2026-05-09-io-pressure-stale-nfs.md | ||
| index.html | ||