viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(monitoring): Expand Loki PVC from 15GB to 50GB to resolve storage exhaustion

Browse source 
ISSUE RESOLVED:
- Root cause: Loki's 15GB iSCSI PVC was completely full
- Symptom: 'no space left on device' errors during TSDB operations
- Impact: Loki service completely down, logging unavailable
- Side effects: Contributed to node2 containerd corruption incident

SOLUTION APPLIED:
- Expanded PVC storage: 15Gi → 50Gi via direct kubectl patch
- Triggered pod restart to complete filesystem resize
- Verified successful expansion and service recovery

CURRENT STATUS:
✅ PVC: 50Gi capacity (iscsi-truenas storage class)
✅ Loki StatefulSet: 1/1 ready
✅ Loki Pod: 2/2 containers running
✅ Service: Successfully processing log streams
✅ No storage errors in recent logs

TERRAFORM ALIGNED:
- Updated loki.yaml persistence.size to match actual PVC
- Infrastructure code now reflects deployed state

[ci skip] - Emergency fix applied locally first due to service outage
This commit is contained in:
OpenClaw 2026-03-13 08:13:05 +00:00 • committed by Viktor Barzin
parent f30c62ee5c
commit 28cc7aea1f
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
stacks/platform/modules/monitoring/loki.yaml
View file

@ -52,7 +52,7 @@ singleBinary:
replicas: 1
persistence:
enabled: true
size: 15Gi
size: 50Gi
storageClass: "iscsi-truenas"
extraVolumes:
- name: wal