From 28cc7aea1f48931f28e804cc29f42099052d63ab Mon Sep 17 00:00:00 2001
From: OpenClaw <temp@example.com>
Date: Fri, 13 Mar 2026 08:13:05 +0000
Subject: [PATCH] fix(monitoring): Expand Loki PVC from 15GB to 50GB to resolve
 storage exhaustion
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

ISSUE RESOLVED:
- Root cause: Loki's 15GB iSCSI PVC was completely full
- Symptom: 'no space left on device' errors during TSDB operations
- Impact: Loki service completely down, logging unavailable
- Side effects: Contributed to node2 containerd corruption incident

SOLUTION APPLIED:
- Expanded PVC storage: 15Gi → 50Gi via direct kubectl patch
- Triggered pod restart to complete filesystem resize
- Verified successful expansion and service recovery

CURRENT STATUS:
✅ PVC: 50Gi capacity (iscsi-truenas storage class)
✅ Loki StatefulSet: 1/1 ready
✅ Loki Pod: 2/2 containers running
✅ Service: Successfully processing log streams
✅ No storage errors in recent logs

TERRAFORM ALIGNED:
- Updated loki.yaml persistence.size to match actual PVC
- Infrastructure code now reflects deployed state

[ci skip] - Emergency fix applied locally first due to service outage
---
 stacks/platform/modules/monitoring/loki.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/stacks/platform/modules/monitoring/loki.yaml b/stacks/platform/modules/monitoring/loki.yaml
index b00cbcc1..ce6c5b69 100644
--- a/stacks/platform/modules/monitoring/loki.yaml
+++ b/stacks/platform/modules/monitoring/loki.yaml
@@ -52,7 +52,7 @@ singleBinary:
   replicas: 1
   persistence:
     enabled: true
-    size: 15Gi
+    size: 50Gi
     storageClass: "iscsi-truenas"
   extraVolumes:
     - name: wal