111 lines
3.6 KiB
Markdown
111 lines
3.6 KiB
Markdown
# OpenClaw Cluster Management Agent — Design
|
|
|
|
**Date**: 2026-02-21
|
|
**Status**: Approved
|
|
|
|
## Goal
|
|
|
|
Build a proactive cluster management agent that runs scheduled health checks every 30 minutes, auto-fixes safe issues, and alerts via Slack. The agent is "taught" via an OpenClaw skill and a reusable health check script.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
CronJob (every 30min)
|
|
└─ kubectl exec into OpenClaw pod
|
|
└─ /workspace/infra/.claude/cluster-health.sh
|
|
├─ kubectl get nodes (check health)
|
|
├─ kubectl get pods -A (find problems)
|
|
├─ kubectl delete pod (evicted/stuck)
|
|
└─ curl Slack webhook (report)
|
|
```
|
|
|
|
Interactive path: User asks OpenClaw via UI -> `cluster-health` skill triggers -> runs same script -> LLM analyzes output and can do deeper investigation.
|
|
|
|
## Components
|
|
|
|
### 1. `cluster-health` skill (`.claude/skills/cluster-health/SKILL.md`)
|
|
|
|
Teaches OpenClaw:
|
|
- What health checks to run
|
|
- What's safe to auto-fix vs alert-only
|
|
- How to format Slack alerts
|
|
- How to do deeper investigation when asked interactively
|
|
|
|
Trigger conditions: "check cluster", "cluster health", "what's wrong", "health check", etc.
|
|
|
|
### 2. `cluster-health.sh` helper script (`.claude/cluster-health.sh`)
|
|
|
|
Reusable script that performs all checks:
|
|
|
|
**Checks:**
|
|
- Node health (NotReady, MemoryPressure, DiskPressure, PIDPressure)
|
|
- Pod health (CrashLoopBackOff, ImagePullBackOff, Error, OOMKilled, Pending)
|
|
- Evicted pods
|
|
- Failed deployments (unavailable replicas)
|
|
- Pending PVCs
|
|
- Resource pressure (high CPU/memory allocation)
|
|
- Failed CronJobs
|
|
- DaemonSet health (missing pods)
|
|
|
|
**Safe auto-fix actions:**
|
|
- Delete evicted pods
|
|
- Delete completed/succeeded pods older than 24h
|
|
- Restart (delete) pods in CrashLoopBackOff for more than 1 hour
|
|
|
|
**Alert-only (never auto-fix):**
|
|
- Node NotReady
|
|
- Persistent OOMKilled
|
|
- ImagePullBackOff
|
|
- Pending PVCs
|
|
- Failed deployments with 0 available replicas
|
|
|
|
**Output:**
|
|
- Structured text summary
|
|
- Posts to Slack via webhook
|
|
- Exit code 0 = healthy, 1 = issues found
|
|
|
|
### 3. Kubernetes CronJob (in `modules/kubernetes/openclaw/main.tf`)
|
|
|
|
- Schedule: `*/30 * * * *`
|
|
- Container: `bitnami/kubectl` (minimal image with kubectl)
|
|
- Command: `kubectl exec deploy/openclaw -n openclaw -- /bin/bash /workspace/infra/.claude/cluster-health.sh`
|
|
- ServiceAccount with RBAC to exec into pods in `openclaw` namespace
|
|
- `concurrencyPolicy: Forbid`
|
|
- `failedJobsHistoryLimit: 3`
|
|
- `successfulJobsHistoryLimit: 3`
|
|
|
|
### 4. Slack Integration
|
|
|
|
- Webhook URL from `openclaw_skill_secrets["slack"]` (already configured)
|
|
- Passed as `SLACK_WEBHOOK_URL` env var to the OpenClaw pod
|
|
|
|
## Slack Message Format
|
|
|
|
```
|
|
:white_check_mark: Cluster Health Check — All Clear
|
|
Nodes: 5/5 Ready | Pods: 142 Running | 0 Issues
|
|
```
|
|
|
|
```
|
|
:warning: Cluster Health Check — 3 Issues Found
|
|
|
|
Auto-fixed:
|
|
- Deleted 4 evicted pods in monitoring namespace
|
|
- Restarted stuck pod calibre-web-xyz (CrashLoopBackOff >1h)
|
|
|
|
Needs attention:
|
|
- Node k8s-node3: MemoryPressure condition detected
|
|
- PVC data-tandoor pending for 45 minutes
|
|
```
|
|
|
|
## Decisions
|
|
|
|
| Decision | Choice | Rationale |
|
|
|----------|--------|-----------|
|
|
| Mode | Proactive (scheduled) | Want automated monitoring |
|
|
| Alert channel | Slack | Existing webhook in openclaw_skill_secrets |
|
|
| Auto-fix | Safe fixes only | Delete evicted, restart stuck; alert for the rest |
|
|
| Frequency | 30 minutes | Balance between detection speed and overhead |
|
|
| Checks scope | Standard K8s health | Pod/node/deployment/PVC/CronJob/DaemonSet |
|
|
| Trigger mechanism | CronJob execs into OpenClaw pod | Reuses OpenClaw's tools; LLM available interactively |
|
|
| Fallback | None | Uptime Kuma monitors OpenClaw availability |
|