diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index be8adb82..a7aa0891 100755 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -78,14 +78,14 @@ Violations cause state drift, which causes future applies to break or silently r ## Resource Management Patterns - **CPU**: All CPU limits removed cluster-wide (CFS throttling). Only set CPU requests based on actual usage. - **Memory**: Set explicit `requests=limits` based on VPA upperBound. Target: upperBound x 1.2 for stable services, x 1.3 for GPU/volatile workloads. -- **VPA (Goldilocks)**: Must be `Initial` mode (not `Auto`) — Auto conflicts with Terraform's declarative resource management. +- **Right-sizing**: VPA/Goldilocks was **REMOVED 2026-06-12** (etcd-load-reduction — 349 VPAs all ran `updateMode=Off`, costing ~800 etcd objects + continuous recommender writes + a pod-creation admission webhook for dashboard-only value). Right-size **on demand with `krr`** (Robusta, Dockerized from the devvm — no cluster install, no admission webhook, no eviction risk; reads Prometheus). Set container resources explicitly in TF from krr output. - **LimitRange**: Tier-based defaults silently apply to pods with `resources: {}`. Always set explicit resources on containers needing more than defaults. Tier 3-edge and 4-aux now use Burstable QoS (request < limit) to reduce scheduler pressure. - **Democratic-CSI sidecars**: Must set explicit resources (32-80Mi) in Helm values — 17 sidecars default to 256Mi each via LimitRange. `csiProxy` is a TOP-LEVEL chart key, not nested under controller/node. - **ResourceQuota blocks rolling updates**: When quota is tight, scale to 0 then back to 1 instead of RollingUpdate. Or use Recreate strategy. - **Kyverno ndots drift**: Kyverno injects dns_config on all pods. Every `kubernetes_deployment`, `kubernetes_stateful_set`, and `kubernetes_cron_job_v1` MUST include `lifecycle { ignore_changes = [spec[0].template[0].spec[0].dns_config] # KYVERNO_LIFECYCLE_V1 }` (use `spec[0].job_template[0].spec[0].template[0].spec[0].dns_config` for CronJobs). The `# KYVERNO_LIFECYCLE_V1` marker is the canonical discoverability tag — grep for it to locate every site. A shared Terraform module was considered but `ignore_changes` only accepts static attribute paths (not module outputs, locals, or expressions), so the snippet convention is the only viable path. Full rationale and copy-paste snippets in `AGENTS.md` → "Kyverno Drift Suppression". - **NVIDIA GPU operator resources**: dcgm-exporter and cuda-validator resources configurable via `dcgmExporter.resources` and `validator.resources` in nvidia values.yaml. - **Pin database versions**: Disable Diun (image update monitoring) for MySQL, PostgreSQL, Redis. -- **Quarterly right-sizing**: Check Goldilocks dashboard. Compare VPA upperBound to current request. Also check for under-provisioned (VPA upper > request x 0.8). +- **Quarterly right-sizing**: Run `krr` (Dockerized, against Prometheus) for recommendations; compare to current requests and adjust in TF. (Goldilocks dashboard removed 2026-06-12.) ## CI/CD Architecture — GHA Builds + Woodpecker Deploy @@ -318,7 +318,7 @@ resource "kubernetes_persistent_volume_claim" "data_encrypted" { - **CrowdSec Helm upgrade times out**: `terragrunt apply` on platform stack causes CrowdSec Helm release to get stuck in `pending-upgrade`. Workaround: `helm rollback crowdsec -n crowdsec`. Root cause: likely ResourceQuota CPU at 302% preventing pods from passing readiness probes. Needs investigation. - **OpenClaw config is writable**: OpenClaw writes to `openclaw.json` at runtime (doctor --fix, plugin auto-enable). Never use subPath ConfigMap mounts for it — use an init container to copy into a writable volume. Needs 2Gi memory + `NODE_OPTIONS=--max-old-space-size=1536`. **`mcp.servers` baked into the ConfigMap-loaded openclaw.json gets stripped by `doctor --fix`** — register MCP servers via `openclaw mcp set ` in the container startup command instead (CLI-written entries persist across doctor runs). Current servers wired this way: `ha`, `context7`, `playwright` (sidecar at `localhost:3000/mcp`). - **OpenClaw memory-core indexes `/workspace/memory/`, not `/home/node/.openclaw/memory/`**: `/home/node/.openclaw/memory/main.sqlite` is the index store, NOT a content source. Files written under `/home/node/.openclaw/memory/projects//*.md` will NOT be indexed. To populate memory-core, write Markdown under `/workspace/memory/projects//` and run `openclaw memory index --force`. This is what the daily `memory-sync` CronJob in `stacks/openclaw/` does for claude-memory → OpenClaw sync. -- **Goldilocks VPA sets limits**: When increasing memory requests, always set explicit `limits` too — Goldilocks may have added a limit that blocks the change. +- **(Obsolete 2026-06-12) Goldilocks VPA**: VPA/Goldilocks was uninstalled (etcd-load-reduction); the old "Goldilocks may have added a limit that blocks the change" gotcha no longer applies. Use `krr` for right-sizing. ## User Preferences - **Calendar**: Nextcloud at `nextcloud.viktorbarzin.me` diff --git a/stacks/descheduler/values.yaml b/stacks/descheduler/values.yaml index dca0d0fc..fe0578f3 100644 --- a/stacks/descheduler/values.yaml +++ b/stacks/descheduler/values.yaml @@ -52,7 +52,7 @@ namespaceOverride: "" commonLabels: {} cronJobApiVersion: "batch/v1" -schedule: "*/5 * * * *" +schedule: "0 * * * *" # hourly (was */5; 2026-06-12 etcd-load-reduction — fewer list/evict cycles, rebalancing isn't time-critical) suspend: false # startingDeadlineSeconds: 200 successfulJobsHistoryLimit: 10 diff --git a/stacks/kyverno/modules/kyverno/main.tf b/stacks/kyverno/modules/kyverno/main.tf index 84ea7d63..3c77d653 100644 --- a/stacks/kyverno/modules/kyverno/main.tf +++ b/stacks/kyverno/modules/kyverno/main.tf @@ -30,9 +30,24 @@ resource "helm_release" "kyverno" { forceFailurePolicyIgnore = { enabled = true } + # Reporting fully disabled (2026-06-12, etcd-load-reduction). policyReports + # were already off, so admission/aggregate/background reporting generated + # ephemeralreports + an hourly all-resource etcd re-scan for NO user-facing + # output. Admission enforcement (deny-* policies) and Keel mutation are + # independent of reporting; policy violations surface via Loki->Slack. This + # removes a steady-state etcd write/scan load (control-plane flap mitigation). policyReports = { enabled = false } + admissionReports = { + enabled = false + } + aggregateReports = { + enabled = false + } + backgroundScan = { + enabled = false + } } reportsController = { diff --git a/stacks/vpa/main.tf b/stacks/vpa/main.tf index b4abe0a5..d51e2f67 100644 --- a/stacks/vpa/main.tf +++ b/stacks/vpa/main.tf @@ -1,7 +1,15 @@ variable "tls_secret_name" { type = string } -module "vpa" { - source = "./modules/vpa" - tls_secret_name = var.tls_secret_name - tier = local.tiers.cluster -} +# VPA / Goldilocks REMOVED 2026-06-12 (etcd-load-reduction; reverses the re-add +# after memory 2431, ties to code-oflt). All 349 VPAs ran updateMode=Off (no +# auto-right-sizing) yet cost ~800 etcd objects, continuous recommender writes, +# and a pod-creation admission webhook — pure etcd overhead feeding only the +# dashboard. Right-size on demand with krr (Dockerized, no cluster footprint). +# +# The `module "vpa"` block was removed so `scripts/tg apply` DESTROYS the helm +# releases (vpa, goldilocks), the goldilocks-vpa-auto-mode ClusterPolicy, the +# dashboard ingress, and the vpa namespace. The chart-installed VPA CRDs (Helm +# keeps CRDs on uninstall) and any leftover VPA/checkpoint CRs are removed +# post-apply (cascade) via: +# kubectl delete crd verticalpodautoscalers.autoscaling.k8s.io \ +# verticalpodautoscalercheckpoints.autoscaling.k8s.io diff --git a/stacks/vpa/modules/vpa/main.tf b/stacks/vpa/modules/vpa/main.tf deleted file mode 100644 index dd73540a..00000000 --- a/stacks/vpa/modules/vpa/main.tf +++ /dev/null @@ -1,175 +0,0 @@ -variable "tls_secret_name" { - type = string - sensitive = true -} -variable "tier" { type = string } - -resource "kubernetes_namespace" "vpa" { - metadata { - name = "vpa" - labels = { - tier = var.tier - "keel.sh/enrolled" = "true" - } - } -} - -module "tls_secret" { - source = "../../../../modules/kubernetes/setup_tls_secret" - namespace = kubernetes_namespace.vpa.metadata[0].name - tls_secret_name = var.tls_secret_name -} - -# ----------------------------------------------------------------------------- -# VPA — Vertical Pod Autoscaler (Fairwinds Helm chart) -# ----------------------------------------------------------------------------- -resource "helm_release" "vpa" { - namespace = kubernetes_namespace.vpa.metadata[0].name - create_namespace = false - name = "vpa" - atomic = true - - repository = "https://charts.fairwinds.com/stable" - chart = "vpa" - - values = [yamlencode({ - recommender = { - enabled = true - resources = { - requests = { - cpu = "50m" - memory = "200Mi" - } - limits = { - memory = "200Mi" - } - } - } - updater = { - enabled = true - resources = { - requests = { - cpu = "50m" - memory = "200Mi" - } - limits = { - memory = "200Mi" - } - } - } - admissionController = { - enabled = true - resources = { - requests = { - cpu = "50m" - memory = "200Mi" - } - limits = { - memory = "200Mi" - } - } - } - })] -} - -# ----------------------------------------------------------------------------- -# Goldilocks — VPA dashboard (Fairwinds Helm chart) -# ----------------------------------------------------------------------------- -resource "helm_release" "goldilocks" { - namespace = kubernetes_namespace.vpa.metadata[0].name - create_namespace = false - name = "goldilocks" - atomic = true - - repository = "https://charts.fairwinds.com/stable" - chart = "goldilocks" - - values = [yamlencode({ - controller = { - flags = { - on-by-default = "true" - } - } - dashboard = { - replicaCount = 1 - flags = { - on-by-default = "true" - } - } - })] - - depends_on = [helm_release.vpa] -} - -# ----------------------------------------------------------------------------- -# Ingress — Goldilocks dashboard at goldilocks.viktorbarzin.me -# ----------------------------------------------------------------------------- -module "ingress" { - source = "../../../../modules/kubernetes/ingress_factory" - dns_type = "proxied" - namespace = kubernetes_namespace.vpa.metadata[0].name - name = "goldilocks" - service_name = "goldilocks-dashboard" - port = 80 - tls_secret_name = var.tls_secret_name - auth = "required" - extra_annotations = { - "gethomepage.dev/enabled" = "true" - "gethomepage.dev/name" = "Goldilocks" - "gethomepage.dev/description" = "Resource recommendations" - "gethomepage.dev/icon" = "mdi-scale-balance" - "gethomepage.dev/group" = "Core Platform" - "gethomepage.dev/pod-selector" = "" - } - - depends_on = [helm_release.goldilocks] -} - -# ----------------------------------------------------------------------------- -# Kyverno policy — label namespaces for VPA observe-only mode -# ----------------------------------------------------------------------------- -# Goldilocks reads the goldilocks.fairwinds.com/vpa-update-mode label on -# namespaces to decide the updateMode for VPA objects it creates. -# All namespaces get "off" — Terraform is the authoritative source of truth -# for container resources. Goldilocks provides recommendations only. - -resource "kubernetes_manifest" "vpa_auto_mode_label" { - manifest = { - apiVersion = "kyverno.io/v1" - kind = "ClusterPolicy" - metadata = { - name = "goldilocks-vpa-auto-mode" - annotations = { - "policies.kyverno.io/title" = "Goldilocks VPA Observe-Only Mode" - "policies.kyverno.io/description" = "Sets VPA update mode to off for all namespaces. Terraform owns container resources; Goldilocks provides recommendations only." - } - } - spec = { - rules = [ - { - name = "label-vpa-off-all" - match = { - any = [ - { - resources = { - kinds = ["Namespace"] - } - } - ] - } - mutate = { - patchStrategicMerge = { - metadata = { - labels = { - "goldilocks.fairwinds.com/vpa-update-mode" = "off" - } - } - } - } - }, - ] - } - } - - depends_on = [helm_release.goldilocks] -}