etcd-load-reduction: remove VPA/Goldilocks, disable kyverno reporting, descheduler hourly
The control-plane flap (etcd lease-renewal timeouts) recurred. Rather than move etcd to SSD (code-oflt, deferred again), the chosen direction is to REDUCE etcd load enough that the leader-election-timeout band-aid (renew 10s->30s) becomes removable. These are the big, clean cuts: 1. Remove VPA/Goldilocks (stacks/vpa emptied). All 349 VPAs ran updateMode=Off (no auto-right-sizing) yet cost ~800 etcd objects + continuous recommender writes + a pod-creation admission webhook, purely to feed a dashboard. krr (Dockerized, on-demand) replaces it. Reverses the re-add after memory 2431. 2. Disable kyverno reporting (admission/aggregate/background). policyReports were already off, so the pipeline generated ephemeralreports + an hourly all-resource etcd re-scan for NO user-facing output. Admission enforcement (deny-* policies) and Keel mutation are unaffected; violations surface via Loki->Slack. 3. descheduler */5 -> hourly (fewer list/evict cycles; rebalancing isn't urgent). Deferred (poor ROI / unsafe as planned): ESO refreshInterval 15m->1h is a ~20-stack sprawl for ~0.1 writes/s; keel background=false is invalid for a mutate-existing policy and its churn is apply-time not steady-state. Both filed as follow-up beads. Post-apply: delete the chart-orphaned VPA CRDs to cascade-clean leftover CRs. Then measure etcd apply-latency and revert the timeouts. Docs updated (VPA/Goldilocks -> krr). See memory 5402-5407. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
16adda2c48
commit
0216e993dc
5 changed files with 32 additions and 184 deletions
|
|
@ -78,14 +78,14 @@ Violations cause state drift, which causes future applies to break or silently r
|
|||
## Resource Management Patterns
|
||||
- **CPU**: All CPU limits removed cluster-wide (CFS throttling). Only set CPU requests based on actual usage.
|
||||
- **Memory**: Set explicit `requests=limits` based on VPA upperBound. Target: upperBound x 1.2 for stable services, x 1.3 for GPU/volatile workloads.
|
||||
- **VPA (Goldilocks)**: Must be `Initial` mode (not `Auto`) — Auto conflicts with Terraform's declarative resource management.
|
||||
- **Right-sizing**: VPA/Goldilocks was **REMOVED 2026-06-12** (etcd-load-reduction — 349 VPAs all ran `updateMode=Off`, costing ~800 etcd objects + continuous recommender writes + a pod-creation admission webhook for dashboard-only value). Right-size **on demand with `krr`** (Robusta, Dockerized from the devvm — no cluster install, no admission webhook, no eviction risk; reads Prometheus). Set container resources explicitly in TF from krr output.
|
||||
- **LimitRange**: Tier-based defaults silently apply to pods with `resources: {}`. Always set explicit resources on containers needing more than defaults. Tier 3-edge and 4-aux now use Burstable QoS (request < limit) to reduce scheduler pressure.
|
||||
- **Democratic-CSI sidecars**: Must set explicit resources (32-80Mi) in Helm values — 17 sidecars default to 256Mi each via LimitRange. `csiProxy` is a TOP-LEVEL chart key, not nested under controller/node.
|
||||
- **ResourceQuota blocks rolling updates**: When quota is tight, scale to 0 then back to 1 instead of RollingUpdate. Or use Recreate strategy.
|
||||
- **Kyverno ndots drift**: Kyverno injects dns_config on all pods. Every `kubernetes_deployment`, `kubernetes_stateful_set`, and `kubernetes_cron_job_v1` MUST include `lifecycle { ignore_changes = [spec[0].template[0].spec[0].dns_config] # KYVERNO_LIFECYCLE_V1 }` (use `spec[0].job_template[0].spec[0].template[0].spec[0].dns_config` for CronJobs). The `# KYVERNO_LIFECYCLE_V1` marker is the canonical discoverability tag — grep for it to locate every site. A shared Terraform module was considered but `ignore_changes` only accepts static attribute paths (not module outputs, locals, or expressions), so the snippet convention is the only viable path. Full rationale and copy-paste snippets in `AGENTS.md` → "Kyverno Drift Suppression".
|
||||
- **NVIDIA GPU operator resources**: dcgm-exporter and cuda-validator resources configurable via `dcgmExporter.resources` and `validator.resources` in nvidia values.yaml.
|
||||
- **Pin database versions**: Disable Diun (image update monitoring) for MySQL, PostgreSQL, Redis.
|
||||
- **Quarterly right-sizing**: Check Goldilocks dashboard. Compare VPA upperBound to current request. Also check for under-provisioned (VPA upper > request x 0.8).
|
||||
- **Quarterly right-sizing**: Run `krr` (Dockerized, against Prometheus) for recommendations; compare to current requests and adjust in TF. (Goldilocks dashboard removed 2026-06-12.)
|
||||
|
||||
## CI/CD Architecture — GHA Builds + Woodpecker Deploy
|
||||
|
||||
|
|
@ -318,7 +318,7 @@ resource "kubernetes_persistent_volume_claim" "data_encrypted" {
|
|||
- **CrowdSec Helm upgrade times out**: `terragrunt apply` on platform stack causes CrowdSec Helm release to get stuck in `pending-upgrade`. Workaround: `helm rollback crowdsec <rev> -n crowdsec`. Root cause: likely ResourceQuota CPU at 302% preventing pods from passing readiness probes. Needs investigation.
|
||||
- **OpenClaw config is writable**: OpenClaw writes to `openclaw.json` at runtime (doctor --fix, plugin auto-enable). Never use subPath ConfigMap mounts for it — use an init container to copy into a writable volume. Needs 2Gi memory + `NODE_OPTIONS=--max-old-space-size=1536`. **`mcp.servers` baked into the ConfigMap-loaded openclaw.json gets stripped by `doctor --fix`** — register MCP servers via `openclaw mcp set <name> <json>` in the container startup command instead (CLI-written entries persist across doctor runs). Current servers wired this way: `ha`, `context7`, `playwright` (sidecar at `localhost:3000/mcp`).
|
||||
- **OpenClaw memory-core indexes `/workspace/memory/`, not `/home/node/.openclaw/memory/`**: `/home/node/.openclaw/memory/main.sqlite` is the index store, NOT a content source. Files written under `/home/node/.openclaw/memory/projects/<x>/*.md` will NOT be indexed. To populate memory-core, write Markdown under `/workspace/memory/projects/<source>/` and run `openclaw memory index --force`. This is what the daily `memory-sync` CronJob in `stacks/openclaw/` does for claude-memory → OpenClaw sync.
|
||||
- **Goldilocks VPA sets limits**: When increasing memory requests, always set explicit `limits` too — Goldilocks may have added a limit that blocks the change.
|
||||
- **(Obsolete 2026-06-12) Goldilocks VPA**: VPA/Goldilocks was uninstalled (etcd-load-reduction); the old "Goldilocks may have added a limit that blocks the change" gotcha no longer applies. Use `krr` for right-sizing.
|
||||
|
||||
## User Preferences
|
||||
- **Calendar**: Nextcloud at `nextcloud.viktorbarzin.me`
|
||||
|
|
|
|||
|
|
@ -52,7 +52,7 @@ namespaceOverride: ""
|
|||
commonLabels: {}
|
||||
|
||||
cronJobApiVersion: "batch/v1"
|
||||
schedule: "*/5 * * * *"
|
||||
schedule: "0 * * * *" # hourly (was */5; 2026-06-12 etcd-load-reduction — fewer list/evict cycles, rebalancing isn't time-critical)
|
||||
suspend: false
|
||||
# startingDeadlineSeconds: 200
|
||||
successfulJobsHistoryLimit: 10
|
||||
|
|
|
|||
|
|
@ -30,9 +30,24 @@ resource "helm_release" "kyverno" {
|
|||
forceFailurePolicyIgnore = {
|
||||
enabled = true
|
||||
}
|
||||
# Reporting fully disabled (2026-06-12, etcd-load-reduction). policyReports
|
||||
# were already off, so admission/aggregate/background reporting generated
|
||||
# ephemeralreports + an hourly all-resource etcd re-scan for NO user-facing
|
||||
# output. Admission enforcement (deny-* policies) and Keel mutation are
|
||||
# independent of reporting; policy violations surface via Loki->Slack. This
|
||||
# removes a steady-state etcd write/scan load (control-plane flap mitigation).
|
||||
policyReports = {
|
||||
enabled = false
|
||||
}
|
||||
admissionReports = {
|
||||
enabled = false
|
||||
}
|
||||
aggregateReports = {
|
||||
enabled = false
|
||||
}
|
||||
backgroundScan = {
|
||||
enabled = false
|
||||
}
|
||||
}
|
||||
|
||||
reportsController = {
|
||||
|
|
|
|||
|
|
@ -1,7 +1,15 @@
|
|||
variable "tls_secret_name" { type = string }
|
||||
|
||||
module "vpa" {
|
||||
source = "./modules/vpa"
|
||||
tls_secret_name = var.tls_secret_name
|
||||
tier = local.tiers.cluster
|
||||
}
|
||||
# VPA / Goldilocks REMOVED 2026-06-12 (etcd-load-reduction; reverses the re-add
|
||||
# after memory 2431, ties to code-oflt). All 349 VPAs ran updateMode=Off (no
|
||||
# auto-right-sizing) yet cost ~800 etcd objects, continuous recommender writes,
|
||||
# and a pod-creation admission webhook — pure etcd overhead feeding only the
|
||||
# dashboard. Right-size on demand with krr (Dockerized, no cluster footprint).
|
||||
#
|
||||
# The `module "vpa"` block was removed so `scripts/tg apply` DESTROYS the helm
|
||||
# releases (vpa, goldilocks), the goldilocks-vpa-auto-mode ClusterPolicy, the
|
||||
# dashboard ingress, and the vpa namespace. The chart-installed VPA CRDs (Helm
|
||||
# keeps CRDs on uninstall) and any leftover VPA/checkpoint CRs are removed
|
||||
# post-apply (cascade) via:
|
||||
# kubectl delete crd verticalpodautoscalers.autoscaling.k8s.io \
|
||||
# verticalpodautoscalercheckpoints.autoscaling.k8s.io
|
||||
|
|
|
|||
|
|
@ -1,175 +0,0 @@
|
|||
variable "tls_secret_name" {
|
||||
type = string
|
||||
sensitive = true
|
||||
}
|
||||
variable "tier" { type = string }
|
||||
|
||||
resource "kubernetes_namespace" "vpa" {
|
||||
metadata {
|
||||
name = "vpa"
|
||||
labels = {
|
||||
tier = var.tier
|
||||
"keel.sh/enrolled" = "true"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
module "tls_secret" {
|
||||
source = "../../../../modules/kubernetes/setup_tls_secret"
|
||||
namespace = kubernetes_namespace.vpa.metadata[0].name
|
||||
tls_secret_name = var.tls_secret_name
|
||||
}
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# VPA — Vertical Pod Autoscaler (Fairwinds Helm chart)
|
||||
# -----------------------------------------------------------------------------
|
||||
resource "helm_release" "vpa" {
|
||||
namespace = kubernetes_namespace.vpa.metadata[0].name
|
||||
create_namespace = false
|
||||
name = "vpa"
|
||||
atomic = true
|
||||
|
||||
repository = "https://charts.fairwinds.com/stable"
|
||||
chart = "vpa"
|
||||
|
||||
values = [yamlencode({
|
||||
recommender = {
|
||||
enabled = true
|
||||
resources = {
|
||||
requests = {
|
||||
cpu = "50m"
|
||||
memory = "200Mi"
|
||||
}
|
||||
limits = {
|
||||
memory = "200Mi"
|
||||
}
|
||||
}
|
||||
}
|
||||
updater = {
|
||||
enabled = true
|
||||
resources = {
|
||||
requests = {
|
||||
cpu = "50m"
|
||||
memory = "200Mi"
|
||||
}
|
||||
limits = {
|
||||
memory = "200Mi"
|
||||
}
|
||||
}
|
||||
}
|
||||
admissionController = {
|
||||
enabled = true
|
||||
resources = {
|
||||
requests = {
|
||||
cpu = "50m"
|
||||
memory = "200Mi"
|
||||
}
|
||||
limits = {
|
||||
memory = "200Mi"
|
||||
}
|
||||
}
|
||||
}
|
||||
})]
|
||||
}
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Goldilocks — VPA dashboard (Fairwinds Helm chart)
|
||||
# -----------------------------------------------------------------------------
|
||||
resource "helm_release" "goldilocks" {
|
||||
namespace = kubernetes_namespace.vpa.metadata[0].name
|
||||
create_namespace = false
|
||||
name = "goldilocks"
|
||||
atomic = true
|
||||
|
||||
repository = "https://charts.fairwinds.com/stable"
|
||||
chart = "goldilocks"
|
||||
|
||||
values = [yamlencode({
|
||||
controller = {
|
||||
flags = {
|
||||
on-by-default = "true"
|
||||
}
|
||||
}
|
||||
dashboard = {
|
||||
replicaCount = 1
|
||||
flags = {
|
||||
on-by-default = "true"
|
||||
}
|
||||
}
|
||||
})]
|
||||
|
||||
depends_on = [helm_release.vpa]
|
||||
}
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Ingress — Goldilocks dashboard at goldilocks.viktorbarzin.me
|
||||
# -----------------------------------------------------------------------------
|
||||
module "ingress" {
|
||||
source = "../../../../modules/kubernetes/ingress_factory"
|
||||
dns_type = "proxied"
|
||||
namespace = kubernetes_namespace.vpa.metadata[0].name
|
||||
name = "goldilocks"
|
||||
service_name = "goldilocks-dashboard"
|
||||
port = 80
|
||||
tls_secret_name = var.tls_secret_name
|
||||
auth = "required"
|
||||
extra_annotations = {
|
||||
"gethomepage.dev/enabled" = "true"
|
||||
"gethomepage.dev/name" = "Goldilocks"
|
||||
"gethomepage.dev/description" = "Resource recommendations"
|
||||
"gethomepage.dev/icon" = "mdi-scale-balance"
|
||||
"gethomepage.dev/group" = "Core Platform"
|
||||
"gethomepage.dev/pod-selector" = ""
|
||||
}
|
||||
|
||||
depends_on = [helm_release.goldilocks]
|
||||
}
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Kyverno policy — label namespaces for VPA observe-only mode
|
||||
# -----------------------------------------------------------------------------
|
||||
# Goldilocks reads the goldilocks.fairwinds.com/vpa-update-mode label on
|
||||
# namespaces to decide the updateMode for VPA objects it creates.
|
||||
# All namespaces get "off" — Terraform is the authoritative source of truth
|
||||
# for container resources. Goldilocks provides recommendations only.
|
||||
|
||||
resource "kubernetes_manifest" "vpa_auto_mode_label" {
|
||||
manifest = {
|
||||
apiVersion = "kyverno.io/v1"
|
||||
kind = "ClusterPolicy"
|
||||
metadata = {
|
||||
name = "goldilocks-vpa-auto-mode"
|
||||
annotations = {
|
||||
"policies.kyverno.io/title" = "Goldilocks VPA Observe-Only Mode"
|
||||
"policies.kyverno.io/description" = "Sets VPA update mode to off for all namespaces. Terraform owns container resources; Goldilocks provides recommendations only."
|
||||
}
|
||||
}
|
||||
spec = {
|
||||
rules = [
|
||||
{
|
||||
name = "label-vpa-off-all"
|
||||
match = {
|
||||
any = [
|
||||
{
|
||||
resources = {
|
||||
kinds = ["Namespace"]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
mutate = {
|
||||
patchStrategicMerge = {
|
||||
metadata = {
|
||||
labels = {
|
||||
"goldilocks.fairwinds.com/vpa-update-mode" = "off"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
depends_on = [helm_release.goldilocks]
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue