viktor/infra
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

etcd-load-reduction: remove VPA/Goldilocks, disable kyverno reporting, descheduler hourly
Some checks failed
ci/woodpecker/push/build-cli Pipeline was successful
Details
ci/woodpecker/push/default Pipeline failed
Details

Browse source 
The control-plane flap (etcd lease-renewal timeouts) recurred. Rather than move
etcd to SSD (code-oflt, deferred again), the chosen direction is to REDUCE etcd
load enough that the leader-election-timeout band-aid (renew 10s->30s) becomes
removable. These are the big, clean cuts:

1. Remove VPA/Goldilocks (stacks/vpa emptied). All 349 VPAs ran updateMode=Off
   (no auto-right-sizing) yet cost ~800 etcd objects + continuous recommender
   writes + a pod-creation admission webhook, purely to feed a dashboard. krr
   (Dockerized, on-demand) replaces it. Reverses the re-add after memory 2431.

2. Disable kyverno reporting (admission/aggregate/background). policyReports were
   already off, so the pipeline generated ephemeralreports + an hourly
   all-resource etcd re-scan for NO user-facing output. Admission enforcement
   (deny-* policies) and Keel mutation are unaffected; violations surface via
   Loki->Slack.

3. descheduler */5 -> hourly (fewer list/evict cycles; rebalancing isn't urgent).

Deferred (poor ROI / unsafe as planned): ESO refreshInterval 15m->1h is a
~20-stack sprawl for ~0.1 writes/s; keel background=false is invalid for a
mutate-existing policy and its churn is apply-time not steady-state. Both filed
as follow-up beads.

Post-apply: delete the chart-orphaned VPA CRDs to cascade-clean leftover CRs.
Then measure etcd apply-latency and revert the timeouts. Docs updated
(VPA/Goldilocks -> krr). See memory 5402-5407.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-06-12 19:41:22 +00:00
parent 16adda2c48
commit 0216e993dc
5 changed files with 32 additions and 184 deletions
Download patch file Download diff file Expand all files Collapse all files

18
stacks/vpa/main.tf
View file

@ -1,7 +1,15 @@
variable "tls_secret_name" { type = string }
module "vpa" {
source = "./modules/vpa"
tls_secret_name = var.tls_secret_name
tier = local.tiers.cluster
}
# VPA / Goldilocks REMOVED 2026-06-12 (etcd-load-reduction; reverses the re-add
# after memory 2431, ties to code-oflt). All 349 VPAs ran updateMode=Off (no
# auto-right-sizing) yet cost ~800 etcd objects, continuous recommender writes,
# and a pod-creation admission webhook — pure etcd overhead feeding only the
# dashboard. Right-size on demand with krr (Dockerized, no cluster footprint).
#
# The `module "vpa"` block was removed so `scripts/tg apply` DESTROYS the helm
# releases (vpa, goldilocks), the goldilocks-vpa-auto-mode ClusterPolicy, the
# dashboard ingress, and the vpa namespace. The chart-installed VPA CRDs (Helm
# keeps CRDs on uninstall) and any leftover VPA/checkpoint CRs are removed
# post-apply (cascade) via:
# kubectl delete crd verticalpodautoscalers.autoscaling.k8s.io \
# verticalpodautoscalercheckpoints.autoscaling.k8s.io