etcd-load-reduction: remove VPA/Goldilocks, disable kyverno reporting, descheduler hourly
The control-plane flap (etcd lease-renewal timeouts) recurred. Rather than move etcd to SSD (code-oflt, deferred again), the chosen direction is to REDUCE etcd load enough that the leader-election-timeout band-aid (renew 10s->30s) becomes removable. These are the big, clean cuts: 1. Remove VPA/Goldilocks (stacks/vpa emptied). All 349 VPAs ran updateMode=Off (no auto-right-sizing) yet cost ~800 etcd objects + continuous recommender writes + a pod-creation admission webhook, purely to feed a dashboard. krr (Dockerized, on-demand) replaces it. Reverses the re-add after memory 2431. 2. Disable kyverno reporting (admission/aggregate/background). policyReports were already off, so the pipeline generated ephemeralreports + an hourly all-resource etcd re-scan for NO user-facing output. Admission enforcement (deny-* policies) and Keel mutation are unaffected; violations surface via Loki->Slack. 3. descheduler */5 -> hourly (fewer list/evict cycles; rebalancing isn't urgent). Deferred (poor ROI / unsafe as planned): ESO refreshInterval 15m->1h is a ~20-stack sprawl for ~0.1 writes/s; keel background=false is invalid for a mutate-existing policy and its churn is apply-time not steady-state. Both filed as follow-up beads. Post-apply: delete the chart-orphaned VPA CRDs to cascade-clean leftover CRs. Then measure etcd apply-latency and revert the timeouts. Docs updated (VPA/Goldilocks -> krr). See memory 5402-5407. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
16adda2c48
commit
0216e993dc
5 changed files with 32 additions and 184 deletions
|
|
@ -30,9 +30,24 @@ resource "helm_release" "kyverno" {
|
|||
forceFailurePolicyIgnore = {
|
||||
enabled = true
|
||||
}
|
||||
# Reporting fully disabled (2026-06-12, etcd-load-reduction). policyReports
|
||||
# were already off, so admission/aggregate/background reporting generated
|
||||
# ephemeralreports + an hourly all-resource etcd re-scan for NO user-facing
|
||||
# output. Admission enforcement (deny-* policies) and Keel mutation are
|
||||
# independent of reporting; policy violations surface via Loki->Slack. This
|
||||
# removes a steady-state etcd write/scan load (control-plane flap mitigation).
|
||||
policyReports = {
|
||||
enabled = false
|
||||
}
|
||||
admissionReports = {
|
||||
enabled = false
|
||||
}
|
||||
aggregateReports = {
|
||||
enabled = false
|
||||
}
|
||||
backgroundScan = {
|
||||
enabled = false
|
||||
}
|
||||
}
|
||||
|
||||
reportsController = {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue