Browsers accumulate one authentik_proxy_<random> cookie per Authentik
Proxy Provider under viktorbarzin.me (Path=/). With 30+ services the
combined Cookie header exceeds nginx's default 4 x 8k
large_client_header_buffers and trips '431 Request Header Fields Too
Large' at the forward-auth nginx (traefik/auth-proxy).
Bumped to:
client_header_buffer_size 8k
large_client_header_buffers 8 64k
Matches the pattern used on the London Flint 2 router nginx
(memory id=647).
Investigated, designed, and planned the 3-master HA control plane
migration triggered by 2026-05-21's autonomous k8s upgrade cascade.
Locked 14 design decisions across two passes:
- 10 initial decisions (LB strategy, IPs, sizing, etcd, kured gate, etc)
- 4 challenger-pass amendments (cloud-init template bump, rbac stack
multi-master refactor, HTTPS /readyz health check, expanded blast
radius to include /home/wizard/code/infra/config root kubeconfig,
config.tfvars, k8s-portal user kubeconfigs, etcd-backup nodeSelector,
k8s-version-upgrade chain extension as Phase 7)
Plan covers 11 phases end-to-end including panic-mode rollback.
DEFERRED before execution. PVE host is 98% RAM-committed
(262 GB allocated / 267 GB physical, 1.5 GB swap active); the
planned 3 x 32 GB masters would push allocation to 326 GB and OOM
the host. k8s-master currently uses only 4.6 GB of its 32 GB
allocation (5-6x oversized).
Revisit triggers documented in design doc:
1. Second PVE host added → hardware HA becomes possible.
2. Right-sizing pass OR planning masters at 16 GB each.
3. Cumulative manual upgrade nursing > ~10h.
Standalone candidate worth lifting independently: Phase 1.5's
rbac stack refactor (apiserver-oidc + audit-policy + etcd-tuning
to loop over k8s_master_hosts list) — future-proofs the cluster
without committing to the HA migration.
Refs: code-n0ow (open, deferred via bd note).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sonnet-4-5 trips Anthropic per-account rate_limit_error on the OAuth
bearer (sk-ant-oat01) tokens after 5-10 burst calls — sticky multi-hour
quota. Haiku-4-5 has much higher RPM and processes the 16-video
backfill cleanly (~30s/video with inter-call throttle).
Comment above the env line documents the rationale for future re-evaluation.
Remove leftover SendGrid references after the Brevo migration was completed:
- Delete TF `cloudflare_record.mail_domainkey` (TXT at `s1._domainkey`,
SendGrid-era DKIM, hidden behind the SendGrid CNAME but would re-emerge
once the CNAME is removed).
- Clean up commented-out `smtp.sendgrid.net` relayhost references and the
`# For sendgrid` comment on `sasl_passwd` in the mailserver module.
DNS records deleted out-of-band (not TF-managed):
- CF: `s1._domainkey CNAME` + `s2._domainkey CNAME` → sendgrid.net (manual entries)
- Technitium internal `viktorbarzin.me`: `em7107`, `s1._domainkey`,
`s2._domainkey` CNAMEs → sendgrid.net
Verified end-to-end mail flow unaffected (Brevo outbound + IMAP receive,
roundtrip 20.4s — identical to baseline). Active DKIM (`mail._domainkey`
local + `brevo1/brevo2._domainkey` Brevo) untouched.
Auth audit on 2026-05-22 — all the broken paths and the one that works:
- openai-codex OAuth: EXPIRED (ChatGPT Plus, ancaelena98@gmail.com)
- secret/openclaw → openai_api_key (sk-svcacct): insufficient_quota
- openrouter_api_key: "Key limit exceeded (total limit)"
- llama_api_key: region-blocked
- anthropic_api_key: sk-ant-oat-… (OAuth refresh token, not a real
x-api-key — won't auth via x-api-key header)
- nvidia_api_key (NIM): WORKS. The key was already baked into the
openclaw.json providers.nim.apiKey from secret/openclaw → nvidia_api_key.
Two NIM models verified end-to-end (call from inside openclaw pod
with tool-call schema, both returned proper {tool_calls:[…]} JSON):
- meta/llama-3.1-70b-instruct — 0.58s, primary
- meta/llama-4-maverick-17b-128e — 16s, smarter, fallback
Fallback chain: maverick → openai-codex (auto-promotes once re-authed)
→ modelrelay/auto-fastest (last resort, hallucinates instead of
tool-calling, but at least responds).
Models registered in both `agents.defaults.models` (allowlist) and
`models.providers.nim.models` (capability declarations) so the agent
sees them as available tools. Startup `models set` updated to pin
the new primary across `doctor --fix` runs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First analysis pass over Calico GNP wave1-egress-observe-tier34 data captured
in Loki since 2026-05-19. Pulled ~10000 flow log lines covering 36 source
namespaces (of 82 selected by tier 3+4). Analysis script outputs preserved
on the dev host at /tmp/{analyze_flows2,build_allowlist}.py.
## Findings
**Universal baseline (every observed ns):**
- DNS to kube-system/kube-dns UDP/53
- Often mysql.dbaas TCP/3306 or pg.dbaas TCP/5432
- Often redis.redis TCP/6379
**Rollout tiering by egress fan-out:**
- Tier A (recruiter-responder only): 2 destinations, ideal pilot
- Tier B (29 namespaces): ≤3 external IPs, ≤5 internal — batch rollout
- Tier C (4 namespaces: f1-stream/openclaw/woodpecker/status-page):
needs per-IP investigation
- Tier D (servarr): 130+ external IPs (BitTorrent P2P) — keep Log+Allow
permanently or move to dedicated egress proxy
## Caveats blocking immediate enforce
- Observation horizon too short: ~6h dense data, ~24h total. Need ≥7 days
to catch weekly CronJobs, Vault token rotations, Keel pulls.
- External IPs are dynamic (Cloudflare/AWS rotate). Static IP allowlists
will break — need DNS-based selectors or CIDR ranges.
- Some intra-namespace traffic bypasses the Calico filter chain.
## Recommended next steps
1. Continue observation through 2026-05-29 (full week). Compare destination
set day-over-day; if stable, allowlist is ready.
2. First enforce: recruiter-responder (allowlist = kube-dns + telegram CIDR
+ vault/ESO service IPs).
3. Tier B phased rollout at 3-5 ns/day after pilot proves out.
Full analysis: docs/architecture/wave1-egress-observation-2026-05-22.md
Tracked under beads code-8ywc.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Discovered via W1.5 enforcement when querying live cluster state:
PolicyViolation events on 5 deployments (council-complaints, ebook2audiobook,
hermes-agent, netbox, whisper/piper) trying to admit images from registries
not in the original enumeration.
Added entries:
- amruthpillai/* (resume — reactive-resume)
- athomasson2/* (ebook2audiobook)
- netboxcommunity/* (netbox)
- nousresearch/* (hermes-agent)
- opentripplanner/* (osm-routing)
- rhasspy/* (whisper, piper)
- registry.viktorbarzin.me/* (legacy private registry — council-complaints
still references; should migrate to forgejo)
The legacy registry.viktorbarzin.me was supposedly decommissioned 2026-05-07
per CLAUDE.md but council-complaints still uses it — separate cleanup task.
## Verification
- kubectl delete + reapply (kubectl_manifest resourceVersion=0 patch gotcha,
same as 2026-05-18 inject-keel-annotations)
- Dry-run admission of previously-blocked images now PASS:
- netboxcommunity/netbox:v4.5.0-beta1 ✓
- rhasspy/wyoming-whisper:3.1.0 ✓
- registry.viktorbarzin.me/council-complaints:1c56f8f ✓
- Policy still in Enforce mode
## Observation status (W1.6)
- Calico GNP wave1-egress-observe-tier34 still applied, 82 ns selected
- Loki `{job="node-journal"} |~ "calico-packet"` returns ~5000 lines/hour
- No errors from observation infrastructure
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of the recurring 'cnpg-webhook-cert' TLS expiry warn:
CNPG default 'expiringCheckThreshold = 7' means the operator only
regenerates the self-signed webhook cert when remaining lifetime drops
BELOW 7 days. Our cluster-health check #22 alerts at <30d. Result:
~23 days of WARN before CNPG would even attempt rotation.
Set EXPIRING_CHECK_THRESHOLD=30 via the chart's config.data map so the
operator now regenerates with 30d buffer, aligning with our monitoring
threshold. Cert lifetime stays at chart default 90d.
Verified after apply: operator runtime config shows
'expiringCheckThreshold:30'. Companion in-session action: deleted the
existing soon-to-expire secret and bounced the operator to force an
immediate fresh 90-day cert (notBefore=May 22, notAfter=Aug 20).
Two latent issues found while diagnosing why the May 2026 META vest
didn't land:
1. broker-sync-imap CronJob's command was 'broker-sync imap', but the
actual CLI subcommand is 'imap-ingest'. Every scheduled run had
been failing with 'No such command imap' since day-one.
2. Pod runs as uid=10001 gid=999; PVC /data dir is mode 2775
group=10001. Without fsGroup in the pod's securityContext the
pod gets only 'other' (r-x) perms on the dir, so sqlite3 can't
create journal/WAL files next to sync.db -- hits
'attempt to write a readonly database'. fsGroup=10001 adds the
matching gid to the pod's supplemental groups so writes work.
Schwab email-sender regex fix is in broker-sync@d860aef.
Operational layer for the new col_snapshot cache shipped in
fire-planner@e72fd22:
stacks/fire-planner:
- fire-planner-col-refresh CronJob — Sun 04:00 UTC, no-op until rows
age toward the 1-year TTL boundary (within 7 days). Calls
python -m fire_planner col-refresh-stale, upserts via cache.upsert.
monitoring/dashboards/cost-of-living.json (Finance folder):
- Two template variables: $city (single-select from col_snapshot),
$baseline_city (for COL ratio computation, defaults London).
- Stat row: total w/rent, w/o rent, 1-bed rent, ratio (color-coded).
- All-cities ranked table with gradient-gauged total + colored ratio.
- Cache-freshness table flags rows approaching TTL expiry.
Initial population needs a one-shot: post-Keel-rollout,
kubectl -n fire-planner exec deploy/fire-planner -- \\
python -m fire_planner col-seed
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous commit promoted modelrelay/auto-fastest to primary as a
workaround for the expired openai-codex OAuth token. But modelrelay
routes to small tool-call-shy models (nvidia/stepfun-ai/step-3.5-flash)
that hallucinate answers instead of using ssh / curl / etc. — exactly
what the v4 learning loop is supposed to leverage.
Revert primary back to openai-codex/gpt-5.4-mini (gpt-5.4-mini is the
only mini variant the Codex backend accepts for ChatGPT Plus tier),
and inline the re-auth command in the model-block comment so future
sessions know exactly what to do when the OAuth token expires:
kubectl -n openclaw exec -it $(kubectl -n openclaw get pods \
-l app=openclaw -o jsonpath='{.items[0].metadata.name}') \
-c openclaw -- node /app/openclaw.mjs models auth login \
--provider openai-codex
modelrelay/auto-fastest stays in the fallback chain so the agent
remains partially usable while the token is expired.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per Viktor: healthy baseline range is 55-65 C; anything above 65 C is a
signal a VM/workload is using too much CPU and warrants investigation.
Previous thresholds were calibrated to the hardware's TjMax (75/83 C) —
that was too lax, since cluster-load-driven elevation arrives a long
time before throttling. The 65 C cutoff matches the live Prometheus
baseline (Apr 20-May 8 2026: peak 61-69 C, avg 51-55 C) and the
session-observed correlation: above 65 C means the cluster is doing
sustained work that should be looked at, even if hardware is still
nowhere near its limit.
Updated:
PASS < 65 C (within 55-65 baseline)
WARN 65-82 C (elevated; check top kvm processes for the culprit)
FAIL >= 83 C (at/above TjMax — throttling imminent)
Verified live: 67 C now WARN (was PASS under the 75 C threshold).
User feedback: "this should work for any task, not just calendar.
this learning flow must be strongly engrained to ensure openclaw
gets better over time."
The v3 rules were buried at the bottom of TOOLS.md and only stated
in workflow language. Three changes to make the rule unavoidable:
1. **SOUL.md** — new marker-delimited section "Learning is your
identity" inserted before ## Boundaries. AGENTS.md tells the
agent to read SOUL.md first every session, so this is now the
FIRST thing the agent loads about itself. Frames learning as
character, not procedure.
2. **TOOLS.md v4** — section moved from the END of the file to
right after the `# TOOLS.md` title (first substantive content
on file load). Title strengthened: "THE FLOW — run this on
EVERY task. Not just hard ones." Concrete examples explicitly
call out diverse domains (calendar, frigate restart, disk
usage, inbox summary, deploys) so the universality is
unmistakable.
3. **learn-from-tasks skill** — opens with "This is universal.
EVERY task runs through this flow — not just hard ones, not
just unfamiliar ones. The save at the end is mandatory."
The actual flow (know → ask devvm → save) is unchanged. What
changed is salience: the rule is now the first thing the agent
encounters in three independent surfaces, with stronger framing
that makes "skipping the save" feel like a violation of identity
rather than a missed optimisation.
Marker bumped v3 → v4. Stripper handles v1-v9 idempotently.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Uncomment the trading-bot stack (disabled 2026-04-06 due to resource
consumption) and add the new meet_kevin_watcher service container.
Changes:
- Uncomment the /* ... */ block enclosing the entire stack
- Fix db_init job: add -d postgres to psql commands (root user has no
root-named database — matches pattern used in claude-memory + others)
- Remove 3 disabled containers from trading-bot-workers Pod spec:
news-fetcher, sentiment-analyzer, trade-executor
- Add new meet-kevin-watcher container (image
viktorbarzin/trading-bot-service:latest, command
python -m services.meet_kevin_watcher.main, mem 128Mi/256Mi)
- Extend ExternalSecret with TRADING_OPENROUTER_API_KEY and
TRADING_MEET_KEVIN_CHANNEL_ID keys (sourced from Vault
secret/trading-bot)
- Add 4 common_env entries for the Meet Kevin pipeline
(poll interval, daily cost cap, model slug, prompt version)
- Update lifecycle.ignore_changes to 4 image indices
vault: re-enable pg-trading static role
- Add pg-trading to vault_database_secret_backend_connection allowed_roles
- Uncomment vault_database_secret_backend_static_role.pg_trading
(was disabled 2026-04-06 with the rest of trading-bot stack)
kyverno: add postgres* to trusted-registries allowlist
- trading-bot db_init uses postgres:16-alpine (Docker Hub library image)
- postgres* was not in the DockerHub bare-name allowlist (unlike mysql*,
alpine*, nginx*, python* which were already there)
Final workers Pod containers (in order):
[0] signal-generator
[1] learning-engine
[2] market-data
[3] meet-kevin-watcher (NEW)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Refines the devvm-fallback into an explicit triage flow that the
agent runs on every task. The default path is to ASK devvm-claude
when uncertain — don't brute-force. Most tasks are solvable there.
## The flow
1. Do I KNOW how? Check `memory_recall` and INDEX.md.
2. If not, SSH devvm and ask claude — and crucially, ask it to
share the steps + credentials needed so I can do it on my own
next time. Save the answer in openclaw memory.
3. (RARE) If devvm-claude says no, try in-pod. Most likely fail —
that's OK.
## Storage moved to memory-indexed location
Learnings now live under
`/workspace/memory/projects/openclaw-learned/` (was
`/workspace/learned/`) so memory-core indexes them and
`memory_recall` surfaces them. Layout:
- `scripts/<task>.md` runnable recipes
- `knowledge/<topic>.md` decisions, paths, gotchas
- `credentials/<name>.md` **POINTERS to Vault, never values**
## Credentials = Vault pointers only
Previous v2 design saved cred values to plaintext NFS files. v3
flips to pointer-only: cred file documents the Vault path + fetch
command (`ssh devvm 'vault kv get -field=foo secret/bar'`), the
consumer, and rotation expectations. The secret stays in Vault.
## Init container also migrates
Strips v1/v2/v3 markers from TOOLS.md before re-inserting v3,
moves any files from the legacy `/workspace/learned/` tree into
the new location, removes the empty legacy dir. User edits
outside the markers always survive.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Refine the init container's devvm-fallback seeding so the OpenClaw
agent treats devvm as its DEFAULT teacher and saves recipes locally
to become independent over time:
1. TOOLS.md v2 section now has two emphatic CRITICAL rules:
- "TRY DEVVM before giving up" — when stuck, ssh devvm before
telling the user "I can't do that".
- "After every task, introspect → save a faster way" — for any
non-trivial task (especially recurring ones), save the recipe
to /workspace/learned/ and update INDEX.md.
2. New cc-skill `learn-from-tasks` at
/home/node/.openclaw/cc-skills/learn-from-tasks/SKILL.md formalises
both triggers: (A) you're stuck → check INDEX → ask devvm → save;
(B) you just finished → introspect → save if recurring.
3. /workspace/learned/ scaffold: INDEX.md table-of-contents +
scripts/, knowledge/, credentials/ (0700) subdirs. Agent checks
INDEX.md BEFORE reaching for devvm, so saved recipes are
findable on the next run.
4. Marker migration: strips both v1 and v2 markers before re-inserting
so user edits outside the markers always survive future restarts.
Security caveat documented inline: credentials in
/workspace/learned/credentials/ are NFS plaintext — acceptable for
home-lab personal scope, NOT for anything more sensitive than what
`ssh devvm` already gives the pod (wizard's access).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The OpenClaw agent reads TOOLS.md on every session per AGENTS.md
("environment-specific notes"), but it does NOT auto-search the
memory-core index for "devvm" before answering. Result: the agent
said "I don't have access to the devvm" even though ssh + the
openclaw-task wrapper were fully wired up (verified e2e in
9ad52dfd).
Updated init 6 (seed-devvm-memory-note) to ALSO append a
marker-delimited section to /workspace/TOOLS.md describing the
devvm SSH capability + openclaw-task usage. Idempotent: strips
any prior v1 section before re-inserting, so user edits outside
the markers survive future pod restarts.
The /workspace/memory/projects/openclaw-runtime/devvm-fallback.md
memory note stays in place — it's still indexed by memory-core
and surfaces for memory_recall queries.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Give the OpenClaw pod two new capabilities:
1. Host-tools bundle. New init container `install-host-tools` extracts
openssh-client + dnsutils + tmux + jq + ripgrep + fd + vault + yq +
friends into /tools/host-tools/, with the bookworm-slim libs the
binaries need. PATH + LD_LIBRARY_PATH on the main container point
ld.so at the bundle. Idempotent via /tools/host-tools/.installed-v1
marker; smoke test (ldd-based) fails the init at deploy time if any
binary has unresolved deps. Bundle is ~558 MB on the existing
/srv/nfs/openclaw/tools NFS.
2. devvm SSH + async task pattern. New init `setup-ssh-config` writes
id_rsa/config/known_hosts under /home/node/.openclaw/.ssh; main
container startup symlinks /home/node/.ssh → there. New
/usr/local/bin/openclaw-task wrapper on devvm manages long-running
work as tmux sessions on devvm (sessions and logs survive pod
restarts — they live on devvm, not in the pod). New init container
`seed-devvm-memory-note` drops a markdown note teaching the pattern;
main container startup now runs `openclaw memory index --force` so
the note is searchable on first boot.
Design + verified E2E flow in
docs/plans/2026-05-22-openclaw-devvm-access-design.md. Persistence test
green: spawned a 50s task from pod A, deleted pod A, new pod B saw the
task finish and read its full log.
Pre-existing keel.sh annotation drift on openclaw/{openlobster,
task_webhook} cleaned up in the same apply.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both new checks SSH read-only to the PVE host and emit PASS/WARN/FAIL
via the standard healthcheck output + JSON. They run alongside the
existing 42 checks and surface the same alerts the 2026-05-20/21
optimization session had to gather by hand.
#43 PVE Host Thermals — Xeon E5-2699v4 package + per-core temps
Reads every /sys/class/hwmon/hwmon0/temp*_input in one SSH round-trip.
Thresholds tuned to the live TjMax=83 / Tcrit=93:
PASS < 75 °C package
WARN 75-82 °C (approaching max, action time)
FAIL >= 83 °C (at/above TjMax, throttling imminent)
Reports hottest core label too so a single hot core doesn't hide in
the package average.
#44 PVE Host Load — load avg vs 44-thread capacity
Reads /proc/loadavg, compares 5-min to thread count (44):
PASS load_5 < 30 (< 70% threads busy)
WARN 30-37 (oversubscribed but not saturating)
FAIL >= 38 (~85%+ threads busy — scheduler saturation)
Uses 5-min so brief work spikes don't false-fail.
Both gracefully WARN-degrade if SSH BatchMode fails, matching the
existing check 36 (LVM PVC snapshots) pattern. TOTAL_CHECKS bumped
42 -> 44 and the dispatcher updated.
Two changes in one commit because they are coupled — the DISABLED_PROVIDERS
addition cannot land safely without the Keel exclusion on temporal:
1. Add DISABLED_PROVIDERS env on postiz Helm chart. Live DB audit showed
only 'instagram-standalone' connected; all other Postiz providers
were idle-polling Temporal task queues. List excludes x, linkedin,
reddit, threads, youtube, tiktok, pinterest, dribbble, slack,
discord, mastodon, bluesky, lemmy, warpcast, vk, beehiiv, telegram,
wordpress, nostr, farcaster. Keeps facebook + instagram + the
standalone variant active.
2. temporal deployment needs keel.sh/policy=never (set live via kubectl
annotate). Keel was rolling temporalio/auto-setup 1.28.1 -> 0.20.0
on every helm reconcile because :0.20.0 is published in the same
registry path but is a DIFFERENT (legacy Cassandra-based) image
stream. Memory id 1933 trap; new variant captured in id 2315-2319.
The annotation is set live (not in TF) because the existing TF block
has lifecycle.ignore_changes = [keel.sh/policy] so the chart
reconcile won't reset it. Long-term fix: add temporal to the
Kyverno keel-mutate-existing exclude list so it survives a
namespace re-label.
Captures today's k8s-upgrade-pipeline session findings — root cause
of repeated upgrade failures is the single-master apiserver outage
window cascading into operator crashloops + storm I/O. HA control
plane with 3 masters + apiserver LB removes the cascade entirely.
Tracked in beads code-n0ow. Plan doc to follow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
kubeadm's `upgrade apply` waits 5min for each static-pod manifest swap
to be picked up by the kubelet (it polls the pod's
`kubernetes.io/config.hash` annotation via apiserver). On a freshly-rebooted
master with apiserver-to-kubelet status sync lagging, that 5min isn't
enough — kubeadm declares the upgrade failed and rolls back.
The thing is: the etcd container HAS already been swapped to the new
image by then (verified live — pod is on registry.k8s.io/etcd:3.6.5-0
when this fires). kubeadm's check is just slow to notice. The 2nd
attempt sees etcd already on target, skips it, and proceeds cleanly.
Wrap `kubeadm upgrade apply` in a 3-attempt loop with 30s between.
Worker phase doesn't need this — `kubeadm upgrade node` has no
static-pod-hash waits.
Today's autonomous-pipeline session: master phase Failed at 5m on
attempt #1 with this exact error, retried, hit same timeout, gave up
(backoffLimit=1). The wrapper turns this from a fatal pipeline halt
into a "wait a bit, try again" that usually completes on attempt #2.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three changes from today's autonomous-pipeline validation session:
1. **Kill-switch ConfigMap** — chain checks for `k8s-upgrade-killswitch`
ConfigMap in `k8s-upgrade` namespace at the top of every phase + at the
start of version-check. Existence halts the chain (exit 0) with a Slack
message. Single-command emergency stop:
kubectl -n k8s-upgrade create configmap k8s-upgrade-killswitch \
--from-literal=reason="storm response"
Resume: kubectl -n k8s-upgrade delete cm k8s-upgrade-killswitch
Role rule for `configmaps` get/list/watch added (resourceName-scoped).
2. **Ignore RecentNodeReboot in halt_on_alert_query everywhere** — the
chain itself causes reboots. The pre-drain master check, post-upgrade
worker check, postflight check, and preflight halt-on-alert all now
pass `RecentNodeReboot` as the extra-ignore. Previously only worker
phase's post-upgrade gate did this. Master Failed silently this morning
on the pre-drain check after my own master reboot.
3. **Preflight quiet-baseline 3600s → 600s** — the 1h cooldown after any
Ready transition meant the chain refused to run for an hour after
every kured reboot. 10 min is enough for kubelet/control-plane to
settle; the 24h-between-cluster-reboots invariant lives in
kured-sentinel-gate, not here.
Validated by running the chain end-to-end: preflight passed in 5s,
master phase now in drain. Today's storm post-mortem (snapshot CoW
amplification + tigera-operator crashloop feedback loop) drove the
kill-switch design.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Workers handle background tasks only (LDAP sync, email, certificate
renewal) — no user-facing traffic, so 2-of-3 redundancy isn't load-
bearing. Reduces sustained CPU by ~100m.
Server replicas unchanged at 3 (PDB minAvailable=2 — user-facing).
PgBouncer pool unchanged at 3 (DB connection pooling).
Bot crawlers were hitting /<owner>/<repo>/archive/<sha>.zip on the
dot_files repo (vim-plugin source trees) — each request synthesised a
fresh ZIP from git history, taking 9.9s and returning 500 under
sustained load. Cost: ~440m sustained forgejo CPU.
Toggle: FORGEJO__repository__DISABLE_DOWNLOAD_SOURCE_ARCHIVES=true.
/archive/* URLs now 404; git clone / OCI registry / API unaffected.
Measured: forgejo pod 440-573m -> 60m steady-state (~85% drop).
(Pod rollout took ~7min on the new RS due to kubelet's recursive
chown of the 2700+ files in the data PVC — fsGroupChangePolicy is
unset and defaults to Always; could be set to OnRootMismatch later.)
Halves sample volume on all default-scrape jobs (cAdvisor, node-exporter,
service-endpoints, etc.). Memory id 559's earlier scrape-2m tuning was
applied live but not codified — this restores the Helm template.
Companion changes to keep alerting fidelity:
- evaluation_interval kept at 1m (alerts evaluate every minute)
- snmp-ups job pinned to scrape_interval=30s so PowerOutage /
LowUPSBattery detect within ~30s instead of 2m
- 3 alerts bumped from for:1m to for:3m (HighGPUTemp, LowUPSBattery,
PowerOutage) for stability above the new 2m global cadence
Other jobs that already had per-job overrides (snmp-idrac 1m,
redfish-idrac 3m, kubernetes-pods 5m, kubernetes-services 5m) unaffected.
Expected: 50-150m sustained CPU saving on Prometheus + apiserver.
Verification ongoing — apiserver settles ~minutes after Prometheus
config reload due to initial-target-scrape burst.
Replaced 'loki.source.kubernetes' with 'loki.source.file' in alloy DS
config. discovery.relabel.pod_logs already sets __path__ to the kubelet
log path (/var/log/pods/*<uid>/<container>/*.log) and varlog host-mount
was already present, so this is a one-line swap.
Why: apiserver was burning ~700m sustained on 'CONNECT pods/log' streams
(13 req/s, ~2200 sec/s of long-lived TCP connections). Streaming pod
logs through the apiserver instead of tailing kubelet's log files was
the dominant residual cost after the recent Loki/Alloy onboarding.
Measured before/after:
- Alloy DS: ~620m total (5 x ~125m) -> ~92m total (5 x ~18m)
- kube-apiserver: peak 1959m midnight burst, settled 632m
(Stuck-pod recovery: alloy-7zg7t on k8s-master needed --force delete
during rollout — FailedKillPod 'unable to signal init: permission denied'
on runc, transient runtime issue, unrelated to this change.)
Both static-roles existed in Vault state (created out-of-band) but
were missing from the postgresql connection's allowed_roles list. Vault
was logging 'is not an allowed role' rotation errors every 10s for both,
sustained CPU waste ~40-70m.
Adopted both via 'import {}' (import blocks removed after first apply
per the canonical adoption pattern).
- pg-matrix: username=matrix, rotation_period=86400 (1d)
- pg-technitium: username=technitium, rotation_period=604800 (7d)
Verified: 'is not an allowed role' errors stopped in vault-0 logs
immediately after apply.
kubectl drain --ignore-daemonsets needs to GET each pod's owner
reference (DaemonSet/StatefulSet/ReplicaSet/Deployment) to classify
which pods can be drained vs ignored. Without these RBAC verbs, drain
bails with 'cannot delete daemonsets ... is forbidden' for every
daemonset-managed pod on the node.
Provider declarations were applied across freshrss, linkwarden,
navidrome, openclaw, tandoor, vault in prior sessions; lock files
regenerated for the 4 stacks where init had run. Commits the WIP so
downstream Terraform plans can proceed.
- kubectl (gavinbunney/kubectl ~> 1.14): kubernetes_manifest panic
workaround for Kyverno CRDs (beads code-e2dp)
- authentik (goauthentik/authentik ~> 2024.10): used where stacks
manage their own Authentik objects
Plugin needs three things to load under OpenClaw 2026.5.x:
1. plugins.allow includes 'recruiter-api' (doctor --fix overwrites the
ConfigMap-baked value, so re-patch via 'openclaw config patch --stdin'
in the startup command after doctor runs).
2. 'openclaw plugins enable recruiter-api' to flip its registry entry.
3. manifest declares contracts.tools (added in recruiter-responder commit
83ffd9fa).
Plus: VIKTOR_CHAT_ID env wired from secret/openclaw.viktor_chat_id so the
plugin's polling loop knows which Telegram chat to deliver into.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
recruiter-responder ExternalSecret gains GPT_MINI_ENDPOINT/_API_KEY/_MODEL
(NIM-served qwen3-coder-480b — gpt-5.4-mini in OpenClaw is OAuth-only and
not HTTP-accessible to external services). OpenClaw gains VIKTOR_CHAT_ID
env consumed by the recruiter-api plugin's announcement loop.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Same `grep -v` / `set -o pipefail` interaction as commit 10b261d2,
in two more callsites the previous fix didn't cover:
Line 354 (phase_master): control-plane Running check —
`grep -v Running | wc -l` returns 1 when all pods are Running
(the happy path), aborting the chain right after master upgrades.
Line 419 (phase_postflight): on-target node check —
`grep -v ":v$TARGET_VERSION$" | wc -l` returns 1 when all nodes
are on the target version (the happy path, exactly when postflight
should succeed). Aborts at the moment of victory.
Forensics on yesterday's master Job failure (see commit message of
10b261d2 for context): the master Job spawned 16s after the previous
fix's TF apply, before configmap propagation completed on the kubelet.
With those two latent bugs also looming, the chain would have died
post-master-upgrade and again at postflight even if propagation had
been timely.
Wrapping each grep in `{ ... || true; }` so a no-matches result
returns success.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
halt_on_alert_query() ends with `grep -vE "$regex" | sort -u`. When
zero alerts are firing (the desired healthy state), grep matches
nothing and exits 1. Under `set -o pipefail`, the whole pipeline
returns 1; under `set -e`, the caller's `alerts=$(...)` assignment
fails and aborts the script in ~1s with no diagnostic output.
The chain effectively required at least one non-meta alert to be
firing to make any forward progress. Today (2026-05-19) the cluster
is fully clean post-MySQL recovery, the daily 12:00 UTC detection
spawned the preflight Job, and it died instantly — blocking the
1.34.7 → 1.34.8 patch chain.
Fix: wrap the grep in `{ ... || true; }` so a no-matches result
returns success. Preflight verified end-to-end after the fix — the
chain is now in flight (preflight ✓, master phase running).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
## Change
- Replaced kubectl_manifest.wave1_egress_observe_recruiter_responder with
kubectl_manifest.wave1_egress_observe_tier34
- namespaceSelector changed from `kubernetes.io/metadata.name == 'recruiter-responder'`
to `tier in {"3-edge", "4-aux"}` — covers 82 namespaces (17 tier-3-edge + 65 tier-4-aux)
- Legacy pilot GNP wave1-egress-observe-recruiter-responder kubectl-deleted
(apply_only=true means TF rename does NOT destroy the live old resource;
cleanup done manually)
- Tier 0/1/2 namespaces explicitly out of wave 1 observation per locked plan
(cluster infra + GPU workloads, deferred)
## Verification (live cluster, 2026-05-19)
- 82 namespaces match `tier in (3-edge,4-aux)`
- Felix translated the new policy into iptables LOG rule in cali-po-* chain
- LogQL `{job="node-journal"} |~ "calico-packet"` returns real packet metadata
from multiple namespaces with distinct destinations:
- east-west pod-to-pod (10.10.108.48, 10.10.122.131)
- in-cluster service VIP (10.96.0.10 — kube-dns)
- external (149.154.166.110 — Telegram API from recruiter-responder)
## W1.7 next step (calendar-bound, ~1 week)
- Let observation run for ~1 week
- Aggregate distinct destinations per namespace via LogQL
- Build per-namespace egress allowlist module `tier3_egress_baseline`
- Flip GNP rules from `[Log, Allow]` to `[Allow <specific dests>, Deny]`
- Phased per-namespace as originally planned
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the abandoned FelixConfiguration.flowLogsFileEnabled approach (Calico
Enterprise-only field, rejected by OSS v3.26) with the supported primitive:
Calico GlobalNetworkPolicy with `action: Log`.
## Mechanics (verified end-to-end on 2026-05-19)
1. kubectl_manifest applies GNP `wave1-egress-observe-recruiter-responder`
with `namespaceSelector: kubernetes.io/metadata.name == 'recruiter-responder'`,
`types: [Egress]`, `egress: [{action: Log}, {action: Allow}]`.
2. Felix translates to iptables LOG rule in
`cali-po-_ZEv_aILlvyT9fbgWN58` chain with prefix `calico-packet: ` log-level=5.
3. Linux kernel emits LOG entries to ring buffer with transport=kernel.
4. systemd-journald captures kernel transport entries.
5. Alloy DaemonSet ships journal to Loki with `job=node-journal,transport=kernel`.
6. LogQL: `{job="node-journal"} |~ "calico-packet"` returns entries showing
SRC/DST/PROTO/PORT for every NEW egress connection.
## Verified output sample
`calico-packet: IN=cali6cfdec4abc1 OUT=ens18 MAC=... SRC=10.10.122.132
DST=9.9.9.9 LEN=60 TOS=0x00 PREC=0x00 TTL=...`
The Allow rule in the GNP keeps egress functional (recruiter-responder
remained 1/1 Running through the apply — verified Python TCP connections to
1.1.1.1, 8.8.8.8, 9.9.9.9 succeed).
## Wave 1 status
W1.6 observation infra is LIVE for the recruiter-responder pilot. W1.7
remains pending: collect 1 week of `{job="node-journal"} |~ "calico-packet"`
samples, build empirical egress allowlist, flip the GNP rules from
`[Log, Allow]` to `[Allow <specific dests>, Deny]`.
Expand observation to additional namespaces by adding entries to
`spec.namespaceSelector` (e.g. `kubernetes.io/metadata.name in {recruiter-responder,X,Y}`).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Keel polls ~175 image manifests hourly against public registries.
Transient i/o timeouts and registry 5xx responses are inherent at
that scale and auto-recover on the next poll, but they were tripping
the Apps row into ⚠ attn — pure noise.
Extend benign_re to cover:
- failed to check digest + (i/o timeout | connection refused
| connection reset | context deadline exceeded | TLS handshake
timeout | no such host | EOF)
- failed to check digest + non-successful response (status=5xx)
Real actionable digest-check failures (HTTP 401 auth, 404 removed
tag) still surface. Persistent registry-side 5xx is owned by the
registry's own monitoring (forgejo-integrity-probe +
RegistryCatalogInaccessible), not by Keel logs.
Tested locally: Apps row flips from ⚠ attn → ✓ healthy after the
filter is in place; remaining errors-line drops to "(none in last
24h)".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two changes to make the 8.4.8 pin durable:
1. Add `keel.sh/policy: never` annotation on the mysql-standalone
StatefulSet. The dbaas namespace was already excluded from the
Kyverno mutate, but the StatefulSet carried orphan Keel annotations
(force/poll/match-tag) from an earlier policy version that lacked
the exclusion list. Keel kept watching :8.4.8 for digest changes.
Now explicitly opted out; Keel logged "image no longer tracked".
2. Expand the inline comment to a banner pointing at the upgrade plan
docs and the gating beads task. Anyone touching this line sees the
warning + the path to do it right.
Closes the loop on the 2026-05-18 outage. Real upgrade tracked in
code-963q + docs/plans/2026-05-19-mysql-8.4.9-upgrade-{design,plan}.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Captures the wipe+reinit strategy (sidestep the broken DD upgrade
path), the IO config bump (innodb_io_capacity 100→2000), root-cause
analysis with explicit uncertainty, verification gates, and rollback.
Not scheduled yet. Tracked in beads code-963q.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Runbook rewritten for the standalone setup (InnoDB Cluster gone since
2026-04-16) and now covers the full disaster-recovery flow we just
executed: stop pod, wipe PVC (incl. PV reclaim-policy flip from Retain
→ Delete), re-apply TF, restore via in-namespace Job, drop+create
static users with fresh Vault passwords, restart dependents.
CLAUDE.md MySQL row notes the 8.4.8 pin + links the runbook.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The mysql:8.4 floating tag let Keel auto-bump to 8.4.9, whose
data-dictionary upgrade got stuck mid-flight on every attempt
(no progress, no CPU, never completing). Pinning to 8.4.8 +
restoring from the 2026-05-18 00:30 UTC mysqldump puts us back
on a known-good binary.
Closes: code-eme8
Closes: code-k40p
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
