Gives direct (non-proxied) apps real client IPs for CrowdSec (were SNAT'd to
the node IP under ETP=Cluster) and working QUIC. Companion change (NOT in TF —
remote cloudflared tunnel config, done via CF API): tunnel ingress repointed
from https://10.0.20.200:443 to https://traefik.traefik.svc.cluster.local:443
so proxied apps are decoupled from the LB IP. pfSense 443 NAT -> traefik_lb
alias (.203). See docs/plans/2026-05-30-traefik-dedicated-ip-etp-local-*.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Attempt rolled back to .200 baseline. Root blocker: cloudflared is a
token/dashboard-managed tunnel whose ingress targets the Traefik LB IP
(10.0.20.200), so moving Traefik to .203 took down all proxied apps. Retry
must also repoint the tunnel ingress (Cloudflare API). Also documents the
vault-ingress circular dep, SIGPIPE->stuck PG state-lock gotcha, and the
ETP=Local hairpin caveat.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move Traefik off shared MetalLB IP 10.0.20.200 to a dedicated 10.0.20.203
with externalTrafficPolicy=Local, to (1) restore real client IPs for CrowdSec
on the 24 non-proxied apps (currently SNAT'd to a node IP) and (2) enable QUIC.
Forced off the shared IP because MetalLB forbids mixed ETP on a shared IP
(10.0.20.200 also carries the Terraform state DB). In-place cutover selected.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Postiz was generating hourly Slack spam and a wedged rollout, both
Keel-driven:
- Bundled redis StatefulSets run docker.io/bitnamilegacy/redis; Keel
tried 7.4.0->7.4.1/7.4.2 every poll but require-trusted-registries
denies bitnamilegacy/* (only bitnami/* allowlisted) -> endless
deny/retry/Slack-ping loop.
- Keel bumped postiz-app v2.21.7->v2.21.8 on 2026-05-26; the surge pod
couldn't schedule under the 3Gi tier-4-aux quota, wedging the rollout
for 3 days.
postiz Terraform state is heavily drifted (~2/30 resources tracked), so
per-workload opt-out can't be applied from the postiz stack. Durable
guard is here (clean kyverno state). Operational steps applied live via
kubectl (postiz stack can't apply): removed keel.sh/enrolled=true from
the namespace, set keel.sh/policy=never (annotation+label) on all 4
workloads, rolled postiz back to the running v2.21.7. Keel restarted
(scale 0->1) to drop postiz-app from its in-memory tracker; confirmed it
no longer tracks postiz.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pin immich-server to the GPU node with a time-sliced nvidia.com/gpu slice
so ffmpeg uses hardware NVENC encode + NVDEC decode instead of software.
This frees the ~3-4 CPU cores the software transcoder was burning inside
the request-serving pod (which was slowing thumbnail/photo browsing), and
makes incompatible (HEVC/iPhone) videos playable in seconds. Activation is
ffmpeg.accel=nvenc + accelDecode=true in the DB system-config (Immich app
config is DB-managed here, like oauth/smtp — not Terraform).
Also give immich-frame the same Keel ignore_changes immich-server already
has, so an untargeted apply no longer churns it (pre-existing drift).
Docs: .claude/CLAUDE.md Immich row + compute.md GPU-workloads list.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
llama-cpp was scaled to 0 during 2026-05-25 IO-storm recovery
(TEMP-SCALEDOWN). Cluster is now stable; only frigate competes for the
GPU on k8s-node1. Restoring to 1 to unblock fire-planner's Reddit
examples ingest, which needs qwen3-8b for structured extraction.
fire-planner's llama_cpp_base_url default pointed at a non-existent
service:port (llama-cpp:8000) — the real service is `llama-swap` on
port 8080. First 2026-05-28 bulk Job exited 0 with 0 rows because of
this. Correcting.
Second-tier keel drift: actualbudget, mailserver (docker-mailserver + roundcube),
servarr (8 deployments), and authentik pgbouncer are live-enrolled (Kyverno injects
keel.sh/policy=patch) and drifting, but never had the V2 block in Terraform. Added
the full block (KYVERNO_LIFECYCLE_V2 + keel.sh/match-tag + per-container
KEEL_IGNORE_IMAGE + KEEL_LIFECYCLE_V1) to all 13 deployments. The docker-mailserver
deployment had no resource-level lifecycle at all — added one.
Also fixes a pre-existing bug in modules/kubernetes/anubis_instance: the `replicas`
validation `var.replicas == null || (...)` doesn't null-short-circuit in the current
TF version, failing apply on every single-replica Anubis site (blog, cyberchef,
f1-stream, homepage, jsoncrack, kms, postiz, real-estate-crawler, travel_blog) with
"argument must not be null". Switched to a null-safe ternary.
Verified: actualbudget plan shows no image drift (http-api 26.5.2 downgrade prevented).
The anubis module change triggers a full platform apply.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirrors the kms-website pattern: deployment image now points to
forgejo.viktorbarzin.me/viktor/tuya_bridge:${var.image_tag} and the
new Woodpecker pipeline in tuya_bridge/.woodpecker.yml drives the
rollout via `kubectl set image` on every push.
Changes:
- Extract `tls_secret_name` and add `image_tag` (default "latest")
to a new variables.tf, matching the kms / fire-planner /
payslip-ingest convention.
- Add `image_pull_secrets { name = "registry-credentials" }` (Kyverno
ClusterPolicy sync-registry-credentials already syncs the Secret
into every namespace).
- Set explicit `image_pull_policy = "IfNotPresent"` — SHA-tagged
images are immutable, no need to re-pull on every restart.
The image attribute remains in `lifecycle.ignore_changes` (line was
already there from the prior Keel-managed era), so future `tg apply`s
do not fight Woodpecker's `kubectl set image`. Keel is still enrolled
on the namespace but will skip SHA-tagged images under `policy: patch`
(non-semver), so the CI pipeline is the sole rollout mechanism.
Backstory: the 2026-05-26 cluster-health incident was tuya-bridge
crashlooping after Keel rewrote `:latest` to a stale broken `:0.1`
tag on Docker Hub (which predated the `prometheus_exporter.py`
addition). Manual rebuild + push was the immediate fix; this commit
plus tuya_bridge/.woodpecker.yml close the underlying gap so a
source change reliably produces a fresh registry image.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Yesterday's session SQL-patched monitor 313 to `https://192.168.1.127:8006/`
+ ignore_tls=1 because the prior URL `http://proxmox.reverse-proxy.svc.cluster.local:8006`
hit a CoreDNS pod-level cache returning stale `10.0.10.1` (pfSense GW)
intermittently, false-tripping ExternalAccessDivergence. A kuma DB
restore would have lost the SQL fix. Declare the monitor in
`internal_monitors` so the existing sync CronJob self-heals it.
Extends the schema with optional `url` / `accepted_statuscodes` /
`ignore_tls` fields (null on the existing DB/port entries) and
teaches the sync script the MonitorType.HTTP branch — url +
accepted_statuscodes + ignoreTls (camelCase on the API), matching
drift fields the same way PORT does for hostname/port.
Verified: manually triggered the sync after apply; it found monitor
313 by name and reported "already in desired state".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the enrolled-workload sweep from cdb7d9a8. fire-planner was held
back because a parallel session was mid-apply on it (presence board); that
claim has since cleared.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Every Keel-enrolled workload (policy=patch, match-tag=true, injected by the
inject-keel-annotations Kyverno policy) was fighting Terraform: Keel rewrites
the image tag and restamps keel.sh/update-time, change-cause and the rollout
revision on each poll; without ignore_changes every `tg apply` reverted those
— downgrading the image and forcing a spurious rollout that Keel then re-did.
Only llama-cpp had the full block (added 2026-05-24); the other ~73 workloads
drifted. This sweep adds, to every enrolled deployment/daemonset lifecycle:
- container[N].image (one per container index + init_container[N]) # KEEL_IGNORE_IMAGE
- keel.sh/match-tag, keel.sh/update-time, kubernetes.io/change-cause,
deployment.kubernetes.io/revision # KEEL_LIFECYCLE_V1
Verified via `tg plan` on speedtest (single-container: image downgrade
0.24.3->0.24.1 + annotation strip now gone) and changedetection (multi-container:
both container images no longer drift). AGENTS.md drift-suppression section
updated with the canonical block + marker legend.
fire-planner deferred (parallel session mid-apply per presence board).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two correctness fixes to the wealth dashboard, found while validating
contribution data against actual-viktor (source of truth):
1. dav_corrected (Fix 1): LOCF gap-fill scoped to the Fidelity pension.
A PlanViewer scrape gap left total_value=0 for 13 days from 2026-02-16,
which cratered net worth and produced a phantom -£97,457 "contribution"
in Feb then +£100,458 in Mar. Carry the last non-zero day forward across
the gap (a £0 pension valuation is always a scrape gap, never real).
2. wealth.json (Fix 3): "Monthly contributions vs market gain" and "Annual
change decomposition" now use consecutive period-end deltas instead of
within-period first-to-last-obs, so contributions landing near a period
boundary are no longer dropped/mis-attributed.
Verified live: Feb-2026 monthly contribution now +£34,000 (real Trading212
RSU-proceeds investment, reconciles with actual-viktor), no spurious
negatives. Brokerage contributions unchanged (already correct).
Applied via scripts/tg (wealthfolio + targeted monitoring ConfigMap).
[ci skip]
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the K8s plumbing for the Reddit FIRE-examples ingest path:
- ExternalSecret fire-planner-examples-reddit (Reddit OAuth from
Vault secret/viktor.trading_bot_reddit_{client_id,client_secret}).
- ExternalSecret fire-planner-examples-claude (claude-agent-service
bearer from Vault secret/claude-agent-service.api_bearer_token).
- kubernetes_job_v1.examples_bulk_ingest — one-shot bulk Job toggled
via var.run_examples_bulk_ingest (default false). Timestamp-named so
each (true) transition creates a fresh Job; lifecycle ignores the
name so re-plans don't propose phantom renames.
- kubernetes_cron_job_v1.examples_weekly_delta — Sunday 04:00 UTC
--top=week --limit=200 incremental run.
Both runners share the env_from plumbing of the existing recompute
CronJob (fire-planner-secrets, fire-planner-db-creds,
wealthfolio-sync-db-creds) plus examples-specific vars
(REDDIT_USER_AGENT, LLAMA_CPP_BASE_URL, CLAUDE_AGENT_SERVICE_URL,
plus the three secret-backed env vars).
Plan-only this commit — actual apply lands in Task 17 after the
ingest image build.
Discovers all *.viktorbarzin.me ingress hosts every 15 minutes and
creates matching CNAME records in Technitium if missing. Prevents
the desync where Cloudflare has the DNS record (via ingress_factory)
but internal DNS returns NXDOMAIN because Technitium was never updated.
Includes ServiceAccount + ClusterRole for ingress list permissions.
Forward 30y net-worth projection on the existing wealth Grafana
dashboard: multi-scenario lines (low/base/high + derived historical
CAGR), pure-SQL over wealth-pg reusing the dashboard's Modified-Dietz
and complete-days patterns, with/without-contributions at base rate,
in a collapsed row that sidesteps Grafana's shared-time-range limit.
[ci skip]
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three new panels comparing employment income to investment returns over
time, via Grafana's -- Mixed -- datasource (salary lives in payslip_ingest,
portfolio in wealthfolio_sync — separate DBs, so per-target datasources):
- cumulative net take-home pay vs cumulative market gain (line race)
- net pay vs market gain per year (grouped bars)
- net pay vs market gain per month (grouped bars)
Inserted after the "Growth over time" panel; existing panels shifted down,
full-width tables remain at the bottom.
User no longer actively using foolery. Removed:
- TF stack stacks/foolery (Cloudflare DNS, Traefik IngressRoute,
Authentik forward-auth integration, K8s Service+Endpoints)
- Devvm systemd unit /etc/systemd/system/foolery.service
- Runtime at ~/.local/share/foolery and launcher ~/.local/bin/foolery
- Stale foolery reference in .claude/CLAUDE.md auth="required" examples
Uptime Kuma [External] foolery monitor will auto-prune on next
external-monitor-sync reconcile.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
curl|bash clients can't solve PoW, so /net-diag.sh has to bypass Anubis.
Adds a second ingress_factory pointing /net-diag.sh at the bare blog
service (port 80), keeping every other path on the existing Anubis
chain. Path-prefix specificity wins in Traefik routing — / stays gated.
dns_type = "none" because the apex viktorbarzin.me CF record already
exists from the main ingress.
Doc update: CLAUDE.md Anubis section notes blog now follows the
wrongmove carve-out pattern.
The catchall-error-pages IngressRoute matches HostRegexp(^(.+\.)?
viktorbarzin\.me$) at priority=1 — it's the wildcard handler that
returns 404 for any unmatched hostname (typos + scanner traffic).
By design its 4xx rate sits at ~100%, so HighService4xxRate was a
permanent false positive for traefik-catchall-error-pages-*@kubernetescrd.
Same exclusion pattern as nextcloud/grafana/linkwarden/claude-memory
(services with legitimately high 4xx counts).
Each check function only reads cluster state and mutates in-memory
counters; that makes it safe to isolate each one in a subshell, write
stdout to a per-check temp file, and replay outputs in original order
after all jobs finish. Counters/JSON_RESULTS replicated through marker
lines (###HCK###PASS:N etc.) so the aggregate state matches the serial
run exactly.
Pre-fetch the HA Sofia cache once in the parent so the four HA checks
share a single API round-trip instead of each subshell re-fetching.
Auto-fix mode forces --serial so mutation order stays deterministic.
New flags: --parallel N (default 12, env HEALTHCHECK_PARALLEL_JOBS),
--serial. Diminishing returns past ~12 workers.
Benchmark (--quiet, 44 checks): 53s serial -> 18s parallel-12.
Keel bumped library/nextcloud :32.0.3-apache → :32.0.9-apache on
2026-05-26 19:42 UTC. The new image needs `occ upgrade` to migrate
the DB schema, which Keel does not run, so Nextcloud landed in
maintenance mode (needsDbUpgrade=true) and stayed there for ~22h —
external probes saw 503, ExternalAccessDivergence kept firing.
Disable Keel for this workload:
- Drop the `keel.sh/enrolled=true` label from the namespace so
Kyverno's `inject-keel-annotations` policy no longer matches.
- Layer `keel.sh/policy=never` label + annotation onto the
Helm-managed Deployment via `kubernetes_labels` /
`kubernetes_annotations` (the chart at 8.8.1 doesn't expose
Deployment-level commonLabels/commonAnnotations). Keel reads the
annotation; the label is defense-in-depth for the Kyverno
exclude rule should the namespace ever get re-enrolled.
Verified: Keel logged `image no longer tracked, removing watcher`
within seconds of the annotation landing, and `tg plan` is clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PVE API endpoint regularly takes ~11s with ~1035 thin LVs on the host
(1002 k8s-csi PVCs + 22 VMs + 11 system), blowing past Prometheus's
default 10s scrape_timeout and flapping ProxmoxMetricsMissing +
ScrapeTargetDown. Switch the Service annotation from prometheus.io/scrape
to prometheus.io/scrape_slow so the scrape moves to the existing
kubernetes-service-endpoints-slow job (5m interval, 30s timeout).
As of fire-planner@4da58fe the account_snapshot cache is refreshed
lazily on each /networth, /networth/history, /progress request when
older than NETWORTH_CACHE_TTL_DAYS (default 1). The recompute CronJob
runs Monte Carlo only — no longer assumed to coordinate with the
wealthfolio-sync schedule.
[ci skip]
E2E test (manual one-shot of all 3 broker-sync CronJobs) confirmed
idempotent behaviour with zero new activities and net worth unchanged.
The IE-via-IMAP path is now default-skipped inside
broker_sync.providers.imap (commit 0d23487), so unsuspending the cron is
safe — Schwab vests get parsed, IE messages get ie_skipped at the parser
level regardless of which entry point triggers the run.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
39 IMAP-source InvestEngine BUYs + their cash-flow DEPOSITs were
re-inserted into Wealthfolio at 2026-05-27T09:22:18 UTC — exactly the
rows the £252k dedup removed yesterday. The broker-sync-imap cron at
02:30 UTC today correctly logged `ie_skipped=53`, so the IMAP cron itself
isn't the immediate culprit, but the rows DO carry broker-sync's IMAP-path
signature (`[rfc2822-v1]` notes + `sync:imap:invest-engine:...` cash-flow
markers).
Suspending kills one possible vector while a researcher subagent
investigates the root cause. Schwab vest ingestion is the only function
lost; can be unsuspended once the IE re-dup source is identified.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Without supplementary GID 10001, the broker user (uid=10001 gid=999)
cannot write sqlite3 journal files next to /data/sync.db. The cron
hits a "readonly database" error in dedup.record() AFTER successfully
importing fills to Wealthfolio — so data lands but the dedup store
never updates, leaving every subsequent run to re-fetch the same
window and exit 1 again. Same fix that's already on imap + ibkr crons.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Net-contribution chart was showing huge negative monthly swings
because broker-sync emits a synthetic cash-flow-match DEPOSIT for every
vest BUY and a WITHDRAWAL for every sell-to-cover SELL. Cumulatively
WITHDRAWALs ($1.06M) exceed DEPOSITs ($498k) — the user perceives this
as having "withdrawn" money even though they never moved cash out of
Schwab. The proceeds left for the bank and surface as real DEPOSITs on
the next account (IE/T212) that the user transfers them to.
Extend the dav_corrected view to subtract Schwab cash-flow-match flows
(DEPOSIT-positive, WITHDRAWAL-negative, account-scoped) in addition to
the existing Fidelity unrealised-gains-offset correction. InvestEngine
and Trading212 cash-flow-match entries are REAL deposits and must be
preserved — scope by Schwab account_id only.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After re-enabling Keel with `policy: patch` (commit f325b949), 3 of the
60 first-hour bumps broke things and need explicit cluster-wide opt-out
so future Kyverno reconciles can't put them back under auto-update:
- `dbaas/mysql-standalone`: patch-bumped `mysql:8.4.8 → :8.4.9` and the
DD upgrade stalled (we explicitly track that as beads `code-963q` —
the 8.4.9 jump needs a wipe+reinit, not a rolling upgrade). The
StatefulSet already had `annotation=never` from TF but was missing the
LABEL — Kyverno's selector exclude reads the LABEL, so a reconcile
that dropped the annotation could resume auto-update. Added the LABEL.
- `redis/redis-v2`: patch-bumped `redis:8-alpine → :8.0.6-alpine` and
the new image rejected the `aof-load-corrupt-tail-max-size` directive
from commit 1eee56d0 → redis-v2-2 CrashLoopBackOff. Plus :8.0.6 is
semantically older than :8-alpine (which resolves to :8.6.2) — same
Keel tag-picking pathology as the 2026-05-26 morning incident, just
in a different shape. LABEL + ANNOTATION both added.
- `nvidia/nvidia-exporter`: Keel rewrote `:latest → :4.5.2-4.8.1-ubuntu22.04`
and the new dcgm-exporter OOMKilled at the 192Mi memory limit
(4 restarts before I caught it). Added LABEL + ANNOTATION for opt-out,
AND bumped memory request/limit 192Mi → 256Mi/512Mi so the bumped image
doesn't OOM (older versions fit in 192Mi; the bumped one needs ~250Mi
steady-state).
The 56 other Keel bumps in that 10-minute window (coredns 1.12.1→1.12.4,
kyverno 1.16.1→1.16.4, nextcloud 32.0.3→32.0.9, grafana 12.3.1→12.3.6,
cnpg, mailserver, csi-nfs, metrics-server, etc.) landed cleanly — the
`patch` policy is the right default. Per-workload `never` opt-out is
the maintenance cost.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The CronJob ran every 5 min on a vanilla python:3.12-alpine image, doing
`apk add git` + `pip install uptime-kuma-api` from scratch on every
invocation. Caught at ~3.2 MB/s on k8s-node4's root LV, contributing to
~8 MB/s sustained on the pve-data thin pool (sdc) — ~804 GB written
over the prior 18 h.
Commented out the kubernetes_cron_job_v1.status_page_pusher resource
(kept ns / SA / RBAC / ConfigMap intact for trivial revert). Re-enable
once a custom image with git + uptime-kuma-api baked in is published so
no per-run cold install happens.
status.viktorbarzin.me stops updating until then.
Sets BROKER_SYNC_IMAP_EXCLUDE_PROVIDERS=invest-engine on broker-sync-imap,
so the IMAP path no longer parses InvestEngine emails (handled by the
bearer-token API path now). Stops duplicate BUYs in Wealthfolio.
The terraform fmt run also realigned two adjacent label assignments.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add new "Data Routing" flowchart up front showing which paths go
where (sda mirror vs Synology-direct vs not-backed-up).
- Overall Backup Flow: split Layer 2 into 2a (nfs-mirror daily 02:00)
and 2b (daily-backup 05:00); show nfs-mirror as an explicit
component; clarify Step 2 is immich-only direct + nfs-ssd.
- Weekly Backup Timeline → Daily Backup Timeline: actual schedule
(00:00 LVM, 00:15 PG, 00:45 MySQL, 02:00 nfs-mirror, 05:00 daily-
backup, 06:00 offsite-sync, 12:00 second LVM); explicit inotify
feeding Step 2.
- Physical Disk Layout: current capacity numbers + dual sdc→sda and
sdc→Synology arrows (immich-only) reflecting the two-leg design.
- Restore Decision Tree: refreshed age tiers (< 12h LVM, 12h-4w sda,
> 4w Synology) + dedicated branch for immich photos (which only
have an offsite copy).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirrors the SSD section's pattern. If the LAST iteration of the
`while IFS= read -r f; do [ -f "$f" ] && echo "${f#/srv/nfs/}"; done`
body sees a file that was deleted between inotify capture and now
(e.g. an immich encoded-video temp file that got cleaned up), the
while loop returns 1, pipefail propagates, set -e kills the script
silently before reaching the rsync. No log line, just disappears.
Pre-existing bug; only exposed today after pruning the bypass regex
to immich-only — when the regex was broader, the last match in the
sorted dedup'd inotify log happened to be a live file often enough
that the bug stayed dormant. Validated by full e2e run:
1120 nfs/immich files + 2285 nfs-ssd files shipped successfully.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
