Commit graph

1696 commits

Author SHA1 Message Date
Viktor Barzin
6478097e2d fix platform stack: k8s_users.domains and sensitive for_each errors [ci skip]
- Use lookup(user, "domains", []) for missing domains attribute
- Wrap user_domains in nonsensitive() for Cloudflare for_each
2026-03-15 23:36:46 +00:00
Viktor Barzin
b87ba5e778 update claude knowledge: secret/viktor is go-to for all personal secrets [ci skip] 2026-03-15 23:21:52 +00:00
Viktor Barzin
a9890a1f27 trigger CI: json webhook 2026-03-15 23:17:58 +00:00
Viktor Barzin
0c6681bc76 fix woodpecker sync: single $ in heredoc, alpine image for jq, port 80 not 8000 2026-03-15 23:12:52 +00:00
Viktor Barzin
a04335d0f3 right-size 14 services and scale down GPU-heavy workloads [ci skip]
Memory right-sizing based on VPA upperBound analysis:
- Increases: stirling-pdf 1200→1536Mi, claude-memory 64→128Mi,
  dawarich 512→768Mi, kyverno-cleanup 128→192Mi, linkwarden 768→1Gi,
  navidrome 64→128Mi, listenarr 768→896Mi, privatebin 64→128Mi,
  ntfy 64→128Mi, health 128→256Mi, dbaas quota 16→20Gi,
  mysql-operator 384→512Mi
- Decreases: rybbit 768→384Mi, nvidia-exporter added explicit 192Mi,
  dcgm-exporter 2560→1536Mi
- Scale to 0: ebook2audiobook/audiblez-web, whisper (GPU node pressure)

Net effect: -496Mi cluster-wide, 13 ContainerNearOOM alerts resolved,
all ResourceQuota pressures cleared, GPU health green.
2026-03-15 23:00:49 +00:00
Viktor Barzin
b6d619e5df fix: increase terragrunt-apply step memory to 2Gi
LimitRange defaults containers to 192Mi which is insufficient for
terragrunt apply on the platform stack (48 vault refs, many modules).
Set explicit 1Gi request / 2Gi limit via backend_options.
2026-03-15 22:59:34 +00:00
Viktor Barzin
0c1239030d fix: CI pipeline - disable corrupted cache, add pull before push
- build-cli.yml: comment out cache_from/cache_to to avoid BuildKit
  "short read" errors from corrupted registry cache
- default.yml: add git pull --rebase before push in cleanup-and-push
  to handle remote having newer commits
2026-03-15 22:51:08 +00:00
Viktor Barzin
c8069f53c8 update claude knowledge: final ESO migration state [ci skip] 2026-03-15 22:32:46 +00:00
Viktor Barzin
6c8a42b4e3 add add-user skill for cluster onboarding
Interactive skill that collects user info, updates Vault KV k8s_users,
and applies vault/platform/woodpecker stacks. Includes verification
checklist and auto-generated resource table.
2026-03-15 22:28:54 +00:00
Viktor Barzin
82b1f82a2c add user onboarding and admin instructions to README
Admin section: how to add a new namespace-owner (Authentik group,
Vault KV entry, three terragrunt applies). Includes auto-generated
resource table.

User section: VPN setup, tool install, Vault/kubectl auth, first app
deployment from template, CI/CD pipeline example, useful commands,
and important rules.
2026-03-15 22:25:43 +00:00
Viktor Barzin
cc55249524 fix ollama: remove conditional count on basicAuth (incompatible with ESO data source) 2026-03-15 22:24:36 +00:00
Viktor Barzin
50620e6047 add generic multi-user cluster onboarding system
Data-driven user onboarding: add a JSON entry to Vault KV k8s_users,
apply vault + platform + woodpecker stacks, and everything is auto-generated.

Vault stack: namespace creation, per-user Vault policies with secret isolation
via identity entities/aliases, K8s deployer roles, CI policy update.

Platform stack: domains field in k8s_users type, TLS secrets per user namespace,
user domains merged into Cloudflare DNS, user-roles ConfigMap mounted in portal.

Woodpecker stack: admin list auto-generated from k8s_users, WOODPECKER_OPEN=true.

K8s-portal: dual-track onboarding (general/namespace-owner), namespace-owner
dashboard with Vault/kubectl commands, setup script adds Vault+Terraform+Terragrunt,
contributing page with CI pipeline template, versioned image tags in CI pipeline.

New: stacks/_template/ with copyable stack template for namespace-owners.
2026-03-15 22:23:36 +00:00
Viktor Barzin
39b3c51709 migrate 16 plan-time stacks: vault data source → ESO + kubernetes_secret
Replaced data "vault_kv_secret_v2" with:
1. ExternalSecret (ESO syncs Vault KV → K8s Secret)
2. data "kubernetes_secret" (reads ESO-created secret at plan time)

This removes the Vault provider dependency at plan time for these
stacks — they now only need K8s API access, not a Vault token.

Stacks: actualbudget, affine, audiobookshelf, calibre, changedetection,
coturn, freedify, freshrss, grampsweb, navidrome, novelapp, ollama,
owntracks, real-estate-crawler, servarr, ytdlp
2026-03-15 22:06:39 +00:00
Viktor Barzin
af3b1b5c90 fix health DB ExternalSecret: use pg-health not postgresql-health role name 2026-03-15 21:53:02 +00:00
root
a186d26ba7 Woodpecker CI deploy commit [CI SKIP] 2026-03-15 21:43:40 +00:00
Viktor Barzin
745e43c983 fix DB password desync + migrate remaining tfvars to Vault
DB desync fix: Stacks with Vault DB engine rotation (24h) now read
the password from vault-database ClusterSecretStore instead of vault-kv.
9 stacks updated with db ExternalSecrets reading from static-creds/*.

Stacks fixed: speedtest, hackmd, health, trading-bot, claude-memory,
woodpecker, linkwarden, nextcloud, url.

terraform.tfvars migration:
- plotting-book: google_client_id/secret → Vault KV + secret_key_ref
- tandoor: email_password var removed (was default="", now optional ESO)
- infra: ssh_private_key, vm_wizard_password, dockerhub_registry_password
  → Vault KV at secret/infra + data source
2026-03-15 21:39:45 +00:00
Viktor Barzin
06a0d0599a regenerate providers.tf: remove vault_root_token variable [ci skip] 2026-03-15 21:21:01 +00:00
Viktor Barzin
2c8f3a607e fix realestate-crawler: access nested notification_settings correctly
Vault KV stores notification_settings as nested JSON ({"slack":{"webhook_url":""}}).
TF code was passing the map object directly as a string env var value.
Fix: access ["slack"]["webhook_url"] with try() fallback.
2026-03-15 21:01:24 +00:00
Viktor Barzin
23dfaa1ac8 update claude knowledge: vault-native secrets migration decisions [ci skip] 2026-03-15 21:00:07 +00:00
Viktor Barzin
1ffce19611 fix woodpecker sync script: escape $ and %{} for HCL heredoc
HCL heredocs always interpolate — use $$ for literal $ and
%%{} for literal %{}. Fixes terraform plan errors.
2026-03-15 19:37:00 +00:00
Viktor Barzin
0f262ceda3 add pod dependency management via Kyverno init container injection
Kyverno ClusterPolicy reads dependency.kyverno.io/wait-for annotation
and injects busybox init containers that block until each dependency
is reachable (nc -z). Annotations added to 18 stacks (24 deployments).

Includes graceful-db-maintenance.sh script for planned DB maintenance
(scales dependents to 0, saves replica counts, restores on startup).
2026-03-15 19:17:57 +00:00
root
dc274ab413 Woodpecker CI deploy commit [CI SKIP] 2026-03-15 19:16:23 +00:00
Viktor Barzin
f0312df2be fix gpu-workload Kyverno policy: use replace with explicit priority value
The API server doesn't re-resolve priority from PriorityClassName after
webhook mutation. Changed from remove+add to replace with explicit
priority=1200000 and preemptionPolicy=PreemptLowerPriority.
2026-03-15 19:11:44 +00:00
Viktor Barzin
29c98aaea8 add Vault→Woodpecker secret sync CronJob (Part E)
Syncs secrets from Vault KV at secret/ci/global to Woodpecker
global secrets via REST API every 6 hours. Authenticates via K8s
SA JWT (woodpecker-sync role). New repos just add secrets to
Vault and use from_secret: in pipeline files.

Also removes k8s-dashboard static admin token — use
vault write kubernetes/creds/dashboard-admin instead.
2026-03-15 19:06:12 +00:00
Viktor Barzin
1acf8cc4e8 migrate consuming stacks to ESO + remove k8s-dashboard static token
Phase 9: ExternalSecret migration across 26 stacks:

Fully migrated (vault data source removed, ESO delivers secrets):
- speedtest, shadowsocks, wealthfolio, plotting-book, f1-stream, tandoor
- n8n, dawarich, diun, netbox, onlyoffice, tuya-bridge
- hackmd (ESO template for DB URL), health (ESO template for DB URL)
- trading-bot (ESO template for DATABASE_URL + 7 secret env vars)
- forgejo (removed unused vault data source)

Partially migrated (vault kept for plan-time, ESO added for runtime):
- immich, linkwarden, nextcloud, paperless-ngx (jsondecode for homepage)
- claude-memory, rybbit, url, webhook_handler (plan-time in locals/jobs)
- woodpecker, openclaw, resume (plan-time in helm values/jobs/modules)

17 stacks unchanged (all plan-time: homepage annotations, configmaps,
module inputs) — vault data source works with OIDC auth.

Phase 17a: Remove k8s-dashboard static admin token secret.
Users now get tokens via: vault write kubernetes/creds/dashboard-admin
2026-03-15 19:05:04 +00:00
Viktor Barzin
cfc30b62e8 enhance devops-engineer agent: deploy + monitor pod health [ci skip]
- Upgrade model from sonnet to opus for subagent orchestration
- Add Write, Edit, Agent tools for spawning monitor subagents
- Add mandatory deployment workflow: pre-deploy snapshot, apply,
  spawn background haiku pod monitor, react to results
- Monitor detects CrashLoopBackOff, OOM, ImagePullBackOff, stuck
  Pending, and probe failures within 3 min timeout
- Allow terragrunt apply and kubectl set image as safe operations
2026-03-15 18:44:20 +00:00
Viktor Barzin
a2720f6a4c claude-memory: read DB password from Vault KV instead of tfvars
Vault DB engine rotates the password every 24h, so the static tfvars
value was stale. Now reads from secret/claude-memory db_password key.
2026-03-15 18:22:29 +00:00
Viktor Barzin
90b7d6ebb5 etcd defrag cronjob: add --command-timeout=60s
Default 5s timeout causes defrag to fail on fragmented DBs.
Discovered during manual defrag that took ~7s.
2026-03-15 17:24:24 +00:00
Viktor Barzin
c034adab5f mitigate cluster instability during terraform applies
- Recreate strategy for heavy single-replica deployments (onlyoffice, stirling-pdf)
- Reduce maxSurge on multi-replica deployments (traefik, authentik, grafana, kyverno)
  to prevent memory request surge overwhelming scheduler
- Weekly etcd defrag CronJob (Sunday 3 AM) to prevent fragmentation buildup
- Disable Kyverno policy reports (ephemeral report cleanup)
- Cloud-init: journald persistence + 4Gi swap for worker nodes
- Kubelet: LimitedSwap behavior for memory pressure relief
2026-03-15 17:23:39 +00:00
Viktor Barzin
1fe7798609 fix openclaw init container: escape shell vars, fix image path [ci skip]
- Use $$ for shell variable escaping in Terraform ($ is Terraform interpolation)
- Fix image: docker.io/alpine/git (not library/alpine/git)
- Inline command instead of heredoc to avoid Terraform interpolation issues
2026-03-15 17:19:03 +00:00
Viktor Barzin
3aba29e7a3 remove SOPS pipeline, deploy ESO + Vault DB/K8s engines
Vault is now the sole source of truth for secrets. SOPS pipeline
removed entirely — auth via `vault login -method=oidc`.

Part A: SOPS removal
- vault/main.tf: delete 990 lines (93 vars + 43 KV write resources),
  add self-read data source for OIDC creds from secret/vault
- terragrunt.hcl: remove SOPS var loading, vault_root_token, check_secrets hook
- scripts/tg: remove SOPS decryption, keep -auto-approve logic
- .woodpecker/default.yml: replace SOPS with Vault K8s auth via curl
- Delete secrets.sops.json, .sops.yaml

Part B: External Secrets Operator
- New stack stacks/external-secrets/ with Helm chart + 2 ClusterSecretStores
  (vault-kv for KV v2, vault-database for DB engine)

Part C: Database secrets engine (in vault/main.tf)
- MySQL + PostgreSQL connections with static role rotation (24h)
- 6 MySQL roles (speedtest, wrongmove, codimd, nextcloud, shlink, grafana)
- 6 PostgreSQL roles (trading, health, linkwarden, affine, woodpecker, claude_memory)

Part D: Kubernetes secrets engine (in vault/main.tf)
- RBAC for Vault SA to manage K8s tokens
- Roles: dashboard-admin, ci-deployer, openclaw, local-admin
- New scripts/vault-kubeconfig helper for dynamic kubeconfig

K8s auth method with scoped policies for CI, ESO, OpenClaw, Woodpecker sync.
2026-03-15 16:37:38 +00:00
Viktor Barzin
d17a6e2fd3 fix calendar-query.py: use get_display_name(), URL-decode names, fix search API
- Replace deprecated cal.name with cal_name() helper using get_display_name()
- URL-decode calendar names (Formula+1 → Formula 1)
- Use cal.search(event=True) instead of deprecated date_search()
- Default to showing all calendars instead of filtering to Personal
2026-03-15 16:12:36 +00:00
Viktor Barzin
deeea5edab openclaw: replace cc-config NFS with dotfiles repo clone [ci skip]
- Add init container "install-dotfiles" that clones the dotfiles repo
  and installs skills/agents/hooks to OpenClaw's home directory
- Remove nfs_cc_config module and its volume mount
- Skills/agents now come from the same chezmoi-managed dotfiles repo
  that manages the Mac config, eliminating the dual-sync problem
2026-03-15 16:04:02 +00:00
Viktor Barzin
944d6d3b22 update claude knowledge: resource management learnings from right-sizing session [ci skip] 2026-03-15 15:38:37 +00:00
Viktor Barzin
5beb481dc4 fix immich TF drift from Kyverno ndots injection, right-size nvidia GPU operator
- immich: add lifecycle ignore_changes for dns_config on all 3 deployments
  to prevent perpetual plan drift from Kyverno ndots:2 mutation policy
- nvidia dcgm-exporter: 768Mi → 2560Mi (VPA upper 2091Mi, was under-provisioned)
- nvidia cuda-validator: 1024Mi → 256Mi (one-shot job, vastly over-provisioned)
2026-03-15 15:36:19 +00:00
Viktor Barzin
a6d281dbc6 vaultwarden: upgrade to 1.35.4, use Recreate strategy
- Upgrade from 1.35.2 to 1.35.4 (fixes API key login userDecryptionOptions bug)
- Switch deployment strategy from RollingUpdate to Recreate (iSCSI PVC can't multi-attach)
2026-03-15 15:35:09 +00:00
Viktor Barzin
3f0e8541f6 claude-memory: pin image to :17, fixes URL-decode crash on sync endpoint
The + in +00:00 timezone offsets was being URL-decoded to a space,
causing ValueError on the /api/memories/sync endpoint. Build :17
includes the fix. Using versioned tag instead of :latest to avoid
pull-through cache serving stale images.
2026-03-15 15:32:50 +00:00
Viktor Barzin
194281e527 right-size cluster memory: reduce overprovisioned, fix under-provisioned services
Phase 1 - Quick wins (~4.5 Gi saved):
- democratic-csi: add explicit sidecar resources (64-80Mi vs 256Mi LimitRange default)
- caretta: 768Mi → 600Mi (VPA upper 485Mi)
- immich-ml: 4Gi → 3584Mi (VPA upper 2.95Gi, GPU margin)
- onlyoffice: 3Gi → 2304Mi (VPA upper 1.82Gi)

Phase 2 - Safety fixes (prevent OOMKills):
- frigate: 2Gi/8Gi → 5Gi/10Gi (VPA upper 7.7Gi, was 4% headroom)
- openclaw: 1280Mi req → 2Gi req=limit (documented 2Gi requirement)

Phase 3 - Additional right-sizing:
- authentik workers: 1Gi → 896Mi x3 (VPA upper 722Mi)
- shlink: 512Mi/768Mi → 960Mi req=limit (VPA upper 780Mi, safety increase)

Phase 4 - Burstable QoS for lower tiers:
- tier-3-edge: 128Mi/128Mi → 96Mi req / 192Mi limit
- tier-4-aux: 128Mi/128Mi → 64Mi req / 256Mi limit

Phase 5 - Monitoring:
- Add ClusterMemoryRequestsHigh alert (>85% allocatable, 15m)
- Add ContainerNearOOM alert (>85% limit, 30m)
- Add PodUnschedulable alert (5m, critical)

Cluster: 92.7% → 90.8% memory requests. Stirling-pdf now schedulable.
2026-03-15 15:30:18 +00:00
Viktor Barzin
8bac6db48f add name/description/tools to review-loop agent frontmatter [ci skip] 2026-03-15 11:14:31 +00:00
Viktor Barzin
616370d34c rename planner agent to review-loop [ci skip] 2026-03-15 11:12:14 +00:00
Viktor Barzin
18d012db11 fix: reduce openclaw memory requests for scheduling
- openclaw: request 1280Mi (limit 2Gi), modelrelay request 128Mi
  (limit 256Mi). Total request 1408Mi fits available capacity.
2026-03-15 10:47:34 +00:00
Viktor Barzin
123e996b04 add planner agent: plan-review-fix convergence loop [ci skip] 2026-03-15 10:46:53 +00:00
Viktor Barzin
307b7f6819 update claude knowledge: infra operational learnings from commit history [ci skip]
Add resource management patterns, networking resilience, service-specific
notes, monitoring patterns, and NFS storage rules extracted from ~963 commits.
2026-03-15 10:46:45 +00:00
Viktor Barzin
56ddee457a fix: openclaw policy violation + reduce memory requests for capacity
- openclaw: fix Kyverno policy violation (node:22-alpine ->
  docker.io/library/node:22-alpine), reduce request to 1536Mi
  with 2Gi limit for overcommit
- rybbit/clickhouse: reduce 1Gi -> 768Mi (frees 256Mi)
- stirling-pdf: reduce 1536Mi -> 1200Mi (frees 336Mi)
2026-03-15 10:37:58 +00:00
Viktor Barzin
683f255c2e fix: increase memory for OOMKilled services
- hackmd: 64Mi -> 256Mi (Node.js app OOMKilled after 14min)
- n8n: limit 512Mi -> 768Mi (DB timeouts at 88% mem usage)
- speedtest: 128Mi -> 256Mi (OOMKilled during startup)
- shlink: limit 512Mi -> 768Mi (OOMKilled after startup)
2026-03-15 10:26:11 +00:00
Viktor Barzin
b219961bd8 fix: MySQL memory overcommit + shlink OOMKill
- dbaas: MySQL requests 4Gi -> 2Gi (limits stay 4Gi) to free 6Gi
  of request capacity. Actual usage is 1-1.5Gi per instance.
- url/shlink: increase memory limit 512Mi -> 768Mi (OOMKilled)
2026-03-15 03:22:07 +00:00
Viktor Barzin
0a69af618d update claude knowledge: vault KV secrets migration [ci skip] 2026-03-15 03:22:07 +00:00
Viktor Barzin
4a27345057 enable memory-core plugin for OpenClaw [ci skip]
- Add memory-core to plugins.allow and plugins.slots.memory
- Add /app/extensions to plugin load paths
- Update CLAUDE.md memory instructions to reference native tools
2026-03-15 03:22:07 +00:00
root
516685c504 Woodpecker CI deploy commit [CI SKIP] 2026-03-15 02:38:30 +00:00
Viktor Barzin
52bce849d7 fix cluster health: resolve 21/23 failures from healthcheck
- nvidia: change GPU taint NoSchedule -> PreferNoSchedule to allow
  overflow scheduling on k8s-node1 (frees ~7Gi capacity)
- kyverno: increase reports-controller memory 256Mi -> 512Mi (OOMKilled)
- speedtest: add missing DB_PORT=3306 env var (nc: service "" unknown)
- realestate-crawler: increase API memory 64Mi -> 256Mi (OOMKilled)
- calibre: increase liveness probe timeout 1s -> 5s (false restarts)
2026-03-15 02:33:46 +00:00