viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1905 commits 2 branches 0 tags 2.3 GiB
Commit graph

1905 commits

This branch
This branch
All branches
Author SHA1 Message Date
Viktor Barzin
efe0cdefc8
[ci skip] add Forgejo task pipeline for OpenClaw AI agent 
Forgejo issues as a task queue for OpenClaw:
- Forgejo OAuth2 with Authentik SSO, self-registration disabled
- Webhook-triggered task processing (instant) + CronJob backup (5min poll)
- Tasks processed via Mistral Large 3 (NVIDIA NIM API)
- Results posted as issue comments, auto-labeled and closed
- Comment follow-ups and reopened issues supported
- n8n RBAC for OpenClaw pod exec (future workflow integration)
 2026-03-07 21:11:07 +00:00
Viktor Barzin
0d03037393
add nginx caching proxy for Homepage widget API requests 
Stale-while-revalidate cache in front of Homepage reduces first-paint
latency by serving cached /api/ responses instantly while refreshing
upstream in background. Non-API paths pass through uncached.
 2026-03-07 21:11:07 +00:00
root
aeee51a1a3 Woodpecker CI deploy commit [CI SKIP] 2026-03-07 20:47:22 +00:00
Viktor Barzin
2b5e4888e0
add Google OAuth env vars to plotting-book deployment 
Deploy GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET, and GOOGLE_CALLBACK_URL
to the plotting-book container. Update CSP to allow accounts.google.com
for connect-src and form-action directives.
 2026-03-07 20:41:08 +00:00
Viktor Barzin
90c3372643
[ci skip] add liveness probe to Send deployment 
Prevents stale Redis connections from silently breaking file uploads.
The old Node.js Redis client doesn't auto-reconnect after Redis restarts,
causing all files to appear expired.
 2026-03-07 20:39:57 +00:00
Viktor Barzin
6dcb6cf55a
[ci skip] fix Navidrome credentials: admin user is wizard not admin 2026-03-07 20:39:56 +00:00
Viktor Barzin
04bed0188e
[ci skip] fix pfSense widget: wan interface is vtnet0 not vmx0 2026-03-07 20:39:56 +00:00
Viktor Barzin
c09e1a0b65
[ci skip] fix pfSense widget: remove wanStatus (API v2 missing gateway endpoint) 
Replace wanStatus with temp field. Remove wan interface param since
the pfSense REST API v2 package doesn't expose /status/gateway.
 2026-03-07 20:39:56 +00:00
Viktor Barzin
8583cd8578
[ci skip] fix widget issues: ports, Immich v2 API, Nextcloud trusted domains 
- qBittorrent: use service port 80 (not container port 8080)
- Immich: add version=2 for new API endpoints (/api/server/*)
- Nextcloud: use external URL (internal rejects untrusted Host header)
- HA London: remove widget (token expired, needs manual regeneration)
- Headscale: remove widget (requires nodeId param, not overview)
 2026-03-07 20:39:56 +00:00
Viktor Barzin
98b6839dd6
[ci skip] fix widget URLs: use correct k8s service ports 
Services expose port 80 via ClusterIP but widgets were using container
target ports (5000, 3001, 4533, 3000). Calibre was using external URL
through Authentik. All now use correct internal service URLs.
 2026-03-07 20:39:56 +00:00
Viktor Barzin
31a8eded7d
[ci skip] upgrade Homepage from v1.8.0 to v1.10.1 2026-03-07 20:39:56 +00:00
Viktor Barzin
2ebbf364d1
[ci skip] add widgets for qbittorrent, navidrome, nextcloud, freshrss, linkwarden, uptime-kuma 
Add API credentials to SOPS and wire homepage_credentials through
stacks. Re-add Uptime Kuma widget with new "infra" status page slug.
 2026-03-07 20:39:55 +00:00
Viktor Barzin
f9efa902ef
[ci skip] update ha-sofia VM: VMID 103, disk 64G, SSH access info 2026-03-07 20:39:55 +00:00
Viktor Barzin
d573e07674
[ci skip] add widgets for audiobookshelf, changedetection, prowlarr, headscale 
Wire homepage_credentials through servarr parent stack for prowlarr.
Fix paperless-ngx widget to use internal service URL.
 2026-03-07 20:39:55 +00:00
Viktor Barzin
ce41f6841f
[ci skip] fix broken Homepage widgets + add service API tokens to SOPS 
- Grafana: fix service URL (grafana not monitoring-grafana)
- Uptime Kuma: remove widget (no status page configured)
- Speedtest/Frigate/Immich: use internal k8s service URLs (external
  goes through Authentik forward auth, blocking API calls)
- pfSense: clean up annotations
- SOPS: add headscale, prowlarr, changedetection, audiobookshelf tokens
 2026-03-07 20:39:55 +00:00
Viktor Barzin
c3df3393a7
[ci skip] add Homepage widget credentials for Authentik, Shlink, Home Assistant 
Wire homepage_credentials tokens through platform stack to enable
live widgets for Authentik, Shlink (URL shortener), and Home Assistant
London. Update SOPS with new credential entries.
 2026-03-07 20:39:54 +00:00
Viktor Barzin
af74aa297d
[ci skip] add Homepage gethomepage.dev annotations to all services 
Add Kubernetes ingress annotations for Homepage auto-discovery across
~88 services organized into 11 groups. Enable serviceAccount for RBAC,
configure group layouts, and add Grafana/Frigate/Speedtest widgets.
 2026-03-07 20:39:54 +00:00
OpenClaw
f7994e012e Update MEMORY.md timestamp 2026-03-07 16:43:15 +00:00
Viktor Barzin
e86db29e80
[ci skip] fix false-positive sensitive=true on kube_config_path 2026-03-07 15:48:19 +00:00
Viktor Barzin
933250bd7b
[ci skip] claudeception: extract 2 skills from today's session 
1. sops-age-secrets-migration: Complete guide for migrating from git-crypt
   to SOPS+age. Covers JSON format requirement, race condition avoidance,
   CI integration, complex types, and migration sequence.

2. iterative-plan-review-with-subagents: Design pattern for reviewing plans
   with parallel security + implementation subagents. 2-3 iterations to
   zero CRITICALs. Used successfully for the SOPS migration design.
 2026-03-07 15:46:36 +00:00
Viktor Barzin
20ec276750
[ci skip] update AGENTS.md + CLAUDE.md with SOPS workflow, add k8s-portal CI pipeline 
AGENTS.md: added SOPS secrets management section, scripts/tg usage,
contributor onboarding steps, pull-through cache bypass notes.

CLAUDE.md: added SOPS workflow note, linux/amd64 build reminder,
versioned tag guidance for pull-through cache.

CI: new .woodpecker/k8s-portal.yml pipeline — auto-builds and deploys
the k8s portal when files under stacks/platform/modules/k8s-portal/files/
change on master push. Uses buildx for linux/amd64.
 2026-03-07 15:37:19 +00:00
Viktor Barzin
ed6a2780fb
[ci skip] fix Svelte 5 table structure (thead/tbody required) + use versioned image tag 
Fixed architecture and services pages to wrap table rows in <thead>/<tbody>
as required by Svelte 5's strict HTML validation.

E2E test passed: clean Alpine container → setup script → kubectl installed →
CA cert verified against API server → TLS SUCCESS
 2026-03-07 15:34:32 +00:00
Viktor Barzin
15f7114c4e
[ci skip] k8s portal: fix setup script + add onboarding hub (5 new pages) 
Bug fixes:
- CA cert now populated in ConfigMap (was empty → TLS failures)
- Remove useless heredoc quote escaping in setup script
- Fix homepage: VPN callout, correct verification command (get namespaces)
- Fix false-positive sensitive=true on ingress_path, tls_secret_name,
  truenas_host, ollama_host, client_certificate_secret_name

New pages (direct Svelte, no mdsvex dependency):
- /onboarding: step-by-step guide (VPN, kubectl, git, first PR)
- /architecture: cluster topology, storage, networking, tiers
- /services: catalog of 70+ services with URLs
- /contributing: PR workflow, what you can/can't change, NEVER list
- /troubleshooting: common issues and fixes

Navigation bar added to layout. All pages use consistent docs styling.

Requires Docker image rebuild: cd stacks/platform/modules/k8s-portal/files
&& docker build -t viktorbarzin/k8s-portal:latest . && docker push
 2026-03-07 15:06:26 +00:00
Viktor Barzin
9ed12587a6
[ci skip] update ha-london skill: SSH is hassio@192.168.8.103 (HA OS) 
Old Pi at 192.168.8.104 no longer runs HA. Updated SSH host, user,
config path, and platform info to reflect HA OS on 192.168.8.103.
 2026-03-07 14:34:44 +00:00
Viktor Barzin
db68067925
[ci skip] phase 5+6: update CI pipelines for SOPS, add sensitive=true to secret vars 
Phase 5 — CI pipelines:
- default.yml: add SOPS decrypt in prepare step, change git add . to
  specific paths (stacks/ state/ .woodpecker/), cleanup on success+failure
- renew-tls.yml: change git add . to git add secrets/ state/

Phase 6 — sensitive=true:
- Add sensitive = true to 256 variable declarations across 149 stack files
- Prevents secret values from appearing in terraform plan output
- Does NOT modify shared modules (ingress_factory, nfs_volume) to avoid
  breaking module interface contracts

Note: CI pipeline SOPS decryption requires sops_age_key Woodpecker secret
to be created before the pipeline will work with SOPS. Until then, the old
terraform.tfvars path continues to function.
 2026-03-07 14:30:36 +00:00
Viktor Barzin
154f8ff0c1
[ci skip] phase 3: switch terragrunt to load config.tfvars + SOPS secrets 
terragrunt.hcl now loads:
- config.tfvars (required, plaintext)
- terraform.tfvars (optional, git-crypt — backward compat)
- secrets.auto.tfvars.json (optional, SOPS-decrypted)

before_hook checks that at least one secrets source exists.
Use `scripts/tg` wrapper for SOPS-based workflow.
Old terraform.tfvars kept for reference and backward compatibility.
 2026-03-07 14:16:28 +00:00
Viktor Barzin
22267fe386
[ci skip] phase 2: split terraform.tfvars into config.tfvars + secrets.sops.json 
config.tfvars (29 vars, plaintext): hostnames, IPs, DNS records, IDs
secrets.sops.json (140 vars, SOPS-encrypted): passwords, tokens, keys, maps

Both files coexist with terraform.tfvars — no functional change yet.
Complex types preserved: maps (mailserver_accounts, k8s_users, homepage_credentials),
lists (xray_reality_clients), heredocs as \n-escaped JSON strings (SSH keys,
WireGuard conf, headscale config).
 2026-03-07 14:04:40 +00:00
Viktor Barzin
7f5dbb82f4
[ci skip] phase 1: SOPS tooling setup (.sops.yaml, scripts/tg, .gitignore) 
Part of SOPS multi-user secrets migration.
- .sops.yaml: defines age recipients (Viktor + CI)
- scripts/tg: wrapper that decrypts secrets before running terragrunt
- .gitignore: excludes decrypted secrets.auto.tfvars.json

No functional change — terraform.tfvars still works as before.
 2026-03-07 13:57:42 +00:00
Viktor Barzin
88989cfad3
[ci skip] add SOPS multi-user secrets migration design (v3, reviewed 3x) 
Replaces git-crypt all-or-nothing encryption with SOPS per-value encryption.
Operators push PRs → Viktor reviews → CI applies. No encryption keys needed
for operators. 7-phase migration plan, reviewed by 2 agents across 3 iterations
(0 remaining CRITICALs).
 2026-03-07 13:55:05 +00:00
Viktor Barzin
b73b2eac33
fix(actualbudget): raise http-api resources to prevent OOM [ci skip] 2026-03-07 00:28:02 +00:00
Viktor Barzin
7d68be870d
[ci skip] add auto-generated tiers.tf, planning docs, and helm chart cache 
- tiers.tf: Terragrunt-generated tier locals for all standalone stacks
- .planning/: resource audit research and plans
- docs/plans/: cluster hardening design doc
- redis-25.3.2.tgz: Bitnami Redis Helm chart cache
 2026-03-06 23:55:57 +00:00
Viktor Barzin
dc85c34069
[ci skip] add AGENTS.md for model-agnostic knowledge, slim CLAUDE.md to Claude-specific layer 
AGENTS.md (63 lines): shared infra knowledge for any AI tool (Codex, Claude,
Cursor). Covers: critical rules, architecture, storage, tiers, common ops.

CLAUDE.md (23 lines): Claude-specific addons — skills, agents, user preferences.
References AGENTS.md for shared knowledge.

Removed generic agents (devops-engineer, fullstack-developer).
 2026-03-06 23:50:26 +00:00
Viktor Barzin
f02f2a5a4d
[ci skip] refactor claude files: compact CLAUDE.md, clean memory, remove generic agents 
CLAUDE.md: 260→72 lines. Moved detailed patterns (NFS, iSCSI, Kyverno
tables, anti-AI, node rebuild) to .claude/reference/patterns.md.
Kept: critical rules, quick patterns, key commands, tier overview, prefs.

Memory: CLAUDE.md is now single source of truth. Auto-memory reduced to
scratch pad (67→25 lines, 5→1 files). MetaClaw DB cleaned from 40→16
entries (removed all infra-specific duplicates, kept cross-project prefs).

Agents: removed generic devops-engineer (885L) and fullstack-developer
(234L). Kept custom cluster-health-checker (48L).
 2026-03-06 23:27:46 +00:00
Viktor Barzin
51f2b040a6
[ci skip] archive 28 unused skills, add runbook index to CLAUDE.md, add cluster-health agent 
- Move 28 never-invoked troubleshooting runbook skills to .claude/skills/archived/
- Keep 7 active workflow skills: cluster-health, uptime-kuma, pfsense,
  home-assistant, setup-project, extend-vm-storage, k8s-ndots
- Add one-line runbook index to CLAUDE.md for quick reference
- Create cluster-health-checker custom agent (haiku model, read-only + bash)
  for autonomous health checks without consuming main context
 2026-03-06 23:17:40 +00:00
Viktor Barzin
1824d2be67
[ci skip] expand Prometheus PVC to 200Gi, increase retention to 180GB for 1-year history 
Storage analysis: ~10.5 GB/month ingestion rate, 1 year = ~125 GB + overhead.
PVC: 30Gi → 200Gi, retention.size: 45GB → 180GB.
Historical TSDB data restored from NFS (39.8 GB total including all blocks).
 2026-03-06 23:16:32 +00:00
Viktor Barzin
a7f3d432ee
[ci skip] expand Prometheus iSCSI PVC to 30Gi for historical data restore 2026-03-06 22:51:38 +00:00
Viktor Barzin
3eaac4b9db
[ci skip] update claude knowledge: iSCSI migration for Redis, Prometheus, Loki 2026-03-06 21:05:21 +00:00
Viktor Barzin
63fb6201c8
[ci skip] migrate Redis, Prometheus, Loki storage to iSCSI 
- Redis: local-path → iscsi-truenas (master + replica persistence)
- Prometheus: NFS PV+PVC → dynamic iSCSI PVC (prometheus-data)
- Loki: NFS PV → dynamic iSCSI via storageClass in Helm values
- Deleted 2 orphaned Released iSCSI PVs (31Gi freed)
 2026-03-06 20:50:55 +00:00
Viktor Barzin
ce0cab7554
[ci skip] replace resource overcommitment check with actual usage 
Check real CPU/memory usage via kubectl top nodes instead of
limits-vs-allocatable ratios. Thresholds: >80% WARN, >90% FAIL.
Limits overcommit is expected with 70+ services on 3 worker nodes.
 2026-03-06 20:28:55 +00:00
Viktor Barzin
bfa4a3ffdf
[ci skip] reduce resource limits per VPA recommendations 
dashy: 4Gi→512Mi mem, 2→500m cpu (actual: 206Mi)
affine: 4Gi→512Mi mem, 2→1 cpu (actual: 186Mi)
rybbit clickhouse: 4Gi→2Gi mem, 2→1 cpu (actual: 618Mi)
 2026-03-06 20:23:21 +00:00
Viktor Barzin
94dcf22db4
[ci skip] exclude linkwarden from HighService4xxRate alert 2026-03-06 20:15:58 +00:00
Viktor Barzin
d4400f8283
[ci skip] remove atuin: destroy stack, DNS, NFS export, PostgreSQL credentials 2026-03-06 20:11:14 +00:00
Viktor Barzin
1d80c49201
[ci skip] iSCSI migration, healthcheck fixes, health probes, etcd backup 
- Migrate MySQL/PostgreSQL storage from local-path to iscsi-truenas
- Add democratic-csi iSCSI driver module for TrueNAS
- Add open-iscsi to cloud-init VM template
- Fix Shlink health probe path (/api/v3 -> /rest/v3 for Shlink 5.0)
- Fix etcd backup: use etcd 3.5.21-0 (3.6.x is distroless, no /bin/sh)
- Fix cluster healthcheck CronJob: always exit 0 to prevent circular
  JobFailed alerts (reporting via Slack, not exit codes)
- Fix Uptime Kuma nested list handling in cluster-health.sh
- Add health probes to: audiobookshelf, immich ML, ntfy, headscale,
  uptime-kuma, vaultwarden, rybbit (clickhouse + server + client),
  shlink, shlink-web
- Add iSCSI storage documentation to CLAUDE.md
 2026-03-06 19:54:21 +00:00
Viktor Barzin
a8e07ad930
[ci skip] fix post-NFS-migration issues: MySQL GR, Loki, grampsweb, alerts 
- Loki: reduce memory limit from 6Gi to 4Gi (within LimitRange max)
- Grampsweb: increase memory to 2Gi (was OOMKilled at 512Mi)
- Fix PostgreSQLDown alert: check pod readiness instead of deployment
- Fix MySQLDown alert: check StatefulSet replicas instead of deployment
- Fix RedisDown alert: check StatefulSet replicas instead of deployment
- Fix NFSServerUnresponsive: aggregate all NFS versions cluster-wide
- Fix Uptime Kuma healthcheck: handle nested list heartbeat format
- Update etcd backup image to registry.k8s.io/etcd:3.6.5-0
 2026-03-03 21:10:26 +00:00
Viktor Barzin
065090dfe0
[ci skip] fix calibre: bump CPU/memory to prevent SIGBUS during calibre_postinstall 2026-03-03 19:48:45 +00:00
Viktor Barzin
31f3fc0773
[ci skip] fix OOMKill: prometheus (4Gi), kyverno-reports (512Mi), grampsweb (512Mi) 
- Prometheus server: explicit 1Gi req / 4Gi limit (was inheriting 512Mi LimitRange default)
- Kyverno reports controller: 128Mi req / 512Mi limit (was 128Mi Helm default)
- Grampsweb: 256Mi req / 512Mi limit for both containers (was 256Mi LimitRange default)
 2026-03-02 21:39:14 +00:00
Viktor Barzin
ea67939525
[ci skip] add security observability layer design document 
Tetragon-centric approach: eBPF runtime security, pfSense syslog
collection, CoreDNS query logging, Calico NetworkPolicies,
on-demand mitmproxy, unified Grafana security dashboard.
~625MB steady-state, <5GB budget.
 2026-03-02 21:13:01 +00:00
Viktor Barzin
51d77369de
[ci skip] fix: add mount_options to all NFS PVs (soft,timeo=30,retrans=3) 
Critical fix: StorageClass mountOptions only apply during dynamic
provisioning. Our static PVs (created by Terraform) were missing
mount_options, so all NFS mounts defaulted to hard,timeo=600 —
the exact stale mount behavior we were trying to eliminate.

Adds mount_options directly to the nfs_volume module PV spec and
to the monitoring PVs (prometheus, loki, alertmanager).

Requires re-applying all stacks to propagate to existing PVs.
 2026-03-02 20:23:36 +00:00
Viktor Barzin
8137c8df63
[ci skip] fix: add mount_options to nfs_volume PV spec 
StorageClass mountOptions only apply during dynamic provisioning.
Static PVs (created by Terraform) need mount_options set explicitly.
Without this, all CSI NFS mounts default to hard,timeo=600 — the
exact problem we were trying to fix.
 2026-03-02 20:22:47 +00:00
Viktor Barzin
70dd172ba7
[ci skip] update CLAUDE.md: NFS volume pattern now uses CSI-backed nfs_volume module 2026-03-02 02:04:47 +00:00