infra/.claude/CLAUDE.md

Infrastructure Repository Knowledge

Instructions for Claude

• When the user says "remember" something: Always update this file (.claude/CLAUDE.md) with the information so it persists across sessions
• When discovering new patterns or versions: Add them to the appropriate section below
• When making infrastructure changes: Always update this file to reflect the current state (new services, removed services, version changes, config changes)
• After every significant change: Proactively update this file (.claude/CLAUDE.md) to reflect what changed — new services, config changes, version bumps, new patterns, etc. This ensures knowledge persists across sessions automatically.
• After updating any .claude/ files: Always commit them immediately (git add .claude/ && git commit -m "[ci skip] update claude knowledge") to avoid building up unstaged changes.
• Skills available: Check .claude/skills/ directory for specialized workflows (e.g., setup-project for deploying new services)
• Reference data: Check .claude/reference/ for inventory tables, API patterns, and current state snapshots
• CRITICAL: All infrastructure changes must go through Terraform/Terragrunt. NEVER modify cluster resources directly (kubectl apply/edit/patch, helm install, docker run). Use kubectl only for read-only operations and ephemeral debugging.
• CRITICAL: NEVER put sensitive data (API keys, passwords, tokens, credentials) into committed files unless encrypted via git-crypt. Secrets belong in terraform.tfvars or secrets/ directory.
• CRITICAL: NEVER commit secrets — triple-check before every commit. Zero exceptions.
• New services MUST have CI/CD (Woodpecker CI pipeline) and monitoring (Prometheus alerts and/or Uptime Kuma).

Execution Environment

• Terraform/Terragrunt: Always run locally: cd stacks/<service> && terragrunt apply --non-interactive
• kubectl: kubectl --kubeconfig $(pwd)/config
• GitHub API: Use curl with tokens from tfvars (see .claude/reference/github-api.md). gh CLI is blocked by sandbox.

---

Overview

Terragrunt-based infrastructure repository managing a home Kubernetes cluster on Proxmox VMs, with per-service state isolation. Each service has its own Terragrunt stack under stacks/. Uses git-crypt for secrets encryption.

Key File Paths

• terraform.tfvars — All secrets, DNS, Cloudflare config, WireGuard peers (git-crypt encrypted)
• terragrunt.hcl — Root config (providers, backend, variable loading)
• stacks/<service>/ — Individual service stacks (terragrunt.hcl + main.tf)
• stacks/platform/ — Core infrastructure (~22 services in modules/ subdir)
• stacks/infra/ — Proxmox VM resources
• modules/kubernetes/ingress_factory/, setup_tls_secret/ — Shared utility modules
• secrets/ — git-crypt encrypted TLS certs and keys

Domains

• Public: viktorbarzin.me (Cloudflare-managed)
• Internal: viktorbarzin.lan (Technitium DNS)

Key Patterns

NFS Volume Pattern

Prefer inline NFS volumes over separate PV/PVC resources. Use var.nfs_server (defined in terraform.tfvars, auto-loaded by Terragrunt):

volume {
  name = "data"
  nfs {
    server = var.nfs_server
    path   = "/mnt/main/<service>"
  }
}

Only use PV/PVC when a Helm chart requires existingClaim.

Adding NFS Exports

1. Edit secrets/nfs_directories.txt — add path, keep sorted
2. Run secrets/nfs_exports.sh from secrets/ to update TrueNAS

Factory Pattern (multi-user services)

Structure: stacks/<service>/main.tf + factory/main.tf. Examples: actualbudget, freedify. To add a user: export NFS share, add Cloudflare route in tfvars, add module block calling factory.

SMTP/Email

• Use: var.mail_host (defaults to mail.viktorbarzin.me) port 587 (STARTTLS). NOT mailserver.mailserver.svc.cluster.local (TLS cert mismatch).
• Credentials: mailserver_accounts in tfvars. Common: info@viktorbarzin.me

Anti-AI Scraping (5-Layer Defense)

All services have anti_ai_scraping = true by default in ingress_factory. Layers:

1. Bot blocking (traefik-ai-bot-block): ForwardAuth → poison-fountain /auth. Returns 403 for GPTBot, ClaudeBot, CCBot, etc.
2. X-Robots-Tag (traefik-anti-ai-headers): Adds noai, noimageai
3. Trap links (traefik-anti-ai-trap-links): rewrite-body injects 5 hidden links before </body> to poison.viktorbarzin.me/article/*
4. Tarpit: /article/* drip-feeds at ~100 bytes/sec
5. Poison content: 50 cached docs from rnsaffn.com/poison2/ (CronJob every 6h, --http1.1 required)

Key files: stacks/poison-fountain/, stacks/platform/modules/traefik/middleware.tf, modules/kubernetes/ingress_factory/main.tf Testing: curl -s -H "Accept: text/html,application/xhtml+xml" https://vaultwarden.viktorbarzin.me/ | grep -oE 'href="https://poison[^"]*"' Disable per-service: anti_ai_scraping = false in ingress_factory call.

Terragrunt Architecture

• Root terragrunt.hcl provides DRY provider, backend, variable loading, and shared tiers locals (via generate "tiers" block)
• Each stack: stacks/<service>/main.tf with resources inline, state at state/stacks/<service>/terraform.tfstate
• Platform modules: stacks/platform/modules/<service>/, shared modules: modules/kubernetes/
• Dependencies via dependency block; variables from terraform.tfvars (unused silently ignored)
• secrets/ symlinks in stacks for TLS cert path resolution
• Syntax: --non-interactive (not --terragrunt-non-interactive), terragrunt run --all -- <command> (not run-all)
• Tiers locals: Auto-generated by Terragrunt into tiers.tf in every stack — do NOT add locals { tiers = { ... } } to stacks manually

Adding a New Service

Use the setup-project skill for the full workflow. Quick reference:

1. Create stacks/<service>/ with terragrunt.hcl, main.tf, secrets symlink
2. Add Cloudflare DNS in terraform.tfvars
3. Apply platform stack (for DNS): cd stacks/platform && terragrunt apply --non-interactive
4. Apply service: cd stacks/<service> && terragrunt apply --non-interactive

Shared Infrastructure Variables

All stacks use variables from terraform.tfvars for shared service endpoints (auto-loaded by Terragrunt). Never hardcode these values:

• var.nfs_server — NFS server IP (10.0.10.15)
• var.redis_host — Redis hostname (redis.redis.svc.cluster.local)
• var.postgresql_host — PostgreSQL hostname (postgresql.dbaas.svc.cluster.local)
• var.mysql_host — MySQL hostname (mysql.dbaas.svc.cluster.local)
• var.ollama_host — Ollama hostname (ollama.ollama.svc.cluster.local)
• var.mail_host — Mail server hostname (mail.viktorbarzin.me)

For standalone stacks: add variable "nfs_server" { type = string } (etc.) to main.tf. For platform submodules: add the variable AND pass it through in stacks/platform/main.tf module block.

Useful Commands

bash scripts/cluster_healthcheck.sh            # Cluster health (24 checks)
bash scripts/cluster_healthcheck.sh --quiet    # Only WARN/FAIL
cd stacks/<service> && terragrunt apply --non-interactive  # Apply single stack
cd stacks && terragrunt run --all --non-interactive -- plan  # Plan all
terraform fmt -recursive                       # Format all

CI/CD

• Woodpecker CI (.woodpecker/): pushes apply platform stack, hosted at https://ci.viktorbarzin.me
• TLS renewal pipeline: cron-triggered renew2.sh (certbot + Cloudflare DNS)
• ALWAYS add [ci skip] to commit messages when you've already applied locally
• After committing, run git push origin master to sync

Infrastructure

• Proxmox hypervisor (192.168.1.127) — see .claude/reference/proxmox-inventory.md for full VM table
• Kubernetes cluster: 5 nodes (k8s-master + k8s-node1-4, v1.34.2), GPU on node1 (Tesla T4)
• NFS: var.nfs_server (10.0.10.15), Redis: var.redis_host (redis.redis.svc.cluster.local)
• PostgreSQL: var.postgresql_host (postgresql.dbaas.svc.cluster.local), MySQL: var.mysql_host (mysql.dbaas.svc.cluster.local)
• Ollama: var.ollama_host (ollama.ollama.svc.cluster.local), Mail: var.mail_host (mail.viktorbarzin.me)
• Docker registry pull-through cache at 10.0.20.10 (ports 5000/5010/5020/5030/5040)
• GPU workloads need: node_selector = { "gpu": "true" } + toleration { key = "nvidia.com/gpu", value = "true", effect = "NoSchedule" }

Node Rebuild Procedure

To rebuild a K8s worker node from scratch (e.g., after disk failure or corruption):

1. Drain the node (if still reachable): kubectl drain k8s-nodeX --ignore-daemonsets --delete-emptydir-data
2. Delete the node from K8s: kubectl delete node k8s-nodeX
3. Destroy the VM in Proxmox (or via Terraform: remove from stacks/infra/main.tf and apply)
4. Ensure K8s template exists: The template ubuntu-2404-cloudinit-k8s-template (VMID 2000) must exist. If not, apply stacks/infra/ to recreate it.
5. Get a fresh join command: ssh wizard@10.0.20.100 'sudo kubeadm token create --print-join-command'
6. Update k8s_join_command in terraform.tfvars with the new join command
7. Create the new VM: Add it back in stacks/infra/main.tf and cd stacks/infra && terragrunt apply --non-interactive
8. Wait for cloud-init: The VM will install packages, configure containerd mirrors, and join the cluster automatically via cloud-init
9. Verify the node joined: kubectl get nodes — should show the new node as Ready
10. For GPU node (k8s-node1) only: Apply the platform stack to re-apply GPU label and taint: cd stacks/platform && terragrunt apply --non-interactive (the null_resource.gpu_node_config in the nvidia module handles this)
11. Verify containerd mirrors: ssh wizard@<node-ip> 'ls /etc/containerd/certs.d/' — should show docker.io, ghcr.io, quay.io, registry.k8s.io, reg.kyverno.io

Note: kubeadm tokens expire after 24h by default. Generate a fresh one just before creating the VM.

Git Operations

• Git is slow — commands can take 30+ seconds. Use GIT_OPTIONAL_LOCKS=0 if git hangs.
• Commit only specific files. ALWAYS ask user before pushing.

Prometheus Alerts

• Rules in stacks/platform/modules/monitoring/prometheus_chart_values.tpl
• Groups: "R730 Host", "Nvidia Tesla T4 GPU", "Power", "Storage", "K8s Health", "Infrastructure Health", "Critical Services", "Cluster", "Traefik Ingress"
• Critical Services group (added 2026-02): PostgreSQLDown, MySQLDown, RedisDown, HeadscaleDown, AuthentikDown, LokiDown
• Predictive alerts: PVPredictedFull (predict_linear 24h ahead)
• Loki alert rules: HighErrorRate, PodCrashLoopBackOff, OOMKilled (in loki.tf ConfigMap)

Tier System & Resource Governance

• 0-core: Critical infra (ingress, DNS, VPN, auth) | 1-cluster: Redis, metrics, security | 2-gpu: GPU workloads | 3-edge: User-facing | 4-aux: Optional
• Tiers locals: Generated by root terragrunt.hcl into tiers.tf — available as local.tiers.core, local.tiers.cluster, etc. in all stacks
• Kyverno-based governance in stacks/platform/modules/kyverno/resource-governance.tf:
  1. PriorityClasses: tier-0-core (1M) through tier-4-aux (200K, preemption=Never)
  2. LimitRange defaults (Kyverno generate): auto-created per namespace tier
  3. ResourceQuotas (Kyverno generate): auto-created per namespace tier (skip with label resource-governance/custom-quota=true)
  4. Priority injection (Kyverno mutate): sets priorityClassName on Pods
• Custom quota override: monitoring, crowdsec, nvidia, realestate-crawler
• Pod Security Policies (Kyverno, audit mode) in stacks/platform/modules/kyverno/security-policies.tf:
  1. deny-privileged-containers: Blocks privileged: true (exempt: frigate, nvidia, monitoring)
  2. deny-host-namespaces: Blocks hostNetwork/PID/IPC (exempt: frigate, monitoring)
  3. restrict-sys-admin: Blocks SYS_ADMIN capability (exempt: nvidia, monitoring)
  4. require-trusted-registries: Validates images from docker.io, ghcr.io, quay.io, registry.k8s.io, 10.0.20.10

Traefik Security (hardened 2026-02)

• API dashboard: api.insecure = false — dashboard only accessible via Authentik-protected ingress
• Forwarded headers: insecure=false with trustedIPs set to Cloudflare IPv4 ranges + internal (10.0.0.0/8, 192.168.0.0/16)
• Security headers middleware (security-headers): HSTS (31536000s, includeSubDomains), X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy: strict-origin-when-cross-origin, Permissions-Policy
• Rate limiting: average=10, burst=50 (default); average=100, burst=1000 (immich-rate-limit for uploads)

---

User Preferences

• Calendar: Nextcloud at https://nextcloud.viktorbarzin.me
• Home Assistant: ha-london (default) at https://ha-london.viktorbarzin.me, ha-sofia at https://ha-sofia.viktorbarzin.me. "ha"/"HA" = ha-london.
• Frontend: Svelte for all new web apps
• Pod monitoring: Never use sleep — spawn background subagent with kubectl get pods -w instead

---

Reference Data

• .claude/reference/service-catalog.md — Full service catalog (70+ services) with Cloudflare domains
• .claude/reference/proxmox-inventory.md — VM table, hardware specs, network topology, GPU config
• .claude/reference/github-api.md — GitHub API patterns with curl examples
• .claude/reference/authentik-state.md — Current applications, groups, users, login sources

---

Service-Specific Notes

Authentik (Identity Provider)

• URL: https://authentik.viktorbarzin.me | API: /api/v3/ | Token: authentik_api_token in tfvars
• Architecture: 3 server + 3 worker + 3 PgBouncer + embedded outpost
• Database: PostgreSQL via postgresql.dbaas:5432, PgBouncer at pgbouncer.authentik:6432
• Traefik integration: Forward auth via protected = true in ingress_factory
• OIDC for K8s: Issuer https://authentik.viktorbarzin.me/application/o/kubernetes/, client kubernetes (public)
• For management tasks, current state, and OIDC gotchas: see authentik and authentik-oidc-kubernetes skills
• For current apps/groups/users snapshot: see .claude/reference/authentik-state.md

AFFiNE (Visual Canvas)

• Image: ghcr.io/toeverything/affine:stable | Port: 3010 | Requires: PostgreSQL + Redis
• Migration: Init container runs node ./scripts/self-host-predeploy.js
• Storage: NFS /mnt/main/affine → /root/.affine/storage and /root/.affine/config

Wyoming Whisper (STT)

• Image: rhasspy/wyoming-whisper:latest | Port: 10300/TCP (Wyoming protocol)
• Model: small-int8 (CPU-only) | Access: 10.0.20.202:10300 (internal, no public DNS)
• HA Integration: Wyoming Protocol in ha-london

Gramps Web (Genealogy)

• Image: ghcr.io/gramps-project/grampsweb:latest | Port: 5000 | URL: https://family.viktorbarzin.me
• Components: Web app + Celery worker (2 containers in 1 pod) | Redis: DB 2 (broker), DB 3 (rate limiting)
• Storage: NFS /mnt/main/grampsweb with sub_paths
• Auth: Protected by Authentik (added 2026-02)

Loki + Alloy (Log Collection)

• Loki: grafana/loki:3.6.5 (single binary, 6Gi RAM, 30d retention / 720h)
• Alloy: grafana/alloy:v1.13.0 (DaemonSet, 512Mi requests / 1Gi limits)
• Storage: NFS PV /mnt/main/loki/loki (15Gi), WAL on tmpfs (2Gi)
• Alert rules: HighErrorRate, PodCrashLoopBackOff, OOMKilled (ConfigMap loki-alert-rules)
• Troubleshooting: "entry too far behind" on first start → restart Alloy DaemonSet

OpenClaw (AI Agent Gateway)

• Image: ghcr.io/openclaw/openclaw:2026.2.9 | Port: 18789 | URL: https://openclaw.viktorbarzin.me
• Init container: Downloads kubectl, terraform, git-crypt; clones infra repo
• ServiceAccount: openclaw with cluster-admin ClusterRoleBinding
• Model providers: Gemini (gemini-2.5-flash), Ollama (qwen2.5-coder:14b, deepseek-r1:14b), Llama API

Service Versions (as of 2026-02)

Immich v2.4.1 | AFFiNE stable | Whisper latest | Loki 3.6.5 | Alloy v1.13.0 | OpenClaw 2026.2.9