viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
infra/AGENTS.md
Viktor Barzin c9d221d578 [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 
## Context

Phase 1 of the state-drift consolidation audit (plan Wave 3) identified that
the entire repo leans on a repeated `lifecycle { ignore_changes = [...dns_config] }`
snippet to suppress Kyverno's admission-webhook dns_config mutation (the ndots=2
override that prevents NxDomain search-domain flooding). 27 occurrences across
19 stacks. Without this suppression, every pod-owning resource shows perpetual
TF plan drift.

The original plan proposed a shared `modules/kubernetes/kyverno_lifecycle/`
module emitting the ignore-paths list as an output that stacks would consume in
their `ignore_changes` blocks. That approach is architecturally impossible:
Terraform's `ignore_changes` meta-argument accepts only static attribute paths
— it rejects module outputs, locals, variables, and any expression (the HCL
spec evaluates `lifecycle` before the regular expression graph). So a DRY
module cannot exist. The canonical pattern IS the repeated snippet.

What the snippet was missing was a *discoverability tag* so that (a) new
resources can be validated for compliance, (b) the existing 27 sites can be
grep'd in a single command, and (c) future maintainers understand the
convention rather than each reinventing it.

## This change

- Introduces `# KYVERNO_LIFECYCLE_V1` as the canonical marker comment.
  Attached inline on every `spec[0].template[0].spec[0].dns_config` line
  (or `spec[0].job_template[0].spec[0]...` for CronJobs) across all 27
  existing suppression sites.
- Documents the convention with rationale and copy-paste snippets in
  `AGENTS.md` → new "Kyverno Drift Suppression" section.
- Expands the existing `.claude/CLAUDE.md` Kyverno ndots note to reference
  the marker and explain why the module approach is blocked.
- Updates `_template/main.tf.example` so every new stack starts compliant.

## What is NOT in this change

- The `kubernetes_manifest` Kyverno annotation drift (beads `code-seq`)
  — that is Phase B with a sibling `# KYVERNO_MANIFEST_V1` marker.
- Behavioral changes — every `ignore_changes` list is byte-identical
  save for the inline comment.
- The fallback module the original plan anticipated — skipped because
  Terraform rejects expressions in `ignore_changes`.
- `terraform fmt` cleanup on adjacent unrelated blocks in three files
  (claude-agent-service, freedify/factory, hermes-agent). Reverted to
  keep this commit scoped to the convention rollout.

## Before / after

Before (cannot distinguish accidental-forgotten from intentional-convention):
```hcl
lifecycle {
  ignore_changes = [spec[0].template[0].spec[0].dns_config]
}
```

After (greppable, self-documenting, discoverable by tooling):
```hcl
lifecycle {
  ignore_changes = [spec[0].template[0].spec[0].dns_config] # KYVERNO_LIFECYCLE_V1
}
```

## Test Plan

### Automated
```
$ rg -c 'KYVERNO_LIFECYCLE_V1' stacks/ --include='*.tf' --include='*.tf.example' \
    | awk -F: '{s+=$2} END {print s}'
27

$ git diff --stat | grep -E '\.(tf|tf\.example|md)$' | wc -l
21

# All code-file diffs are 1 insertion + 1 deletion per marker site,
# except beads-server (3), ebooks (4), immich (3), uptime-kuma (2).
$ git diff --stat stacks/ | tail -1
20 files changed, 45 insertions(+), 28 deletions(-)
```

### Manual Verification

No apply required — HCL comments only. Zero effect on any stack's plan output.
Future audits: `rg 'KYVERNO_LIFECYCLE_V1' stacks/ | wc -l` must grow as new
pod-owning resources are added.

## Reproduce locally
1. `cd infra && git pull`
2. `rg 'KYVERNO_LIFECYCLE_V1' stacks/` → expect 27 hits in 19 files
3. Grep any new `kubernetes_deployment` for the marker; absence = missing
   suppression.

Closes: code-28m

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 14:15:51 +00:00

11 KiB
Raw Blame History

Infrastructure Repository — AI Agent Instructions

Critical Rules (MUST FOLLOW)

  • ALL changes through Terraform/Terragrunt — NEVER kubectl apply/edit/patch/delete for persistent changes. Read-only kubectl is fine.
  • NEVER put secrets in plaintext — use secrets.sops.json (SOPS-encrypted) or terraform.tfvars (git-crypt, legacy)
  • NEVER restart NFS on the Proxmox host — causes cluster-wide mount failures across all pods
  • NEVER commit secrets — triple-check before every commit
  • [ci skip] in commit messages when changes were already applied locally
  • Ask before git push — always confirm with the user first

Execution

  • Apply a service: scripts/tg apply --non-interactive (auto-decrypts SOPS secrets)
  • Legacy apply: cd stacks/<service> && terragrunt apply --non-interactive (uses terraform.tfvars)
  • kubectl: kubectl --kubeconfig $(pwd)/config
  • Health check: bash scripts/cluster_healthcheck.sh --quiet
  • Plan all: cd stacks && terragrunt run --all --non-interactive -- plan

Secrets Management (SOPS)

  • config.tfvars — plaintext config (hostnames, IPs, DNS records, public keys)
  • secrets.sops.json — SOPS-encrypted secrets (passwords, tokens, SSH keys, API keys)
  • .sops.yaml — defines who can decrypt (age public keys: Viktor + CI)
  • scripts/tg — wrapper that auto-decrypts SOPS before running terragrunt
  • Edit secrets: sops secrets.sops.json (opens $EDITOR, re-encrypts on save)
  • Add a secret: sops set secrets.sops.json '["new_key"]' '"value"'
  • Operators push PRs → Viktor reviews → CI decrypts and applies. No encryption keys needed for operators.

Sealed Secrets (User-Managed Secrets)

For secrets that users manage themselves (no SOPS/git-crypt access needed):

  1. Create: kubectl create secret generic <name> --from-literal=key=value -n <ns> --dry-run=client -o yaml | kubeseal --controller-name sealed-secrets --controller-namespace sealed-secrets -o yaml > sealed-<name>.yaml
  2. Commit: Place sealed-*.yaml files in the stack directory (stacks/<service>/)
  3. Terraform picks them up automatically via fileset + for_each: 
    resource "kubernetes_manifest" "sealed_secrets" {
  for_each = fileset(path.module, "sealed-*.yaml")
  manifest = yamldecode(file("${path.module}/${each.value}"))
}
  4. Deploy: Push → CI runs terragrunt apply → controller decrypts into real K8s Secrets
  • Only the in-cluster controller has the private key. kubeseal uses the public key — safe to distribute.
  • Naming convention: files MUST match sealed-*.yaml glob pattern.
  • The kubernetes_manifest block is safe to add even with zero sealed-*.yaml files (empty for_each).

Architecture

Terragrunt-based homelab managing a Kubernetes cluster (5 nodes, v1.34.2) on Proxmox VMs.

  • 100+ stacks, each in stacks/<service>/ with its own Terraform state
  • Core platform: stacks/platform/ is now an empty shell — all modules have been extracted to independent stacks under stacks/
  • Public domain: viktorbarzin.me (Cloudflare) | Internal: viktorbarzin.lan (Technitium DNS)
  • Onboarding portal: https://k8s-portal.viktorbarzin.me — self-service kubectl setup + docs
  • CI/CD: Woodpecker CI — PRs run plan, merges to master auto-apply all stacks

Key Paths

  • stacks/<service>/main.tf — service definition
  • stacks/platform/modules/<service>/ — core infra modules
  • modules/kubernetes/ingress_factory/ — standardized ingress with auth, rate limiting, anti-AI, and auto Cloudflare DNS (dns_type = "proxied" or "non-proxied")
  • modules/kubernetes/nfs_volume/ — NFS volume module (CSI-backed, soft mount)
  • config.tfvars — non-secret configuration (plaintext)
  • secrets.sops.json — all secrets (SOPS-encrypted JSON)
  • terraform.tfvars — legacy secrets file (git-crypt, kept for reference)
  • scripts/cluster_healthcheck.sh — 25-check cluster health script

Storage

  • NFS (nfs-proxmox StorageClass): For app data. Use the nfs_volume module, never inline nfs {} blocks.
  • proxmox-lvm-encrypted (proxmox-lvm-encrypted StorageClass): Default for all sensitive data — databases, auth, email, passwords, git repos, health data. LUKS2 encryption via Proxmox CSI. Passphrase in Vault, backup key on PVE host.
  • proxmox-lvm (proxmox-lvm StorageClass): For non-sensitive stateful apps (configs, caches, tools). Proxmox CSI driver.
  • NFS server: Proxmox host at 192.168.1.127. HDD NFS at /srv/nfs (2TB ext4 LV pve/nfs-data), SSD NFS at /srv/nfs-ssd (100GB ext4 LV ssd/nfs-ssd-data). Exports use async mode (safe with UPS + databases on block storage). TrueNAS (10.0.10.15) decommissioned.
  • SQLite on NFS is unreliable (fsync issues) — always use proxmox-lvm or local disk for databases.
  • NFS mount options: Always soft,timeo=30,retrans=3 to prevent uninterruptible sleep (D state).
  • NFS export directory must exist on the Proxmox host before Terraform can create the PV.
  • Backup (3-2-1): Copy 1 = live PVCs on sdc. Copy 2 = sda /mnt/backup (PVC file backups, auto SQLite backups, pfSense, PVE config). Copy 3 = Synology offsite (two-tier: sda→pve-backup/, NFS→nfs/+nfs-ssd/ via inotify change tracking).
  • daily-backup (Daily 05:00): Auto-discovered BACKUP_DIRS (glob), auto SQLite backup (magic number + ?mode=ro), pfSense, PVE config. No NFS mirror step (NFS syncs directly to Synology via inotify).
  • offsite-sync-backup (Daily 06:00): Step 1: sda→Synology pve-backup/. Step 2: NFS→Synology nfs/+nfs-ssd/ via rsync --files-from (inotify change log). Monthly full --delete.
  • nfs-change-tracker.service: inotifywait on /srv/nfs + /srv/nfs-ssd, logs to /mnt/backup/.nfs-changes.log. Incremental syncs complete in seconds.
  • Synology layout (/volume1/Backup/Viki/): pve-backup/ (from sda), nfs/ (from /srv/nfs), nfs-ssd/ (from /srv/nfs-ssd). truenas/ renamed to nfs/, pve-backup/nfs-mirror/ removed.

Shared Variables (never hardcode)

var.nfs_server (192.168.1.127), var.redis_host, var.postgresql_host, var.mysql_host, var.ollama_host, var.mail_host

Kyverno Drift Suppression (# KYVERNO_LIFECYCLE_V1)

Kyverno's admission webhook mutates every pod with a dns_config { option { name = "ndots"; value = "2" } } block (fixes NxDomain search-domain floods — see k8s-ndots-search-domain-nxdomain-flood skill). Terraform does not manage that field, so without suppression every pod-owning resource shows perpetual spec[0].template[0].spec[0].dns_config drift.

Rule: every kubernetes_deployment, kubernetes_stateful_set, kubernetes_daemon_set, and kubernetes_cron_job_v1 MUST include the following lifecycle block, tagged with the # KYVERNO_LIFECYCLE_V1 marker so every site is greppable:

# kubernetes_deployment / kubernetes_stateful_set / kubernetes_daemon_set
lifecycle {
  ignore_changes = [spec[0].template[0].spec[0].dns_config] # KYVERNO_LIFECYCLE_V1
}

# kubernetes_cron_job_v1 (extra job_template nesting)
lifecycle {
  ignore_changes = [spec[0].job_template[0].spec[0].template[0].spec[0].dns_config] # KYVERNO_LIFECYCLE_V1
}

Why not a shared module? Terraform's ignore_changes meta-argument only accepts static attribute paths. It rejects module outputs, locals, variables, and any expression. A DRY module is therefore impossible — the canonical pattern IS the snippet + marker. When kubernetes_manifest resources get Kyverno generate.kyverno.io/* annotations mutated, a sibling convention # KYVERNO_MANIFEST_V1 will be introduced (Phase B).

Audit: rg "KYVERNO_LIFECYCLE_V1" stacks/ | wc -l — should grow (never shrink). Add the marker to every new pod-owning resource. The _template/main.tf.example stub shows the canonical form.

Tier System

0-core | 1-cluster | 2-gpu | 3-edge | 4-aux — Kyverno auto-generates LimitRange + ResourceQuota per namespace based on tier label.

  • Containers without explicit resources {} get default limits (256Mi for edge/aux — causes OOMKill for heavy apps)
  • Always set explicit resources on containers that need more than defaults
  • Opt-out: labels resource-governance/custom-quota=true / resource-governance/custom-limitrange=true

Infrastructure

  • Proxmox: 192.168.1.127 (Dell R730, 22c/44t, 142GB RAM)
  • Nodes: k8s-master (10.0.20.100), node1 (GPU, Tesla T4), node2-4
  • GPU: node_selector = { "gpu": "true" } + toleration nvidia.com/gpu
  • Pull-through cache: 10.0.20.10 — docker.io (:5000), ghcr.io (:5010) only. Caches stale manifests for :latest tags — use versioned tags or pre-pull with ctr --hosts-dir '' to bypass.
  • pfSense: 10.0.20.1 (gateway, firewall, DNS forwarding)
  • MySQL InnoDB Cluster: 1 instance on proxmox-lvm (scaled from 3 — only Uptime Kuma + phpIPAM remain), PriorityClass mysql-critical + PDB, anti-affinity excludes k8s-node1 (GPU node)
  • SMTP: var.mail_host port 587 STARTTLS (not internal svc address — cert mismatch)

Contributor Onboarding

  1. Get Authentik account + Headscale VPN access (ask Viktor)
  2. Clone repo — AGENTS.md is auto-loaded by Codex
  3. Create branch → edit → push → open PR
  4. Viktor reviews → CI applies → Slack notification
  5. Portal: https://k8s-portal.viktorbarzin.me/onboarding for full guide

Common Operations

  • Deploy new service: Use stacks/<existing-service>/ as template. Create stack, add DNS in tfvars, apply platform then service.
  • Fix crashed pods: Run healthcheck first. Safe to delete evicted/failed pods and CrashLoopBackOff pods with >10 restarts.
  • OOMKilled: Check kubectl describe limitrange tier-defaults -n <ns>. Increase resources.limits.memory in the stack's main.tf.
  • Add a secret: sops set secrets.sops.json '["key"]' '"value"' then commit.
  • NFS exports: Create dir on Proxmox host (ssh root@192.168.1.127 "mkdir -p /srv/nfs/<service>"), add to /etc/exports, run exportfs -ra.

Automated Service Upgrades

  • Pipeline: DIUN (detect) → n8n webhook (filter + rate limit) → HTTP POST → claude-agent-service (K8s) → claude -p (upgrade agent)
  • Agent: .claude/agents/service-upgrade.md — analyzes changelogs, backs up DBs, bumps versions, verifies health, rolls back on failure
  • Config: .claude/reference/upgrade-config.json — GitHub repo mappings, DB-backed services, skip patterns
  • Rate limit: Max 5 upgrades per 6h DIUN scan cycle (configured in n8n workflow)
  • Skipped: databases, :latest, custom images (viktorbarzin/*), infrastructure images
  • Risk: SAFE (2min verify) vs CAUTION (10min, DB backup, step through versions) based on changelog analysis
  • Docs: docs/architecture/automated-upgrades.md

Detailed Reference

See .claude/reference/patterns.md for: NFS volume code examples, iSCSI details, Kyverno governance tables, anti-AI scraping layers, Terragrunt architecture, node rebuild procedure, archived troubleshooting runbooks index.