Commit graph

1754 commits

Author SHA1 Message Date
Viktor Barzin
62d42657e6 state(redis): update encrypted state 2026-03-19 20:32:27 +00:00
Viktor Barzin
5be9f70a0d state(infra-maintenance): update encrypted state 2026-03-19 20:32:19 +00:00
Viktor Barzin
13759e58da state(redis): update encrypted state 2026-03-19 20:31:13 +00:00
Viktor Barzin
2511c1d78d state(infra-maintenance): update encrypted state 2026-03-19 20:30:50 +00:00
Viktor Barzin
414232cf5e state(redis): update encrypted state 2026-03-19 20:27:38 +00:00
Viktor Barzin
4680dd5fbc state(infra-maintenance): update encrypted state 2026-03-19 20:27:15 +00:00
Viktor Barzin
e54bc016ba reduce alert noise: raise memory thresholds, exclude claude-memory 4xx, right-size mysql-operator
- ContainerNearOOM: 85% → 90% (silences forgejo, changedetection, immich-pg, mysql-cluster)
- ClusterMemoryRequestsHigh: 85% → 92% (intentional overcommit)
- NodeMemoryPressureTrending: 85% → 92%
- HighService4xxRate: exclude claude-memory (401s from unauth requests are expected)
- mysql-operator memory limit: 512Mi → 580Mi (VPA upperBound 481Mi × 1.2)
2026-03-19 20:25:36 +00:00
Viktor Barzin
21bb3036af state(dbaas): update encrypted state 2026-03-19 20:23:59 +00:00
Viktor Barzin
67d1ce453c add /sentinel dir to cloud-init for kured reboot gating
The kured sentinel gate DaemonSet requires /sentinel to exist on
all nodes. Without it, kured pods get stuck in ContainerCreating
with hostPath mount failure. Previously created manually; now
provisioned automatically for new nodes.
2026-03-19 19:57:27 +00:00
Viktor Barzin
01eb9dd121 fix(monitoring): patch idrac-redfish-exporter to restore PSU voltage metric
Upstream v2.4.1 accidentally dropped idrac_power_supply_input_voltage from
the legacy RefreshPowerOld code path during a Huawei OEM support refactor.
Built a patched image that restores the single missing line:
  mc.NewPowerSupplyInputVoltage(ch, psu.LineInputVoltage, id)

Image: viktorbarzin/idrac-redfish-exporter:2.4.1-voltage-fix

[ci skip]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 13:37:14 +00:00
Viktor Barzin
b05421dbb5 add comment explaining prometheus 4Gi minimum memory requirement [ci skip] 2026-03-18 21:45:26 +00:00
Viktor Barzin
9d87ce605f revert prometheus memory 3Gi→4Gi: WAL tmpfs shares cgroup limit
The 2Gi WAL tmpfs (medium: Memory) counts against the container's
memory limit. At 3Gi, Prometheus OOM-kills during WAL replay on
startup (heap + tmpfs > 3Gi). Reverting to 4Gi restores headroom.
2026-03-18 21:44:14 +00:00
Viktor Barzin
03f55d969f state(vault): update encrypted state 2026-03-18 21:30:59 +00:00
Viktor Barzin
410c893647 fix(provision): security hardening from code review
- Add input validation: username regex + email format check in pipeline
- Quote variables in .provision-env to prevent shell injection
- Remove dead source command (each Woodpecker command is separate shell)
- Use jq to build JSON payloads (prevents injection via group names)
- Clean up git-crypt key on failure (use ; instead of &&)
- Add Kyverno ndots lifecycle ignore to webhook-handler deployment
2026-03-18 21:25:03 +00:00
Viktor Barzin
e51c063600 docs(add-user): update skill with actual working flow (no auto TF apply) 2026-03-18 00:28:46 +00:00
Viktor Barzin
82403a933c fix(provision): remove TF apply from pipeline, notify for manual apply
Vault stack can't be applied in CI (git-crypt TLS certs + sensitive
for_each on k8s_users). Pipeline now automates Vault KV update +
Authentik group creation, then notifies admin to apply stacks manually.
This matches the existing pattern — vault is not in default.yml either.
2026-03-18 00:23:06 +00:00
Viktor Barzin
d76b4b698f fix(provision): targeted vault apply + git-crypt in terragrunt step
- Two-pass vault apply: first target new user resources, then full apply
- Add git-crypt unlock to terragrunt step (TLS certs needed at plan time)
2026-03-18 00:19:16 +00:00
Viktor Barzin
6fad484126 fix(provision): reduce memory limit to 4Gi (LimitRange max) 2026-03-18 00:15:26 +00:00
Viktor Barzin
de6a5caecc fix(provision): merge terragrunt-apply into single shell block for env persistence 2026-03-18 00:11:14 +00:00
Viktor Barzin
7a24ff6702 fix(provision): use $USERNAME/$EMAIL directly — Woodpecker 3.x env vars
Woodpecker 3.x exposes pipeline variables with their original key names
(USERNAME, EMAIL), not CI_PIPELINE_VARIABLE_ prefix.
2026-03-18 00:04:51 +00:00
Viktor Barzin
52dc657af5 debug(provision): dump env vars to find correct variable names 2026-03-18 00:00:33 +00:00
Viktor Barzin
0a05343d86 fix(provision): use $VAR instead of ${VAR} to avoid Woodpecker interpolation
Woodpecker performs compile-time substitution on ${...} patterns,
replacing pipeline variables with empty strings. Using $VAR without
braces lets the shell evaluate them at runtime.
2026-03-17 23:58:46 +00:00
Viktor Barzin
fd130971aa feat(provision): automated user provisioning via Authentik webhook
- Expand CI Vault policy: write secret/data/platform + Transit SOPS keys
- Add Woodpecker provision-user.yml pipeline (manual event, API-triggered)
- Add env vars to webhook-handler deployment for Woodpecker/Authentik integration
- Update add-user skill with automated flow documentation
- Update Woodpecker repo ID list in CLAUDE.md
2026-03-17 23:56:30 +00:00
Viktor Barzin
82b9dd9e8a state(webhook_handler): update encrypted state 2026-03-17 23:52:32 +00:00
Viktor Barzin
5b29cfc73a state(vault): update encrypted state 2026-03-17 23:46:56 +00:00
Viktor Barzin
0fff155f17 feat(k8s-portal): update onboarding + architecture with SOPS state docs
Onboarding (namespace-owner):
- Add steps for sops/terragrunt install, state decrypt, apply workflow
- Add flow diagram showing auth → decrypt → apply → encrypt → push
- Add architecture overview with security model table
- Add access control callout explaining per-stack Transit keys

Architecture:
- Add secrets & state encryption section with ASCII diagrams
- Add request flow diagram (Cloudflare → Traefik → pods)
- Add CI/CD pipeline diagram (GHA → Woodpecker → K8s)

[ci skip]
2026-03-17 23:17:47 +00:00
Viktor Barzin
ccbcebb670 feat(vault): automate SOPS onboarding for namespace-owners
- Add Transit mount + per-stack Transit keys to vault stack TF
- Auto-create sops-user-<name> policy scoping decrypt to owned stacks
- Auto-create sops-<name> external group + alias for Authentik mapping
- Add sops-admin policy to authentik-admins group
- Attach sops-user policy to namespace-owner identity entities
- Update add-user skill with SOPS onboarding steps and Authentik group
- Adding a user to k8s_users + applying vault stack = full SOPS access

[ci skip]
2026-03-17 23:15:25 +00:00
Viktor Barzin
4d40c51a97 state(vault): update encrypted state 2026-03-17 23:14:24 +00:00
Viktor Barzin
7a8452e4c7 state(vault): update encrypted state 2026-03-17 23:14:16 +00:00
Viktor Barzin
0215d81622 state(vault): update encrypted state 2026-03-17 23:13:57 +00:00
Viktor Barzin
750cfcce7c state(vault): update encrypted state 2026-03-17 23:13:55 +00:00
Viktor Barzin
e54ad33315 state(vault): update encrypted state 2026-03-17 23:13:19 +00:00
Viktor Barzin
02d0291797 state(vault): update encrypted state 2026-03-17 23:12:58 +00:00
Viktor Barzin
468df3c5c4 state(vault): update encrypted state 2026-03-17 23:12:35 +00:00
Viktor Barzin
cf570c3d3b state(vault): update encrypted state 2026-03-17 23:12:03 +00:00
Viktor Barzin
4277b41c28 state(vault): update encrypted state 2026-03-17 23:11:55 +00:00
Viktor Barzin
77143dfd6b state: per-stack Transit keys for namespace-owner access control
- Each stack gets its own Vault Transit key (transit/keys/sops-state-<stack>)
- state-sync passes per-stack Transit URI + age keys on encrypt
- Vault policies scope namespace-owners to their stacks only:
  - sops-admin: wildcard access to all transit keys
  - sops-user-<name>: access only to owned stack keys
- Anca (plotting-book) can only decrypt plotting-book state
- Admin can decrypt everything (via admin Transit policy or age fallback)
- External group sops-plotting-book maps Authentik group to Vault policy
- Updated CLAUDE.md with state sync documentation
2026-03-17 23:08:18 +00:00
Viktor Barzin
6239e07dd5 docs: add plotting-book to GHA-migrated list and repo IDs [ci skip] 2026-03-17 23:07:32 +00:00
Viktor Barzin
4e7ca1ad61 state: add Vault Transit as primary SOPS backend, age as fallback
- .sops.yaml: add hc_vault_transit_uri for transit/keys/sops-state
- state-sync: try Vault Transit first, fall back to age key on disk
- Re-encrypted all 101 state files with both Vault Transit + age
- Normal workflow: vault login → decrypt via Transit (no key files)
- Bootstrap/DR: age key at ~/.config/sops/age/keys.txt
2026-03-17 22:56:33 +00:00
Viktor Barzin
9f80eb7ba0 state: add devvm as SOPS recipient
Add devvm age public key to .sops.yaml and re-encrypt all 101 state
files with both laptop and devvm keys.
2026-03-17 22:41:19 +00:00
Viktor Barzin
b6faa24349 state: add SOPS-encrypted terraform state to git
- SOPS + age encrypts all 101 .tfstate files (JSON-aware: keys visible, values encrypted)
- scripts/state-sync: encrypt/decrypt/commit wrapper
- scripts/tg: auto-decrypt before ops, auto-encrypt+commit after apply/destroy
- terragrunt.hcl: -backup=- prevents backup file accumulation
- .gitignore: track .tfstate.enc, ignore plaintext .tfstate
- Cleaned 964MB of stale backups (state/backups/, .backup files)
2026-03-17 22:37:56 +00:00
Viktor Barzin
12a51c4ffa right-size memory requests to unblock GPU workloads and fix dbaas quota [ci skip]
- nvidia: custom LimitRange (128Mi default, was 1Gi from Kyverno) to stop
  inflating GPU operator init containers; saves ~2.5Gi on GPU node
- nvidia: dcgm-exporter 1536Mi → 768Mi (actual usage 489Mi)
- monitoring: prometheus server 4Gi → 3Gi (actual usage 2.6Gi)
- onlyoffice: 2304Mi → 1536Mi (actual usage 1.3Gi)
- immich: frame explicit 64Mi resources (was getting 1Gi LimitRange default)
- dbaas: quota limits.memory 20Gi → 24Gi to fit 3rd MySQL replica

Root cause: Kyverno tier-2-gpu LimitRange injected 1Gi on every NVIDIA init
container (no explicit resources), wasting ~2.5Gi scheduling overhead on the
GPU node. Combined with over-requesting, frigate and immich-ml couldn't schedule.
2026-03-17 22:35:54 +00:00
Viktor Barzin
73511b1230 extract remaining 19 modules from platform, complete stack split [ci skip]
Phase 3: all 27 platform modules now run as independent stacks.
Platform reduced to empty shell (outputs only) for backward compat
with 72 app stacks that declare dependency "platform".
Fixed technitium cross-module dashboard reference by copying file.
Woodpecker pipeline applies all 27+1 stacks in parallel via loop.
All applied with zero destroys.
2026-03-17 21:42:16 +00:00
Viktor Barzin
ae36dc253b extract monitoring, nvidia, mailserver, cloudflared, kyverno from platform [ci skip]
Phase 2 of platform stack split. 5 more modules extracted into
independent stacks. All applied successfully with zero destroys.
Cloudflared now reads k8s_users from Vault directly to compute
user_domains. Woodpecker pipeline runs all 8 extracted stacks
in parallel. Memory bumped to 6Gi for 9 concurrent TF processes.
Platform reduced from 27 to 19 modules.
2026-03-17 21:34:11 +00:00
Viktor Barzin
3c804aedf8 extract dbaas, authentik, crowdsec from platform into independent stacks [ci skip]
Phase 1 of platform stack split for parallel CI applies.
All 3 modules were fully independent (no cross-module refs).
State migrated via terraform state mv. All 3 stacks applied
with zero changes (dbaas had pre-existing ResourceQuota drift).
Woodpecker pipeline updated to run extracted stacks in parallel.
2026-03-17 18:11:53 +00:00
Viktor Barzin
c8b42f78df fix DB password rotation desync in 5 stacks
Vault DB engine rotates passwords weekly but 5 stacks baked passwords
at Terraform plan time, causing stale credentials until next apply.

- real-estate-crawler: add vault-database ESO, use secret_key_ref in 3 deployments
- nextcloud: switch Helm chart to existingSecret for DB password
- grafana: add vault-database ESO, use envFromSecrets in Helm values
- woodpecker: use extraSecretNamesForEnvFrom, remove plan-time data source chain
- affine: add vault-database ESO, use secret_key_ref in deployment + init container
2026-03-17 07:39:29 +00:00
Viktor Barzin
8d8c8db737 increase DB password rotation from 24h to weekly (604800s) 2026-03-16 23:17:01 +00:00
Viktor Barzin
c31ba2c50c k8s-portal: use Recreate strategy, limit revision history to 3
Prevents stale pods serving old content during rapid successive deploys.
With 1 replica + RollingUpdate, old and new pods briefly coexist.
2026-03-16 22:55:15 +00:00
Viktor Barzin
6cc4d526f1 add GitHub Pages for post-mortems
- Index page listing all incident reports
- GHA workflow deploys post-mortems/ on push
- Available at viktorbarzin.github.io/infra/
2026-03-16 22:16:05 +00:00
Viktor Barzin
fb66676d7b post-mortem: kured + containerd cascade outage — alerts + report
26h outage caused by unattended-upgrades kernel update → kured reboot →
containerd overlayfs snapshotter corruption → image pull failures →
calico down → cascading cluster outage.

Remediation:
- Add "Node Runtime Health" Prometheus alert group (6 alerts):
  KubeletImagePullErrors, KubeletPLEGUnhealthy, PodsStuckContainerCreating,
  KubeletRuntimeOperationsLatency, KubeletRunningContainersDrop, CalicoNodeNotReady
- Add containerd cascade inhibition rule
- Save post-mortem report as HTML in post-mortems/

Also applied via kubectl (needs Terraform codification):
- Sentinel gate DaemonSet gating kured reboots on cluster health
- Fixed kured Helm values: reboot window + gated sentinel path
2026-03-16 22:06:10 +00:00