viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
infra/stacks
History
Viktor Barzin 83079758bb monitoring(wave1): re-enable Loki+Alloy, deploy wave1 alert rules, add #security Slack lane 
## Loki + Alloy re-enabled (code-146x)
- Uncommented helm_release.loki, helm_release.alloy, kubernetes_daemon_set_v1.sysctl-inotify,
  kubernetes_config_map.loki_alert_rules, kubernetes_config_map.grafana_loki_datasource
- Reverses the documented "operational overhead vs benefit after node2 incident"
  decision. Re-evaluated because wave 1 security detection layer (beads code-8ywc)
  needs Loki + ruler + alert routing.
- SingleBinary mode, 2-4Gi mem, 50Gi proxmox-lvm PVC, 30-day retention, ruler enabled
  pointed at prometheus-alertmanager.monitoring.svc:9093
- Alloy DaemonSet (4 pods on worker nodes) discovers pod logs via K8s API + pushes
  to Loki
- Loki canaries running (4)
- Vault audit-tail sidecar logs now flowing to Loki: queried
  {namespace="vault",container="audit-tail"} returns live audit JSON

## Wave 1 alert rules deployed (W1.3 partial)
Added "Security Wave 1" rule group to loki_alert_rules configmap:
- V1: VaultRootTokenCreated — auth/token/create with policies=[root]
- V2: VaultAuditDeviceModified — sys/audit/* create/delete/update
- V3: VaultSealChanged — sys/seal update
- V4: VaultPolicyModified — sys/policies/acl/* create/update/delete
- V5: VaultAuthFailureSpike — >10 permission denied/min
- V7: VaultViktorFromUnexpectedIP — auth as me@viktorbarzin.me from non-allowlist source IP
  (allowlist: 10.0.20.0/22, 192.168.1.0/24, 10.10.0.0/16 pod, 10.96.0.0/12 svc, 100.64-127 tailnet)
- S1: PVEsshLoginFromUnexpectedIP — sshd "Accepted" from non-allowlist IP (rule defined,
  fires once promtail/Alloy ships sshd journal with job=sshd-pve)

Verified rules visible via /loki/api/v1/rules. K2-K9 (K8s API audit) deferred to W1.1
which needs the audit policy + apiserver log shipping codified.

## #security Slack lane (Alertmanager)
- New `slack-security` receiver in prometheus_chart_values.tpl, channel #security
- Higher-priority route at top of routes list: matchers `lane = security` →
  slack-security, continue: false (so wave 1 alerts never fall through to #alerts)
- Slack message format includes summary + description + runbook link annotation
- All wave 1 rules set `lane = "security"` label

## Resource summary
- 6 added: helm_release.loki, helm_release.alloy, kubernetes_config_map.grafana_loki_datasource,
  kubernetes_config_map.loki_alert_rules, kubernetes_daemon_set_v1.sysctl-inotify,
  + 1 other
- 5 changed: helm_release.prometheus (alertmanager config — new receiver + route),
  4 deployments (image tag drift from Keel-managed images, unrelated)
- 1 destroyed: null_resource grafana_admin_only_folder_acl["Finance (Personal)"]
  (timestamp-triggered always recreates — not destructive)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Closes: code-146x
2026-05-22 14:16:58 +00:00
..
_template ingress_factory: replace protected bool with auth enum + audit pass across 100 stacks 2026-05-22 14:16:42 +00:00
actualbudget recruiter-responder: bump image_tag to 189ef901 2026-05-22 14:16:49 +00:00
affine recruiter-responder: bump image_tag to 189ef901 2026-05-22 14:16:49 +00:00
authentik keel: enroll 11 more namespaces (operators + critical infra) 2026-05-22 14:16:56 +00:00
beads-server beads-server: codify Keel annotations on Dolt deployment (drift cleanup) 2026-05-22 14:16:57 +00:00
blog final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
broker-sync broker-sync(fidelity): un-suspend monthly CronJob 2026-05-22 14:16:56 +00:00
calico final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
changedetection enrolled-patch stacks: ignore image drift from Keel auto-update 2026-05-22 14:16:51 +00:00
chrome-service recruiter-responder: bump image_tag to 189ef901 2026-05-22 14:16:49 +00:00
city-guesser enrolled-patch stacks: ignore image drift from Keel auto-update 2026-05-22 14:16:51 +00:00
claude-agent-service recruiter-triage: AI culture & tooling section + warm-engage AI ask 2026-05-22 14:16:50 +00:00
claude-memory recruiter-responder: bump image_tag to 189ef901 2026-05-22 14:16:49 +00:00
cloudflared keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
cnpg [infra] Suppress Goldilocks vpa-update-mode label drift on all namespaces [ci skip] 2026-04-18 21:15:27 +00:00
coturn enrolled-patch stacks: ignore image drift from Keel auto-update 2026-05-22 14:16:51 +00:00
crowdsec keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
cyberchef final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
dashy enrolled-patch stacks: ignore image drift from Keel auto-update 2026-05-22 14:16:51 +00:00
dawarich enrolled-patch stacks: ignore image drift from Keel auto-update 2026-05-22 14:16:51 +00:00
dbaas kured + cnpg: drain-safe defaults ahead of Monday reboot wave 2026-05-22 14:16:48 +00:00
descheduler keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
diun enrolled-patch stacks: ignore image drift from Keel auto-update 2026-05-22 14:16:51 +00:00
ebook2audiobook enrolled-patch stacks: ignore image drift from Keel auto-update 2026-05-22 14:16:51 +00:00
ebooks enrolled-patch stacks: ignore image drift from Keel auto-update 2026-05-22 14:16:51 +00:00
echo enrolled-patch stacks: ignore image drift from Keel auto-update 2026-05-22 14:16:51 +00:00
excalidraw enrolled-patch stacks: ignore image drift from Keel auto-update 2026-05-22 14:16:51 +00:00
external-secrets recruiter-responder: bump image_tag to 189ef901 2026-05-22 14:16:49 +00:00
f1-stream final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
fire-planner ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
foolery recruiter-responder: bump image_tag to 189ef901 2026-05-22 14:16:49 +00:00
forgejo enrolled-patch stacks: ignore image drift from Keel auto-update 2026-05-22 14:16:51 +00:00
freedify recruiter-responder: bump image_tag to 189ef901 2026-05-22 14:16:49 +00:00
freshrss ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
frigate ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
grampsweb ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
hackmd ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
headscale keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
health ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
hermes-agent ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
homepage final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
immich final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
infra [forgejo] Phases 3+4+5: cutover, decommission, docs sweep 2026-05-07 23:29:34 +00:00
infra-maintenance [infra] Sweep dns_config ignore_changes across all pod-owning resources [ci skip] 2026-04-18 21:19:48 +00:00
insta2spotify ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
instagram-poster Bucket A retrigger + Bucket D enrollment (5 module-nested stacks) 2026-05-22 14:16:55 +00:00
isponsorblocktv ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
job-hunter ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
jsoncrack final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
k8s-dashboard final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
k8s-portal Bucket A retrigger + Bucket D enrollment (5 module-nested stacks) 2026-05-22 14:16:55 +00:00
k8s-version-upgrade k8s-version-upgrade: switch detection cron from weekly to daily 2026-05-22 14:16:57 +00:00
keel upgrade-state: skill + script + Keel scrape for periodic three-pipeline audit 2026-05-22 14:16:57 +00:00
kms final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
kured ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
kyverno security(wave1): W1.2 Vault XFF (applied) + W1.4/W1.5 Kyverno code prep (apply blocked on provider crash) 2026-05-22 14:16:57 +00:00
linkwarden ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
llama-cpp Bucket C: enroll 5 raw-deploy stacks in Keel auto-update 2026-05-22 14:16:55 +00:00
local-path final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
mailserver keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
matrix ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
meshcentral ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
metallb keel: enroll 11 more namespaces (operators + critical infra) 2026-05-22 14:16:56 +00:00
metrics-server keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
monitoring monitoring(wave1): re-enable Loki+Alloy, deploy wave1 alert rules, add #security Slack lane 2026-05-22 14:16:58 +00:00
n8n ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
navidrome ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
netbox ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
networking-toolbox ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
nextcloud ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
nfs-csi keel: enroll 11 more namespaces (operators + critical infra) 2026-05-22 14:16:56 +00:00
nodelocal-dns [dns] NodeLocal DNSCache — deploy DaemonSet to all nodes (WS C) 2026-04-19 15:46:41 +00:00
novelapp Woodpecker CI deploy [CI SKIP] 2026-05-22 14:16:55 +00:00
ntfy ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
nvidia nvidia: bump driver container memory limit 128Mi → 2Gi 2026-05-22 14:16:56 +00:00
onlyoffice ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
openclaw openclaw: native MCP servers + daily claude-memory sync 2026-05-22 14:16:53 +00:00
osm_routing final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
owntracks ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
paperless-mcp paperless-mcp: deploy MCP for AI document search 2026-05-22 14:16:56 +00:00
paperless-ngx ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
payslip-ingest ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
phpipam ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
platform [infra] Add Cloudflare provider to all stack lock files and generated providers 2026-04-16 16:31:36 +00:00
plotting-book Woodpecker CI deploy [CI SKIP] 2026-05-22 14:16:55 +00:00
poison-fountain ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
postiz Bucket A retrigger + Bucket D enrollment (5 module-nested stacks) 2026-05-22 14:16:55 +00:00
priority-pass ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
privatebin ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
proxmox-csi keel: enroll 11 more namespaces (operators + critical infra) 2026-05-22 14:16:56 +00:00
pvc-autoresizer [infra] Suppress Goldilocks vpa-update-mode label drift on all namespaces [ci skip] 2026-04-18 21:15:27 +00:00
rbac [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
real-estate-crawler realestate-crawler: dockerhub pull-secret + lift image-pin on ui/api 2026-05-22 14:16:57 +00:00
recruiter-responder recruiter-responder: bump image to 05b95943 (split callback routes) 2026-05-22 14:16:56 +00:00
redis keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
reloader keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
resume ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
reverse-proxy keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
rybbit ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
sealed-secrets keel: enroll 11 more namespaces (operators + critical infra) 2026-05-22 14:16:56 +00:00
send ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
servarr keel: enroll 11 more namespaces (operators + critical infra) 2026-05-22 14:16:56 +00:00
shadowsocks ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
speedtest ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
status-page [infra] Establish KYVERNO_LIFECYCLE_V1 drift-suppression convention [ci skip] 2026-04-18 14:15:51 +00:00
stirling-pdf ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
tandoor ci: retrigger v2 — apply pending Keel-enrolled stacks (#697 was cancelled by #698) 2026-05-22 14:16:53 +00:00
technitium keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
terminal terminal: probe + alerts after Traefik replica routing-table skew 2026-05-22 14:16:56 +00:00
tor-proxy ci: retrigger v3 — apply remaining 22 Keel-enrolled stacks 2026-05-22 14:16:54 +00:00
trading-bot final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
traefik keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
travel_blog final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
tuya-bridge ci: retrigger v3 — apply remaining 22 Keel-enrolled stacks 2026-05-22 14:16:54 +00:00
uptime-kuma Bucket A retrigger + Bucket D enrollment (5 module-nested stacks) 2026-05-22 14:16:55 +00:00
url ci: retrigger v3 — apply remaining 22 Keel-enrolled stacks 2026-05-22 14:16:54 +00:00
vault security(wave1): Vault audit-tail sidecar (live) + doc reality-check 2026-05-22 14:16:57 +00:00
vaultwarden Bucket A retrigger + Bucket D enrollment (5 module-nested stacks) 2026-05-22 14:16:55 +00:00
vpa keel: enroll 11 more namespaces (operators + critical infra) 2026-05-22 14:16:56 +00:00
wealthfolio Woodpecker CI deploy [CI SKIP] 2026-05-22 14:16:53 +00:00
webhook_handler final wave: enroll immich + status-page, retrigger 17 pending Bucket A 2026-05-22 14:16:55 +00:00
whisper ci: retrigger v3 — apply remaining 22 Keel-enrolled stacks 2026-05-22 14:16:54 +00:00
wireguard keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
woodpecker ci: retrigger v3 — apply remaining 22 Keel-enrolled stacks 2026-05-22 14:16:54 +00:00
xray keel: enroll 15 critical-path namespaces for digest-only auto-update 2026-05-22 14:16:56 +00:00
ytdlp ci: retrigger v3 — apply remaining 22 Keel-enrolled stacks 2026-05-22 14:16:54 +00:00