security(wave1): W1.1 audit-log shipping LIVE + W1.5 trusted-registries Enforce LIVE

## W1.1 — K8s API audit log shipping (LIVE)
- alloy.yaml: added control-plane toleration so Alloy DaemonSet runs on
  k8s-master node. Verified alloy-7zg7t scheduled on master, tailing
  /var/log/kubernetes/audit.log
- loki.tf "Security Wave 1" rule group: added K2-K9 alert rules
  (skipped K1 per Q7 decision):
  - K2 K8sSATokenFromUnexpectedIP
  - K3 K8sSensitiveSecretReadByUnexpectedActor
  - K4 K8sExecIntoSensitiveNamespace
  - K5 K8sMassDelete (>5 Pod/Secret/CM in 60s by single user)
  - K6 K8sAuditPolicyModified (kubeadm-config CM change)
  - K7 K8sClusterRoleWildcardCreated (verbs=* + resources=*)
  - K8 K8sAnonymousBindingGranted
  - K9 K8sViktorFromUnexpectedIP
- All rules use source-IP regex matching the wave-1 allowlist
  (10.0.20.0/22, 192.168.1.0/24, 10.10.0.0/16 pod, 10.96.0.0/12 svc,
  100.64-127 tailnet) and `lane = "security"` → #security Slack route.
- Verified: kubectl-audit logs flowing in Loki query
  {job="kubernetes-audit"} returns events with node=k8s-master.
- Verified: /loki/api/v1/rules lists all K2-K9 + V1-V7 + S1.

## W1.5 — require-trusted-registries Enforce (LIVE)
- security-policies.tf: flipped Audit→Enforce with explicit allowlist
  built by `kubectl get pods -A -o jsonpath='{..image}'` enumeration.
- Removed `*/*` catch-all (which made Audit→Enforce a no-op).
- Pattern includes 15 explicit registries, 6 DockerHub library bare
  names, 56 DockerHub user repos.
- Verified by admission dry-run:
  - evilcorp.example/malware:v1 → BLOCKED with custom message
  - alpine:3.20 → ALLOWED (matches `alpine*`)
  - docker.io/library/alpine:3.20 → ALLOWED (matches `docker.io/*`)

## W1.6 — Calico flow logs (BLOCKED — Calico OSS limitation)
- Tried adding FelixConfiguration with flowLogsFileEnabled=true via
  kubectl_manifest in stacks/calico/main.tf
- Calico OSS rejected with "strict decoding error: unknown field
  spec.flowLogsFileEnabled" — these fields are Calico Enterprise/Tigera-only
- Removed the failed resource. Documented alternative paths in main.tf
  comment block: GNP with action=Log (iptables NFLOG → journal), Cilium
  migration, eBPF tooling, or Tigera Operator adoption.

## Docs updates
- security.md status table refreshed: W1.1/W1.2/W1.3/W1.4/W1.5 LIVE,
  W1.6/W1.7 blocked
- monitoring.md: Loki marked DEPLOYED (was incorrectly NOT-DEPLOYED in
  prior session before today's apply)

## Cleanup
- Removed stacks/kyverno/imports.tf (TF 1.5+ import blocks completed
  their job in the 2026-05-18 apply; should not stay in tree per TF docs)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-05-19 06:37:54 +00:00 committed by Viktor Barzin
parent 51365937b1
commit a048b37f60
7 changed files with 166 additions and 88 deletions

View file

@ -57,7 +57,7 @@ graph TB
|-----------|---------|----------|---------|
| Prometheus | Latest (Diun monitored) | `stacks/monitoring/modules/monitoring/` | Metrics collection and storage, scrape configs for all services |
| Grafana | Latest (Diun monitored) | `stacks/monitoring/modules/monitoring/` | Visualization, 14+ dashboards (API server, CoreDNS, GPU, UPS, etc.) |
| Loki | **NOT DEPLOYED as of 2026-05-18** — TF code exists in `stacks/monitoring/modules/monitoring/loki.tf` but `helm_release.loki` has a self-referencing `depends_on` that prevented apply. No `loki` Helm release in cluster, no Loki pods or Service. All "log aggregation" claims below are aspirational. Tracked under beads `code-146x`. | `stacks/monitoring/modules/monitoring/` | Log aggregation and querying (planned) |
| Loki | **DEPLOYED 2026-05-18** (SingleBinary mode, 30d retention, 50Gi PVC on `proxmox-lvm`, ruler enabled → Alertmanager). Re-enabled from previous "operational overhead" disable. Ships logs via Alloy DaemonSet (now on all nodes including master after 2026-05-19 toleration add). | `stacks/monitoring/modules/monitoring/` | Log aggregation and querying |
| Alertmanager | Latest (Diun monitored) | `stacks/monitoring/modules/monitoring/` | Alert routing with cascade inhibitions |
| Uptime Kuma | Latest (Diun monitored) | `stacks/uptime-kuma/` | Internal + external HTTP monitors, status page |
| External Monitor Sync | Python 3.12 | `stacks/uptime-kuma/` | CronJob (10min) syncs `[External]` monitors from `cloudflare_proxied_names` |

View file

@ -175,13 +175,13 @@ Beads epic: `code-8ywc`. **Status: partially live as of 2026-05-18.**
|---|---|
| W1.2 Vault `file` audit device | **LIVE**`vault_audit.file` in `stacks/vault/main.tf:287`, writing to `/vault/audit/vault-audit.log` on `proxmox-lvm-encrypted` PVC |
| W1.2 Vault `x_forwarded_for_authorized_addrs = 10.10.0.0/16` | **LIVE** — applied via `tg apply -target=helm_release.vault` on 2026-05-18; all 3 vault pods restarted cleanly |
| W1.2 Vault audit log shipping to Loki | **PARTIAL** — `audit-tail` sidecar live in vault pods (emits JSON audit lines to stdout, viewable via `kubectl logs -n vault vault-X -c audit-tail`). Actual shipping to Loki BLOCKED on `code-146x` (Loki not deployed in the cluster despite TF code existing). |
| W1.1 K8s API audit policy | **PENDING** — needs `stacks/infra` kubeadm-config templating |
| W1.3 Source-IP anomaly rules (K9, V7, S1) | **PENDING** — gated on `code-146x` (Loki + Alloy not deployed) and W1.1 audit-policy codification |
| W1.4 Kyverno security policies → Enforce | **CODE READY, APPLY BLOCKED** by `code-e2dp` (terraform-provider-kubernetes v3.1.0 crash on `kubernetes_manifest` plan) |
| W1.5 Kyverno trusted-registries enforce | **CODE PARTIAL** (exclude list added; allowlist tightening + enforce-flip deferred until `code-e2dp` resolved) |
| W1.6 Calico flow logs + log-only GNP | **BLOCKED** on `code-3ad` (Calico stack adopts only namespaces today; `Installation` CR + Felix config not under TF) |
| W1.7 NetworkPolicy phased enforce | **BLOCKED** on W1.6 observation window |
| W1.2 Vault audit log shipping to Loki | **LIVE** — `audit-tail` sidecar in vault pods + Alloy DaemonSet ships to Loki with `container="audit-tail"`. Verified via `{namespace="vault",container="audit-tail"}` LogQL query. |
| W1.1 K8s API audit policy + shipping | **LIVE** — kube-apiserver audit policy was already configured (Metadata level, `/var/log/kubernetes/audit.log`, 7d retention). Alloy DaemonSet now tolerates control-plane taint, scrapes the audit log file, ships to Loki with `job=kubernetes-audit`. K2-K9 alert rules in Loki ruler. |
| W1.3 Source-IP anomaly rules (K9, V7, S1) | **LIVE** (K9, V7); **S1 PENDING** — fires once promtail/Alloy on PVE host ships sshd journal with `job=sshd-pve`. |
| W1.4 Kyverno security policies → Enforce | **LIVE** — 3 policies in Enforce mode with 35-namespace exclude list. |
| W1.5 Kyverno trusted-registries → Enforce | **LIVE** — explicit allowlist (15 registries + 6 DockerHub library bare names + 56 DockerHub user repos). Verified by admission dry-run: `evilcorp.example/malware:v1` BLOCKED, `alpine:3.20` and `docker.io/library/alpine:3.20` ALLOWED. |
| W1.6 Calico flow logs + log-only GNP | **BLOCKED** — Calico OSS doesn't support `FelixConfiguration.flowLogsFileEnabled` (Calico Enterprise/Tigera-only, rejected 2026-05-19 with "strict decoding error"). Alternative paths: Calico GlobalNetworkPolicy `action: Log` → iptables NFLOG → node journal, OR Cilium migration, OR Tigera Operator adoption. See stacks/calico/main.tf comment block. |
| W1.7 NetworkPolicy phased enforce | **BLOCKED** on W1.6 observation-method decision |
The block below documents the locked design.

View file

@ -69,6 +69,22 @@ resource "kubernetes_namespace" "tigera_operator" {
}
}
# Wave 1 W1.6 (beads code-8ywc): Calico OSS does NOT support flow-log-to-file
# export via FelixConfiguration `flowLogsFileEnabled` and related fields are
# Calico Enterprise / Tigera Cloud features and are rejected by the OSS API
# (verified 2026-05-19: "strict decoding error: unknown field spec.flowLogsFileEnabled").
#
# Alternative observe-then-enforce paths for W1.6/W1.7:
# 1. Calico GlobalNetworkPolicy with `action: Log` on tier 3+4 Log action
# writes to iptables NFLOG which lands in node syslog. Alloy already
# scrapes journal, but the format needs parsing.
# 2. Cilium replacement with Hubble flow observability (large migration).
# 3. Tigera Operator + Calico Enterprise (commercial).
# 4. eBPF-based flow capture (e.g. inspektor-gadget, retina) sidecar approach.
#
# Wave 1 stops at this fork. The observe phase requires further design choice
# tracked under code-8ywc as a separate W1.6/W1.7 follow-up.
# CI retrigger 2026-05-16T13:42:57+00:00 bulk enrollment apply (pipeline #689 killed)
# CI retrigger v2 2026-05-16T13:46:35+00:00

View file

@ -1,72 +0,0 @@
# Import existing live Kyverno resources into kubectl_manifest state.
# Created during code-e2dp fix (kubernetes_manifest kubectl_manifest swap).
# Once applied successfully, these import blocks can be deleted in a cleanup commit.
import {
to = module.kyverno.kubectl_manifest.cleanup_failed_pods
id = "kyverno.io/v2beta1//ClusterCleanupPolicy//cleanup-failed-pods"
}
import {
to = module.kyverno.kubectl_manifest.generate_limitrange_by_tier
id = "kyverno.io/v1//ClusterPolicy//generate-limitrange-by-tier"
}
import {
to = module.kyverno.kubectl_manifest.generate_resourcequota_by_tier
id = "kyverno.io/v1//ClusterPolicy//generate-resourcequota-by-tier"
}
import {
to = module.kyverno.kubectl_manifest.inject_dependency_init_containers
id = "kyverno.io/v1//ClusterPolicy//inject-dependency-init-containers"
}
import {
to = module.kyverno.kubectl_manifest.mutate_gpu_priority
id = "kyverno.io/v1//ClusterPolicy//mutate-gpu-priority"
}
import {
to = module.kyverno.kubectl_manifest.mutate_ndots
id = "kyverno.io/v1//ClusterPolicy//mutate-ndots"
}
import {
to = module.kyverno.kubectl_manifest.mutate_priority_from_tier
id = "kyverno.io/v1//ClusterPolicy//mutate-priority-from-tier"
}
import {
to = module.kyverno.kubectl_manifest.mutate_strip_cpu_limits
id = "kyverno.io/v1//ClusterPolicy//mutate-strip-cpu-limits"
}
import {
to = module.kyverno.kubectl_manifest.mutate_tier_from_namespace
id = "kyverno.io/v1//ClusterPolicy//mutate-tier-from-namespace"
}
import {
to = module.kyverno.kubectl_manifest.policy_deny_host_namespaces
id = "kyverno.io/v1//ClusterPolicy//deny-host-namespaces"
}
import {
to = module.kyverno.kubectl_manifest.policy_deny_privileged
id = "kyverno.io/v1//ClusterPolicy//deny-privileged-containers"
}
import {
to = module.kyverno.kubectl_manifest.policy_inject_keel_annotations
id = "kyverno.io/v1//ClusterPolicy//inject-keel-annotations"
}
import {
to = module.kyverno.kubectl_manifest.policy_require_trusted_registries
id = "kyverno.io/v1//ClusterPolicy//require-trusted-registries"
}
import {
to = module.kyverno.kubectl_manifest.policy_restrict_capabilities
id = "kyverno.io/v1//ClusterPolicy//restrict-sys-admin"
}
import {
to = module.kyverno.kubectl_manifest.policy_set_image_pull_policy
id = "kyverno.io/v1//ClusterPolicy//set-image-pull-policy"
}
import {
to = module.kyverno.kubectl_manifest.sync_registry_credentials
id = "kyverno.io/v1//ClusterPolicy//sync-registry-credentials"
}
import {
to = module.kyverno.kubectl_manifest.sync_tls_secret
id = "kyverno.io/v1//ClusterPolicy//sync-tls-secret"
}

View file

@ -290,12 +290,13 @@ resource "kubectl_manifest" "policy_require_trusted_registries" {
}
}
spec = {
# NOTE: Stays in Audit mode pending allowlist tightening. The current
# pattern includes `*/*` which matches any image with a registry flipping
# to Enforce would not actually restrict supply chain. Tightening the
# allowlist to a precise enumeration of in-use registries is tracked
# separately under beads code-8ywc (W1.5 follow-up).
validationFailureAction = "Audit"
# Wave 1 W1.5: flipped Audit Enforce 2026-05-19 with explicit allowlist.
# Allowlist enumerated from `kubectl get pods -A -o jsonpath='{..image}'`
# on 2026-05-18; covers all in-cluster image sources. Update on adding new
# workloads from a registry NOT in this list (and ask if the new registry
# is trusted before opening it). The `*/*` catch-all was deliberately
# removed so unknown registries fail closed at admission.
validationFailureAction = "Enforce"
background = true
rules = [{
name = "validate-registries"
@ -314,11 +315,39 @@ resource "kubectl_manifest" "policy_require_trusted_registries" {
}]
}
validate = {
message = "Images must be from trusted registries (docker.io, ghcr.io, quay.io, registry.k8s.io, or local cache)."
message = "Images must be from trusted registries. Allowlist defined in stacks/kyverno/modules/kyverno/security-policies.tf — add the new registry there if intentional, otherwise switch the workload to a trusted source."
pattern = {
spec = {
containers = [{
image = "docker.io/* | ghcr.io/* | quay.io/* | registry.k8s.io/* | 10.0.20.10* | */*"
image = join(" | ", [
# Explicit registries
"docker.io/*", "ghcr.io/*", "quay.io/*", "registry.k8s.io/*",
"gcr.io/*", "us-docker.pkg.dev/*", "lscr.io/*",
"codeberg.org/*", "mcr.microsoft.com/*", "nvcr.io/*",
"oci.external-secrets.io/*", "reg.kyverno.io/*",
"docker.n8n.io/*", "registry.gitlab.com/*",
# Private
"forgejo.viktorbarzin.me/*", "10.0.20.10*",
# DockerHub library (bare image names without slash)
"alpine*", "busybox*", "kong*", "mysql*", "nginx*", "python*",
# DockerHub user repos (no registry prefix, has slash)
# enumerated from current cluster state.
"actualbudget/*", "afadil/*", "binwiederhier/*", "bitnami/*",
"clickhouse/*", "cloudflare/*", "coturn/*", "crowdsecurity/*",
"curlimages/*", "deluan/*", "dgtlmoon/*", "dolthub/*",
"dpage/*", "dperson/*", "edoburu/*", "esanchezm/*",
"freikin/*", "freshrss/*", "hackmdio/*", "hashicorp/*",
"headscale/*", "jhonderson/*", "kebe/*", "library/*",
"lissy93/*", "louislam/*", "matrixdotorg/*", "mendhak/*",
"mghee/*", "mindflavor/*", "mpepping/*", "netsampler/*",
"nvidia/*", "onlyoffice/*", "openresty/*", "owntracks/*",
"phpipam/*", "phpmyadmin/*", "privatebin/*", "prom/*",
"prompve/*", "rancher/*", "roundcube/*", "sclevine/*",
"shadowsocks/*", "shlinkio/*", "stirlingtools/*",
"technitium/*", "teddysun/*", "temporalio/*",
"typhonragewind/*", "tzahi12345/*", "vabene1111/*",
"vaultwarden/*", "viktorbarzin/*", "viren070/*", "zelest/*",
])
}]
}
}

View file

@ -195,6 +195,14 @@ controller:
path: /etc/machine-id
type: File
# Schedule on control-plane node too so we can tail /var/log/kubernetes/audit.log
# from kube-apiserver. Without this, K8s audit log shipping (wave 1 K2-K9 alert
# rules) has no source. control-plane has the standard NoSchedule taint.
tolerations:
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"
# Resource limits for DaemonSet pods
# Alloy tails logs from all containers on the node via K8s API and batches
# them to Loki. Memory scales with number of active log streams (~30-50 per node).

View file

@ -258,6 +258,103 @@ resource "kubernetes_config_map" "loki_alert_rules" {
runbook = "docs/runbooks/security-incident.md#v7-viktors-vault-identity-from-unexpected-source-ip"
}
},
# K2: ServiceAccount token used from outside cluster.
# Allowlist = pod CIDR + LAN + Headscale tailnet. Anything else =
# likely stolen SA token used externally.
{
alert = "K8sSATokenFromUnexpectedIP"
expr = "sum(count_over_time({job=\"kubernetes-audit\"} | json | user_username=~\"system:serviceaccount:.+\" | sourceIPs_0!~\"^(10\\\\.0\\\\.2[0-3]\\\\.|192\\\\.168\\\\.1\\\\.|10\\\\.10\\\\.|10\\\\.(9[6-9]|1[01][0-9]|111)\\\\.|100\\\\.(6[4-9]|[7-9][0-9]|1[01][0-9]|12[0-7])\\\\.).*\" [5m])) > 0"
for = "0m"
labels = { severity = "critical", lane = "security" }
annotations = {
summary = "K8s ServiceAccount token used from non-allowlist source IP — possible stolen SA token"
runbook = "docs/runbooks/security-incident.md#k2-serviceaccount-token-used-from-outside-cluster"
}
},
# K3: Secret read in sensitive namespace by unexpected actor.
# Allowlisted readers: ESO controller, sealed-secrets controller,
# Vault SA, me@viktorbarzin.me. Anyone else = alert.
{
alert = "K8sSensitiveSecretReadByUnexpectedActor"
expr = "sum(count_over_time({job=\"kubernetes-audit\"} | json | verb=~\"get|list\" | objectRef_resource=\"secrets\" | objectRef_namespace=~\"vault|sealed-secrets|external-secrets\" | user_username!~\"^(me@viktorbarzin\\\\.me|system:serviceaccount:external-secrets:.+|system:serviceaccount:sealed-secrets:.+|system:serviceaccount:vault:.+)$\" [5m])) > 0"
for = "0m"
labels = { severity = "critical", lane = "security" }
annotations = {
summary = "Sensitive Secret read in vault/sealed-secrets/external-secrets by non-allowlisted actor"
runbook = "docs/runbooks/security-incident.md#k3-secret-read-in-sensitive-namespace-by-unexpected-actor"
}
},
# K4: Exec into pod in sensitive namespace.
{
alert = "K8sExecIntoSensitiveNamespace"
expr = "sum(count_over_time({job=\"kubernetes-audit\"} | json | verb=\"create\" | objectRef_resource=\"pods\" | objectRef_subresource=\"exec\" | objectRef_namespace=~\"vault|kube-system|dbaas|cnpg-system\" | user_username!=\"me@viktorbarzin.me\" [5m])) > 0"
for = "0m"
labels = { severity = "warning", lane = "security" }
annotations = {
summary = "kubectl exec into sensitive namespace (vault/kube-system/dbaas/cnpg-system) by non-Viktor actor"
runbook = "docs/runbooks/security-incident.md#k4-exec-into-sensitive-pod"
}
},
# K5: Mass delete of pods/secrets/configmaps in 60s by single actor.
{
alert = "K8sMassDelete"
expr = "sum by (user_username) (count_over_time({job=\"kubernetes-audit\"} | json | verb=\"delete\" | objectRef_resource=~\"pods|secrets|configmaps\" [1m])) > 5"
for = "1m"
labels = { severity = "critical", lane = "security" }
annotations = {
summary = "Mass delete (>5 Pod/Secret/ConfigMap in 60s) by {{ $labels.user_username }}"
runbook = "docs/runbooks/security-incident.md#k5-mass-delete"
}
},
# K6: Audit policy or audit-log path modified attacker silencing
# visibility. The audit policy file is /etc/kubernetes/policies/audit-policy.yaml
# on master; changes go via kubeadm reconfig. Detect via API access
# to apiserver kubeadm-config ConfigMap.
{
alert = "K8sAuditPolicyModified"
expr = "sum(count_over_time({job=\"kubernetes-audit\"} | json | verb=~\"update|patch\" | objectRef_resource=\"configmaps\" | objectRef_name=\"kubeadm-config\" | objectRef_namespace=\"kube-system\" [5m])) > 0"
for = "0m"
labels = { severity = "critical", lane = "security" }
annotations = {
summary = "kubeadm-config ConfigMap modified — could be audit policy change"
runbook = "docs/runbooks/security-incident.md#k6-audit-policy-modified"
}
},
# K7: New ClusterRole created with verbs=* and resources=*.
# Allowlist excludes calico-system, kyverno, nvidia, etc. which legitimately
# create such ClusterRoles via Helm.
{
alert = "K8sClusterRoleWildcardCreated"
expr = "sum(count_over_time({job=\"kubernetes-audit\"} | json | verb=\"create\" | objectRef_resource=\"clusterroles\" |~ \"\\\"verbs\\\":\\\\[\\\"\\\\*\\\"\\\\]\" |~ \"\\\"resources\\\":\\\\[\\\"\\\\*\\\"\\\\]\" [5m])) > 0"
for = "0m"
labels = { severity = "warning", lane = "security" }
annotations = {
summary = "New ClusterRole with verbs=[*]+resources=[*] created — privilege escalation primitive"
runbook = "docs/runbooks/security-incident.md#k7-new-clusterrole-with-full-wildcards"
}
},
# K8: Anonymous binding granted catastrophic.
{
alert = "K8sAnonymousBindingGranted"
expr = "sum(count_over_time({job=\"kubernetes-audit\"} | json | verb=\"create\" | objectRef_resource=~\"rolebindings|clusterrolebindings\" |~ \"system:(anonymous|unauthenticated)\" [5m])) > 0"
for = "0m"
labels = { severity = "critical", lane = "security" }
annotations = {
summary = "Binding granted to system:anonymous or system:unauthenticated — full cluster compromise risk"
runbook = "docs/runbooks/security-incident.md#k8-anonymous-binding"
}
},
# K9: Viktor's identity from non-allowlist source IP. Same regex as V7.
{
alert = "K8sViktorFromUnexpectedIP"
expr = "sum(count_over_time({job=\"kubernetes-audit\"} | json | user_username=\"me@viktorbarzin.me\" | sourceIPs_0!~\"^(10\\\\.0\\\\.2[0-3]\\\\.|192\\\\.168\\\\.1\\\\.|10\\\\.10\\\\.|10\\\\.(9[6-9]|1[01][0-9]|111)\\\\.|100\\\\.(6[4-9]|[7-9][0-9]|1[01][0-9]|12[0-7])\\\\.).*\" [5m])) > 0"
for = "0m"
labels = { severity = "critical", lane = "security" }
annotations = {
summary = "K8s API request as me@viktorbarzin.me from non-allowlist source IP — possible stolen kubeconfig/OIDC token"
runbook = "docs/runbooks/security-incident.md#k9-viktors-identity-from-unexpected-source-ip"
}
},
# S1: PVE sshd auth success from non-allowlist IP.
# Conditional on the pve-sshd promtail unit being live on PVE host
# (deployed via stacks/infra/scripts out of scope until W1.3 host