apiserver: enable audit logging (low-write Metadata) + ship to Loki

Resource changes/deletions are now attributable (the novelapp deletion this week
was untraceable because apiserver audit was off). Low-write policy: drops
reads/noise, Metadata level on mutations, omitStages RequestReceived. Wired into
the kube-apiserver static-pod manifest + kubeadm-config (v1beta4
extraArgs/extraVolumes -> survives kubeadm upgrade) on k8s-master; Alloy tails
/var/log/kubernetes/audit/audit.log -> Loki {job=kubernetes-audit}.

Root cause that had silently blocked this AND OIDC for weeks: a stray
kube-apiserver.yaml.bak inside /etc/kubernetes/manifests/ was a duplicate
static-pod manifest kubelet ran instead of the real one, dropping every flag
added to the real manifest. Removed it. Runbook added.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

scripts/k8s-apiserver-audit-policy.yaml

@@ -0,0 +1,57 @@
+# kube-apiserver audit policy -- k8s-master (10.0.20.100), single control-plane.
+#
+# Goal: a durable "who/when/what" trail for MUTATIONS (create/update/patch/
+# delete) so resource deletions can be attributed even though direct
+# kubectl-to-apiserver calls otherwise leave no trace (see the 2026-06-06
+# novelapp incident: a dashboard delete was attributable, a direct-kubectl
+# recreate was not). Deployed OUTSIDE Terraform (the k8s VMs are not TF-managed,
+# see memory id=1575); this file is the source of truth, scp'd to
+# /etc/kubernetes/audit-policy.yaml and wired into the apiserver static-pod
+# manifest + the kubeadm-config ConfigMap (so "kubeadm upgrade" preserves it).
+#
+# Tuned for LOW WRITE VOLUME (the cluster's sdc HDD is write-sensitive, see
+# memory id=559): reads are dropped entirely, high-churn resources and probe
+# endpoints are dropped, and the verbose RequestReceived stage is omitted, so
+# only one Metadata-level line is written per mutating request.
+apiVersion: audit.k8s.io/v1
+kind: Policy
+# Only emit the post-execution stage -- halves volume vs logging both stages.
+omitStages:
+  - RequestReceived
+rules:
+  # 1. Never log read-only verbs -- the overwhelming majority of traffic and
+  #    irrelevant to "who changed/deleted X".
+  - level: None
+    verbs: ["get", "list", "watch"]
+
+  # 2. Drop high-churn / low-value resources even on writes.
+  - level: None
+    resources:
+      - group: ""
+        resources: ["events", "endpoints", "nodes/status", "pods/status"]
+      - group: "coordination.k8s.io"
+        resources: ["leases"]
+      - group: "discovery.k8s.io"
+        resources: ["endpointslices"]
+      - group: "metrics.k8s.io"
+      - group: "authentication.k8s.io"
+        resources: ["tokenreviews"]
+      - group: "authorization.k8s.io"
+        resources: ["subjectaccessreviews", "selfsubjectaccessreviews"]
+
+  # 3. Drop noisy non-resource probe / discovery URLs.
+  - level: None
+    nonResourceURLs:
+      - "/healthz*"
+      - "/readyz*"
+      - "/livez*"
+      - "/version"
+      - "/metrics"
+      - "/openapi/*"
+      - "/swagger*"
+
+  # 4. Everything else (every create/update/patch/delete on real resources):
+  #    record WHO (user + sourceIP + userAgent), WHAT (resource/namespace/name),
+  #    WHEN, and the verb -- at Metadata level (no request/response bodies, so
+  #    each entry stays small).
+  - level: Metadata