infra/docs/runbooks/apiserver-audit-logging.md

Runbook: kube-apiserver Audit Logging

Status: enabled 2026-06-06 on k8s-master (10.0.20.100, the single control-plane node). Motivated by the novelapp incident — a workload was deleted with no way to attribute it, because apiserver audit logging had never been on (see post-incident note below).

What is configured

• Audit policy: infra/scripts/k8s-apiserver-audit-policy.yaml (source of truth), deployed to /etc/kubernetes/audit-policy.yaml on k8s-master. Low-write by design: drops reads (get/list/watch), high-churn resources (events, leases, endpointslices, token/subjectaccess reviews), and probe URLs; logs everything else (create/update/patch/delete) at Metadata level (who/verb/resource/namespace/name/time/sourceIP — no bodies). omitStages: [RequestReceived] → one line per mutating request.
• kube-apiserver static-pod manifest (/etc/kubernetes/manifests/kube-apiserver.yaml): --audit-policy-file=/etc/kubernetes/audit-policy.yaml, --audit-log-path=/var/log/kubernetes/audit/audit.log, --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 (≤1 GB on disk, 30-day rotation), plus the audit-policy (File, RO) and audit-logs (DirectoryOrCreate) hostPath volumes/mounts.
• Persistence across kubeadm upgrade: the same flags + volumes are in the kubeadm-config ConfigMap (kube-system), ClusterConfiguration.apiServer.{extraArgs,extraVolumes} (v1beta4). Without this, a control-plane upgrade regenerates the manifest and silently drops audit (and oidc). The OIDC flags are recorded there too (see below).
• Shipping to Loki: the Alloy DaemonSet (infra/stacks/monitoring/modules/monitoring/alloy.yaml) tails /var/log/kubernetes/audit/audit.log (it schedules on the control-plane node and mounts host /var/log). Query in Loki/Grafana with {job="kubernetes-audit"}.

How to attribute a change ("who deleted X, when")

# In Loki (Grafana Explore or logcli), last 24h:
{job="kubernetes-audit"} |= "delete" |= "<resource-name>"

Each entry is a JSON audit.k8s.io/v1 Event: user.username, verb, objectRef.{resource,namespace,name}, requestReceivedTimestamp, sourceIPs, userAgent. On-node fallback (Loki down): sudo grep <name> /var/log/kubernetes/audit/audit.log on k8s-master.

Note: direct kubectl/dashboard calls now show the real identity (user SA or OIDC email). Pre-2026-06-06 deletions are NOT recoverable (audit was off).

CRITICAL gotcha that blocked this (and OIDC) for weeks

kubelet runs every non-dotfile in its staticPodPath (/etc/kubernetes/manifests) as a static pod. A stray kube-apiserver.yaml.bak.<epoch> left in that directory (from an earlier manual edit) was a second manifest defining pod kube-apiserver. kubelet ran the older .bak copy and ignored edits to the real kube-apiserver.yaml — so newly added flags (the OIDC flags, then these audit flags) never reached the running process even though the file clearly had them. Symptom: the running apiserver's /proc/<pid>/cmdline (or crictl inspect args) is SHORTER than the manifest's command: list. Fix: move any *.bak/backup OUT of /etc/kubernetes/manifests/. Always back up control-plane manifests to a sibling dir (e.g. /etc/kubernetes/), never inside manifests/. This also un-blocked OIDC (memory id=4042) as a side effect.

Rollback

Backups live in /etc/kubernetes/apiserver-manifest-archive/ on k8s-master (the 27-arg pre-audit known-good, and the 36-arg desired). To disable audit: remove the --audit-* flags + audit volumes from the manifest (kubelet restarts the apiserver in ~30-40s), and remove them from kubeadm-config. A bad manifest edit only needs the known-good copied back over /etc/kubernetes/manifests/kube-apiserver.yaml.

Editing the apiserver manifest restarts the apiserver → ~30-40s API blip on this single-control-plane cluster. Always edit from a backup + watch curl -sk https://10.0.20.100:6443/livez before declaring success.