nfs-csi: pin chart v4.13.1 + controller affinity (post-mortem)

Keel rolled csi-driver-nfs 4.13.1→4.13.2 today. The 4.13.2 chart dropped
control-plane exclusion from the controller Deployment, so both replicas
landed on k8s-master, fought for hostNetwork ports 19809/29653, and one
went CrashLoopBackOff. Helm rollback left orphan containerd sandboxes
holding the ports — only a kubelet restart on master cleared them.

- Pin helm_release.version = "4.13.1" so terraform apply can't drift to
  the broken chart (defense in depth; nfs-csi namespace is already in the
  Kyverno-Keel exclude list)
- Add controller.affinity: podAntiAffinity between replicas +
  nodeAffinity excluding node-role.kubernetes.io/control-plane
- docs/post-mortems/2026-05-17-nfs-csi-keel-upgrade-master-port-conflict.md
  captures the root cause + recovery procedure (kubelet restart via
  nsenter is the escalation path when crictl rmp -f fails)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

stacks/nfs-csi/modules/nfs-csi/main.tf

@@ -23,10 +23,54 @@ resource "helm_release" "nfs_csi_driver" {
 
   repository = "https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts"
   chart      = "csi-driver-nfs"
+  # Pinned 2026-05-17. Keel polled and rolled csi-driver-nfs 4.13.1 → 4.13.2,
+  # which broke the cluster:
+  #   * Controller pods ended up on k8s-master because the new chart removed
+  #     control-plane exclusion from the default node selector.
+  #   * Two controller replicas on the same node fought over hostNetwork ports
+  #     19809 (node-driver-registrar) and 29653 (liveness-probe). One replica
+  #     CrashLoopBackOff'd with `bind: address already in use`.
+  #   * Rolling back live (helm rollback) left zombie containerd containers
+  #     holding the ports — only a kubelet restart cleared them.
+  # nfs-csi namespace is in the Kyverno keel exclude list (keel-annotations.tf)
+  # so Keel will not touch it again. This version pin is the second line of
+  # defense against accidental floating-version drift on `terraform apply`.
+  version = "4.13.1"
 
   values = [yamlencode({
     controller = {
       replicas = 2
+      # Required to coexist with the v4.13.1 chart on a 1-master + 4-worker
+      # cluster:
+      #   * podAntiAffinity forces the 2 controller replicas onto DIFFERENT
+      #     hosts (host network ports 19809/29653 are per-host).
+      #   * nodeAffinity excludes the control-plane node entirely so the
+      #     scheduler can't pick master when a worker is briefly NotReady.
+      # Without these, Kubernetes can schedule both replicas on the same node
+      # (port conflict) or on master itself (which already runs the DaemonSet
+      # pod and would conflict with it).
+      affinity = {
+        nodeAffinity = {
+          requiredDuringSchedulingIgnoredDuringExecution = {
+            nodeSelectorTerms = [{
+              matchExpressions = [{
+                key      = "node-role.kubernetes.io/control-plane"
+                operator = "DoesNotExist"
+              }]
+            }]
+          }
+        }
+        podAntiAffinity = {
+          requiredDuringSchedulingIgnoredDuringExecution = [{
+            labelSelector = {
+              matchLabels = {
+                app = "csi-nfs-controller"
+              }
+            }
+            topologyKey = "kubernetes.io/hostname"
+          }]
+        }
+      }
       livenessProbe = {
         httpPort = 29653
       }