viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

vault: move audit-PVC autoresizer annotations to kubernetes_annotations

Browse source 
Background: 2026-05-10 someone added `server.auditStorage.annotations`
to vault/main.tf attempting to enable pvc-autoresizer on audit-vault-N
PVCs. The vault helm chart maps that block into the StatefulSet's
volumeClaimTemplates, which is immutable post-creation on existing
StatefulSets. Result: 4 consecutive helm upgrade attempts (rev 16-19)
all rejected with "StatefulSet spec: Forbidden", leaving the release
stuck in failed state since 22:47 UTC that day. Live PVCs were
hand-annotated via `kubectl annotate` as a workaround, but the IaC
declared a path that couldn't be applied — every subsequent tg apply
on the vault stack would re-fail.

Fix:
  * Remove `annotations` block from `server.auditStorage` values
    (with a comment recording why it can't live there).
  * Add `kubernetes_annotations` resources for audit-vault-{0,1,2}
    with `force = true`, so Terraform adopts the existing annotations
    and tracks the desired-state in IaC going forward. The autoresizer
    cares about PVC annotations, not StatefulSet template annotations,
    so this is functionally equivalent.

Done out-of-band before commit (helm state was already corrupted):
  `helm rollback vault 15 -n vault` → revision 20 deployed (clean).

Verified: helm status vault = deployed; audit-vault-0 still has
threshold=10% storage_limit=10Gi annotations; cluster healthcheck
no longer reports vault/vault=failed.
This commit is contained in:
Viktor Barzin 2026-05-11 19:41:35 +00:00
parent 18a17891c4
commit a699d5bedf
1 changed files with 30 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

38
stacks/vault/main.tf
View file

@ -63,14 +63,13 @@ resource "helm_release" "vault" {
enabled = true enabled = true
size = "2Gi" size = "2Gi"
storageClass = "proxmox-lvm-encrypted" # Migrated 2026-04-25 from nfs-proxmox storageClass = "proxmox-lvm-encrypted" # Migrated 2026-04-25 from nfs-proxmox
# Vault audit logs grow unbounded per request; let pvc-autoresizer # Note: pvc-autoresizer annotations on audit-vault-{0,1,2} are
# expand the volume up to 10Gi rather than ride a stuck-Pending # NOT declared here. The chart maps `annotations` into the
# vault-0 the moment the PVC fills. # StatefulSet's volumeClaimTemplates, which is immutable
annotations = { # post-creation — every helm upgrade with this block set fails
"resize.topolvm.io/threshold" = "10%" # with "StatefulSet spec: Forbidden" (rev 16-19 on 2026-05-10).
"resize.topolvm.io/increase" = "100%" # Instead, the annotations are applied directly to the live
"resize.topolvm.io/storage_limit" = "10Gi" # PVCs via the kubernetes_annotations resources below.
}
} }
standalone = { enabled = false } standalone = { enabled = false }
@ -166,6 +165,29 @@ resource "helm_release" "vault" {
})] })]
} }
# pvc-autoresizer annotations on the audit PVCs. Applied here (not via
# the chart's `server.auditStorage.annotations`) because StatefulSet
# volumeClaimTemplates are immutable post-creation — the chart-mediated
# path fails the helm upgrade with "spec: Forbidden". Audit logs grow
# unbounded per request; allow the volume to expand to 10Gi rather
# than ride a stuck-Pending vault-N the moment the PVC fills.
resource "kubernetes_annotations" "audit_vault_autoresizer" {
for_each = toset(["0", "1", "2"])
api_version = "v1"
kind = "PersistentVolumeClaim"
metadata {
name = "audit-vault-${each.key}"
namespace = "vault"
}
annotations = {
"resize.topolvm.io/threshold" = "10%"
"resize.topolvm.io/increase" = "100%"
"resize.topolvm.io/storage_limit" = "10Gi"
}
force = true
depends_on = [helm_release.vault]
}
# --- Self-read: Vault's own OIDC credentials from KV --- # --- Self-read: Vault's own OIDC credentials from KV ---
data "vault_kv_secret_v2" "vault" { data "vault_kv_secret_v2" "vault" {