2026-03-14 09:49:38 +00:00
variable " tls_secret_name " {
type = string
sensitive = true
}
variable " postgresql_host " { type = string }
variable " claude_memory_db_password " {
type = string
sensitive = true
}
2026-03-14 17:15:48 +00:00
data " vault_kv_secret_v2 " " secrets " {
mount = " secret "
name = " claude-memory "
2026-03-14 09:49:38 +00:00
}
resource " kubernetes_namespace " " claude-memory " {
metadata {
name = " claude-memory "
labels = {
tier = local . tiers . aux
}
}
[infra] Suppress Goldilocks vpa-update-mode label drift on all namespaces [ci skip]
## Context
Wave 3B-continued: the Goldilocks VPA dashboard (stacks/vpa) runs a Kyverno
ClusterPolicy `goldilocks-vpa-auto-mode` that mutates every namespace with
`metadata.labels["goldilocks.fairwinds.com/vpa-update-mode"] = "off"`. This
is intentional — Terraform owns container resource limits, and Goldilocks
should only provide recommendations, never auto-update. The label is how
Goldilocks decides per-namespace whether to run its VPA in `off` mode.
Effect on Terraform: every `kubernetes_namespace` resource shows the label
as pending-removal (`-> null`) on every `scripts/tg plan`. Dawarich survey
2026-04-18 confirmed the drift. Cluster-side count: 88 namespaces carry the
label (`kubectl get ns -o json | jq ... | wc -l`). Every TF-managed namespace
is affected.
This commit brings the intentional admission drift under the same
`# KYVERNO_LIFECYCLE_V1` discoverability marker introduced in c9d221d5 for
the ndots dns_config pattern. The marker now stands generically for any
Kyverno admission-webhook drift suppression; the inline comment records
which specific policy stamps which specific field so future grep audits
show why each suppression exists.
## This change
107 `.tf` files touched — every stack's `resource "kubernetes_namespace"`
resource gets:
```hcl
lifecycle {
# KYVERNO_LIFECYCLE_V1: goldilocks-vpa-auto-mode ClusterPolicy stamps this label on every namespace
ignore_changes = [metadata[0].labels["goldilocks.fairwinds.com/vpa-update-mode"]]
}
```
Injection was done with a brace-depth-tracking Python pass (`/tmp/add_goldilocks_ignore.py`):
match `^resource "kubernetes_namespace" ` → track `{` / `}` until the
outermost closing brace → insert the lifecycle block before the closing
brace. The script is idempotent (skips any file that already mentions
`goldilocks.fairwinds.com/vpa-update-mode`) so re-running is safe.
Vault stack picked up 2 namespaces in the same file (k8s-users produces
one, plus a second explicit ns) — confirmed via file diff (+8 lines).
## What is NOT in this change
- `stacks/trading-bot/main.tf` — entire file is `/* … */` commented out
(paused 2026-04-06 per user decision). Reverted after the script ran.
- `stacks/_template/main.tf.example` — per-stack skeleton, intentionally
minimal. User keeps it that way. Not touched by the script (file
has no real `resource "kubernetes_namespace"` — only a placeholder
comment).
- `.terraform/` copies (e.g. `stacks/metallb/.terraform/modules/...`) —
gitignored, won't commit; the live path was edited.
- `terraform fmt` cleanup of adjacent pre-existing alignment issues in
authentik, freedify, hermes-agent, nvidia, vault, meshcentral. Reverted
to keep the commit scoped to the Goldilocks sweep. Those files will
need a separate fmt-only commit or will be cleaned up on next real
apply to that stack.
## Verification
Dawarich (one of the hundred-plus touched stacks) showed the pattern
before and after:
```
$ cd stacks/dawarich && ../../scripts/tg plan
Before:
Plan: 0 to add, 2 to change, 0 to destroy.
# kubernetes_namespace.dawarich will be updated in-place
(goldilocks.fairwinds.com/vpa-update-mode -> null)
# module.tls_secret.kubernetes_secret.tls_secret will be updated in-place
(Kyverno generate.* labels — fixed in 8d94688d)
After:
No changes. Your infrastructure matches the configuration.
```
Injection count check:
```
$ rg -c 'KYVERNO_LIFECYCLE_V1: goldilocks-vpa-auto-mode' stacks/ | awk -F: '{s+=$2} END {print s}'
108
```
## Reproduce locally
1. `git pull`
2. Pick any stack: `cd stacks/<name> && ../../scripts/tg plan`
3. Expect: no drift on the namespace's goldilocks.fairwinds.com/vpa-update-mode label.
Closes: code-dwx
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 21:15:27 +00:00
lifecycle {
# KYVERNO_LIFECYCLE_V1: goldilocks-vpa-auto-mode ClusterPolicy stamps this label on every namespace
ignore_changes = [ metadata [ 0 ] . labels [ " goldilocks.fairwinds.com/vpa-update-mode " ] ]
}
2026-03-14 09:49:38 +00:00
}
migrate consuming stacks to ESO + remove k8s-dashboard static token
Phase 9: ExternalSecret migration across 26 stacks:
Fully migrated (vault data source removed, ESO delivers secrets):
- speedtest, shadowsocks, wealthfolio, plotting-book, f1-stream, tandoor
- n8n, dawarich, diun, netbox, onlyoffice, tuya-bridge
- hackmd (ESO template for DB URL), health (ESO template for DB URL)
- trading-bot (ESO template for DATABASE_URL + 7 secret env vars)
- forgejo (removed unused vault data source)
Partially migrated (vault kept for plan-time, ESO added for runtime):
- immich, linkwarden, nextcloud, paperless-ngx (jsondecode for homepage)
- claude-memory, rybbit, url, webhook_handler (plan-time in locals/jobs)
- woodpecker, openclaw, resume (plan-time in helm values/jobs/modules)
17 stacks unchanged (all plan-time: homepage annotations, configmaps,
module inputs) — vault data source works with OIDC auth.
Phase 17a: Remove k8s-dashboard static admin token secret.
Users now get tokens via: vault write kubernetes/creds/dashboard-admin
2026-03-15 19:05:04 +00:00
resource " kubernetes_manifest " " external_secret " {
manifest = {
apiVersion = " external-secrets.io/v1beta1 "
kind = " ExternalSecret "
metadata = {
name = " claude-memory-secrets "
namespace = " claude-memory "
}
spec = {
refreshInterval = " 15m "
secretStoreRef = {
name = " vault-kv "
kind = " ClusterSecretStore "
}
target = {
name = " claude-memory-secrets "
}
dataFrom = [ {
extract = {
key = " claude-memory "
}
} ]
}
}
depends_on = [ kubernetes_namespace . claude - memory ]
}
fix DB password desync + migrate remaining tfvars to Vault
DB desync fix: Stacks with Vault DB engine rotation (24h) now read
the password from vault-database ClusterSecretStore instead of vault-kv.
9 stacks updated with db ExternalSecrets reading from static-creds/*.
Stacks fixed: speedtest, hackmd, health, trading-bot, claude-memory,
woodpecker, linkwarden, nextcloud, url.
terraform.tfvars migration:
- plotting-book: google_client_id/secret → Vault KV + secret_key_ref
- tandoor: email_password var removed (was default="", now optional ESO)
- infra: ssh_private_key, vm_wizard_password, dockerhub_registry_password
→ Vault KV at secret/infra + data source
2026-03-15 21:39:45 +00:00
# DB credentials from Vault database engine (rotated every 24h)
resource " kubernetes_manifest " " db_external_secret " {
manifest = {
apiVersion = " external-secrets.io/v1beta1 "
kind = " ExternalSecret "
metadata = {
name = " claude-memory-db-creds "
namespace = " claude-memory "
}
spec = {
refreshInterval = " 15m "
secretStoreRef = {
name = " vault-database "
kind = " ClusterSecretStore "
}
target = {
name = " claude-memory-db-creds "
template = {
data = {
DATABASE_URL = " postgresql://claude_memory:{{ .password }}@ ${ var . postgresql_host } :5432/claude_memory "
DB_PASSWORD = " {{ .password }} "
}
}
}
data = [ {
secretKey = " password "
remoteRef = {
key = " static-creds/pg-claude-memory "
property = " password "
}
} ]
}
}
depends_on = [ kubernetes_namespace . claude - memory ]
}
2026-03-14 09:49:38 +00:00
module " tls_secret " {
source = " ../../modules/kubernetes/setup_tls_secret "
namespace = kubernetes_namespace . claude - memory . metadata [ 0 ] . name
tls_secret_name = var . tls_secret_name
}
# Database init job
resource " kubernetes_job " " db_init " {
metadata {
name = " claude-memory-db-init "
namespace = kubernetes_namespace . claude - memory . metadata [ 0 ] . name
}
spec {
template {
metadata { }
spec {
container {
name = " db-init "
image = " postgres:16-alpine "
command = [
" sh " , " -c " ,
< < - EOT
set - e
2026-03-14 17:15:48 +00:00
PGPASSWORD =' $ { data . vault_kv_secret_v2 . secrets . data [ " dbaas_root_password " ] } ' psql - h $ { var . postgresql_host } - U root - tc " SELECT 1 FROM pg_roles WHERE rolname='claude_memory' " | grep - q 1 | | \
PGPASSWORD =' $ { data . vault_kv_secret_v2 . secrets . data [ " dbaas_root_password " ] } ' psql - h $ { var . postgresql_host } - U root - c " CREATE ROLE claude_memory WITH LOGIN PASSWORD ' ${ var . claude_memory_db_password } ' "
PGPASSWORD =' $ { data . vault_kv_secret_v2 . secrets . data [ " dbaas_root_password " ] } ' psql - h $ { var . postgresql_host } - U root - tc " SELECT 1 FROM pg_database WHERE datname='claude_memory' " | grep - q 1 | | \
PGPASSWORD =' $ { data . vault_kv_secret_v2 . secrets . data [ " dbaas_root_password " ] } ' psql - h $ { var . postgresql_host } - U root - c " CREATE DATABASE claude_memory OWNER claude_memory "
PGPASSWORD =' $ { data . vault_kv_secret_v2 . secrets . data [ " dbaas_root_password " ] } ' psql - h $ { var . postgresql_host } - U root - c " GRANT ALL PRIVILEGES ON DATABASE claude_memory TO claude_memory "
2026-03-14 09:49:38 +00:00
echo " Database init complete "
EOT
]
}
restart_policy = " Never "
}
}
backoff_limit = 3
}
wait_for_completion = true
timeouts {
create = " 2m "
}
}
resource " kubernetes_deployment " " claude-memory " {
depends_on = [ kubernetes_job . db_init ]
metadata {
name = " claude-memory "
namespace = kubernetes_namespace . claude - memory . metadata [ 0 ] . name
labels = {
app = " claude-memory "
tier = local . tiers . aux
}
migrate consuming stacks to ESO + remove k8s-dashboard static token
Phase 9: ExternalSecret migration across 26 stacks:
Fully migrated (vault data source removed, ESO delivers secrets):
- speedtest, shadowsocks, wealthfolio, plotting-book, f1-stream, tandoor
- n8n, dawarich, diun, netbox, onlyoffice, tuya-bridge
- hackmd (ESO template for DB URL), health (ESO template for DB URL)
- trading-bot (ESO template for DATABASE_URL + 7 secret env vars)
- forgejo (removed unused vault data source)
Partially migrated (vault kept for plan-time, ESO added for runtime):
- immich, linkwarden, nextcloud, paperless-ngx (jsondecode for homepage)
- claude-memory, rybbit, url, webhook_handler (plan-time in locals/jobs)
- woodpecker, openclaw, resume (plan-time in helm values/jobs/modules)
17 stacks unchanged (all plan-time: homepage annotations, configmaps,
module inputs) — vault data source works with OIDC auth.
Phase 17a: Remove k8s-dashboard static admin token secret.
Users now get tokens via: vault write kubernetes/creds/dashboard-admin
2026-03-15 19:05:04 +00:00
annotations = {
" reloader.stakater.com/auto " = " true "
}
2026-03-14 09:49:38 +00:00
}
spec {
2026-03-22 03:10:12 +02:00
replicas = 1
2026-03-14 09:49:38 +00:00
selector {
match_labels = {
app = " claude-memory "
}
}
template {
metadata {
labels = {
app = " claude-memory "
}
2026-03-15 19:17:44 +00:00
annotations = {
" dependency.kyverno.io/wait-for " = " postgresql.dbaas:5432 "
}
2026-03-14 09:49:38 +00:00
}
spec {
2026-03-14 17:15:48 +00:00
affinity {
pod_anti_affinity {
required_during_scheduling_ignored_during_execution {
label_selector {
match_labels = {
app = " claude-memory "
}
}
topology_key = " kubernetes.io/hostname "
}
}
}
2026-03-14 09:49:38 +00:00
container {
name = " claude-memory "
2026-03-15 15:32:50 +00:00
image = " viktorbarzin/claude-memory-mcp:17 "
2026-03-14 09:49:38 +00:00
port {
container_port = 8000
}
env {
fix DB password desync + migrate remaining tfvars to Vault
DB desync fix: Stacks with Vault DB engine rotation (24h) now read
the password from vault-database ClusterSecretStore instead of vault-kv.
9 stacks updated with db ExternalSecrets reading from static-creds/*.
Stacks fixed: speedtest, hackmd, health, trading-bot, claude-memory,
woodpecker, linkwarden, nextcloud, url.
terraform.tfvars migration:
- plotting-book: google_client_id/secret → Vault KV + secret_key_ref
- tandoor: email_password var removed (was default="", now optional ESO)
- infra: ssh_private_key, vm_wizard_password, dockerhub_registry_password
→ Vault KV at secret/infra + data source
2026-03-15 21:39:45 +00:00
name = " DATABASE_URL "
value_from {
secret_key_ref {
name = " claude-memory-db-creds "
key = " DATABASE_URL "
}
}
2026-03-14 09:49:38 +00:00
}
env {
2026-03-22 20:08:07 +02:00
name = " API_KEYS "
migrate consuming stacks to ESO + remove k8s-dashboard static token
Phase 9: ExternalSecret migration across 26 stacks:
Fully migrated (vault data source removed, ESO delivers secrets):
- speedtest, shadowsocks, wealthfolio, plotting-book, f1-stream, tandoor
- n8n, dawarich, diun, netbox, onlyoffice, tuya-bridge
- hackmd (ESO template for DB URL), health (ESO template for DB URL)
- trading-bot (ESO template for DATABASE_URL + 7 secret env vars)
- forgejo (removed unused vault data source)
Partially migrated (vault kept for plan-time, ESO added for runtime):
- immich, linkwarden, nextcloud, paperless-ngx (jsondecode for homepage)
- claude-memory, rybbit, url, webhook_handler (plan-time in locals/jobs)
- woodpecker, openclaw, resume (plan-time in helm values/jobs/modules)
17 stacks unchanged (all plan-time: homepage annotations, configmaps,
module inputs) — vault data source works with OIDC auth.
Phase 17a: Remove k8s-dashboard static admin token secret.
Users now get tokens via: vault write kubernetes/creds/dashboard-admin
2026-03-15 19:05:04 +00:00
value_from {
secret_key_ref {
name = " claude-memory-secrets "
2026-03-22 20:08:07 +02:00
key = " api_keys "
migrate consuming stacks to ESO + remove k8s-dashboard static token
Phase 9: ExternalSecret migration across 26 stacks:
Fully migrated (vault data source removed, ESO delivers secrets):
- speedtest, shadowsocks, wealthfolio, plotting-book, f1-stream, tandoor
- n8n, dawarich, diun, netbox, onlyoffice, tuya-bridge
- hackmd (ESO template for DB URL), health (ESO template for DB URL)
- trading-bot (ESO template for DATABASE_URL + 7 secret env vars)
- forgejo (removed unused vault data source)
Partially migrated (vault kept for plan-time, ESO added for runtime):
- immich, linkwarden, nextcloud, paperless-ngx (jsondecode for homepage)
- claude-memory, rybbit, url, webhook_handler (plan-time in locals/jobs)
- woodpecker, openclaw, resume (plan-time in helm values/jobs/modules)
17 stacks unchanged (all plan-time: homepage annotations, configmaps,
module inputs) — vault data source works with OIDC auth.
Phase 17a: Remove k8s-dashboard static admin token secret.
Users now get tokens via: vault write kubernetes/creds/dashboard-admin
2026-03-15 19:05:04 +00:00
}
}
2026-03-14 09:49:38 +00:00
}
2026-03-14 17:15:48 +00:00
startup_probe {
http_get {
path = " /health "
port = 8000
}
failure_threshold = 30
period_seconds = 2
}
2026-03-14 09:49:38 +00:00
liveness_probe {
http_get {
path = " /health "
port = 8000
}
initial_delay_seconds = 5
period_seconds = 30
}
readiness_probe {
http_get {
path = " /health "
port = 8000
}
initial_delay_seconds = 3
period_seconds = 10
}
resources {
requests = {
right-size 14 services and scale down GPU-heavy workloads [ci skip]
Memory right-sizing based on VPA upperBound analysis:
- Increases: stirling-pdf 1200→1536Mi, claude-memory 64→128Mi,
dawarich 512→768Mi, kyverno-cleanup 128→192Mi, linkwarden 768→1Gi,
navidrome 64→128Mi, listenarr 768→896Mi, privatebin 64→128Mi,
ntfy 64→128Mi, health 128→256Mi, dbaas quota 16→20Gi,
mysql-operator 384→512Mi
- Decreases: rybbit 768→384Mi, nvidia-exporter added explicit 192Mi,
dcgm-exporter 2560→1536Mi
- Scale to 0: ebook2audiobook/audiblez-web, whisper (GPU node pressure)
Net effect: -496Mi cluster-wide, 13 ContainerNearOOM alerts resolved,
all ResourceQuota pressures cleared, GPU health green.
2026-03-15 23:00:49 +00:00
memory = " 128Mi "
2026-03-14 09:49:38 +00:00
cpu = " 10m "
}
limits = {
right-size 14 services and scale down GPU-heavy workloads [ci skip]
Memory right-sizing based on VPA upperBound analysis:
- Increases: stirling-pdf 1200→1536Mi, claude-memory 64→128Mi,
dawarich 512→768Mi, kyverno-cleanup 128→192Mi, linkwarden 768→1Gi,
navidrome 64→128Mi, listenarr 768→896Mi, privatebin 64→128Mi,
ntfy 64→128Mi, health 128→256Mi, dbaas quota 16→20Gi,
mysql-operator 384→512Mi
- Decreases: rybbit 768→384Mi, nvidia-exporter added explicit 192Mi,
dcgm-exporter 2560→1536Mi
- Scale to 0: ebook2audiobook/audiblez-web, whisper (GPU node pressure)
Net effect: -496Mi cluster-wide, 13 ContainerNearOOM alerts resolved,
all ResourceQuota pressures cleared, GPU health green.
2026-03-15 23:00:49 +00:00
memory = " 128Mi "
2026-03-14 09:49:38 +00:00
}
}
}
}
}
}
lifecycle {
2026-04-18 14:08:10 +00:00
# DRIFT_WORKAROUND: CI pipeline owns image tag (kubectl set image from Woodpecker/GHA). Reviewed 2026-04-18.
2026-03-14 09:49:38 +00:00
ignore_changes = [
spec [ 0 ] . template [ 0 ] . spec [ 0 ] . container [ 0 ] . image
]
}
}
2026-04-06 00:47:40 +03:00
# PDB removed — single replica with minAvailable=1 blocks all node drains.
# claude-memory is non-critical and recovers quickly after rescheduling.
2026-03-14 17:15:48 +00:00
2026-03-14 09:49:38 +00:00
resource " kubernetes_service " " claude-memory " {
metadata {
name = " claude-memory "
namespace = kubernetes_namespace . claude - memory . metadata [ 0 ] . name
labels = {
app = " claude-memory "
}
}
spec {
selector = {
app = " claude-memory "
}
port {
name = " http "
port = 80
target_port = 8000
}
}
}
module " ingress " {
source = " ../../modules/kubernetes/ingress_factory "
2026-04-16 13:45:04 +00:00
dns_type = " proxied "
2026-03-14 09:49:38 +00:00
namespace = kubernetes_namespace . claude - memory . metadata [ 0 ] . name
name = " claude-memory "
tls_secret_name = var . tls_secret_name
extra_annotations = {
" gethomepage.dev/enabled " = " true "
" gethomepage.dev/name " = " Claude Memory "
" gethomepage.dev/description " = " Shared persistent memory for Claude sessions "
" gethomepage.dev/icon " = " claude-ai.png "
" gethomepage.dev/group " = " Core Platform "
" gethomepage.dev/pod-selector " = " "
}
}