viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
infra/stacks
History
Viktor Barzin 50dea8f0a7 [monitoring] Add Claude OAuth token expiry monitoring + alerts 
## Context

The new CLAUDE_CODE_OAUTH_TOKEN mechanism (commit 8a054752) uses
long-lived 1-year tokens minted via `claude setup-token`. Tokens don't
auto-refresh — at the 1-year mark they expire hard and the upgrade
agent stops working. We need to be told 30 days ahead, not find out
when DIUN fires and gets 401 again.

A cron rotator doesn't make sense here (tokens don't refresh, they
just expire) so we alert instead. Two spares at
`secret/claude-agent-service-spare-{1,2}` provide failover runway —
monitor covers all three.

## This change

**CronJob** (`claude-agent` ns, every 6h): reads a ConfigMap
containing `<path> → expiry_unix_timestamp` entries, pushes
`claude_oauth_token_expiry_timestamp{path="..."}` and
`claude_oauth_expiry_monitor_last_push_timestamp` to Pushgateway at
`prometheus-prometheus-pushgateway.monitoring:9091`.

**ConfigMap** generated from a Terraform local `claude_oauth_token_mint_epochs`
— source of truth for mint times. On rotation, update the map + apply.
TTL is a shared local (365d).

**PrometheusRules** (in prometheus_chart_values.tpl):
- `ClaudeOAuthTokenExpiringSoon`  — <30d, warning, for 1h
- `ClaudeOAuthTokenCritical`      — <7d,  critical, for 10m
- `ClaudeOAuthTokenMonitorStale`  — last push >48h, warning
- `ClaudeOAuthTokenMonitorNeverRun` — metric absent for 2h, warning

Alert labels include `{{ $labels.path }}` so we know which token is
expiring (primary / spare-1 / spare-2).

## Verification

```
$ kubectl -n claude-agent create job --from=cronjob/claude-oauth-expiry-monitor manual
$ curl pushgateway/metrics | grep claude_oauth_token_expiry
claude_oauth_token_expiry_timestamp{...,path="primary"} 1.808064429e+09
claude_oauth_token_expiry_timestamp{...,path="spare-1"} 1.80806428e+09
claude_oauth_token_expiry_timestamp{...,path="spare-2"} 1.808064429e+09

$ query: (claude_oauth_token_expiry_timestamp - time()) / 86400
  primary: 365.2 days
  spare-1: 365.2 days
  spare-2: 365.2 days
```

## Rotation playbook (future)

1. `kubectl run -it --rm --image=registry.viktorbarzin.me/claude-agent-service:latest tokmint -- claude setup-token`
   (or harvest via `harvest3.py` pattern in memory for headless flow)
2. `vault kv patch secret/claude-agent-service claude_oauth_token=<new>`
3. Update `claude_oauth_token_mint_epochs["primary"]` in
   `stacks/claude-agent-service/main.tf` with new unix timestamp
4. `scripts/tg apply` claude-agent-service + monitoring
5. Alert clears within 6h (next cron tick) + 1h of the
   `ClaudeOAuthTokenExpiringSoon` "for:" duration

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 12:27:11 +00:00
..
_template [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
actualbudget [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
affine [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
authentik [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
beads-server Add broker-sync Terraform stack (#7) 2026-04-17 21:17:45 +01:00
blog [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
broker-sync Add broker-sync Terraform stack (#7) 2026-04-17 21:17:45 +01:00
changedetection [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
city-guesser [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
claude-agent-service [monitoring] Add Claude OAuth token expiry monitoring + alerts 2026-04-18 12:27:11 +00:00
claude-memory Add broker-sync Terraform stack (#7) 2026-04-17 21:17:45 +01:00
cloudflared [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
cnpg [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
coturn [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
crowdsec [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
cyberchef [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
dashy [cleanup] Remove ollama from dashy + docs + nfs_directories 2026-04-18 11:17:59 +00:00
dawarich [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
dbaas [dbaas] Fix mysql_static_user heredoc quoting 2026-04-17 22:34:12 +00:00
descheduler [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
diun [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
ebook2audiobook [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
ebooks [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
echo [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
excalidraw [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
external-secrets [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
f1-stream [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
foolery Add broker-sync Terraform stack (#7) 2026-04-17 21:17:45 +01:00
forgejo [forgejo] Probe /api/healthz for external monitor 2026-04-17 22:06:23 +00:00
freedify [freedify] Remove stale sed patches from container startup 2026-04-17 06:17:13 +00:00
freshrss [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
frigate [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
grampsweb [grampsweb] Align PVC resource to encrypted storage; imported state 2026-04-18 11:37:45 +00:00
hackmd [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
headscale [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
health [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
hermes-agent [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
homepage [cleanup] Remove ollama from dashy + docs + nfs_directories 2026-04-18 11:17:59 +00:00
immich [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
infra [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
infra-maintenance [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
insta2spotify [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
iscsi-csi [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
isponsorblocktv [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
jsoncrack [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
k8s-dashboard [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
k8s-portal [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
kms [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
kyverno kyverno: strip resources.limits.cpu cluster-wide via ClusterPolicy 2026-04-18 11:34:39 +00:00
linkwarden [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
mailserver [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
matrix [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
meshcentral [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
metallb [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
metrics-server [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
monitoring [monitoring] Add Claude OAuth token expiry monitoring + alerts 2026-04-18 12:27:11 +00:00
n8n [n8n] Fix broken DIUN auto-upgrade pipeline — missing auth token to claude-agent-service 2026-04-18 10:41:09 +00:00
navidrome [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
netbox [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
networking-toolbox [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
nextcloud [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
nfs-csi [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
novelapp [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
ntfy [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
nvidia [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
onlyoffice [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
openclaw [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
osm_routing [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
owntracks [storage] Fix owntracks + wealthfolio: switch to encrypted PVCs 2026-04-17 20:29:57 +00:00
paperless-ngx [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
phpipam [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
platform [infra] Add Cloudflare provider to all stack lock files and generated providers 2026-04-16 16:31:36 +00:00
plotting-book [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
poison-fountain [infra] Scale down unused services + remove DoH ingress 2026-04-17 18:55:52 +00:00
priority-pass [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
privatebin [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
proxmox-csi feat(storage): migrate all sensitive services to proxmox-lvm-encrypted 2026-04-15 20:15:30 +00:00
pvc-autoresizer fix: disable cert-manager webhook for pvc-autoresizer, use self-signed cert [ci skip] 2026-04-03 23:44:49 +03:00
rbac [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
real-estate-crawler [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
redis [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
reloader [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
resume [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
reverse-proxy [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
rybbit [rybbit] Remove ollama favicon cache entry (deploy on next manual wrangler) 2026-04-18 11:14:38 +00:00
sealed-secrets [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
send [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
servarr [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
shadowsocks [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
speedtest [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
status-page [alerts] Fix status-page-pusher crash + Prometheus backup push 2026-04-17 18:29:43 +00:00
stirling-pdf [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
tandoor [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
technitium [infra] Scale down unused services + remove DoH ingress 2026-04-17 18:55:52 +00:00
terminal Add broker-sync Terraform stack (#7) 2026-04-17 21:17:45 +01:00
tor-proxy [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
trading-bot [trading-bot] Remove ollama refs from commented-out source 2026-04-18 11:14:22 +00:00
traefik [traefik] Remove ollama-tcp entrypoint 2026-04-18 11:12:59 +00:00
travel_blog [infra] Scale down unused services + remove DoH ingress 2026-04-17 18:55:52 +00:00
tuya-bridge [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
uptime-kuma [uptime-kuma] Codify MySQL monitor (id=663) via idempotent sync CronJob 2026-04-18 12:04:17 +00:00
url [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
vault [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
vaultwarden [traefik] Remove broken rewrite-body plugin and all rybbit/anti-AI injection 2026-04-17 12:41:17 +00:00
vpa [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
wealthfolio [storage] Fix owntracks + wealthfolio: switch to encrypted PVCs 2026-04-17 20:29:57 +00:00
webhook_handler [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
whisper [whisper] Remove ollama_tcp IngressRouteTCP (ollama decom) 2026-04-18 11:11:21 +00:00
wireguard [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
woodpecker [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
xray [infra] Migrate Terraform state from local SOPS to PostgreSQL backend 2026-04-16 19:33:12 +00:00
ytdlp [ytdlp] Remove ollama_host variable and fallback env vars 2026-04-18 11:13:42 +00:00