diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 55cbc0d1..98dacd41 100755 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -117,7 +117,7 @@ Repo IDs: infra=1, Website=2, finance=3, health=4, travel_blog=5, webhook-handle - **Rate limiting**: Return 429 (not 503). Per-service tuning: Immich/Nextcloud need higher limits. - **Retry middleware**: 2 attempts, 100ms — in default ingress chain. - **HTTP/3 (QUIC)**: Enabled cluster-wide via Traefik. -- **IPAM & DNS auto-registration**: pfSense Kea DHCP serves all 3 subnets (VLAN 10, VLAN 20, 192.168.1.x). Kea DDNS auto-registers every DHCP client in Technitium (RFC 2136, A+PTR). CronJob `phpipam-pfsense-import` (5min) pulls Kea leases + ARP into phpIPAM via SSH (passive, no scanning). CronJob `phpipam-dns-sync` (15min) bidirectional sync phpIPAM ↔ Technitium. 42 MAC reservations for 192.168.1.x. +- **IPAM & DNS auto-registration**: pfSense Kea DHCP serves all 3 subnets (VLAN 10, VLAN 20, 192.168.1.x). Kea DDNS auto-registers every DHCP client in Technitium (RFC 2136, A+PTR). CronJob `phpipam-pfsense-import` (hourly) pulls Kea leases + ARP into phpIPAM via SSH (passive, no scanning). CronJob `phpipam-dns-sync` (15min) bidirectional sync phpIPAM ↔ Technitium. 42 MAC reservations for 192.168.1.x. ## Service-Specific Notes | Service | Key Operational Knowledge | @@ -129,7 +129,7 @@ Repo IDs: infra=1, Website=2, finance=3, health=4, travel_blog=5, webhook-handle | Authentik | 3 replicas, PgBouncer in front of PostgreSQL, strip auth headers before forwarding | | Kyverno | failurePolicy=Ignore to prevent blocking cluster, pin chart version | | MySQL Standalone | Raw `kubernetes_stateful_set_v1` with `mysql:8.4` (migrated from InnoDB Cluster 2026-04-16). `skip-log-bin`, `innodb_flush_log_at_trx_commit=2`, `innodb_doublewrite=ON`. ConfigMap `mysql-standalone-cnf`. PVC `data-mysql-standalone-0` (15Gi, `proxmox-lvm-encrypted`). Service `mysql.dbaas` unchanged. Anti-affinity excludes k8s-node1. Old InnoDB Cluster + operator still in TF (Phase 4 cleanup pending). Bitnami charts deprecated (Broadcom Aug 2025) — use official images. | -| phpIPAM | IPAM — no active scanning. `pfsense-import` CronJob (5min) pulls Kea leases + ARP via SSH. `dns-sync` CronJob (15min) bidirectional sync with Technitium. Kea DDNS on pfSense handles all 3 subnets. API app `claude` (ssl_token). | +| phpIPAM | IPAM — no active scanning. `pfsense-import` CronJob (hourly) pulls Kea leases + ARP via SSH. `dns-sync` CronJob (15min) bidirectional sync with Technitium. Kea DDNS on pfSense handles all 3 subnets. API app `claude` (ssl_token). | ## Monitoring & Alerting - Alert cascade inhibitions: if node is down, suppress pod alerts on that node. diff --git a/.claude/reference/authentik-state.md b/.claude/reference/authentik-state.md index 2005bb09..f76dd325 100644 --- a/.claude/reference/authentik-state.md +++ b/.claude/reference/authentik-state.md @@ -119,3 +119,18 @@ Removed bindings from: - `default-source-authentication` (PK: via policybindingmodel `1a779f24`) — Google/GitHub/Facebook OAuth Policy still exists with 0 bindings. If brute-force protection is needed, bind to the **password stage** (not the flow level). + +## Session Duration (2026-05-01) + +Pinned via Terraform in `stacks/authentik/`: + +| Knob | Value | Surface | Effect | +|------|-------|---------|--------| +| `UserLoginStage.session_duration` on `default-authentication-login` | `weeks=4` | `authentik_stage_user_login.default_login` in `authentik_provider.tf` | Authenticated users stay logged in 4 weeks across browser restarts. No sliding refresh — resets on each login. | +| `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE` (server + worker) | `hours=2` | `server.env` + `worker.env` in `modules/authentik/values.yaml` | Anonymous Django sessions (bots, healthcheckers, partial flows) are reaped within 2h instead of the 1d default. | + +Notes: +- There is **no** `Brand.session_duration`; `UserLoginStage` is the only correct lever for authenticated session lifetime. +- Embedded outpost session storage moved from `/dev/shm` → Postgres table `authentik_providers_proxy_proxysession` in authentik 2025.10. The 2026-04-18 `/dev/shm`-fill outage class is no longer load-bearing in 2026.2.2; the `unauthenticated_age` cap is still the right lever for anonymous-session bloat from external monitors. +- `ProxyProvider.access_token_validity` and `remember_me_offset` stay UI-managed via `ignore_changes`. +- The `unauthenticated_age` env var is injected via `server.env` / `worker.env` (not `authentik.sessions.unauthenticated_age`) because we set `authentik.existingSecret.secretName: goauthentik`, which makes the chart skip rendering its own `AUTHENTIK_*` Secret. The `authentik.*` value block is therefore inert in this stack — anything new under `authentik.*` must use the `*.env` arrays instead. The same applies to the existing `authentik.cache.*`, `authentik.web.*`, `authentik.worker.*` blocks (currently inert; live values come from the orphaned, helm-keep-policy `goauthentik` Secret created by chart 2025.10.3 before `existingSecret` was introduced). diff --git a/docs/architecture/dns.md b/docs/architecture/dns.md index 97e609f3..eec99830 100644 --- a/docs/architecture/dns.md +++ b/docs/architecture/dns.md @@ -377,7 +377,7 @@ Devices get automatic DNS registration without manual intervention. See [network Summary: 1. **Kea DHCP** on pfSense assigns IP (53 reservations across 3 subnets). DHCP option 6 (DNS servers) is pushed with two IPs per internal subnet: internal resolver + AdGuard public fallback (`94.140.14.14`) — clients survive an internal DNS outage. 2. **Kea DDNS** sends **TSIG-signed** RFC 2136 dynamic update to Technitium (A + PTR records) — immediate. Key `kea-ddns` (HMAC-SHA256); Technitium enforces both source-IP ACL and TSIG signature on `viktorbarzin.lan` + reverse zones. -3. **phpipam-pfsense-import** CronJob (5min) pulls Kea leases + ARP table into phpIPAM +3. **phpipam-pfsense-import** CronJob (hourly) pulls Kea leases + ARP table into phpIPAM 4. **phpipam-dns-sync** CronJob (15min) pushes named phpIPAM hosts → Technitium A + PTR, pulls Technitium PTR → phpIPAM hostnames ## Automation CronJobs @@ -389,7 +389,7 @@ Summary: | `technitium-split-horizon-sync` | `15 */6 * * *` | technitium | Split Horizon + DNS Rebinding Protection on all 3 instances | | `technitium-dns-optimization` | `30 */6 * * *` | technitium | Min cache TTL 60s, emrsn.org stub zone | | `phpipam-dns-sync` | `*/15 * * * *` | phpipam | Bidirectional phpIPAM ↔ Technitium DNS sync | -| `phpipam-pfsense-import` | `*/5 * * * *` | phpipam | Import Kea DHCP leases + ARP from pfSense | +| `phpipam-pfsense-import` | `0 * * * *` | phpipam | Import Kea DHCP leases + ARP from pfSense | ### Password Rotation Flow diff --git a/docs/architecture/networking.md b/docs/architecture/networking.md index 3c75e4fd..e7959589 100644 --- a/docs/architecture/networking.md +++ b/docs/architecture/networking.md @@ -104,7 +104,7 @@ flowchart LR end subgraph K8s["Kubernetes"] - Import[CronJob
pfsense-import
every 5min] + Import[CronJob
pfsense-import
hourly] Sync[CronJob
dns-sync
every 15min] IPAM[phpIPAM
Web UI + API] MySQL[(MySQL
InnoDB)] @@ -338,7 +338,7 @@ Containerd on all K8s nodes uses `hosts.toml` to redirect pulls to the local cac - Stack: `stacks/phpipam/` - Web UI: `phpipam.viktorbarzin.me` (Authentik-protected) - Database: MySQL InnoDB cluster (`mysql.dbaas.svc.cluster.local`) -- Device import: CronJob `phpipam-pfsense-import` every 5min — queries Kea DHCP leases + pfSense ARP table via SSH (no active scanning) +- Device import: CronJob `phpipam-pfsense-import` hourly — queries Kea DHCP leases + pfSense ARP table via SSH (no active scanning) - DNS sync: CronJob `phpipam-dns-sync` every 15min — bidirectional sync between phpIPAM and Technitium DNS (push named hosts → A+PTR, pull DNS hostnames → unnamed phpIPAM entries) - Subnets tracked: 10.0.10.0/24, 10.0.20.0/24, 192.168.1.0/24, 10.3.2.0/24, 192.168.8.0/24, 192.168.0.0/24 - API: REST API enabled (app `claude`, ssl_token auth), MCP server available for agent access diff --git a/docs/runbooks/mailserver-pfsense-haproxy.md b/docs/runbooks/mailserver-pfsense-haproxy.md index 564554eb..329be214 100644 --- a/docs/runbooks/mailserver-pfsense-haproxy.md +++ b/docs/runbooks/mailserver-pfsense-haproxy.md @@ -12,7 +12,11 @@ so pfSense runs a small HAProxy that: 1. Listens on the pfSense VLAN20 IP (`10.0.20.1`) on all 4 mail ports, 2. Forwards each connection to a k8s node's NodePort with `send-proxy-v2`, 3. Injects PROXY v2 framing so Postfix/Dovecot see the original client IP, -4. TCP health-checks every k8s worker — any node can serve (ETP:Cluster). +4. TCP-checks every k8s worker via dedicated **non-PROXY healthcheck NodePorts** + (30145/30146/30147 → pod stock 25/465/587 listeners, no PROXY required). + This split path avoids the `smtpd_peer_hostaddr_to_sockaddr` fatal that + used to fire on every PROXY-aware health probe and throttled real client + connections. Corresponding k8s-side setup (`stacks/mailserver/modules/mailserver/`): @@ -23,14 +27,20 @@ Corresponding k8s-side setup (`stacks/mailserver/modules/mailserver/`): - `:5587` smtpd (alt :587 submission) with `smtpd_upstream_proxy_protocol=haproxy` - ConfigMap `mailserver.config` adds Dovecot `inet_listener imaps_proxy` on port 10993 with `haproxy = yes` and `haproxy_trusted_networks = 10.0.20.0/24`. -- Service `mailserver-proxy` (NodePort, ETP:Cluster) with 4 NodePorts: - - `port 25 → targetPort 2525 → nodePort 30125` - - `port 465 → targetPort 4465 → nodePort 30126` - - `port 587 → targetPort 5587 → nodePort 30127` - - `port 993 → targetPort 10993 → nodePort 30128` +- Service `mailserver-proxy` (NodePort, ETP:Cluster) — 4 PROXY data ports + + 3 non-PROXY healthcheck ports: + - Data (PROXY v2): + - `port 25 → targetPort 2525 → nodePort 30125` + - `port 465 → targetPort 4465 → nodePort 30126` + - `port 587 → targetPort 5587 → nodePort 30127` + - `port 993 → targetPort 10993 → nodePort 30128` + - Healthcheck (no PROXY, stock SMTP/SMTPS/Submission listeners): + - `port 2500 → targetPort 25 → nodePort 30145` (smtp-check) + - `port 4650 → targetPort 465 → nodePort 30146` (smtps-check) + - `port 5870 → targetPort 587 → nodePort 30147` (sub-check) - Service `mailserver` (ClusterIP) — unchanged stock ports 25/465/587/993 for intra-cluster clients (Roundcube pod, `email-roundtrip-monitor` - CronJob). These listeners are PROXY-free. + CronJob, book-search). These listeners are PROXY-free. bd: `code-yiu`. @@ -46,7 +56,9 @@ External mail (WAN) path — PROXY v2 │ │ NAT rdr → 10.0.20.1:{same} │ │ ▼ │ │ pfSense HAProxy (mode tcp, 4 frontends, 4 backend pools) │ -│ │ send-proxy-v2 + tcp-check inter 120000 │ +│ │ data: send-proxy-v2 → :{30125..30128} (PROXY-aware pod) │ +│ │ health: TCP-check → :{30145..30147} (no-PROXY pod) │ +│ │ inter 5000 │ │ ▼ │ │ k8s-node<1-4>:{30125..30128} ← any node (ETP:Cluster) │ │ │ kube-proxy SNAT (source IP lost on the wire) │ @@ -186,11 +198,18 @@ Full restore: pfSense WebUI → Diagnostics → Backup & Restore → Upload that ## Known warts -- HAProxy TCP health-check with `send-proxy-v2` generates `getpeername: - Transport endpoint not connected` warnings on postscreen every check cycle. - Mitigated with `inter 120000` (2 min). To reduce further, switch to - `option smtpchk` — but that requires a separate non-PROXY health-check - port on the pod (not done yet). +- ~~HAProxy TCP health-check with `send-proxy-v2` generates `getpeername: + Transport endpoint not connected` warnings on postscreen every check cycle.~~ + **Resolved 2026-05-05**: dedicated non-PROXY healthcheck NodePorts + (30145/30146/30147 → stock pod 25/465/587) added; HAProxy now checks + those, eliminating both the `getpeername` postscreen warnings and the + `smtpd_peer_hostaddr_to_sockaddr: ... Servname not supported` fatals + that were throttling smtpd respawns and causing ~50% client timeouts on + the public 587 path. `inter` dropped 120000 → 5000 (fast failover, no + log-spam concern). `option smtpchk` was tried but flapped against + postscreen (multi-line greet + DNSBL silence + anti-pre-greet detection + trip HAProxy's parser → L7RSP). Plain TCP check on the no-PROXY ports + is sufficient. - Frontend binds on all pfSense interfaces (`bind :25` instead of `10.0.20.1:25`). `` is set in XML but pfSense templates it port-only. Low concern in practice because WAN firewall rules plus the diff --git a/scripts/pfsense-haproxy-bootstrap.php b/scripts/pfsense-haproxy-bootstrap.php index 3834d852..5452b198 100644 --- a/scripts/pfsense-haproxy-bootstrap.php +++ b/scripts/pfsense-haproxy-bootstrap.php @@ -68,7 +68,35 @@ $NODES = [ ['k8s-node4', '10.0.20.104'], ]; -function build_pool(string $name, string $nodeport, array $nodes): array { +// Build a pool with optional split healthcheck path. +// +// $check_port: if non-null, HAProxy sends health probes to that NodePort +// (which Service `mailserver-proxy` maps to the pod's stock no-PROXY +// listener — see infra/stacks/mailserver/.../mailserver_proxy ports +// 30145/30146/30147). Real client traffic still goes to $nodeport with +// PROXY v2 framing. +// $check_type: 'TCP' for plain accept-on-port checks, 'ESMTP' for +// `option smtpchk EHLO ` (real SMTP banner+EHLO+250). +// +// Why split: smtpd-proxy587/4465 fatal on every PROXY-v2-aware health +// probe with `smtpd_peer_hostaddr_to_sockaddr: ... Servname not supported` +// — the daemon respawns get throttled by Postfix master and real clients +// land mid-respawn → 6s TCP timeout. Routing health probes to the stock +// no-PROXY port sidesteps the bug entirely while data path still gets +// PROXY v2 for CrowdSec/Postfix client-IP visibility. The HAProxy package +// has no `checkport` field, so `port N` is appended via the server's +// `advanced` string (HAProxy parses server keywords in any order). +function build_pool( + string $name, + string $nodeport, + array $nodes, + string $check_type = 'TCP', + ?string $check_port = null, + string $monitor_domain = '' +): array { + $advanced_check = $check_port !== null + ? "send-proxy-v2 port {$check_port}" + : 'send-proxy-v2'; $servers = []; foreach ($nodes as $n) { $servers[] = [ @@ -77,18 +105,19 @@ function build_pool(string $name, string $nodeport, array $nodes): array { 'port' => $nodeport, 'weight' => '10', 'ssl' => '', - // check every 2 min — send-proxy-v2 check + close generates - // noise on postscreen, not worth doing more often. - 'checkinter' => '120000', - 'advanced' => 'send-proxy-v2', + // 5s = sub-block-window failover when a NodePort goes sour. + // Safe to be aggressive once health probes don't fatal smtpd. + 'checkinter' => '5000', + 'advanced' => $advanced_check, 'status' => 'active', ]; } return [ 'name' => $name, 'balance' => 'roundrobin', - 'check_type' => 'TCP', - 'checkinter' => '120000', + 'check_type' => $check_type, + 'monitor_domain' => $monitor_domain, + 'checkinter' => '5000', 'retries' => '3', 'ha_servers' => ['item' => $servers], 'advanced_bind' => '', @@ -132,9 +161,28 @@ $h['ha_pools']['item'] = array_values(array_filter( $h['ha_pools']['item'][] = build_pool('mailserver_nodes', '30125', $NODES); // Production pools — one per mail port. -$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtp', '30125', $NODES); -$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtps', '30126', $NODES); -$h['ha_pools']['item'][] = build_pool('mailserver_nodes_sub', '30127', $NODES); +// +// All SMTP/SMTPS/Submission backends use plain TCP checks against +// dedicated non-PROXY healthcheck NodePorts (30145/30146/30147 → pod +// stock 25/465/587) so probes hit the no-PROXY listeners and avoid +// the smtpd_peer_hostaddr_to_sockaddr fatal that fires on PROXY-v2 +// LOCAL frames. Real client traffic still goes to 30125-30128 with +// PROXY v2 for client-IP visibility. +// +// We tried `option smtpchk EHLO` initially — it works on the plain +// `submission` daemon (587) but flaps the `postscreen` listener on +// port 25 (multi-line greet + DNSBL silence + anti-pre-greet +// detection makes HAProxy's simple smtpchk parser hit L7RSP). A +// plain TCP accept-on-port check is enough for both: HAProxy still +// gets fast failover when the listener actually goes away, and we +// stop triggering the Postfix fatal entirely. +// +// IMAPS stays on its existing TCP-check-with-PROXY-frame for now — +// Dovecot's PROXY parser doesn't show the same fatal pattern; adding +// a separate IMAP healthcheck path would require another svc port. +$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtp', '30125', $NODES, 'TCP', '30145'); +$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtps', '30126', $NODES, 'TCP', '30146'); +$h['ha_pools']['item'][] = build_pool('mailserver_nodes_sub', '30127', $NODES, 'TCP', '30147'); $h['ha_pools']['item'][] = build_pool('mailserver_nodes_imaps', '30128', $NODES); // ── Frontends ─────────────────────────────────────────────────────────── diff --git a/secrets/fullchain.pem b/secrets/fullchain.pem index e4bc0d60..435a3239 100644 Binary files a/secrets/fullchain.pem and b/secrets/fullchain.pem differ diff --git a/secrets/privkey.pem b/secrets/privkey.pem index 1f38edfe..b6cf256c 100644 Binary files a/secrets/privkey.pem and b/secrets/privkey.pem differ diff --git a/stacks/authentik/authentik_provider.tf b/stacks/authentik/authentik_provider.tf index e9db3985..f34214fe 100644 --- a/stacks/authentik/authentik_provider.tf +++ b/stacks/authentik/authentik_provider.tf @@ -57,3 +57,25 @@ resource "authentik_provider_proxy" "catchall" { ignore_changes = [property_mappings, jwt_federation_sources, skip_path_regex, internal_host, basic_auth_enabled, basic_auth_password_attribute, basic_auth_username_attribute, intercept_header_auth, access_token_validity] } } + +# ----------------------------------------------------------------------------- +# Default User Login stage — bound to default-authentication-flow. +# Adopted into Terraform 2026-05-01 to set session_duration=weeks=4 so users +# stay logged in across browser restarts. There is no Brand.session_duration +# in authentik 2026.2.x — UserLoginStage is the correct knob. +# ----------------------------------------------------------------------------- + +resource "authentik_stage_user_login" "default_login" { + name = "default-authentication-login" + session_duration = "weeks=4" + lifecycle { + # Pin only session_duration; everything else stays UI-managed so the + # plan doesn't churn unrelated knobs (e.g. remember_me_offset toggles). + ignore_changes = [ + remember_me_offset, + terminate_other_sessions, + geoip_binding, + network_binding, + ] + } +} diff --git a/stacks/authentik/modules/authentik/values.yaml b/stacks/authentik/modules/authentik/values.yaml index e8c7d5ea..9822516c 100644 --- a/stacks/authentik/modules/authentik/values.yaml +++ b/stacks/authentik/modules/authentik/values.yaml @@ -37,6 +37,15 @@ authentik: server: replicas: 3 + # Anonymous Django sessions (no completed login: bots, healthcheckers, + # partial flows) expire in 2h. Default is days=1. Once login completes, + # UserLoginStage.session_duration takes over via request.session.set_expiry. + # Injected via server.env (not authentik.sessions.*) because we use + # authentik.existingSecret.secretName, which makes the chart skip + # rendering the AUTHENTIK_* secret — so the values block doesn't reach env. + env: + - name: AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE + value: "hours=2" strategy: type: RollingUpdate rollingUpdate: @@ -70,6 +79,11 @@ global: worker: replicas: 3 + # Same unauthenticated_age cap as server — both the server (Django session + # middleware) and worker (cleanup tasks) need to see the value. + env: + - name: AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE + value: "hours=2" strategy: type: RollingUpdate rollingUpdate: diff --git a/stacks/ebooks/main.tf b/stacks/ebooks/main.tf index 7500e579..c6978a05 100644 --- a/stacks/ebooks/main.tf +++ b/stacks/ebooks/main.tf @@ -785,8 +785,18 @@ resource "kubernetes_deployment" "book_search" { } } env { - name = "SMTP_HOST" - value = "mail.viktorbarzin.me" + name = "SMTP_HOST" + # Use intra-cluster ClusterIP path — bypasses pfSense HAProxy + + # PROXY v2 (the public path hairpins through HAProxy:587 → + # NodePort → pod :5587 where Postfix's smtpd-proxy587 daemon + # crashes ~50% of HAProxy healthchecks with + # `smtpd_peer_hostaddr_to_sockaddr: ... Servname not supported`, + # producing intermittent 6s TCP timeouts for clients that land + # mid-respawn). The ClusterIP service points to pod port 587 + # (stock submission daemon, no PROXY) and is rock-solid (12/12 + # in <31ms vs 6/12 timeouts on the public path). + # See docs/runbooks/mailserver-pfsense-haproxy.md. + value = "mailserver.mailserver.svc.cluster.local" } env { name = "SMTP_PORT" diff --git a/stacks/kms/main.tf b/stacks/kms/main.tf index 1ad91cd2..3b758159 100644 --- a/stacks/kms/main.tf +++ b/stacks/kms/main.tf @@ -124,7 +124,7 @@ resource "kubernetes_service" "kms-web-page" { module "ingress" { source = "../../modules/kubernetes/ingress_factory" - dns_type = "proxied" + dns_type = "non-proxied" namespace = kubernetes_namespace.kms.metadata[0].name name = "kms" tls_secret_name = var.tls_secret_name diff --git a/stacks/mailserver/modules/mailserver/main.tf b/stacks/mailserver/modules/mailserver/main.tf index 43a113e3..c3c33d26 100644 --- a/stacks/mailserver/modules/mailserver/main.tf +++ b/stacks/mailserver/modules/mailserver/main.tf @@ -733,6 +733,35 @@ resource "kubernetes_service" "mailserver_proxy" { target_port = 10993 node_port = 30128 } + # Dedicated non-PROXY healthcheck NodePorts. HAProxy on pfSense uses + # `option smtpchk` against these stock pod ports (25/465/587, no PROXY) + # so health probes don't hit the smtpd_peer_hostaddr_to_sockaddr fatal + # that fires on PROXY-v2 LOCAL/AF_UNSPEC frames sent during checks. The + # data path (30125-30128 → 2525/4465/5587/10993) still gets PROXY v2 for + # real client IP visibility — only the healthcheck path is split off. + # See infra/scripts/pfsense-haproxy-bootstrap.php (`check port` directive) + # and docs/runbooks/mailserver-pfsense-haproxy.md. + port { + name = "smtp-check" + protocol = "TCP" + port = 2500 + target_port = 25 + node_port = 30145 + } + port { + name = "smtps-check" + protocol = "TCP" + port = 4650 + target_port = 465 + node_port = 30146 + } + port { + name = "sub-check" + protocol = "TCP" + port = 5870 + target_port = 587 + node_port = 30147 + } } } @@ -829,11 +858,32 @@ DOMAIN = "viktorbarzin.me" marker = f"e2e-probe-{uuid.uuid4().hex[:12]}" subject = f"[E2E Monitor] {marker}" +recipient = f"smoke-test@{DOMAIN}" start = time.time() success = 0 duration = 0 try: + # Step 0: Defensive unblock. Brevo permanently blocks a recipient after a + # single hardBounce — once blocked, every subsequent /smtp/email request + # returns 201 but the message is silently dropped (event=blocked). + # Single transient pod outage → permanent probe outage. Idempotent: 204 if + # the recipient was blocked, 404 if not blocked — both are fine. + # NOTE: this script is wrapped in shell single quotes (see the python3 -c + # invocation above). Do NOT use apostrophes anywhere here, including in + # comments — a stray apostrophe terminates the shell string and Python + # only sees the prefix, raising IndentationError on this try block. + try: + unblock = requests.delete( + f"https://api.brevo.com/v3/smtp/blockedContacts/{recipient}", + headers={"api-key": BREVO_API_KEY, "Accept": "application/json"}, + timeout=10, + ) + if unblock.status_code == 204: + print(f"WARN: {recipient} was blocked at Brevo, unblocked") + except Exception as ue: + print(f"Unblock attempt failed (non-critical): {ue}") + # Step 1: Send via Brevo Transactional Email API to smoke-test@ (hits catch-all -> spam@) resp = requests.post( "https://api.brevo.com/v3/smtp/email", @@ -844,7 +894,7 @@ try: }, json={ "sender": {"name": "Monitoring", "email": f"monitoring@{DOMAIN}"}, - "to": [{"email": f"smoke-test@{DOMAIN}"}], + "to": [{"email": recipient}], "subject": subject, "textContent": f"E2E email monitoring probe {marker}. Auto-generated, will be deleted.", }, diff --git a/stacks/monitoring/modules/monitoring/dashboards/wealth.json b/stacks/monitoring/modules/monitoring/dashboards/wealth.json index 1b8a8aed..c5719a53 100644 --- a/stacks/monitoring/modules/monitoring/dashboards/wealth.json +++ b/stacks/monitoring/modules/monitoring/dashboards/wealth.json @@ -9,6 +9,20 @@ "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "type": "dashboard" + }, + { + "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, + "enable": true, + "hide": false, + "iconColor": "purple", + "name": "Milestones", + "target": { + "rawQuery": true, + "editorMode": "code", + "format": "table", + "refId": "Anno", + "rawSql": "WITH daily AS (SELECT d.valuation_date, SUM(d.total_value) AS nw FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date), crossings AS (SELECT t, (SELECT MIN(valuation_date) FROM daily WHERE nw >= t::numeric) AS d FROM unnest(ARRAY[100000, 250000, 500000, 750000, 1000000]) AS t) SELECT d::timestamptz AS time, '£' || CASE WHEN t >= 1000000 THEN (t/1000000)::int::text || 'M' ELSE (t/1000)::int::text || 'k' END AS text FROM crossings WHERE d IS NOT NULL ORDER BY d" + } } ] }, @@ -23,12 +37,12 @@ "title": "Net worth (current)", "type": "stat", "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, - "gridPos": {"h": 4, "w": 5, "x": 0, "y": 0}, + "gridPos": {"h": 4, "w": 4, "x": 0, "y": 0}, "fieldConfig": { "defaults": { "unit": "currencyGBP", "color": {"mode": "fixed", "fixedColor": "green"}, - "decimals": 0 + "decimals": 2 }, "overrides": [] }, @@ -47,7 +61,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "SELECT SUM(total_value) AS net_worth FROM daily_account_valuation WHERE valuation_date = (SELECT MAX(valuation_date) FROM daily_account_valuation)" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.total_value, d.net_contribution, d.cash_balance, d.investment_market_value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC) SELECT SUM(total_value) AS net_worth FROM latest" } ] }, @@ -57,12 +71,12 @@ "description": "Total deposits minus withdrawals across all accounts.", "type": "stat", "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, - "gridPos": {"h": 4, "w": 5, "x": 5, "y": 0}, + "gridPos": {"h": 4, "w": 4, "x": 4, "y": 0}, "fieldConfig": { "defaults": { "unit": "currencyGBP", "color": {"mode": "fixed", "fixedColor": "blue"}, - "decimals": 0 + "decimals": 2 }, "overrides": [] }, @@ -81,7 +95,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "SELECT SUM(net_contribution) AS contribution FROM daily_account_valuation WHERE valuation_date = (SELECT MAX(valuation_date) FROM daily_account_valuation)" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.total_value, d.net_contribution, d.cash_balance, d.investment_market_value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC) SELECT SUM(net_contribution) AS contribution FROM latest" } ] }, @@ -91,12 +105,12 @@ "description": "Net worth minus net contribution — the gain on everything you've put in.", "type": "stat", "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, - "gridPos": {"h": 4, "w": 5, "x": 10, "y": 0}, + "gridPos": {"h": 4, "w": 4, "x": 8, "y": 0}, "fieldConfig": { "defaults": { "unit": "currencyGBP", "color": {"mode": "thresholds"}, - "decimals": 0, + "decimals": 2, "thresholds": { "mode": "absolute", "steps": [ @@ -122,7 +136,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "SELECT (SUM(total_value) - SUM(net_contribution)) AS growth FROM daily_account_valuation WHERE valuation_date = (SELECT MAX(valuation_date) FROM daily_account_valuation)" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.total_value, d.net_contribution, d.cash_balance, d.investment_market_value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC) SELECT (SUM(total_value) - SUM(net_contribution)) AS growth FROM latest" } ] }, @@ -132,12 +146,12 @@ "description": "Growth / net contribution × 100. Excludes accounts with zero/negative contribution (Schwab) to avoid distortion.", "type": "stat", "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, - "gridPos": {"h": 4, "w": 5, "x": 15, "y": 0}, + "gridPos": {"h": 4, "w": 3, "x": 12, "y": 0}, "fieldConfig": { "defaults": { "unit": "percent", "color": {"mode": "thresholds"}, - "decimals": 1, + "decimals": 2, "thresholds": { "mode": "absolute", "steps": [ @@ -164,7 +178,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "WITH latest AS (SELECT * FROM daily_account_valuation WHERE valuation_date = (SELECT MAX(valuation_date) FROM daily_account_valuation) AND net_contribution > 0) SELECT (SUM(total_value - net_contribution) / NULLIF(SUM(net_contribution), 0) * 100) AS roi_pct FROM latest" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.total_value, d.net_contribution, d.cash_balance, d.investment_market_value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC) SELECT (SUM(total_value - net_contribution) / NULLIF(SUM(net_contribution), 0) * 100) AS roi_pct FROM latest WHERE net_contribution > 0" } ] }, @@ -179,6 +193,7 @@ "defaults": { "color": {"mode": "fixed", "fixedColor": "green"}, "unit": "currencyGBP", + "decimals": 2, "custom": { "drawStyle": "line", "lineWidth": 2, @@ -208,7 +223,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "SELECT valuation_date::timestamp AS \"time\", SUM(total_value) AS net_worth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) GROUP BY valuation_date ORDER BY valuation_date" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)) SELECT valuation_date::timestamp AS \"time\", SUM(total_value) AS net_worth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date <= (SELECT d FROM max_complete) GROUP BY valuation_date ORDER BY valuation_date" } ] }, @@ -223,6 +238,7 @@ "defaults": { "color": {"mode": "palette-classic"}, "unit": "currencyGBP", + "decimals": 2, "custom": { "drawStyle": "line", "lineWidth": 2, @@ -262,7 +278,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "SELECT valuation_date::timestamp AS \"time\", SUM(net_contribution) AS net_contribution, SUM(total_value) AS market_value FROM daily_account_valuation WHERE $__timeFilter(valuation_date) GROUP BY valuation_date ORDER BY valuation_date" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)) SELECT valuation_date::timestamp AS \"time\", SUM(net_contribution) AS net_contribution, SUM(total_value) AS market_value FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date <= (SELECT d FROM max_complete) GROUP BY valuation_date ORDER BY valuation_date" } ] }, @@ -277,6 +293,7 @@ "defaults": { "color": {"mode": "fixed", "fixedColor": "#56A64B"}, "unit": "currencyGBP", + "decimals": 2, "custom": { "drawStyle": "line", "lineWidth": 2, @@ -307,7 +324,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "SELECT valuation_date::timestamp AS \"time\", (SUM(total_value) - SUM(net_contribution)) AS growth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) GROUP BY valuation_date ORDER BY valuation_date" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)) SELECT valuation_date::timestamp AS \"time\", (SUM(total_value) - SUM(net_contribution)) AS growth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date <= (SELECT d FROM max_complete) GROUP BY valuation_date ORDER BY valuation_date" } ] }, @@ -322,6 +339,7 @@ "defaults": { "color": {"mode": "palette-classic"}, "unit": "currencyGBP", + "decimals": 2, "custom": { "drawStyle": "line", "lineWidth": 1, @@ -346,7 +364,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "SELECT d.valuation_date::timestamp AS \"time\", a.name AS metric, d.total_value AS value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) ORDER BY d.valuation_date, a.name" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)) SELECT d.valuation_date::timestamp AS \"time\", a.name AS metric, d.total_value AS value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) AND d.valuation_date <= (SELECT d FROM max_complete) ORDER BY d.valuation_date, a.name" } ] }, @@ -361,6 +379,7 @@ "defaults": { "color": {"mode": "palette-classic"}, "unit": "currencyGBP", + "decimals": 2, "custom": { "drawStyle": "line", "lineWidth": 1, @@ -400,7 +419,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "SELECT d.valuation_date::timestamp AS \"time\", SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN 0 ELSE d.cash_balance END) AS cash, SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN d.cash_balance + d.investment_market_value ELSE d.investment_market_value END) AS invested FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) GROUP BY d.valuation_date ORDER BY d.valuation_date" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)) SELECT d.valuation_date::timestamp AS \"time\", SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN 0 ELSE d.cash_balance END) AS cash, SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN d.cash_balance + d.investment_market_value ELSE d.investment_market_value END) AS invested FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) AND d.valuation_date <= (SELECT d FROM max_complete) GROUP BY d.valuation_date ORDER BY d.valuation_date" } ] }, @@ -443,7 +462,7 @@ "description": "Modified-Dietz return over the trailing 12 months: market_gain / (nw_12mo_ago + 0.5 × contributions_12mo). Excludes new money in — answers 'how did my investments perform' rather than 'how much did my net worth change'.", "type": "stat", "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, - "gridPos": {"h": 4, "w": 4, "x": 20, "y": 0}, + "gridPos": {"h": 4, "w": 3, "x": 15, "y": 0}, "fieldConfig": { "defaults": { "unit": "percent", @@ -475,7 +494,82 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "WITH bounds AS (SELECT (SELECT MAX(valuation_date) FROM daily_account_valuation) AS d_now, (SELECT MIN(valuation_date) FROM daily_account_valuation WHERE valuation_date >= (SELECT MAX(valuation_date) - INTERVAL '12 months' FROM daily_account_valuation)) AS d_ago), agg AS (SELECT (SELECT SUM(total_value) FROM daily_account_valuation WHERE valuation_date = b.d_now) AS nw_now, (SELECT SUM(net_contribution) FROM daily_account_valuation WHERE valuation_date = b.d_now) AS contrib_now, (SELECT SUM(total_value) FROM daily_account_valuation WHERE valuation_date = b.d_ago) AS nw_ago, (SELECT SUM(net_contribution) FROM daily_account_valuation WHERE valuation_date = b.d_ago) AS contrib_ago FROM bounds b) SELECT ROUND((((nw_now - nw_ago - (contrib_now - contrib_ago)) / NULLIF(nw_ago + 0.5 * (contrib_now - contrib_ago), 0)) * 100)::numeric, 2) AS pct_12mo FROM agg" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.valuation_date AS d_now, d.total_value AS nw_now, d.net_contribution AS contrib_now FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC), ago AS (SELECT DISTINCT ON (l.account_id) l.account_id, d.total_value AS nw_ago, d.net_contribution AS contrib_ago FROM latest l JOIN daily_account_valuation d ON d.account_id = l.account_id AND d.valuation_date <= l.d_now - INTERVAL '12 months' ORDER BY l.account_id, d.valuation_date DESC), agg AS (SELECT (SELECT SUM(nw_now) FROM latest) AS nw_now, (SELECT SUM(contrib_now) FROM latest) AS contrib_now, (SELECT SUM(nw_ago) FROM ago) AS nw_ago, (SELECT SUM(contrib_ago) FROM ago) AS contrib_ago) SELECT ROUND((((nw_now - nw_ago - (contrib_now - contrib_ago)) / NULLIF(nw_ago + 0.5 * (contrib_now - contrib_ago), 0)) * 100)::numeric, 2) AS pct_12mo FROM agg" + } + ] + }, + { + "id": 15, + "title": "12mo contrib", + "description": "Net contributions (deposits − withdrawals) over the trailing 12 months. How much new money you put in — independent of market movement.", + "type": "stat", + "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, + "gridPos": {"h": 4, "w": 3, "x": 18, "y": 0}, + "fieldConfig": { + "defaults": { + "unit": "currencyGBP", + "color": {"mode": "fixed", "fixedColor": "blue"}, + "decimals": 2 + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "center", + "orientation": "auto", + "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, + "textMode": "auto" + }, + "targets": [ + { + "refId": "A", + "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, + "rawQuery": true, + "editorMode": "code", + "format": "table", + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.valuation_date AS d_now, d.total_value AS nw_now, d.net_contribution AS contrib_now FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC), ago AS (SELECT DISTINCT ON (l.account_id) l.account_id, d.total_value AS nw_ago, d.net_contribution AS contrib_ago FROM latest l JOIN daily_account_valuation d ON d.account_id = l.account_id AND d.valuation_date <= l.d_now - INTERVAL '12 months' ORDER BY l.account_id, d.valuation_date DESC), agg AS (SELECT (SELECT SUM(nw_now) FROM latest) AS nw_now, (SELECT SUM(contrib_now) FROM latest) AS contrib_now, (SELECT SUM(nw_ago) FROM ago) AS nw_ago, (SELECT SUM(contrib_ago) FROM ago) AS contrib_ago) SELECT (contrib_now - contrib_ago) AS contrib_12mo FROM agg" + } + ] + }, + { + "id": 16, + "title": "12mo gain", + "description": "Trailing 12-month market gain in £ — the change in net worth minus net contributions. What the markets gave you, separate from money you added in.", + "type": "stat", + "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, + "gridPos": {"h": 4, "w": 3, "x": 21, "y": 0}, + "fieldConfig": { + "defaults": { + "unit": "currencyGBP", + "color": {"mode": "thresholds"}, + "decimals": 2, + "thresholds": { + "mode": "absolute", + "steps": [ + {"color": "red", "value": null}, + {"color": "green", "value": 0} + ] + } + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "center", + "orientation": "auto", + "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, + "textMode": "auto" + }, + "targets": [ + { + "refId": "A", + "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, + "rawQuery": true, + "editorMode": "code", + "format": "table", + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.valuation_date AS d_now, d.total_value AS nw_now, d.net_contribution AS contrib_now FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC), ago AS (SELECT DISTINCT ON (l.account_id) l.account_id, d.total_value AS nw_ago, d.net_contribution AS contrib_ago FROM latest l JOIN daily_account_valuation d ON d.account_id = l.account_id AND d.valuation_date <= l.d_now - INTERVAL '12 months' ORDER BY l.account_id, d.valuation_date DESC), agg AS (SELECT (SELECT SUM(nw_now) FROM latest) AS nw_now, (SELECT SUM(contrib_now) FROM latest) AS contrib_now, (SELECT SUM(nw_ago) FROM ago) AS nw_ago, (SELECT SUM(contrib_ago) FROM ago) AS contrib_ago) SELECT ((nw_now - nw_ago) - (contrib_now - contrib_ago)) AS gain_12mo FROM agg" } ] }, @@ -535,7 +629,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "WITH yearly AS (SELECT EXTRACT(YEAR FROM valuation_date)::int AS yr, valuation_date, SUM(total_value) AS nw, SUM(net_contribution) AS contrib FROM daily_account_valuation GROUP BY valuation_date), endpoints AS (SELECT yr, (array_agg(nw ORDER BY valuation_date ASC))[1] AS nw_start, (array_agg(nw ORDER BY valuation_date DESC))[1] AS nw_end, (array_agg(contrib ORDER BY valuation_date ASC))[1] AS contrib_start, (array_agg(contrib ORDER BY valuation_date DESC))[1] AS contrib_end FROM yearly GROUP BY yr) SELECT yr::text AS year, ROUND((((nw_end - nw_start - (contrib_end - contrib_start)) / NULLIF(nw_start + 0.5 * (contrib_end - contrib_start), 0)) * 100)::numeric, 2) AS return_pct FROM endpoints WHERE (nw_start + 0.5 * (contrib_end - contrib_start)) > 0 ORDER BY yr" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)), yearly AS (SELECT EXTRACT(YEAR FROM valuation_date)::int AS yr, valuation_date, SUM(total_value) AS nw, SUM(net_contribution) AS contrib FROM daily_account_valuation WHERE valuation_date <= (SELECT d FROM max_complete) GROUP BY valuation_date), endpoints AS (SELECT yr, (array_agg(nw ORDER BY valuation_date ASC))[1] AS nw_start, (array_agg(nw ORDER BY valuation_date DESC))[1] AS nw_end, (array_agg(contrib ORDER BY valuation_date ASC))[1] AS contrib_start, (array_agg(contrib ORDER BY valuation_date DESC))[1] AS contrib_end FROM yearly GROUP BY yr) SELECT yr::text AS year, ROUND((((nw_end - nw_start - (contrib_end - contrib_start)) / NULLIF(nw_start + 0.5 * (contrib_end - contrib_start), 0)) * 100)::numeric, 2) AS return_pct FROM endpoints WHERE (nw_start + 0.5 * (contrib_end - contrib_start)) > 0 ORDER BY yr" } ] }, @@ -550,7 +644,7 @@ "defaults": { "color": {"mode": "palette-classic"}, "unit": "currencyGBP", - "decimals": 0, + "decimals": 2, "custom": { "axisPlacement": "auto", "axisLabel": "", @@ -601,7 +695,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "WITH yearly AS (SELECT EXTRACT(YEAR FROM valuation_date)::int AS yr, valuation_date, SUM(total_value) AS nw, SUM(net_contribution) AS contrib FROM daily_account_valuation GROUP BY valuation_date), endpoints AS (SELECT yr, (array_agg(nw ORDER BY valuation_date ASC))[1] AS nw_start, (array_agg(nw ORDER BY valuation_date DESC))[1] AS nw_end, (array_agg(contrib ORDER BY valuation_date ASC))[1] AS contrib_start, (array_agg(contrib ORDER BY valuation_date DESC))[1] AS contrib_end FROM yearly GROUP BY yr) SELECT yr::text AS year, ROUND((contrib_end - contrib_start)::numeric, 0) AS contributions, ROUND((nw_end - nw_start - (contrib_end - contrib_start))::numeric, 0) AS market_gain FROM endpoints ORDER BY yr" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)), yearly AS (SELECT EXTRACT(YEAR FROM valuation_date)::int AS yr, valuation_date, SUM(total_value) AS nw, SUM(net_contribution) AS contrib FROM daily_account_valuation WHERE valuation_date <= (SELECT d FROM max_complete) GROUP BY valuation_date), endpoints AS (SELECT yr, (array_agg(nw ORDER BY valuation_date ASC))[1] AS nw_start, (array_agg(nw ORDER BY valuation_date DESC))[1] AS nw_end, (array_agg(contrib ORDER BY valuation_date ASC))[1] AS contrib_start, (array_agg(contrib ORDER BY valuation_date DESC))[1] AS contrib_end FROM yearly GROUP BY yr) SELECT yr::text AS year, ROUND((contrib_end - contrib_start)::numeric, 0) AS contributions, ROUND((nw_end - nw_start - (contrib_end - contrib_start))::numeric, 0) AS market_gain FROM endpoints ORDER BY yr" } ] }, @@ -616,7 +710,7 @@ "defaults": { "color": {"mode": "thresholds"}, "unit": "percent", - "decimals": 1, + "decimals": 2, "thresholds": { "mode": "absolute", "steps": [ @@ -653,7 +747,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "SELECT a.name AS account, ROUND(((d.total_value - d.net_contribution) / NULLIF(d.net_contribution, 0) * 100)::numeric, 2) AS roi_pct FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE d.valuation_date = (SELECT MAX(valuation_date) FROM daily_account_valuation) AND d.net_contribution > 0 ORDER BY roi_pct DESC" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) a.name, d.total_value, d.net_contribution FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC) SELECT name AS account, ROUND(((total_value - net_contribution) / NULLIF(net_contribution, 0) * 100)::numeric, 2) AS roi_pct FROM latest WHERE net_contribution > 0 ORDER BY roi_pct DESC" } ] } @@ -662,7 +756,7 @@ "schemaVersion": 39, "tags": ["finance", "personal", "wealth"], "templating": {"list": []}, - "time": {"from": "now-5y", "to": "now"}, + "time": {"from": "now-180d", "to": "now"}, "timepicker": {}, "timezone": "browser", "title": "Wealth", diff --git a/stacks/phpipam/main.tf b/stacks/phpipam/main.tf index c277ba21..9fce6c14 100644 --- a/stacks/phpipam/main.tf +++ b/stacks/phpipam/main.tf @@ -386,7 +386,7 @@ resource "kubernetes_cron_job_v1" "phpipam_pfsense_import" { namespace = kubernetes_namespace.phpipam.metadata[0].name } spec { - schedule = "*/5 * * * *" + schedule = "0 * * * *" successful_jobs_history_limit = 1 failed_jobs_history_limit = 3 concurrency_policy = "Forbid" diff --git a/stacks/priority-pass/main.tf b/stacks/priority-pass/main.tf index a9e62e7d..d02aa651 100644 --- a/stacks/priority-pass/main.tf +++ b/stacks/priority-pass/main.tf @@ -1,8 +1,19 @@ +variable "image_tag" { + type = string + default = "7c01448d" + description = "priority-pass image tag (applies to both frontend + backend). Use 8-char git SHA in CI; :latest only for local trials." +} + variable "tls_secret_name" { type = string sensitive = true } +locals { + frontend_image = "docker.io/viktorbarzin/priority-pass-frontend:${var.image_tag}" + backend_image = "docker.io/viktorbarzin/priority-pass-backend:${var.image_tag}" +} + resource "kubernetes_namespace" "priority-pass" { metadata { name = "priority-pass" @@ -23,6 +34,26 @@ module "tls_secret" { tls_secret_name = var.tls_secret_name } +resource "kubernetes_persistent_volume_claim" "uploads" { + wait_until_bound = false + metadata { + name = "priority-pass-uploads" + namespace = kubernetes_namespace.priority-pass.metadata[0].name + annotations = { + "resize.topolvm.io/threshold" = "80%" + "resize.topolvm.io/increase" = "100%" + "resize.topolvm.io/storage_limit" = "10Gi" + } + } + spec { + access_modes = ["ReadWriteOnce"] + storage_class_name = "proxmox-lvm-encrypted" + resources { + requests = { storage = "1Gi" } + } + } +} + resource "kubernetes_deployment" "priority-pass" { metadata { name = "priority-pass" @@ -34,6 +65,9 @@ resource "kubernetes_deployment" "priority-pass" { } spec { replicas = 1 + strategy { + type = "Recreate" + } selector { match_labels = { run = "priority-pass" @@ -49,9 +83,15 @@ resource "kubernetes_deployment" "priority-pass" { image_pull_secrets { name = "registry-credentials" } + volume { + name = "uploads" + persistent_volume_claim { + claim_name = kubernetes_persistent_volume_claim.uploads.metadata[0].name + } + } container { name = "frontend" - image = "registry.viktorbarzin.me/priority-pass-frontend:v5" + image = local.frontend_image port { container_port = 3000 } @@ -75,10 +115,18 @@ resource "kubernetes_deployment" "priority-pass" { } container { name = "backend" - image = "registry.viktorbarzin.me/priority-pass-backend:v8" + image = local.backend_image port { container_port = 8000 } + env { + name = "UPLOAD_DIR" + value = "/data/uploads" + } + volume_mount { + name = "uploads" + mount_path = "/data/uploads" + } resources { limits = { memory = "512Mi" diff --git a/stacks/priority-pass/terragrunt.hcl b/stacks/priority-pass/terragrunt.hcl deleted file mode 120000 index af58a92e..00000000 --- a/stacks/priority-pass/terragrunt.hcl +++ /dev/null @@ -1 +0,0 @@ -../../terragrunt.hcl \ No newline at end of file diff --git a/stacks/priority-pass/terragrunt.hcl b/stacks/priority-pass/terragrunt.hcl new file mode 100644 index 00000000..42b52856 --- /dev/null +++ b/stacks/priority-pass/terragrunt.hcl @@ -0,0 +1,25 @@ +include "root" { + path = find_in_parent_folders() +} + +dependency "platform" { + config_path = "../platform" + skip_outputs = true +} + +dependency "vault" { + config_path = "../vault" + skip_outputs = true +} + +dependency "external-secrets" { + config_path = "../external-secrets" + skip_outputs = true +} + +inputs = { + # priority-pass repo HEAD — auto-bumped by GHA `build-and-deploy.yml` + # on every successful build. Manual edits welcome for local trials, + # but CI will overwrite on the next push to main. + image_tag = "88f18e53" +} diff --git a/stacks/wealthfolio/main.tf b/stacks/wealthfolio/main.tf index df4dca48..672f4ca1 100644 --- a/stacks/wealthfolio/main.tf +++ b/stacks/wealthfolio/main.tf @@ -660,3 +660,101 @@ resource "kubernetes_config_map" "grafana_wealth_datasource" { # See `resource "kubernetes_deployment" "wealthfolio"` above — the sidecar # is wired in via the deployment's container/volume blocks. ############################################################################ + +############################################################################ +# Daily portfolio-recalc CronJob — keeps the Grafana wealth dashboard fresh. +# +# Wealthfolio writes new `daily_account_valuation` rows ONLY when a +# PortfolioJob fires with ValuationRecalcMode != None. None of its built-in +# schedulers do that for our deployment: +# * Internal 6h quote scheduler — refreshes the `quotes` table only. +# * Internal 4h broker scheduler — short-circuits if `sync_refresh_token` +# is unset (it is — we route broker imports through the external +# wealthfolio-sync CronJob). +# Result: valuations only update when the Tauri/web UI hits +# /api/v1/market-data/sync — i.e. when someone opens the dashboard. +# +# This CronJob mimics that: login → POST /api/v1/market-data/sync. The +# server runs the portfolio job (Incremental quote sync + IncrementalFromLast +# valuation recalc), backfilling missing daily_account_valuation rows up to +# today. The pg-sync sidecar's :07 hourly tick mirrors them to PG, and +# Grafana auto-refreshes within 5 min. +# +# Schedule 16:00 UTC (= 17:00 BST in summer): +# - After UK market close (15:30 UTC BST), so EOD UK prices are settled +# - US market open ~2.5h (good intra-day US quotes) +# - pg-sync next tick at 16:07 → Grafana fresh by ~16:12 UTC ≈ 17:12 BST, +# well before the 18:00 BST "fresh data by 6pm" target. +# +# Plaintext password lives at Vault `secret/wealthfolio.web_password`, +# pulled into the existing `wealthfolio-secrets` K8s Secret by the +# `dataFrom.extract` ExternalSecret above (no extra ESO wiring needed — +# the new key flows through automatically). +############################################################################ +resource "kubernetes_cron_job_v1" "wealthfolio_daily_sync" { + metadata { + name = "wealthfolio-daily-sync" + namespace = kubernetes_namespace.wealthfolio.metadata[0].name + } + + spec { + schedule = "0 16 * * *" + successful_jobs_history_limit = 1 + failed_jobs_history_limit = 3 + concurrency_policy = "Forbid" + + job_template { + metadata {} + spec { + active_deadline_seconds = 180 + backoff_limit = 1 + template { + metadata {} + spec { + restart_policy = "Never" + + container { + name = "curl" + image = "curlimages/curl:8.11.1" + env { + name = "WF_PASSWORD" + value_from { + secret_key_ref { + name = "wealthfolio-secrets" + key = "web_password" + } + } + } + command = ["/bin/sh", "-c"] + args = [ + <<-EOT + set -eu + BASE=http://wealthfolio.wealthfolio.svc.cluster.local + JAR=$(mktemp) + trap 'rm -f "$JAR"' EXIT + + echo "[$(date -u +%FT%TZ)] login" + curl -sS --max-time 15 --fail -X POST "$BASE/api/v1/auth/login" \ + -H "Content-Type: application/json" \ + -d "{\"password\":\"$WF_PASSWORD\"}" \ + -c "$JAR" -o /dev/null + + echo "[$(date -u +%FT%TZ)] POST /api/v1/market-data/sync" + curl -sS --max-time 60 --fail -X POST "$BASE/api/v1/market-data/sync" \ + -H "Content-Type: application/json" \ + -b "$JAR" \ + -d '{"refetchAll":false}' -o /dev/null + echo "[$(date -u +%FT%TZ)] sync queued (204) — portfolio job runs async" + EOT + ] + } + } + } + } + } + } + lifecycle { + # KYVERNO_LIFECYCLE_V1: Kyverno admission webhook mutates dns_config with ndots=2 + ignore_changes = [spec[0].job_template[0].spec[0].template[0].spec[0].dns_config] + } +}