From cd96fb64a8881b1fd1b632f9abae595f012de1f0 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Sun, 26 Apr 2026 22:48:43 +0000 Subject: [PATCH 01/25] =?UTF-8?q?phpipam-pfsense-import:=20every=205min=20?= =?UTF-8?q?=E2=86=92=20hourly?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reduces 5-min disk-write spikes on PVE sdc. The cronjob was the heaviest single contributor in our hourly fan-out investigation (11.2 MB/s burst when it fired). Kea DDNS still handles real-time DNS auto-registration; phpIPAM inventory just lags by up to 1h, which we don't need fresher. Docs (dns.md, networking.md, .claude/CLAUDE.md) updated to match. --- .claude/CLAUDE.md | 4 ++-- docs/architecture/dns.md | 4 ++-- docs/architecture/networking.md | 4 ++-- stacks/phpipam/main.tf | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 55cbc0d1..98dacd41 100755 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -117,7 +117,7 @@ Repo IDs: infra=1, Website=2, finance=3, health=4, travel_blog=5, webhook-handle - **Rate limiting**: Return 429 (not 503). Per-service tuning: Immich/Nextcloud need higher limits. - **Retry middleware**: 2 attempts, 100ms — in default ingress chain. - **HTTP/3 (QUIC)**: Enabled cluster-wide via Traefik. -- **IPAM & DNS auto-registration**: pfSense Kea DHCP serves all 3 subnets (VLAN 10, VLAN 20, 192.168.1.x). Kea DDNS auto-registers every DHCP client in Technitium (RFC 2136, A+PTR). CronJob `phpipam-pfsense-import` (5min) pulls Kea leases + ARP into phpIPAM via SSH (passive, no scanning). CronJob `phpipam-dns-sync` (15min) bidirectional sync phpIPAM ↔ Technitium. 42 MAC reservations for 192.168.1.x. +- **IPAM & DNS auto-registration**: pfSense Kea DHCP serves all 3 subnets (VLAN 10, VLAN 20, 192.168.1.x). Kea DDNS auto-registers every DHCP client in Technitium (RFC 2136, A+PTR). CronJob `phpipam-pfsense-import` (hourly) pulls Kea leases + ARP into phpIPAM via SSH (passive, no scanning). CronJob `phpipam-dns-sync` (15min) bidirectional sync phpIPAM ↔ Technitium. 42 MAC reservations for 192.168.1.x. ## Service-Specific Notes | Service | Key Operational Knowledge | @@ -129,7 +129,7 @@ Repo IDs: infra=1, Website=2, finance=3, health=4, travel_blog=5, webhook-handle | Authentik | 3 replicas, PgBouncer in front of PostgreSQL, strip auth headers before forwarding | | Kyverno | failurePolicy=Ignore to prevent blocking cluster, pin chart version | | MySQL Standalone | Raw `kubernetes_stateful_set_v1` with `mysql:8.4` (migrated from InnoDB Cluster 2026-04-16). `skip-log-bin`, `innodb_flush_log_at_trx_commit=2`, `innodb_doublewrite=ON`. ConfigMap `mysql-standalone-cnf`. PVC `data-mysql-standalone-0` (15Gi, `proxmox-lvm-encrypted`). Service `mysql.dbaas` unchanged. Anti-affinity excludes k8s-node1. Old InnoDB Cluster + operator still in TF (Phase 4 cleanup pending). Bitnami charts deprecated (Broadcom Aug 2025) — use official images. | -| phpIPAM | IPAM — no active scanning. `pfsense-import` CronJob (5min) pulls Kea leases + ARP via SSH. `dns-sync` CronJob (15min) bidirectional sync with Technitium. Kea DDNS on pfSense handles all 3 subnets. API app `claude` (ssl_token). | +| phpIPAM | IPAM — no active scanning. `pfsense-import` CronJob (hourly) pulls Kea leases + ARP via SSH. `dns-sync` CronJob (15min) bidirectional sync with Technitium. Kea DDNS on pfSense handles all 3 subnets. API app `claude` (ssl_token). | ## Monitoring & Alerting - Alert cascade inhibitions: if node is down, suppress pod alerts on that node. diff --git a/docs/architecture/dns.md b/docs/architecture/dns.md index 97e609f3..eec99830 100644 --- a/docs/architecture/dns.md +++ b/docs/architecture/dns.md @@ -377,7 +377,7 @@ Devices get automatic DNS registration without manual intervention. See [network Summary: 1. **Kea DHCP** on pfSense assigns IP (53 reservations across 3 subnets). DHCP option 6 (DNS servers) is pushed with two IPs per internal subnet: internal resolver + AdGuard public fallback (`94.140.14.14`) — clients survive an internal DNS outage. 2. **Kea DDNS** sends **TSIG-signed** RFC 2136 dynamic update to Technitium (A + PTR records) — immediate. Key `kea-ddns` (HMAC-SHA256); Technitium enforces both source-IP ACL and TSIG signature on `viktorbarzin.lan` + reverse zones. -3. **phpipam-pfsense-import** CronJob (5min) pulls Kea leases + ARP table into phpIPAM +3. **phpipam-pfsense-import** CronJob (hourly) pulls Kea leases + ARP table into phpIPAM 4. **phpipam-dns-sync** CronJob (15min) pushes named phpIPAM hosts → Technitium A + PTR, pulls Technitium PTR → phpIPAM hostnames ## Automation CronJobs @@ -389,7 +389,7 @@ Summary: | `technitium-split-horizon-sync` | `15 */6 * * *` | technitium | Split Horizon + DNS Rebinding Protection on all 3 instances | | `technitium-dns-optimization` | `30 */6 * * *` | technitium | Min cache TTL 60s, emrsn.org stub zone | | `phpipam-dns-sync` | `*/15 * * * *` | phpipam | Bidirectional phpIPAM ↔ Technitium DNS sync | -| `phpipam-pfsense-import` | `*/5 * * * *` | phpipam | Import Kea DHCP leases + ARP from pfSense | +| `phpipam-pfsense-import` | `0 * * * *` | phpipam | Import Kea DHCP leases + ARP from pfSense | ### Password Rotation Flow diff --git a/docs/architecture/networking.md b/docs/architecture/networking.md index 3c75e4fd..e7959589 100644 --- a/docs/architecture/networking.md +++ b/docs/architecture/networking.md @@ -104,7 +104,7 @@ flowchart LR end subgraph K8s["Kubernetes"] - Import[CronJob
pfsense-import
every 5min] + Import[CronJob
pfsense-import
hourly] Sync[CronJob
dns-sync
every 15min] IPAM[phpIPAM
Web UI + API] MySQL[(MySQL
InnoDB)] @@ -338,7 +338,7 @@ Containerd on all K8s nodes uses `hosts.toml` to redirect pulls to the local cac - Stack: `stacks/phpipam/` - Web UI: `phpipam.viktorbarzin.me` (Authentik-protected) - Database: MySQL InnoDB cluster (`mysql.dbaas.svc.cluster.local`) -- Device import: CronJob `phpipam-pfsense-import` every 5min — queries Kea DHCP leases + pfSense ARP table via SSH (no active scanning) +- Device import: CronJob `phpipam-pfsense-import` hourly — queries Kea DHCP leases + pfSense ARP table via SSH (no active scanning) - DNS sync: CronJob `phpipam-dns-sync` every 15min — bidirectional sync between phpIPAM and Technitium DNS (push named hosts → A+PTR, pull DNS hostnames → unnamed phpIPAM entries) - Subnets tracked: 10.0.10.0/24, 10.0.20.0/24, 192.168.1.0/24, 10.3.2.0/24, 192.168.8.0/24, 192.168.0.0/24 - API: REST API enabled (app `claude`, ssl_token auth), MCP server available for agent access diff --git a/stacks/phpipam/main.tf b/stacks/phpipam/main.tf index c277ba21..9fce6c14 100644 --- a/stacks/phpipam/main.tf +++ b/stacks/phpipam/main.tf @@ -386,7 +386,7 @@ resource "kubernetes_cron_job_v1" "phpipam_pfsense_import" { namespace = kubernetes_namespace.phpipam.metadata[0].name } spec { - schedule = "*/5 * * * *" + schedule = "0 * * * *" successful_jobs_history_limit = 1 failed_jobs_history_limit = 3 concurrency_policy = "Forbid" From 31b9e5d4a9ea7a6e9372625dcde3dd301c8049a5 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Mon, 27 Apr 2026 06:32:53 +0000 Subject: [PATCH 02/25] monitoring(wealth): add 12mo contrib + 12mo gain to top row MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Top row goes from 5 → 7 stat panels (widths 4+4+4+3+3+3+3=24): - Net worth, Net contribution, Growth shrink from w=5 to w=4. - ROI % shrinks from w=5 to w=3 (now sits at x=12). - 12mo return slides from x=20/w=4 to x=15/w=3. - New: 12mo contrib (id=15, currency, blue) at x=18 — net contributions added in the trailing 12 months. - New: 12mo gain (id=16, currency, red/green) at x=21 — pure market gain in £ over the trailing 12 months (12mo Δnet-worth − 12mo contribs). Live values verified against PG: contrib_12mo=£245k, gain_12mo=£172k, sum = £417k = nw_now − nw_ago, return = 23.51%. --- .../modules/monitoring/dashboards/wealth.json | 85 +++++++++++++++++-- 1 file changed, 80 insertions(+), 5 deletions(-) diff --git a/stacks/monitoring/modules/monitoring/dashboards/wealth.json b/stacks/monitoring/modules/monitoring/dashboards/wealth.json index 1b8a8aed..a576f28c 100644 --- a/stacks/monitoring/modules/monitoring/dashboards/wealth.json +++ b/stacks/monitoring/modules/monitoring/dashboards/wealth.json @@ -23,7 +23,7 @@ "title": "Net worth (current)", "type": "stat", "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, - "gridPos": {"h": 4, "w": 5, "x": 0, "y": 0}, + "gridPos": {"h": 4, "w": 4, "x": 0, "y": 0}, "fieldConfig": { "defaults": { "unit": "currencyGBP", @@ -57,7 +57,7 @@ "description": "Total deposits minus withdrawals across all accounts.", "type": "stat", "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, - "gridPos": {"h": 4, "w": 5, "x": 5, "y": 0}, + "gridPos": {"h": 4, "w": 4, "x": 4, "y": 0}, "fieldConfig": { "defaults": { "unit": "currencyGBP", @@ -91,7 +91,7 @@ "description": "Net worth minus net contribution — the gain on everything you've put in.", "type": "stat", "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, - "gridPos": {"h": 4, "w": 5, "x": 10, "y": 0}, + "gridPos": {"h": 4, "w": 4, "x": 8, "y": 0}, "fieldConfig": { "defaults": { "unit": "currencyGBP", @@ -132,7 +132,7 @@ "description": "Growth / net contribution × 100. Excludes accounts with zero/negative contribution (Schwab) to avoid distortion.", "type": "stat", "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, - "gridPos": {"h": 4, "w": 5, "x": 15, "y": 0}, + "gridPos": {"h": 4, "w": 3, "x": 12, "y": 0}, "fieldConfig": { "defaults": { "unit": "percent", @@ -443,7 +443,7 @@ "description": "Modified-Dietz return over the trailing 12 months: market_gain / (nw_12mo_ago + 0.5 × contributions_12mo). Excludes new money in — answers 'how did my investments perform' rather than 'how much did my net worth change'.", "type": "stat", "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, - "gridPos": {"h": 4, "w": 4, "x": 20, "y": 0}, + "gridPos": {"h": 4, "w": 3, "x": 15, "y": 0}, "fieldConfig": { "defaults": { "unit": "percent", @@ -479,6 +479,81 @@ } ] }, + { + "id": 15, + "title": "12mo contrib", + "description": "Net contributions (deposits − withdrawals) over the trailing 12 months. How much new money you put in — independent of market movement.", + "type": "stat", + "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, + "gridPos": {"h": 4, "w": 3, "x": 18, "y": 0}, + "fieldConfig": { + "defaults": { + "unit": "currencyGBP", + "color": {"mode": "fixed", "fixedColor": "blue"}, + "decimals": 0 + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "center", + "orientation": "auto", + "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, + "textMode": "auto" + }, + "targets": [ + { + "refId": "A", + "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, + "rawQuery": true, + "editorMode": "code", + "format": "table", + "rawSql": "WITH bounds AS (SELECT (SELECT MAX(valuation_date) FROM daily_account_valuation) AS d_now, (SELECT MIN(valuation_date) FROM daily_account_valuation WHERE valuation_date >= (SELECT MAX(valuation_date) - INTERVAL '12 months' FROM daily_account_valuation)) AS d_ago) SELECT ((SELECT SUM(net_contribution) FROM daily_account_valuation WHERE valuation_date = b.d_now) - (SELECT SUM(net_contribution) FROM daily_account_valuation WHERE valuation_date = b.d_ago)) AS contrib_12mo FROM bounds b" + } + ] + }, + { + "id": 16, + "title": "12mo gain", + "description": "Trailing 12-month market gain in £ — the change in net worth minus net contributions. What the markets gave you, separate from money you added in.", + "type": "stat", + "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, + "gridPos": {"h": 4, "w": 3, "x": 21, "y": 0}, + "fieldConfig": { + "defaults": { + "unit": "currencyGBP", + "color": {"mode": "thresholds"}, + "decimals": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + {"color": "red", "value": null}, + {"color": "green", "value": 0} + ] + } + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "center", + "orientation": "auto", + "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, + "textMode": "auto" + }, + "targets": [ + { + "refId": "A", + "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, + "rawQuery": true, + "editorMode": "code", + "format": "table", + "rawSql": "WITH bounds AS (SELECT (SELECT MAX(valuation_date) FROM daily_account_valuation) AS d_now, (SELECT MIN(valuation_date) FROM daily_account_valuation WHERE valuation_date >= (SELECT MAX(valuation_date) - INTERVAL '12 months' FROM daily_account_valuation)) AS d_ago), agg AS (SELECT (SELECT SUM(total_value) FROM daily_account_valuation WHERE valuation_date = b.d_now) AS nw_now, (SELECT SUM(net_contribution) FROM daily_account_valuation WHERE valuation_date = b.d_now) AS contrib_now, (SELECT SUM(total_value) FROM daily_account_valuation WHERE valuation_date = b.d_ago) AS nw_ago, (SELECT SUM(net_contribution) FROM daily_account_valuation WHERE valuation_date = b.d_ago) AS contrib_ago FROM bounds b) SELECT ((nw_now - nw_ago) - (contrib_now - contrib_ago)) AS gain_12mo FROM agg" + } + ] + }, { "id": 12, "title": "Yearly investment return %", From 1d3ae01aacac465034123f6e825a49fd28081a33 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Wed, 29 Apr 2026 21:21:24 +0000 Subject: [PATCH 03/25] wealthfolio(daily-sync): API call CronJob, replaces rollout-restart MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restart-only didn't refresh the wealth Grafana dashboard — verified empirically: a fresh `daily_account_valuation` row only lands when a PortfolioJob runs with ValuationRecalcMode != None, and Wealthfolio's internal schedulers don't trigger that path: - 6h quotes scheduler refreshes the `quotes` table only. - 4h broker scheduler short-circuits on missing `sync_refresh_token`. The right knob is `POST /api/v1/market-data/sync`. Replaced the rollout-restart CronJob (+ its SA/Role/RoleBinding) with a curl-based CronJob that logs in (`POST /api/v1/auth/login`) then POSTs to `/api/v1/market-data/sync` with the session cookie. Backfills missing days via IncrementalFromLast in one call. Schedule 16:00 UTC (= 17:00 BST): * After UK market close (15:30 UTC BST), EOD UK prices settled. * US market open ~2.5h, intra-day US quotes fresh. * pg-sync next :07 tick mirrors → Grafana refresh ≤5m → fresh data by ~17:12 BST, comfortably before the 18:00 BST target. Plaintext password lives in Vault `secret/wealthfolio.web_password`, flows via the existing `dataFrom.extract` ExternalSecret — no extra ESO wiring needed. Verified end-to-end: API call backfilled 04-26 through 04-29, pg-sync mirrored, PG now shows rows up to today. --- stacks/wealthfolio/main.tf | 98 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/stacks/wealthfolio/main.tf b/stacks/wealthfolio/main.tf index df4dca48..672f4ca1 100644 --- a/stacks/wealthfolio/main.tf +++ b/stacks/wealthfolio/main.tf @@ -660,3 +660,101 @@ resource "kubernetes_config_map" "grafana_wealth_datasource" { # See `resource "kubernetes_deployment" "wealthfolio"` above — the sidecar # is wired in via the deployment's container/volume blocks. ############################################################################ + +############################################################################ +# Daily portfolio-recalc CronJob — keeps the Grafana wealth dashboard fresh. +# +# Wealthfolio writes new `daily_account_valuation` rows ONLY when a +# PortfolioJob fires with ValuationRecalcMode != None. None of its built-in +# schedulers do that for our deployment: +# * Internal 6h quote scheduler — refreshes the `quotes` table only. +# * Internal 4h broker scheduler — short-circuits if `sync_refresh_token` +# is unset (it is — we route broker imports through the external +# wealthfolio-sync CronJob). +# Result: valuations only update when the Tauri/web UI hits +# /api/v1/market-data/sync — i.e. when someone opens the dashboard. +# +# This CronJob mimics that: login → POST /api/v1/market-data/sync. The +# server runs the portfolio job (Incremental quote sync + IncrementalFromLast +# valuation recalc), backfilling missing daily_account_valuation rows up to +# today. The pg-sync sidecar's :07 hourly tick mirrors them to PG, and +# Grafana auto-refreshes within 5 min. +# +# Schedule 16:00 UTC (= 17:00 BST in summer): +# - After UK market close (15:30 UTC BST), so EOD UK prices are settled +# - US market open ~2.5h (good intra-day US quotes) +# - pg-sync next tick at 16:07 → Grafana fresh by ~16:12 UTC ≈ 17:12 BST, +# well before the 18:00 BST "fresh data by 6pm" target. +# +# Plaintext password lives at Vault `secret/wealthfolio.web_password`, +# pulled into the existing `wealthfolio-secrets` K8s Secret by the +# `dataFrom.extract` ExternalSecret above (no extra ESO wiring needed — +# the new key flows through automatically). +############################################################################ +resource "kubernetes_cron_job_v1" "wealthfolio_daily_sync" { + metadata { + name = "wealthfolio-daily-sync" + namespace = kubernetes_namespace.wealthfolio.metadata[0].name + } + + spec { + schedule = "0 16 * * *" + successful_jobs_history_limit = 1 + failed_jobs_history_limit = 3 + concurrency_policy = "Forbid" + + job_template { + metadata {} + spec { + active_deadline_seconds = 180 + backoff_limit = 1 + template { + metadata {} + spec { + restart_policy = "Never" + + container { + name = "curl" + image = "curlimages/curl:8.11.1" + env { + name = "WF_PASSWORD" + value_from { + secret_key_ref { + name = "wealthfolio-secrets" + key = "web_password" + } + } + } + command = ["/bin/sh", "-c"] + args = [ + <<-EOT + set -eu + BASE=http://wealthfolio.wealthfolio.svc.cluster.local + JAR=$(mktemp) + trap 'rm -f "$JAR"' EXIT + + echo "[$(date -u +%FT%TZ)] login" + curl -sS --max-time 15 --fail -X POST "$BASE/api/v1/auth/login" \ + -H "Content-Type: application/json" \ + -d "{\"password\":\"$WF_PASSWORD\"}" \ + -c "$JAR" -o /dev/null + + echo "[$(date -u +%FT%TZ)] POST /api/v1/market-data/sync" + curl -sS --max-time 60 --fail -X POST "$BASE/api/v1/market-data/sync" \ + -H "Content-Type: application/json" \ + -b "$JAR" \ + -d '{"refetchAll":false}' -o /dev/null + echo "[$(date -u +%FT%TZ)] sync queued (204) — portfolio job runs async" + EOT + ] + } + } + } + } + } + } + lifecycle { + # KYVERNO_LIFECYCLE_V1: Kyverno admission webhook mutates dns_config with ndots=2 + ignore_changes = [spec[0].job_template[0].spec[0].template[0].spec[0].dns_config] + } +} From 628f5a0d26a62569881bcd6b133c081be045a502 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Fri, 1 May 2026 16:08:18 +0000 Subject: [PATCH 04/25] monitoring(wealth): skew-resilient queries, no more partial-day dips MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bug witnessed 2026-05-01: dashboard "Net worth (current)" showed £88k instead of £1.03M because at 02:00 UTC an external trigger refreshed ONE account (Trading212 ISA), creating its 05-01 daily_account_valuation row. The 5 other accounts still had their last row at 04-30. The panel SQL `WHERE valuation_date = (SELECT MAX(valuation_date))` then summed only the single account that had a 05-01 row. Two new SQL patterns adopted across all 15 affected panels: 1. Stat / barchart "current snapshot" panels (1, 2, 3, 4, 11, 14, 15, 16): latest-per-account stitching — WITH latest AS (SELECT DISTINCT ON (d.account_id) ... FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC) gives a coherent "now" snapshot regardless of refresh skew, and the inner join filters out orphan/deleted accounts (one such was adding a stale £33k from 04-17). 12-month panels add a parallel `ago` CTE picking each account's row closest to (d_now - 12mo). 2. Time-series / yearly panels (5, 6, 7, 8, 9, 12, 13): complete-days- only filter — WITH active_accounts AS (SELECT COUNT(*) FROM accounts), complete_dates AS (SELECT valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id=d.account_id GROUP BY valuation_date HAVING COUNT(*) >= active.n) so a partial today never renders as a chart dip. The day rejoins the chart automatically once the daily 16:00 UTC sync writes rows for every account. Verified end-to-end against live PG: new queries produce £1,033,734 (matches the 6 active accounts' true latest sum) where the old query gave £88k. --- .../modules/monitoring/dashboards/wealth.json | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/stacks/monitoring/modules/monitoring/dashboards/wealth.json b/stacks/monitoring/modules/monitoring/dashboards/wealth.json index a576f28c..a09ab92d 100644 --- a/stacks/monitoring/modules/monitoring/dashboards/wealth.json +++ b/stacks/monitoring/modules/monitoring/dashboards/wealth.json @@ -47,7 +47,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "SELECT SUM(total_value) AS net_worth FROM daily_account_valuation WHERE valuation_date = (SELECT MAX(valuation_date) FROM daily_account_valuation)" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.total_value, d.net_contribution, d.cash_balance, d.investment_market_value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC) SELECT SUM(total_value) AS net_worth FROM latest" } ] }, @@ -81,7 +81,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "SELECT SUM(net_contribution) AS contribution FROM daily_account_valuation WHERE valuation_date = (SELECT MAX(valuation_date) FROM daily_account_valuation)" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.total_value, d.net_contribution, d.cash_balance, d.investment_market_value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC) SELECT SUM(net_contribution) AS contribution FROM latest" } ] }, @@ -122,7 +122,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "SELECT (SUM(total_value) - SUM(net_contribution)) AS growth FROM daily_account_valuation WHERE valuation_date = (SELECT MAX(valuation_date) FROM daily_account_valuation)" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.total_value, d.net_contribution, d.cash_balance, d.investment_market_value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC) SELECT (SUM(total_value) - SUM(net_contribution)) AS growth FROM latest" } ] }, @@ -164,7 +164,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "WITH latest AS (SELECT * FROM daily_account_valuation WHERE valuation_date = (SELECT MAX(valuation_date) FROM daily_account_valuation) AND net_contribution > 0) SELECT (SUM(total_value - net_contribution) / NULLIF(SUM(net_contribution), 0) * 100) AS roi_pct FROM latest" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.total_value, d.net_contribution, d.cash_balance, d.investment_market_value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC) SELECT (SUM(total_value - net_contribution) / NULLIF(SUM(net_contribution), 0) * 100) AS roi_pct FROM latest WHERE net_contribution > 0" } ] }, @@ -208,7 +208,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "SELECT valuation_date::timestamp AS \"time\", SUM(total_value) AS net_worth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) GROUP BY valuation_date ORDER BY valuation_date" + "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT valuation_date::timestamp AS \\\"time\\\", SUM(total_value) AS net_worth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date ORDER BY valuation_date" } ] }, @@ -262,7 +262,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "SELECT valuation_date::timestamp AS \"time\", SUM(net_contribution) AS net_contribution, SUM(total_value) AS market_value FROM daily_account_valuation WHERE $__timeFilter(valuation_date) GROUP BY valuation_date ORDER BY valuation_date" + "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT valuation_date::timestamp AS \\\"time\\\", SUM(net_contribution) AS net_contribution, SUM(total_value) AS market_value FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date ORDER BY valuation_date" } ] }, @@ -307,7 +307,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "SELECT valuation_date::timestamp AS \"time\", (SUM(total_value) - SUM(net_contribution)) AS growth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) GROUP BY valuation_date ORDER BY valuation_date" + "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT valuation_date::timestamp AS \\\"time\\\", (SUM(total_value) - SUM(net_contribution)) AS growth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date ORDER BY valuation_date" } ] }, @@ -346,7 +346,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "SELECT d.valuation_date::timestamp AS \"time\", a.name AS metric, d.total_value AS value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) ORDER BY d.valuation_date, a.name" + "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT d.valuation_date::timestamp AS \\\"time\\\", a.name AS metric, d.total_value AS value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) AND d.valuation_date IN (SELECT valuation_date FROM complete_dates) ORDER BY d.valuation_date, a.name" } ] }, @@ -400,7 +400,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "SELECT d.valuation_date::timestamp AS \"time\", SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN 0 ELSE d.cash_balance END) AS cash, SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN d.cash_balance + d.investment_market_value ELSE d.investment_market_value END) AS invested FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) GROUP BY d.valuation_date ORDER BY d.valuation_date" + "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT d.valuation_date::timestamp AS \\\"time\\\", SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN 0 ELSE d.cash_balance END) AS cash, SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN d.cash_balance + d.investment_market_value ELSE d.investment_market_value END) AS invested FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) AND d.valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY d.valuation_date ORDER BY d.valuation_date" } ] }, @@ -475,7 +475,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "WITH bounds AS (SELECT (SELECT MAX(valuation_date) FROM daily_account_valuation) AS d_now, (SELECT MIN(valuation_date) FROM daily_account_valuation WHERE valuation_date >= (SELECT MAX(valuation_date) - INTERVAL '12 months' FROM daily_account_valuation)) AS d_ago), agg AS (SELECT (SELECT SUM(total_value) FROM daily_account_valuation WHERE valuation_date = b.d_now) AS nw_now, (SELECT SUM(net_contribution) FROM daily_account_valuation WHERE valuation_date = b.d_now) AS contrib_now, (SELECT SUM(total_value) FROM daily_account_valuation WHERE valuation_date = b.d_ago) AS nw_ago, (SELECT SUM(net_contribution) FROM daily_account_valuation WHERE valuation_date = b.d_ago) AS contrib_ago FROM bounds b) SELECT ROUND((((nw_now - nw_ago - (contrib_now - contrib_ago)) / NULLIF(nw_ago + 0.5 * (contrib_now - contrib_ago), 0)) * 100)::numeric, 2) AS pct_12mo FROM agg" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.valuation_date AS d_now, d.total_value AS nw_now, d.net_contribution AS contrib_now FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC), ago AS (SELECT DISTINCT ON (l.account_id) l.account_id, d.total_value AS nw_ago, d.net_contribution AS contrib_ago FROM latest l JOIN daily_account_valuation d ON d.account_id = l.account_id AND d.valuation_date <= l.d_now - INTERVAL '12 months' ORDER BY l.account_id, d.valuation_date DESC), agg AS (SELECT (SELECT SUM(nw_now) FROM latest) AS nw_now, (SELECT SUM(contrib_now) FROM latest) AS contrib_now, (SELECT SUM(nw_ago) FROM ago) AS nw_ago, (SELECT SUM(contrib_ago) FROM ago) AS contrib_ago) SELECT ROUND((((nw_now - nw_ago - (contrib_now - contrib_ago)) / NULLIF(nw_ago + 0.5 * (contrib_now - contrib_ago), 0)) * 100)::numeric, 2) AS pct_12mo FROM agg" } ] }, @@ -509,7 +509,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "WITH bounds AS (SELECT (SELECT MAX(valuation_date) FROM daily_account_valuation) AS d_now, (SELECT MIN(valuation_date) FROM daily_account_valuation WHERE valuation_date >= (SELECT MAX(valuation_date) - INTERVAL '12 months' FROM daily_account_valuation)) AS d_ago) SELECT ((SELECT SUM(net_contribution) FROM daily_account_valuation WHERE valuation_date = b.d_now) - (SELECT SUM(net_contribution) FROM daily_account_valuation WHERE valuation_date = b.d_ago)) AS contrib_12mo FROM bounds b" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.valuation_date AS d_now, d.total_value AS nw_now, d.net_contribution AS contrib_now FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC), ago AS (SELECT DISTINCT ON (l.account_id) l.account_id, d.total_value AS nw_ago, d.net_contribution AS contrib_ago FROM latest l JOIN daily_account_valuation d ON d.account_id = l.account_id AND d.valuation_date <= l.d_now - INTERVAL '12 months' ORDER BY l.account_id, d.valuation_date DESC), agg AS (SELECT (SELECT SUM(nw_now) FROM latest) AS nw_now, (SELECT SUM(contrib_now) FROM latest) AS contrib_now, (SELECT SUM(nw_ago) FROM ago) AS nw_ago, (SELECT SUM(contrib_ago) FROM ago) AS contrib_ago) SELECT (contrib_now - contrib_ago) AS contrib_12mo FROM agg" } ] }, @@ -550,7 +550,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "WITH bounds AS (SELECT (SELECT MAX(valuation_date) FROM daily_account_valuation) AS d_now, (SELECT MIN(valuation_date) FROM daily_account_valuation WHERE valuation_date >= (SELECT MAX(valuation_date) - INTERVAL '12 months' FROM daily_account_valuation)) AS d_ago), agg AS (SELECT (SELECT SUM(total_value) FROM daily_account_valuation WHERE valuation_date = b.d_now) AS nw_now, (SELECT SUM(net_contribution) FROM daily_account_valuation WHERE valuation_date = b.d_now) AS contrib_now, (SELECT SUM(total_value) FROM daily_account_valuation WHERE valuation_date = b.d_ago) AS nw_ago, (SELECT SUM(net_contribution) FROM daily_account_valuation WHERE valuation_date = b.d_ago) AS contrib_ago FROM bounds b) SELECT ((nw_now - nw_ago) - (contrib_now - contrib_ago)) AS gain_12mo FROM agg" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) d.account_id, d.valuation_date AS d_now, d.total_value AS nw_now, d.net_contribution AS contrib_now FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC), ago AS (SELECT DISTINCT ON (l.account_id) l.account_id, d.total_value AS nw_ago, d.net_contribution AS contrib_ago FROM latest l JOIN daily_account_valuation d ON d.account_id = l.account_id AND d.valuation_date <= l.d_now - INTERVAL '12 months' ORDER BY l.account_id, d.valuation_date DESC), agg AS (SELECT (SELECT SUM(nw_now) FROM latest) AS nw_now, (SELECT SUM(contrib_now) FROM latest) AS contrib_now, (SELECT SUM(nw_ago) FROM ago) AS nw_ago, (SELECT SUM(contrib_ago) FROM ago) AS contrib_ago) SELECT ((nw_now - nw_ago) - (contrib_now - contrib_ago)) AS gain_12mo FROM agg" } ] }, @@ -610,7 +610,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "WITH yearly AS (SELECT EXTRACT(YEAR FROM valuation_date)::int AS yr, valuation_date, SUM(total_value) AS nw, SUM(net_contribution) AS contrib FROM daily_account_valuation GROUP BY valuation_date), endpoints AS (SELECT yr, (array_agg(nw ORDER BY valuation_date ASC))[1] AS nw_start, (array_agg(nw ORDER BY valuation_date DESC))[1] AS nw_end, (array_agg(contrib ORDER BY valuation_date ASC))[1] AS contrib_start, (array_agg(contrib ORDER BY valuation_date DESC))[1] AS contrib_end FROM yearly GROUP BY yr) SELECT yr::text AS year, ROUND((((nw_end - nw_start - (contrib_end - contrib_start)) / NULLIF(nw_start + 0.5 * (contrib_end - contrib_start), 0)) * 100)::numeric, 2) AS return_pct FROM endpoints WHERE (nw_start + 0.5 * (contrib_end - contrib_start)) > 0 ORDER BY yr" + "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)), yearly AS (SELECT EXTRACT(YEAR FROM valuation_date)::int AS yr, valuation_date, SUM(total_value) AS nw, SUM(net_contribution) AS contrib FROM daily_account_valuation WHERE valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date), endpoints AS (SELECT yr, (array_agg(nw ORDER BY valuation_date ASC))[1] AS nw_start, (array_agg(nw ORDER BY valuation_date DESC))[1] AS nw_end, (array_agg(contrib ORDER BY valuation_date ASC))[1] AS contrib_start, (array_agg(contrib ORDER BY valuation_date DESC))[1] AS contrib_end FROM yearly GROUP BY yr) SELECT yr::text AS year, ROUND((((nw_end - nw_start - (contrib_end - contrib_start)) / NULLIF(nw_start + 0.5 * (contrib_end - contrib_start), 0)) * 100)::numeric, 2) AS return_pct FROM endpoints WHERE (nw_start + 0.5 * (contrib_end - contrib_start)) > 0 ORDER BY yr" } ] }, @@ -676,7 +676,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "WITH yearly AS (SELECT EXTRACT(YEAR FROM valuation_date)::int AS yr, valuation_date, SUM(total_value) AS nw, SUM(net_contribution) AS contrib FROM daily_account_valuation GROUP BY valuation_date), endpoints AS (SELECT yr, (array_agg(nw ORDER BY valuation_date ASC))[1] AS nw_start, (array_agg(nw ORDER BY valuation_date DESC))[1] AS nw_end, (array_agg(contrib ORDER BY valuation_date ASC))[1] AS contrib_start, (array_agg(contrib ORDER BY valuation_date DESC))[1] AS contrib_end FROM yearly GROUP BY yr) SELECT yr::text AS year, ROUND((contrib_end - contrib_start)::numeric, 0) AS contributions, ROUND((nw_end - nw_start - (contrib_end - contrib_start))::numeric, 0) AS market_gain FROM endpoints ORDER BY yr" + "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)), yearly AS (SELECT EXTRACT(YEAR FROM valuation_date)::int AS yr, valuation_date, SUM(total_value) AS nw, SUM(net_contribution) AS contrib FROM daily_account_valuation WHERE valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date), endpoints AS (SELECT yr, (array_agg(nw ORDER BY valuation_date ASC))[1] AS nw_start, (array_agg(nw ORDER BY valuation_date DESC))[1] AS nw_end, (array_agg(contrib ORDER BY valuation_date ASC))[1] AS contrib_start, (array_agg(contrib ORDER BY valuation_date DESC))[1] AS contrib_end FROM yearly GROUP BY yr) SELECT yr::text AS year, ROUND((contrib_end - contrib_start)::numeric, 0) AS contributions, ROUND((nw_end - nw_start - (contrib_end - contrib_start))::numeric, 0) AS market_gain FROM endpoints ORDER BY yr" } ] }, @@ -728,7 +728,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "SELECT a.name AS account, ROUND(((d.total_value - d.net_contribution) / NULLIF(d.net_contribution, 0) * 100)::numeric, 2) AS roi_pct FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE d.valuation_date = (SELECT MAX(valuation_date) FROM daily_account_valuation) AND d.net_contribution > 0 ORDER BY roi_pct DESC" + "rawSql": "WITH latest AS (SELECT DISTINCT ON (d.account_id) a.name, d.total_value, d.net_contribution FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id ORDER BY d.account_id, d.valuation_date DESC) SELECT name AS account, ROUND(((total_value - net_contribution) / NULLIF(net_contribution, 0) * 100)::numeric, 2) AS roi_pct FROM latest WHERE net_contribution > 0 ORDER BY roi_pct DESC" } ] } From d67416d4cac9819b8fc753baa731ac424af28d55 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Fri, 1 May 2026 16:15:39 +0000 Subject: [PATCH 05/25] monitoring(wealth): tighten default time range, bump decimals for granularity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two adjustments to make daily movements visible: 1. Default time range: now-5y → now-180d. The timeseries charts (Net worth, Net contribution vs market value, Growth, Per-account stacked, Cash vs invested) auto-fit their y-axis to the data range in view. Over 5 years, daily £1k–£10k moves are ~1% of axis range and visually invisible against the cumulative trend. Over 6 months, the same daily moves dominate. Yearly bar charts (12, 13) are unaffected — they aggregate by calendar year and don't filter on $__timeFilter. 2. Decimals → 2 on every currency panel (1, 2, 3, 5–9, 13, 15, 16) and every percent panel (4, 14). Stat panels now show pennies on currency and 0.01% on rates; chart y-axis ticks are likewise more precise. Honest caveat: pennies on a £1M number don't make the absolute readout easier — to see "today changed by £8,358" cleanly we'd want a dedicated delta panel; pending user direction. Widen the time picker manually to recover the 5-year view; default just zooms into the last 6 months. --- .../modules/monitoring/dashboards/wealth.json | 23 +++++++++++-------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/stacks/monitoring/modules/monitoring/dashboards/wealth.json b/stacks/monitoring/modules/monitoring/dashboards/wealth.json index a09ab92d..e86e0431 100644 --- a/stacks/monitoring/modules/monitoring/dashboards/wealth.json +++ b/stacks/monitoring/modules/monitoring/dashboards/wealth.json @@ -28,7 +28,7 @@ "defaults": { "unit": "currencyGBP", "color": {"mode": "fixed", "fixedColor": "green"}, - "decimals": 0 + "decimals": 2 }, "overrides": [] }, @@ -62,7 +62,7 @@ "defaults": { "unit": "currencyGBP", "color": {"mode": "fixed", "fixedColor": "blue"}, - "decimals": 0 + "decimals": 2 }, "overrides": [] }, @@ -96,7 +96,7 @@ "defaults": { "unit": "currencyGBP", "color": {"mode": "thresholds"}, - "decimals": 0, + "decimals": 2, "thresholds": { "mode": "absolute", "steps": [ @@ -137,7 +137,7 @@ "defaults": { "unit": "percent", "color": {"mode": "thresholds"}, - "decimals": 1, + "decimals": 2, "thresholds": { "mode": "absolute", "steps": [ @@ -179,6 +179,7 @@ "defaults": { "color": {"mode": "fixed", "fixedColor": "green"}, "unit": "currencyGBP", + "decimals": 2, "custom": { "drawStyle": "line", "lineWidth": 2, @@ -223,6 +224,7 @@ "defaults": { "color": {"mode": "palette-classic"}, "unit": "currencyGBP", + "decimals": 2, "custom": { "drawStyle": "line", "lineWidth": 2, @@ -277,6 +279,7 @@ "defaults": { "color": {"mode": "fixed", "fixedColor": "#56A64B"}, "unit": "currencyGBP", + "decimals": 2, "custom": { "drawStyle": "line", "lineWidth": 2, @@ -322,6 +325,7 @@ "defaults": { "color": {"mode": "palette-classic"}, "unit": "currencyGBP", + "decimals": 2, "custom": { "drawStyle": "line", "lineWidth": 1, @@ -361,6 +365,7 @@ "defaults": { "color": {"mode": "palette-classic"}, "unit": "currencyGBP", + "decimals": 2, "custom": { "drawStyle": "line", "lineWidth": 1, @@ -490,7 +495,7 @@ "defaults": { "unit": "currencyGBP", "color": {"mode": "fixed", "fixedColor": "blue"}, - "decimals": 0 + "decimals": 2 }, "overrides": [] }, @@ -524,7 +529,7 @@ "defaults": { "unit": "currencyGBP", "color": {"mode": "thresholds"}, - "decimals": 0, + "decimals": 2, "thresholds": { "mode": "absolute", "steps": [ @@ -625,7 +630,7 @@ "defaults": { "color": {"mode": "palette-classic"}, "unit": "currencyGBP", - "decimals": 0, + "decimals": 2, "custom": { "axisPlacement": "auto", "axisLabel": "", @@ -691,7 +696,7 @@ "defaults": { "color": {"mode": "thresholds"}, "unit": "percent", - "decimals": 1, + "decimals": 2, "thresholds": { "mode": "absolute", "steps": [ @@ -737,7 +742,7 @@ "schemaVersion": 39, "tags": ["finance", "personal", "wealth"], "templating": {"list": []}, - "time": {"from": "now-5y", "to": "now"}, + "time": {"from": "now-180d", "to": "now"}, "timepicker": {}, "timezone": "browser", "title": "Wealth", From 2722260ce9d7cab3aeaae5934c98eb40cc3493ca Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Fri, 1 May 2026 16:19:07 +0000 Subject: [PATCH 06/25] =?UTF-8?q?monitoring(wealth):=20unbreak=20timeserie?= =?UTF-8?q?s=20SQL=20=E2=80=94=20over-escaped=20time=20alias?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix: panels 5–9 had `AS \"time\"` (literal backslash-quote sequence embedded in the SQL string). PostgreSQL parsed that as a syntax error at the leading backslash: ERROR: syntax error at or near "\" LINE 1: ...complete_dates)) SELECT valuation_date::timestamp AS \"time\" Root cause: the patch script for the skew-resilient queries (commit 628f5a0d) used a Python f-string with `\\\"time\\\"`, which produces a literal backslash-quote in the Python string. When that string was JSON-encoded the backslash was preserved verbatim instead of collapsed to plain `"time"`. Replaces all five occurrences with the correct `AS "time"` form. Verified the corrected query against PG returns 7 daily net-worth rows for 04-25..05-01 as expected. --- .../modules/monitoring/dashboards/wealth.json | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/stacks/monitoring/modules/monitoring/dashboards/wealth.json b/stacks/monitoring/modules/monitoring/dashboards/wealth.json index e86e0431..f9a22e19 100644 --- a/stacks/monitoring/modules/monitoring/dashboards/wealth.json +++ b/stacks/monitoring/modules/monitoring/dashboards/wealth.json @@ -209,7 +209,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT valuation_date::timestamp AS \\\"time\\\", SUM(total_value) AS net_worth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date ORDER BY valuation_date" + "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT valuation_date::timestamp AS \"time\", SUM(total_value) AS net_worth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date ORDER BY valuation_date" } ] }, @@ -264,7 +264,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT valuation_date::timestamp AS \\\"time\\\", SUM(net_contribution) AS net_contribution, SUM(total_value) AS market_value FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date ORDER BY valuation_date" + "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT valuation_date::timestamp AS \"time\", SUM(net_contribution) AS net_contribution, SUM(total_value) AS market_value FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date ORDER BY valuation_date" } ] }, @@ -310,7 +310,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT valuation_date::timestamp AS \\\"time\\\", (SUM(total_value) - SUM(net_contribution)) AS growth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date ORDER BY valuation_date" + "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT valuation_date::timestamp AS \"time\", (SUM(total_value) - SUM(net_contribution)) AS growth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date ORDER BY valuation_date" } ] }, @@ -350,7 +350,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT d.valuation_date::timestamp AS \\\"time\\\", a.name AS metric, d.total_value AS value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) AND d.valuation_date IN (SELECT valuation_date FROM complete_dates) ORDER BY d.valuation_date, a.name" + "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT d.valuation_date::timestamp AS \"time\", a.name AS metric, d.total_value AS value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) AND d.valuation_date IN (SELECT valuation_date FROM complete_dates) ORDER BY d.valuation_date, a.name" } ] }, @@ -405,7 +405,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT d.valuation_date::timestamp AS \\\"time\\\", SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN 0 ELSE d.cash_balance END) AS cash, SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN d.cash_balance + d.investment_market_value ELSE d.investment_market_value END) AS invested FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) AND d.valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY d.valuation_date ORDER BY d.valuation_date" + "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT d.valuation_date::timestamp AS \"time\", SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN 0 ELSE d.cash_balance END) AS cash, SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN d.cash_balance + d.investment_market_value ELSE d.investment_market_value END) AS invested FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) AND d.valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY d.valuation_date ORDER BY d.valuation_date" } ] }, From 5472720c752d566a252fd07e4ff41702c60a6e2a Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Fri, 1 May 2026 16:23:25 +0000 Subject: [PATCH 07/25] monitoring(wealth): show daily points + lighter fill on timeseries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make daily movements visible on the line charts. The y-axis still spans ~£700k–£1M so an £8k daily move is ~1% of vertical range and easy to miss when only the line is drawn. Changes per panel: * 5 (Net worth): showPoints never→always, pointSize 4→5, fillOpacity 20→10 * 6 (Net contrib vs market): showPoints never→always, pointSize 4→5 * 7 (Growth over time): showPoints never→always, pointSize 4→5, fillOpacity 50→25 * 8 (Per-account stacked): showPoints never→always (kept stacking fill at 70) * 9 (Cash vs invested stacked): showPoints never→always (kept stacking fill at 70) Each daily value now renders as a visible dot, so even if the line appears flat at this scale, the per-day points trace the wiggle. Lighter fill on the unstacked panels lets the line + points dominate visually. Caveat: the fundamental "£8k on a £1M base" visibility issue is best solved with a dedicated "Daily change" delta panel — happy to add one on next pass if this isn't enough. --- .../modules/monitoring/dashboards/wealth.json | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/stacks/monitoring/modules/monitoring/dashboards/wealth.json b/stacks/monitoring/modules/monitoring/dashboards/wealth.json index f9a22e19..9b0e7962 100644 --- a/stacks/monitoring/modules/monitoring/dashboards/wealth.json +++ b/stacks/monitoring/modules/monitoring/dashboards/wealth.json @@ -183,9 +183,9 @@ "custom": { "drawStyle": "line", "lineWidth": 2, - "fillOpacity": 20, - "pointSize": 4, - "showPoints": "never", + "fillOpacity": 10, + "pointSize": 5, + "showPoints": "always", "spanNulls": true, "axisPlacement": "auto", "stacking": {"group": "A", "mode": "none"} @@ -229,8 +229,8 @@ "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, - "pointSize": 4, - "showPoints": "never", + "pointSize": 5, + "showPoints": "always", "spanNulls": true, "axisPlacement": "auto", "stacking": {"group": "A", "mode": "none"} @@ -283,10 +283,10 @@ "custom": { "drawStyle": "line", "lineWidth": 2, - "fillOpacity": 50, + "fillOpacity": 25, "gradientMode": "opacity", - "pointSize": 4, - "showPoints": "never", + "pointSize": 5, + "showPoints": "always", "spanNulls": true, "axisPlacement": "auto", "stacking": {"group": "A", "mode": "none"} @@ -330,8 +330,8 @@ "drawStyle": "line", "lineWidth": 1, "fillOpacity": 70, - "pointSize": 3, - "showPoints": "never", + "pointSize": 4, + "showPoints": "always", "spanNulls": true, "axisPlacement": "auto", "stacking": {"group": "A", "mode": "normal"} @@ -370,8 +370,8 @@ "drawStyle": "line", "lineWidth": 1, "fillOpacity": 70, - "pointSize": 3, - "showPoints": "never", + "pointSize": 4, + "showPoints": "always", "spanNulls": true, "axisPlacement": "auto", "stacking": {"group": "A", "mode": "normal"} From 664a85ef1eb01d0afb11e5b80e6fc4faf7379b24 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Fri, 1 May 2026 16:24:18 +0000 Subject: [PATCH 08/25] Revert "monitoring(wealth): show daily points + lighter fill on timeseries" This reverts commit 5472720c752d566a252fd07e4ff41702c60a6e2a. --- .../modules/monitoring/dashboards/wealth.json | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/stacks/monitoring/modules/monitoring/dashboards/wealth.json b/stacks/monitoring/modules/monitoring/dashboards/wealth.json index 9b0e7962..f9a22e19 100644 --- a/stacks/monitoring/modules/monitoring/dashboards/wealth.json +++ b/stacks/monitoring/modules/monitoring/dashboards/wealth.json @@ -183,9 +183,9 @@ "custom": { "drawStyle": "line", "lineWidth": 2, - "fillOpacity": 10, - "pointSize": 5, - "showPoints": "always", + "fillOpacity": 20, + "pointSize": 4, + "showPoints": "never", "spanNulls": true, "axisPlacement": "auto", "stacking": {"group": "A", "mode": "none"} @@ -229,8 +229,8 @@ "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, - "pointSize": 5, - "showPoints": "always", + "pointSize": 4, + "showPoints": "never", "spanNulls": true, "axisPlacement": "auto", "stacking": {"group": "A", "mode": "none"} @@ -283,10 +283,10 @@ "custom": { "drawStyle": "line", "lineWidth": 2, - "fillOpacity": 25, + "fillOpacity": 50, "gradientMode": "opacity", - "pointSize": 5, - "showPoints": "always", + "pointSize": 4, + "showPoints": "never", "spanNulls": true, "axisPlacement": "auto", "stacking": {"group": "A", "mode": "none"} @@ -330,8 +330,8 @@ "drawStyle": "line", "lineWidth": 1, "fillOpacity": 70, - "pointSize": 4, - "showPoints": "always", + "pointSize": 3, + "showPoints": "never", "spanNulls": true, "axisPlacement": "auto", "stacking": {"group": "A", "mode": "normal"} @@ -370,8 +370,8 @@ "drawStyle": "line", "lineWidth": 1, "fillOpacity": 70, - "pointSize": 4, - "showPoints": "always", + "pointSize": 3, + "showPoints": "never", "spanNulls": true, "axisPlacement": "auto", "stacking": {"group": "A", "mode": "normal"} From ce7a584801b61cd8d5e248db1d4257fcbd14e3d0 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Fri, 1 May 2026 18:38:30 +0000 Subject: [PATCH 09/25] priority-pass: frontend ea9176f8 (gallery upload), sync backend pin to live MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Frontend bug fix: photo input forced camera on mobile via capture="environment". Added separate "Choose from Gallery" / "Take Photo" buttons so users can pick from their photo library. Backend image unchanged; pin synced from stale v8 to live SHA ae1420a0 (the v8 tag was never pushed to the registry). Image built locally and pushed to registry.viktorbarzin.me. The priority-pass project directory isn't under git, so deployment was applied via kubectl set image (matches existing pattern — TF state for this stack is empty). [ci skip] --- stacks/priority-pass/main.tf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stacks/priority-pass/main.tf b/stacks/priority-pass/main.tf index a9e62e7d..b19777a8 100644 --- a/stacks/priority-pass/main.tf +++ b/stacks/priority-pass/main.tf @@ -51,7 +51,7 @@ resource "kubernetes_deployment" "priority-pass" { } container { name = "frontend" - image = "registry.viktorbarzin.me/priority-pass-frontend:v5" + image = "registry.viktorbarzin.me/priority-pass-frontend:ea9176f8" port { container_port = 3000 } @@ -75,7 +75,7 @@ resource "kubernetes_deployment" "priority-pass" { } container { name = "backend" - image = "registry.viktorbarzin.me/priority-pass-backend:v8" + image = "registry.viktorbarzin.me/priority-pass-backend:ae1420a0" port { container_port = 8000 } From dfbf6faf3d969686934ace1e629cdb50af197062 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Fri, 1 May 2026 18:50:51 +0000 Subject: [PATCH 10/25] priority-pass: backend f4246691 (QR fit fix + persist uploads), add encrypted PVC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Backend changes: - transformers.py: QR container now sized to actual qr_bbox + 8% padding (was fixed at 45% of card width). When QR was wider than 45% of card, the leftover-pixel branch color-remapped QR pixels outside the container, breaking the scan. New container always encloses qr_mask. - main.py: persist input + output + json metadata under $UPLOAD_DIR//-{input.,output.png,*.json} for future training. Failure to save is logged, never breaks the API. Infra: - New PVC priority-pass-uploads (1Gi proxmox-lvm-encrypted, 10Gi autoresize cap) — encrypted because boarding passes contain PII. - Deployment strategy → Recreate (RWO requirement). - Volume + volumeMount + UPLOAD_DIR env on backend container. Applied via kubectl (TF state for this stack is empty — see prior commit). New pod priority-pass-77956b64fb rolled out, PVC bound, test transform succeeded, sample written to /data/uploads/ryanair/. [ci skip] --- stacks/priority-pass/main.tf | 39 +++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/stacks/priority-pass/main.tf b/stacks/priority-pass/main.tf index b19777a8..02497af8 100644 --- a/stacks/priority-pass/main.tf +++ b/stacks/priority-pass/main.tf @@ -23,6 +23,26 @@ module "tls_secret" { tls_secret_name = var.tls_secret_name } +resource "kubernetes_persistent_volume_claim" "uploads" { + wait_until_bound = false + metadata { + name = "priority-pass-uploads" + namespace = kubernetes_namespace.priority-pass.metadata[0].name + annotations = { + "resize.topolvm.io/threshold" = "80%" + "resize.topolvm.io/increase" = "100%" + "resize.topolvm.io/storage_limit" = "10Gi" + } + } + spec { + access_modes = ["ReadWriteOnce"] + storage_class_name = "proxmox-lvm-encrypted" + resources { + requests = { storage = "1Gi" } + } + } +} + resource "kubernetes_deployment" "priority-pass" { metadata { name = "priority-pass" @@ -34,6 +54,9 @@ resource "kubernetes_deployment" "priority-pass" { } spec { replicas = 1 + strategy { + type = "Recreate" + } selector { match_labels = { run = "priority-pass" @@ -49,6 +72,12 @@ resource "kubernetes_deployment" "priority-pass" { image_pull_secrets { name = "registry-credentials" } + volume { + name = "uploads" + persistent_volume_claim { + claim_name = kubernetes_persistent_volume_claim.uploads.metadata[0].name + } + } container { name = "frontend" image = "registry.viktorbarzin.me/priority-pass-frontend:ea9176f8" @@ -75,10 +104,18 @@ resource "kubernetes_deployment" "priority-pass" { } container { name = "backend" - image = "registry.viktorbarzin.me/priority-pass-backend:ae1420a0" + image = "registry.viktorbarzin.me/priority-pass-backend:f4246691" port { container_port = 8000 } + env { + name = "UPLOAD_DIR" + value = "/data/uploads" + } + volume_mount { + name = "uploads" + mount_path = "/data/uploads" + } resources { limits = { memory = "512Mi" From 40a6cd067b79b24fb0bc0832065f8500bb361d49 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Fri, 1 May 2026 19:03:50 +0000 Subject: [PATCH 11/25] authentik: long-lived authenticated sessions, short-lived anonymous ones - Adopt UserLoginStage (default-authentication-login) into Terraform and pin session_duration=weeks=4 so users stay logged in across browser restarts. There is no Brand.session_duration in 2026.2.x; UserLoginStage is the only correct lever. - Cap anonymous Django sessions at 2h via AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE on server + worker pods (default is days=1). Bots, healthcheckers, and partial flows now get reaped within 2h instead of accumulating for a day. Implementation note: the env var is injected via server.env / worker.env rather than authentik.sessions.unauthenticated_age, because authentik.existingSecret.secretName is set, which makes the chart skip rendering its own AUTHENTIK_* Secret. authentik.* values are therefore inert in this stack -- this is documented in .claude/reference/authentik-state.md so future edits use the right surface. Co-Authored-By: Claude Opus 4.7 --- .claude/reference/authentik-state.md | 15 +++++++++ stacks/authentik/authentik_provider.tf | 31 +++++++++++++++++++ .../authentik/modules/authentik/values.yaml | 14 +++++++++ 3 files changed, 60 insertions(+) diff --git a/.claude/reference/authentik-state.md b/.claude/reference/authentik-state.md index 2005bb09..f76dd325 100644 --- a/.claude/reference/authentik-state.md +++ b/.claude/reference/authentik-state.md @@ -119,3 +119,18 @@ Removed bindings from: - `default-source-authentication` (PK: via policybindingmodel `1a779f24`) — Google/GitHub/Facebook OAuth Policy still exists with 0 bindings. If brute-force protection is needed, bind to the **password stage** (not the flow level). + +## Session Duration (2026-05-01) + +Pinned via Terraform in `stacks/authentik/`: + +| Knob | Value | Surface | Effect | +|------|-------|---------|--------| +| `UserLoginStage.session_duration` on `default-authentication-login` | `weeks=4` | `authentik_stage_user_login.default_login` in `authentik_provider.tf` | Authenticated users stay logged in 4 weeks across browser restarts. No sliding refresh — resets on each login. | +| `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE` (server + worker) | `hours=2` | `server.env` + `worker.env` in `modules/authentik/values.yaml` | Anonymous Django sessions (bots, healthcheckers, partial flows) are reaped within 2h instead of the 1d default. | + +Notes: +- There is **no** `Brand.session_duration`; `UserLoginStage` is the only correct lever for authenticated session lifetime. +- Embedded outpost session storage moved from `/dev/shm` → Postgres table `authentik_providers_proxy_proxysession` in authentik 2025.10. The 2026-04-18 `/dev/shm`-fill outage class is no longer load-bearing in 2026.2.2; the `unauthenticated_age` cap is still the right lever for anonymous-session bloat from external monitors. +- `ProxyProvider.access_token_validity` and `remember_me_offset` stay UI-managed via `ignore_changes`. +- The `unauthenticated_age` env var is injected via `server.env` / `worker.env` (not `authentik.sessions.unauthenticated_age`) because we set `authentik.existingSecret.secretName: goauthentik`, which makes the chart skip rendering its own `AUTHENTIK_*` Secret. The `authentik.*` value block is therefore inert in this stack — anything new under `authentik.*` must use the `*.env` arrays instead. The same applies to the existing `authentik.cache.*`, `authentik.web.*`, `authentik.worker.*` blocks (currently inert; live values come from the orphaned, helm-keep-policy `goauthentik` Secret created by chart 2025.10.3 before `existingSecret` was introduced). diff --git a/stacks/authentik/authentik_provider.tf b/stacks/authentik/authentik_provider.tf index e9db3985..f33a225a 100644 --- a/stacks/authentik/authentik_provider.tf +++ b/stacks/authentik/authentik_provider.tf @@ -57,3 +57,34 @@ resource "authentik_provider_proxy" "catchall" { ignore_changes = [property_mappings, jwt_federation_sources, skip_path_regex, internal_host, basic_auth_enabled, basic_auth_password_attribute, basic_auth_username_attribute, intercept_header_auth, access_token_validity] } } + +# ----------------------------------------------------------------------------- +# Default User Login stage — bound to default-authentication-flow. +# Adopted into Terraform 2026-05-01 to set session_duration=weeks=4 so users +# stay logged in across browser restarts. There is no Brand.session_duration +# in authentik 2026.2.x — UserLoginStage is the correct knob. +# ----------------------------------------------------------------------------- + +data "authentik_stage" "default_authentication_login" { + name = "default-authentication-login" +} + +import { + to = authentik_stage_user_login.default_login + id = data.authentik_stage.default_authentication_login.id +} + +resource "authentik_stage_user_login" "default_login" { + name = "default-authentication-login" + session_duration = "weeks=4" + lifecycle { + # Pin only session_duration; everything else stays UI-managed so the + # plan doesn't churn unrelated knobs (e.g. remember_me_offset toggles). + ignore_changes = [ + remember_me_offset, + terminate_other_sessions, + geoip_binding, + network_binding, + ] + } +} diff --git a/stacks/authentik/modules/authentik/values.yaml b/stacks/authentik/modules/authentik/values.yaml index e8c7d5ea..9822516c 100644 --- a/stacks/authentik/modules/authentik/values.yaml +++ b/stacks/authentik/modules/authentik/values.yaml @@ -37,6 +37,15 @@ authentik: server: replicas: 3 + # Anonymous Django sessions (no completed login: bots, healthcheckers, + # partial flows) expire in 2h. Default is days=1. Once login completes, + # UserLoginStage.session_duration takes over via request.session.set_expiry. + # Injected via server.env (not authentik.sessions.*) because we use + # authentik.existingSecret.secretName, which makes the chart skip + # rendering the AUTHENTIK_* secret — so the values block doesn't reach env. + env: + - name: AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE + value: "hours=2" strategy: type: RollingUpdate rollingUpdate: @@ -70,6 +79,11 @@ global: worker: replicas: 3 + # Same unauthenticated_age cap as server — both the server (Django session + # middleware) and worker (cleanup tasks) need to see the value. + env: + - name: AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE + value: "hours=2" strategy: type: RollingUpdate rollingUpdate: From d76b5dbc4b1ec144c23c92685803d4dc049aa5ba Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Fri, 1 May 2026 19:06:02 +0000 Subject: [PATCH 12/25] =?UTF-8?q?priority-pass:=20backend=20c2b4ac50=20?= =?UTF-8?q?=E2=80=94=20crop=20to=20card=20before=20transforming?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three fixes for boarding passes uploaded as iPhone screenshots (input includes phone status bar, partial Tesco card below, etc.): 1. Detect the card region first and crop to it. All proportional coordinates (Step 8 text replacement, Step 9 logo removal) are now card-relative instead of full-image-relative — they were landing in the wrong region on tall screenshots, putting "Priority" text inside the QR area and leaving a yellow icon box at the bottom. 2. Step 8 now picks the LONGEST contiguous dark-row run inside a wider y-band, instead of using the dark-row [first, last] span. This distinguishes the QUEUE value text from the QUEUE label above it (both are dark blue in the original) so the erase rectangle no longer eats into the labels. 3. QR container padding bumped 8% → 12% so QR/container ratio matches the ~74-80% golden look. Verified end-to-end against three real samples saved by the previous build's training-data feature, plus the original non-priority.jpeg fixture: outputs now match priority.jpeg layout. [ci skip] --- stacks/authentik/authentik_provider.tf | 9 --------- stacks/priority-pass/main.tf | 2 +- 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/stacks/authentik/authentik_provider.tf b/stacks/authentik/authentik_provider.tf index f33a225a..f34214fe 100644 --- a/stacks/authentik/authentik_provider.tf +++ b/stacks/authentik/authentik_provider.tf @@ -65,15 +65,6 @@ resource "authentik_provider_proxy" "catchall" { # in authentik 2026.2.x — UserLoginStage is the correct knob. # ----------------------------------------------------------------------------- -data "authentik_stage" "default_authentication_login" { - name = "default-authentication-login" -} - -import { - to = authentik_stage_user_login.default_login - id = data.authentik_stage.default_authentication_login.id -} - resource "authentik_stage_user_login" "default_login" { name = "default-authentication-login" session_duration = "weeks=4" diff --git a/stacks/priority-pass/main.tf b/stacks/priority-pass/main.tf index 02497af8..9bd9792e 100644 --- a/stacks/priority-pass/main.tf +++ b/stacks/priority-pass/main.tf @@ -104,7 +104,7 @@ resource "kubernetes_deployment" "priority-pass" { } container { name = "backend" - image = "registry.viktorbarzin.me/priority-pass-backend:f4246691" + image = "registry.viktorbarzin.me/priority-pass-backend:c2b4ac50" port { container_port = 8000 } From 86385f5842ba927a97ec235b868d74c4ae0b55b8 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Fri, 1 May 2026 19:27:33 +0000 Subject: [PATCH 13/25] priority-pass: pin to DockerHub viktorbarzin/* (GHA-built, sha 50a432ad) Now that priority-pass has its own GitHub repo (ViktorBarzin/priority-pass) with a working GHA build pipeline that pushes to DockerHub, switch the TF deployment pin from registry.viktorbarzin.me to docker.io/viktorbarzin so future automated rollouts (once the repo is registered with Woodpecker) land on the matching image source. [ci skip] --- stacks/priority-pass/main.tf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stacks/priority-pass/main.tf b/stacks/priority-pass/main.tf index 9bd9792e..983d4ae6 100644 --- a/stacks/priority-pass/main.tf +++ b/stacks/priority-pass/main.tf @@ -80,7 +80,7 @@ resource "kubernetes_deployment" "priority-pass" { } container { name = "frontend" - image = "registry.viktorbarzin.me/priority-pass-frontend:ea9176f8" + image = "docker.io/viktorbarzin/priority-pass-frontend:50a432ad" port { container_port = 3000 } @@ -104,7 +104,7 @@ resource "kubernetes_deployment" "priority-pass" { } container { name = "backend" - image = "registry.viktorbarzin.me/priority-pass-backend:c2b4ac50" + image = "docker.io/viktorbarzin/priority-pass-backend:50a432ad" port { container_port = 8000 } From 5a00b9c096724ed9ed63ee4e9f49af6a0e05395d Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Sat, 2 May 2026 08:42:21 +0000 Subject: [PATCH 14/25] monitoring(wealth): milestone annotations on every timeseries chart MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Inspired by the user's "Journey to £1M" reference — adds vertical dashed lines on every timeseries panel at the date net worth first crossed each round threshold (£100k, £250k, £500k, £750k, £1M). Implementation: a dashboard-level annotation source ("Milestones", purple) backed by a PG query that finds the MIN(valuation_date) where SUM(total_value) >= each threshold. The query returns (time, text) pairs, e.g. "2026-04-18 → £1M 🎉". Annotations attach to all timeseries panels automatically; auto-extends as future thresholds are crossed. Verified against current data: £100k → 2021-11-01 £250k → 2023-07-18 £500k → 2024-09-19 £750k → 2025-08-26 £1M → 2026-04-18 🎉 Future work (per user request): add a "Journey" stat-card row at the top mirroring the reference (date achieved + months from previous). --- .../modules/monitoring/dashboards/wealth.json | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/stacks/monitoring/modules/monitoring/dashboards/wealth.json b/stacks/monitoring/modules/monitoring/dashboards/wealth.json index f9a22e19..8c165b88 100644 --- a/stacks/monitoring/modules/monitoring/dashboards/wealth.json +++ b/stacks/monitoring/modules/monitoring/dashboards/wealth.json @@ -9,6 +9,20 @@ "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "type": "dashboard" + }, + { + "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, + "enable": true, + "hide": false, + "iconColor": "purple", + "name": "Milestones", + "target": { + "rawQuery": true, + "editorMode": "code", + "format": "table", + "refId": "Anno", + "rawSql": "WITH daily AS (SELECT d.valuation_date, SUM(d.total_value) AS nw FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date), crossings AS (SELECT t, (SELECT MIN(valuation_date) FROM daily WHERE nw >= t::numeric) AS d FROM unnest(ARRAY[100000, 250000, 500000, 750000, 1000000]) AS t) SELECT d::timestamp AS \"time\", '£' || CASE WHEN t >= 1000000 THEN (t/1000000)::int::text || 'M 🎉' ELSE (t/1000)::int::text || 'k' END AS text FROM crossings WHERE d IS NOT NULL ORDER BY d" + } } ] }, From 0ef36aec363172642d96915b325fc876d614ccb1 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Sat, 2 May 2026 20:20:18 +0000 Subject: [PATCH 15/25] Revert "monitoring(wealth): milestone annotations on every timeseries chart" This reverts commit 5a00b9c096724ed9ed63ee4e9f49af6a0e05395d. --- .../modules/monitoring/dashboards/wealth.json | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/stacks/monitoring/modules/monitoring/dashboards/wealth.json b/stacks/monitoring/modules/monitoring/dashboards/wealth.json index 8c165b88..f9a22e19 100644 --- a/stacks/monitoring/modules/monitoring/dashboards/wealth.json +++ b/stacks/monitoring/modules/monitoring/dashboards/wealth.json @@ -9,20 +9,6 @@ "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "type": "dashboard" - }, - { - "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, - "enable": true, - "hide": false, - "iconColor": "purple", - "name": "Milestones", - "target": { - "rawQuery": true, - "editorMode": "code", - "format": "table", - "refId": "Anno", - "rawSql": "WITH daily AS (SELECT d.valuation_date, SUM(d.total_value) AS nw FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date), crossings AS (SELECT t, (SELECT MIN(valuation_date) FROM daily WHERE nw >= t::numeric) AS d FROM unnest(ARRAY[100000, 250000, 500000, 750000, 1000000]) AS t) SELECT d::timestamp AS \"time\", '£' || CASE WHEN t >= 1000000 THEN (t/1000000)::int::text || 'M 🎉' ELSE (t/1000)::int::text || 'k' END AS text FROM crossings WHERE d IS NOT NULL ORDER BY d" - } } ] }, From 6715cdc51f86a980c8041a43e6370665643ab20e Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Sat, 2 May 2026 20:27:21 +0000 Subject: [PATCH 16/25] monitoring(wealth): re-add milestone annotations (now that PG creds rotated) Re-applies the milestone annotation commit reverted in 0ef36aec. The earlier "nothing loads / syntax error" was a red herring: Vault had rotated the wealthfolio_sync DB password 7 days prior, the K8s Secret picked it up automatically (pg-sync sidecar still working), but the Grafana datasource ConfigMap is baked at TF-apply time so Grafana was sending the old password. Every panel + the new annotation alike failed with: pq password authentication failed for user wealthfolio_sync. Fix today: refresh the datasource ConfigMap and roll Grafana. scripts/tg apply -target=kubernetes_config_map.grafana_wealth_datasource kubectl -n monitoring rollout restart deploy/grafana Annotation source verified live via /api/ds/query: SQL returns 5 milestone rows correctly. Dashboard charts now show vertical dashed lines at GBP100k 2021-11-01, GBP250k 2023-07-18, GBP500k 2024-09-19, GBP750k 2025-08-26, GBP1M 2026-04-18. KNOWN FOLLOW-UP: Vault rotates pg-wealthfolio-sync every 7 days (static role). Todays failure will recur unless the Grafana datasource auto-refreshes. Options: 1. Annotate Grafana deploy with stakater/reloader so it restarts when wealthfolio-sync-db-creds Secret changes. 2. Switch datasource provisioning to read password from an env var sourced from the Secret instead of baking into the ConfigMap. Combined with reloader, picks up rotation cleanly. --- .../modules/monitoring/dashboards/wealth.json | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/stacks/monitoring/modules/monitoring/dashboards/wealth.json b/stacks/monitoring/modules/monitoring/dashboards/wealth.json index f9a22e19..67536e29 100644 --- a/stacks/monitoring/modules/monitoring/dashboards/wealth.json +++ b/stacks/monitoring/modules/monitoring/dashboards/wealth.json @@ -9,6 +9,20 @@ "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "type": "dashboard" + }, + { + "datasource": {"type": "grafana-postgresql-datasource", "uid": "wealth-pg"}, + "enable": true, + "hide": false, + "iconColor": "purple", + "name": "Milestones", + "target": { + "rawQuery": true, + "editorMode": "code", + "format": "table", + "refId": "Anno", + "rawSql": "WITH daily AS (SELECT d.valuation_date, SUM(d.total_value) AS nw FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date), crossings AS (SELECT t, (SELECT MIN(valuation_date) FROM daily WHERE nw >= t::numeric) AS d FROM unnest(ARRAY[100000, 250000, 500000, 750000, 1000000]) AS t) SELECT d::timestamptz AS time, '£' || CASE WHEN t >= 1000000 THEN (t/1000000)::int::text || 'M' ELSE (t/1000)::int::text || 'k' END AS text FROM crossings WHERE d IS NOT NULL ORDER BY d" + } } ] }, From 0aea98f2258d6b9c462b4769bd54d0f7b330bfd7 Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 00:02:02 +0000 Subject: [PATCH 17/25] Woodpecker CI Update TLS Certificates Commit --- secrets/fullchain.pem | Bin 2898 -> 2898 bytes secrets/privkey.pem | Bin 263 -> 263 bytes 2 files changed, 0 insertions(+), 0 deletions(-) diff --git a/secrets/fullchain.pem b/secrets/fullchain.pem index e4bc0d60252237703652405a6052592e96466d0a..435a323916c24196808a20592132e0397b865df5 100644 GIT binary patch literal 2898 zcmV-Y3$643M@dveQdv+`0M)o#IjV6Q(yj}I$H~Ravi`T*aL z>_!R(%Cb3zQ6T!?1BJ?NBAX1PgWEE2TT|D+Wq8$2{!%Ak2sYox>?2Y}@CiiJS>}hV z2)2arxjZ6swTyn(sPiEQbtS}b-gl8hL|*DCkz`~^5;{k;Ja*zpUXyTTij$sTL!HOKm*z}6c);XK=-nL9{dl(rl$X?%@?1hYJa+LxotBsue zrN+v`3qDgJ7cK-NqDt~lo5G^*VFbC3RnIx=otTbjPaYB=MqRr-41L<%I-lvbfAo3g z1}uag-LPGv71gL3q^8jkOm-G)k2a>es)IpOi32~c##B*l>H7m$kZ>5tKtuU>n{X=X z&7VEqaEL3yU#!3a!6;a2 zz(1Q;a(QpFa5R;#Z`kJ5UvLCVzkM=7 z0&~z*C}bgpr@cyPdV(RmE#&;RhS3xeKjjL9spR_=bgb*GBlBQie`o(hrJCKhkOta0 zhrETr;LiE+VxVp&)vyw0R<*aL+6hI&77I)a6Fv|Wy_-f}6dQoT_H6<;w$|&~WP)=b zO0zIF0iUue%$Xr;SDjM#vpyLklB%u=rymw1n?D~Ys+5dHP&pyF2eg=brM6VqHX4I^sLTDKeMCBlzb79y2iWX zMUZwaPCO)VUmf{JH#bEM&G{h?D@N)9GlErXEWzj`#?1ZnvN$%Aa8nPyXf$|u=r}}- zepFmz@8DN6rQbimRW!Y6Y_b7^?VX?kGc&Q5{#C)0iAoRmu7I9ciJpbP>765u2;w&L z2lgIytFB5#T=_pfdNpz0uXd`EA=da*Jk$wUOM4kAui676;z4yjbs15d^WmiR0Cwh9 z&J$zTT-x}6svpTGm3LzIHOVRUs0~3yJXJ@NJxQGaFqm6ZK@>82DGqWa3nCMD4ye5K zq|>(M!8Jui9wwZ*f+MPf44(a={q#uAVQ)}o*s`FN~-rw)^y5+$t-Y-y+yy!^EE^u!cu`|bW*;iI!g2V;1?AO zg>cwkKR0?^+$(osDK7rX`=zcLba#>MLlkk;aDKYQ?3<{s&hT3Zlx!WraniZiyfA(YcPQ>MMBOq`PG41UE^%L2t|t3(tZ7~cHhToE%(4&h z+ej+7nPu}5M>!xmmrYyqdbx~t4I!S;Eh8BBHSWNvjd0`uS9bMyI_m_swjdU0H2Qy4 z2yBnb-;ojWAlTsG6i=CRzwi3V(= zh=p{*7+7*M^-vzmwSh>q?dc9Rt?9r;+=XiMC6V zw{Y8_My`#S^vwQB6pOfXF|*kgMxNw=v>-fI2qxrl`ojaj4>V>G>TSV()-b2b>0&Kf z4DO&*3N6Bt<7VYEl3X5dRMWc}u!Ls^H%oIGMT+~D>eVSPKl=@Z%uF@ExbFfhH)xO3 zBQ|J99K!qIAG*8!xjTogy6g96Esp7<8@r|^wIwc<8L=|i5Uh_2>=8uQIn4JP-Q>rf zh5pIG#@@oNO+kF^cqLlTcIs*MH~d zr1>Ha`uE(h*xQz4auumDaD$u|3CUCy9w5Z2ADAmhIEG5JHwK$OpS#_N@*WIYmOxoJ zl>KqP{dWT6dLG@FjhpDOB;iHReB?9WlW=^n4FM;`8r94feH_kKkkSwVLB;AiTG+Z_ zl-Ff^jTK!M392zEhv2}5L zAbkFS`|?r`&N9&{Qm-nlvO`bUWt90bLDz7k2p6~ zU1~e-wM6fICgmpoe<4LjwYyAV88CVrh-ZghRCgFsq5%kh9@ZvpYyM;<7l~UR4A$I< zKGo~w__8rhC1WE6T*08#>D3|=f>l-6u6jC4HgSOd5@-fOj$LzsbOHzN-~RstQ=EvH z`2|xq#ieUcxM?UGq_4z@RohEwnKG@l(cg|DXl+2YxR#^D|6gB#-KtF3W?u8d61&?{ z&Qa;Gs+>+s-?G+Na7ubbS00T5UBZdk7d%N|$mTyK-G9wm7FrSdup!G%y(#5h#8a$_ z>tNJh-iM$)iMJq(7KFxt=Ge4nk3$svo@9%|8BEPQ41cbmu-K(SvsuG2gy#e}K7R&& z3<`>nR8v&&mGX#4qR`m4{Yq*O`L=5TIg0RMJa;oaj`0aE--a_4Fc2b~-EWT&$a6rQ z6ma3JQlAWi1i$51GfJ`?Og#Eh&$q5maIRm@zGf}awDRKh?VNV5-#|W9K}(8vPlVD@ zrV~ekuwbC~)xoM}+MCr@xoggPbwWw&LG9F@NKdE>iQkTvQ+bm|KH&&_6JY5ocRfrw zE80M6f{L12_5%Kxf)JhE0w+e;u#8mdgLV#R9GXQpnwHXy4mEFfMh?JclFTPTHI|6Q zZ#H-{*%^zI^(JR?)KjqXh`st9)o}^2=IV{mJdV(H zZW>l<8*kA=YR*+UkvCa=W#diy-5Zj8zpblg0LK=OUJ<5!?=N%9tlKvu*hOBXL$@DmhmKksXrQ4H!L;{A0>SFdoq>w@bo28%D537lo7l6vu>f zoXEfc#q7BnAYDHa`Ry(m%w~^EmR0x+^Kt0gM_Q5YY=;*IoW2=~ zGtTk>JXBg1VKBeKoz?R}3Z(+Bjqoj_D3JcaHn>&fX3#qU$f_H?$?%w+elBdJ4;yMG w&)MjEx{cXe6c~omQCwM9&|CXc`ENSag3Chx*$`R zMh&8ml+nEx&Mx^+r__i`i~~#ofd;Kdz?{Ty11>IZG*~)A{$Md;r>vEfb+sY(fV5L! zhS+aAMM6%hR>MXy{?%-=rBhjybDhuXG%V8&NAVY}OR%RI?m2LT79D;ynPEf))kX&U zS0w#eN?_I{_axb$yV(E+q+Ie`?-esy9M9JWyW3Ovo(|=jcopOVH0ClMjn%8xAhK2r zdF&VoX8m$F_KSGn-rkvit~Yv<()ToujvlZqCAnld=aTR;xpnTJM8*w}gyi|>7=#X? z4-`;}T?qz<%jMwO@F4Qf!TaO>5Fu4=vuNvUM8Tp1$6zb#J^dr|t#9O`@S!&ygBzAM zleQQA-(!>$%-x`vUU4XFCNMgo>NAoGBCB}bJL9?CVRsQU@_sA36j|R$wZ9Ez^mY*! z4eG%U2DaX;57nr18rD{Opr6geQ%g_qe|d^o?Ka;EjCL0*?6`9vI za7|Z`pSs0b62cCod$Xx(uJB3w$G&~e7(lg1WNra2 zrIO-}9g_xNc^~!R6ii%T-kz=}2@^%t8J%}NpIR}b7t2DJ=OHGF)OVTK*%7tx6sXOv zLckcYuLB|fIpP}k>9+id-$9lr>JDxIx{+hVp_xRF*}RGb>nJ;%A`QGA&cImykZ@Js z>1ix1mI-&b5PNpj3>=1xYBz4;YY+-R@EANuR7n^^jWU=;%*$69$upve9b&vNi!HY02lX7BD_~;8Y{Qn zfry`$$kySY#1I7}y|^fz5YzP%^UWB=#aU)~P19@Abr43OGZ&N3`?yx6bLTq7wpAcltLO|-&2ZwelSoSN><|nSNb@<4 z4LX2f7U0;><;9ELV&mVGvp&HBIG1YbMt}sX=5&oCj2)0h8-L( z#reqat2go``DmAYGP0xi!gd@Dtf#zz0wJ1dU7+LwB3Z6W>?j%9rqSelwL9AxB<>hh zZE>KO4%3(zO>xbZjV6BT#Oe(YQrA=B)r3t*%wq|5ne}k(OMgqVK-Expz-mvsasgxb%{8rS0v;iVQ!8^ChgJ%&2P?3V32EDeg;waZ;!WRvi&*F zRn!1l;}>CLZY=X%6gS*+VYex@qA+x%Wh-C-r}N7-*#yfCz+pO7XsPad>QFN4h$tSl z-wq`8HOYC1;iL#9FacT-=&T+W7C`>vkc=W>%Oj@WhwbZ@AnH3WEqXCqDT>vMqOGHw z)PxEpaqCQHQH8O3# zecNR|lMcoex1|FQMkPL>fu+8${+D1a)hE`rtzOu=S#SVP!juYmd^G-OU+4iymD1;h z$2>3QS9bdxKMEI$=$*YeSLT#U(=8_4czIRIwk}q8iejj|QU5jbL^k40f7tLyCh0AI z=HwzCq=73+@P%1KE*`7J+@+toGh7tsllVLGm9kaLR|!DTxk4aj3jZz5&7Wo*f~z4d zn9yrsOByd)aa3loX6s{b(vi)XtwOc|3sDh*U@3Y{if_x_4MJNZ8B*4hiuxcb^bH47 zsv?^n3cmkCFk`hs0As(BhXYsevv61@a&+}C=6F~rpWGgUV?z_s50)EMj5-Cc4K*dP zIF{CUpxrETT>c4V*9Nr}rY+cZ4n76uuqnH?qVFZ)YstG*@&@MurWN!QQtR0mXOVLQ zQVM~Nz{eO{Ew0>`(J&{eg_-pt`hrvMFLf-I|L*A6a*JGC=Tx)U;R@I??6gA(i zaDSh15P5C$>suVVF;v<8WIJ@{{MOC8jeqv4a+I?VHuLBtF!UP%&9ntv27J{3amc_r z8pRVgt`a)ZTL?h~u;?xN+J*;;7SN)Mp&|jr5;Gwue$x=pzSP^-7dmwrLwh)1(1~cS z${jv(phESvjKB)>q_r7(g-nklxDXdmahtFjSaKOBi|?8i8h=1C`ATbE7YpyqM2brq z!h6mVd?fi`9I230x59s|Zea;Z-#SXx`go{(l@Wf8`xZn{24mY)-UjnYyAe?GFKYKX zPGJ(EyjomFdL6)(q|^%wQY8}oizrvF4=(`Z#LI1w9D3+$-!Qk^t-$whn+Mws zT#qT;wJNPM)k+Jhu5ThdD3e|C8o~sE;3_i$Fx{g@kvf!bX#+>dpLDMsTuA3>0}N!D z=^Do?R~1_H@W2k%`~Yy|h;Nv7-Q%J`B%Umk_Hb1Xw#XLZ)g3I;G>$&@a;5_Qt)l2C!WDnh4R4=bq3` z@QQztfodAEl&j<*ube^ReW%o{z2F+!*!6oG;J>t`Y?*Ztp z1&#c!Bhy=p$0}i!&DCQGAw1Y=(D&?e*tSg^4t$*4W)2#u(>%c`73Swa$fNbBV1q^hZY8qpfrq4MHu<757 zTfI?K?^D!S$-TMy`_VFDhvA_{-nSONf{%Odi)tk|Y!#gQvX5e1XD3!-hCR>yl%n8C2?vAhdSg`~_B5#(^Gf!mJBX%N|Ej#GP$U?A?00000 diff --git a/secrets/privkey.pem b/secrets/privkey.pem index 1f38edfe7f64e5da802c1f3daa94167c11b0008c..b6cf256cb1d1f46d83c38f809eef74e4c60bb775 100644 GIT binary patch literal 263 zcmV+i0r>s^M@dveQdv+`03vZ8CXo!^@Z zL?)U66bRs6SgQ1RGKiGlzR#={g+Pt=r>`Z3qMnqM-RSu%&GyKLiDWZ6 NpmOw+_Ece)w#9Z{dP)EQ literal 263 zcmV+i0r>s^M@dveQdv+`0N&mghVKvigB;3ayK&7`eW=``aM3;VEPI}FiS0aQ-c6BE zSUn+TG~Ou2O$VyW)chr&Fs+ByhapO*YDi;T*%yrn2pVa@53yDu)-9r0(2O8M95@md zA0sp*y!f#6*L;lg9@R=~+4o^|t2vH5S8;BwpVUob9%`FlTju@mYqe_4EG4(T+qg!5 z%c2!hWZ`^opl3YwSC)>pE>)T;E#DlO)u_d%J`8Rx8-5mCO;&yrS+(bQf6|+kl0oMO zxkUv)=QCV*z*7~$(=CVC2Sr^wE_u7~7(C%8$Qu=WL14OSrGbI;jTR-1#)+7tO*7G7 NHojt+9=7Oj(0E@Wd%pkx From 6e77d1870e5e4d9dec9bafa68cb258b3c7d8f392 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Mon, 4 May 2026 07:52:09 +0000 Subject: [PATCH 18/25] mailserver: fix e2e probe shell-quoting bug (apostrophe in comment) The 2026-05-02 change that added the Brevo defensive-unblock step to the email-roundtrip-monitor cron contained an apostrophe in a Python comment ("wasn't"). The whole script is wrapped in shell single quotes (python3 -c '...'), so the apostrophe terminated the shell string. Python only parsed up to the apostrophe and raised IndentationError on the now-bodyless try: block; everything after was handed to /bin/sh which complained about "try::" and unmatched parens. Result: every probe run since 2026-05-02 00:41 UTC crashed before it could push, and the "Email Roundtrip E2E" Uptime Kuma push monitor went DOWN with "No heartbeat in the time window". Fix: rewrite the comment without an apostrophe and add a banner warning so the next person editing this heredoc does not regress. Validated: shell parses (bash -n), Python compiles (py_compile) with the wrapping single quotes intact. --- stacks/mailserver/modules/mailserver/main.tf | 23 +++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/stacks/mailserver/modules/mailserver/main.tf b/stacks/mailserver/modules/mailserver/main.tf index 43a113e3..dd9dd6bf 100644 --- a/stacks/mailserver/modules/mailserver/main.tf +++ b/stacks/mailserver/modules/mailserver/main.tf @@ -829,11 +829,32 @@ DOMAIN = "viktorbarzin.me" marker = f"e2e-probe-{uuid.uuid4().hex[:12]}" subject = f"[E2E Monitor] {marker}" +recipient = f"smoke-test@{DOMAIN}" start = time.time() success = 0 duration = 0 try: + # Step 0: Defensive unblock. Brevo permanently blocks a recipient after a + # single hardBounce — once blocked, every subsequent /smtp/email request + # returns 201 but the message is silently dropped (event=blocked). + # Single transient pod outage → permanent probe outage. Idempotent: 204 if + # the recipient was blocked, 404 if not blocked — both are fine. + # NOTE: this script is wrapped in shell single quotes (see the python3 -c + # invocation above). Do NOT use apostrophes anywhere here, including in + # comments — a stray apostrophe terminates the shell string and Python + # only sees the prefix, raising IndentationError on this try block. + try: + unblock = requests.delete( + f"https://api.brevo.com/v3/smtp/blockedContacts/{recipient}", + headers={"api-key": BREVO_API_KEY, "Accept": "application/json"}, + timeout=10, + ) + if unblock.status_code == 204: + print(f"WARN: {recipient} was blocked at Brevo, unblocked") + except Exception as ue: + print(f"Unblock attempt failed (non-critical): {ue}") + # Step 1: Send via Brevo Transactional Email API to smoke-test@ (hits catch-all -> spam@) resp = requests.post( "https://api.brevo.com/v3/smtp/email", @@ -844,7 +865,7 @@ try: }, json={ "sender": {"name": "Monitoring", "email": f"monitoring@{DOMAIN}"}, - "to": [{"email": f"smoke-test@{DOMAIN}"}], + "to": [{"email": recipient}], "subject": subject, "textContent": f"E2E email monitoring probe {marker}. Auto-generated, will be deleted.", }, From 11a615e723afff2b49eb136c37c025f13dacb674 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Mon, 4 May 2026 08:05:53 +0000 Subject: [PATCH 19/25] mailserver: retrigger CI to apply 6e77d187 Pipeline 1846 apply step errored at ~5s post-start, leaving the e2e probe shell-quoting fix unapplied. K8s state was patched manually as a temporary unblock; this empty commit retriggers CI to land the source fix and resolve drift. [ci] Co-Authored-By: Claude Opus 4.7 (1M context) From 1cb2bb30f7624221632c79523ade9c7ae6ab9380 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 5 May 2026 18:43:26 +0000 Subject: [PATCH 20/25] monitoring(wealth): show pre-2024 historical data on timeseries Bug: timeseries panels were empty before 2024-04-10. Cause was the complete_dates CTE filtering to "every active account has a row for this date" -- which excluded every day before the most-recently-added account first appeared. The 6th account (Trading212 Invest GIA) only started 2024-04-10, so 4 years of legitimate historical data (2020-06-07 onwards, when the user genuinely had fewer accounts) got hidden. New pattern across panels 5/6/7/8/9/12/13: replace complete_dates with max_complete cutoff. Compute the most-recent date where all current accounts have a row, then include every historical date up to and including that day. Partial-today is still excluded automatically. Historical days with fewer accounts now show as their actual smaller sums -- which is the correct historical net worth at the time. Verified via PG: new pattern returns 2,159 distinct days from 2020-06-07 to 2026-05-05 (vs the previous 391 from 2024-04-10). Per-account first-seen dates: InvestEngine ISA - 2020-06-07 Schwab US workplace - 2020-11-17 InvestEngine GIA - 2022-03-17 Fidelity UK Pension - 2022-05-16 Trading212 ISA - 2024-04-08 Trading212 Invest GIA - 2024-04-10 (was the bottleneck) --- .../modules/monitoring/dashboards/wealth.json | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/stacks/monitoring/modules/monitoring/dashboards/wealth.json b/stacks/monitoring/modules/monitoring/dashboards/wealth.json index 67536e29..c5719a53 100644 --- a/stacks/monitoring/modules/monitoring/dashboards/wealth.json +++ b/stacks/monitoring/modules/monitoring/dashboards/wealth.json @@ -223,7 +223,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT valuation_date::timestamp AS \"time\", SUM(total_value) AS net_worth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date ORDER BY valuation_date" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)) SELECT valuation_date::timestamp AS \"time\", SUM(total_value) AS net_worth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date <= (SELECT d FROM max_complete) GROUP BY valuation_date ORDER BY valuation_date" } ] }, @@ -278,7 +278,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT valuation_date::timestamp AS \"time\", SUM(net_contribution) AS net_contribution, SUM(total_value) AS market_value FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date ORDER BY valuation_date" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)) SELECT valuation_date::timestamp AS \"time\", SUM(net_contribution) AS net_contribution, SUM(total_value) AS market_value FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date <= (SELECT d FROM max_complete) GROUP BY valuation_date ORDER BY valuation_date" } ] }, @@ -324,7 +324,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT valuation_date::timestamp AS \"time\", (SUM(total_value) - SUM(net_contribution)) AS growth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date ORDER BY valuation_date" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)) SELECT valuation_date::timestamp AS \"time\", (SUM(total_value) - SUM(net_contribution)) AS growth FROM daily_account_valuation WHERE $__timeFilter(valuation_date) AND valuation_date <= (SELECT d FROM max_complete) GROUP BY valuation_date ORDER BY valuation_date" } ] }, @@ -364,7 +364,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT d.valuation_date::timestamp AS \"time\", a.name AS metric, d.total_value AS value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) AND d.valuation_date IN (SELECT valuation_date FROM complete_dates) ORDER BY d.valuation_date, a.name" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)) SELECT d.valuation_date::timestamp AS \"time\", a.name AS metric, d.total_value AS value FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) AND d.valuation_date <= (SELECT d FROM max_complete) ORDER BY d.valuation_date, a.name" } ] }, @@ -419,7 +419,7 @@ "rawQuery": true, "editorMode": "code", "format": "time_series", - "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)) SELECT d.valuation_date::timestamp AS \"time\", SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN 0 ELSE d.cash_balance END) AS cash, SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN d.cash_balance + d.investment_market_value ELSE d.investment_market_value END) AS invested FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) AND d.valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY d.valuation_date ORDER BY d.valuation_date" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)) SELECT d.valuation_date::timestamp AS \"time\", SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN 0 ELSE d.cash_balance END) AS cash, SUM(CASE WHEN a.account_type = 'WORKPLACE_PENSION' THEN d.cash_balance + d.investment_market_value ELSE d.investment_market_value END) AS invested FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id WHERE $__timeFilter(d.valuation_date) AND d.valuation_date <= (SELECT d FROM max_complete) GROUP BY d.valuation_date ORDER BY d.valuation_date" } ] }, @@ -629,7 +629,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)), yearly AS (SELECT EXTRACT(YEAR FROM valuation_date)::int AS yr, valuation_date, SUM(total_value) AS nw, SUM(net_contribution) AS contrib FROM daily_account_valuation WHERE valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date), endpoints AS (SELECT yr, (array_agg(nw ORDER BY valuation_date ASC))[1] AS nw_start, (array_agg(nw ORDER BY valuation_date DESC))[1] AS nw_end, (array_agg(contrib ORDER BY valuation_date ASC))[1] AS contrib_start, (array_agg(contrib ORDER BY valuation_date DESC))[1] AS contrib_end FROM yearly GROUP BY yr) SELECT yr::text AS year, ROUND((((nw_end - nw_start - (contrib_end - contrib_start)) / NULLIF(nw_start + 0.5 * (contrib_end - contrib_start), 0)) * 100)::numeric, 2) AS return_pct FROM endpoints WHERE (nw_start + 0.5 * (contrib_end - contrib_start)) > 0 ORDER BY yr" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)), yearly AS (SELECT EXTRACT(YEAR FROM valuation_date)::int AS yr, valuation_date, SUM(total_value) AS nw, SUM(net_contribution) AS contrib FROM daily_account_valuation WHERE valuation_date <= (SELECT d FROM max_complete) GROUP BY valuation_date), endpoints AS (SELECT yr, (array_agg(nw ORDER BY valuation_date ASC))[1] AS nw_start, (array_agg(nw ORDER BY valuation_date DESC))[1] AS nw_end, (array_agg(contrib ORDER BY valuation_date ASC))[1] AS contrib_start, (array_agg(contrib ORDER BY valuation_date DESC))[1] AS contrib_end FROM yearly GROUP BY yr) SELECT yr::text AS year, ROUND((((nw_end - nw_start - (contrib_end - contrib_start)) / NULLIF(nw_start + 0.5 * (contrib_end - contrib_start), 0)) * 100)::numeric, 2) AS return_pct FROM endpoints WHERE (nw_start + 0.5 * (contrib_end - contrib_start)) > 0 ORDER BY yr" } ] }, @@ -695,7 +695,7 @@ "rawQuery": true, "editorMode": "code", "format": "table", - "rawSql": "WITH active_accounts AS (SELECT COUNT(*) AS n FROM accounts), complete_dates AS (SELECT d.valuation_date FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date HAVING COUNT(*) >= (SELECT n FROM active_accounts)), yearly AS (SELECT EXTRACT(YEAR FROM valuation_date)::int AS yr, valuation_date, SUM(total_value) AS nw, SUM(net_contribution) AS contrib FROM daily_account_valuation WHERE valuation_date IN (SELECT valuation_date FROM complete_dates) GROUP BY valuation_date), endpoints AS (SELECT yr, (array_agg(nw ORDER BY valuation_date ASC))[1] AS nw_start, (array_agg(nw ORDER BY valuation_date DESC))[1] AS nw_end, (array_agg(contrib ORDER BY valuation_date ASC))[1] AS contrib_start, (array_agg(contrib ORDER BY valuation_date DESC))[1] AS contrib_end FROM yearly GROUP BY yr) SELECT yr::text AS year, ROUND((contrib_end - contrib_start)::numeric, 0) AS contributions, ROUND((nw_end - nw_start - (contrib_end - contrib_start))::numeric, 0) AS market_gain FROM endpoints ORDER BY yr" + "rawSql": "WITH active_count AS (SELECT COUNT(*) AS n FROM accounts), max_complete AS (SELECT MAX(valuation_date) AS d FROM (SELECT d.valuation_date, COUNT(*) AS c FROM daily_account_valuation d JOIN accounts a ON a.id = d.account_id GROUP BY d.valuation_date) x WHERE c >= (SELECT n FROM active_count)), yearly AS (SELECT EXTRACT(YEAR FROM valuation_date)::int AS yr, valuation_date, SUM(total_value) AS nw, SUM(net_contribution) AS contrib FROM daily_account_valuation WHERE valuation_date <= (SELECT d FROM max_complete) GROUP BY valuation_date), endpoints AS (SELECT yr, (array_agg(nw ORDER BY valuation_date ASC))[1] AS nw_start, (array_agg(nw ORDER BY valuation_date DESC))[1] AS nw_end, (array_agg(contrib ORDER BY valuation_date ASC))[1] AS contrib_start, (array_agg(contrib ORDER BY valuation_date DESC))[1] AS contrib_end FROM yearly GROUP BY yr) SELECT yr::text AS year, ROUND((contrib_end - contrib_start)::numeric, 0) AS contributions, ROUND((nw_end - nw_start - (contrib_end - contrib_start))::numeric, 0) AS market_gain FROM endpoints ORDER BY yr" } ] }, From c4c5057edce28401760e3bd607af66d3e5e86d8d Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 5 May 2026 19:14:11 +0000 Subject: [PATCH 21/25] priority-pass: pin to backend 7c01448d (transplant QR into golden-position container) [ci skip] --- stacks/priority-pass/main.tf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stacks/priority-pass/main.tf b/stacks/priority-pass/main.tf index 983d4ae6..4b320df7 100644 --- a/stacks/priority-pass/main.tf +++ b/stacks/priority-pass/main.tf @@ -80,7 +80,7 @@ resource "kubernetes_deployment" "priority-pass" { } container { name = "frontend" - image = "docker.io/viktorbarzin/priority-pass-frontend:50a432ad" + image = "docker.io/viktorbarzin/priority-pass-frontend:7c01448d" port { container_port = 3000 } @@ -104,7 +104,7 @@ resource "kubernetes_deployment" "priority-pass" { } container { name = "backend" - image = "docker.io/viktorbarzin/priority-pass-backend:50a432ad" + image = "docker.io/viktorbarzin/priority-pass-backend:7c01448d" port { container_port = 8000 } From 4c8d12229f428e4a4cbb7fd4d3bd8176c917b2ca Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 5 May 2026 19:45:33 +0000 Subject: [PATCH 22/25] mailserver: split healthcheck path off PROXY-aware listeners + book-search uses ClusterIP MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two coordinated fixes for the same root cause: Postfix's smtpd_upstream_proxy_protocol listener fatals on every HAProxy health probe with `smtpd_peer_hostaddr_to_sockaddr: ... Servname not supported for ai_socktype` — the daemon respawns get throttled by postfix master, and real client connections that land mid-respawn time out. We saw this as ~50% timeout rate on public 587 from inside the cluster. Layer 1 (book-search) — stacks/ebooks/main.tf: SMTP_HOST mail.viktorbarzin.me → mailserver.mailserver.svc.cluster.local Internal services should use ClusterIP, not hairpin through pfSense+HAProxy. 12/12 OK in <28ms vs ~6/12 timeouts on the public path. Layer 2 (pfSense HAProxy) — stacks/mailserver + scripts/pfsense-haproxy-bootstrap.php: Add 3 non-PROXY healthcheck NodePorts to mailserver-proxy svc: 30145 → pod 25 (stock postscreen) 30146 → pod 465 (stock smtps) 30147 → pod 587 (stock submission) HAProxy uses `port ` (per-server in advanced field) to redirect L4 health probes to those ports while real client traffic keeps going to 30125-30128 with PROXY v2. Result: 0 fatals/min (was 96), 30/30 probes OK on 587, e2e roundtrip 20.4s. Inter dropped 120000 → 5000 since log-spam concern is gone. `option smtpchk EHLO` was tried first but flapped against postscreen (multi-line greet + DNSBL silence + anti-pre-greet detection trip HAProxy's parser → L7RSP). Plain TCP accept-on-port check is sufficient for both submission and postscreen. Updated docs/runbooks/mailserver-pfsense-haproxy.md to reflect the new healthcheck path and mark the "Known warts" entry as resolved. Co-Authored-By: Claude Opus 4.7 --- docs/runbooks/mailserver-pfsense-haproxy.md | 45 +++++++++---- scripts/pfsense-haproxy-bootstrap.php | 68 +++++++++++++++++--- stacks/ebooks/main.tf | 14 +++- stacks/mailserver/modules/mailserver/main.tf | 29 +++++++++ 4 files changed, 131 insertions(+), 25 deletions(-) diff --git a/docs/runbooks/mailserver-pfsense-haproxy.md b/docs/runbooks/mailserver-pfsense-haproxy.md index 564554eb..329be214 100644 --- a/docs/runbooks/mailserver-pfsense-haproxy.md +++ b/docs/runbooks/mailserver-pfsense-haproxy.md @@ -12,7 +12,11 @@ so pfSense runs a small HAProxy that: 1. Listens on the pfSense VLAN20 IP (`10.0.20.1`) on all 4 mail ports, 2. Forwards each connection to a k8s node's NodePort with `send-proxy-v2`, 3. Injects PROXY v2 framing so Postfix/Dovecot see the original client IP, -4. TCP health-checks every k8s worker — any node can serve (ETP:Cluster). +4. TCP-checks every k8s worker via dedicated **non-PROXY healthcheck NodePorts** + (30145/30146/30147 → pod stock 25/465/587 listeners, no PROXY required). + This split path avoids the `smtpd_peer_hostaddr_to_sockaddr` fatal that + used to fire on every PROXY-aware health probe and throttled real client + connections. Corresponding k8s-side setup (`stacks/mailserver/modules/mailserver/`): @@ -23,14 +27,20 @@ Corresponding k8s-side setup (`stacks/mailserver/modules/mailserver/`): - `:5587` smtpd (alt :587 submission) with `smtpd_upstream_proxy_protocol=haproxy` - ConfigMap `mailserver.config` adds Dovecot `inet_listener imaps_proxy` on port 10993 with `haproxy = yes` and `haproxy_trusted_networks = 10.0.20.0/24`. -- Service `mailserver-proxy` (NodePort, ETP:Cluster) with 4 NodePorts: - - `port 25 → targetPort 2525 → nodePort 30125` - - `port 465 → targetPort 4465 → nodePort 30126` - - `port 587 → targetPort 5587 → nodePort 30127` - - `port 993 → targetPort 10993 → nodePort 30128` +- Service `mailserver-proxy` (NodePort, ETP:Cluster) — 4 PROXY data ports + + 3 non-PROXY healthcheck ports: + - Data (PROXY v2): + - `port 25 → targetPort 2525 → nodePort 30125` + - `port 465 → targetPort 4465 → nodePort 30126` + - `port 587 → targetPort 5587 → nodePort 30127` + - `port 993 → targetPort 10993 → nodePort 30128` + - Healthcheck (no PROXY, stock SMTP/SMTPS/Submission listeners): + - `port 2500 → targetPort 25 → nodePort 30145` (smtp-check) + - `port 4650 → targetPort 465 → nodePort 30146` (smtps-check) + - `port 5870 → targetPort 587 → nodePort 30147` (sub-check) - Service `mailserver` (ClusterIP) — unchanged stock ports 25/465/587/993 for intra-cluster clients (Roundcube pod, `email-roundtrip-monitor` - CronJob). These listeners are PROXY-free. + CronJob, book-search). These listeners are PROXY-free. bd: `code-yiu`. @@ -46,7 +56,9 @@ External mail (WAN) path — PROXY v2 │ │ NAT rdr → 10.0.20.1:{same} │ │ ▼ │ │ pfSense HAProxy (mode tcp, 4 frontends, 4 backend pools) │ -│ │ send-proxy-v2 + tcp-check inter 120000 │ +│ │ data: send-proxy-v2 → :{30125..30128} (PROXY-aware pod) │ +│ │ health: TCP-check → :{30145..30147} (no-PROXY pod) │ +│ │ inter 5000 │ │ ▼ │ │ k8s-node<1-4>:{30125..30128} ← any node (ETP:Cluster) │ │ │ kube-proxy SNAT (source IP lost on the wire) │ @@ -186,11 +198,18 @@ Full restore: pfSense WebUI → Diagnostics → Backup & Restore → Upload that ## Known warts -- HAProxy TCP health-check with `send-proxy-v2` generates `getpeername: - Transport endpoint not connected` warnings on postscreen every check cycle. - Mitigated with `inter 120000` (2 min). To reduce further, switch to - `option smtpchk` — but that requires a separate non-PROXY health-check - port on the pod (not done yet). +- ~~HAProxy TCP health-check with `send-proxy-v2` generates `getpeername: + Transport endpoint not connected` warnings on postscreen every check cycle.~~ + **Resolved 2026-05-05**: dedicated non-PROXY healthcheck NodePorts + (30145/30146/30147 → stock pod 25/465/587) added; HAProxy now checks + those, eliminating both the `getpeername` postscreen warnings and the + `smtpd_peer_hostaddr_to_sockaddr: ... Servname not supported` fatals + that were throttling smtpd respawns and causing ~50% client timeouts on + the public 587 path. `inter` dropped 120000 → 5000 (fast failover, no + log-spam concern). `option smtpchk` was tried but flapped against + postscreen (multi-line greet + DNSBL silence + anti-pre-greet detection + trip HAProxy's parser → L7RSP). Plain TCP check on the no-PROXY ports + is sufficient. - Frontend binds on all pfSense interfaces (`bind :25` instead of `10.0.20.1:25`). `` is set in XML but pfSense templates it port-only. Low concern in practice because WAN firewall rules plus the diff --git a/scripts/pfsense-haproxy-bootstrap.php b/scripts/pfsense-haproxy-bootstrap.php index 3834d852..5452b198 100644 --- a/scripts/pfsense-haproxy-bootstrap.php +++ b/scripts/pfsense-haproxy-bootstrap.php @@ -68,7 +68,35 @@ $NODES = [ ['k8s-node4', '10.0.20.104'], ]; -function build_pool(string $name, string $nodeport, array $nodes): array { +// Build a pool with optional split healthcheck path. +// +// $check_port: if non-null, HAProxy sends health probes to that NodePort +// (which Service `mailserver-proxy` maps to the pod's stock no-PROXY +// listener — see infra/stacks/mailserver/.../mailserver_proxy ports +// 30145/30146/30147). Real client traffic still goes to $nodeport with +// PROXY v2 framing. +// $check_type: 'TCP' for plain accept-on-port checks, 'ESMTP' for +// `option smtpchk EHLO ` (real SMTP banner+EHLO+250). +// +// Why split: smtpd-proxy587/4465 fatal on every PROXY-v2-aware health +// probe with `smtpd_peer_hostaddr_to_sockaddr: ... Servname not supported` +// — the daemon respawns get throttled by Postfix master and real clients +// land mid-respawn → 6s TCP timeout. Routing health probes to the stock +// no-PROXY port sidesteps the bug entirely while data path still gets +// PROXY v2 for CrowdSec/Postfix client-IP visibility. The HAProxy package +// has no `checkport` field, so `port N` is appended via the server's +// `advanced` string (HAProxy parses server keywords in any order). +function build_pool( + string $name, + string $nodeport, + array $nodes, + string $check_type = 'TCP', + ?string $check_port = null, + string $monitor_domain = '' +): array { + $advanced_check = $check_port !== null + ? "send-proxy-v2 port {$check_port}" + : 'send-proxy-v2'; $servers = []; foreach ($nodes as $n) { $servers[] = [ @@ -77,18 +105,19 @@ function build_pool(string $name, string $nodeport, array $nodes): array { 'port' => $nodeport, 'weight' => '10', 'ssl' => '', - // check every 2 min — send-proxy-v2 check + close generates - // noise on postscreen, not worth doing more often. - 'checkinter' => '120000', - 'advanced' => 'send-proxy-v2', + // 5s = sub-block-window failover when a NodePort goes sour. + // Safe to be aggressive once health probes don't fatal smtpd. + 'checkinter' => '5000', + 'advanced' => $advanced_check, 'status' => 'active', ]; } return [ 'name' => $name, 'balance' => 'roundrobin', - 'check_type' => 'TCP', - 'checkinter' => '120000', + 'check_type' => $check_type, + 'monitor_domain' => $monitor_domain, + 'checkinter' => '5000', 'retries' => '3', 'ha_servers' => ['item' => $servers], 'advanced_bind' => '', @@ -132,9 +161,28 @@ $h['ha_pools']['item'] = array_values(array_filter( $h['ha_pools']['item'][] = build_pool('mailserver_nodes', '30125', $NODES); // Production pools — one per mail port. -$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtp', '30125', $NODES); -$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtps', '30126', $NODES); -$h['ha_pools']['item'][] = build_pool('mailserver_nodes_sub', '30127', $NODES); +// +// All SMTP/SMTPS/Submission backends use plain TCP checks against +// dedicated non-PROXY healthcheck NodePorts (30145/30146/30147 → pod +// stock 25/465/587) so probes hit the no-PROXY listeners and avoid +// the smtpd_peer_hostaddr_to_sockaddr fatal that fires on PROXY-v2 +// LOCAL frames. Real client traffic still goes to 30125-30128 with +// PROXY v2 for client-IP visibility. +// +// We tried `option smtpchk EHLO` initially — it works on the plain +// `submission` daemon (587) but flaps the `postscreen` listener on +// port 25 (multi-line greet + DNSBL silence + anti-pre-greet +// detection makes HAProxy's simple smtpchk parser hit L7RSP). A +// plain TCP accept-on-port check is enough for both: HAProxy still +// gets fast failover when the listener actually goes away, and we +// stop triggering the Postfix fatal entirely. +// +// IMAPS stays on its existing TCP-check-with-PROXY-frame for now — +// Dovecot's PROXY parser doesn't show the same fatal pattern; adding +// a separate IMAP healthcheck path would require another svc port. +$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtp', '30125', $NODES, 'TCP', '30145'); +$h['ha_pools']['item'][] = build_pool('mailserver_nodes_smtps', '30126', $NODES, 'TCP', '30146'); +$h['ha_pools']['item'][] = build_pool('mailserver_nodes_sub', '30127', $NODES, 'TCP', '30147'); $h['ha_pools']['item'][] = build_pool('mailserver_nodes_imaps', '30128', $NODES); // ── Frontends ─────────────────────────────────────────────────────────── diff --git a/stacks/ebooks/main.tf b/stacks/ebooks/main.tf index 7500e579..c6978a05 100644 --- a/stacks/ebooks/main.tf +++ b/stacks/ebooks/main.tf @@ -785,8 +785,18 @@ resource "kubernetes_deployment" "book_search" { } } env { - name = "SMTP_HOST" - value = "mail.viktorbarzin.me" + name = "SMTP_HOST" + # Use intra-cluster ClusterIP path — bypasses pfSense HAProxy + + # PROXY v2 (the public path hairpins through HAProxy:587 → + # NodePort → pod :5587 where Postfix's smtpd-proxy587 daemon + # crashes ~50% of HAProxy healthchecks with + # `smtpd_peer_hostaddr_to_sockaddr: ... Servname not supported`, + # producing intermittent 6s TCP timeouts for clients that land + # mid-respawn). The ClusterIP service points to pod port 587 + # (stock submission daemon, no PROXY) and is rock-solid (12/12 + # in <31ms vs 6/12 timeouts on the public path). + # See docs/runbooks/mailserver-pfsense-haproxy.md. + value = "mailserver.mailserver.svc.cluster.local" } env { name = "SMTP_PORT" diff --git a/stacks/mailserver/modules/mailserver/main.tf b/stacks/mailserver/modules/mailserver/main.tf index dd9dd6bf..c3c33d26 100644 --- a/stacks/mailserver/modules/mailserver/main.tf +++ b/stacks/mailserver/modules/mailserver/main.tf @@ -733,6 +733,35 @@ resource "kubernetes_service" "mailserver_proxy" { target_port = 10993 node_port = 30128 } + # Dedicated non-PROXY healthcheck NodePorts. HAProxy on pfSense uses + # `option smtpchk` against these stock pod ports (25/465/587, no PROXY) + # so health probes don't hit the smtpd_peer_hostaddr_to_sockaddr fatal + # that fires on PROXY-v2 LOCAL/AF_UNSPEC frames sent during checks. The + # data path (30125-30128 → 2525/4465/5587/10993) still gets PROXY v2 for + # real client IP visibility — only the healthcheck path is split off. + # See infra/scripts/pfsense-haproxy-bootstrap.php (`check port` directive) + # and docs/runbooks/mailserver-pfsense-haproxy.md. + port { + name = "smtp-check" + protocol = "TCP" + port = 2500 + target_port = 25 + node_port = 30145 + } + port { + name = "smtps-check" + protocol = "TCP" + port = 4650 + target_port = 465 + node_port = 30146 + } + port { + name = "sub-check" + protocol = "TCP" + port = 5870 + target_port = 587 + node_port = 30147 + } } } From fb454e16d5bdc0e7eea80f2cdb32cdae81750a1d Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Tue, 5 May 2026 21:03:46 +0000 Subject: [PATCH 23/25] priority-pass: parameterise image_tag via var pattern (matches job-hunter) Adopts the always-latest convention used by job-hunter, payslip-ingest, and fire-planner: image SHA lives in stacks/priority-pass/terragrunt.hcl inputs, default in main.tf var. The priority-pass GHA build workflow auto-commits new SHAs to this file on every successful push. - Add `variable "image_tag"` (default = current value 7c01448d). - Both containers now use `local.{frontend,backend}_image` interpolation. - Replace symlinked terragrunt.hcl with a real file so the stack-local inputs block can override image_tag (mirrors payslip-ingest exactly). State note: priority-pass TF state is currently empty (Tier 1 PG migration skipped this stack). A subsequent `terragrunt import` is required to adopt the live deployment + namespace + ingress before running apply. Co-Authored-By: Claude Opus 4.7 --- stacks/priority-pass/main.tf | 15 +++++++++++++-- stacks/priority-pass/terragrunt.hcl | 26 +++++++++++++++++++++++++- 2 files changed, 38 insertions(+), 3 deletions(-) mode change 120000 => 100644 stacks/priority-pass/terragrunt.hcl diff --git a/stacks/priority-pass/main.tf b/stacks/priority-pass/main.tf index 4b320df7..d02aa651 100644 --- a/stacks/priority-pass/main.tf +++ b/stacks/priority-pass/main.tf @@ -1,8 +1,19 @@ +variable "image_tag" { + type = string + default = "7c01448d" + description = "priority-pass image tag (applies to both frontend + backend). Use 8-char git SHA in CI; :latest only for local trials." +} + variable "tls_secret_name" { type = string sensitive = true } +locals { + frontend_image = "docker.io/viktorbarzin/priority-pass-frontend:${var.image_tag}" + backend_image = "docker.io/viktorbarzin/priority-pass-backend:${var.image_tag}" +} + resource "kubernetes_namespace" "priority-pass" { metadata { name = "priority-pass" @@ -80,7 +91,7 @@ resource "kubernetes_deployment" "priority-pass" { } container { name = "frontend" - image = "docker.io/viktorbarzin/priority-pass-frontend:7c01448d" + image = local.frontend_image port { container_port = 3000 } @@ -104,7 +115,7 @@ resource "kubernetes_deployment" "priority-pass" { } container { name = "backend" - image = "docker.io/viktorbarzin/priority-pass-backend:7c01448d" + image = local.backend_image port { container_port = 8000 } diff --git a/stacks/priority-pass/terragrunt.hcl b/stacks/priority-pass/terragrunt.hcl deleted file mode 120000 index af58a92e..00000000 --- a/stacks/priority-pass/terragrunt.hcl +++ /dev/null @@ -1 +0,0 @@ -../../terragrunt.hcl \ No newline at end of file diff --git a/stacks/priority-pass/terragrunt.hcl b/stacks/priority-pass/terragrunt.hcl new file mode 100644 index 00000000..6cb142d9 --- /dev/null +++ b/stacks/priority-pass/terragrunt.hcl @@ -0,0 +1,25 @@ +include "root" { + path = find_in_parent_folders() +} + +dependency "platform" { + config_path = "../platform" + skip_outputs = true +} + +dependency "vault" { + config_path = "../vault" + skip_outputs = true +} + +dependency "external-secrets" { + config_path = "../external-secrets" + skip_outputs = true +} + +inputs = { + # priority-pass repo HEAD — auto-bumped by GHA `build-and-deploy.yml` + # on every successful build. Manual edits welcome for local trials, + # but CI will overwrite on the next push to main. + image_tag = "7c01448d" +} From b45c45e41981a7f07a5f1150c25a147f4461404b Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Tue, 5 May 2026 21:13:14 +0000 Subject: [PATCH 24/25] priority-pass: bump image_tag to 88f18e53 [ci skip] Auto-committed by ViktorBarzin/priority-pass GHA on push to main. Source: https://github.com/ViktorBarzin/priority-pass/commit/88f18e532b2d0a220957768c889635a19dfe6c39 --- stacks/priority-pass/terragrunt.hcl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stacks/priority-pass/terragrunt.hcl b/stacks/priority-pass/terragrunt.hcl index 6cb142d9..42b52856 100644 --- a/stacks/priority-pass/terragrunt.hcl +++ b/stacks/priority-pass/terragrunt.hcl @@ -21,5 +21,5 @@ inputs = { # priority-pass repo HEAD — auto-bumped by GHA `build-and-deploy.yml` # on every successful build. Manual edits welcome for local trials, # but CI will overwrite on the next push to main. - image_tag = "7c01448d" + image_tag = "88f18e53" } From 813148c4af3e2d358d037ea5cdf8d378ddfe76ea Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Wed, 6 May 2026 18:02:25 +0000 Subject: [PATCH 25/25] kms: switch to non-proxied DNS so port 1688 is reachable externally Cloudflare cannot proxy raw TCP/1688 (KMS protocol). Switch kms.viktorbarzin.me from CF-proxied CNAME to direct A/AAAA so clients can reach the vlmcsd LoadBalancer (10.0.20.200) via the existing pfSense WAN port-forward for 1688. Verified end-to-end: vlmcs against 176.12.22.76:1688 completes the KMS V4 handshake for Office Professional Plus 2019. Co-Authored-By: Claude Opus 4.7 --- stacks/kms/main.tf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stacks/kms/main.tf b/stacks/kms/main.tf index 1ad91cd2..3b758159 100644 --- a/stacks/kms/main.tf +++ b/stacks/kms/main.tf @@ -124,7 +124,7 @@ resource "kubernetes_service" "kms-web-page" { module "ingress" { source = "../../modules/kubernetes/ingress_factory" - dns_type = "proxied" + dns_type = "non-proxied" namespace = kubernetes_namespace.kms.metadata[0].name name = "kms" tls_secret_name = var.tls_secret_name