viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

[payslip-ingest] Deploy stack + Grafana dashboard + Vault DB role

Browse source 
## Context

New service `payslip-ingest` (code lives in `/home/wizard/code/payslip-ingest/`)
needs in-cluster deployment, its own Postgres DB + rotating user, a Grafana
datasource, a dashboard, and a Claude agent definition for PDF extraction.

Cluster-internal only — webhook fires from Paperless-ngx in a sibling namespace.
No ingress, no TLS cert, no DNS record.

## What

### New stack `stacks/payslip-ingest/`
- `kubernetes_namespace` payslip-ingest, tier=aux.
- ExternalSecret (vault-kv) projects PAPERLESS_API_TOKEN, CLAUDE_AGENT_BEARER_TOKEN,
  WEBHOOK_BEARER_TOKEN into `payslip-ingest-secrets`.
- ExternalSecret (vault-database) reads rotating password from
  `static-creds/pg-payslip-ingest` and templates `DATABASE_URL` into
  `payslip-ingest-db-creds` with `reloader.stakater.com/match=true`.
- Deployment: single replica, Recreate strategy (matches single-worker queue
  design), `wait-for postgresql.dbaas:5432` annotation, init container runs
  `alembic upgrade head`, main container serves FastAPI on 8080, Kyverno
  dns_config lifecycle ignore.
- ClusterIP Service :8080.
- Grafana datasource ConfigMap in `monitoring` ns (label `grafana_datasource=1`,
  uid `payslips-pg`) reading password from the db-creds K8s Secret.

### Grafana dashboard `uk-payslip.json` (4 panels)
- Monthly gross/net/tax/NI (timeseries, currencyGBP).
- YTD tax-band progression with threshold lines at £12,570 / £50,270 / £125,140.
- Deductions breakdown (stacked bars).
- Effective rate + take-home % (timeseries, percent).

### Vault DB role `pg-payslip-ingest`
- Added to `allowed_roles` in `vault_database_secret_backend_connection.postgresql`.
- New `vault_database_secret_backend_static_role.pg_payslip_ingest`
  (username `payslip_ingest`, 7d rotation).

### DBaaS — DB + role creation
- New `null_resource.pg_payslip_ingest_db` mirrors `pg_terraform_state_db`:
  idempotent CREATE ROLE + CREATE DATABASE + GRANT ALL via `kubectl exec` into
  `pg-cluster-1`.

### Claude agent `.claude/agents/payslip-extractor.md`
- Haiku-backed agent invoked by `claude-agent-service`.
- Decodes base64 PDF from prompt, tries pdftotext → pypdf fallback, emits a single
  JSON object matching the schema to stdout. No network, no file writes outside /tmp,
  no markdown fences.

## Trade-offs / decisions

- Own DB per service (convention), NOT a schema in a shared `app` DB as the plan
  initially described. The Alembic migration still creates a `payslip_ingest`
  schema inside the `payslip_ingest` DB for table organisation.
- Paperless URL uses port 80 (the Service port), not 8000 (the pod target port).
- Grafana datasource uses the primary RW user — separate `_ro` role is aspirational
  and not yet a pattern in this repo.
- No ingress — webhook is cluster-internal; external exposure is unnecessary attack
  surface.
- No Uptime Kuma monitor yet: the internal-monitor list is a static block in
  `stacks/uptime-kuma/`; will add in a follow-up tied to code-z29 (internal monitor
  auto-creator).

## Test Plan

### Automated
```
terraform init -backend=false && terraform validate
Success! The configuration is valid.

terraform fmt -check -recursive
(exit 0)

python3 -c "import json; json.load(open('uk-payslip.json'))"
(exit 0)
```

### Manual Verification (post-merge)

Prerequisites:
1. Seed Vault: `vault kv put secret/payslip-ingest webhook_bearer_token=$(openssl rand -hex 32)`.
2. Seed Vault: `vault kv patch secret/paperless-ngx api_token=<paperless token>`.

Apply:
3. `scripts/tg apply vault` → creates pg-payslip-ingest static role.
4. `scripts/tg apply dbaas` → creates payslip_ingest DB + role.
5. `cd stacks/payslip-ingest && ../../scripts/tg apply -target=kubernetes_manifest.db_external_secret`
   (first-apply ESO bootstrap).
6. `scripts/tg apply payslip-ingest` (full).
7. `kubectl -n payslip-ingest get pods` → Running 1/1.
8. `kubectl -n payslip-ingest port-forward svc/payslip-ingest 8080:8080 && curl localhost:8080/healthz` → 200.

End-to-end:
9. Configure Paperless workflow (README in code repo has steps).
10. Upload sample payslip tagged `payslip` → row in `payslip_ingest.payslip` within 60s.
11. Grafana → Dashboards → UK Payslip → 4 panels render.

Closes: code-do7

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-04-18 19:07:05 +00:00
parent 81e7c3d6ee
commit 43b4e1d372
6 changed files with 774 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

24
stacks/dbaas/modules/dbaas/main.tf
View file

@ -1440,6 +1440,30 @@ resource "null_resource" "pg_terraform_state_db" {
}
}
# Create payslip_ingest database for the payslip-ingest webhook service.
# Role password is managed by Vault Database Secrets Engine (static role `pg-payslip-ingest`, 7d rotation).
resource "null_resource" "pg_payslip_ingest_db" {
depends_on = [null_resource.pg_cluster]
triggers = {
db_name = "payslip_ingest"
username = "payslip_ingest"
}
provisioner "local-exec" {
command = <<-EOT
kubectl --kubeconfig ${var.kube_config_path} exec -n dbaas pg-cluster-1 -c postgres -- \
bash -c '
psql -U postgres -tc "SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = '"'"'payslip_ingest'"'"'" | grep -q 1 || \
psql -U postgres -c "CREATE ROLE payslip_ingest WITH LOGIN PASSWORD '"'"'changeme-vault-will-rotate'"'"'"
psql -U postgres -tc "SELECT 1 FROM pg_catalog.pg_database WHERE datname = '"'"'payslip_ingest'"'"'" | grep -q 1 || \
psql -U postgres -c "CREATE DATABASE payslip_ingest OWNER payslip_ingest"
psql -U postgres -c "GRANT ALL PRIVILEGES ON DATABASE payslip_ingest TO payslip_ingest"
'
EOT
}
}
# Old PostgreSQL deployment — kept commented for rollback reference
# resource "kubernetes_deployment" "postgres" {
# metadata {

224
stacks/monitoring/modules/monitoring/dashboards/uk-payslip.json Normal file
View file

@ -0,0 +1,224 @@
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": { "type": "datasource", "uid": "grafana" },
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "UK payslip breakdown — gross/net/tax/NI trends, YTD progression against income tax bands, deductions split, and effective rate.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"panels": [
{
"id": 1,
"title": "Monthly gross / net / tax / NI",
"type": "timeseries",
"datasource": { "type": "postgres", "uid": "payslips-pg" },
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 0 },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"unit": "currencyGBP",
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": { "legend": false, "tooltip": false, "viz": false },
"lineWidth": 2,
"pointSize": 5,
"scaleDistribution": { "type": "linear" },
"showPoints": "auto",
"spanNulls": false,
"stacking": { "group": "A", "mode": "none" },
"thresholdsStyle": { "mode": "off" }
}
},
"overrides": []
},
"options": {
"legend": { "calcs": ["last", "mean"], "displayMode": "table", "placement": "bottom" },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"datasource": { "type": "postgres", "uid": "payslips-pg" },
"rawSql": "SELECT pay_date AS \"time\", gross_pay, net_pay, income_tax, national_insurance FROM payslip_ingest.payslip WHERE $__timeFilter(pay_date) ORDER BY pay_date",
"format": "time_series",
"refId": "A"
}
]
},
{
"id": 2,
"title": "YTD gross (this tax year) with UK band thresholds",
"type": "timeseries",
"datasource": { "type": "postgres", "uid": "payslips-pg" },
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 0 },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"unit": "currencyGBP",
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "YTD gross",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 15,
"gradientMode": "none",
"hideFrom": { "legend": false, "tooltip": false, "viz": false },
"lineWidth": 2,
"pointSize": 5,
"scaleDistribution": { "type": "linear" },
"showPoints": "auto",
"spanNulls": false,
"stacking": { "group": "A", "mode": "none" },
"thresholdsStyle": { "mode": "line" }
},
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 12570 },
{ "color": "orange", "value": 50270 },
{ "color": "red", "value": 125140 }
]
}
},
"overrides": []
},
"options": {
"legend": { "calcs": ["last", "max"], "displayMode": "table", "placement": "bottom" },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"datasource": { "type": "postgres", "uid": "payslips-pg" },
"rawSql": "SELECT pay_date AS \"time\", SUM(gross_pay) OVER (PARTITION BY tax_year ORDER BY pay_date) AS ytd_gross FROM payslip_ingest.payslip WHERE $__timeFilter(pay_date) ORDER BY pay_date",
"format": "time_series",
"refId": "A"
}
]
},
{
"id": 3,
"title": "Deductions breakdown per payslip",
"type": "timeseries",
"datasource": { "type": "postgres", "uid": "payslips-pg" },
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 9 },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"unit": "currencyGBP",
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "bars",
"fillOpacity": 80,
"gradientMode": "none",
"hideFrom": { "legend": false, "tooltip": false, "viz": false },
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": { "type": "linear" },
"showPoints": "never",
"spanNulls": false,
"stacking": { "group": "A", "mode": "normal" },
"thresholdsStyle": { "mode": "off" }
}
},
"overrides": []
},
"options": {
"legend": { "calcs": ["sum", "mean"], "displayMode": "table", "placement": "bottom" },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"datasource": { "type": "postgres", "uid": "payslips-pg" },
"rawSql": "SELECT pay_date AS \"time\", income_tax, national_insurance, pension_employee, student_loan FROM payslip_ingest.payslip WHERE $__timeFilter(pay_date) ORDER BY pay_date",
"format": "time_series",
"refId": "A"
}
]
},
{
"id": 4,
"title": "Latest effective rate & take-home %",
"type": "timeseries",
"datasource": { "type": "postgres", "uid": "payslips-pg" },
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 9 },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"unit": "percent",
"min": 0,
"max": 100,
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": { "legend": false, "tooltip": false, "viz": false },
"lineWidth": 2,
"pointSize": 5,
"scaleDistribution": { "type": "linear" },
"showPoints": "auto",
"spanNulls": false,
"stacking": { "group": "A", "mode": "none" },
"thresholdsStyle": { "mode": "off" }
}
},
"overrides": []
},
"options": {
"legend": { "calcs": ["last", "mean"], "displayMode": "table", "placement": "bottom" },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"datasource": { "type": "postgres", "uid": "payslips-pg" },
"rawSql": "SELECT pay_date AS \"time\", ROUND(((income_tax + national_insurance)::numeric / NULLIF(gross_pay, 0)) * 100, 2) AS \"effective_rate_pct\", ROUND((net_pay::numeric / NULLIF(gross_pay, 0)) * 100, 2) AS \"take_home_pct\" FROM payslip_ingest.payslip WHERE $__timeFilter(pay_date) ORDER BY pay_date",
"format": "time_series",
"refId": "A"
}
]
}
],
"refresh": "5m",
"schemaVersion": 39,
"tags": ["finance", "personal", "uk-tax"],
"templating": { "list": [] },
"time": { "from": "now-2y", "to": "now" },
"timepicker": {},
"timezone": "browser",
"title": "UK Payslip",
"uid": "uk-payslip",
"version": 1
}

330
stacks/payslip-ingest/main.tf Normal file
View file

@ -0,0 +1,330 @@
variable "image_tag" {
type = string
default = "latest"
description = "payslip-ingest image tag. Use 8-char git SHA in CI; :latest only for local trials."
}
variable "postgresql_host" { type = string }
locals {
namespace = "payslip-ingest"
image = "registry.viktorbarzin.me/payslip-ingest:${var.image_tag}"
labels = {
app = "payslip-ingest"
}
}
resource "kubernetes_namespace" "payslip_ingest" {
metadata {
name = local.namespace
labels = {
tier = local.tiers.aux
"istio-injection" = "disabled"
}
}
}
# App secrets sourced from multiple Vault KV keys.
# Seed these manually in Vault before applying:
# secret/paperless-ngx -> property `api_token`
# secret/claude-agent-service -> property `api_bearer_token`
# secret/payslip-ingest -> property `webhook_bearer_token`
resource "kubernetes_manifest" "external_secret" {
manifest = {
apiVersion = "external-secrets.io/v1beta1"
kind = "ExternalSecret"
metadata = {
name = "payslip-ingest-secrets"
namespace = local.namespace
}
spec = {
refreshInterval = "15m"
secretStoreRef = {
name = "vault-kv"
kind = "ClusterSecretStore"
}
target = {
name = "payslip-ingest-secrets"
template = {
metadata = {
annotations = {
"reloader.stakater.com/match" = "true"
}
}
}
}
data = [
{
secretKey = "PAPERLESS_API_TOKEN"
remoteRef = {
key = "paperless-ngx"
property = "api_token"
}
},
{
secretKey = "CLAUDE_AGENT_BEARER_TOKEN"
remoteRef = {
key = "claude-agent-service"
property = "api_bearer_token"
}
},
{
secretKey = "WEBHOOK_BEARER_TOKEN"
remoteRef = {
key = "payslip-ingest"
property = "webhook_bearer_token"
}
},
]
}
}
depends_on = [kubernetes_namespace.payslip_ingest]
}
# DB credentials from Vault database engine (rotated every 7 days).
# Template builds the asyncpg DSN consumed by the FastAPI app as DB_CONNECTION_STRING.
resource "kubernetes_manifest" "db_external_secret" {
manifest = {
apiVersion = "external-secrets.io/v1beta1"
kind = "ExternalSecret"
metadata = {
name = "payslip-ingest-db-creds"
namespace = local.namespace
}
spec = {
refreshInterval = "15m"
secretStoreRef = {
name = "vault-database"
kind = "ClusterSecretStore"
}
target = {
name = "payslip-ingest-db-creds"
template = {
metadata = {
annotations = {
"reloader.stakater.com/match" = "true"
}
}
data = {
DB_CONNECTION_STRING = "postgresql+asyncpg://payslip_ingest:{{ .password }}@${var.postgresql_host}:5432/payslip_ingest"
DB_PASSWORD = "{{ .password }}"
}
}
}
data = [{
secretKey = "password"
remoteRef = {
key = "static-creds/pg-payslip-ingest"
property = "password"
}
}]
}
}
depends_on = [kubernetes_namespace.payslip_ingest]
}
resource "kubernetes_deployment" "payslip_ingest" {
metadata {
name = "payslip-ingest"
namespace = kubernetes_namespace.payslip_ingest.metadata[0].name
labels = merge(local.labels, {
tier = local.tiers.aux
})
annotations = {
"reloader.stakater.com/search" = "true"
}
}
spec {
replicas = 1
strategy {
type = "Recreate"
}
selector {
match_labels = local.labels
}
template {
metadata {
labels = local.labels
annotations = {
"dependency.kyverno.io/wait-for" = "postgresql.dbaas:5432"
}
}
spec {
image_pull_secrets {
name = "registry-credentials"
}
init_container {
name = "alembic-migrate"
image = local.image
command = ["python", "-m", "payslip_ingest", "migrate"]
env_from {
secret_ref {
name = "payslip-ingest-secrets"
}
}
env_from {
secret_ref {
name = "payslip-ingest-db-creds"
}
}
env {
name = "PAPERLESS_URL"
value = "http://paperless-ngx.paperless-ngx.svc.cluster.local"
}
env {
name = "CLAUDE_AGENT_URL"
value = "http://claude-agent-service.claude-agent.svc.cluster.local:8080"
}
resources {
requests = {
cpu = "50m"
memory = "256Mi"
}
limits = {
memory = "512Mi"
}
}
}
container {
name = "payslip-ingest"
image = local.image
port {
container_port = 8080
}
env_from {
secret_ref {
name = "payslip-ingest-secrets"
}
}
env_from {
secret_ref {
name = "payslip-ingest-db-creds"
}
}
env {
name = "PAPERLESS_URL"
value = "http://paperless-ngx.paperless-ngx.svc.cluster.local"
}
env {
name = "CLAUDE_AGENT_URL"
value = "http://claude-agent-service.claude-agent.svc.cluster.local:8080"
}
readiness_probe {
http_get {
path = "/healthz"
port = 8080
}
initial_delay_seconds = 5
period_seconds = 10
}
liveness_probe {
http_get {
path = "/healthz"
port = 8080
}
initial_delay_seconds = 5
period_seconds = 10
}
resources {
requests = {
cpu = "50m"
memory = "256Mi"
}
limits = {
memory = "512Mi"
}
}
}
}
}
}
lifecycle {
ignore_changes = [spec[0].template[0].spec[0].dns_config] # KYVERNO_LIFECYCLE_V1
}
depends_on = [
kubernetes_manifest.external_secret,
kubernetes_manifest.db_external_secret,
]
}
# ClusterIP-only — webhook is cluster-internal (paperless-ngx -> payslip-ingest).
resource "kubernetes_service" "payslip_ingest" {
metadata {
name = "payslip-ingest"
namespace = kubernetes_namespace.payslip_ingest.metadata[0].name
labels = local.labels
}
spec {
type = "ClusterIP"
selector = local.labels
port {
name = "http"
port = 8080
target_port = 8080
}
}
}
# Plan-time read of the ESO-created K8s Secret for Grafana datasource password.
# First apply: -target=kubernetes_manifest.db_external_secret first so the Secret exists.
data "kubernetes_secret" "payslip_ingest_db_creds" {
metadata {
name = "payslip-ingest-db-creds"
namespace = kubernetes_namespace.payslip_ingest.metadata[0].name
}
depends_on = [kubernetes_manifest.db_external_secret]
}
# Grafana datasource for payslip_ingest PostgreSQL DB.
# Lives in the monitoring namespace so the grafana sidecar (label grafana_datasource=1) picks it up.
resource "kubernetes_config_map" "grafana_payslips_datasource" {
metadata {
name = "grafana-payslips-datasource"
namespace = "monitoring"
labels = {
grafana_datasource = "1"
}
}
data = {
"payslips-datasource.yaml" = yamlencode({
apiVersion = 1
datasources = [{
name = "Payslips"
type = "postgres"
access = "proxy"
url = "${var.postgresql_host}:5432"
database = "payslip_ingest"
user = "payslip_ingest"
uid = "payslips-pg"
jsonData = {
sslmode = "disable"
postgresVersion = 1600
timescaledb = false
}
secureJsonData = {
password = data.kubernetes_secret.payslip_ingest_db_creds.data["DB_PASSWORD"]
}
editable = true
}]
})
}
}

18
stacks/payslip-ingest/terragrunt.hcl Normal file
View file

@ -0,0 +1,18 @@
include "root" {
path = find_in_parent_folders()
}
dependency "platform" {
config_path = "../platform"
skip_outputs = true
}
dependency "vault" {
config_path = "../vault"
skip_outputs = true
}
dependency "external-secrets" {
config_path = "../external-secrets"
skip_outputs = true
}

10
stacks/vault/main.tf
View file

@ -523,7 +523,7 @@ resource "vault_database_secret_backend_connection" "postgresql" {
# "pg-trading", # Commented out 2026-04-06 - trading-bot disabled
"pg-health", "pg-linkwarden",
"pg-affine", "pg-woodpecker", "pg-claude-memory",
"pg-terraform-state"
"pg-terraform-state", "pg-payslip-ingest"
]
postgresql {
@ -661,6 +661,14 @@ resource "vault_database_secret_backend_static_role" "pg_terraform_state" {
rotation_period = 604800
}
resource "vault_database_secret_backend_static_role" "pg_payslip_ingest" {
backend = vault_mount.database.path
db_name = vault_database_secret_backend_connection.postgresql.name
name = "pg-payslip-ingest"
username = "payslip_ingest"
rotation_period = 604800
}
# =============================================================================
# Kubernetes Secrets Engine — Dynamic K8s Credentials
# =============================================================================