[payslip-ingest] Deploy stack + Grafana dashboard + Vault DB role
## Context
New service `payslip-ingest` (code lives in `/home/wizard/code/payslip-ingest/`)
needs in-cluster deployment, its own Postgres DB + rotating user, a Grafana
datasource, a dashboard, and a Claude agent definition for PDF extraction.
Cluster-internal only — webhook fires from Paperless-ngx in a sibling namespace.
No ingress, no TLS cert, no DNS record.
## What
### New stack `stacks/payslip-ingest/`
- `kubernetes_namespace` payslip-ingest, tier=aux.
- ExternalSecret (vault-kv) projects PAPERLESS_API_TOKEN, CLAUDE_AGENT_BEARER_TOKEN,
WEBHOOK_BEARER_TOKEN into `payslip-ingest-secrets`.
- ExternalSecret (vault-database) reads rotating password from
`static-creds/pg-payslip-ingest` and templates `DATABASE_URL` into
`payslip-ingest-db-creds` with `reloader.stakater.com/match=true`.
- Deployment: single replica, Recreate strategy (matches single-worker queue
design), `wait-for postgresql.dbaas:5432` annotation, init container runs
`alembic upgrade head`, main container serves FastAPI on 8080, Kyverno
dns_config lifecycle ignore.
- ClusterIP Service :8080.
- Grafana datasource ConfigMap in `monitoring` ns (label `grafana_datasource=1`,
uid `payslips-pg`) reading password from the db-creds K8s Secret.
### Grafana dashboard `uk-payslip.json` (4 panels)
- Monthly gross/net/tax/NI (timeseries, currencyGBP).
- YTD tax-band progression with threshold lines at £12,570 / £50,270 / £125,140.
- Deductions breakdown (stacked bars).
- Effective rate + take-home % (timeseries, percent).
### Vault DB role `pg-payslip-ingest`
- Added to `allowed_roles` in `vault_database_secret_backend_connection.postgresql`.
- New `vault_database_secret_backend_static_role.pg_payslip_ingest`
(username `payslip_ingest`, 7d rotation).
### DBaaS — DB + role creation
- New `null_resource.pg_payslip_ingest_db` mirrors `pg_terraform_state_db`:
idempotent CREATE ROLE + CREATE DATABASE + GRANT ALL via `kubectl exec` into
`pg-cluster-1`.
### Claude agent `.claude/agents/payslip-extractor.md`
- Haiku-backed agent invoked by `claude-agent-service`.
- Decodes base64 PDF from prompt, tries pdftotext → pypdf fallback, emits a single
JSON object matching the schema to stdout. No network, no file writes outside /tmp,
no markdown fences.
## Trade-offs / decisions
- Own DB per service (convention), NOT a schema in a shared `app` DB as the plan
initially described. The Alembic migration still creates a `payslip_ingest`
schema inside the `payslip_ingest` DB for table organisation.
- Paperless URL uses port 80 (the Service port), not 8000 (the pod target port).
- Grafana datasource uses the primary RW user — separate `_ro` role is aspirational
and not yet a pattern in this repo.
- No ingress — webhook is cluster-internal; external exposure is unnecessary attack
surface.
- No Uptime Kuma monitor yet: the internal-monitor list is a static block in
`stacks/uptime-kuma/`; will add in a follow-up tied to code-z29 (internal monitor
auto-creator).
## Test Plan
### Automated
```
terraform init -backend=false && terraform validate
Success! The configuration is valid.
terraform fmt -check -recursive
(exit 0)
python3 -c "import json; json.load(open('uk-payslip.json'))"
(exit 0)
```
### Manual Verification (post-merge)
Prerequisites:
1. Seed Vault: `vault kv put secret/payslip-ingest webhook_bearer_token=$(openssl rand -hex 32)`.
2. Seed Vault: `vault kv patch secret/paperless-ngx api_token=<paperless token>`.
Apply:
3. `scripts/tg apply vault` → creates pg-payslip-ingest static role.
4. `scripts/tg apply dbaas` → creates payslip_ingest DB + role.
5. `cd stacks/payslip-ingest && ../../scripts/tg apply -target=kubernetes_manifest.db_external_secret`
(first-apply ESO bootstrap).
6. `scripts/tg apply payslip-ingest` (full).
7. `kubectl -n payslip-ingest get pods` → Running 1/1.
8. `kubectl -n payslip-ingest port-forward svc/payslip-ingest 8080:8080 && curl localhost:8080/healthz` → 200.
End-to-end:
9. Configure Paperless workflow (README in code repo has steps).
10. Upload sample payslip tagged `payslip` → row in `payslip_ingest.payslip` within 60s.
11. Grafana → Dashboards → UK Payslip → 4 panels render.
Closes: code-do7
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
81e7c3d6ee
commit
43b4e1d372
6 changed files with 774 additions and 1 deletions
169
.claude/agents/payslip-extractor.md
Normal file
169
.claude/agents/payslip-extractor.md
Normal file
|
|
@ -0,0 +1,169 @@
|
||||||
|
---
|
||||||
|
name: payslip-extractor
|
||||||
|
description: "Extract structured UK payslip fields from a base64-encoded PDF into strict JSON."
|
||||||
|
model: haiku
|
||||||
|
allowedTools:
|
||||||
|
- Bash
|
||||||
|
- Read
|
||||||
|
---
|
||||||
|
|
||||||
|
You are a headless payslip-field extractor. You receive a prompt containing a base64-encoded UK payslip PDF plus a target JSON schema, and you produce exactly one JSON object that matches the schema.
|
||||||
|
|
||||||
|
## Your single job
|
||||||
|
|
||||||
|
Given a prompt that contains:
|
||||||
|
- A line of the form `PDF_BASE64: <base64-blob>`
|
||||||
|
- A JSON schema describing the target fields
|
||||||
|
|
||||||
|
Produce EXACTLY ONE JSON object on stdout matching the schema. No prose. No markdown fences. No preamble. No trailing commentary. The final message content must be a single valid JSON object and nothing else.
|
||||||
|
|
||||||
|
## Processing steps
|
||||||
|
|
||||||
|
### Step 1. Extract and decode the base64 PDF
|
||||||
|
|
||||||
|
The prompt will include a line that starts with `PDF_BASE64:` followed by the base64 blob. Decode it to `/tmp/payslip.pdf`.
|
||||||
|
|
||||||
|
Preferred method (handles whitespace and very long blobs robustly):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 - <<'PY'
|
||||||
|
import base64, re, pathlib, sys, os
|
||||||
|
prompt = os.environ.get("PAYSLIP_PROMPT", "")
|
||||||
|
# If the orchestrator didn't set an env var, fall back to reading the transcript via CWD stdin mechanism.
|
||||||
|
# In practice the agent receives the prompt in its conversation — you extract the PDF_BASE64 value
|
||||||
|
# from the prompt text you were given, strip whitespace, and base64-decode.
|
||||||
|
PY
|
||||||
|
```
|
||||||
|
|
||||||
|
In practice: read the `PDF_BASE64:` value out of the prompt you have been given (you can see the full prompt), then run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -c "
|
||||||
|
import base64, sys
|
||||||
|
data = sys.stdin.read().strip()
|
||||||
|
open('/tmp/payslip.pdf','wb').write(base64.b64decode(data))
|
||||||
|
print('decoded bytes:', len(base64.b64decode(data)))
|
||||||
|
" <<'B64'
|
||||||
|
<paste-the-base64-here>
|
||||||
|
B64
|
||||||
|
```
|
||||||
|
|
||||||
|
Or pipe via shell `base64 -d`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
printf '%s' '<base64>' | base64 -d > /tmp/payslip.pdf
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify the file looks like a PDF:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
head -c 8 /tmp/payslip.pdf | xxd
|
||||||
|
# Expected: 25 50 44 46 2d (i.e. "%PDF-")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2. Extract text from the PDF
|
||||||
|
|
||||||
|
Try tools in this order. Use the first one that works; do not chain all of them.
|
||||||
|
|
||||||
|
1. `pdftotext` from `poppler-utils` (preferred — fastest, most reliable on layout-preserving payslips):
|
||||||
|
```bash
|
||||||
|
pdftotext -layout /tmp/payslip.pdf - 2>/dev/null
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Python `pypdf` fallback:
|
||||||
|
```bash
|
||||||
|
python3 -c "
|
||||||
|
from pypdf import PdfReader
|
||||||
|
r = PdfReader('/tmp/payslip.pdf')
|
||||||
|
for p in r.pages:
|
||||||
|
print(p.extract_text() or '')
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Python `pdfplumber` fallback:
|
||||||
|
```bash
|
||||||
|
python3 -c "
|
||||||
|
import pdfplumber
|
||||||
|
with pdfplumber.open('/tmp/payslip.pdf') as pdf:
|
||||||
|
for page in pdf.pages:
|
||||||
|
print(page.extract_text() or '')
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
4. If none of those are installed, check what IS available:
|
||||||
|
```bash
|
||||||
|
which pdftotext pdf2txt.py mutool
|
||||||
|
python3 -c "import pypdf, pdfplumber, pdfminer" 2>&1
|
||||||
|
```
|
||||||
|
and use whatever you find (e.g. `mutool draw -F txt`).
|
||||||
|
|
||||||
|
If every text-extraction tool fails, emit the failure JSON (see "Failure mode" below).
|
||||||
|
|
||||||
|
### Step 3. Parse the extracted text
|
||||||
|
|
||||||
|
UK payslips are laid out in a few common templates (Sage, Iris, QuickBooks, Xero, in-house ADP/Workday layouts). Common landmarks:
|
||||||
|
|
||||||
|
- "Pay Date" / "Payment Date" / "Date Paid" — the date wages hit the account. Usually at the top or in a header box.
|
||||||
|
- "Tax Period" / "Period" / "Month" — e.g. "Month 1", "Week 12".
|
||||||
|
- Two numeric columns per line: "This Period" (or "Amount", "Current") and "Year to Date" (or "YTD"). **Always take the This Period column**, never YTD.
|
||||||
|
- Payments / Earnings block: "Basic Pay", "Salary", "Bonus", "Overtime", "Commission", "Holiday Pay".
|
||||||
|
- Deductions block: "Income Tax" / "PAYE", "National Insurance" / "NI" / "NIC", "Pension" / "Pension Contribution" / "Salary Sacrifice Pension", "Student Loan" / "SL", optional: "Union Dues", "Charity", "Season Ticket Loan", "Private Medical", etc.
|
||||||
|
- "Gross Pay" / "Total Gross" — sum of payments.
|
||||||
|
- "Net Pay" / "Take Home" / "Amount Payable" — the money actually paid.
|
||||||
|
- "Tax Code" — e.g. "1257L", "BR", "D0", "NT".
|
||||||
|
- "NI Number" / "National Insurance Number" — `AA123456A` format. Never invent one.
|
||||||
|
- "Employer" / "Company" — usually in the letterhead. "Employee" / "Name".
|
||||||
|
- Currency: almost always GBP / "£" for UK payslips. If the PDF is not in GBP or not a UK payslip, still return the numbers as-is but include a best-effort `currency` field.
|
||||||
|
|
||||||
|
### Step 4. Map to the schema and emit JSON
|
||||||
|
|
||||||
|
Rules that apply regardless of the caller's exact schema:
|
||||||
|
|
||||||
|
- **Dates**: `pay_date` MUST be `YYYY-MM-DD`. If the PDF prints `12/03/2026`, interpret as `DD/MM/YYYY` (UK format) → `2026-03-12`. If ambiguous (`01/02/2026`), prefer UK ordering. If impossible to determine a year, use the pay_period year.
|
||||||
|
- **Money fields**: emit as JSON numbers, not strings. Two decimal places are acceptable (`2450.17`). Strip `£`, commas, and trailing spaces. Negative values stay negative.
|
||||||
|
- **Missing numeric fields**: emit `0` (zero), not `null`, not an empty string, not `"N/A"`.
|
||||||
|
- **`other_deductions`**: an object mapping `{ "<label>": <number>, ... }` for any deduction that isn't one of the first-class fields in the schema (tax, NI, pension, student loan). Use the exact label from the payslip (e.g. `"Season Ticket Loan"`, `"Private Medical"`). If there are no other deductions, emit `{}` — NEVER `null` and NEVER omit the key.
|
||||||
|
- **Column discipline**: ALWAYS use the "This Period" column, NEVER the YTD column. If only one column exists, that's the period column.
|
||||||
|
- **Currency default**: `"GBP"` unless the payslip explicitly shows another currency symbol or ISO code.
|
||||||
|
- **No invented data**: If a field genuinely isn't on the payslip, use the documented default (`0` for money, `""` for strings, `{}` for objects). Do NOT make up names, NI numbers, tax codes, or employers.
|
||||||
|
|
||||||
|
Follow the exact field names and types given in the prompt's schema. If the prompt's schema adds fields not listed above, produce them too using the same discipline.
|
||||||
|
|
||||||
|
## Failure mode
|
||||||
|
|
||||||
|
If the PDF cannot be read at all — unreadable base64, not a PDF, encrypted PDF with no text layer, no text-extraction tool available, or clearly not a UK payslip — emit a single JSON object:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"error": "<short human reason>"}
|
||||||
|
```
|
||||||
|
|
||||||
|
Examples of acceptable error reasons:
|
||||||
|
- `"base64 did not decode to a valid PDF"`
|
||||||
|
- `"pdf has no extractable text layer (image-only scan)"`
|
||||||
|
- `"no pdf text extraction tool available (pdftotext/pypdf/pdfplumber all missing)"`
|
||||||
|
- `"document does not appear to be a UK payslip"`
|
||||||
|
- `"pay_date not found on document"`
|
||||||
|
|
||||||
|
The caller treats the `error` key as a non-retriable parse failure. Do not include any other keys when emitting an error object.
|
||||||
|
|
||||||
|
## Hard constraints — things you MUST NOT do
|
||||||
|
|
||||||
|
1. **No network calls.** Do not curl, wget, dig, or otherwise talk to the network. Everything you need is in the prompt.
|
||||||
|
2. **No modifications to `/workspace/infra/**`.** Do not edit, write, or commit any file under the infra repo. The only file you may create is the scratch PDF at `/tmp/payslip.pdf` (and intermediate text dumps under `/tmp/`).
|
||||||
|
3. **No git operations.** No `git add`, `git commit`, `git push`, nothing.
|
||||||
|
4. **No kubectl, no terraform, no vault.** You are not an infra agent — you are a narrow extractor.
|
||||||
|
5. **No markdown in output.** No ` ```json ` fences, no preamble like "Here's the extraction:", no trailing notes. The ENTIRE final assistant message is exactly one JSON object.
|
||||||
|
6. **No verbose logging in the final message.** It is fine to run bash commands and see their output during processing, but your final assistant message is JSON and nothing else.
|
||||||
|
7. **No hallucinated fields.** If the payslip does not show a pension line, do not invent one. Use the documented default instead.
|
||||||
|
|
||||||
|
## Output discipline — summary
|
||||||
|
|
||||||
|
- Exactly one JSON object, UTF-8, no BOM.
|
||||||
|
- Keys match the schema the caller gave you.
|
||||||
|
- Numeric fields are JSON numbers, not strings.
|
||||||
|
- `pay_date` is `YYYY-MM-DD`.
|
||||||
|
- `other_deductions` is always present and is an object (possibly `{}`).
|
||||||
|
- Missing money → `0`, missing string → `""`, missing object → `{}`.
|
||||||
|
- On unrecoverable failure, one JSON object with a single `error` key.
|
||||||
|
|
||||||
|
That's the whole job. Decode, extract, parse, emit JSON. Be boring and exact.
|
||||||
|
|
@ -1440,6 +1440,30 @@ resource "null_resource" "pg_terraform_state_db" {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Create payslip_ingest database for the payslip-ingest webhook service.
|
||||||
|
# Role password is managed by Vault Database Secrets Engine (static role `pg-payslip-ingest`, 7d rotation).
|
||||||
|
resource "null_resource" "pg_payslip_ingest_db" {
|
||||||
|
depends_on = [null_resource.pg_cluster]
|
||||||
|
|
||||||
|
triggers = {
|
||||||
|
db_name = "payslip_ingest"
|
||||||
|
username = "payslip_ingest"
|
||||||
|
}
|
||||||
|
|
||||||
|
provisioner "local-exec" {
|
||||||
|
command = <<-EOT
|
||||||
|
kubectl --kubeconfig ${var.kube_config_path} exec -n dbaas pg-cluster-1 -c postgres -- \
|
||||||
|
bash -c '
|
||||||
|
psql -U postgres -tc "SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = '"'"'payslip_ingest'"'"'" | grep -q 1 || \
|
||||||
|
psql -U postgres -c "CREATE ROLE payslip_ingest WITH LOGIN PASSWORD '"'"'changeme-vault-will-rotate'"'"'"
|
||||||
|
psql -U postgres -tc "SELECT 1 FROM pg_catalog.pg_database WHERE datname = '"'"'payslip_ingest'"'"'" | grep -q 1 || \
|
||||||
|
psql -U postgres -c "CREATE DATABASE payslip_ingest OWNER payslip_ingest"
|
||||||
|
psql -U postgres -c "GRANT ALL PRIVILEGES ON DATABASE payslip_ingest TO payslip_ingest"
|
||||||
|
'
|
||||||
|
EOT
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
# Old PostgreSQL deployment — kept commented for rollback reference
|
# Old PostgreSQL deployment — kept commented for rollback reference
|
||||||
# resource "kubernetes_deployment" "postgres" {
|
# resource "kubernetes_deployment" "postgres" {
|
||||||
# metadata {
|
# metadata {
|
||||||
|
|
|
||||||
224
stacks/monitoring/modules/monitoring/dashboards/uk-payslip.json
Normal file
224
stacks/monitoring/modules/monitoring/dashboards/uk-payslip.json
Normal file
|
|
@ -0,0 +1,224 @@
|
||||||
|
{
|
||||||
|
"annotations": {
|
||||||
|
"list": [
|
||||||
|
{
|
||||||
|
"builtIn": 1,
|
||||||
|
"datasource": { "type": "datasource", "uid": "grafana" },
|
||||||
|
"enable": true,
|
||||||
|
"hide": true,
|
||||||
|
"iconColor": "rgba(0, 211, 255, 1)",
|
||||||
|
"name": "Annotations & Alerts",
|
||||||
|
"type": "dashboard"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"description": "UK payslip breakdown — gross/net/tax/NI trends, YTD progression against income tax bands, deductions split, and effective rate.",
|
||||||
|
"editable": true,
|
||||||
|
"fiscalYearStartMonth": 0,
|
||||||
|
"graphTooltip": 1,
|
||||||
|
"id": null,
|
||||||
|
"links": [],
|
||||||
|
"panels": [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"title": "Monthly gross / net / tax / NI",
|
||||||
|
"type": "timeseries",
|
||||||
|
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||||
|
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 0 },
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": { "mode": "palette-classic" },
|
||||||
|
"unit": "currencyGBP",
|
||||||
|
"custom": {
|
||||||
|
"axisBorderShow": false,
|
||||||
|
"axisCenteredZero": false,
|
||||||
|
"axisColorMode": "text",
|
||||||
|
"axisLabel": "",
|
||||||
|
"axisPlacement": "auto",
|
||||||
|
"barAlignment": 0,
|
||||||
|
"drawStyle": "line",
|
||||||
|
"fillOpacity": 10,
|
||||||
|
"gradientMode": "none",
|
||||||
|
"hideFrom": { "legend": false, "tooltip": false, "viz": false },
|
||||||
|
"lineWidth": 2,
|
||||||
|
"pointSize": 5,
|
||||||
|
"scaleDistribution": { "type": "linear" },
|
||||||
|
"showPoints": "auto",
|
||||||
|
"spanNulls": false,
|
||||||
|
"stacking": { "group": "A", "mode": "none" },
|
||||||
|
"thresholdsStyle": { "mode": "off" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"options": {
|
||||||
|
"legend": { "calcs": ["last", "mean"], "displayMode": "table", "placement": "bottom" },
|
||||||
|
"tooltip": { "mode": "multi", "sort": "desc" }
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||||
|
"rawSql": "SELECT pay_date AS \"time\", gross_pay, net_pay, income_tax, national_insurance FROM payslip_ingest.payslip WHERE $__timeFilter(pay_date) ORDER BY pay_date",
|
||||||
|
"format": "time_series",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 2,
|
||||||
|
"title": "YTD gross (this tax year) with UK band thresholds",
|
||||||
|
"type": "timeseries",
|
||||||
|
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||||
|
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 0 },
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": { "mode": "palette-classic" },
|
||||||
|
"unit": "currencyGBP",
|
||||||
|
"custom": {
|
||||||
|
"axisBorderShow": false,
|
||||||
|
"axisCenteredZero": false,
|
||||||
|
"axisColorMode": "text",
|
||||||
|
"axisLabel": "YTD gross",
|
||||||
|
"axisPlacement": "auto",
|
||||||
|
"barAlignment": 0,
|
||||||
|
"drawStyle": "line",
|
||||||
|
"fillOpacity": 15,
|
||||||
|
"gradientMode": "none",
|
||||||
|
"hideFrom": { "legend": false, "tooltip": false, "viz": false },
|
||||||
|
"lineWidth": 2,
|
||||||
|
"pointSize": 5,
|
||||||
|
"scaleDistribution": { "type": "linear" },
|
||||||
|
"showPoints": "auto",
|
||||||
|
"spanNulls": false,
|
||||||
|
"stacking": { "group": "A", "mode": "none" },
|
||||||
|
"thresholdsStyle": { "mode": "line" }
|
||||||
|
},
|
||||||
|
"thresholds": {
|
||||||
|
"mode": "absolute",
|
||||||
|
"steps": [
|
||||||
|
{ "color": "green", "value": null },
|
||||||
|
{ "color": "yellow", "value": 12570 },
|
||||||
|
{ "color": "orange", "value": 50270 },
|
||||||
|
{ "color": "red", "value": 125140 }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"options": {
|
||||||
|
"legend": { "calcs": ["last", "max"], "displayMode": "table", "placement": "bottom" },
|
||||||
|
"tooltip": { "mode": "multi", "sort": "desc" }
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||||
|
"rawSql": "SELECT pay_date AS \"time\", SUM(gross_pay) OVER (PARTITION BY tax_year ORDER BY pay_date) AS ytd_gross FROM payslip_ingest.payslip WHERE $__timeFilter(pay_date) ORDER BY pay_date",
|
||||||
|
"format": "time_series",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 3,
|
||||||
|
"title": "Deductions breakdown per payslip",
|
||||||
|
"type": "timeseries",
|
||||||
|
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||||
|
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 9 },
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": { "mode": "palette-classic" },
|
||||||
|
"unit": "currencyGBP",
|
||||||
|
"custom": {
|
||||||
|
"axisBorderShow": false,
|
||||||
|
"axisCenteredZero": false,
|
||||||
|
"axisColorMode": "text",
|
||||||
|
"axisLabel": "",
|
||||||
|
"axisPlacement": "auto",
|
||||||
|
"barAlignment": 0,
|
||||||
|
"drawStyle": "bars",
|
||||||
|
"fillOpacity": 80,
|
||||||
|
"gradientMode": "none",
|
||||||
|
"hideFrom": { "legend": false, "tooltip": false, "viz": false },
|
||||||
|
"lineWidth": 1,
|
||||||
|
"pointSize": 5,
|
||||||
|
"scaleDistribution": { "type": "linear" },
|
||||||
|
"showPoints": "never",
|
||||||
|
"spanNulls": false,
|
||||||
|
"stacking": { "group": "A", "mode": "normal" },
|
||||||
|
"thresholdsStyle": { "mode": "off" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"options": {
|
||||||
|
"legend": { "calcs": ["sum", "mean"], "displayMode": "table", "placement": "bottom" },
|
||||||
|
"tooltip": { "mode": "multi", "sort": "desc" }
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||||
|
"rawSql": "SELECT pay_date AS \"time\", income_tax, national_insurance, pension_employee, student_loan FROM payslip_ingest.payslip WHERE $__timeFilter(pay_date) ORDER BY pay_date",
|
||||||
|
"format": "time_series",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 4,
|
||||||
|
"title": "Latest effective rate & take-home %",
|
||||||
|
"type": "timeseries",
|
||||||
|
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||||
|
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 9 },
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": { "mode": "palette-classic" },
|
||||||
|
"unit": "percent",
|
||||||
|
"min": 0,
|
||||||
|
"max": 100,
|
||||||
|
"custom": {
|
||||||
|
"axisBorderShow": false,
|
||||||
|
"axisCenteredZero": false,
|
||||||
|
"axisColorMode": "text",
|
||||||
|
"axisLabel": "",
|
||||||
|
"axisPlacement": "auto",
|
||||||
|
"barAlignment": 0,
|
||||||
|
"drawStyle": "line",
|
||||||
|
"fillOpacity": 10,
|
||||||
|
"gradientMode": "none",
|
||||||
|
"hideFrom": { "legend": false, "tooltip": false, "viz": false },
|
||||||
|
"lineWidth": 2,
|
||||||
|
"pointSize": 5,
|
||||||
|
"scaleDistribution": { "type": "linear" },
|
||||||
|
"showPoints": "auto",
|
||||||
|
"spanNulls": false,
|
||||||
|
"stacking": { "group": "A", "mode": "none" },
|
||||||
|
"thresholdsStyle": { "mode": "off" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"options": {
|
||||||
|
"legend": { "calcs": ["last", "mean"], "displayMode": "table", "placement": "bottom" },
|
||||||
|
"tooltip": { "mode": "multi", "sort": "desc" }
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||||
|
"rawSql": "SELECT pay_date AS \"time\", ROUND(((income_tax + national_insurance)::numeric / NULLIF(gross_pay, 0)) * 100, 2) AS \"effective_rate_pct\", ROUND((net_pay::numeric / NULLIF(gross_pay, 0)) * 100, 2) AS \"take_home_pct\" FROM payslip_ingest.payslip WHERE $__timeFilter(pay_date) ORDER BY pay_date",
|
||||||
|
"format": "time_series",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"refresh": "5m",
|
||||||
|
"schemaVersion": 39,
|
||||||
|
"tags": ["finance", "personal", "uk-tax"],
|
||||||
|
"templating": { "list": [] },
|
||||||
|
"time": { "from": "now-2y", "to": "now" },
|
||||||
|
"timepicker": {},
|
||||||
|
"timezone": "browser",
|
||||||
|
"title": "UK Payslip",
|
||||||
|
"uid": "uk-payslip",
|
||||||
|
"version": 1
|
||||||
|
}
|
||||||
330
stacks/payslip-ingest/main.tf
Normal file
330
stacks/payslip-ingest/main.tf
Normal file
|
|
@ -0,0 +1,330 @@
|
||||||
|
variable "image_tag" {
|
||||||
|
type = string
|
||||||
|
default = "latest"
|
||||||
|
description = "payslip-ingest image tag. Use 8-char git SHA in CI; :latest only for local trials."
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "postgresql_host" { type = string }
|
||||||
|
|
||||||
|
locals {
|
||||||
|
namespace = "payslip-ingest"
|
||||||
|
image = "registry.viktorbarzin.me/payslip-ingest:${var.image_tag}"
|
||||||
|
labels = {
|
||||||
|
app = "payslip-ingest"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "kubernetes_namespace" "payslip_ingest" {
|
||||||
|
metadata {
|
||||||
|
name = local.namespace
|
||||||
|
labels = {
|
||||||
|
tier = local.tiers.aux
|
||||||
|
"istio-injection" = "disabled"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# App secrets sourced from multiple Vault KV keys.
|
||||||
|
# Seed these manually in Vault before applying:
|
||||||
|
# secret/paperless-ngx -> property `api_token`
|
||||||
|
# secret/claude-agent-service -> property `api_bearer_token`
|
||||||
|
# secret/payslip-ingest -> property `webhook_bearer_token`
|
||||||
|
resource "kubernetes_manifest" "external_secret" {
|
||||||
|
manifest = {
|
||||||
|
apiVersion = "external-secrets.io/v1beta1"
|
||||||
|
kind = "ExternalSecret"
|
||||||
|
metadata = {
|
||||||
|
name = "payslip-ingest-secrets"
|
||||||
|
namespace = local.namespace
|
||||||
|
}
|
||||||
|
spec = {
|
||||||
|
refreshInterval = "15m"
|
||||||
|
secretStoreRef = {
|
||||||
|
name = "vault-kv"
|
||||||
|
kind = "ClusterSecretStore"
|
||||||
|
}
|
||||||
|
target = {
|
||||||
|
name = "payslip-ingest-secrets"
|
||||||
|
template = {
|
||||||
|
metadata = {
|
||||||
|
annotations = {
|
||||||
|
"reloader.stakater.com/match" = "true"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
data = [
|
||||||
|
{
|
||||||
|
secretKey = "PAPERLESS_API_TOKEN"
|
||||||
|
remoteRef = {
|
||||||
|
key = "paperless-ngx"
|
||||||
|
property = "api_token"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
secretKey = "CLAUDE_AGENT_BEARER_TOKEN"
|
||||||
|
remoteRef = {
|
||||||
|
key = "claude-agent-service"
|
||||||
|
property = "api_bearer_token"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
secretKey = "WEBHOOK_BEARER_TOKEN"
|
||||||
|
remoteRef = {
|
||||||
|
key = "payslip-ingest"
|
||||||
|
property = "webhook_bearer_token"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
depends_on = [kubernetes_namespace.payslip_ingest]
|
||||||
|
}
|
||||||
|
|
||||||
|
# DB credentials from Vault database engine (rotated every 7 days).
|
||||||
|
# Template builds the asyncpg DSN consumed by the FastAPI app as DB_CONNECTION_STRING.
|
||||||
|
resource "kubernetes_manifest" "db_external_secret" {
|
||||||
|
manifest = {
|
||||||
|
apiVersion = "external-secrets.io/v1beta1"
|
||||||
|
kind = "ExternalSecret"
|
||||||
|
metadata = {
|
||||||
|
name = "payslip-ingest-db-creds"
|
||||||
|
namespace = local.namespace
|
||||||
|
}
|
||||||
|
spec = {
|
||||||
|
refreshInterval = "15m"
|
||||||
|
secretStoreRef = {
|
||||||
|
name = "vault-database"
|
||||||
|
kind = "ClusterSecretStore"
|
||||||
|
}
|
||||||
|
target = {
|
||||||
|
name = "payslip-ingest-db-creds"
|
||||||
|
template = {
|
||||||
|
metadata = {
|
||||||
|
annotations = {
|
||||||
|
"reloader.stakater.com/match" = "true"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
data = {
|
||||||
|
DB_CONNECTION_STRING = "postgresql+asyncpg://payslip_ingest:{{ .password }}@${var.postgresql_host}:5432/payslip_ingest"
|
||||||
|
DB_PASSWORD = "{{ .password }}"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
data = [{
|
||||||
|
secretKey = "password"
|
||||||
|
remoteRef = {
|
||||||
|
key = "static-creds/pg-payslip-ingest"
|
||||||
|
property = "password"
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
depends_on = [kubernetes_namespace.payslip_ingest]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "kubernetes_deployment" "payslip_ingest" {
|
||||||
|
metadata {
|
||||||
|
name = "payslip-ingest"
|
||||||
|
namespace = kubernetes_namespace.payslip_ingest.metadata[0].name
|
||||||
|
labels = merge(local.labels, {
|
||||||
|
tier = local.tiers.aux
|
||||||
|
})
|
||||||
|
annotations = {
|
||||||
|
"reloader.stakater.com/search" = "true"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
spec {
|
||||||
|
replicas = 1
|
||||||
|
strategy {
|
||||||
|
type = "Recreate"
|
||||||
|
}
|
||||||
|
|
||||||
|
selector {
|
||||||
|
match_labels = local.labels
|
||||||
|
}
|
||||||
|
|
||||||
|
template {
|
||||||
|
metadata {
|
||||||
|
labels = local.labels
|
||||||
|
annotations = {
|
||||||
|
"dependency.kyverno.io/wait-for" = "postgresql.dbaas:5432"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
spec {
|
||||||
|
image_pull_secrets {
|
||||||
|
name = "registry-credentials"
|
||||||
|
}
|
||||||
|
|
||||||
|
init_container {
|
||||||
|
name = "alembic-migrate"
|
||||||
|
image = local.image
|
||||||
|
command = ["python", "-m", "payslip_ingest", "migrate"]
|
||||||
|
|
||||||
|
env_from {
|
||||||
|
secret_ref {
|
||||||
|
name = "payslip-ingest-secrets"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
env_from {
|
||||||
|
secret_ref {
|
||||||
|
name = "payslip-ingest-db-creds"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
env {
|
||||||
|
name = "PAPERLESS_URL"
|
||||||
|
value = "http://paperless-ngx.paperless-ngx.svc.cluster.local"
|
||||||
|
}
|
||||||
|
env {
|
||||||
|
name = "CLAUDE_AGENT_URL"
|
||||||
|
value = "http://claude-agent-service.claude-agent.svc.cluster.local:8080"
|
||||||
|
}
|
||||||
|
|
||||||
|
resources {
|
||||||
|
requests = {
|
||||||
|
cpu = "50m"
|
||||||
|
memory = "256Mi"
|
||||||
|
}
|
||||||
|
limits = {
|
||||||
|
memory = "512Mi"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
container {
|
||||||
|
name = "payslip-ingest"
|
||||||
|
image = local.image
|
||||||
|
|
||||||
|
port {
|
||||||
|
container_port = 8080
|
||||||
|
}
|
||||||
|
|
||||||
|
env_from {
|
||||||
|
secret_ref {
|
||||||
|
name = "payslip-ingest-secrets"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
env_from {
|
||||||
|
secret_ref {
|
||||||
|
name = "payslip-ingest-db-creds"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
env {
|
||||||
|
name = "PAPERLESS_URL"
|
||||||
|
value = "http://paperless-ngx.paperless-ngx.svc.cluster.local"
|
||||||
|
}
|
||||||
|
env {
|
||||||
|
name = "CLAUDE_AGENT_URL"
|
||||||
|
value = "http://claude-agent-service.claude-agent.svc.cluster.local:8080"
|
||||||
|
}
|
||||||
|
|
||||||
|
readiness_probe {
|
||||||
|
http_get {
|
||||||
|
path = "/healthz"
|
||||||
|
port = 8080
|
||||||
|
}
|
||||||
|
initial_delay_seconds = 5
|
||||||
|
period_seconds = 10
|
||||||
|
}
|
||||||
|
|
||||||
|
liveness_probe {
|
||||||
|
http_get {
|
||||||
|
path = "/healthz"
|
||||||
|
port = 8080
|
||||||
|
}
|
||||||
|
initial_delay_seconds = 5
|
||||||
|
period_seconds = 10
|
||||||
|
}
|
||||||
|
|
||||||
|
resources {
|
||||||
|
requests = {
|
||||||
|
cpu = "50m"
|
||||||
|
memory = "256Mi"
|
||||||
|
}
|
||||||
|
limits = {
|
||||||
|
memory = "512Mi"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
lifecycle {
|
||||||
|
ignore_changes = [spec[0].template[0].spec[0].dns_config] # KYVERNO_LIFECYCLE_V1
|
||||||
|
}
|
||||||
|
|
||||||
|
depends_on = [
|
||||||
|
kubernetes_manifest.external_secret,
|
||||||
|
kubernetes_manifest.db_external_secret,
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
# ClusterIP-only — webhook is cluster-internal (paperless-ngx -> payslip-ingest).
|
||||||
|
resource "kubernetes_service" "payslip_ingest" {
|
||||||
|
metadata {
|
||||||
|
name = "payslip-ingest"
|
||||||
|
namespace = kubernetes_namespace.payslip_ingest.metadata[0].name
|
||||||
|
labels = local.labels
|
||||||
|
}
|
||||||
|
|
||||||
|
spec {
|
||||||
|
type = "ClusterIP"
|
||||||
|
selector = local.labels
|
||||||
|
|
||||||
|
port {
|
||||||
|
name = "http"
|
||||||
|
port = 8080
|
||||||
|
target_port = 8080
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Plan-time read of the ESO-created K8s Secret for Grafana datasource password.
|
||||||
|
# First apply: -target=kubernetes_manifest.db_external_secret first so the Secret exists.
|
||||||
|
data "kubernetes_secret" "payslip_ingest_db_creds" {
|
||||||
|
metadata {
|
||||||
|
name = "payslip-ingest-db-creds"
|
||||||
|
namespace = kubernetes_namespace.payslip_ingest.metadata[0].name
|
||||||
|
}
|
||||||
|
depends_on = [kubernetes_manifest.db_external_secret]
|
||||||
|
}
|
||||||
|
|
||||||
|
# Grafana datasource for payslip_ingest PostgreSQL DB.
|
||||||
|
# Lives in the monitoring namespace so the grafana sidecar (label grafana_datasource=1) picks it up.
|
||||||
|
resource "kubernetes_config_map" "grafana_payslips_datasource" {
|
||||||
|
metadata {
|
||||||
|
name = "grafana-payslips-datasource"
|
||||||
|
namespace = "monitoring"
|
||||||
|
labels = {
|
||||||
|
grafana_datasource = "1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
data = {
|
||||||
|
"payslips-datasource.yaml" = yamlencode({
|
||||||
|
apiVersion = 1
|
||||||
|
datasources = [{
|
||||||
|
name = "Payslips"
|
||||||
|
type = "postgres"
|
||||||
|
access = "proxy"
|
||||||
|
url = "${var.postgresql_host}:5432"
|
||||||
|
database = "payslip_ingest"
|
||||||
|
user = "payslip_ingest"
|
||||||
|
uid = "payslips-pg"
|
||||||
|
jsonData = {
|
||||||
|
sslmode = "disable"
|
||||||
|
postgresVersion = 1600
|
||||||
|
timescaledb = false
|
||||||
|
}
|
||||||
|
secureJsonData = {
|
||||||
|
password = data.kubernetes_secret.payslip_ingest_db_creds.data["DB_PASSWORD"]
|
||||||
|
}
|
||||||
|
editable = true
|
||||||
|
}]
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
18
stacks/payslip-ingest/terragrunt.hcl
Normal file
18
stacks/payslip-ingest/terragrunt.hcl
Normal file
|
|
@ -0,0 +1,18 @@
|
||||||
|
include "root" {
|
||||||
|
path = find_in_parent_folders()
|
||||||
|
}
|
||||||
|
|
||||||
|
dependency "platform" {
|
||||||
|
config_path = "../platform"
|
||||||
|
skip_outputs = true
|
||||||
|
}
|
||||||
|
|
||||||
|
dependency "vault" {
|
||||||
|
config_path = "../vault"
|
||||||
|
skip_outputs = true
|
||||||
|
}
|
||||||
|
|
||||||
|
dependency "external-secrets" {
|
||||||
|
config_path = "../external-secrets"
|
||||||
|
skip_outputs = true
|
||||||
|
}
|
||||||
|
|
@ -523,7 +523,7 @@ resource "vault_database_secret_backend_connection" "postgresql" {
|
||||||
# "pg-trading", # Commented out 2026-04-06 - trading-bot disabled
|
# "pg-trading", # Commented out 2026-04-06 - trading-bot disabled
|
||||||
"pg-health", "pg-linkwarden",
|
"pg-health", "pg-linkwarden",
|
||||||
"pg-affine", "pg-woodpecker", "pg-claude-memory",
|
"pg-affine", "pg-woodpecker", "pg-claude-memory",
|
||||||
"pg-terraform-state"
|
"pg-terraform-state", "pg-payslip-ingest"
|
||||||
]
|
]
|
||||||
|
|
||||||
postgresql {
|
postgresql {
|
||||||
|
|
@ -661,6 +661,14 @@ resource "vault_database_secret_backend_static_role" "pg_terraform_state" {
|
||||||
rotation_period = 604800
|
rotation_period = 604800
|
||||||
}
|
}
|
||||||
|
|
||||||
|
resource "vault_database_secret_backend_static_role" "pg_payslip_ingest" {
|
||||||
|
backend = vault_mount.database.path
|
||||||
|
db_name = vault_database_secret_backend_connection.postgresql.name
|
||||||
|
name = "pg-payslip-ingest"
|
||||||
|
username = "payslip_ingest"
|
||||||
|
rotation_period = 604800
|
||||||
|
}
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Kubernetes Secrets Engine — Dynamic K8s Credentials
|
# Kubernetes Secrets Engine — Dynamic K8s Credentials
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue