[payslip-ingest] Deploy stack + Grafana dashboard + Vault DB role
## Context
New service `payslip-ingest` (code lives in `/home/wizard/code/payslip-ingest/`)
needs in-cluster deployment, its own Postgres DB + rotating user, a Grafana
datasource, a dashboard, and a Claude agent definition for PDF extraction.
Cluster-internal only — webhook fires from Paperless-ngx in a sibling namespace.
No ingress, no TLS cert, no DNS record.
## What
### New stack `stacks/payslip-ingest/`
- `kubernetes_namespace` payslip-ingest, tier=aux.
- ExternalSecret (vault-kv) projects PAPERLESS_API_TOKEN, CLAUDE_AGENT_BEARER_TOKEN,
WEBHOOK_BEARER_TOKEN into `payslip-ingest-secrets`.
- ExternalSecret (vault-database) reads rotating password from
`static-creds/pg-payslip-ingest` and templates `DATABASE_URL` into
`payslip-ingest-db-creds` with `reloader.stakater.com/match=true`.
- Deployment: single replica, Recreate strategy (matches single-worker queue
design), `wait-for postgresql.dbaas:5432` annotation, init container runs
`alembic upgrade head`, main container serves FastAPI on 8080, Kyverno
dns_config lifecycle ignore.
- ClusterIP Service :8080.
- Grafana datasource ConfigMap in `monitoring` ns (label `grafana_datasource=1`,
uid `payslips-pg`) reading password from the db-creds K8s Secret.
### Grafana dashboard `uk-payslip.json` (4 panels)
- Monthly gross/net/tax/NI (timeseries, currencyGBP).
- YTD tax-band progression with threshold lines at £12,570 / £50,270 / £125,140.
- Deductions breakdown (stacked bars).
- Effective rate + take-home % (timeseries, percent).
### Vault DB role `pg-payslip-ingest`
- Added to `allowed_roles` in `vault_database_secret_backend_connection.postgresql`.
- New `vault_database_secret_backend_static_role.pg_payslip_ingest`
(username `payslip_ingest`, 7d rotation).
### DBaaS — DB + role creation
- New `null_resource.pg_payslip_ingest_db` mirrors `pg_terraform_state_db`:
idempotent CREATE ROLE + CREATE DATABASE + GRANT ALL via `kubectl exec` into
`pg-cluster-1`.
### Claude agent `.claude/agents/payslip-extractor.md`
- Haiku-backed agent invoked by `claude-agent-service`.
- Decodes base64 PDF from prompt, tries pdftotext → pypdf fallback, emits a single
JSON object matching the schema to stdout. No network, no file writes outside /tmp,
no markdown fences.
## Trade-offs / decisions
- Own DB per service (convention), NOT a schema in a shared `app` DB as the plan
initially described. The Alembic migration still creates a `payslip_ingest`
schema inside the `payslip_ingest` DB for table organisation.
- Paperless URL uses port 80 (the Service port), not 8000 (the pod target port).
- Grafana datasource uses the primary RW user — separate `_ro` role is aspirational
and not yet a pattern in this repo.
- No ingress — webhook is cluster-internal; external exposure is unnecessary attack
surface.
- No Uptime Kuma monitor yet: the internal-monitor list is a static block in
`stacks/uptime-kuma/`; will add in a follow-up tied to code-z29 (internal monitor
auto-creator).
## Test Plan
### Automated
```
terraform init -backend=false && terraform validate
Success! The configuration is valid.
terraform fmt -check -recursive
(exit 0)
python3 -c "import json; json.load(open('uk-payslip.json'))"
(exit 0)
```
### Manual Verification (post-merge)
Prerequisites:
1. Seed Vault: `vault kv put secret/payslip-ingest webhook_bearer_token=$(openssl rand -hex 32)`.
2. Seed Vault: `vault kv patch secret/paperless-ngx api_token=<paperless token>`.
Apply:
3. `scripts/tg apply vault` → creates pg-payslip-ingest static role.
4. `scripts/tg apply dbaas` → creates payslip_ingest DB + role.
5. `cd stacks/payslip-ingest && ../../scripts/tg apply -target=kubernetes_manifest.db_external_secret`
(first-apply ESO bootstrap).
6. `scripts/tg apply payslip-ingest` (full).
7. `kubectl -n payslip-ingest get pods` → Running 1/1.
8. `kubectl -n payslip-ingest port-forward svc/payslip-ingest 8080:8080 && curl localhost:8080/healthz` → 200.
End-to-end:
9. Configure Paperless workflow (README in code repo has steps).
10. Upload sample payslip tagged `payslip` → row in `payslip_ingest.payslip` within 60s.
11. Grafana → Dashboards → UK Payslip → 4 panels render.
Closes: code-do7
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
81e7c3d6ee
commit
43b4e1d372
6 changed files with 774 additions and 1 deletions
169
.claude/agents/payslip-extractor.md
Normal file
169
.claude/agents/payslip-extractor.md
Normal file
|
|
@ -0,0 +1,169 @@
|
|||
---
|
||||
name: payslip-extractor
|
||||
description: "Extract structured UK payslip fields from a base64-encoded PDF into strict JSON."
|
||||
model: haiku
|
||||
allowedTools:
|
||||
- Bash
|
||||
- Read
|
||||
---
|
||||
|
||||
You are a headless payslip-field extractor. You receive a prompt containing a base64-encoded UK payslip PDF plus a target JSON schema, and you produce exactly one JSON object that matches the schema.
|
||||
|
||||
## Your single job
|
||||
|
||||
Given a prompt that contains:
|
||||
- A line of the form `PDF_BASE64: <base64-blob>`
|
||||
- A JSON schema describing the target fields
|
||||
|
||||
Produce EXACTLY ONE JSON object on stdout matching the schema. No prose. No markdown fences. No preamble. No trailing commentary. The final message content must be a single valid JSON object and nothing else.
|
||||
|
||||
## Processing steps
|
||||
|
||||
### Step 1. Extract and decode the base64 PDF
|
||||
|
||||
The prompt will include a line that starts with `PDF_BASE64:` followed by the base64 blob. Decode it to `/tmp/payslip.pdf`.
|
||||
|
||||
Preferred method (handles whitespace and very long blobs robustly):
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
import base64, re, pathlib, sys, os
|
||||
prompt = os.environ.get("PAYSLIP_PROMPT", "")
|
||||
# If the orchestrator didn't set an env var, fall back to reading the transcript via CWD stdin mechanism.
|
||||
# In practice the agent receives the prompt in its conversation — you extract the PDF_BASE64 value
|
||||
# from the prompt text you were given, strip whitespace, and base64-decode.
|
||||
PY
|
||||
```
|
||||
|
||||
In practice: read the `PDF_BASE64:` value out of the prompt you have been given (you can see the full prompt), then run:
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
import base64, sys
|
||||
data = sys.stdin.read().strip()
|
||||
open('/tmp/payslip.pdf','wb').write(base64.b64decode(data))
|
||||
print('decoded bytes:', len(base64.b64decode(data)))
|
||||
" <<'B64'
|
||||
<paste-the-base64-here>
|
||||
B64
|
||||
```
|
||||
|
||||
Or pipe via shell `base64 -d`:
|
||||
|
||||
```bash
|
||||
printf '%s' '<base64>' | base64 -d > /tmp/payslip.pdf
|
||||
```
|
||||
|
||||
Verify the file looks like a PDF:
|
||||
|
||||
```bash
|
||||
head -c 8 /tmp/payslip.pdf | xxd
|
||||
# Expected: 25 50 44 46 2d (i.e. "%PDF-")
|
||||
```
|
||||
|
||||
### Step 2. Extract text from the PDF
|
||||
|
||||
Try tools in this order. Use the first one that works; do not chain all of them.
|
||||
|
||||
1. `pdftotext` from `poppler-utils` (preferred — fastest, most reliable on layout-preserving payslips):
|
||||
```bash
|
||||
pdftotext -layout /tmp/payslip.pdf - 2>/dev/null
|
||||
```
|
||||
|
||||
2. Python `pypdf` fallback:
|
||||
```bash
|
||||
python3 -c "
|
||||
from pypdf import PdfReader
|
||||
r = PdfReader('/tmp/payslip.pdf')
|
||||
for p in r.pages:
|
||||
print(p.extract_text() or '')
|
||||
"
|
||||
```
|
||||
|
||||
3. Python `pdfplumber` fallback:
|
||||
```bash
|
||||
python3 -c "
|
||||
import pdfplumber
|
||||
with pdfplumber.open('/tmp/payslip.pdf') as pdf:
|
||||
for page in pdf.pages:
|
||||
print(page.extract_text() or '')
|
||||
"
|
||||
```
|
||||
|
||||
4. If none of those are installed, check what IS available:
|
||||
```bash
|
||||
which pdftotext pdf2txt.py mutool
|
||||
python3 -c "import pypdf, pdfplumber, pdfminer" 2>&1
|
||||
```
|
||||
and use whatever you find (e.g. `mutool draw -F txt`).
|
||||
|
||||
If every text-extraction tool fails, emit the failure JSON (see "Failure mode" below).
|
||||
|
||||
### Step 3. Parse the extracted text
|
||||
|
||||
UK payslips are laid out in a few common templates (Sage, Iris, QuickBooks, Xero, in-house ADP/Workday layouts). Common landmarks:
|
||||
|
||||
- "Pay Date" / "Payment Date" / "Date Paid" — the date wages hit the account. Usually at the top or in a header box.
|
||||
- "Tax Period" / "Period" / "Month" — e.g. "Month 1", "Week 12".
|
||||
- Two numeric columns per line: "This Period" (or "Amount", "Current") and "Year to Date" (or "YTD"). **Always take the This Period column**, never YTD.
|
||||
- Payments / Earnings block: "Basic Pay", "Salary", "Bonus", "Overtime", "Commission", "Holiday Pay".
|
||||
- Deductions block: "Income Tax" / "PAYE", "National Insurance" / "NI" / "NIC", "Pension" / "Pension Contribution" / "Salary Sacrifice Pension", "Student Loan" / "SL", optional: "Union Dues", "Charity", "Season Ticket Loan", "Private Medical", etc.
|
||||
- "Gross Pay" / "Total Gross" — sum of payments.
|
||||
- "Net Pay" / "Take Home" / "Amount Payable" — the money actually paid.
|
||||
- "Tax Code" — e.g. "1257L", "BR", "D0", "NT".
|
||||
- "NI Number" / "National Insurance Number" — `AA123456A` format. Never invent one.
|
||||
- "Employer" / "Company" — usually in the letterhead. "Employee" / "Name".
|
||||
- Currency: almost always GBP / "£" for UK payslips. If the PDF is not in GBP or not a UK payslip, still return the numbers as-is but include a best-effort `currency` field.
|
||||
|
||||
### Step 4. Map to the schema and emit JSON
|
||||
|
||||
Rules that apply regardless of the caller's exact schema:
|
||||
|
||||
- **Dates**: `pay_date` MUST be `YYYY-MM-DD`. If the PDF prints `12/03/2026`, interpret as `DD/MM/YYYY` (UK format) → `2026-03-12`. If ambiguous (`01/02/2026`), prefer UK ordering. If impossible to determine a year, use the pay_period year.
|
||||
- **Money fields**: emit as JSON numbers, not strings. Two decimal places are acceptable (`2450.17`). Strip `£`, commas, and trailing spaces. Negative values stay negative.
|
||||
- **Missing numeric fields**: emit `0` (zero), not `null`, not an empty string, not `"N/A"`.
|
||||
- **`other_deductions`**: an object mapping `{ "<label>": <number>, ... }` for any deduction that isn't one of the first-class fields in the schema (tax, NI, pension, student loan). Use the exact label from the payslip (e.g. `"Season Ticket Loan"`, `"Private Medical"`). If there are no other deductions, emit `{}` — NEVER `null` and NEVER omit the key.
|
||||
- **Column discipline**: ALWAYS use the "This Period" column, NEVER the YTD column. If only one column exists, that's the period column.
|
||||
- **Currency default**: `"GBP"` unless the payslip explicitly shows another currency symbol or ISO code.
|
||||
- **No invented data**: If a field genuinely isn't on the payslip, use the documented default (`0` for money, `""` for strings, `{}` for objects). Do NOT make up names, NI numbers, tax codes, or employers.
|
||||
|
||||
Follow the exact field names and types given in the prompt's schema. If the prompt's schema adds fields not listed above, produce them too using the same discipline.
|
||||
|
||||
## Failure mode
|
||||
|
||||
If the PDF cannot be read at all — unreadable base64, not a PDF, encrypted PDF with no text layer, no text-extraction tool available, or clearly not a UK payslip — emit a single JSON object:
|
||||
|
||||
```json
|
||||
{"error": "<short human reason>"}
|
||||
```
|
||||
|
||||
Examples of acceptable error reasons:
|
||||
- `"base64 did not decode to a valid PDF"`
|
||||
- `"pdf has no extractable text layer (image-only scan)"`
|
||||
- `"no pdf text extraction tool available (pdftotext/pypdf/pdfplumber all missing)"`
|
||||
- `"document does not appear to be a UK payslip"`
|
||||
- `"pay_date not found on document"`
|
||||
|
||||
The caller treats the `error` key as a non-retriable parse failure. Do not include any other keys when emitting an error object.
|
||||
|
||||
## Hard constraints — things you MUST NOT do
|
||||
|
||||
1. **No network calls.** Do not curl, wget, dig, or otherwise talk to the network. Everything you need is in the prompt.
|
||||
2. **No modifications to `/workspace/infra/**`.** Do not edit, write, or commit any file under the infra repo. The only file you may create is the scratch PDF at `/tmp/payslip.pdf` (and intermediate text dumps under `/tmp/`).
|
||||
3. **No git operations.** No `git add`, `git commit`, `git push`, nothing.
|
||||
4. **No kubectl, no terraform, no vault.** You are not an infra agent — you are a narrow extractor.
|
||||
5. **No markdown in output.** No ` ```json ` fences, no preamble like "Here's the extraction:", no trailing notes. The ENTIRE final assistant message is exactly one JSON object.
|
||||
6. **No verbose logging in the final message.** It is fine to run bash commands and see their output during processing, but your final assistant message is JSON and nothing else.
|
||||
7. **No hallucinated fields.** If the payslip does not show a pension line, do not invent one. Use the documented default instead.
|
||||
|
||||
## Output discipline — summary
|
||||
|
||||
- Exactly one JSON object, UTF-8, no BOM.
|
||||
- Keys match the schema the caller gave you.
|
||||
- Numeric fields are JSON numbers, not strings.
|
||||
- `pay_date` is `YYYY-MM-DD`.
|
||||
- `other_deductions` is always present and is an object (possibly `{}`).
|
||||
- Missing money → `0`, missing string → `""`, missing object → `{}`.
|
||||
- On unrecoverable failure, one JSON object with a single `error` key.
|
||||
|
||||
That's the whole job. Decode, extract, parse, emit JSON. Be boring and exact.
|
||||
|
|
@ -1440,6 +1440,30 @@ resource "null_resource" "pg_terraform_state_db" {
|
|||
}
|
||||
}
|
||||
|
||||
# Create payslip_ingest database for the payslip-ingest webhook service.
|
||||
# Role password is managed by Vault Database Secrets Engine (static role `pg-payslip-ingest`, 7d rotation).
|
||||
resource "null_resource" "pg_payslip_ingest_db" {
|
||||
depends_on = [null_resource.pg_cluster]
|
||||
|
||||
triggers = {
|
||||
db_name = "payslip_ingest"
|
||||
username = "payslip_ingest"
|
||||
}
|
||||
|
||||
provisioner "local-exec" {
|
||||
command = <<-EOT
|
||||
kubectl --kubeconfig ${var.kube_config_path} exec -n dbaas pg-cluster-1 -c postgres -- \
|
||||
bash -c '
|
||||
psql -U postgres -tc "SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = '"'"'payslip_ingest'"'"'" | grep -q 1 || \
|
||||
psql -U postgres -c "CREATE ROLE payslip_ingest WITH LOGIN PASSWORD '"'"'changeme-vault-will-rotate'"'"'"
|
||||
psql -U postgres -tc "SELECT 1 FROM pg_catalog.pg_database WHERE datname = '"'"'payslip_ingest'"'"'" | grep -q 1 || \
|
||||
psql -U postgres -c "CREATE DATABASE payslip_ingest OWNER payslip_ingest"
|
||||
psql -U postgres -c "GRANT ALL PRIVILEGES ON DATABASE payslip_ingest TO payslip_ingest"
|
||||
'
|
||||
EOT
|
||||
}
|
||||
}
|
||||
|
||||
# Old PostgreSQL deployment — kept commented for rollback reference
|
||||
# resource "kubernetes_deployment" "postgres" {
|
||||
# metadata {
|
||||
|
|
|
|||
224
stacks/monitoring/modules/monitoring/dashboards/uk-payslip.json
Normal file
224
stacks/monitoring/modules/monitoring/dashboards/uk-payslip.json
Normal file
|
|
@ -0,0 +1,224 @@
|
|||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": { "type": "datasource", "uid": "grafana" },
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": "UK payslip breakdown — gross/net/tax/NI trends, YTD progression against income tax bands, deductions split, and effective rate.",
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Monthly gross / net / tax / NI",
|
||||
"type": "timeseries",
|
||||
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 0 },
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "palette-classic" },
|
||||
"unit": "currencyGBP",
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": { "legend": false, "tooltip": false, "viz": false },
|
||||
"lineWidth": 2,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"stacking": { "group": "A", "mode": "none" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": { "calcs": ["last", "mean"], "displayMode": "table", "placement": "bottom" },
|
||||
"tooltip": { "mode": "multi", "sort": "desc" }
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||
"rawSql": "SELECT pay_date AS \"time\", gross_pay, net_pay, income_tax, national_insurance FROM payslip_ingest.payslip WHERE $__timeFilter(pay_date) ORDER BY pay_date",
|
||||
"format": "time_series",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "YTD gross (this tax year) with UK band thresholds",
|
||||
"type": "timeseries",
|
||||
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 0 },
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "palette-classic" },
|
||||
"unit": "currencyGBP",
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "YTD gross",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": { "legend": false, "tooltip": false, "viz": false },
|
||||
"lineWidth": 2,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"stacking": { "group": "A", "mode": "none" },
|
||||
"thresholdsStyle": { "mode": "line" }
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "yellow", "value": 12570 },
|
||||
{ "color": "orange", "value": 50270 },
|
||||
{ "color": "red", "value": 125140 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": { "calcs": ["last", "max"], "displayMode": "table", "placement": "bottom" },
|
||||
"tooltip": { "mode": "multi", "sort": "desc" }
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||
"rawSql": "SELECT pay_date AS \"time\", SUM(gross_pay) OVER (PARTITION BY tax_year ORDER BY pay_date) AS ytd_gross FROM payslip_ingest.payslip WHERE $__timeFilter(pay_date) ORDER BY pay_date",
|
||||
"format": "time_series",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "Deductions breakdown per payslip",
|
||||
"type": "timeseries",
|
||||
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 9 },
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "palette-classic" },
|
||||
"unit": "currencyGBP",
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "bars",
|
||||
"fillOpacity": 80,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": { "legend": false, "tooltip": false, "viz": false },
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": { "group": "A", "mode": "normal" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": { "calcs": ["sum", "mean"], "displayMode": "table", "placement": "bottom" },
|
||||
"tooltip": { "mode": "multi", "sort": "desc" }
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||
"rawSql": "SELECT pay_date AS \"time\", income_tax, national_insurance, pension_employee, student_loan FROM payslip_ingest.payslip WHERE $__timeFilter(pay_date) ORDER BY pay_date",
|
||||
"format": "time_series",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"title": "Latest effective rate & take-home %",
|
||||
"type": "timeseries",
|
||||
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 9 },
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "palette-classic" },
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100,
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": { "legend": false, "tooltip": false, "viz": false },
|
||||
"lineWidth": 2,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"stacking": { "group": "A", "mode": "none" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": { "calcs": ["last", "mean"], "displayMode": "table", "placement": "bottom" },
|
||||
"tooltip": { "mode": "multi", "sort": "desc" }
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "payslips-pg" },
|
||||
"rawSql": "SELECT pay_date AS \"time\", ROUND(((income_tax + national_insurance)::numeric / NULLIF(gross_pay, 0)) * 100, 2) AS \"effective_rate_pct\", ROUND((net_pay::numeric / NULLIF(gross_pay, 0)) * 100, 2) AS \"take_home_pct\" FROM payslip_ingest.payslip WHERE $__timeFilter(pay_date) ORDER BY pay_date",
|
||||
"format": "time_series",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"refresh": "5m",
|
||||
"schemaVersion": 39,
|
||||
"tags": ["finance", "personal", "uk-tax"],
|
||||
"templating": { "list": [] },
|
||||
"time": { "from": "now-2y", "to": "now" },
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "UK Payslip",
|
||||
"uid": "uk-payslip",
|
||||
"version": 1
|
||||
}
|
||||
330
stacks/payslip-ingest/main.tf
Normal file
330
stacks/payslip-ingest/main.tf
Normal file
|
|
@ -0,0 +1,330 @@
|
|||
variable "image_tag" {
|
||||
type = string
|
||||
default = "latest"
|
||||
description = "payslip-ingest image tag. Use 8-char git SHA in CI; :latest only for local trials."
|
||||
}
|
||||
|
||||
variable "postgresql_host" { type = string }
|
||||
|
||||
locals {
|
||||
namespace = "payslip-ingest"
|
||||
image = "registry.viktorbarzin.me/payslip-ingest:${var.image_tag}"
|
||||
labels = {
|
||||
app = "payslip-ingest"
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_namespace" "payslip_ingest" {
|
||||
metadata {
|
||||
name = local.namespace
|
||||
labels = {
|
||||
tier = local.tiers.aux
|
||||
"istio-injection" = "disabled"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# App secrets sourced from multiple Vault KV keys.
|
||||
# Seed these manually in Vault before applying:
|
||||
# secret/paperless-ngx -> property `api_token`
|
||||
# secret/claude-agent-service -> property `api_bearer_token`
|
||||
# secret/payslip-ingest -> property `webhook_bearer_token`
|
||||
resource "kubernetes_manifest" "external_secret" {
|
||||
manifest = {
|
||||
apiVersion = "external-secrets.io/v1beta1"
|
||||
kind = "ExternalSecret"
|
||||
metadata = {
|
||||
name = "payslip-ingest-secrets"
|
||||
namespace = local.namespace
|
||||
}
|
||||
spec = {
|
||||
refreshInterval = "15m"
|
||||
secretStoreRef = {
|
||||
name = "vault-kv"
|
||||
kind = "ClusterSecretStore"
|
||||
}
|
||||
target = {
|
||||
name = "payslip-ingest-secrets"
|
||||
template = {
|
||||
metadata = {
|
||||
annotations = {
|
||||
"reloader.stakater.com/match" = "true"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
data = [
|
||||
{
|
||||
secretKey = "PAPERLESS_API_TOKEN"
|
||||
remoteRef = {
|
||||
key = "paperless-ngx"
|
||||
property = "api_token"
|
||||
}
|
||||
},
|
||||
{
|
||||
secretKey = "CLAUDE_AGENT_BEARER_TOKEN"
|
||||
remoteRef = {
|
||||
key = "claude-agent-service"
|
||||
property = "api_bearer_token"
|
||||
}
|
||||
},
|
||||
{
|
||||
secretKey = "WEBHOOK_BEARER_TOKEN"
|
||||
remoteRef = {
|
||||
key = "payslip-ingest"
|
||||
property = "webhook_bearer_token"
|
||||
}
|
||||
},
|
||||
]
|
||||
}
|
||||
}
|
||||
depends_on = [kubernetes_namespace.payslip_ingest]
|
||||
}
|
||||
|
||||
# DB credentials from Vault database engine (rotated every 7 days).
|
||||
# Template builds the asyncpg DSN consumed by the FastAPI app as DB_CONNECTION_STRING.
|
||||
resource "kubernetes_manifest" "db_external_secret" {
|
||||
manifest = {
|
||||
apiVersion = "external-secrets.io/v1beta1"
|
||||
kind = "ExternalSecret"
|
||||
metadata = {
|
||||
name = "payslip-ingest-db-creds"
|
||||
namespace = local.namespace
|
||||
}
|
||||
spec = {
|
||||
refreshInterval = "15m"
|
||||
secretStoreRef = {
|
||||
name = "vault-database"
|
||||
kind = "ClusterSecretStore"
|
||||
}
|
||||
target = {
|
||||
name = "payslip-ingest-db-creds"
|
||||
template = {
|
||||
metadata = {
|
||||
annotations = {
|
||||
"reloader.stakater.com/match" = "true"
|
||||
}
|
||||
}
|
||||
data = {
|
||||
DB_CONNECTION_STRING = "postgresql+asyncpg://payslip_ingest:{{ .password }}@${var.postgresql_host}:5432/payslip_ingest"
|
||||
DB_PASSWORD = "{{ .password }}"
|
||||
}
|
||||
}
|
||||
}
|
||||
data = [{
|
||||
secretKey = "password"
|
||||
remoteRef = {
|
||||
key = "static-creds/pg-payslip-ingest"
|
||||
property = "password"
|
||||
}
|
||||
}]
|
||||
}
|
||||
}
|
||||
depends_on = [kubernetes_namespace.payslip_ingest]
|
||||
}
|
||||
|
||||
resource "kubernetes_deployment" "payslip_ingest" {
|
||||
metadata {
|
||||
name = "payslip-ingest"
|
||||
namespace = kubernetes_namespace.payslip_ingest.metadata[0].name
|
||||
labels = merge(local.labels, {
|
||||
tier = local.tiers.aux
|
||||
})
|
||||
annotations = {
|
||||
"reloader.stakater.com/search" = "true"
|
||||
}
|
||||
}
|
||||
|
||||
spec {
|
||||
replicas = 1
|
||||
strategy {
|
||||
type = "Recreate"
|
||||
}
|
||||
|
||||
selector {
|
||||
match_labels = local.labels
|
||||
}
|
||||
|
||||
template {
|
||||
metadata {
|
||||
labels = local.labels
|
||||
annotations = {
|
||||
"dependency.kyverno.io/wait-for" = "postgresql.dbaas:5432"
|
||||
}
|
||||
}
|
||||
|
||||
spec {
|
||||
image_pull_secrets {
|
||||
name = "registry-credentials"
|
||||
}
|
||||
|
||||
init_container {
|
||||
name = "alembic-migrate"
|
||||
image = local.image
|
||||
command = ["python", "-m", "payslip_ingest", "migrate"]
|
||||
|
||||
env_from {
|
||||
secret_ref {
|
||||
name = "payslip-ingest-secrets"
|
||||
}
|
||||
}
|
||||
env_from {
|
||||
secret_ref {
|
||||
name = "payslip-ingest-db-creds"
|
||||
}
|
||||
}
|
||||
|
||||
env {
|
||||
name = "PAPERLESS_URL"
|
||||
value = "http://paperless-ngx.paperless-ngx.svc.cluster.local"
|
||||
}
|
||||
env {
|
||||
name = "CLAUDE_AGENT_URL"
|
||||
value = "http://claude-agent-service.claude-agent.svc.cluster.local:8080"
|
||||
}
|
||||
|
||||
resources {
|
||||
requests = {
|
||||
cpu = "50m"
|
||||
memory = "256Mi"
|
||||
}
|
||||
limits = {
|
||||
memory = "512Mi"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
container {
|
||||
name = "payslip-ingest"
|
||||
image = local.image
|
||||
|
||||
port {
|
||||
container_port = 8080
|
||||
}
|
||||
|
||||
env_from {
|
||||
secret_ref {
|
||||
name = "payslip-ingest-secrets"
|
||||
}
|
||||
}
|
||||
env_from {
|
||||
secret_ref {
|
||||
name = "payslip-ingest-db-creds"
|
||||
}
|
||||
}
|
||||
|
||||
env {
|
||||
name = "PAPERLESS_URL"
|
||||
value = "http://paperless-ngx.paperless-ngx.svc.cluster.local"
|
||||
}
|
||||
env {
|
||||
name = "CLAUDE_AGENT_URL"
|
||||
value = "http://claude-agent-service.claude-agent.svc.cluster.local:8080"
|
||||
}
|
||||
|
||||
readiness_probe {
|
||||
http_get {
|
||||
path = "/healthz"
|
||||
port = 8080
|
||||
}
|
||||
initial_delay_seconds = 5
|
||||
period_seconds = 10
|
||||
}
|
||||
|
||||
liveness_probe {
|
||||
http_get {
|
||||
path = "/healthz"
|
||||
port = 8080
|
||||
}
|
||||
initial_delay_seconds = 5
|
||||
period_seconds = 10
|
||||
}
|
||||
|
||||
resources {
|
||||
requests = {
|
||||
cpu = "50m"
|
||||
memory = "256Mi"
|
||||
}
|
||||
limits = {
|
||||
memory = "512Mi"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
lifecycle {
|
||||
ignore_changes = [spec[0].template[0].spec[0].dns_config] # KYVERNO_LIFECYCLE_V1
|
||||
}
|
||||
|
||||
depends_on = [
|
||||
kubernetes_manifest.external_secret,
|
||||
kubernetes_manifest.db_external_secret,
|
||||
]
|
||||
}
|
||||
|
||||
# ClusterIP-only — webhook is cluster-internal (paperless-ngx -> payslip-ingest).
|
||||
resource "kubernetes_service" "payslip_ingest" {
|
||||
metadata {
|
||||
name = "payslip-ingest"
|
||||
namespace = kubernetes_namespace.payslip_ingest.metadata[0].name
|
||||
labels = local.labels
|
||||
}
|
||||
|
||||
spec {
|
||||
type = "ClusterIP"
|
||||
selector = local.labels
|
||||
|
||||
port {
|
||||
name = "http"
|
||||
port = 8080
|
||||
target_port = 8080
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Plan-time read of the ESO-created K8s Secret for Grafana datasource password.
|
||||
# First apply: -target=kubernetes_manifest.db_external_secret first so the Secret exists.
|
||||
data "kubernetes_secret" "payslip_ingest_db_creds" {
|
||||
metadata {
|
||||
name = "payslip-ingest-db-creds"
|
||||
namespace = kubernetes_namespace.payslip_ingest.metadata[0].name
|
||||
}
|
||||
depends_on = [kubernetes_manifest.db_external_secret]
|
||||
}
|
||||
|
||||
# Grafana datasource for payslip_ingest PostgreSQL DB.
|
||||
# Lives in the monitoring namespace so the grafana sidecar (label grafana_datasource=1) picks it up.
|
||||
resource "kubernetes_config_map" "grafana_payslips_datasource" {
|
||||
metadata {
|
||||
name = "grafana-payslips-datasource"
|
||||
namespace = "monitoring"
|
||||
labels = {
|
||||
grafana_datasource = "1"
|
||||
}
|
||||
}
|
||||
data = {
|
||||
"payslips-datasource.yaml" = yamlencode({
|
||||
apiVersion = 1
|
||||
datasources = [{
|
||||
name = "Payslips"
|
||||
type = "postgres"
|
||||
access = "proxy"
|
||||
url = "${var.postgresql_host}:5432"
|
||||
database = "payslip_ingest"
|
||||
user = "payslip_ingest"
|
||||
uid = "payslips-pg"
|
||||
jsonData = {
|
||||
sslmode = "disable"
|
||||
postgresVersion = 1600
|
||||
timescaledb = false
|
||||
}
|
||||
secureJsonData = {
|
||||
password = data.kubernetes_secret.payslip_ingest_db_creds.data["DB_PASSWORD"]
|
||||
}
|
||||
editable = true
|
||||
}]
|
||||
})
|
||||
}
|
||||
}
|
||||
18
stacks/payslip-ingest/terragrunt.hcl
Normal file
18
stacks/payslip-ingest/terragrunt.hcl
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
include "root" {
|
||||
path = find_in_parent_folders()
|
||||
}
|
||||
|
||||
dependency "platform" {
|
||||
config_path = "../platform"
|
||||
skip_outputs = true
|
||||
}
|
||||
|
||||
dependency "vault" {
|
||||
config_path = "../vault"
|
||||
skip_outputs = true
|
||||
}
|
||||
|
||||
dependency "external-secrets" {
|
||||
config_path = "../external-secrets"
|
||||
skip_outputs = true
|
||||
}
|
||||
|
|
@ -523,7 +523,7 @@ resource "vault_database_secret_backend_connection" "postgresql" {
|
|||
# "pg-trading", # Commented out 2026-04-06 - trading-bot disabled
|
||||
"pg-health", "pg-linkwarden",
|
||||
"pg-affine", "pg-woodpecker", "pg-claude-memory",
|
||||
"pg-terraform-state"
|
||||
"pg-terraform-state", "pg-payslip-ingest"
|
||||
]
|
||||
|
||||
postgresql {
|
||||
|
|
@ -661,6 +661,14 @@ resource "vault_database_secret_backend_static_role" "pg_terraform_state" {
|
|||
rotation_period = 604800
|
||||
}
|
||||
|
||||
resource "vault_database_secret_backend_static_role" "pg_payslip_ingest" {
|
||||
backend = vault_mount.database.path
|
||||
db_name = vault_database_secret_backend_connection.postgresql.name
|
||||
name = "pg-payslip-ingest"
|
||||
username = "payslip_ingest"
|
||||
rotation_period = 604800
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# Kubernetes Secrets Engine — Dynamic K8s Credentials
|
||||
# =============================================================================
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue