Event-driven UK payslip ingest: Paperless-ngx webhook -> claude-agent-service extraction -> Postgres -> Grafana

Find a file

Viktor Barzin d91f34ddb4 processor + parser: fix 3 backfill failure modes ## Context After the first v2 backfill (commit `f62c533`), 72 of 73 real payslips landed correctly, but three residual failure modes remained: 1. doc_id=215 — a 1442-byte empty-text PDF that Claude hallucinated a `pay_date=1900-01-01 / gross=0 / net=0` row for. Data poison waiting to happen. 2. doc_id=39 — a P60 End of Year Certificate. Got tagged `payslip` in Paperless, has no Paperless title, so the title-based filter couldn't catch it; the regex parser then happily pulled bogus numbers out of the P60 layout. 3. doc_id=49 — a real June 2021 variant-A payslip with an `EE Discount BIK` line in BOTH Payments and Deductions at 12.00. The parser was configured to drop `EE Discount BIK` from `other_deductions` (treating it as a known mapped field), which caused validate_totals to fail by exactly 12.00. ## This change ### processor.py — defence in depth - `NON_PAYSLIP_CONTENT_RE` — new regex run against the first 500 chars of pdftotext output. Catches `P60 End of Year Certificate` and `Take-home income per month` (Viktor's comp estimation spreadsheet). First-500-char scoping keeps it from false-positiving a legit payslip that mentions "P60" in a footer. - Post-extraction sanity checks — reject a ProcessResult if `pay_date.year < 2010` (Viktor joined Meta in 2019) or if `gross_pay == net_pay == 0`. These raise rather than insert, so the backfill's existing `except Exception` block logs and continues without poisoning the DB. Supersedes the 1900-01-01 case that would otherwise slip through. ### meta_uk.py — variant A BIK fix Removed `EE Discount BIK` from `VARIANT_A_DEDUCTIONS_KNOWN`. That set filters items OUT of `other_deductions` (because they have dedicated schema fields). `EE Discount BIK` has no dedicated field — it should stay in `other_deductions` like Private Dental and Private Medical so the validation math balances. ### Fixtures + tests - New fixture `meta_uk_2021_06_variant_a_bik.txt` — real pdftotext from doc_id=49 — encodes the BIK-in-both-columns case so a regression would fail this fixture's validation test. - `test_parses_variant_a_with_ee_discount_bik` — explicitly asserts `EE Discount BIK` lands in `other_deductions`. - `test_rejects_implausible_pay_date`, `test_rejects_zero_gross_zero_net` — cover the two sanity-check branches. - `test_skips_p60_by_content_when_title_is_null` — covers the content-based non-payslip filter. ## Test Plan ### Automated ``` $ poetry run pytest ============================== 57 passed in 2.42s ============================== $ poetry run ruff check . All checks passed! $ poetry run mypy . Success: no issues found in 24 source files ``` ### Manual verification (after deploy + re-run backfill) Expected DB shape: - Total rows ≈ 71 (88 paperless tags − 15 non-payslip titles − 2 null-title non-payslips caught by content filter) - `validated = true` on ≥99% of rows - No `pay_date < 2010` rows - No rows with employer IS NULL ## Reproduce locally 1. `cd payslip-ingest && poetry run pytest` 2. Expected: 57 passed, including the 3 new processor tests and the 5 parametrised fixture-total-validation tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>		2026-04-19 12:00:00 +00:00
alembic	v2: regex parser for Meta UK template + accurate RSU tax attribution	2026-04-19 10:53:52 +00:00
payslip_ingest	processor + parser: fix 3 backfill failure modes	2026-04-19 12:00:00 +00:00
tests	processor + parser: fix 3 backfill failure modes	2026-04-19 12:00:00 +00:00
.gitignore	Initial commit: event-driven UK payslip ingest service	2026-04-18 22:10:23 +00:00
.woodpecker.yml	Initial commit: event-driven UK payslip ingest service	2026-04-18 22:10:23 +00:00
alembic.ini	Initial commit: event-driven UK payslip ingest service	2026-04-18 22:10:23 +00:00
Dockerfile	extractor: preextract PDF text with pdftotext before calling Claude	2026-04-18 22:48:04 +00:00
poetry.lock	Initial commit: event-driven UK payslip ingest service	2026-04-18 22:10:23 +00:00
pyproject.toml	Initial commit: event-driven UK payslip ingest service	2026-04-18 22:10:23 +00:00
README.md	Initial commit: event-driven UK payslip ingest service	2026-04-18 22:10:23 +00:00

README.md

payslip-ingest

Event-driven UK payslip ingest: Paperless-ngx fires a webhook when a document is tagged payslip; this service fetches the PDF, calls claude-agent-service to extract structured fields, and upserts into Postgres keyed by paperless_doc_id (idempotent). A CLI backfill mode enumerates every existing payslip in Paperless for initial population.

Local dev

poetry install
poetry run pytest -q
poetry run mypy .
poetry run ruff check .

# Smoke-test extraction against a real PDF (no DB writes):
export CLAUDE_AGENT_URL=http://claude-agent-service.claude-agent.svc.cluster.local:8080
export CLAUDE_AGENT_BEARER_TOKEN=...
poetry run python -m payslip_ingest extract-one /tmp/sample.pdf

Env vars

Variable	Purpose
`PAPERLESS_URL`	Paperless-ngx base URL (e.g. `https://paperless.viktorbarzin.me`)
`PAPERLESS_API_TOKEN`	Paperless API token (User → My Profile → API Auth Token)
`CLAUDE_AGENT_URL`	claude-agent-service URL (`http://claude-agent-service.claude-agent.svc.cluster.local:8080`)
`CLAUDE_AGENT_BEARER_TOKEN`	Vault `secret/claude-agent-service` → `api_bearer_token`
`DB_CONNECTION_STRING`	SQLAlchemy async URL: `postgresql+asyncpg://user:pass@host/db`
`WEBHOOK_BEARER_TOKEN`	Shared secret Paperless sends in `Authorization: Bearer ...`

Paperless workflow configuration

In Paperless-ngx, create a workflow:

Name: payslip-ingest
Trigger: Document Added, matching tag payslip
Action type: Webhook
URL: http://payslip-ingest.payslip-ingest.svc.cluster.local:8080/webhook
Method: POST
Headers:
- Authorization: Bearer <WEBHOOK_BEARER_TOKEN>
- Content-Type: application/json
Body (template):
```
{"document_id": {{ document_id }}}
```

Deployment

Ship to the payslip-ingest namespace. The service serializes incoming webhooks onto an in-process queue so it never collides with claude-agent-service's single-job lock.

Run the initial backfill once the deployment is live:

kubectl -n payslip-ingest create job \
  --from=deployment/payslip-ingest \
  payslip-backfill-$(date +%s) \
  -- python -m payslip_ingest backfill --all

Architecture notes

extract-one never touches the DB — safe for ad-hoc re-extraction on disk.
The backfill command is idempotent (skips rows whose paperless_doc_id already exists) so it can be re-run freely.
Totals validation is a best-effort sanity check; mismatches are stored with validated=false and raw_extraction retained for manual review, rather than rejected.
The agent service is single-threaded. The webhook handler enqueues and returns 202; a single background worker drains the queue one at a time and absorbs 409-busy responses from the agent with retry-with-backoff.
New agent prompt lives at .claude/agents/payslip-extractor in the infra repo — this is a separate deliverable (see TODOs).