974181674d
## Context
v1 shipped a Claude Haiku-based extractor that validated only 10/71
backfilled rows. Haiku fumbles the arithmetic on pension salary-sacrifice,
conflates RSU vest with regular earnings, and occasionally misreads YTD
vs this-period columns — so 86% of rows land with validated=false and the
downstream dashboards under-report take-home.
Meta UK uses a stable two-variant template (pre/post 2022-01-31 boundary),
so a regex parser is both faster (ms vs. 30-90s + $0.01-0.05/call) and
more accurate. v2 introduces that parser as the primary path, keeps
Claude as the fallback for non-Meta payslips, and surfaces new fields
the dashboard needs to attribute PAYE between cash salary and RSU vests
correctly.
## This change
### Parser (new)
`payslip_ingest/parsers/meta_uk.py` detects the layout variant by header
presence:
- **Variant A** (pre-2022): vertical Description/This Period/This Year.
`AE Pension EE` is a positive deduction against a pre-sacrifice gross —
maps to `pension_employee` for the existing validation formula to hold.
- **Variant B** (post-2022): side-by-side Payments | Deductions | Year to
Date. `AE Pension EE` is NEGATIVE in Payments (salary sacrifice) — maps
to `pension_sacrifice` and is already netted into Total Payment.
`rsu_vest = RSU Tax Offset + RSU Excs Refund` (Meta's template inflates
Taxable Pay without using a matching offset deduction).
Column boundaries come from the header row's anchor positions; each data
row slices into 3 cells and the last numeric token per cell is the amount.
Anchor misses raise ParserError so the caller falls back to Claude rather
than silently returning bad data.
### New fields
Schema + DB + Claude prompt gain:
- `salary`, `bonus`, `pension_sacrifice` — earnings decomposition for the
dashboard's bonus-sacrifice visibility and earnings-breakdown chart
- `taxable_pay`, `ytd_tax_paid`, `ytd_taxable_pay`, `ytd_gross` — powers
the YTD-effective-rate method of attributing cash tax vs RSU tax, which
is the only method that's accurate month-to-month
All new columns default to 0 / null so v1 rows continue to round-trip.
### Orchestration
processor.py tries `parse_meta_uk(pdftotext(pdf))` first. On success the
result goes straight to the DB — zero Claude tokens spent, extraction in
milliseconds. On ParserError it falls through to ClaudeExtractor as before.
ProcessResult gains an `extractor` field ("meta_uk_regex" | "claude") so
backfill logs show the hit rate.
## Tests
- `test_meta_uk_parser.py` — 11 tests covering variant A, variant B
(standard + bonus month + bonus-sacrificed month), malformed inputs,
and end-to-end totals validation for all 4 golden fixtures.
- `test_processor.py` — 2 new tests proving the regex-first short-circuit
and the Claude fallback on non-Meta inputs.
Fixtures under `tests/fixtures/` are hand-crafted `pdftotext -layout`
emulations — real Meta numbers from the plan's sample payslips for
variant B, synthesized realistic variant A and bonus-sacrificed samples.
0001_initial.py reformat is yapf cleanup touched during the session's
format pass; not a behavior change.
## Test Plan
### Automated
```
$ poetry run pytest
============================= test session starts ==============================
collected 53 items
tests/test_extractor.py ..... [ 9%]
tests/test_meta_uk_parser.py ........... [ 30%]
tests/test_paperless.py ...... [ 41%]
tests/test_processor.py .............. [ 67%]
tests/test_schema.py .... [ 75%]
tests/test_tax_year.py ........ [ 90%]
tests/test_webhook.py ..... [100%]
============================== 53 passed in 1.66s ==============================
$ poetry run ruff check .
All checks passed!
$ poetry run mypy .
Success: no issues found in 24 source files
$ poetry run yapf --style pyproject.toml --diff --recursive payslip_ingest tests
(no output — all files are yapf-clean)
```
### Manual Verification
Smoke-test the parser against a real Meta payslip PDF on the deploy host:
```
# After 0003 migration applied to prod DB
$ poetry run python -c "
from payslip_ingest.parsers import parse_meta_uk
import subprocess
text = subprocess.check_output(['pdftotext', '-layout', '/path/to/real.pdf', '-']).decode()
p = parse_meta_uk(text)
print(p.model_dump_json(indent=2))
"
```
Expected: JSON with salary/bonus/rsu_vest/pension_sacrifice populated and
`validate_totals(p)` returning True.
## Reproduce locally
1. `cd payslip-ingest && poetry install`
2. `poetry run pytest tests/test_meta_uk_parser.py -v`
3. Expected: 11 tests pass, each fixture validates totals within 2p.
Closes: code-un1
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
76 lines
3.5 KiB
Python
76 lines
3.5 KiB
Python
from datetime import date
|
|
from decimal import Decimal
|
|
|
|
from pydantic import BaseModel, ConfigDict, Field
|
|
|
|
TOTALS_TOLERANCE = Decimal("0.02")
|
|
|
|
|
|
class ExtractedPayslip(BaseModel):
|
|
model_config = ConfigDict(extra="forbid")
|
|
|
|
pay_date: date
|
|
pay_period_start: date | None = None
|
|
pay_period_end: date | None = None
|
|
employer: str | None = None
|
|
currency: str = "GBP"
|
|
gross_pay: Decimal
|
|
income_tax: Decimal = Field(default=Decimal("0"))
|
|
national_insurance: Decimal = Field(default=Decimal("0"))
|
|
pension_employee: Decimal = Field(default=Decimal("0"))
|
|
pension_employer: Decimal = Field(default=Decimal("0"))
|
|
student_loan: Decimal = Field(default=Decimal("0"))
|
|
# RSU vest reported on the UK payslip is notional — the share grant is
|
|
# handled by Schwab which withholds US-side tax by selling shares. The
|
|
# UK payslip only lists it for HMRC reporting; no cash flows through
|
|
# UK payroll. Track it separately so dashboards can derive cash-only
|
|
# gross = gross_pay - rsu_vest.
|
|
rsu_vest: Decimal = Field(default=Decimal("0"))
|
|
# Corresponding offset deduction that nets the RSU out of cash pay on the
|
|
# UK slip (labels vary: "Shares Retained", "Stock Tax Withholding",
|
|
# "RSU Offset", "Notional Pay Offset"). Same as rsu_vest in magnitude.
|
|
# Meta's template doesn't carry one — rsu_vest grosses up Taxable Pay
|
|
# directly and PAYE is computed on the grossed-up figure.
|
|
rsu_offset: Decimal = Field(default=Decimal("0"))
|
|
# v2 additions: earnings decomposition + YTD snapshot for accurate
|
|
# cash-vs-RSU tax attribution. All default to 0/None so v1 extractor
|
|
# output continues to validate.
|
|
salary: Decimal = Field(default=Decimal("0"))
|
|
bonus: Decimal = Field(default=Decimal("0"))
|
|
# Absolute value of negative "AE Pension EE" in Payments block — the
|
|
# employee-side salary-sacrifice contribution that reduces gross before
|
|
# PAYE. pension_employee stays reserved for the rare case where pension
|
|
# is posted as a positive Deduction.
|
|
pension_sacrifice: Decimal = Field(default=Decimal("0"))
|
|
# Post-sacrifice Taxable Pay = gross_pay + rsu_vest (PAYE base). Nullable
|
|
# because variant A payslips (pre-2022) don't surface the summary block.
|
|
taxable_pay: Decimal | None = None
|
|
# YTD values from the summary block — powers the ytd-effective-tax-rate
|
|
# formula used by the dashboard.
|
|
ytd_tax_paid: Decimal | None = None
|
|
ytd_taxable_pay: Decimal | None = None
|
|
ytd_gross: Decimal | None = None
|
|
other_deductions: dict[str, Decimal] = Field(default_factory=dict)
|
|
net_pay: Decimal
|
|
|
|
|
|
class WebhookPayload(BaseModel):
|
|
model_config = ConfigDict(extra="forbid")
|
|
|
|
document_id: int
|
|
|
|
|
|
def validate_totals(p: ExtractedPayslip) -> bool:
|
|
"""Check that gross - deductions ≈ net within a 2p tolerance.
|
|
|
|
- Employer pension is excluded — it never leaves the employer's books.
|
|
- `rsu_offset` is included as a deduction: it's the line that nets
|
|
the RSU notional back out of cash pay on UK payslips with stock comp.
|
|
The gross + rsu_vest inflation is offset by rsu_offset of equal size.
|
|
Meta's template doesn't carry rsu_offset — the grossing happens via
|
|
Taxable Pay and PAYE, so `gross_pay` already excludes the RSU uplift.
|
|
"""
|
|
deductions = (p.income_tax + p.national_insurance + p.pension_employee + p.student_loan +
|
|
p.rsu_offset + sum(p.other_deductions.values(), start=Decimal("0")))
|
|
diff = abs(p.gross_pay - deductions - p.net_pay)
|
|
return diff < TOTALS_TOLERANCE
|