payslip-ingest/tests/test_meta_uk_parser.py
Viktor Barzin 974181674d v2: regex parser for Meta UK template + accurate RSU tax attribution
## Context

v1 shipped a Claude Haiku-based extractor that validated only 10/71
backfilled rows. Haiku fumbles the arithmetic on pension salary-sacrifice,
conflates RSU vest with regular earnings, and occasionally misreads YTD
vs this-period columns — so 86% of rows land with validated=false and the
downstream dashboards under-report take-home.

Meta UK uses a stable two-variant template (pre/post 2022-01-31 boundary),
so a regex parser is both faster (ms vs. 30-90s + $0.01-0.05/call) and
more accurate. v2 introduces that parser as the primary path, keeps
Claude as the fallback for non-Meta payslips, and surfaces new fields
the dashboard needs to attribute PAYE between cash salary and RSU vests
correctly.

## This change

### Parser (new)

`payslip_ingest/parsers/meta_uk.py` detects the layout variant by header
presence:

- **Variant A** (pre-2022): vertical Description/This Period/This Year.
  `AE Pension EE` is a positive deduction against a pre-sacrifice gross —
  maps to `pension_employee` for the existing validation formula to hold.
- **Variant B** (post-2022): side-by-side Payments | Deductions | Year to
  Date. `AE Pension EE` is NEGATIVE in Payments (salary sacrifice) — maps
  to `pension_sacrifice` and is already netted into Total Payment.
  `rsu_vest = RSU Tax Offset + RSU Excs Refund` (Meta's template inflates
  Taxable Pay without using a matching offset deduction).

Column boundaries come from the header row's anchor positions; each data
row slices into 3 cells and the last numeric token per cell is the amount.
Anchor misses raise ParserError so the caller falls back to Claude rather
than silently returning bad data.

### New fields

Schema + DB + Claude prompt gain:

- `salary`, `bonus`, `pension_sacrifice` — earnings decomposition for the
  dashboard's bonus-sacrifice visibility and earnings-breakdown chart
- `taxable_pay`, `ytd_tax_paid`, `ytd_taxable_pay`, `ytd_gross` — powers
  the YTD-effective-rate method of attributing cash tax vs RSU tax, which
  is the only method that's accurate month-to-month

All new columns default to 0 / null so v1 rows continue to round-trip.

### Orchestration

processor.py tries `parse_meta_uk(pdftotext(pdf))` first. On success the
result goes straight to the DB — zero Claude tokens spent, extraction in
milliseconds. On ParserError it falls through to ClaudeExtractor as before.
ProcessResult gains an `extractor` field ("meta_uk_regex" | "claude") so
backfill logs show the hit rate.

## Tests

- `test_meta_uk_parser.py` — 11 tests covering variant A, variant B
  (standard + bonus month + bonus-sacrificed month), malformed inputs,
  and end-to-end totals validation for all 4 golden fixtures.
- `test_processor.py` — 2 new tests proving the regex-first short-circuit
  and the Claude fallback on non-Meta inputs.

Fixtures under `tests/fixtures/` are hand-crafted `pdftotext -layout`
emulations — real Meta numbers from the plan's sample payslips for
variant B, synthesized realistic variant A and bonus-sacrificed samples.

0001_initial.py reformat is yapf cleanup touched during the session's
format pass; not a behavior change.

## Test Plan

### Automated

```
$ poetry run pytest
============================= test session starts ==============================
collected 53 items

tests/test_extractor.py .....                                            [  9%]
tests/test_meta_uk_parser.py ...........                                 [ 30%]
tests/test_paperless.py ......                                           [ 41%]
tests/test_processor.py ..............                                   [ 67%]
tests/test_schema.py ....                                                [ 75%]
tests/test_tax_year.py ........                                          [ 90%]
tests/test_webhook.py .....                                              [100%]
============================== 53 passed in 1.66s ==============================

$ poetry run ruff check .
All checks passed!

$ poetry run mypy .
Success: no issues found in 24 source files

$ poetry run yapf --style pyproject.toml --diff --recursive payslip_ingest tests
(no output — all files are yapf-clean)
```

### Manual Verification

Smoke-test the parser against a real Meta payslip PDF on the deploy host:

```
# After 0003 migration applied to prod DB
$ poetry run python -c "
from payslip_ingest.parsers import parse_meta_uk
import subprocess
text = subprocess.check_output(['pdftotext', '-layout', '/path/to/real.pdf', '-']).decode()
p = parse_meta_uk(text)
print(p.model_dump_json(indent=2))
"
```

Expected: JSON with salary/bonus/rsu_vest/pension_sacrifice populated and
`validate_totals(p)` returning True.

## Reproduce locally

1. `cd payslip-ingest && poetry install`
2. `poetry run pytest tests/test_meta_uk_parser.py -v`
3. Expected: 11 tests pass, each fixture validates totals within 2p.

Closes: code-un1

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 10:53:52 +00:00

146 lines
5.7 KiB
Python

from datetime import date
from decimal import Decimal
from pathlib import Path
import pytest
from payslip_ingest.parsers.meta_uk import ParserError, parse_meta_uk
FIXTURES = Path(__file__).parent / "fixtures"
def _load(name: str) -> str:
return (FIXTURES / name).read_text(encoding="utf-8")
def test_parses_variant_b_standard_month() -> None:
"""Feb 2026 — variant B, RSU vesting, no bonus, salary-sacrifice pension."""
result = parse_meta_uk(_load("meta_uk_2026_02.txt"))
assert result.pay_date == date(2026, 2, 27)
assert result.pay_period_start == date(2026, 2, 1)
assert result.pay_period_end == date(2026, 2, 27)
assert result.employer == "Facebook UK Limited"
assert result.currency == "GBP"
assert result.salary == Decimal("10003.33")
assert result.bonus == Decimal("0")
assert result.pension_sacrifice == Decimal("600.20")
# rsu_vest = RSU Tax Offset + RSU Excs Refund
assert result.rsu_vest == Decimal("30479.76")
assert result.rsu_offset == Decimal("0")
assert result.gross_pay == Decimal("39882.89")
assert result.income_tax == Decimal("31311.90")
assert result.national_insurance == Decimal("1602.89")
assert result.pension_employee == Decimal("0")
assert result.student_loan == Decimal("0")
assert result.net_pay == Decimal("6968.10")
assert result.taxable_pay == Decimal("72096.92")
assert result.ytd_tax_paid == Decimal("155626.37")
assert result.ytd_taxable_pay == Decimal("373601.64")
assert result.ytd_gross == Decimal("232630.34")
def test_parses_variant_b_with_bonus_and_rsu() -> None:
"""March 2025 — variant B, bonus month, RSU vesting, multiple other deductions."""
result = parse_meta_uk(_load("meta_uk_2025_03.txt"))
assert result.pay_date == date(2025, 3, 27)
assert result.salary == Decimal("10000.00")
assert result.bonus == Decimal("25000.00")
assert result.pension_sacrifice == Decimal("1200.00")
assert result.rsu_vest == Decimal("20000.00")
assert result.gross_pay == Decimal("53720.00")
assert result.income_tax == Decimal("45210.44")
assert result.national_insurance == Decimal("2750.12")
assert result.student_loan == Decimal("850.00")
assert result.net_pay == Decimal("4753.69")
# Private Medical comes from the Deductions column. Cycle To Work is a
# negative Payments line — already subtracted from Total Payment, so it
# does NOT belong in other_deductions (that would double-count).
assert "Private Medical" in result.other_deductions
assert result.other_deductions["Private Medical"] == Decimal("155.75")
assert "Cycle To Work" not in result.other_deductions
def test_parses_variant_b_bonus_sacrificed() -> None:
"""March 2024 — variant B, full bonus sacrificed into pension, bonus line = 0."""
result = parse_meta_uk(_load("meta_uk_2024_03_bonus_sacrificed.txt"))
assert result.pay_date == date(2024, 3, 27)
assert result.salary == Decimal("9500.00")
# Bonus line present but zero — parser should surface this so the dashboard
# can highlight the "bonus sacrificed" dip.
assert result.bonus == Decimal("0")
# Big pension sacrifice dwarfs the salary — this is the signal we care about.
assert result.pension_sacrifice == Decimal("6200.00")
assert result.rsu_vest == Decimal("0")
assert result.gross_pay == Decimal("3300.00")
assert result.net_pay == Decimal("2130.00")
def test_parses_variant_a_pre_2022() -> None:
"""July 2019 — variant A, pre-RSU, single-column layout.
Variant A lists AE Pension EE as a positive deduction (pre-sacrifice gross),
so it maps to `pension_employee` for the standard validation formula to hold.
Variant B lists it as a negative payment (post-sacrifice gross) and maps to
`pension_sacrifice` instead. Both represent money going into the pension.
"""
result = parse_meta_uk(_load("meta_uk_2019_07.txt"))
assert result.pay_date == date(2019, 7, 31)
assert result.employer == "Facebook UK Limited"
assert result.salary == Decimal("7083.33")
assert result.bonus == Decimal("0")
assert result.rsu_vest == Decimal("0")
assert result.pension_sacrifice == Decimal("0")
assert result.pension_employee == Decimal("500.00")
assert result.gross_pay == Decimal("7083.33")
assert result.income_tax == Decimal("1480.00")
assert result.national_insurance == Decimal("564.73")
assert result.student_loan == Decimal("120.00")
assert result.net_pay == Decimal("4418.60")
# Variant A carries a "Taxable Pay" line inline
assert result.taxable_pay == Decimal("6583.33")
def test_raises_on_non_meta_payslip() -> None:
with pytest.raises(ParserError):
parse_meta_uk("This is not a Meta payslip\nRandom text\n")
def test_raises_on_empty_text() -> None:
with pytest.raises(ParserError):
parse_meta_uk("")
def test_raises_when_pay_date_missing() -> None:
broken = "Facebook UK Limited\nPayslip\nSalary 1000.00\nNet Pay: 800.00\n"
with pytest.raises(ParserError):
parse_meta_uk(broken)
@pytest.mark.parametrize("fixture_name", [
"meta_uk_2026_02.txt",
"meta_uk_2025_03.txt",
"meta_uk_2024_03_bonus_sacrificed.txt",
"meta_uk_2019_07.txt",
])
def test_all_fixtures_validate_totals(fixture_name: str) -> None:
"""Every fixture must satisfy gross - deductions ≈ net within 2p."""
from payslip_ingest.schema import validate_totals
result = parse_meta_uk(_load(fixture_name))
assert validate_totals(result), (
f"{fixture_name}: gross={result.gross_pay} "
f"tax={result.income_tax} nic={result.national_insurance} "
f"student={result.student_loan} other={result.other_deductions} "
f"net={result.net_pay}")