payslip-ingest/tests/test_meta_uk_parser.py
Viktor Barzin f62c5332e3 meta_uk parser: add variant A (2019-2022) + variant C (2022-2023)
## Context

The initial v2 parser (commit 9741816) only handled the modern template
(variant B, 2024+). Of Viktor's 73 real payslips in Paperless, 30 from
2021-07 through 2023-11 failed entirely — Claude fallback hit errors on
them and the rows never landed. Investigation via `kubectl exec` +
pdftotext on a sample of the failing docs revealed two previously-unseen
layouts that the parser needs to handle directly:

- **Variant A** (2019 → mid-2022): single-column Description/This Period/
  This Year. Parenthesized negatives `(152.90)`. Date format `Date : 31
  Aug 2021`. Employer is `Facebook UK Ltd` (not `Limited`). RSU lines:
  `RSU Gain Taxable` + `RSU Gain Nicable` + `RSU Net Cash UK` on the
  earnings side with a matching `RSU Net Gain` on the deductions side.
  BIK items (Private Dental/Medical) appear on both sides — net zero in
  the gross, but the deduction-side copy must land in other_deductions
  for the validation formula to hold.

- **Variant C** (late-2022 → 2023): side-by-side Payments|Deductions|
  Year To Date (note capital "To", vs variant B's lowercase "to"). Date
  format `Pay Date : 30.11.2022` (dots, not slashes). RSU labels use the
  abbreviated `RSU Gain Taxabl` / `Nicabl` and still include the `RSU
  Net Gain` offset. `Company Name : Facebook UK Limited` preamble.

Variant B (2024+) is unchanged.

## This change

### Parser refactor

- `EMPLOYER_RE = re.compile(r"Facebook UK (?:Limited|Ltd)\b")` — matches
  all three eras.
- `AMOUNT_RE` now accepts both `-1,234.56` and `(1,234.56)` — variant A's
  accounting-style parenthesized negatives normalize to `-1234.56` in
  `_to_decimal`.
- `_parse_date` tries three formats in order: slash (B), dot (C), word (A).
- `_is_variant_b_or_c` collapses B and C into one detector (both have the
  side-by-side header with `Year [Tt]o Date`); their parsers share code
  because the column mechanics are identical — only the RSU-label set and
  date format differ.
- `_parse_variant_a` is a full rewrite: single-column rows split by the
  two `Total ...` anchors (payments → deductions), pay_date from the
  header's `Date : ...`, gross from first Total, net from the trailing
  `Net Pay` line, taxable_pay from the `Taxable Pay : This Period £X`
  line at the bottom.
- RSU_VEST_LABELS is a shared set covering 8 aliases; rsu_vest sums every
  matching payment line. rsu_offset maps to `RSU Net Gain` on the
  deduction side when present (absent in variant B, present in A and C).

### Fixtures switched to real pdftotext output

Removed the two synthetic fixtures that no longer reflected real Meta
output (`meta_uk_2019_07.txt`, `meta_uk_2024_03_bonus_sacrificed.txt`)
and replaced with real pdftotext captures:

- `meta_uk_2021_08_variant_a.txt` (doc_id=43)
- `meta_uk_2022_11_variant_c.txt` (doc_id=53)

The remaining synthetic fixtures (`2025_03`, `2026_02`) stay because
they encode specific bonus/no-bonus scenarios and the numbers are
derived from the real Feb-2026 sample in the plan.

## Tests

- 10 parser tests: one per variant (A/B/C) + totals validation across
  all 4 fixtures + the existing non-Meta/empty-input guards. All pass.
- 52 total tests across the repo, all green.

## Test Plan

### Automated

```
$ poetry run pytest
============================== 52 passed in 1.66s ==============================
$ poetry run ruff check .
All checks passed!
$ poetry run mypy .
Success: no issues found in 24 source files
```

### Manual verification (after deploy)

1. TRUNCATE + re-run backfill — expect 73 real payslips to extract via
   regex (≥95% hit rate), 42 → 70+ validated rows.
2. Sample a row for each variant via psql: employer, rsu_vest, and
   taxable_pay should all be populated.

## Reproduce locally

1. `poetry run pytest tests/test_meta_uk_parser.py -v`
2. Expected: 10 passed, each fixture validates totals to within 2p.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 11:52:59 +00:00

148 lines
5.8 KiB
Python

from datetime import date
from decimal import Decimal
from pathlib import Path
import pytest
from payslip_ingest.parsers.meta_uk import ParserError, parse_meta_uk
FIXTURES = Path(__file__).parent / "fixtures"
def _load(name: str) -> str:
return (FIXTURES / name).read_text(encoding="utf-8")
def test_parses_variant_b_modern() -> None:
"""Feb 2026 — variant B (post-2024), RSU vest, salary-sacrifice pension."""
result = parse_meta_uk(_load("meta_uk_2026_02.txt"))
assert result.pay_date == date(2026, 2, 27)
assert result.pay_period_start == date(2026, 2, 1)
assert result.pay_period_end == date(2026, 2, 27)
assert result.employer == "Facebook UK Limited"
assert result.salary == Decimal("10003.33")
assert result.bonus == Decimal("0")
assert result.pension_sacrifice == Decimal("600.20")
assert result.rsu_vest == Decimal("30479.76") # RSU Tax Offset + RSU Excs Refund
assert result.rsu_offset == Decimal("0") # modern Meta template omits offset
assert result.gross_pay == Decimal("39882.89")
assert result.income_tax == Decimal("31311.90")
assert result.national_insurance == Decimal("1602.89")
assert result.net_pay == Decimal("6968.10")
assert result.taxable_pay == Decimal("72096.92")
assert result.ytd_tax_paid == Decimal("155626.37")
assert result.ytd_taxable_pay == Decimal("373601.64")
assert result.ytd_gross == Decimal("232630.34")
def test_parses_variant_b_with_bonus() -> None:
"""March 2025 — variant B, bonus + RSU + multiple other deductions."""
result = parse_meta_uk(_load("meta_uk_2025_03.txt"))
assert result.pay_date == date(2025, 3, 27)
assert result.salary == Decimal("10000.00")
assert result.bonus == Decimal("25000.00")
assert result.pension_sacrifice == Decimal("1200.00")
assert result.rsu_vest == Decimal("20000.00")
assert result.gross_pay == Decimal("53720.00")
assert result.net_pay == Decimal("4753.69")
assert "Private Medical" in result.other_deductions
def test_parses_variant_c_2022_11() -> None:
"""Nov 2022 — mid-era template. Real pdftotext from doc_id=53.
Side-by-side Payments | Deductions | Year To Date (capital "To"), dot-
separated date, RSU labels use `RSU Gain Taxabl` / `Nicabl` (abbreviated)
and a matching `RSU Net Gain` offset on the deductions side.
"""
result = parse_meta_uk(_load("meta_uk_2022_11_variant_c.txt"))
assert result.pay_date == date(2022, 11, 30)
assert result.employer == "Facebook UK Limited"
assert result.salary == Decimal("8983.33")
assert result.bonus == Decimal("0")
assert result.pension_sacrifice == Decimal("539.00")
# rsu_vest = RSU Gain Taxabl + RSU Gain Nicabl + RSU Net Cash
assert result.rsu_vest == Decimal("15192.00")
# rsu_offset = RSU Net Gain (the matching deduction)
assert result.rsu_offset == Decimal("11522.91")
assert result.gross_pay == Decimal("23636.33")
assert result.income_tax == Decimal("5800.07")
assert result.national_insurance == Decimal("612.65")
assert result.student_loan == Decimal("1233.00")
assert result.net_pay == Decimal("4467.70")
assert result.taxable_pay == Decimal("16070.14")
assert result.ytd_tax_paid == Decimal("34886.93")
assert result.ytd_taxable_pay == Decimal("99784.08")
assert result.ytd_gross == Decimal("131034.64")
def test_parses_variant_a_2021_08() -> None:
"""Aug 2021 — variant A. Real pdftotext from doc_id=43.
Single-column Description | This Period | This Year layout. Parenthesized
negatives `(152.90)`, Facebook UK Ltd (not Limited), date `Date : 31 Aug
2021`. BIK items (Dental/Medical) appear as both earnings and deductions.
"""
result = parse_meta_uk(_load("meta_uk_2021_08_variant_a.txt"))
assert result.pay_date == date(2021, 8, 31)
assert result.employer == "Facebook UK Ltd"
assert result.salary == Decimal("5096.65")
assert result.bonus == Decimal("0")
assert result.pension_sacrifice == Decimal("152.90")
# rsu_vest = RSU Gain Taxable + RSU Gain Nicable + RSU Net Cash UK
assert result.rsu_vest == Decimal("20654.51")
assert result.rsu_offset == Decimal("15666.13")
assert result.gross_pay == Decimal("25738.37")
assert result.income_tax == Decimal("5500.87")
assert result.national_insurance == Decimal("627.72")
assert result.student_loan == Decimal("1165.00")
assert result.net_pay == Decimal("2678.54")
# BIK offsets on the deductions side
assert result.other_deductions.get("Private Dental Insurance") == Decimal("15.61")
assert result.other_deductions.get("Private Medical Insurance") == Decimal("84.50")
# Variant A surfaces Taxable Pay via a trailing line `Taxable Pay : This
# Period £XXXX.XX : To Date £YYYY.YY`.
assert result.taxable_pay == Decimal("15323.16")
def test_raises_on_non_meta_payslip() -> None:
with pytest.raises(ParserError):
parse_meta_uk("This is not a Meta payslip\nRandom text\n")
def test_raises_on_empty_text() -> None:
with pytest.raises(ParserError):
parse_meta_uk("")
@pytest.mark.parametrize("fixture_name", [
"meta_uk_2026_02.txt",
"meta_uk_2025_03.txt",
"meta_uk_2022_11_variant_c.txt",
"meta_uk_2021_08_variant_a.txt",
])
def test_all_fixtures_validate_totals(fixture_name: str) -> None:
"""Every fixture must satisfy gross - deductions ≈ net within 2p."""
from payslip_ingest.schema import validate_totals
result = parse_meta_uk(_load(fixture_name))
assert validate_totals(result), (
f"{fixture_name}: gross={result.gross_pay} tax={result.income_tax} "
f"nic={result.national_insurance} student={result.student_loan} "
f"pension_employee={result.pension_employee} rsu_offset={result.rsu_offset} "
f"other={result.other_deductions} net={result.net_pay}")