payslip-ingest/tests/test_meta_uk_parser.py
Viktor Barzin d91f34ddb4 processor + parser: fix 3 backfill failure modes
## Context

After the first v2 backfill (commit f62c533), 72 of 73 real payslips
landed correctly, but three residual failure modes remained:

1. **doc_id=215** — a 1442-byte empty-text PDF that Claude
   hallucinated a `pay_date=1900-01-01 / gross=0 / net=0` row for.
   Data poison waiting to happen.
2. **doc_id=39** — a P60 End of Year Certificate. Got tagged
   `payslip` in Paperless, has no Paperless title, so the title-based
   filter couldn't catch it; the regex parser then happily pulled
   bogus numbers out of the P60 layout.
3. **doc_id=49** — a real June 2021 variant-A payslip with an
   `EE Discount BIK` line in BOTH Payments and Deductions at 12.00.
   The parser was configured to drop `EE Discount BIK` from
   `other_deductions` (treating it as a known mapped field), which
   caused validate_totals to fail by exactly 12.00.

## This change

### processor.py — defence in depth

- **`NON_PAYSLIP_CONTENT_RE`** — new regex run against the first
  500 chars of pdftotext output. Catches `P60 End of Year
  Certificate` and `Take-home income per month` (Viktor's comp
  estimation spreadsheet). First-500-char scoping keeps it from
  false-positiving a legit payslip that mentions "P60" in a
  footer.
- **Post-extraction sanity checks** — reject a ProcessResult if
  `pay_date.year < 2010` (Viktor joined Meta in 2019) or if
  `gross_pay == net_pay == 0`. These raise rather than insert,
  so the backfill's existing `except Exception` block logs and
  continues without poisoning the DB. Supersedes the 1900-01-01
  case that would otherwise slip through.

### meta_uk.py — variant A BIK fix

Removed `EE Discount BIK` from `VARIANT_A_DEDUCTIONS_KNOWN`. That
set filters items OUT of `other_deductions` (because they have
dedicated schema fields). `EE Discount BIK` has no dedicated
field — it should stay in `other_deductions` like Private Dental
and Private Medical so the validation math balances.

### Fixtures + tests

- New fixture `meta_uk_2021_06_variant_a_bik.txt` — real
  pdftotext from doc_id=49 — encodes the BIK-in-both-columns
  case so a regression would fail this fixture's validation test.
- `test_parses_variant_a_with_ee_discount_bik` — explicitly
  asserts `EE Discount BIK` lands in `other_deductions`.
- `test_rejects_implausible_pay_date`, `test_rejects_zero_gross_zero_net`
  — cover the two sanity-check branches.
- `test_skips_p60_by_content_when_title_is_null` — covers the
  content-based non-payslip filter.

## Test Plan

### Automated

```
$ poetry run pytest
============================== 57 passed in 2.42s ==============================
$ poetry run ruff check .
All checks passed!
$ poetry run mypy .
Success: no issues found in 24 source files
```

### Manual verification (after deploy + re-run backfill)

Expected DB shape:
- Total rows ≈ 71 (88 paperless tags − 15 non-payslip titles −
  2 null-title non-payslips caught by content filter)
- `validated = true` on ≥99% of rows
- No `pay_date < 2010` rows
- No rows with employer IS NULL

## Reproduce locally

1. `cd payslip-ingest && poetry run pytest`
2. Expected: 57 passed, including the 3 new processor tests and
   the 5 parametrised fixture-total-validation tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 12:00:00 +00:00

174 lines
7 KiB
Python

from datetime import date
from decimal import Decimal
from pathlib import Path
import pytest
from payslip_ingest.parsers.meta_uk import ParserError, parse_meta_uk
FIXTURES = Path(__file__).parent / "fixtures"
def _load(name: str) -> str:
return (FIXTURES / name).read_text(encoding="utf-8")
def test_parses_variant_b_modern() -> None:
"""Feb 2026 — variant B (post-2024), RSU vest, salary-sacrifice pension."""
result = parse_meta_uk(_load("meta_uk_2026_02.txt"))
assert result.pay_date == date(2026, 2, 27)
assert result.pay_period_start == date(2026, 2, 1)
assert result.pay_period_end == date(2026, 2, 27)
assert result.employer == "Facebook UK Limited"
assert result.salary == Decimal("10003.33")
assert result.bonus == Decimal("0")
assert result.pension_sacrifice == Decimal("600.20")
assert result.rsu_vest == Decimal("30479.76") # RSU Tax Offset + RSU Excs Refund
assert result.rsu_offset == Decimal("0") # modern Meta template omits offset
assert result.gross_pay == Decimal("39882.89")
assert result.income_tax == Decimal("31311.90")
assert result.national_insurance == Decimal("1602.89")
assert result.net_pay == Decimal("6968.10")
assert result.taxable_pay == Decimal("72096.92")
assert result.ytd_tax_paid == Decimal("155626.37")
assert result.ytd_taxable_pay == Decimal("373601.64")
assert result.ytd_gross == Decimal("232630.34")
def test_parses_variant_b_with_bonus() -> None:
"""March 2025 — variant B, bonus + RSU + multiple other deductions."""
result = parse_meta_uk(_load("meta_uk_2025_03.txt"))
assert result.pay_date == date(2025, 3, 27)
assert result.salary == Decimal("10000.00")
assert result.bonus == Decimal("25000.00")
assert result.pension_sacrifice == Decimal("1200.00")
assert result.rsu_vest == Decimal("20000.00")
assert result.gross_pay == Decimal("53720.00")
assert result.net_pay == Decimal("4753.69")
assert "Private Medical" in result.other_deductions
def test_parses_variant_c_2022_11() -> None:
"""Nov 2022 — mid-era template. Real pdftotext from doc_id=53.
Side-by-side Payments | Deductions | Year To Date (capital "To"), dot-
separated date, RSU labels use `RSU Gain Taxabl` / `Nicabl` (abbreviated)
and a matching `RSU Net Gain` offset on the deductions side.
"""
result = parse_meta_uk(_load("meta_uk_2022_11_variant_c.txt"))
assert result.pay_date == date(2022, 11, 30)
assert result.employer == "Facebook UK Limited"
assert result.salary == Decimal("8983.33")
assert result.bonus == Decimal("0")
assert result.pension_sacrifice == Decimal("539.00")
# rsu_vest = RSU Gain Taxabl + RSU Gain Nicabl + RSU Net Cash
assert result.rsu_vest == Decimal("15192.00")
# rsu_offset = RSU Net Gain (the matching deduction)
assert result.rsu_offset == Decimal("11522.91")
assert result.gross_pay == Decimal("23636.33")
assert result.income_tax == Decimal("5800.07")
assert result.national_insurance == Decimal("612.65")
assert result.student_loan == Decimal("1233.00")
assert result.net_pay == Decimal("4467.70")
assert result.taxable_pay == Decimal("16070.14")
assert result.ytd_tax_paid == Decimal("34886.93")
assert result.ytd_taxable_pay == Decimal("99784.08")
assert result.ytd_gross == Decimal("131034.64")
def test_parses_variant_a_with_ee_discount_bik() -> None:
"""June 2021 — variant A. Real pdftotext from doc_id=49.
Has an `EE Discount BIK` line present in BOTH Payments AND Deductions
blocks with value 12.00. Needs to land in `other_deductions` so the
validation formula accounts for it (earlier parser version filtered
it out, causing an off-by-12.00 validation failure).
"""
result = parse_meta_uk(_load("meta_uk_2021_06_variant_a_bik.txt"))
assert result.pay_date == date(2021, 6, 30)
assert result.salary == Decimal("5096.65")
assert result.pension_sacrifice == Decimal("152.90")
assert result.rsu_vest == Decimal("0") # RSU lines are YTD-only this period
assert result.rsu_offset == Decimal("0")
assert result.gross_pay == Decimal("5095.86")
assert result.income_tax == Decimal("1410.07")
assert result.national_insurance == Decimal("423.17")
assert result.student_loan == Decimal("244.00")
assert result.net_pay == Decimal("2906.51")
assert result.other_deductions.get("Private Dental Insurance") == Decimal("15.61")
assert result.other_deductions.get("Private Medical Insurance") == Decimal("84.50")
assert result.other_deductions.get("EE Discount BIK") == Decimal("12.00")
def test_parses_variant_a_2021_08() -> None:
"""Aug 2021 — variant A. Real pdftotext from doc_id=43.
Single-column Description | This Period | This Year layout. Parenthesized
negatives `(152.90)`, Facebook UK Ltd (not Limited), date `Date : 31 Aug
2021`. BIK items (Dental/Medical) appear as both earnings and deductions.
"""
result = parse_meta_uk(_load("meta_uk_2021_08_variant_a.txt"))
assert result.pay_date == date(2021, 8, 31)
assert result.employer == "Facebook UK Ltd"
assert result.salary == Decimal("5096.65")
assert result.bonus == Decimal("0")
assert result.pension_sacrifice == Decimal("152.90")
# rsu_vest = RSU Gain Taxable + RSU Gain Nicable + RSU Net Cash UK
assert result.rsu_vest == Decimal("20654.51")
assert result.rsu_offset == Decimal("15666.13")
assert result.gross_pay == Decimal("25738.37")
assert result.income_tax == Decimal("5500.87")
assert result.national_insurance == Decimal("627.72")
assert result.student_loan == Decimal("1165.00")
assert result.net_pay == Decimal("2678.54")
# BIK offsets on the deductions side
assert result.other_deductions.get("Private Dental Insurance") == Decimal("15.61")
assert result.other_deductions.get("Private Medical Insurance") == Decimal("84.50")
# Variant A surfaces Taxable Pay via a trailing line `Taxable Pay : This
# Period £XXXX.XX : To Date £YYYY.YY`.
assert result.taxable_pay == Decimal("15323.16")
def test_raises_on_non_meta_payslip() -> None:
with pytest.raises(ParserError):
parse_meta_uk("This is not a Meta payslip\nRandom text\n")
def test_raises_on_empty_text() -> None:
with pytest.raises(ParserError):
parse_meta_uk("")
@pytest.mark.parametrize("fixture_name", [
"meta_uk_2026_02.txt",
"meta_uk_2025_03.txt",
"meta_uk_2022_11_variant_c.txt",
"meta_uk_2021_08_variant_a.txt",
"meta_uk_2021_06_variant_a_bik.txt",
])
def test_all_fixtures_validate_totals(fixture_name: str) -> None:
"""Every fixture must satisfy gross - deductions ≈ net within 2p."""
from payslip_ingest.schema import validate_totals
result = parse_meta_uk(_load(fixture_name))
assert validate_totals(result), (
f"{fixture_name}: gross={result.gross_pay} tax={result.income_tax} "
f"nic={result.national_insurance} student={result.student_loan} "
f"pension_employee={result.pension_employee} rsu_offset={result.rsu_offset} "
f"other={result.other_deductions} net={result.net_pay}")