processor + parser: fix 3 backfill failure modes
## Context
After the first v2 backfill (commit f62c533), 72 of 73 real payslips
landed correctly, but three residual failure modes remained:
1. **doc_id=215** — a 1442-byte empty-text PDF that Claude
hallucinated a `pay_date=1900-01-01 / gross=0 / net=0` row for.
Data poison waiting to happen.
2. **doc_id=39** — a P60 End of Year Certificate. Got tagged
`payslip` in Paperless, has no Paperless title, so the title-based
filter couldn't catch it; the regex parser then happily pulled
bogus numbers out of the P60 layout.
3. **doc_id=49** — a real June 2021 variant-A payslip with an
`EE Discount BIK` line in BOTH Payments and Deductions at 12.00.
The parser was configured to drop `EE Discount BIK` from
`other_deductions` (treating it as a known mapped field), which
caused validate_totals to fail by exactly 12.00.
## This change
### processor.py — defence in depth
- **`NON_PAYSLIP_CONTENT_RE`** — new regex run against the first
500 chars of pdftotext output. Catches `P60 End of Year
Certificate` and `Take-home income per month` (Viktor's comp
estimation spreadsheet). First-500-char scoping keeps it from
false-positiving a legit payslip that mentions "P60" in a
footer.
- **Post-extraction sanity checks** — reject a ProcessResult if
`pay_date.year < 2010` (Viktor joined Meta in 2019) or if
`gross_pay == net_pay == 0`. These raise rather than insert,
so the backfill's existing `except Exception` block logs and
continues without poisoning the DB. Supersedes the 1900-01-01
case that would otherwise slip through.
### meta_uk.py — variant A BIK fix
Removed `EE Discount BIK` from `VARIANT_A_DEDUCTIONS_KNOWN`. That
set filters items OUT of `other_deductions` (because they have
dedicated schema fields). `EE Discount BIK` has no dedicated
field — it should stay in `other_deductions` like Private Dental
and Private Medical so the validation math balances.
### Fixtures + tests
- New fixture `meta_uk_2021_06_variant_a_bik.txt` — real
pdftotext from doc_id=49 — encodes the BIK-in-both-columns
case so a regression would fail this fixture's validation test.
- `test_parses_variant_a_with_ee_discount_bik` — explicitly
asserts `EE Discount BIK` lands in `other_deductions`.
- `test_rejects_implausible_pay_date`, `test_rejects_zero_gross_zero_net`
— cover the two sanity-check branches.
- `test_skips_p60_by_content_when_title_is_null` — covers the
content-based non-payslip filter.
## Test Plan
### Automated
```
$ poetry run pytest
============================== 57 passed in 2.42s ==============================
$ poetry run ruff check .
All checks passed!
$ poetry run mypy .
Success: no issues found in 24 source files
```
### Manual verification (after deploy + re-run backfill)
Expected DB shape:
- Total rows ≈ 71 (88 paperless tags − 15 non-payslip titles −
2 null-title non-payslips caught by content filter)
- `validated = true` on ≥99% of rows
- No `pay_date < 2010` rows
- No rows with employer IS NULL
## Reproduce locally
1. `cd payslip-ingest && poetry run pytest`
2. Expected: 57 passed, including the 3 new processor tests and
the 5 parametrised fixture-total-validation tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
f62c5332e3
commit
d91f34ddb4
5 changed files with 141 additions and 1 deletions
|
|
@ -86,6 +86,31 @@ def test_parses_variant_c_2022_11() -> None:
|
|||
assert result.ytd_gross == Decimal("131034.64")
|
||||
|
||||
|
||||
def test_parses_variant_a_with_ee_discount_bik() -> None:
|
||||
"""June 2021 — variant A. Real pdftotext from doc_id=49.
|
||||
|
||||
Has an `EE Discount BIK` line present in BOTH Payments AND Deductions
|
||||
blocks with value 12.00. Needs to land in `other_deductions` so the
|
||||
validation formula accounts for it (earlier parser version filtered
|
||||
it out, causing an off-by-12.00 validation failure).
|
||||
"""
|
||||
result = parse_meta_uk(_load("meta_uk_2021_06_variant_a_bik.txt"))
|
||||
|
||||
assert result.pay_date == date(2021, 6, 30)
|
||||
assert result.salary == Decimal("5096.65")
|
||||
assert result.pension_sacrifice == Decimal("152.90")
|
||||
assert result.rsu_vest == Decimal("0") # RSU lines are YTD-only this period
|
||||
assert result.rsu_offset == Decimal("0")
|
||||
assert result.gross_pay == Decimal("5095.86")
|
||||
assert result.income_tax == Decimal("1410.07")
|
||||
assert result.national_insurance == Decimal("423.17")
|
||||
assert result.student_loan == Decimal("244.00")
|
||||
assert result.net_pay == Decimal("2906.51")
|
||||
assert result.other_deductions.get("Private Dental Insurance") == Decimal("15.61")
|
||||
assert result.other_deductions.get("Private Medical Insurance") == Decimal("84.50")
|
||||
assert result.other_deductions.get("EE Discount BIK") == Decimal("12.00")
|
||||
|
||||
|
||||
def test_parses_variant_a_2021_08() -> None:
|
||||
"""Aug 2021 — variant A. Real pdftotext from doc_id=43.
|
||||
|
||||
|
|
@ -135,6 +160,7 @@ def test_raises_on_empty_text() -> None:
|
|||
"meta_uk_2025_03.txt",
|
||||
"meta_uk_2022_11_variant_c.txt",
|
||||
"meta_uk_2021_08_variant_a.txt",
|
||||
"meta_uk_2021_06_variant_a_bik.txt",
|
||||
])
|
||||
def test_all_fixtures_validate_totals(fixture_name: str) -> None:
|
||||
"""Every fixture must satisfy gross - deductions ≈ net within 2p."""
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue