processor + parser: fix 3 backfill failure modes

## Context

After the first v2 backfill (commit f62c533), 72 of 73 real payslips
landed correctly, but three residual failure modes remained:

1. **doc_id=215** — a 1442-byte empty-text PDF that Claude
   hallucinated a `pay_date=1900-01-01 / gross=0 / net=0` row for.
   Data poison waiting to happen.
2. **doc_id=39** — a P60 End of Year Certificate. Got tagged
   `payslip` in Paperless, has no Paperless title, so the title-based
   filter couldn't catch it; the regex parser then happily pulled
   bogus numbers out of the P60 layout.
3. **doc_id=49** — a real June 2021 variant-A payslip with an
   `EE Discount BIK` line in BOTH Payments and Deductions at 12.00.
   The parser was configured to drop `EE Discount BIK` from
   `other_deductions` (treating it as a known mapped field), which
   caused validate_totals to fail by exactly 12.00.

## This change

### processor.py — defence in depth

- **`NON_PAYSLIP_CONTENT_RE`** — new regex run against the first
  500 chars of pdftotext output. Catches `P60 End of Year
  Certificate` and `Take-home income per month` (Viktor's comp
  estimation spreadsheet). First-500-char scoping keeps it from
  false-positiving a legit payslip that mentions "P60" in a
  footer.
- **Post-extraction sanity checks** — reject a ProcessResult if
  `pay_date.year < 2010` (Viktor joined Meta in 2019) or if
  `gross_pay == net_pay == 0`. These raise rather than insert,
  so the backfill's existing `except Exception` block logs and
  continues without poisoning the DB. Supersedes the 1900-01-01
  case that would otherwise slip through.

### meta_uk.py — variant A BIK fix

Removed `EE Discount BIK` from `VARIANT_A_DEDUCTIONS_KNOWN`. That
set filters items OUT of `other_deductions` (because they have
dedicated schema fields). `EE Discount BIK` has no dedicated
field — it should stay in `other_deductions` like Private Dental
and Private Medical so the validation math balances.

### Fixtures + tests

- New fixture `meta_uk_2021_06_variant_a_bik.txt` — real
  pdftotext from doc_id=49 — encodes the BIK-in-both-columns
  case so a regression would fail this fixture's validation test.
- `test_parses_variant_a_with_ee_discount_bik` — explicitly
  asserts `EE Discount BIK` lands in `other_deductions`.
- `test_rejects_implausible_pay_date`, `test_rejects_zero_gross_zero_net`
  — cover the two sanity-check branches.
- `test_skips_p60_by_content_when_title_is_null` — covers the
  content-based non-payslip filter.

## Test Plan

### Automated

```
$ poetry run pytest
============================== 57 passed in 2.42s ==============================
$ poetry run ruff check .
All checks passed!
$ poetry run mypy .
Success: no issues found in 24 source files
```

### Manual verification (after deploy + re-run backfill)

Expected DB shape:
- Total rows ≈ 71 (88 paperless tags − 15 non-payslip titles −
  2 null-title non-payslips caught by content filter)
- `validated = true` on ≥99% of rows
- No `pay_date < 2010` rows
- No rows with employer IS NULL

## Reproduce locally

1. `cd payslip-ingest && poetry run pytest`
2. Expected: 57 passed, including the 3 new processor tests and
   the 5 parametrised fixture-total-validation tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-04-19 12:00:00 +00:00
parent f62c5332e3
commit d91f34ddb4
5 changed files with 141 additions and 1 deletions

View file

@ -202,3 +202,47 @@ async def test_regex_miss_falls_back_to_claude(paperless: AsyncMock, extractor:
assert result.status == "inserted"
assert result.extractor == "claude"
extractor.extract.assert_awaited_once()
async def test_rejects_implausible_pay_date(paperless: AsyncMock, extractor: AsyncMock) -> None:
"""Reject 1900-01-01-style hallucinations before they poison the DB."""
bad = _sample_extraction()
bad_dict = bad.model_dump()
bad_dict["pay_date"] = date(1900, 1, 1)
extractor.extract.return_value = ExtractedPayslip.model_validate(bad_dict)
factory = _SessionFactory([_FakeSession(existing_ids=[])])
with pytest.raises(ValueError, match="implausible pay_date"):
await process_document(42, factory, paperless, extractor)
async def test_skips_p60_by_content_when_title_is_null(paperless: AsyncMock, extractor: AsyncMock,
monkeypatch: pytest.MonkeyPatch) -> None:
"""P60s get the `payslip` tag sometimes, and some have no title in Paperless.
The title filter can't catch them, so we also check the pdftotext output
for the `P60 End of Year Certificate` signature before hitting the
extractor.
"""
paperless.get_document.return_value = {"id": 42, "title": None}
monkeypatch.setattr(processor, "_pdftotext",
lambda _: "P60 End of Year Certificate\nTax year to 5 April 2021\n")
factory = _SessionFactory([_FakeSession(existing_ids=[])])
result = await process_document(42, factory, paperless, extractor)
assert result.status == "skipped_non_payslip"
extractor.extract.assert_not_called()
async def test_rejects_zero_gross_zero_net(paperless: AsyncMock, extractor: AsyncMock) -> None:
"""Reject the other common hallucination: all zeros on a non-payslip."""
bad = _sample_extraction()
bad_dict = bad.model_dump()
bad_dict["gross_pay"] = Decimal("0")
bad_dict["net_pay"] = Decimal("0")
extractor.extract.return_value = ExtractedPayslip.model_validate(bad_dict)
factory = _SessionFactory([_FakeSession(existing_ids=[])])
with pytest.raises(ValueError, match="zero gross and net"):
await process_document(42, factory, paperless, extractor)