Add CSV attachment fallback for InvestEngine email parser
Context: IE has not (yet) sent CSV-attached statements in production,
but the upstream parser had _extract_positions_csv as a third fallback
for exactly this case. Keeping the fallback preserves behaviour-parity
with the legacy parser and makes future statement support one fixture
away — the shape is documented by column set, not scraped live.
Unlike the upstream which split the body on whitespace and broke on any
embedded commas in names, this port walks real MIME attachments using
Python's csv.DictReader. A part qualifies as CSV if:
- its Content-Type is text/csv / application/csv / application/vnd.ms-excel, OR
- its filename ends in .csv (defence against IE mis-labelling the part)
Rows missing required columns or containing unparseable numbers/dates
are skipped silently — consistent with the "partial match" contract:
a half-corrupt CSV yields whatever rows were intact. Required columns:
ticker, unit_price, quantity, date (YYYY-MM-DD), currency. Non-GBP
rows are filtered because the IE ISA is strictly sterling — flagging
this assumption in the review notes.
This change:
- Adds `_parse_csv_attachment(raw_email)` as the third strategy after
text/plain and text/html; it re-parses the raw email bytes so we can
inspect Content-Type/filename on each part.
- Flags symbols/currencies, filters non-GBP, and runs each row through
the shared `_build_activity` so external_id formation matches every
other strategy (dedup stays consistent across strategies).
- Fixture `csv_attachment.eml` has three rows (VUAG, SWDA, VUSA) in a
`text/csv` part with a `.csv` filename — covers both detection paths.
Test plan:
poetry run pytest tests/providers/parsers/ -q → 6 passed in 0.15s
poetry run mypy broker_sync/providers/parsers/invest_engine.py tests/providers/parsers/test_invest_engine.py → clean
poetry run ruff check broker_sync/providers/parsers/invest_engine.py tests/providers/parsers/test_invest_engine.py → All checks passed!
poetry run yapf --diff → clean (no diff)
Manual verification: load csv_attachment.eml, call parse_invest_engine_email,
assert 3 activities each with symbol in {VUAG,SWDA,VUSA}, currency=GBP,
notes containing "csv".
This commit is contained in:
parent
72d348e294
commit
020ba16723
3 changed files with 123 additions and 3 deletions
22
tests/fixtures/invest_engine/csv_attachment.eml
vendored
Normal file
22
tests/fixtures/invest_engine/csv_attachment.eml
vendored
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
From: InvestEngine <no-reply@investengine.com>
|
||||
To: viktorbarzin@example.com
|
||||
Subject: Your InvestEngine statement
|
||||
Date: Mon, 07 Apr 2025 09:00:00 +0000
|
||||
MIME-Version: 1.0
|
||||
Content-Type: multipart/mixed; boundary="----=_MIXED_1"
|
||||
|
||||
------=_MIXED_1
|
||||
Content-Type: text/plain; charset=UTF-8
|
||||
|
||||
Your monthly statement is attached as a CSV.
|
||||
|
||||
------=_MIXED_1
|
||||
Content-Type: text/csv; charset=UTF-8; name="statement.csv"
|
||||
Content-Disposition: attachment; filename="statement.csv"
|
||||
|
||||
ticker,unit_price,quantity,date,currency
|
||||
VUAG,63.21,12.5,2025-04-02,GBP
|
||||
SWDA,86.40,4.75,2025-04-03,GBP
|
||||
VUSA,90.10,1.0,2025-04-04,GBP
|
||||
|
||||
------=_MIXED_1--
|
||||
|
|
@ -68,3 +68,24 @@ def test_html_notes_record_html_strategy() -> None:
|
|||
a = parse_invest_engine_email(_load("html_two_orders.eml"))[0]
|
||||
assert a.notes is not None
|
||||
assert "html" in a.notes
|
||||
|
||||
|
||||
# -- CSV attachment body --
|
||||
|
||||
|
||||
def test_csv_attachment_parses_all_rows() -> None:
|
||||
activities = parse_invest_engine_email(_load("csv_attachment.eml"))
|
||||
assert len(activities) == 3
|
||||
by_symbol = {a.symbol: a for a in activities}
|
||||
assert by_symbol["VUAG"].quantity == Decimal("12.5")
|
||||
assert by_symbol["VUAG"].unit_price == Decimal("63.21")
|
||||
assert by_symbol["VUAG"].date == datetime(2025, 4, 2)
|
||||
assert by_symbol["SWDA"].quantity == Decimal("4.75")
|
||||
assert by_symbol["VUSA"].date == datetime(2025, 4, 4)
|
||||
for a in activities:
|
||||
assert a.activity_type is ActivityType.BUY
|
||||
assert a.currency == "GBP"
|
||||
assert a.account_id == "invest-engine-primary"
|
||||
assert a.account_type is AccountType.ISA
|
||||
assert a.notes is not None
|
||||
assert "csv" in a.notes
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue