imap: route IE BUYs to ISA first-£20k / GIA overflow per UK tax year
## Context Viktor's InvestEngine account has both an ISA and a GIA wrapper. Trade confirmation emails (info@investengine.com) are identical between them — subject "Here's how your portfolio looks now", body shows "Client name: Viktor Barzin" with no portfolio/account type. That left the IMAP parser hardcoded to route every IE BUY to the ISA (invest-engine-primary), which produced a 2339-share over-count when 2023-24 GIA buys landed in the ISA during the 2026-04-18 reconciliation. Viktor's rule: from 6 April each tax year, BUYs fill ISA up to the £20,000 cap, then overflow to GIA. This commit codifies that rule in a standalone batch splitter and applies it at the ImapProvider boundary. Also picks up a silent-drop bug surfaced during the same reconciliation: WF's /import (unlike /import/check) rejects naive datetimes with "Invalid date". The sink now coerces tzinfo=UTC defensively so every provider gets the same guarantee. ## This change - `_split_ie_by_isa_cap(activities)` — sorts all IE-ISA BUYs by date and walks them once per UK tax year (6 April boundary). A BUY whose running tax-year total BEFORE it is strictly below £20k stays on the ISA; otherwise it flips to a new `invest-engine-gia` account_id. No fractional splits — boundary activities go whole to whichever bucket their pre-running-total dictates. Non-IE and non-BUY activities pass through unchanged. - `ImapProvider.accounts()` gains an `invest-engine-gia` Account so the pipeline's `_ensure_accounts` can resolve both. - `ImapProvider.fetch()` calls the splitter on the full batch before applying the `since`/`before` date filter — batch-level sort guarantees consistent routing regardless of the order IMAP returns messages. - `WealthfolioSink._activity_to_import_row` coerces naive datetimes to UTC so the row passes WF /import validation. ## What is NOT in this change - No retroactive re-routing of data already in WF. Historical finance-mysql rows (all lumped to `invest-engine-primary` or `invest-engine-gia` by the existing heuristic) keep their current account assignment. If a past tax-year was routed "wrong" under the new rule, that's corrected manually via the WF API, not here. - No change to the Schwab or trading212 paths. ## Verification ### Automated \`\`\` $ poetry run pytest tests/providers/test_imap.py -v tests/providers/test_imap.py::test_uk_tax_year_start_before_april_6_rolls_back PASSED tests/providers/test_imap.py::test_single_tax_year_under_cap_stays_isa PASSED tests/providers/test_imap.py::test_overflow_past_cap_flips_to_gia PASSED tests/providers/test_imap.py::test_tax_year_boundary_resets_cap PASSED tests/providers/test_imap.py::test_out_of_order_activities_sorted_before_cap_applied PASSED tests/providers/test_imap.py::test_non_ie_activities_passed_through_unchanged PASSED 6 passed in 0.36s $ poetry run pytest -q --ignore=tests/test_cli.py 116 passed, 1 skipped in 2.76s $ poetry run ruff check broker_sync/providers/imap.py broker_sync/sinks/wealthfolio.py All checks passed! $ poetry run mypy broker_sync/providers/imap.py broker_sync/sinks/wealthfolio.py Success: no issues found in 2 source files \`\`\` ### Manual verification The tzinfo fix was validated against the live WF instance during the 2026-04-18 reconciliation — before the fix, /import returned \`"errors": {"symbol": ["Invalid date '2022-05-24T00:00:00'."]}\` for every IMAP activity; after, the same payload imported cleanly. The splitter was not exercised against live IMAP data because Viktor's mailbox only has Apr 2022 → Feb 2024 emails, all inside finance.position's existing coverage. Running IMAP ingest with \`since=2024-04-06\` yields fetched=0. The unit tests cover the boundary arithmetic; a live run will happen when newer emails are parsed (or when finance coverage is re-scoped). ## Reproduce locally 1. \`poetry install\` 2. \`poetry run pytest tests/providers/test_imap.py\` 3. Expected: 6 passed, 0 failed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
a190875f63
commit
c830856ba1
3 changed files with 173 additions and 5 deletions
|
|
@ -19,14 +19,66 @@ import logging
|
|||
import re
|
||||
import ssl
|
||||
from collections.abc import AsyncIterator, Iterator
|
||||
from datetime import datetime
|
||||
from datetime import date, datetime
|
||||
from decimal import Decimal
|
||||
from email.message import Message
|
||||
from typing import NamedTuple
|
||||
|
||||
from broker_sync.models import Account, AccountType, Activity
|
||||
from broker_sync.models import Account, AccountType, Activity, ActivityType
|
||||
from broker_sync.providers.parsers import invest_engine as ie_parser
|
||||
from broker_sync.providers.parsers.schwab import parse_schwab_email
|
||||
|
||||
_IE_ISA_ACCOUNT_ID = "invest-engine-primary"
|
||||
_IE_GIA_ACCOUNT_ID = "invest-engine-gia"
|
||||
_ISA_ANNUAL_CAP = Decimal("20000")
|
||||
_UK_TAX_YEAR_START = (4, 6) # (month, day) — UK tax year starts 6 April
|
||||
|
||||
|
||||
def _uk_tax_year_start(d: datetime) -> date:
|
||||
"""Return the start date (6 April of year N) of the UK tax year containing `d`."""
|
||||
month, day = _UK_TAX_YEAR_START
|
||||
cutoff = date(d.year, month, day)
|
||||
return cutoff if d.date() >= cutoff else date(d.year - 1, month, day)
|
||||
|
||||
|
||||
def _split_ie_by_isa_cap(
|
||||
activities: list[Activity],
|
||||
*,
|
||||
isa_cap: Decimal = _ISA_ANNUAL_CAP,
|
||||
) -> list[Activity]:
|
||||
"""Re-route IE BUYs: first `isa_cap` GBP of each UK tax year → ISA, rest → GIA.
|
||||
|
||||
Viktor's IE account has both an ISA and a GIA wrapper, and his trade
|
||||
confirmation emails don't indicate which one a given buy hit. Empirically,
|
||||
he fills the ISA allowance first each tax year (6 April) and any excess
|
||||
lands in GIA. This function partitions an already-parsed batch of Activity
|
||||
objects by that rule.
|
||||
|
||||
Rule for boundary buys: a BUY is assigned to ISA iff the running tax-year
|
||||
total BEFORE it is still strictly below the cap; otherwise GIA. Whole-
|
||||
activity assignment — no fractional splits.
|
||||
|
||||
Non-IE activities and non-BUYs are passed through unchanged.
|
||||
"""
|
||||
ie_buys = [
|
||||
a for a in activities
|
||||
if a.account_id == _IE_ISA_ACCOUNT_ID and a.activity_type is ActivityType.BUY
|
||||
]
|
||||
ie_buys.sort(key=lambda a: a.date)
|
||||
cumulative: dict[date, Decimal] = {}
|
||||
for a in ie_buys:
|
||||
ty = _uk_tax_year_start(a.date)
|
||||
running = cumulative.get(ty, Decimal(0))
|
||||
trade_value = (a.quantity or Decimal(0)) * (a.unit_price or Decimal(0))
|
||||
if running < isa_cap:
|
||||
a.account_id = _IE_ISA_ACCOUNT_ID
|
||||
a.account_type = AccountType.ISA
|
||||
else:
|
||||
a.account_id = _IE_GIA_ACCOUNT_ID
|
||||
a.account_type = AccountType.GIA
|
||||
cumulative[ty] = running + trade_value
|
||||
return activities
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_IE_SENDERS = {"noreply@investengine.com", "hello@investengine.com"}
|
||||
|
|
@ -142,12 +194,19 @@ class ImapProvider:
|
|||
def accounts(self) -> list[Account]:
|
||||
return [
|
||||
Account(
|
||||
id="invest-engine-primary",
|
||||
id=_IE_ISA_ACCOUNT_ID,
|
||||
name="InvestEngine ISA",
|
||||
account_type=AccountType.ISA,
|
||||
currency="GBP",
|
||||
provider="invest-engine",
|
||||
),
|
||||
Account(
|
||||
id=_IE_GIA_ACCOUNT_ID,
|
||||
name="InvestEngine GIA",
|
||||
account_type=AccountType.GIA,
|
||||
currency="GBP",
|
||||
provider="invest-engine",
|
||||
),
|
||||
Account(
|
||||
id="schwab-workplace",
|
||||
name="Schwab (US workplace)",
|
||||
|
|
@ -165,7 +224,12 @@ class ImapProvider:
|
|||
) -> AsyncIterator[Activity]:
|
||||
# IMAP doesn't give us a server-side date range directly without
|
||||
# constructing IMAP SEARCH criteria; filter client-side.
|
||||
for a in fetch_activities(self._creds):
|
||||
all_activities = fetch_activities(self._creds)
|
||||
# Apply ISA/GIA £20k-cap routing in one batch-level pass so each UK tax
|
||||
# year's cumulative total is computed consistently regardless of email
|
||||
# order on the server.
|
||||
routed = _split_ie_by_isa_cap(all_activities)
|
||||
for a in routed:
|
||||
if since is not None and a.date < since:
|
||||
continue
|
||||
if before is not None and a.date >= before:
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ from __future__ import annotations
|
|||
|
||||
import json
|
||||
from collections.abc import Iterable
|
||||
from datetime import UTC
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
|
@ -164,8 +165,11 @@ class WealthfolioSink:
|
|||
@staticmethod
|
||||
def _activity_to_import_row(a: Activity) -> dict[str, Any]:
|
||||
"""Match Wealthfolio's ActivityImport struct (camelCase JSON)."""
|
||||
# WF /import rejects naive datetimes with "Invalid date" (even though
|
||||
# /import/check accepts them) — coerce to UTC if tzinfo is missing.
|
||||
date = a.date if a.date.tzinfo is not None else a.date.replace(tzinfo=UTC)
|
||||
row: dict[str, Any] = {
|
||||
"date": a.date.isoformat(),
|
||||
"date": date.isoformat(),
|
||||
"symbol": a.symbol or "$CASH",
|
||||
"activityType": str(a.activity_type),
|
||||
"currency": a.currency,
|
||||
|
|
|
|||
100
tests/providers/test_imap.py
Normal file
100
tests/providers/test_imap.py
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, date, datetime
|
||||
from decimal import Decimal
|
||||
|
||||
from broker_sync.models import AccountType, Activity, ActivityType
|
||||
from broker_sync.providers.imap import (
|
||||
_IE_GIA_ACCOUNT_ID,
|
||||
_IE_ISA_ACCOUNT_ID,
|
||||
_split_ie_by_isa_cap,
|
||||
_uk_tax_year_start,
|
||||
)
|
||||
|
||||
|
||||
def _buy(on: datetime, qty: str, price: str) -> Activity:
|
||||
return Activity(
|
||||
external_id=f"invest-engine:{on.isoformat()}|{qty}|{price}",
|
||||
account_id=_IE_ISA_ACCOUNT_ID,
|
||||
account_type=AccountType.ISA,
|
||||
date=on,
|
||||
activity_type=ActivityType.BUY,
|
||||
currency="GBP",
|
||||
symbol="VUAG",
|
||||
quantity=Decimal(qty),
|
||||
unit_price=Decimal(price),
|
||||
)
|
||||
|
||||
|
||||
def test_uk_tax_year_start_before_april_6_rolls_back() -> None:
|
||||
assert _uk_tax_year_start(datetime(2025, 4, 5, tzinfo=UTC)) == date(2024, 4, 6)
|
||||
assert _uk_tax_year_start(datetime(2025, 4, 6, tzinfo=UTC)) == date(2025, 4, 6)
|
||||
assert _uk_tax_year_start(datetime(2025, 1, 15, tzinfo=UTC)) == date(2024, 4, 6)
|
||||
assert _uk_tax_year_start(datetime(2024, 4, 7, tzinfo=UTC)) == date(2024, 4, 6)
|
||||
|
||||
|
||||
def test_single_tax_year_under_cap_stays_isa() -> None:
|
||||
acts = [
|
||||
_buy(datetime(2024, 5, 1, tzinfo=UTC), "100", "50"), # £5000
|
||||
_buy(datetime(2024, 8, 1, tzinfo=UTC), "100", "80"), # £8000
|
||||
]
|
||||
routed = _split_ie_by_isa_cap(acts)
|
||||
assert all(a.account_id == _IE_ISA_ACCOUNT_ID for a in routed)
|
||||
assert all(a.account_type is AccountType.ISA for a in routed)
|
||||
|
||||
|
||||
def test_overflow_past_cap_flips_to_gia() -> None:
|
||||
acts = [
|
||||
_buy(datetime(2024, 5, 1, tzinfo=UTC), "100", "80"), # £8,000
|
||||
_buy(datetime(2024, 6, 1, tzinfo=UTC), "150", "80"), # +£12,000 → £20,000 total; prev £8k < cap → ISA
|
||||
_buy(datetime(2024, 7, 1, tzinfo=UTC), "10", "80"), # prev £20,000 ≥ cap → GIA
|
||||
_buy(datetime(2024, 8, 1, tzinfo=UTC), "10", "80"), # GIA
|
||||
]
|
||||
routed = _split_ie_by_isa_cap(acts)
|
||||
assert routed[0].account_id == _IE_ISA_ACCOUNT_ID
|
||||
assert routed[1].account_id == _IE_ISA_ACCOUNT_ID
|
||||
assert routed[2].account_id == _IE_GIA_ACCOUNT_ID
|
||||
assert routed[2].account_type is AccountType.GIA
|
||||
assert routed[3].account_id == _IE_GIA_ACCOUNT_ID
|
||||
|
||||
|
||||
def test_tax_year_boundary_resets_cap() -> None:
|
||||
acts = [
|
||||
# 2023-24 tax year: £20k in ISA, plus one in GIA
|
||||
_buy(datetime(2023, 5, 1, tzinfo=UTC), "400", "50"), # £20,000 → ISA (prev 0 < cap)
|
||||
_buy(datetime(2024, 1, 1, tzinfo=UTC), "100", "50"), # GIA (prev 20k)
|
||||
# 2024-25 tax year starts 2024-04-06 — cap resets
|
||||
_buy(datetime(2024, 5, 1, tzinfo=UTC), "100", "50"), # ISA (prev 0 for new year)
|
||||
]
|
||||
routed = _split_ie_by_isa_cap(acts)
|
||||
assert routed[0].account_id == _IE_ISA_ACCOUNT_ID
|
||||
assert routed[1].account_id == _IE_GIA_ACCOUNT_ID
|
||||
assert routed[2].account_id == _IE_ISA_ACCOUNT_ID
|
||||
|
||||
|
||||
def test_out_of_order_activities_sorted_before_cap_applied() -> None:
|
||||
acts = [
|
||||
_buy(datetime(2024, 8, 1, tzinfo=UTC), "10", "80"), # later date but given first
|
||||
_buy(datetime(2024, 5, 1, tzinfo=UTC), "250", "80"), # earlier, £20,000 → ISA
|
||||
]
|
||||
routed = _split_ie_by_isa_cap(acts)
|
||||
by_date = {a.date: a for a in routed}
|
||||
assert by_date[datetime(2024, 5, 1, tzinfo=UTC)].account_id == _IE_ISA_ACCOUNT_ID
|
||||
assert by_date[datetime(2024, 8, 1, tzinfo=UTC)].account_id == _IE_GIA_ACCOUNT_ID
|
||||
|
||||
|
||||
def test_non_ie_activities_passed_through_unchanged() -> None:
|
||||
schwab_act = Activity(
|
||||
external_id="schwab:abc",
|
||||
account_id="schwab-workplace",
|
||||
account_type=AccountType.GIA,
|
||||
date=datetime(2024, 5, 1, tzinfo=UTC),
|
||||
activity_type=ActivityType.SELL,
|
||||
currency="USD",
|
||||
symbol="META",
|
||||
quantity=Decimal("10"),
|
||||
unit_price=Decimal("500"),
|
||||
)
|
||||
routed = _split_ie_by_isa_cap([schwab_act])
|
||||
assert routed[0].account_id == "schwab-workplace"
|
||||
assert routed[0].account_type is AccountType.GIA
|
||||
Loading…
Add table
Add a link
Reference in a new issue