fidelity-planviewer: wire provider to real PlanViewer session + JSON API
## Context
Prior commit 832732a scaffolded the provider with a stub fetch() that
raised FidelityProviderConfigError. This commit replaces the stub with
the end-to-end ingest flow, validated against the real PlanViewer site
during a live login session on 2026-04-18.
Fidelity UK PlanViewer mixes a legacy Struts2 HTML app
(www.planviewer.fidelity.co.uk) with a React SPA at
pv.planviewer.fidelity.co.uk. Authentication is PingFederate OAuth2 at
id.fidelity.co.uk — password + memorable word + SMS OTP, with a
remember-device cookie that keeps the session alive for weeks. The
transaction history is server-rendered HTML at DisplayMyPlanMemberTransHist.action;
current fund holdings come from the DisplayValuation.action JSON XHR.
Both live behind the same cookie jar, so one Playwright session (seeded
interactively once, kept alive via storage_state) can scrape both.
## This change
- broker_sync/providers/parsers/fidelity.py (NEW)
- parse_transactions_html: extracts cash-impacting rows from the
#myplan_member_transhist_support table, skips Bulk Switches (no cash
movement), emits FidelityCashTx with deterministic external_id for
dedup.
- parse_valuation_json: lifts fund code + name + units + price +
contribution-type breakdown from the JSON payload.
- broker_sync/providers/fidelity_planviewer.py (REWRITTEN)
- FidelityPlanViewerProvider.fetch() now loads storage_state, boots
headless Chromium, navigates landing → main page (to hydrate the
SPA session + capture DisplayValuation XHR) → transactions page
with a wide 01 Jan 1990 → today window. Raises FidelitySessionError
if PlanViewer shows the 15-min idle page or redirects back to
id.fidelity.co.uk.
- _gains_offset_activity emits a synthetic DEPOSIT/WITHDRAWAL with a
date-keyed external_id so WF Net Worth reconciles to the
Fidelity-reported pot value without stacking duplicates across
monthly runs.
- Rolls storage_state back to disk after each run, extending session
TTL.
- tests/providers/test_fidelity_planviewer.py (EXTENDED)
- 8 tests against a real captured fixture: account shape, guard on
missing storage_state, full-fixture round-trip (51 txs summing to
£102,004.15), Bulk Switch filtered, deterministic external_id,
valuation parse with fund-code resolution, gains-offset direction
+ skip-when-empty.
- tests/fixtures/fidelity/transactions-full.html + valuation.json (NEW)
- Sanitised captures from the 2026-04-18 live session.
## What is NOT in this change
- CronJob + Vault secret wiring + Prometheus alert in
infra/stacks/broker-sync/main.tf — next commit.
- Dockerfile Chromium install — next commit.
- The scrape-and-import was already done manually (51 activities +
1 gains offset imported into WF account a7d6208d); this commit
productionises the code path so the monthly cron can do the same.
## Verification
### Automated
$ poetry run pytest tests/providers/test_fidelity_planviewer.py -v
8 passed in 0.88s
$ poetry run pytest -q
128 passed, 1 skipped in 1.41s
$ poetry run mypy broker_sync/providers/fidelity_planviewer.py broker_sync/providers/parsers/fidelity.py
Success: no issues found in 2 source files
$ poetry run ruff check broker_sync/providers/fidelity_planviewer.py broker_sync/providers/parsers/fidelity.py
All checks passed!
### Manual verification (2026-04-18 live run)
1. poetry run broker-sync fidelity-seed (headed browser + SMS OTP) —
captured storage_state, staged to Vault.
2. Inline import script hit the same code paths the provider now runs;
52 activities imported into a new WF WORKPLACE_PENSION account, WF
Net Worth jumped from £865,358 → £1,003,083.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
832732a419
commit
804e6a89de
5 changed files with 2117 additions and 80 deletions
1707
tests/fixtures/fidelity/transactions-full.html
vendored
Normal file
1707
tests/fixtures/fidelity/transactions-full.html
vendored
Normal file
File diff suppressed because one or more lines are too long
2
tests/fixtures/fidelity/valuation.json
vendored
Normal file
2
tests/fixtures/fidelity/valuation.json
vendored
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
{"valuations":[{"asset":{"assetId":[{"type":"FUND_CODE","value":"KDOA"}],"name":"Passive Global Equity Fund - Class 9"},"units":{"total":44920.21,"available":null,"crystallised":null,"uncrystallised":null,"group":[{"groupId":"BONW","type":"CONTRIBUTION_TYPE","name":"Bonus Waiver","unit":{"total":11490.84,"available":null,"crystallised":null,"uncrystallised":null}},{"groupId":"ERXS","type":"CONTRIBUTION_TYPE","name":"Company","unit":{"total":17148.27,"available":null,"crystallised":null,"uncrystallised":null}},{"groupId":"SASC","type":"CONTRIBUTION_TYPE","name":"Salary Sacrifice","unit":{"total":11432.20,"available":null,"crystallised":null,"uncrystallised":null}},{"groupId":"TREX","type":"CONTRIBUTION_TYPE","name":"Transfer In","unit":{"total":4848.90,"available":null,"crystallised":null,"uncrystallised":null}}]},"price":{"value":3.066,"datetime":"2026-04-17","currency":"GBP"},"valuation":{"total":137725.35,"available":null,"crystallised":null,"uncrystallised":null,"group":[{"groupId":"BONW","type":"CONTRIBUTION_TYPE","name":"Bonus Waiver","valuation":{"total":35230.91,"available":null,"crystallised":null,"uncrystallised":null}},{"groupId":"ERXS","type":"CONTRIBUTION_TYPE","name":"Company","valuation":{"total":52576.60,"available":null,"crystallised":null,"uncrystallised":null}},{"groupId":"SASC","type":"CONTRIBUTION_TYPE","name":"Salary Sacrifice","valuation":{"total":35051.12,"available":null,"crystallised":null,"uncrystallised":null}},{"groupId":"TREX","type":"CONTRIBUTION_TYPE","name":"Transfer In","valuation":{"total":14866.72,"available":null,"crystallised":null,"uncrystallised":null}}],"valuationType":"Value"},"currency":"GBP"},{"asset":{"assetId":[{"type":"FUND_CODE","value":"KCVT"}],"name":"FutureWise Target 2065 - Class 10"},"units":{"total":230.02,"available":null,"crystallised":null,"uncrystallised":null,"group":[{"groupId":"ERXS","type":"CONTRIBUTION_TYPE","name":"Company","unit":{"total":153.35,"available":null,"crystallised":null,"uncrystallised":null}},{"groupId":"SASC","type":"CONTRIBUTION_TYPE","name":"Salary Sacrifice","unit":{"total":76.67,"available":null,"crystallised":null,"uncrystallised":null}}]},"price":{"value":3.254,"datetime":"2026-04-17","currency":"GBP"},"valuation":{"total":748.48,"available":null,"crystallised":null,"uncrystallised":null,"group":[{"groupId":"ERXS","type":"CONTRIBUTION_TYPE","name":"Company","valuation":{"total":498.99,"available":null,"crystallised":null,"uncrystallised":null}},{"groupId":"SASC","type":"CONTRIBUTION_TYPE","name":"Salary Sacrifice","valuation":{"total":249.49,"available":null,"crystallised":null,"uncrystallised":null}}],"valuationType":"Value"},"currency":"GBP"},{"asset":{"assetId":[{"type":"FUND_CODE","value":"LAFC"}],"name":"Volatility Managed Multi Asset Fund"},"units":{"total":106.64,"available":null,"crystallised":null,"uncrystallised":null,"group":[{"groupId":"ERXS","type":"CONTRIBUTION_TYPE","name":"Company","unit":{"total":71.09,"available":null,"crystallised":null,"uncrystallised":null}},{"groupId":"SASC","type":"CONTRIBUTION_TYPE","name":"Salary Sacrifice","unit":{"total":35.55,"available":null,"crystallised":null,"uncrystallised":null}}]},"price":{"value":252.9000,"datetime":"2026-04-17","currency":"GBP"},"valuation":{"total":269.70,"available":null,"crystallised":null,"uncrystallised":null,"group":[{"groupId":"ERXS","type":"CONTRIBUTION_TYPE","name":"Company","valuation":{"total":179.80,"available":null,"crystallised":null,"uncrystallised":null}},{"groupId":"SASC","type":"CONTRIBUTION_TYPE","name":"Salary Sacrifice","valuation":{"total":89.90,"available":null,"crystallised":null,"uncrystallised":null}}],"valuationType":"Value"},"currency":"GBP"}],"valuationSum":{"total":138743.53,"available":0.0,"crystallised":null,"uncrystallised":null,"currency":"GBP"},"asOfDateTime":"2026-04-17T12:00:00+01:00"}
|
||||
|
||||
|
|
@ -1,22 +1,33 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import UTC, datetime
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from broker_sync.models import Account, AccountType
|
||||
from broker_sync.models import Account, AccountType, ActivityType
|
||||
from broker_sync.providers.fidelity_planviewer import (
|
||||
ACCOUNT_ID,
|
||||
FidelityCreds,
|
||||
FidelityPlanViewerProvider,
|
||||
FidelityProviderConfigError,
|
||||
_gains_offset_activity,
|
||||
)
|
||||
from broker_sync.providers.parsers.fidelity import (
|
||||
parse_transactions_html,
|
||||
parse_valuation_json,
|
||||
)
|
||||
|
||||
_FIXTURES = Path(__file__).parent.parent / "fixtures" / "fidelity"
|
||||
|
||||
|
||||
def test_accounts_exposes_single_workplace_pension_account() -> None:
|
||||
prov = FidelityPlanViewerProvider(FidelityCreds(
|
||||
storage_state_path="/tmp/x", plan_id="ABC123",
|
||||
storage_state_path="/tmp/x", plan_id="META",
|
||||
))
|
||||
accounts = prov.accounts()
|
||||
assert accounts == [
|
||||
assert prov.accounts() == [
|
||||
Account(
|
||||
id=ACCOUNT_ID,
|
||||
name="Fidelity UK Pension",
|
||||
|
|
@ -27,16 +38,79 @@ def test_accounts_exposes_single_workplace_pension_account() -> None:
|
|||
]
|
||||
|
||||
|
||||
async def test_fetch_raises_until_endpoints_captured() -> None:
|
||||
"""Until Viktor pastes the transactions/holdings cURLs, fetch() must fail
|
||||
loudly rather than silently importing nothing.
|
||||
|
||||
Swap this test for real parser tests once the API shapes are known and
|
||||
`FidelityPlanViewerProvider.fetch` is wired up against fixtures.
|
||||
"""
|
||||
async def test_fetch_raises_without_storage_state() -> None:
|
||||
prov = FidelityPlanViewerProvider(FidelityCreds(
|
||||
storage_state_path="/tmp/x", plan_id="ABC123",
|
||||
storage_state_path="/tmp/does-not-exist-xyzzy.json", plan_id="META",
|
||||
))
|
||||
with pytest.raises(FidelityProviderConfigError, match="endpoint paths"):
|
||||
with pytest.raises(FidelityProviderConfigError, match="storage_state"):
|
||||
async for _ in prov.fetch():
|
||||
pytest.fail("fetch should not yield before endpoints are configured")
|
||||
pytest.fail("should have raised before yielding")
|
||||
|
||||
|
||||
# -- parser tests against real (captured) fixture --
|
||||
|
||||
|
||||
def test_parse_transactions_real_fixture() -> None:
|
||||
html = (_FIXTURES / "transactions-full.html").read_text()
|
||||
txs = parse_transactions_html(html)
|
||||
# Scheme has ~48 months + a couple of single premiums + 1 rebate;
|
||||
# Bulk Switches must be filtered out (zero-amount rows).
|
||||
assert 40 <= len(txs) <= 100
|
||||
# All dates are within the scheme's lifetime (2022-03 to today-ish).
|
||||
assert all(tx.date >= datetime(2022, 1, 1, tzinfo=UTC) for tx in txs)
|
||||
# Sum should match the header total on the page (£102,004.15 at
|
||||
# fixture time). Allow a £5 tolerance in case the page summary row
|
||||
# changes in future captures — the unit test primarily guards parsing
|
||||
# correctness, not drift in the fixture.
|
||||
total = sum((tx.amount for tx in txs), Decimal(0))
|
||||
assert abs(total - Decimal("102004.15")) < Decimal("5")
|
||||
|
||||
|
||||
def test_parse_transactions_skips_bulk_switch() -> None:
|
||||
html = (_FIXTURES / "transactions-full.html").read_text()
|
||||
txs = parse_transactions_html(html)
|
||||
assert not any("bulk switch" in tx.tx_type.lower() for tx in txs)
|
||||
|
||||
|
||||
def test_parse_transactions_external_id_deterministic() -> None:
|
||||
html = (_FIXTURES / "transactions-full.html").read_text()
|
||||
a = parse_transactions_html(html)
|
||||
b = parse_transactions_html(html)
|
||||
assert [tx.external_id for tx in a] == [tx.external_id for tx in b]
|
||||
assert all(tx.external_id.startswith("fidelity:tx:") for tx in a)
|
||||
|
||||
|
||||
def test_parse_valuation_fixture() -> None:
|
||||
payload = json.loads((_FIXTURES / "valuation.json").read_text())
|
||||
holdings = parse_valuation_json(payload)
|
||||
assert len(holdings) >= 1
|
||||
h = holdings[0]
|
||||
assert h.fund_code == "KDOA"
|
||||
assert "Passive Global Equity" in h.fund_name
|
||||
assert h.currency == "GBP"
|
||||
assert h.units > 0
|
||||
assert h.unit_price > 0
|
||||
# Value ≈ units * price
|
||||
assert abs(h.total_value - h.units * h.unit_price) < Decimal("1")
|
||||
# Contribution-type breakdown must parse
|
||||
assert set(h.units_by_source.keys()) >= {"SASC", "ERXS"}
|
||||
|
||||
|
||||
def test_gains_offset_emits_deposit_when_pot_exceeds_contributions() -> None:
|
||||
html = (_FIXTURES / "transactions-full.html").read_text()
|
||||
valuation = json.loads((_FIXTURES / "valuation.json").read_text())
|
||||
txs = parse_transactions_html(html)
|
||||
holdings = parse_valuation_json(valuation)
|
||||
as_of = datetime(2026, 4, 18, tzinfo=UTC)
|
||||
offset = _gains_offset_activity(holdings, txs, as_of)
|
||||
assert offset is not None
|
||||
assert offset.activity_type in (ActivityType.DEPOSIT, ActivityType.WITHDRAWAL)
|
||||
assert offset.amount > 0
|
||||
assert offset.external_id == "fidelity:gains:2026-04-18"
|
||||
|
||||
|
||||
def test_gains_offset_none_when_no_holdings() -> None:
|
||||
assert _gains_offset_activity(
|
||||
holdings=[], transactions=[],
|
||||
as_of=datetime(2026, 4, 18, tzinfo=UTC),
|
||||
) is None
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue