fidelity-planviewer: scaffold provider + CLI (seed + stub ingest)

## Context

UK workplace pension at planviewer.fidelity.co.uk has no public API; the SPA
calls a private JSON backend at prd.wiciam.fidelity.co.uk/cvmfe/api/*. Viktor
confirmed in DevTools that an OPTIONS preflight lists auth headers
(ch, fid, rid, sid, tbid, theosreferer, ua). Full reverse-engineering of the
endpoint paths is pending Viktor's POST cURL paste for transactions +
holdings views.

Until those endpoints are captured, ship the scaffold: provider module, CLI
commands, tests, docs. This unblocks installing Playwright in the image and
lets Viktor run the one-off seed command on his laptop ahead of the data
integration.

## This change

- broker_sync/providers/fidelity_planviewer.py
  - FidelityCreds namedtuple (storage_state_path, plan_id).
  - FidelitySessionError (401 → re-seed), FidelityProviderConfigError.
  - FidelityPlanViewerProvider: .accounts() returns a single
    WORKPLACE_PENSION account, .fetch() raises until endpoints are wired.
- broker_sync/cli.py
  - fidelity-seed: launches headed Chromium so Viktor can log in and tick
    "Remember device", then dumps storage_state.json.
  - fidelity-ingest: stub matching the invest-engine / trading212 CLI
    shape; reads storage_state + plan_id, pipes through the shared pipeline.
- tests/providers/test_fidelity_planviewer.py
  - Asserts the single-account shape + the loud-failure guard.
- docs/providers/fidelity-planviewer.md
  - Architecture diagram, one-time seed procedure, backfill + monthly
    commands, alert runbook.
- pyproject.toml
  - playwright ^1.47 as a first-class dep (used only by fidelity-seed and
    later by the session-refresh step in fidelity-ingest).

## What is NOT in this change

- Endpoint wiring in provider.fetch() — blocked on DevTools POST cURL.
- Infra CronJob + Vault secret + Prometheus alert — lands once the first
  manual backfill succeeds and we know the Chromium image size is fine.
- Dockerfile Chromium install — same trigger.

## Verification

### Automated

$ poetry run pytest tests/providers/test_fidelity_planviewer.py -v
2 passed in 0.08s

$ poetry run pytest -q
122 passed, 1 skipped in 1.07s

$ poetry run mypy broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py
Success: no issues found in 2 source files

$ poetry run ruff check broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py tests/providers/test_fidelity_planviewer.py
All checks passed!

### Manual (Viktor, later)

1. poetry install && poetry run playwright install chromium
2. poetry run broker-sync fidelity-seed --out /tmp/state.json
3. Chromium opens → log in → tick "Remember device" → press Enter
4. vault kv patch secret/broker-sync fidelity_storage_state=@/tmp/state.json

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-04-18 14:09:04 +00:00
parent c830856ba1
commit 832732a419
6 changed files with 508 additions and 1 deletions

View file

@ -358,6 +358,115 @@ def imap_ingest(
asyncio.run(_run())
@app.command("fidelity-seed")
def fidelity_seed(
out: str = typer.Option(
"fidelity_storage_state.json",
help="Where to write the storage_state JSON (stage it to Vault afterwards)",
),
url: str = typer.Option(
"https://pv.planviewer.fidelity.co.uk/",
help="PlanViewer SPA URL — defaults to the production UK landing",
),
) -> None:
"""One-off: launch a headed Chromium so Viktor can log into PlanViewer and
capture a long-lived storage_state (cookies + localStorage) for the monthly
cron.
Expected flow:
1. Chromium opens on the PlanViewer login page.
2. Viktor enters username, password, memorable word, MFA code.
3. Viktor ticks "Remember device" / "Trust this browser" if offered.
4. Viktor waits until the dashboard loads, then presses Enter in the terminal.
5. Script dumps storage_state.json and exits.
6. Viktor runs ``vault kv patch secret/broker-sync fidelity_storage_state=@...``.
"""
_setup_logging()
try:
from playwright.sync_api import sync_playwright
except ImportError as e:
typer.echo(
"Playwright is not installed — run `poetry install` first.", err=True)
raise typer.Exit(code=2) from e
typer.echo(f"Opening {url} in a headed browser — log in, tick "
"'Remember device' if offered, then press Enter here.")
with sync_playwright() as pw:
browser = pw.chromium.launch(headless=False)
context = browser.new_context()
page = context.new_page()
page.goto(url)
input("Press Enter once you're fully logged in and the dashboard is visible… ")
context.storage_state(path=out)
browser.close()
typer.echo(f"Wrote {out} — stage it to Vault:")
typer.echo(f" vault kv patch secret/broker-sync fidelity_storage_state=@{out}")
@app.command("fidelity-ingest")
def fidelity_ingest(
wf_base_url: str = typer.Option(..., envvar="WF_BASE_URL"),
wf_username: str = typer.Option(..., envvar="WF_USERNAME"),
wf_password: str = typer.Option(..., envvar="WF_PASSWORD"),
wf_session_path: str = typer.Option("/data/wealthfolio_session.json", envvar="WF_SESSION_PATH"),
storage_state_path: str = typer.Option(
...,
envvar="FIDELITY_STORAGE_STATE_PATH",
help="Path on disk to storage_state.json (materialised from Vault by the init container)",
),
plan_id: str = typer.Option(..., envvar="FIDELITY_PLAN_ID"),
data_dir: str = typer.Option("/data", envvar="BROKER_SYNC_DATA_DIR"),
mode: str = typer.Option("steady", help="steady = last-60-days; backfill = full history"),
) -> None:
"""Sync Fidelity UK PlanViewer contributions + fund purchases into Wealthfolio."""
from broker_sync.dedup import SyncRecordStore
from broker_sync.pipeline import sync_provider_to_wealthfolio
from broker_sync.providers.fidelity_planviewer import (
FidelityCreds,
FidelityPlanViewerProvider,
)
from broker_sync.sinks.wealthfolio import WealthfolioSink
_setup_logging()
if mode == "steady":
since: datetime | None = datetime.now(UTC) - timedelta(days=60)
elif mode == "backfill":
since = None
else:
typer.echo(f"Unknown mode: {mode!r}. Use 'steady' or 'backfill'.", err=True)
sys.exit(2)
async def _run() -> None:
sink = WealthfolioSink(
base_url=wf_base_url,
username=wf_username,
password=wf_password,
session_path=wf_session_path,
)
provider = FidelityPlanViewerProvider(FidelityCreds(
storage_state_path=storage_state_path,
plan_id=plan_id,
))
dedup = SyncRecordStore(Path(data_dir) / "sync.db")
try:
if not Path(wf_session_path).exists():
await sink.login()
result = await sync_provider_to_wealthfolio(
provider=provider, sink=sink, dedup=dedup, since=since,
)
finally:
await sink.close()
typer.echo(f"fidelity-ingest: fetched={result.fetched} "
f"new={result.new_after_dedup} "
f"imported={result.imported} "
f"failed={result.failed}")
if result.failed > 0:
sys.exit(1)
asyncio.run(_run())
def _setup_logging() -> None:
logging.basicConfig(
level=logging.INFO,

View file

@ -0,0 +1,128 @@
"""Fidelity UK PlanViewer provider — workplace pension backfill + monthly sync.
PlanViewer has no public individual-member API; Fidelity International's
developer portal only catalogues B2B scheme/HR endpoints. The SPA (at
``pv.planviewer.fidelity.co.uk``) does call a private JSON backend at
``prd.wiciam.fidelity.co.uk/cvmfe/api/*`` we reverse-engineer that and feed
it through a Playwright-maintained session.
## Session lifecycle
1. **One-off seed** (``broker-sync fidelity-seed``): Viktor runs a headed
Chromium, logs in (password + memorable word + MFA), clicks "Remember
device". Playwright dumps the resulting ``storage_state.json`` (cookies +
localStorage) which we stash in Vault.
2. **Monthly cron**: loads storage_state, boots headless Chromium, navigates
to the SPA once to let it refresh rolling session tokens, intercepts the
first outbound XHR to capture the ``sid``/``fid``/``tbid``/``rid`` headers,
then closes the browser and continues with plain httpx.
3. **Re-seed trigger**: on any 401 from the JSON API we raise
:class:`FidelitySessionError`; the CronJob fails loudly and Prometheus
alerts Viktor to run the seed command again.
Remember-device typically survives 30-90 days on Fidelity, so we expect the
re-seed to be a quarterly manual step not monthly.
## Data model
Salary-sacrifice scheme with two contribution streams (employee + employer),
both pre-tax. Each contribution buys units across one or more funds. We emit:
- ``DEPOSIT`` per employee-or-employer cash inflow (external_id carries
``fidelity:<doc-id>:<source>``).
- ``BUY`` per fund-unit purchase (``symbol`` = fund ISIN or Fidelity code,
``quantity`` = units, ``unit_price`` = GBp or GBP per unit).
All currency is GBP. The single WF account is ``AccountType.WORKPLACE_PENSION``.
"""
from __future__ import annotations
import logging
from collections.abc import AsyncIterator
from datetime import datetime
from typing import NamedTuple
from broker_sync.models import Account, AccountType, Activity
log = logging.getLogger(__name__)
ACCOUNT_ID = "fidelity-workplace-pension"
_CCY = "GBP"
# PlanViewer's private JSON backend. Endpoint paths are reverse-engineered from
# Viktor's DevTools cURLs and validated by the unit tests' fixtures.
_API_BASE = "https://prd.wiciam.fidelity.co.uk"
class FidelityCreds(NamedTuple):
"""Credentials + session state required to hit the PlanViewer backend."""
storage_state_path: str
plan_id: str
headless: bool = True
class FidelitySessionError(Exception):
"""Raised when PlanViewer returns 401/403 — storage_state is stale.
Recovery: run ``broker-sync fidelity-seed`` in a browser to refresh the
storage_state blob in Vault, then re-run the CronJob.
"""
class FidelityProviderConfigError(Exception):
"""Raised when the provider is asked to run but required config (plan id,
storage_state path) is missing or obviously wrong."""
class FidelityPlanViewerProvider:
"""Read-only provider against Fidelity UK PlanViewer.
Per the Provider protocol consumed by ``broker_sync.pipeline``:
- ``.accounts()`` advertises the single workplace-pension WF account we
write into.
- ``.fetch(since, before)`` is an async generator that yields canonical
``Activity`` objects.
"""
name = "fidelity-planviewer"
def __init__(self, creds: FidelityCreds) -> None:
self._creds = creds
def accounts(self) -> list[Account]:
return [
Account(
id=ACCOUNT_ID,
name="Fidelity UK Pension",
account_type=AccountType.WORKPLACE_PENSION,
currency=_CCY,
provider=self.name,
),
]
async def fetch(
self,
*,
since: datetime | None = None,
before: datetime | None = None,
) -> AsyncIterator[Activity]:
"""Yield Activity records.
Implementation blocked on captured endpoint shapes. Viktor will paste
the transactions + holdings POST cURLs from DevTools, then we wire the
parsers and this method lights up.
"""
# Guard against accidentally running before endpoint reverse-engineering
# is done — makes the CronJob fail loudly with an actionable message
# rather than silently importing nothing.
raise FidelityProviderConfigError(
"Fidelity ingest not yet enabled — PlanViewer endpoint paths have "
"not been captured. Paste the POST cURLs from DevTools for the "
"transactions + holdings views and re-apply the provider update."
)
# Unreachable yield — keeps the return type AsyncIterator[Activity]
# once the raise above is removed.
if False: # pragma: no cover
yield