fidelity-planviewer: scaffold provider + CLI (seed + stub ingest)
## Context
UK workplace pension at planviewer.fidelity.co.uk has no public API; the SPA
calls a private JSON backend at prd.wiciam.fidelity.co.uk/cvmfe/api/*. Viktor
confirmed in DevTools that an OPTIONS preflight lists auth headers
(ch, fid, rid, sid, tbid, theosreferer, ua). Full reverse-engineering of the
endpoint paths is pending Viktor's POST cURL paste for transactions +
holdings views.
Until those endpoints are captured, ship the scaffold: provider module, CLI
commands, tests, docs. This unblocks installing Playwright in the image and
lets Viktor run the one-off seed command on his laptop ahead of the data
integration.
## This change
- broker_sync/providers/fidelity_planviewer.py
- FidelityCreds namedtuple (storage_state_path, plan_id).
- FidelitySessionError (401 → re-seed), FidelityProviderConfigError.
- FidelityPlanViewerProvider: .accounts() returns a single
WORKPLACE_PENSION account, .fetch() raises until endpoints are wired.
- broker_sync/cli.py
- fidelity-seed: launches headed Chromium so Viktor can log in and tick
"Remember device", then dumps storage_state.json.
- fidelity-ingest: stub matching the invest-engine / trading212 CLI
shape; reads storage_state + plan_id, pipes through the shared pipeline.
- tests/providers/test_fidelity_planviewer.py
- Asserts the single-account shape + the loud-failure guard.
- docs/providers/fidelity-planviewer.md
- Architecture diagram, one-time seed procedure, backfill + monthly
commands, alert runbook.
- pyproject.toml
- playwright ^1.47 as a first-class dep (used only by fidelity-seed and
later by the session-refresh step in fidelity-ingest).
## What is NOT in this change
- Endpoint wiring in provider.fetch() — blocked on DevTools POST cURL.
- Infra CronJob + Vault secret + Prometheus alert — lands once the first
manual backfill succeeds and we know the Chromium image size is fine.
- Dockerfile Chromium install — same trigger.
## Verification
### Automated
$ poetry run pytest tests/providers/test_fidelity_planviewer.py -v
2 passed in 0.08s
$ poetry run pytest -q
122 passed, 1 skipped in 1.07s
$ poetry run mypy broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py
Success: no issues found in 2 source files
$ poetry run ruff check broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py tests/providers/test_fidelity_planviewer.py
All checks passed!
### Manual (Viktor, later)
1. poetry install && poetry run playwright install chromium
2. poetry run broker-sync fidelity-seed --out /tmp/state.json
3. Chromium opens → log in → tick "Remember device" → press Enter
4. vault kv patch secret/broker-sync fidelity_storage_state=@/tmp/state.json
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
c830856ba1
commit
832732a419
6 changed files with 508 additions and 1 deletions
|
|
@ -358,6 +358,115 @@ def imap_ingest(
|
|||
asyncio.run(_run())
|
||||
|
||||
|
||||
@app.command("fidelity-seed")
|
||||
def fidelity_seed(
|
||||
out: str = typer.Option(
|
||||
"fidelity_storage_state.json",
|
||||
help="Where to write the storage_state JSON (stage it to Vault afterwards)",
|
||||
),
|
||||
url: str = typer.Option(
|
||||
"https://pv.planviewer.fidelity.co.uk/",
|
||||
help="PlanViewer SPA URL — defaults to the production UK landing",
|
||||
),
|
||||
) -> None:
|
||||
"""One-off: launch a headed Chromium so Viktor can log into PlanViewer and
|
||||
capture a long-lived storage_state (cookies + localStorage) for the monthly
|
||||
cron.
|
||||
|
||||
Expected flow:
|
||||
1. Chromium opens on the PlanViewer login page.
|
||||
2. Viktor enters username, password, memorable word, MFA code.
|
||||
3. Viktor ticks "Remember device" / "Trust this browser" if offered.
|
||||
4. Viktor waits until the dashboard loads, then presses Enter in the terminal.
|
||||
5. Script dumps storage_state.json and exits.
|
||||
6. Viktor runs ``vault kv patch secret/broker-sync fidelity_storage_state=@...``.
|
||||
"""
|
||||
_setup_logging()
|
||||
try:
|
||||
from playwright.sync_api import sync_playwright
|
||||
except ImportError as e:
|
||||
typer.echo(
|
||||
"Playwright is not installed — run `poetry install` first.", err=True)
|
||||
raise typer.Exit(code=2) from e
|
||||
|
||||
typer.echo(f"Opening {url} in a headed browser — log in, tick "
|
||||
"'Remember device' if offered, then press Enter here.")
|
||||
with sync_playwright() as pw:
|
||||
browser = pw.chromium.launch(headless=False)
|
||||
context = browser.new_context()
|
||||
page = context.new_page()
|
||||
page.goto(url)
|
||||
input("Press Enter once you're fully logged in and the dashboard is visible… ")
|
||||
context.storage_state(path=out)
|
||||
browser.close()
|
||||
typer.echo(f"Wrote {out} — stage it to Vault:")
|
||||
typer.echo(f" vault kv patch secret/broker-sync fidelity_storage_state=@{out}")
|
||||
|
||||
|
||||
@app.command("fidelity-ingest")
|
||||
def fidelity_ingest(
|
||||
wf_base_url: str = typer.Option(..., envvar="WF_BASE_URL"),
|
||||
wf_username: str = typer.Option(..., envvar="WF_USERNAME"),
|
||||
wf_password: str = typer.Option(..., envvar="WF_PASSWORD"),
|
||||
wf_session_path: str = typer.Option("/data/wealthfolio_session.json", envvar="WF_SESSION_PATH"),
|
||||
storage_state_path: str = typer.Option(
|
||||
...,
|
||||
envvar="FIDELITY_STORAGE_STATE_PATH",
|
||||
help="Path on disk to storage_state.json (materialised from Vault by the init container)",
|
||||
),
|
||||
plan_id: str = typer.Option(..., envvar="FIDELITY_PLAN_ID"),
|
||||
data_dir: str = typer.Option("/data", envvar="BROKER_SYNC_DATA_DIR"),
|
||||
mode: str = typer.Option("steady", help="steady = last-60-days; backfill = full history"),
|
||||
) -> None:
|
||||
"""Sync Fidelity UK PlanViewer contributions + fund purchases into Wealthfolio."""
|
||||
from broker_sync.dedup import SyncRecordStore
|
||||
from broker_sync.pipeline import sync_provider_to_wealthfolio
|
||||
from broker_sync.providers.fidelity_planviewer import (
|
||||
FidelityCreds,
|
||||
FidelityPlanViewerProvider,
|
||||
)
|
||||
from broker_sync.sinks.wealthfolio import WealthfolioSink
|
||||
|
||||
_setup_logging()
|
||||
|
||||
if mode == "steady":
|
||||
since: datetime | None = datetime.now(UTC) - timedelta(days=60)
|
||||
elif mode == "backfill":
|
||||
since = None
|
||||
else:
|
||||
typer.echo(f"Unknown mode: {mode!r}. Use 'steady' or 'backfill'.", err=True)
|
||||
sys.exit(2)
|
||||
|
||||
async def _run() -> None:
|
||||
sink = WealthfolioSink(
|
||||
base_url=wf_base_url,
|
||||
username=wf_username,
|
||||
password=wf_password,
|
||||
session_path=wf_session_path,
|
||||
)
|
||||
provider = FidelityPlanViewerProvider(FidelityCreds(
|
||||
storage_state_path=storage_state_path,
|
||||
plan_id=plan_id,
|
||||
))
|
||||
dedup = SyncRecordStore(Path(data_dir) / "sync.db")
|
||||
try:
|
||||
if not Path(wf_session_path).exists():
|
||||
await sink.login()
|
||||
result = await sync_provider_to_wealthfolio(
|
||||
provider=provider, sink=sink, dedup=dedup, since=since,
|
||||
)
|
||||
finally:
|
||||
await sink.close()
|
||||
typer.echo(f"fidelity-ingest: fetched={result.fetched} "
|
||||
f"new={result.new_after_dedup} "
|
||||
f"imported={result.imported} "
|
||||
f"failed={result.failed}")
|
||||
if result.failed > 0:
|
||||
sys.exit(1)
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
|
||||
def _setup_logging() -> None:
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
|
|
|
|||
128
broker_sync/providers/fidelity_planviewer.py
Normal file
128
broker_sync/providers/fidelity_planviewer.py
Normal file
|
|
@ -0,0 +1,128 @@
|
|||
"""Fidelity UK PlanViewer provider — workplace pension backfill + monthly sync.
|
||||
|
||||
PlanViewer has no public individual-member API; Fidelity International's
|
||||
developer portal only catalogues B2B scheme/HR endpoints. The SPA (at
|
||||
``pv.planviewer.fidelity.co.uk``) does call a private JSON backend at
|
||||
``prd.wiciam.fidelity.co.uk/cvmfe/api/*`` — we reverse-engineer that and feed
|
||||
it through a Playwright-maintained session.
|
||||
|
||||
## Session lifecycle
|
||||
|
||||
1. **One-off seed** (``broker-sync fidelity-seed``): Viktor runs a headed
|
||||
Chromium, logs in (password + memorable word + MFA), clicks "Remember
|
||||
device". Playwright dumps the resulting ``storage_state.json`` (cookies +
|
||||
localStorage) which we stash in Vault.
|
||||
|
||||
2. **Monthly cron**: loads storage_state, boots headless Chromium, navigates
|
||||
to the SPA once to let it refresh rolling session tokens, intercepts the
|
||||
first outbound XHR to capture the ``sid``/``fid``/``tbid``/``rid`` headers,
|
||||
then closes the browser and continues with plain httpx.
|
||||
|
||||
3. **Re-seed trigger**: on any 401 from the JSON API we raise
|
||||
:class:`FidelitySessionError`; the CronJob fails loudly and Prometheus
|
||||
alerts Viktor to run the seed command again.
|
||||
|
||||
Remember-device typically survives 30-90 days on Fidelity, so we expect the
|
||||
re-seed to be a quarterly manual step — not monthly.
|
||||
|
||||
## Data model
|
||||
|
||||
Salary-sacrifice scheme with two contribution streams (employee + employer),
|
||||
both pre-tax. Each contribution buys units across one or more funds. We emit:
|
||||
|
||||
- ``DEPOSIT`` per employee-or-employer cash inflow (external_id carries
|
||||
``fidelity:<doc-id>:<source>``).
|
||||
- ``BUY`` per fund-unit purchase (``symbol`` = fund ISIN or Fidelity code,
|
||||
``quantity`` = units, ``unit_price`` = GBp or GBP per unit).
|
||||
|
||||
All currency is GBP. The single WF account is ``AccountType.WORKPLACE_PENSION``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import AsyncIterator
|
||||
from datetime import datetime
|
||||
from typing import NamedTuple
|
||||
|
||||
from broker_sync.models import Account, AccountType, Activity
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
ACCOUNT_ID = "fidelity-workplace-pension"
|
||||
_CCY = "GBP"
|
||||
|
||||
# PlanViewer's private JSON backend. Endpoint paths are reverse-engineered from
|
||||
# Viktor's DevTools cURLs and validated by the unit tests' fixtures.
|
||||
_API_BASE = "https://prd.wiciam.fidelity.co.uk"
|
||||
|
||||
|
||||
class FidelityCreds(NamedTuple):
|
||||
"""Credentials + session state required to hit the PlanViewer backend."""
|
||||
storage_state_path: str
|
||||
plan_id: str
|
||||
headless: bool = True
|
||||
|
||||
|
||||
class FidelitySessionError(Exception):
|
||||
"""Raised when PlanViewer returns 401/403 — storage_state is stale.
|
||||
|
||||
Recovery: run ``broker-sync fidelity-seed`` in a browser to refresh the
|
||||
storage_state blob in Vault, then re-run the CronJob.
|
||||
"""
|
||||
|
||||
|
||||
class FidelityProviderConfigError(Exception):
|
||||
"""Raised when the provider is asked to run but required config (plan id,
|
||||
storage_state path) is missing or obviously wrong."""
|
||||
|
||||
|
||||
class FidelityPlanViewerProvider:
|
||||
"""Read-only provider against Fidelity UK PlanViewer.
|
||||
|
||||
Per the Provider protocol consumed by ``broker_sync.pipeline``:
|
||||
|
||||
- ``.accounts()`` advertises the single workplace-pension WF account we
|
||||
write into.
|
||||
- ``.fetch(since, before)`` is an async generator that yields canonical
|
||||
``Activity`` objects.
|
||||
"""
|
||||
name = "fidelity-planviewer"
|
||||
|
||||
def __init__(self, creds: FidelityCreds) -> None:
|
||||
self._creds = creds
|
||||
|
||||
def accounts(self) -> list[Account]:
|
||||
return [
|
||||
Account(
|
||||
id=ACCOUNT_ID,
|
||||
name="Fidelity UK Pension",
|
||||
account_type=AccountType.WORKPLACE_PENSION,
|
||||
currency=_CCY,
|
||||
provider=self.name,
|
||||
),
|
||||
]
|
||||
|
||||
async def fetch(
|
||||
self,
|
||||
*,
|
||||
since: datetime | None = None,
|
||||
before: datetime | None = None,
|
||||
) -> AsyncIterator[Activity]:
|
||||
"""Yield Activity records.
|
||||
|
||||
Implementation blocked on captured endpoint shapes. Viktor will paste
|
||||
the transactions + holdings POST cURLs from DevTools, then we wire the
|
||||
parsers and this method lights up.
|
||||
"""
|
||||
# Guard against accidentally running before endpoint reverse-engineering
|
||||
# is done — makes the CronJob fail loudly with an actionable message
|
||||
# rather than silently importing nothing.
|
||||
raise FidelityProviderConfigError(
|
||||
"Fidelity ingest not yet enabled — PlanViewer endpoint paths have "
|
||||
"not been captured. Paste the POST cURLs from DevTools for the "
|
||||
"transactions + holdings views and re-apply the provider update."
|
||||
)
|
||||
# Unreachable yield — keeps the return type AsyncIterator[Activity]
|
||||
# once the raise above is removed.
|
||||
if False: # pragma: no cover
|
||||
yield
|
||||
Loading…
Add table
Add a link
Reference in a new issue