fidelity-planviewer: scaffold provider + CLI (seed + stub ingest)
## Context
UK workplace pension at planviewer.fidelity.co.uk has no public API; the SPA
calls a private JSON backend at prd.wiciam.fidelity.co.uk/cvmfe/api/*. Viktor
confirmed in DevTools that an OPTIONS preflight lists auth headers
(ch, fid, rid, sid, tbid, theosreferer, ua). Full reverse-engineering of the
endpoint paths is pending Viktor's POST cURL paste for transactions +
holdings views.
Until those endpoints are captured, ship the scaffold: provider module, CLI
commands, tests, docs. This unblocks installing Playwright in the image and
lets Viktor run the one-off seed command on his laptop ahead of the data
integration.
## This change
- broker_sync/providers/fidelity_planviewer.py
- FidelityCreds namedtuple (storage_state_path, plan_id).
- FidelitySessionError (401 → re-seed), FidelityProviderConfigError.
- FidelityPlanViewerProvider: .accounts() returns a single
WORKPLACE_PENSION account, .fetch() raises until endpoints are wired.
- broker_sync/cli.py
- fidelity-seed: launches headed Chromium so Viktor can log in and tick
"Remember device", then dumps storage_state.json.
- fidelity-ingest: stub matching the invest-engine / trading212 CLI
shape; reads storage_state + plan_id, pipes through the shared pipeline.
- tests/providers/test_fidelity_planviewer.py
- Asserts the single-account shape + the loud-failure guard.
- docs/providers/fidelity-planviewer.md
- Architecture diagram, one-time seed procedure, backfill + monthly
commands, alert runbook.
- pyproject.toml
- playwright ^1.47 as a first-class dep (used only by fidelity-seed and
later by the session-refresh step in fidelity-ingest).
## What is NOT in this change
- Endpoint wiring in provider.fetch() — blocked on DevTools POST cURL.
- Infra CronJob + Vault secret + Prometheus alert — lands once the first
manual backfill succeeds and we know the Chromium image size is fine.
- Dockerfile Chromium install — same trigger.
## Verification
### Automated
$ poetry run pytest tests/providers/test_fidelity_planviewer.py -v
2 passed in 0.08s
$ poetry run pytest -q
122 passed, 1 skipped in 1.07s
$ poetry run mypy broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py
Success: no issues found in 2 source files
$ poetry run ruff check broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py tests/providers/test_fidelity_planviewer.py
All checks passed!
### Manual (Viktor, later)
1. poetry install && poetry run playwright install chromium
2. poetry run broker-sync fidelity-seed --out /tmp/state.json
3. Chromium opens → log in → tick "Remember device" → press Enter
4. vault kv patch secret/broker-sync fidelity_storage_state=@/tmp/state.json
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
c830856ba1
commit
832732a419
6 changed files with 508 additions and 1 deletions
|
|
@ -358,6 +358,115 @@ def imap_ingest(
|
|||
asyncio.run(_run())
|
||||
|
||||
|
||||
@app.command("fidelity-seed")
|
||||
def fidelity_seed(
|
||||
out: str = typer.Option(
|
||||
"fidelity_storage_state.json",
|
||||
help="Where to write the storage_state JSON (stage it to Vault afterwards)",
|
||||
),
|
||||
url: str = typer.Option(
|
||||
"https://pv.planviewer.fidelity.co.uk/",
|
||||
help="PlanViewer SPA URL — defaults to the production UK landing",
|
||||
),
|
||||
) -> None:
|
||||
"""One-off: launch a headed Chromium so Viktor can log into PlanViewer and
|
||||
capture a long-lived storage_state (cookies + localStorage) for the monthly
|
||||
cron.
|
||||
|
||||
Expected flow:
|
||||
1. Chromium opens on the PlanViewer login page.
|
||||
2. Viktor enters username, password, memorable word, MFA code.
|
||||
3. Viktor ticks "Remember device" / "Trust this browser" if offered.
|
||||
4. Viktor waits until the dashboard loads, then presses Enter in the terminal.
|
||||
5. Script dumps storage_state.json and exits.
|
||||
6. Viktor runs ``vault kv patch secret/broker-sync fidelity_storage_state=@...``.
|
||||
"""
|
||||
_setup_logging()
|
||||
try:
|
||||
from playwright.sync_api import sync_playwright
|
||||
except ImportError as e:
|
||||
typer.echo(
|
||||
"Playwright is not installed — run `poetry install` first.", err=True)
|
||||
raise typer.Exit(code=2) from e
|
||||
|
||||
typer.echo(f"Opening {url} in a headed browser — log in, tick "
|
||||
"'Remember device' if offered, then press Enter here.")
|
||||
with sync_playwright() as pw:
|
||||
browser = pw.chromium.launch(headless=False)
|
||||
context = browser.new_context()
|
||||
page = context.new_page()
|
||||
page.goto(url)
|
||||
input("Press Enter once you're fully logged in and the dashboard is visible… ")
|
||||
context.storage_state(path=out)
|
||||
browser.close()
|
||||
typer.echo(f"Wrote {out} — stage it to Vault:")
|
||||
typer.echo(f" vault kv patch secret/broker-sync fidelity_storage_state=@{out}")
|
||||
|
||||
|
||||
@app.command("fidelity-ingest")
|
||||
def fidelity_ingest(
|
||||
wf_base_url: str = typer.Option(..., envvar="WF_BASE_URL"),
|
||||
wf_username: str = typer.Option(..., envvar="WF_USERNAME"),
|
||||
wf_password: str = typer.Option(..., envvar="WF_PASSWORD"),
|
||||
wf_session_path: str = typer.Option("/data/wealthfolio_session.json", envvar="WF_SESSION_PATH"),
|
||||
storage_state_path: str = typer.Option(
|
||||
...,
|
||||
envvar="FIDELITY_STORAGE_STATE_PATH",
|
||||
help="Path on disk to storage_state.json (materialised from Vault by the init container)",
|
||||
),
|
||||
plan_id: str = typer.Option(..., envvar="FIDELITY_PLAN_ID"),
|
||||
data_dir: str = typer.Option("/data", envvar="BROKER_SYNC_DATA_DIR"),
|
||||
mode: str = typer.Option("steady", help="steady = last-60-days; backfill = full history"),
|
||||
) -> None:
|
||||
"""Sync Fidelity UK PlanViewer contributions + fund purchases into Wealthfolio."""
|
||||
from broker_sync.dedup import SyncRecordStore
|
||||
from broker_sync.pipeline import sync_provider_to_wealthfolio
|
||||
from broker_sync.providers.fidelity_planviewer import (
|
||||
FidelityCreds,
|
||||
FidelityPlanViewerProvider,
|
||||
)
|
||||
from broker_sync.sinks.wealthfolio import WealthfolioSink
|
||||
|
||||
_setup_logging()
|
||||
|
||||
if mode == "steady":
|
||||
since: datetime | None = datetime.now(UTC) - timedelta(days=60)
|
||||
elif mode == "backfill":
|
||||
since = None
|
||||
else:
|
||||
typer.echo(f"Unknown mode: {mode!r}. Use 'steady' or 'backfill'.", err=True)
|
||||
sys.exit(2)
|
||||
|
||||
async def _run() -> None:
|
||||
sink = WealthfolioSink(
|
||||
base_url=wf_base_url,
|
||||
username=wf_username,
|
||||
password=wf_password,
|
||||
session_path=wf_session_path,
|
||||
)
|
||||
provider = FidelityPlanViewerProvider(FidelityCreds(
|
||||
storage_state_path=storage_state_path,
|
||||
plan_id=plan_id,
|
||||
))
|
||||
dedup = SyncRecordStore(Path(data_dir) / "sync.db")
|
||||
try:
|
||||
if not Path(wf_session_path).exists():
|
||||
await sink.login()
|
||||
result = await sync_provider_to_wealthfolio(
|
||||
provider=provider, sink=sink, dedup=dedup, since=since,
|
||||
)
|
||||
finally:
|
||||
await sink.close()
|
||||
typer.echo(f"fidelity-ingest: fetched={result.fetched} "
|
||||
f"new={result.new_after_dedup} "
|
||||
f"imported={result.imported} "
|
||||
f"failed={result.failed}")
|
||||
if result.failed > 0:
|
||||
sys.exit(1)
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
|
||||
def _setup_logging() -> None:
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue