Viktor Barzin 832732a419 fidelity-planviewer: scaffold provider + CLI (seed + stub ingest)

## Context

UK workplace pension at planviewer.fidelity.co.uk has no public API; the SPA
calls a private JSON backend at prd.wiciam.fidelity.co.uk/cvmfe/api/*. Viktor
confirmed in DevTools that an OPTIONS preflight lists auth headers
(ch, fid, rid, sid, tbid, theosreferer, ua). Full reverse-engineering of the
endpoint paths is pending Viktor's POST cURL paste for transactions +
holdings views.

Until those endpoints are captured, ship the scaffold: provider module, CLI
commands, tests, docs. This unblocks installing Playwright in the image and
lets Viktor run the one-off seed command on his laptop ahead of the data
integration.

## This change

- broker_sync/providers/fidelity_planviewer.py
  - FidelityCreds namedtuple (storage_state_path, plan_id).
  - FidelitySessionError (401 → re-seed), FidelityProviderConfigError.
  - FidelityPlanViewerProvider: .accounts() returns a single
    WORKPLACE_PENSION account, .fetch() raises until endpoints are wired.
- broker_sync/cli.py
  - fidelity-seed: launches headed Chromium so Viktor can log in and tick
    "Remember device", then dumps storage_state.json.
  - fidelity-ingest: stub matching the invest-engine / trading212 CLI
    shape; reads storage_state + plan_id, pipes through the shared pipeline.
- tests/providers/test_fidelity_planviewer.py
  - Asserts the single-account shape + the loud-failure guard.
- docs/providers/fidelity-planviewer.md
  - Architecture diagram, one-time seed procedure, backfill + monthly
    commands, alert runbook.
- pyproject.toml
  - playwright ^1.47 as a first-class dep (used only by fidelity-seed and
    later by the session-refresh step in fidelity-ingest).

## What is NOT in this change

- Endpoint wiring in provider.fetch() — blocked on DevTools POST cURL.
- Infra CronJob + Vault secret + Prometheus alert — lands once the first
  manual backfill succeeds and we know the Chromium image size is fine.
- Dockerfile Chromium install — same trigger.

## Verification

### Automated

$ poetry run pytest tests/providers/test_fidelity_planviewer.py -v
2 passed in 0.08s

$ poetry run pytest -q
122 passed, 1 skipped in 1.07s

$ poetry run mypy broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py
Success: no issues found in 2 source files

$ poetry run ruff check broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py tests/providers/test_fidelity_planviewer.py
All checks passed!

### Manual (Viktor, later)

1. poetry install && poetry run playwright install chromium
2. poetry run broker-sync fidelity-seed --out /tmp/state.json
3. Chromium opens → log in → tick "Remember device" → press Enter
4. vault kv patch secret/broker-sync fidelity_storage_state=@/tmp/state.json

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-18 14:09:04 +00:00

6 KiB

Raw Permalink Blame History

Fidelity UK PlanViewer provider

Viktor's UK workplace pension is hosted at pv.planviewer.fidelity.co.uk. There is no public API for individual members — the provider reverse-engineers the private JSON backend at prd.wiciam.fidelity.co.uk/cvmfe/api/* that the SPA itself calls, and uses Playwright only to keep a long-lived login session alive.

Architecture

┌─────────────┐    storage_state.json     ┌──────────────────┐
│  Vault KV   │◀─── (quarterly reseed) ───│  fidelity-seed   │
│ broker-sync │                           │  (headed browser) │
└──────┬──────┘                           └──────────────────┘
       │                                           ▲
       │ loads on start                            │ Viktor runs once
       ▼                                          when session expires
┌────────────────────┐
│   Monthly CronJob   │
│ broker-sync-fidelity│
└────────────┬────────┘
             │ headless Chromium
             ▼
┌─────────────────────────────────┐      ┌────────────────────────────────┐
│ pv.planviewer.fidelity.co.uk    │◀─────│ navigate dashboard → capture   │
│ (SPA)                           │      │ fresh sid/fid/tbid/rid headers │
└─────────────────────────────────┘      └──────────────┬─────────────────┘
                                                         │
                                             ┌───────────▼─────────────┐
                                             │  httpx JSON calls       │
                                             │  prd.wiciam.../cvmfe/api│
                                             └───────────┬─────────────┘
                                                         │
                                    ┌────────────────────▼────────────────────┐
                                    │ DEPOSIT × N (employee + employer)        │
                                    │ BUY × N (fund unit purchases, per date) │
                                    └────────────────────┬────────────────────┘
                                                         │
                                        ┌────────────────▼────────────────┐
                                        │ Wealthfolio account              │
                                        │ type = WORKPLACE_PENSION         │
                                        │ currency = GBP                   │
                                        └──────────────────────────────────┘

One-time seed (Viktor)

# on your laptop (macOS / Linux with a desktop):
cd broker-sync
poetry install
poetry run playwright install chromium
poetry run broker-sync fidelity-seed --out /tmp/fidelity_storage_state.json
# chromium opens — log in to PlanViewer, tick "Remember device", press Enter

# stage to Vault
vault kv patch secret/broker-sync \
  fidelity_storage_state=@/tmp/fidelity_storage_state.json \
  fidelity_plan_id=<your-plan-id>

rm /tmp/fidelity_storage_state.json   # don't leave credentials lying around

Re-seed when the monthly CronJob fails with FidelitySessionError (expect every 30-90 days, depending on how long Fidelity honours the remember-device cookie).

One-time backfill

kubectl -n broker-sync create job fidelity-backfill \
  --from=cronjob/broker-sync-fidelity
kubectl -n broker-sync logs -f job/fidelity-backfill
# expect: fidelity-ingest: fetched=N new=N imported=N failed=0

Monthly cron

Schedule: 0 3 5 * * (3am UTC on the 5th of each month — after mid-month payroll settles in Viktor's scheme)
CronJob: broker-sync-fidelity in namespace broker-sync
Resource: small, ≤512 MiB memory (Chromium for ~2 min, then idle)
Alert: BrokerSyncFidelityFailed fires on 2 consecutive failures

Runbook — `BrokerSyncFidelityFailed`

Check pod logs: kubectl -n broker-sync logs job/broker-sync-fidelity-<timestamp>.
If the error is FidelitySessionError: session expired, re-run the seed on Viktor's laptop (see above).
If the error is a 404 / 5xx from prd.wiciam.fidelity.co.uk: likely an API path change. Check DevTools for the new endpoint, update the provider, ship a new image.
If Playwright can't launch Chromium: check that the image still has Chromium installed (playwright install chromium at build time).

Data model notes

Salary sacrifice scheme: all employee + employer contributions are pre-tax from gross salary. No HMRC basic-rate relief line.
Emits two DEPOSIT per month (employee, employer) with comment carrying the source tag fidelity:<doc-id>:<source> for audit.
Emits one BUY per fund unit purchase, symbol = Fidelity fund code / ISIN. Units × unit price should reconcile to the cash deposited ±pennies.

Not yet implemented

Endpoint paths: waiting on Viktor's DevTools POST cURL for transactions + holdings views. Until pasted, fidelity-ingest raises FidelityProviderConfigError to fail loudly.
Infra: CronJob + Vault secret wiring + Prometheus alert in infra/stacks/broker-sync/main.tf — pending first successful manual run.

6 KiB Raw Permalink Blame History Unescape Escape