broker-sync/docs/providers/fidelity-planviewer.md

112 lines
6 KiB
Markdown
Raw Normal View History

fidelity-planviewer: scaffold provider + CLI (seed + stub ingest) ## Context UK workplace pension at planviewer.fidelity.co.uk has no public API; the SPA calls a private JSON backend at prd.wiciam.fidelity.co.uk/cvmfe/api/*. Viktor confirmed in DevTools that an OPTIONS preflight lists auth headers (ch, fid, rid, sid, tbid, theosreferer, ua). Full reverse-engineering of the endpoint paths is pending Viktor's POST cURL paste for transactions + holdings views. Until those endpoints are captured, ship the scaffold: provider module, CLI commands, tests, docs. This unblocks installing Playwright in the image and lets Viktor run the one-off seed command on his laptop ahead of the data integration. ## This change - broker_sync/providers/fidelity_planviewer.py - FidelityCreds namedtuple (storage_state_path, plan_id). - FidelitySessionError (401 → re-seed), FidelityProviderConfigError. - FidelityPlanViewerProvider: .accounts() returns a single WORKPLACE_PENSION account, .fetch() raises until endpoints are wired. - broker_sync/cli.py - fidelity-seed: launches headed Chromium so Viktor can log in and tick "Remember device", then dumps storage_state.json. - fidelity-ingest: stub matching the invest-engine / trading212 CLI shape; reads storage_state + plan_id, pipes through the shared pipeline. - tests/providers/test_fidelity_planviewer.py - Asserts the single-account shape + the loud-failure guard. - docs/providers/fidelity-planviewer.md - Architecture diagram, one-time seed procedure, backfill + monthly commands, alert runbook. - pyproject.toml - playwright ^1.47 as a first-class dep (used only by fidelity-seed and later by the session-refresh step in fidelity-ingest). ## What is NOT in this change - Endpoint wiring in provider.fetch() — blocked on DevTools POST cURL. - Infra CronJob + Vault secret + Prometheus alert — lands once the first manual backfill succeeds and we know the Chromium image size is fine. - Dockerfile Chromium install — same trigger. ## Verification ### Automated $ poetry run pytest tests/providers/test_fidelity_planviewer.py -v 2 passed in 0.08s $ poetry run pytest -q 122 passed, 1 skipped in 1.07s $ poetry run mypy broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py Success: no issues found in 2 source files $ poetry run ruff check broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py tests/providers/test_fidelity_planviewer.py All checks passed! ### Manual (Viktor, later) 1. poetry install && poetry run playwright install chromium 2. poetry run broker-sync fidelity-seed --out /tmp/state.json 3. Chromium opens → log in → tick "Remember device" → press Enter 4. vault kv patch secret/broker-sync fidelity_storage_state=@/tmp/state.json Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 14:09:04 +00:00
# Fidelity UK PlanViewer provider
Viktor's UK workplace pension is hosted at `pv.planviewer.fidelity.co.uk`. There
is no public API for individual members — the provider reverse-engineers the
private JSON backend at `prd.wiciam.fidelity.co.uk/cvmfe/api/*` that the SPA
itself calls, and uses Playwright only to keep a long-lived login session
alive.
## Architecture
```
┌─────────────┐ storage_state.json ┌──────────────────┐
│ Vault KV │◀─── (quarterly reseed) ───│ fidelity-seed │
│ broker-sync │ │ (headed browser) │
└──────┬──────┘ └──────────────────┘
│ ▲
│ loads on start │ Viktor runs once
▼ when session expires
┌────────────────────┐
│ Monthly CronJob │
│ broker-sync-fidelity│
└────────────┬────────┘
│ headless Chromium
┌─────────────────────────────────┐ ┌────────────────────────────────┐
│ pv.planviewer.fidelity.co.uk │◀─────│ navigate dashboard → capture │
│ (SPA) │ │ fresh sid/fid/tbid/rid headers │
└─────────────────────────────────┘ └──────────────┬─────────────────┘
┌───────────▼─────────────┐
│ httpx JSON calls │
│ prd.wiciam.../cvmfe/api│
└───────────┬─────────────┘
┌────────────────────▼────────────────────┐
│ DEPOSIT × N (employee + employer) │
│ BUY × N (fund unit purchases, per date) │
└────────────────────┬────────────────────┘
┌────────────────▼────────────────┐
│ Wealthfolio account │
│ type = WORKPLACE_PENSION │
│ currency = GBP │
└──────────────────────────────────┘
```
## One-time seed (Viktor)
```bash
# on your laptop (macOS / Linux with a desktop):
cd broker-sync
poetry install
poetry run playwright install chromium
poetry run broker-sync fidelity-seed --out /tmp/fidelity_storage_state.json
# chromium opens — log in to PlanViewer, tick "Remember device", press Enter
# stage to Vault
vault kv patch secret/broker-sync \
fidelity_storage_state=@/tmp/fidelity_storage_state.json \
fidelity_plan_id=<your-plan-id>
rm /tmp/fidelity_storage_state.json # don't leave credentials lying around
```
Re-seed when the monthly CronJob fails with `FidelitySessionError` (expect
every 30-90 days, depending on how long Fidelity honours the remember-device
cookie).
## One-time backfill
```bash
kubectl -n broker-sync create job fidelity-backfill \
--from=cronjob/broker-sync-fidelity
kubectl -n broker-sync logs -f job/fidelity-backfill
# expect: fidelity-ingest: fetched=N new=N imported=N failed=0
```
## Monthly cron
- Schedule: `0 3 5 * *` (3am UTC on the 5th of each month — after mid-month payroll settles in Viktor's scheme)
- CronJob: `broker-sync-fidelity` in namespace `broker-sync`
- Resource: small, ≤512 MiB memory (Chromium for ~2 min, then idle)
- Alert: `BrokerSyncFidelityFailed` fires on 2 consecutive failures
## Runbook — `BrokerSyncFidelityFailed`
1. Check pod logs: `kubectl -n broker-sync logs job/broker-sync-fidelity-<timestamp>`.
2. If the error is `FidelitySessionError`: session expired, re-run the seed on
Viktor's laptop (see above).
3. If the error is a 404 / 5xx from `prd.wiciam.fidelity.co.uk`: likely an API
path change. Check DevTools for the new endpoint, update the provider, ship
a new image.
4. If Playwright can't launch Chromium: check that the image still has Chromium
installed (`playwright install chromium` at build time).
## Data model notes
- **Salary sacrifice scheme**: all employee + employer contributions are
pre-tax from gross salary. No HMRC basic-rate relief line.
- Emits two `DEPOSIT` per month (employee, employer) with `comment` carrying
the source tag `fidelity:<doc-id>:<source>` for audit.
- Emits one `BUY` per fund unit purchase, `symbol` = Fidelity fund code / ISIN.
Units × unit price should reconcile to the cash deposited ±pennies.
## Not yet implemented
- Endpoint paths: waiting on Viktor's DevTools POST cURL for transactions +
holdings views. Until pasted, `fidelity-ingest` raises
`FidelityProviderConfigError` to fail loudly.
- Infra: CronJob + Vault secret wiring + Prometheus alert in
`infra/stacks/broker-sync/main.tf` — pending first successful manual run.