fidelity-planviewer: scaffold provider + CLI (seed + stub ingest)
## Context
UK workplace pension at planviewer.fidelity.co.uk has no public API; the SPA
calls a private JSON backend at prd.wiciam.fidelity.co.uk/cvmfe/api/*. Viktor
confirmed in DevTools that an OPTIONS preflight lists auth headers
(ch, fid, rid, sid, tbid, theosreferer, ua). Full reverse-engineering of the
endpoint paths is pending Viktor's POST cURL paste for transactions +
holdings views.
Until those endpoints are captured, ship the scaffold: provider module, CLI
commands, tests, docs. This unblocks installing Playwright in the image and
lets Viktor run the one-off seed command on his laptop ahead of the data
integration.
## This change
- broker_sync/providers/fidelity_planviewer.py
- FidelityCreds namedtuple (storage_state_path, plan_id).
- FidelitySessionError (401 → re-seed), FidelityProviderConfigError.
- FidelityPlanViewerProvider: .accounts() returns a single
WORKPLACE_PENSION account, .fetch() raises until endpoints are wired.
- broker_sync/cli.py
- fidelity-seed: launches headed Chromium so Viktor can log in and tick
"Remember device", then dumps storage_state.json.
- fidelity-ingest: stub matching the invest-engine / trading212 CLI
shape; reads storage_state + plan_id, pipes through the shared pipeline.
- tests/providers/test_fidelity_planviewer.py
- Asserts the single-account shape + the loud-failure guard.
- docs/providers/fidelity-planviewer.md
- Architecture diagram, one-time seed procedure, backfill + monthly
commands, alert runbook.
- pyproject.toml
- playwright ^1.47 as a first-class dep (used only by fidelity-seed and
later by the session-refresh step in fidelity-ingest).
## What is NOT in this change
- Endpoint wiring in provider.fetch() — blocked on DevTools POST cURL.
- Infra CronJob + Vault secret + Prometheus alert — lands once the first
manual backfill succeeds and we know the Chromium image size is fine.
- Dockerfile Chromium install — same trigger.
## Verification
### Automated
$ poetry run pytest tests/providers/test_fidelity_planviewer.py -v
2 passed in 0.08s
$ poetry run pytest -q
122 passed, 1 skipped in 1.07s
$ poetry run mypy broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py
Success: no issues found in 2 source files
$ poetry run ruff check broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py tests/providers/test_fidelity_planviewer.py
All checks passed!
### Manual (Viktor, later)
1. poetry install && poetry run playwright install chromium
2. poetry run broker-sync fidelity-seed --out /tmp/state.json
3. Chromium opens → log in → tick "Remember device" → press Enter
4. vault kv patch secret/broker-sync fidelity_storage_state=@/tmp/state.json
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
c830856ba1
commit
832732a419
6 changed files with 508 additions and 1 deletions
111
docs/providers/fidelity-planviewer.md
Normal file
111
docs/providers/fidelity-planviewer.md
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
# Fidelity UK PlanViewer provider
|
||||
|
||||
Viktor's UK workplace pension is hosted at `pv.planviewer.fidelity.co.uk`. There
|
||||
is no public API for individual members — the provider reverse-engineers the
|
||||
private JSON backend at `prd.wiciam.fidelity.co.uk/cvmfe/api/*` that the SPA
|
||||
itself calls, and uses Playwright only to keep a long-lived login session
|
||||
alive.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────┐ storage_state.json ┌──────────────────┐
|
||||
│ Vault KV │◀─── (quarterly reseed) ───│ fidelity-seed │
|
||||
│ broker-sync │ │ (headed browser) │
|
||||
└──────┬──────┘ └──────────────────┘
|
||||
│ ▲
|
||||
│ loads on start │ Viktor runs once
|
||||
▼ when session expires
|
||||
┌────────────────────┐
|
||||
│ Monthly CronJob │
|
||||
│ broker-sync-fidelity│
|
||||
└────────────┬────────┘
|
||||
│ headless Chromium
|
||||
▼
|
||||
┌─────────────────────────────────┐ ┌────────────────────────────────┐
|
||||
│ pv.planviewer.fidelity.co.uk │◀─────│ navigate dashboard → capture │
|
||||
│ (SPA) │ │ fresh sid/fid/tbid/rid headers │
|
||||
└─────────────────────────────────┘ └──────────────┬─────────────────┘
|
||||
│
|
||||
┌───────────▼─────────────┐
|
||||
│ httpx JSON calls │
|
||||
│ prd.wiciam.../cvmfe/api│
|
||||
└───────────┬─────────────┘
|
||||
│
|
||||
┌────────────────────▼────────────────────┐
|
||||
│ DEPOSIT × N (employee + employer) │
|
||||
│ BUY × N (fund unit purchases, per date) │
|
||||
└────────────────────┬────────────────────┘
|
||||
│
|
||||
┌────────────────▼────────────────┐
|
||||
│ Wealthfolio account │
|
||||
│ type = WORKPLACE_PENSION │
|
||||
│ currency = GBP │
|
||||
└──────────────────────────────────┘
|
||||
```
|
||||
|
||||
## One-time seed (Viktor)
|
||||
|
||||
```bash
|
||||
# on your laptop (macOS / Linux with a desktop):
|
||||
cd broker-sync
|
||||
poetry install
|
||||
poetry run playwright install chromium
|
||||
poetry run broker-sync fidelity-seed --out /tmp/fidelity_storage_state.json
|
||||
# chromium opens — log in to PlanViewer, tick "Remember device", press Enter
|
||||
|
||||
# stage to Vault
|
||||
vault kv patch secret/broker-sync \
|
||||
fidelity_storage_state=@/tmp/fidelity_storage_state.json \
|
||||
fidelity_plan_id=<your-plan-id>
|
||||
|
||||
rm /tmp/fidelity_storage_state.json # don't leave credentials lying around
|
||||
```
|
||||
|
||||
Re-seed when the monthly CronJob fails with `FidelitySessionError` (expect
|
||||
every 30-90 days, depending on how long Fidelity honours the remember-device
|
||||
cookie).
|
||||
|
||||
## One-time backfill
|
||||
|
||||
```bash
|
||||
kubectl -n broker-sync create job fidelity-backfill \
|
||||
--from=cronjob/broker-sync-fidelity
|
||||
kubectl -n broker-sync logs -f job/fidelity-backfill
|
||||
# expect: fidelity-ingest: fetched=N new=N imported=N failed=0
|
||||
```
|
||||
|
||||
## Monthly cron
|
||||
|
||||
- Schedule: `0 3 5 * *` (3am UTC on the 5th of each month — after mid-month payroll settles in Viktor's scheme)
|
||||
- CronJob: `broker-sync-fidelity` in namespace `broker-sync`
|
||||
- Resource: small, ≤512 MiB memory (Chromium for ~2 min, then idle)
|
||||
- Alert: `BrokerSyncFidelityFailed` fires on 2 consecutive failures
|
||||
|
||||
## Runbook — `BrokerSyncFidelityFailed`
|
||||
|
||||
1. Check pod logs: `kubectl -n broker-sync logs job/broker-sync-fidelity-<timestamp>`.
|
||||
2. If the error is `FidelitySessionError`: session expired, re-run the seed on
|
||||
Viktor's laptop (see above).
|
||||
3. If the error is a 404 / 5xx from `prd.wiciam.fidelity.co.uk`: likely an API
|
||||
path change. Check DevTools for the new endpoint, update the provider, ship
|
||||
a new image.
|
||||
4. If Playwright can't launch Chromium: check that the image still has Chromium
|
||||
installed (`playwright install chromium` at build time).
|
||||
|
||||
## Data model notes
|
||||
|
||||
- **Salary sacrifice scheme**: all employee + employer contributions are
|
||||
pre-tax from gross salary. No HMRC basic-rate relief line.
|
||||
- Emits two `DEPOSIT` per month (employee, employer) with `comment` carrying
|
||||
the source tag `fidelity:<doc-id>:<source>` for audit.
|
||||
- Emits one `BUY` per fund unit purchase, `symbol` = Fidelity fund code / ISIN.
|
||||
Units × unit price should reconcile to the cash deposited ±pennies.
|
||||
|
||||
## Not yet implemented
|
||||
|
||||
- Endpoint paths: waiting on Viktor's DevTools POST cURL for transactions +
|
||||
holdings views. Until pasted, `fidelity-ingest` raises
|
||||
`FidelityProviderConfigError` to fail loudly.
|
||||
- Infra: CronJob + Vault secret wiring + Prometheus alert in
|
||||
`infra/stacks/broker-sync/main.tf` — pending first successful manual run.
|
||||
Loading…
Add table
Add a link
Reference in a new issue