Commit graph

5 commits

Author SHA1 Message Date
Viktor Barzin
832732a419 fidelity-planviewer: scaffold provider + CLI (seed + stub ingest)
## Context

UK workplace pension at planviewer.fidelity.co.uk has no public API; the SPA
calls a private JSON backend at prd.wiciam.fidelity.co.uk/cvmfe/api/*. Viktor
confirmed in DevTools that an OPTIONS preflight lists auth headers
(ch, fid, rid, sid, tbid, theosreferer, ua). Full reverse-engineering of the
endpoint paths is pending Viktor's POST cURL paste for transactions +
holdings views.

Until those endpoints are captured, ship the scaffold: provider module, CLI
commands, tests, docs. This unblocks installing Playwright in the image and
lets Viktor run the one-off seed command on his laptop ahead of the data
integration.

## This change

- broker_sync/providers/fidelity_planviewer.py
  - FidelityCreds namedtuple (storage_state_path, plan_id).
  - FidelitySessionError (401 → re-seed), FidelityProviderConfigError.
  - FidelityPlanViewerProvider: .accounts() returns a single
    WORKPLACE_PENSION account, .fetch() raises until endpoints are wired.
- broker_sync/cli.py
  - fidelity-seed: launches headed Chromium so Viktor can log in and tick
    "Remember device", then dumps storage_state.json.
  - fidelity-ingest: stub matching the invest-engine / trading212 CLI
    shape; reads storage_state + plan_id, pipes through the shared pipeline.
- tests/providers/test_fidelity_planviewer.py
  - Asserts the single-account shape + the loud-failure guard.
- docs/providers/fidelity-planviewer.md
  - Architecture diagram, one-time seed procedure, backfill + monthly
    commands, alert runbook.
- pyproject.toml
  - playwright ^1.47 as a first-class dep (used only by fidelity-seed and
    later by the session-refresh step in fidelity-ingest).

## What is NOT in this change

- Endpoint wiring in provider.fetch() — blocked on DevTools POST cURL.
- Infra CronJob + Vault secret + Prometheus alert — lands once the first
  manual backfill succeeds and we know the Chromium image size is fine.
- Dockerfile Chromium install — same trigger.

## Verification

### Automated

$ poetry run pytest tests/providers/test_fidelity_planviewer.py -v
2 passed in 0.08s

$ poetry run pytest -q
122 passed, 1 skipped in 1.07s

$ poetry run mypy broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py
Success: no issues found in 2 source files

$ poetry run ruff check broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py tests/providers/test_fidelity_planviewer.py
All checks passed!

### Manual (Viktor, later)

1. poetry install && poetry run playwright install chromium
2. poetry run broker-sync fidelity-seed --out /tmp/state.json
3. Chromium opens → log in → tick "Remember device" → press Enter
4. vault kv patch secret/broker-sync fidelity_storage_state=@/tmp/state.json

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 14:09:04 +00:00
Viktor Barzin
a190875f63 Add finance_mysql provider + CLI for historical backfill
finance.position (171 rows, 2020-06-07 to 2025-12-19) is the only source
of InvestEngine + Schwab trade history pre-dating the broker-sync project.
This provider reads it once and pushes every row into the correct WF
account (.L tickers → IE ISA, others → Schwab).

Dedup: external_id = 'finance-mysql:position:<PK>' — idempotent on re-run.
Auth: aiomysql as MySQL root (user-authorized) against the standalone
mysql:8.4 in-cluster service.

New CLI: broker-sync finance-mysql-import
New tests: 5 unit tests covering route, symbol normalise, BUY/SELL
detection.

poetry run pytest -q   →  114 passed, 1 skipped
poetry run mypy        →  clean (aiomysql shielded with type: ignore)
poetry run ruff check  →  clean
2026-04-17 22:38:21 +00:00
Viktor Barzin
f089b8b93a Add Schwab email parser (port from finance/)
Schwab's workplace-RSU confirmation emails have 5 data td elements
with class='dark-background-body' align='right': date, direction, qty,
ticker, price-with-currency-sign. One email → one Activity.

- parse_schwab_email(raw_html) -> list[Activity] (1-item or empty)
- Empty on any parse failure (IMAP batch shouldn't crash on one bad mail)
- Deterministic external_id ('schwab📅ticker:type:qty') — stable
  across re-pulls so dedup works
- Hardcoded to account 'schwab-workplace' / AccountType.GIA / USD
- 6 unit tests: SELL + BUY happy path, malformed, missing cells,
  external-id stability, commas in price

Dropped from the original finance port:
- msg_timestamp-based external id (non-deterministic — would re-import
  on every IMAP walk). Replaced with a hash-stable key.
- Currency.from_sign() currency hack. Schwab US is USD-only; we'll add
  FX when that changes.

poetry run pytest -q   →  109 passed, 1 skipped
poetry run mypy        →  clean (added types-python-dateutil)
poetry run ruff check  →  clean
2026-04-17 22:08:40 +00:00
Viktor Barzin
1eb3f78ea5 Wire T212 pagination, retries, and click<8.2 pin
Context
-------
Closes out the Trading212 provider's retry + pagination surface so
the "Add Trading212Provider core fetch" commit has everything the
CronJob needs: cursor-based pagination, 429 honouring Retry-After,
jittered exponential backoff for 429-without-header and 5xx, bailout
after _MAX_RETRIES, and checkpoint-after-page semantics so a crashed
run resumes at the start of the unfinished page.

Also pins click<8.2 — typer 0.12 calls Parameter.make_metavar()
without a ctx argument, which click 8.2 removed; `broker-sync --help`
was crashing with TypeError until this pin. typer 0.15+ would also
fix it; the pin is lower friction.

One test fix: test_checkpoint_advances_only_after_page_yielded had a
handler that unconditionally returned a next_path → infinite loop. The
assertion was always about "a cursor was saved after page 1", so I
changed the handler to return page 2 as empty-with-no-next, which
terminates the loop cleanly.

Test plan
---------
## Automated
- poetry run pytest -q  →  70 passed
- poetry run mypy broker_sync tests  →  Success: no issues found in 29 source files
- poetry run ruff check .  →  All checks passed!
- poetry run broker-sync --help  →  renders without crash; lists version + auth-spike

## Manual Verification
End-to-end against a live T212 key is in the next commit once the
CLI subcommand and pipeline land.
2026-04-17 19:45:23 +00:00
Viktor Barzin
a2aa7ec486 Initial scaffold + canonical Activity model
Context
-------
New connector suite that syncs UK brokerage activity (Trading212,
InvestEngine, Schwab email-parsed, CSV drop-folder) into Wealthfolio.
Lives outside finance/ intentionally — finance/ is untouched per the
plan at ~/.claude/plans/let-s-work-on-linking-temporal-valiant.md.

This change
-----------
- Poetry project with httpx, typer, bs4, dev tools (pytest, mypy strict,
  ruff, yapf).
- Canonical Activity + Account models with the 6 UK tax wrappers
  (ISA/SIPP/GIA/LISA/JISA/WORKPLACE_PENSION) and the 12 Wealthfolio
  activity types from docs/activities/activity-types.md on the upstream.
- Validation invariants: BUY/SELL need qty+price, DIVIDEND/DEPOSIT/etc
  need amount — raises early so providers can't silently emit broken
  rows.
- to_wealthfolio_csv_row() shape matches Wealthfolio's CSV import;
  primary sink path per the plan.

Test plan
---------
## Automated
- poetry run pytest -q  →  7 passed in 0.03s
- poetry run mypy broker_sync tests  →  Success: no issues found in 4 source files
- poetry run ruff check .  →  All checks passed!
- poetry run yapf --diff --recursive broker_sync tests  →  no diff

## Manual Verification
Not applicable — pure data model, no runtime behaviour.

Closes: code-thw.1
2026-04-17 19:16:11 +00:00