## Context
UK workplace pension at planviewer.fidelity.co.uk has no public API; the SPA
calls a private JSON backend at prd.wiciam.fidelity.co.uk/cvmfe/api/*. Viktor
confirmed in DevTools that an OPTIONS preflight lists auth headers
(ch, fid, rid, sid, tbid, theosreferer, ua). Full reverse-engineering of the
endpoint paths is pending Viktor's POST cURL paste for transactions +
holdings views.
Until those endpoints are captured, ship the scaffold: provider module, CLI
commands, tests, docs. This unblocks installing Playwright in the image and
lets Viktor run the one-off seed command on his laptop ahead of the data
integration.
## This change
- broker_sync/providers/fidelity_planviewer.py
- FidelityCreds namedtuple (storage_state_path, plan_id).
- FidelitySessionError (401 → re-seed), FidelityProviderConfigError.
- FidelityPlanViewerProvider: .accounts() returns a single
WORKPLACE_PENSION account, .fetch() raises until endpoints are wired.
- broker_sync/cli.py
- fidelity-seed: launches headed Chromium so Viktor can log in and tick
"Remember device", then dumps storage_state.json.
- fidelity-ingest: stub matching the invest-engine / trading212 CLI
shape; reads storage_state + plan_id, pipes through the shared pipeline.
- tests/providers/test_fidelity_planviewer.py
- Asserts the single-account shape + the loud-failure guard.
- docs/providers/fidelity-planviewer.md
- Architecture diagram, one-time seed procedure, backfill + monthly
commands, alert runbook.
- pyproject.toml
- playwright ^1.47 as a first-class dep (used only by fidelity-seed and
later by the session-refresh step in fidelity-ingest).
## What is NOT in this change
- Endpoint wiring in provider.fetch() — blocked on DevTools POST cURL.
- Infra CronJob + Vault secret + Prometheus alert — lands once the first
manual backfill succeeds and we know the Chromium image size is fine.
- Dockerfile Chromium install — same trigger.
## Verification
### Automated
$ poetry run pytest tests/providers/test_fidelity_planviewer.py -v
2 passed in 0.08s
$ poetry run pytest -q
122 passed, 1 skipped in 1.07s
$ poetry run mypy broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py
Success: no issues found in 2 source files
$ poetry run ruff check broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py tests/providers/test_fidelity_planviewer.py
All checks passed!
### Manual (Viktor, later)
1. poetry install && poetry run playwright install chromium
2. poetry run broker-sync fidelity-seed --out /tmp/state.json
3. Chromium opens → log in → tick "Remember device" → press Enter
4. vault kv patch secret/broker-sync fidelity_storage_state=@/tmp/state.json
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
finance.position (171 rows, 2020-06-07 to 2025-12-19) is the only source
of InvestEngine + Schwab trade history pre-dating the broker-sync project.
This provider reads it once and pushes every row into the correct WF
account (.L tickers → IE ISA, others → Schwab).
Dedup: external_id = 'finance-mysql:position:<PK>' — idempotent on re-run.
Auth: aiomysql as MySQL root (user-authorized) against the standalone
mysql:8.4 in-cluster service.
New CLI: broker-sync finance-mysql-import
New tests: 5 unit tests covering route, symbol normalise, BUY/SELL
detection.
poetry run pytest -q → 114 passed, 1 skipped
poetry run mypy → clean (aiomysql shielded with type: ignore)
poetry run ruff check → clean
Wires the IE + Schwab email parsers into an actual runnable sync. Walks
the IMAP mailbox, routes each message by sender domain:
- *@investengine.com → invest_engine.parse_invest_engine_email
- *@schwab.com → schwab.parse_schwab_email
then pushes the resulting Activities through the shared pipeline.
broker-sync imap-ingest — new CLI command taking IMAP_HOST/USER/PASSWORD/
DIRECTORY (mirrors the old wealthfolio-sync image's env shape so the
Terraform CronJob's existing env wiring works unchanged).
Verified: poetry run pytest -q → 109 passed + 1 skipped; mypy strict
clean (37 files); ruff + yapf clean.
Context: Phase 2b wiring — hand the bearer-token InvestEngineProvider
into the existing sync pipeline (sync_provider_to_wealthfolio), mirroring
the trading212 subcommand.
Environment contract:
WF_BASE_URL, WF_USERNAME, WF_PASSWORD, WF_SESSION_PATH (shared with trading212)
IE_BEARER_TOKEN (devtools-pasted)
IE_TOKEN_EXPIRES_AT (ISO-8601; Viktor sets on paste)
BROKER_SYNC_DATA_DIR (sync.db + checkpoint state)
Exit codes:
0 = clean run
1 = some rows failed to import (mirrors trading212 behaviour)
2 = token already expired per IE_TOKEN_EXPIRES_AT, or malformed ISO
timestamp, or live 401 response from IE (InvestEngineTokenExpiredError),
or unknown --mode flag
The pre-request expiry check is deliberate: a CronJob that runs during
the refresh window would otherwise waste a request on a dead token and
get the same 401 that we already know about from the clock. Exit 2
from the clock-only path also separates "token is old" from "wealthfolio
rejected a batch" in the CronJob alert pipeline.
Mode defaults:
--mode steady → since = now - 30d (bigger window than T212's 7d
because the IE sync only runs once a month in steady
state; 30d guarantees no gap even after a missed run)
--mode backfill → since = None (full history)
This change:
- `invest-engine` subcommand added to broker_sync/cli.py
- Token-expiry pre-check (clock), IE_TOKEN_EXPIRES_AT ISO parsing with
a UTC default for naive timestamps, and graceful handling of
InvestEngineTokenExpiredError surfaced during pipeline run
- 3 new tests in tests/test_cli.py covering the 3 exit-2 paths
## Automated
poetry run pytest tests/test_cli.py -v
======================== 4 passed in 0.28s =========================
poetry run pytest -q
98 passed, 1 skipped in 0.85s
poetry run mypy --strict .
Success: no issues found in 34 source files
poetry run ruff check .
All checks passed!
## Manual Verification
1. Populate Vault keys per the docstring in
broker_sync/providers/invest_engine.py (Viktor pastes token + sets
expires_at to the Monday morning of next month).
2. Set env:
export WF_BASE_URL=https://wealthfolio.viktorbarzin.me
export WF_USERNAME=viktor
export WF_PASSWORD=<from Vault>
export IE_BEARER_TOKEN=<from Vault>
export IE_TOKEN_EXPIRES_AT=<from Vault>
export BROKER_SYNC_DATA_DIR=/tmp/ie-smoke
3. poetry run broker-sync invest-engine --mode backfill
Expected: single line "invest-engine: fetched=N new=M imported=M failed=0"
on success; exit 2 with "InvestEngine token expired..." if the clock
or server disagrees; exit 2 with "IE_TOKEN_EXPIRES_AT not a valid
ISO-8601 timestamp..." if the env var is malformed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Context
-------
Live run of `broker-sync trading212` hit a PermissionError and typer's
rich traceback printed every local variable, including the cleartext
WF_PASSWORD and the T212 api_key strings, into pod logs. Kubernetes
pod logs are world-readable cluster-wide — that's a security incident.
This change
-----------
- Pass `pretty_exceptions_enable=False` to the typer.Typer constructor.
Plain stdlib tracebacks don't dump frame locals.
- Rich is still available for help text; only crash formatting changes.
Follow-up in infra/stacks/broker-sync: add `security_context.fs_group = 10001`
to every pod spec so the PVC is owned by the broker user (the original
PermissionError that triggered the traceback was the broker user being
unable to write /data/watermarks).
Test plan
---------
## Automated
- poetry run pytest -q → 70 passed
- poetry run mypy broker_sync tests → clean
- poetry run ruff check . → clean
## Manual Verification
Re-run the backfill Job after the image is rebuilt + the infra
fsGroup change is applied.
Context
-------
Closes the gap between "Trading212 provider yields Activities" and
"activities land in Wealthfolio with dedup". One generic pipeline
function works for every provider (Phase 2 IMAP ingest and Phase 3
CSV drop will reuse it).
This change
-----------
- `broker_sync/pipeline.py` — sync_provider_to_wealthfolio():
ensure accounts exist in Wealthfolio, fetch, dedup against the local
SQLite store, batch into Wealthfolio's CSV import at 200 rows each,
record successful imports in the dedup store with the returned
Wealthfolio activity id. Failed batches don't touch the dedup store
so the next run retries.
- Notes field stamped with `sync:<provider>:<external_id>` for human
auditability — NOT used for dedup (the SQLite store owns that).
- `broker_sync/cli.py` — new `trading212` subcommand driven by
T212_API_KEYS_JSON + WF_* + BROKER_SYNC_DATA_DIR env vars. Two modes:
`steady` fetches last 7 days; `backfill` pulls all history. Exits 0
on clean run, 1 if any batch failed, 2 on config errors.
- Pipeline tests with MockTransport: dedup-skip-then-import happy path
(verifies imported CSV contains only the unseen rows and all three
are recorded after the run); import-rejected path (verifies the
failed row is NOT recorded so the next run retries).
Test plan
---------
## Automated
- poetry run pytest -q → 70 passed
- poetry run mypy broker_sync tests → Success: no issues found in 29 source files
- poetry run ruff check . → All checks passed!
- poetry run broker-sync trading212 --help → shows all env vars + mode flag
## Manual Verification
Live smoke test blocked on:
1. Vault secret/broker-sync seeded (wf_base_url, wf_username, wf_password,
trading212_api_keys).
2. Terraform stack applied (infra/stacks/broker-sync/ — staged, not yet applied).
3. Image pushed to viktorbarzin/broker-sync on DockerHub via GHA.
Once those land:
kubectl -n broker-sync create job t212-backfill \
--from=cronjob/broker-sync-trading212 -- \
broker-sync trading212 --mode=backfill
Context
-------
Closes Phase 0 scaffolding. Image must build and run so infra can
schedule an initial no-op CronJob (the plan's Phase 0 exit criterion)
while Phase 0.5 / 0.75 / 1 land.
This change
-----------
- broker_sync/cli.py: typer app with two commands.
* `version` — prints __version__; used as the no-op CronJob
liveness check.
* `auth-spike` — Phase 0.5 end-to-end live probe: log in to
Wealthfolio, list accounts, exit 0 on success. Credentials read
from env (WF_BASE_URL/USERNAME/PASSWORD) so CronJob + ESO can
inject them without CLI flags.
- Dockerfile: multi-stage, Python 3.12-slim, non-root user 10001
with /data as the shared PVC mount. Poetry virtualenv baked into
/app/.venv, entrypoint is `broker-sync`, default command `version`.
- CLI test via typer.testing.CliRunner.
Test plan
---------
## Automated
- poetry run pytest -q → 32 passed
- poetry run mypy broker_sync tests → Success: no issues found in 19 source files
- poetry run ruff check . → All checks passed!
- poetry run broker-sync version → broker-sync 0.1.0
## Manual Verification
Docker build + run deferred — image will be built via GHA after the
repo is pushed to GitHub in a follow-up session; the pyproject install
has already been verified locally.
