Context
-------
Phase 1 needs live FX rates for USD-denominated RSU vestings (Schwab),
EUR-denominated deposits, and any multi-currency dividend. The FxCache
from an earlier commit stores (currency, date) → rate_to_gbp but was
intentionally left empty — this commit wires the ingestion path.
Design decisions
----------------
- ECB publishes EUR→X, not X→GBP. Everything pivots through EUR:
rate(X→GBP) = rate(EUR→GBP) / rate(EUR→X)
GBP goes into the result at 1.0 so callers iterating the dict get
a consistent shape; `populate_fx_cache` then skips GBP because
`convert_to_gbp` has a dedicated passthrough branch.
- `on_date` parameter is accepted for API symmetry with the future
historical fetcher even though the daily endpoint only serves the
most recent publication. The docstring calls this out explicitly.
- XML is parsed with stdlib `xml.etree.ElementTree`. No `lxml` —
the file is 30 lines, no performance concern, and keeping deps
minimal matters for the container image.
- The HTTP layer takes an optional `httpx.AsyncBaseTransport` the same
way WealthfolioSink does — MockTransport drives all tests, the
production caller just leaves it None.
This change
-----------
- broker_sync/fx_ecb.py:
* `fetch_ecb_rates(on_date, *, transport=None)` — GETs the daily
XML, parses, pivots through EUR, returns `{ccy: rate_to_gbp}`.
Raises `RuntimeError` on non-2xx or if GBP is missing (cannot
pivot). No retries — caller handles resilience.
* `populate_fx_cache(cache, rates, on_date)` — writes every
non-GBP rate with `FxRateSource.ECB_LIVE`.
* `fetch_ecb_rates_historical(start, end)` — `NotImplementedError`
stub; filed as beads task code-thw.2.2. Needed for backfilling
years of T212 history (daily endpoint only covers today).
- tests/fixtures/ecb_2026-04-01.xml: realistic 5-currency ECB snapshot.
- tests/test_fx_ecb.py: fixture-driven tests covering the pivot math,
the 503 failure path, the cache-skip-GBP rule, and the NotImplemented
guard on the historical stub.
Test plan
---------
## Automated
- poetry run pytest -q → 52 passed in 0.50s
- poetry run mypy broker_sync tests → Success: no issues found in 26 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Live daily endpoint hit deferred to the CLI integration commit — the
fetcher is pure + transport-injectable, so the unit tests cover the
parsing and pivot logic, and the CLI wiring will be the place where
the live call is exercised end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Context
-------
Trading212's `/equity/history/orders` is cursor-paginated via a
`nextPagePath` query-param in each response. Steady-state runs must
resume where the previous run finished, or we either miss fills (if
we start from 'now') or waste the 6/min rate limit walking history
we already imported (if we start from epoch).
A shared checkpoint store must live alongside the SyncRecordStore's
dedup DB on the /data PVC so CronJob pods can see progress from the
previous invocation. One file per (provider, account_id) because:
- T212 issues one API key per wrapper — ISA + Invest share no data.
- Plain JSON files are trivial to hand-edit during backfill if a
resume cursor gets stuck at a bad point.
This change
-----------
- broker_sync/providers/_checkpoint.py: `Checkpoint(dir, provider,
account_id)` with `load() -> str | None` and `save(cursor)`. Writes
`{cursor, updated_at}` to `<provider>-<account_id>.json`. Creates
parent directory lazily on first save so the PVC only needs a
mountpoint, not a pre-seeded layout.
- Provider-agnostic: no T212 knowledge. Will be reused for
InvestEngine in Phase 2.
- tests/providers/test_checkpoint.py: roundtrip, filename shape,
overwrite, per-account isolation, parent-dir creation, and a
malformed-file fallback (returns None rather than raising) so a
manual edit gone wrong does not brick the CronJob.
Test plan
---------
## Automated
- poetry run pytest -q → 48 passed in 0.47s
- poetry run mypy broker_sync tests → Success: no issues found in 24 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Not applicable — pure local-filesystem helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Context
-------
Phase 1 kickoff: Trading212 tags every ticker with `_EQ`, sometimes
preceded by a lowercase exchange letter ("l" = LSE) or `_US`. Raw
symbols like `VUAGl_EQ` are an implementation detail that would leak
into Wealthfolio and diverge from other providers (InvestEngine and
Schwab emit `VUAG` / `META`). The canonical form has to match across
providers so portfolio aggregation lines up.
Unlike the finance/ reference code, we do NOT restrict to a
SUPPORTED_TICKERS allowlist here — Wealthfolio is the source of truth,
everything gets imported, and the user decides what to track.
This change
-----------
- broker_sync/providers/trading212.py: pure `_normalise_ticker`
helper backed by a single regex that peels `(_US)?[a-z]?_EQ`. No
lookup tables — the rule covers all observed shapes.
- tests/providers/test_trading212_ticker.py: parametrised cases for
every mapping called out in the Phase 1 plan plus pass-through of
already-canonical symbols.
Test plan
---------
## Automated
- poetry run pytest -q → 41 passed in 0.46s
- poetry run mypy broker_sync tests → Success: no issues found in 22 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Not applicable — pure function, no external side effects.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Context
-------
Closes Phase 0 scaffolding. Image must build and run so infra can
schedule an initial no-op CronJob (the plan's Phase 0 exit criterion)
while Phase 0.5 / 0.75 / 1 land.
This change
-----------
- broker_sync/cli.py: typer app with two commands.
* `version` — prints __version__; used as the no-op CronJob
liveness check.
* `auth-spike` — Phase 0.5 end-to-end live probe: log in to
Wealthfolio, list accounts, exit 0 on success. Credentials read
from env (WF_BASE_URL/USERNAME/PASSWORD) so CronJob + ESO can
inject them without CLI flags.
- Dockerfile: multi-stage, Python 3.12-slim, non-root user 10001
with /data as the shared PVC mount. Poetry virtualenv baked into
/app/.venv, entrypoint is `broker-sync`, default command `version`.
- CLI test via typer.testing.CliRunner.
Test plan
---------
## Automated
- poetry run pytest -q → 32 passed
- poetry run mypy broker_sync tests → Success: no issues found in 19 source files
- poetry run ruff check . → All checks passed!
- poetry run broker-sync version → broker-sync 0.1.0
## Manual Verification
Docker build + run deferred — image will be built via GHA after the
repo is pushed to GitHub in a follow-up session; the pyproject install
has already been verified locally.
Context
-------
This is the Phase 0.5 deliverable — the hardest-to-validate unknown
in the plan. Wealthfolio auth is JWT HttpOnly cookie with a 5-req/min
login rate limit. CronJob pods are ephemeral, so we persist cookies
to disk between runs (shared PVC in production).
Plan stress-test also flagged: use the CSV import path, not per-row
JSON POST. Wealthfolio's UI uses /activities/import and its dedup
logic is battle-tested; CSVs double as audit artefacts we can replay.
This change
-----------
- WealthfolioSink (httpx async): login with username/password, persists
cookie dict to session_path on disk, attaches it as a Cookie header
on subsequent calls.
- 401 on a non-login endpoint triggers a single re-login + retry.
- ensure_account() is idempotent — GETs the account list first, only
POSTs /accounts if id is missing.
- import_activities() always runs /activities/import/check first; any
non-2xx there raises ImportValidationError and we never touch the
real import endpoint. Protects against half-written state when the
broker emits a symbol Wealthfolio doesn't know.
- httpx.MockTransport-based tests cover: login persistence, 401 on
login raises UnauthorizedError, session reuse from disk, 401 retry
path, ensure_account idempotency + creation, import dry-run-then-real
sequencing, halt on check failure.
Not yet covered (deferred):
- Multi-process file lock on session_path (single-process enough for
now; Phase 1 adds it when multiple CronJobs run concurrently).
- 429 jittered backoff (TBD when Wealthfolio actually rate-limits us).
Test plan
---------
## Automated
- poetry run pytest -q → 31 passed
- poetry run mypy broker_sync tests → Success: no issues found in 17 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Live auth spike against https://wealthfolio.viktorbarzin.me deferred
until the password is seeded into Vault at secret/broker-sync/wealthfolio
in a follow-up commit (needs Viktor's Vault session).
Context
-------
Every broker connector needs a uniform shape so the orchestrator can
fan out without knowing provider-specific details. Normalisation (GBP
conversion) lives outside providers on purpose — keeping providers
native-currency-emitters means we can re-normalise historical activity
when HMRC rates land without re-fetching from the broker.
This change
-----------
- providers/base.py: Provider Protocol with `accounts()` and async
`fetch(since, before)` iterator. No abstract base class — duck-typed
Protocol so each concrete provider stays independent.
- normaliser.py: takes a native Activity + FxCache, returns a copy
with amount_gbp/fx_rate_gbp/fx_rate_source filled in. Two modes:
qty*price for BUY/SELL, amount for DIVIDEND/DEPOSIT/etc.
- Namespace packages for providers/, providers/parsers/, sinks/ so
future modules slot in cleanly.
Test plan
---------
## Automated
- poetry run pytest -q → 23 passed
- poetry run mypy broker_sync tests → Success: no issues found in 14 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Not applicable at this layer.
Context
-------
FX for UK users has two lives: live ECB rates for portfolio display
(available same-day), and HMRC monthly/daily rates for CGT basis
(published after month-end). The plan keeps both in one cache table
with an upgradable `source` column, so a later reconciliation job can
replace ECB_LIVE values with HMRC_MONTHLY for the same date without
schema work.
This change
-----------
- FxCache: SQLite table (currency, on_date) -> (rate_gbp, source) with
ON CONFLICT UPDATE semantics so reconciliation is a single put().
- convert_to_gbp(): GBP short-circuits to identity; any other currency
must be in the cache (network fetch is the caller's responsibility,
separately implemented by the ECB and HMRC fetchers).
- Explicit LookupError on cache miss — deliberate, we do NOT want a
silent fallback that produces wrong cost-basis numbers.
Decisions deferred to later commits:
- Actual ECB daily reference-rate fetcher (eurofxref XML) — lands with
the Trading212 provider in Phase 1 when non-GBP trades first appear.
- HMRC monthly-rate fetcher + reconciliation CronJob — Phase 1 tail.
Test plan
---------
## Automated
- poetry run pytest tests/test_fx.py -v → 6 passed
- poetry run mypy broker_sync tests → Success: no issues found in 8 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Not applicable — no network yet.
Context
-------
Wealthfolio's activity `notes` field is user-editable via the UI, so
using it as the dedup key would let a single note-edit in Wealthfolio
cause the next sync to create a duplicate. Stress-testing the plan
flagged this as the top structural risk.
This change
-----------
- SQLite-backed store at `/data/broker_sync.db` in production; keyed on
(provider, account, external_id) so each provider's id space is
scoped to its own account.
- `INSERT OR IGNORE` makes record() idempotent — second call with the
same key is a no-op and preserves the original wealthfolio_activity_id
plus first_seen timestamp.
- `filter_new()` is the integration point: provider fetches activities,
hands them to the store, gets back only the unseen subset to submit
to the Wealthfolio sink.
- Wealthfolio activity id returned by the API is persisted alongside
each record so the HMRC FX reconciliation job can later PATCH the
original activity rather than creating a new one.
Test plan
---------
## Automated
- poetry run pytest tests/test_dedup.py -v → 6 passed
- poetry run mypy broker_sync tests → Success: no issues found in 6 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Not applicable for this layer — full end-to-end verification happens
once a provider + sink land (Phase 1 Trading212 and the auth spike).
Context
-------
New connector suite that syncs UK brokerage activity (Trading212,
InvestEngine, Schwab email-parsed, CSV drop-folder) into Wealthfolio.
Lives outside finance/ intentionally — finance/ is untouched per the
plan at ~/.claude/plans/let-s-work-on-linking-temporal-valiant.md.
This change
-----------
- Poetry project with httpx, typer, bs4, dev tools (pytest, mypy strict,
ruff, yapf).
- Canonical Activity + Account models with the 6 UK tax wrappers
(ISA/SIPP/GIA/LISA/JISA/WORKPLACE_PENSION) and the 12 Wealthfolio
activity types from docs/activities/activity-types.md on the upstream.
- Validation invariants: BUY/SELL need qty+price, DIVIDEND/DEPOSIT/etc
need amount — raises early so providers can't silently emit broken
rows.
- to_wealthfolio_csv_row() shape matches Wealthfolio's CSV import;
primary sink path per the plan.
Test plan
---------
## Automated
- poetry run pytest -q → 7 passed in 0.03s
- poetry run mypy broker_sync tests → Success: no issues found in 4 source files
- poetry run ruff check . → All checks passed!
- poetry run yapf --diff --recursive broker_sync tests → no diff
## Manual Verification
Not applicable — pure data model, no runtime behaviour.
Closes: code-thw.1
