Context
-------
Phase 1 needs live FX rates for USD-denominated RSU vestings (Schwab),
EUR-denominated deposits, and any multi-currency dividend. The FxCache
from an earlier commit stores (currency, date) → rate_to_gbp but was
intentionally left empty — this commit wires the ingestion path.
Design decisions
----------------
- ECB publishes EUR→X, not X→GBP. Everything pivots through EUR:
rate(X→GBP) = rate(EUR→GBP) / rate(EUR→X)
GBP goes into the result at 1.0 so callers iterating the dict get
a consistent shape; `populate_fx_cache` then skips GBP because
`convert_to_gbp` has a dedicated passthrough branch.
- `on_date` parameter is accepted for API symmetry with the future
historical fetcher even though the daily endpoint only serves the
most recent publication. The docstring calls this out explicitly.
- XML is parsed with stdlib `xml.etree.ElementTree`. No `lxml` —
the file is 30 lines, no performance concern, and keeping deps
minimal matters for the container image.
- The HTTP layer takes an optional `httpx.AsyncBaseTransport` the same
way WealthfolioSink does — MockTransport drives all tests, the
production caller just leaves it None.
This change
-----------
- broker_sync/fx_ecb.py:
* `fetch_ecb_rates(on_date, *, transport=None)` — GETs the daily
XML, parses, pivots through EUR, returns `{ccy: rate_to_gbp}`.
Raises `RuntimeError` on non-2xx or if GBP is missing (cannot
pivot). No retries — caller handles resilience.
* `populate_fx_cache(cache, rates, on_date)` — writes every
non-GBP rate with `FxRateSource.ECB_LIVE`.
* `fetch_ecb_rates_historical(start, end)` — `NotImplementedError`
stub; filed as beads task code-thw.2.2. Needed for backfilling
years of T212 history (daily endpoint only covers today).
- tests/fixtures/ecb_2026-04-01.xml: realistic 5-currency ECB snapshot.
- tests/test_fx_ecb.py: fixture-driven tests covering the pivot math,
the 503 failure path, the cache-skip-GBP rule, and the NotImplemented
guard on the historical stub.
Test plan
---------
## Automated
- poetry run pytest -q → 52 passed in 0.50s
- poetry run mypy broker_sync tests → Success: no issues found in 26 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Live daily endpoint hit deferred to the CLI integration commit — the
fetcher is pure + transport-injectable, so the unit tests cover the
parsing and pivot logic, and the CLI wiring will be the place where
the live call is exercised end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Context
-------
Matches the pattern used by claude-memory-mcp (infra CLAUDE.md §CI/CD).
GHA is cheap and parallel — build+push happens there. Woodpecker runs
in-cluster and has kubectl access, so it owns the `kubectl set image`
step.
This change
-----------
- `.github/workflows/ci.yml` — push to main runs ruff + mypy strict +
pytest, then builds `viktorbarzin/broker-sync:<8-char-sha>` +
`:latest` and pushes to DockerHub, then triggers Woodpecker.
- `.woodpecker/deploy.yml` — kubectl set image on all 5 CronJobs in
the broker-sync namespace (version probe + 4 sync jobs), then
spawns a one-shot Job from the version CronJob as a smoke test
and waits for completion.
- Woodpecker repo ID is `TBD` — needs filling in once the repo is
registered with Woodpecker (see infra CLAUDE.md Repo IDs list).
The workflow skips deploy cleanly if still TBD, so this doesn't
block green builds.
Test plan
---------
## Automated
Nothing to run locally — CI is verified by pushing and watching the
run on GitHub.
## Manual Verification
1. Push this branch to GitHub, confirm `test` job runs and passes.
2. Push to `main`, confirm `build` job produces
`viktorbarzin/broker-sync:<sha>` on DockerHub.
3. Register repo with Woodpecker, note the numeric repo ID, replace
`TBD` in ci.yml, push again.
4. Confirm Woodpecker deploy pipeline runs `kubectl set image` on
all 5 CronJobs and the smoke-test job returns `broker-sync 0.1.0`.
Context
-------
Trading212's `/equity/history/orders` is cursor-paginated via a
`nextPagePath` query-param in each response. Steady-state runs must
resume where the previous run finished, or we either miss fills (if
we start from 'now') or waste the 6/min rate limit walking history
we already imported (if we start from epoch).
A shared checkpoint store must live alongside the SyncRecordStore's
dedup DB on the /data PVC so CronJob pods can see progress from the
previous invocation. One file per (provider, account_id) because:
- T212 issues one API key per wrapper — ISA + Invest share no data.
- Plain JSON files are trivial to hand-edit during backfill if a
resume cursor gets stuck at a bad point.
This change
-----------
- broker_sync/providers/_checkpoint.py: `Checkpoint(dir, provider,
account_id)` with `load() -> str | None` and `save(cursor)`. Writes
`{cursor, updated_at}` to `<provider>-<account_id>.json`. Creates
parent directory lazily on first save so the PVC only needs a
mountpoint, not a pre-seeded layout.
- Provider-agnostic: no T212 knowledge. Will be reused for
InvestEngine in Phase 2.
- tests/providers/test_checkpoint.py: roundtrip, filename shape,
overwrite, per-account isolation, parent-dir creation, and a
malformed-file fallback (returns None rather than raising) so a
manual edit gone wrong does not brick the CronJob.
Test plan
---------
## Automated
- poetry run pytest -q → 48 passed in 0.47s
- poetry run mypy broker_sync tests → Success: no issues found in 24 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Not applicable — pure local-filesystem helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Context
-------
Phase 1 kickoff: Trading212 tags every ticker with `_EQ`, sometimes
preceded by a lowercase exchange letter ("l" = LSE) or `_US`. Raw
symbols like `VUAGl_EQ` are an implementation detail that would leak
into Wealthfolio and diverge from other providers (InvestEngine and
Schwab emit `VUAG` / `META`). The canonical form has to match across
providers so portfolio aggregation lines up.
Unlike the finance/ reference code, we do NOT restrict to a
SUPPORTED_TICKERS allowlist here — Wealthfolio is the source of truth,
everything gets imported, and the user decides what to track.
This change
-----------
- broker_sync/providers/trading212.py: pure `_normalise_ticker`
helper backed by a single regex that peels `(_US)?[a-z]?_EQ`. No
lookup tables — the rule covers all observed shapes.
- tests/providers/test_trading212_ticker.py: parametrised cases for
every mapping called out in the Phase 1 plan plus pass-through of
already-canonical symbols.
Test plan
---------
## Automated
- poetry run pytest -q → 41 passed in 0.46s
- poetry run mypy broker_sync tests → Success: no issues found in 22 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Not applicable — pure function, no external side effects.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Context
-------
Closes Phase 0 scaffolding. Image must build and run so infra can
schedule an initial no-op CronJob (the plan's Phase 0 exit criterion)
while Phase 0.5 / 0.75 / 1 land.
This change
-----------
- broker_sync/cli.py: typer app with two commands.
* `version` — prints __version__; used as the no-op CronJob
liveness check.
* `auth-spike` — Phase 0.5 end-to-end live probe: log in to
Wealthfolio, list accounts, exit 0 on success. Credentials read
from env (WF_BASE_URL/USERNAME/PASSWORD) so CronJob + ESO can
inject them without CLI flags.
- Dockerfile: multi-stage, Python 3.12-slim, non-root user 10001
with /data as the shared PVC mount. Poetry virtualenv baked into
/app/.venv, entrypoint is `broker-sync`, default command `version`.
- CLI test via typer.testing.CliRunner.
Test plan
---------
## Automated
- poetry run pytest -q → 32 passed
- poetry run mypy broker_sync tests → Success: no issues found in 19 source files
- poetry run ruff check . → All checks passed!
- poetry run broker-sync version → broker-sync 0.1.0
## Manual Verification
Docker build + run deferred — image will be built via GHA after the
repo is pushed to GitHub in a follow-up session; the pyproject install
has already been verified locally.
Context
-------
This is the Phase 0.5 deliverable — the hardest-to-validate unknown
in the plan. Wealthfolio auth is JWT HttpOnly cookie with a 5-req/min
login rate limit. CronJob pods are ephemeral, so we persist cookies
to disk between runs (shared PVC in production).
Plan stress-test also flagged: use the CSV import path, not per-row
JSON POST. Wealthfolio's UI uses /activities/import and its dedup
logic is battle-tested; CSVs double as audit artefacts we can replay.
This change
-----------
- WealthfolioSink (httpx async): login with username/password, persists
cookie dict to session_path on disk, attaches it as a Cookie header
on subsequent calls.
- 401 on a non-login endpoint triggers a single re-login + retry.
- ensure_account() is idempotent — GETs the account list first, only
POSTs /accounts if id is missing.
- import_activities() always runs /activities/import/check first; any
non-2xx there raises ImportValidationError and we never touch the
real import endpoint. Protects against half-written state when the
broker emits a symbol Wealthfolio doesn't know.
- httpx.MockTransport-based tests cover: login persistence, 401 on
login raises UnauthorizedError, session reuse from disk, 401 retry
path, ensure_account idempotency + creation, import dry-run-then-real
sequencing, halt on check failure.
Not yet covered (deferred):
- Multi-process file lock on session_path (single-process enough for
now; Phase 1 adds it when multiple CronJobs run concurrently).
- 429 jittered backoff (TBD when Wealthfolio actually rate-limits us).
Test plan
---------
## Automated
- poetry run pytest -q → 31 passed
- poetry run mypy broker_sync tests → Success: no issues found in 17 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Live auth spike against https://wealthfolio.viktorbarzin.me deferred
until the password is seeded into Vault at secret/broker-sync/wealthfolio
in a follow-up commit (needs Viktor's Vault session).
Context
-------
Every broker connector needs a uniform shape so the orchestrator can
fan out without knowing provider-specific details. Normalisation (GBP
conversion) lives outside providers on purpose — keeping providers
native-currency-emitters means we can re-normalise historical activity
when HMRC rates land without re-fetching from the broker.
This change
-----------
- providers/base.py: Provider Protocol with `accounts()` and async
`fetch(since, before)` iterator. No abstract base class — duck-typed
Protocol so each concrete provider stays independent.
- normaliser.py: takes a native Activity + FxCache, returns a copy
with amount_gbp/fx_rate_gbp/fx_rate_source filled in. Two modes:
qty*price for BUY/SELL, amount for DIVIDEND/DEPOSIT/etc.
- Namespace packages for providers/, providers/parsers/, sinks/ so
future modules slot in cleanly.
Test plan
---------
## Automated
- poetry run pytest -q → 23 passed
- poetry run mypy broker_sync tests → Success: no issues found in 14 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Not applicable at this layer.
Context
-------
FX for UK users has two lives: live ECB rates for portfolio display
(available same-day), and HMRC monthly/daily rates for CGT basis
(published after month-end). The plan keeps both in one cache table
with an upgradable `source` column, so a later reconciliation job can
replace ECB_LIVE values with HMRC_MONTHLY for the same date without
schema work.
This change
-----------
- FxCache: SQLite table (currency, on_date) -> (rate_gbp, source) with
ON CONFLICT UPDATE semantics so reconciliation is a single put().
- convert_to_gbp(): GBP short-circuits to identity; any other currency
must be in the cache (network fetch is the caller's responsibility,
separately implemented by the ECB and HMRC fetchers).
- Explicit LookupError on cache miss — deliberate, we do NOT want a
silent fallback that produces wrong cost-basis numbers.
Decisions deferred to later commits:
- Actual ECB daily reference-rate fetcher (eurofxref XML) — lands with
the Trading212 provider in Phase 1 when non-GBP trades first appear.
- HMRC monthly-rate fetcher + reconciliation CronJob — Phase 1 tail.
Test plan
---------
## Automated
- poetry run pytest tests/test_fx.py -v → 6 passed
- poetry run mypy broker_sync tests → Success: no issues found in 8 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Not applicable — no network yet.
Context
-------
Wealthfolio's activity `notes` field is user-editable via the UI, so
using it as the dedup key would let a single note-edit in Wealthfolio
cause the next sync to create a duplicate. Stress-testing the plan
flagged this as the top structural risk.
This change
-----------
- SQLite-backed store at `/data/broker_sync.db` in production; keyed on
(provider, account, external_id) so each provider's id space is
scoped to its own account.
- `INSERT OR IGNORE` makes record() idempotent — second call with the
same key is a no-op and preserves the original wealthfolio_activity_id
plus first_seen timestamp.
- `filter_new()` is the integration point: provider fetches activities,
hands them to the store, gets back only the unseen subset to submit
to the Wealthfolio sink.
- Wealthfolio activity id returned by the API is persisted alongside
each record so the HMRC FX reconciliation job can later PATCH the
original activity rather than creating a new one.
Test plan
---------
## Automated
- poetry run pytest tests/test_dedup.py -v → 6 passed
- poetry run mypy broker_sync tests → Success: no issues found in 6 source files
- poetry run ruff check . → All checks passed!
## Manual Verification
Not applicable for this layer — full end-to-end verification happens
once a provider + sink land (Phase 1 Trading212 and the auth spike).
Context
-------
New connector suite that syncs UK brokerage activity (Trading212,
InvestEngine, Schwab email-parsed, CSV drop-folder) into Wealthfolio.
Lives outside finance/ intentionally — finance/ is untouched per the
plan at ~/.claude/plans/let-s-work-on-linking-temporal-valiant.md.
This change
-----------
- Poetry project with httpx, typer, bs4, dev tools (pytest, mypy strict,
ruff, yapf).
- Canonical Activity + Account models with the 6 UK tax wrappers
(ISA/SIPP/GIA/LISA/JISA/WORKPLACE_PENSION) and the 12 Wealthfolio
activity types from docs/activities/activity-types.md on the upstream.
- Validation invariants: BUY/SELL need qty+price, DIVIDEND/DEPOSIT/etc
need amount — raises early so providers can't silently emit broken
rows.
- to_wealthfolio_csv_row() shape matches Wealthfolio's CSV import;
primary sink path per the plan.
Test plan
---------
## Automated
- poetry run pytest -q → 7 passed in 0.03s
- poetry run mypy broker_sync tests → Success: no issues found in 4 source files
- poetry run ruff check . → All checks passed!
- poetry run yapf --diff --recursive broker_sync tests → no diff
## Manual Verification
Not applicable — pure data model, no runtime behaviour.
Closes: code-thw.1
