viktor/broker-sync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
broker-sync/pyproject.toml

59 lines
1.6 KiB
TOML
Raw Permalink Normal View History

Initial scaffold + canonical Activity model Context ------- New connector suite that syncs UK brokerage activity (Trading212, InvestEngine, Schwab email-parsed, CSV drop-folder) into Wealthfolio. Lives outside finance/ intentionally — finance/ is untouched per the plan at ~/.claude/plans/let-s-work-on-linking-temporal-valiant.md. This change ----------- - Poetry project with httpx, typer, bs4, dev tools (pytest, mypy strict, ruff, yapf). - Canonical Activity + Account models with the 6 UK tax wrappers (ISA/SIPP/GIA/LISA/JISA/WORKPLACE_PENSION) and the 12 Wealthfolio activity types from docs/activities/activity-types.md on the upstream. - Validation invariants: BUY/SELL need qty+price, DIVIDEND/DEPOSIT/etc need amount — raises early so providers can't silently emit broken rows. - to_wealthfolio_csv_row() shape matches Wealthfolio's CSV import; primary sink path per the plan. Test plan --------- ## Automated - poetry run pytest -q → 7 passed in 0.03s - poetry run mypy broker_sync tests → Success: no issues found in 4 source files - poetry run ruff check . → All checks passed! - poetry run yapf --diff --recursive broker_sync tests → no diff ## Manual Verification Not applicable — pure data model, no runtime behaviour. Closes: code-thw.1
2026-04-17 19:16:11 +00:00
[tool.poetry]
name = "broker-sync"
version = "0.1.0"
description = "Sync UK brokerage activity (Trading212, InvestEngine, Schwab, CSV) into Wealthfolio"
authors = ["Viktor Barzin <viktorbarzin@meta.com>"]
readme = "pyproject.toml"
packages = [{ include = "broker_sync" }]
[tool.poetry.dependencies]
python = ">=3.11,<3.13"
httpx = "^0.27"
beautifulsoup4 = "^4.12"
python-dateutil = "^2.9"
typer = "^0.12"
Wire T212 pagination, retries, and click<8.2 pin Context ------- Closes out the Trading212 provider's retry + pagination surface so the "Add Trading212Provider core fetch" commit has everything the CronJob needs: cursor-based pagination, 429 honouring Retry-After, jittered exponential backoff for 429-without-header and 5xx, bailout after _MAX_RETRIES, and checkpoint-after-page semantics so a crashed run resumes at the start of the unfinished page. Also pins click<8.2 — typer 0.12 calls Parameter.make_metavar() without a ctx argument, which click 8.2 removed; `broker-sync --help` was crashing with TypeError until this pin. typer 0.15+ would also fix it; the pin is lower friction. One test fix: test_checkpoint_advances_only_after_page_yielded had a handler that unconditionally returned a next_path → infinite loop. The assertion was always about "a cursor was saved after page 1", so I changed the handler to return page 2 as empty-with-no-next, which terminates the loop cleanly. Test plan --------- ## Automated - poetry run pytest -q → 70 passed - poetry run mypy broker_sync tests → Success: no issues found in 29 source files - poetry run ruff check . → All checks passed! - poetry run broker-sync --help → renders without crash; lists version + auth-spike ## Manual Verification End-to-end against a live T212 key is in the next commit once the CLI subcommand and pipeline land.
2026-04-17 19:45:23 +00:00
click = "<8.2" # typer 0.12 uses make_metavar() without ctx; click 8.2 made ctx required
Add finance_mysql provider + CLI for historical backfill finance.position (171 rows, 2020-06-07 to 2025-12-19) is the only source of InvestEngine + Schwab trade history pre-dating the broker-sync project. This provider reads it once and pushes every row into the correct WF account (.L tickers → IE ISA, others → Schwab). Dedup: external_id = 'finance-mysql:position:<PK>' — idempotent on re-run. Auth: aiomysql as MySQL root (user-authorized) against the standalone mysql:8.4 in-cluster service. New CLI: broker-sync finance-mysql-import New tests: 5 unit tests covering route, symbol normalise, BUY/SELL detection. poetry run pytest -q → 114 passed, 1 skipped poetry run mypy → clean (aiomysql shielded with type: ignore) poetry run ruff check → clean
2026-04-17 22:38:21 +00:00
aiomysql = "^0.3.2"
fidelity-planviewer: scaffold provider + CLI (seed + stub ingest) ## Context UK workplace pension at planviewer.fidelity.co.uk has no public API; the SPA calls a private JSON backend at prd.wiciam.fidelity.co.uk/cvmfe/api/*. Viktor confirmed in DevTools that an OPTIONS preflight lists auth headers (ch, fid, rid, sid, tbid, theosreferer, ua). Full reverse-engineering of the endpoint paths is pending Viktor's POST cURL paste for transactions + holdings views. Until those endpoints are captured, ship the scaffold: provider module, CLI commands, tests, docs. This unblocks installing Playwright in the image and lets Viktor run the one-off seed command on his laptop ahead of the data integration. ## This change - broker_sync/providers/fidelity_planviewer.py - FidelityCreds namedtuple (storage_state_path, plan_id). - FidelitySessionError (401 → re-seed), FidelityProviderConfigError. - FidelityPlanViewerProvider: .accounts() returns a single WORKPLACE_PENSION account, .fetch() raises until endpoints are wired. - broker_sync/cli.py - fidelity-seed: launches headed Chromium so Viktor can log in and tick "Remember device", then dumps storage_state.json. - fidelity-ingest: stub matching the invest-engine / trading212 CLI shape; reads storage_state + plan_id, pipes through the shared pipeline. - tests/providers/test_fidelity_planviewer.py - Asserts the single-account shape + the loud-failure guard. - docs/providers/fidelity-planviewer.md - Architecture diagram, one-time seed procedure, backfill + monthly commands, alert runbook. - pyproject.toml - playwright ^1.47 as a first-class dep (used only by fidelity-seed and later by the session-refresh step in fidelity-ingest). ## What is NOT in this change - Endpoint wiring in provider.fetch() — blocked on DevTools POST cURL. - Infra CronJob + Vault secret + Prometheus alert — lands once the first manual backfill succeeds and we know the Chromium image size is fine. - Dockerfile Chromium install — same trigger. ## Verification ### Automated $ poetry run pytest tests/providers/test_fidelity_planviewer.py -v 2 passed in 0.08s $ poetry run pytest -q 122 passed, 1 skipped in 1.07s $ poetry run mypy broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py Success: no issues found in 2 source files $ poetry run ruff check broker_sync/providers/fidelity_planviewer.py broker_sync/cli.py tests/providers/test_fidelity_planviewer.py All checks passed! ### Manual (Viktor, later) 1. poetry install && poetry run playwright install chromium 2. poetry run broker-sync fidelity-seed --out /tmp/state.json 3. Chromium opens → log in → tick "Remember device" → press Enter 4. vault kv patch secret/broker-sync fidelity_storage_state=@/tmp/state.json Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 14:09:04 +00:00
# Fidelity UK PlanViewer has no public API — we use Playwright only to keep a
# long-lived session alive (storage_state + device-trust cookie); actual data
# is fetched via httpx against the SPA's private JSON backend.
playwright = "^1.47"
deps: add ibflex with web extras for IBKR Flex Web Service ingestion
2026-05-26 22:21:54 +00:00
# IBKR Flex Web Service: pulls Activity Flex Query XML reports (token-auth)
# and parses to typed dataclasses. No Gateway / daily re-auth needed.
ibflex = { version = "^1.1", extras = ["web"] }
Initial scaffold + canonical Activity model Context ------- New connector suite that syncs UK brokerage activity (Trading212, InvestEngine, Schwab email-parsed, CSV drop-folder) into Wealthfolio. Lives outside finance/ intentionally — finance/ is untouched per the plan at ~/.claude/plans/let-s-work-on-linking-temporal-valiant.md. This change ----------- - Poetry project with httpx, typer, bs4, dev tools (pytest, mypy strict, ruff, yapf). - Canonical Activity + Account models with the 6 UK tax wrappers (ISA/SIPP/GIA/LISA/JISA/WORKPLACE_PENSION) and the 12 Wealthfolio activity types from docs/activities/activity-types.md on the upstream. - Validation invariants: BUY/SELL need qty+price, DIVIDEND/DEPOSIT/etc need amount — raises early so providers can't silently emit broken rows. - to_wealthfolio_csv_row() shape matches Wealthfolio's CSV import; primary sink path per the plan. Test plan --------- ## Automated - poetry run pytest -q → 7 passed in 0.03s - poetry run mypy broker_sync tests → Success: no issues found in 4 source files - poetry run ruff check . → All checks passed! - poetry run yapf --diff --recursive broker_sync tests → no diff ## Manual Verification Not applicable — pure data model, no runtime behaviour. Closes: code-thw.1
2026-04-17 19:16:11 +00:00
[tool.poetry.group.dev.dependencies]
pytest = "^8.3"
pytest-asyncio = "^0.23"
mypy = "^1.11"
ruff = "^0.6"
yapf = "^0.43"
Add Schwab email parser (port from finance/) Schwab's workplace-RSU confirmation emails have 5 data td elements with class='dark-background-body' align='right': date, direction, qty, ticker, price-with-currency-sign. One email → one Activity. - parse_schwab_email(raw_html) -> list[Activity] (1-item or empty) - Empty on any parse failure (IMAP batch shouldn't crash on one bad mail) - Deterministic external_id ('schwab:date:ticker:type:qty') — stable across re-pulls so dedup works - Hardcoded to account 'schwab-workplace' / AccountType.GIA / USD - 6 unit tests: SELL + BUY happy path, malformed, missing cells, external-id stability, commas in price Dropped from the original finance port: - msg_timestamp-based external id (non-deterministic — would re-import on every IMAP walk). Replaced with a hash-stable key. - Currency.from_sign() currency hack. Schwab US is USD-only; we'll add FX when that changes. poetry run pytest -q → 109 passed, 1 skipped poetry run mypy → clean (added types-python-dateutil) poetry run ruff check → clean
2026-04-17 22:08:40 +00:00
types-python-dateutil = "^2.9.0.20260408"
Initial scaffold + canonical Activity model Context ------- New connector suite that syncs UK brokerage activity (Trading212, InvestEngine, Schwab email-parsed, CSV drop-folder) into Wealthfolio. Lives outside finance/ intentionally — finance/ is untouched per the plan at ~/.claude/plans/let-s-work-on-linking-temporal-valiant.md. This change ----------- - Poetry project with httpx, typer, bs4, dev tools (pytest, mypy strict, ruff, yapf). - Canonical Activity + Account models with the 6 UK tax wrappers (ISA/SIPP/GIA/LISA/JISA/WORKPLACE_PENSION) and the 12 Wealthfolio activity types from docs/activities/activity-types.md on the upstream. - Validation invariants: BUY/SELL need qty+price, DIVIDEND/DEPOSIT/etc need amount — raises early so providers can't silently emit broken rows. - to_wealthfolio_csv_row() shape matches Wealthfolio's CSV import; primary sink path per the plan. Test plan --------- ## Automated - poetry run pytest -q → 7 passed in 0.03s - poetry run mypy broker_sync tests → Success: no issues found in 4 source files - poetry run ruff check . → All checks passed! - poetry run yapf --diff --recursive broker_sync tests → no diff ## Manual Verification Not applicable — pure data model, no runtime behaviour. Closes: code-thw.1
2026-04-17 19:16:11 +00:00
[tool.poetry.scripts]
broker-sync = "broker_sync.cli:app"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["tests"]
[tool.mypy]
python_version = "3.11"
strict = true
files = ["broker_sync", "tests"]
[tool.ruff]
line-length = 100
target-version = "py311"
[tool.ruff.lint]
select = ["E", "F", "W", "I", "UP", "B", "SIM", "RUF"]
[tool.yapf]
based_on_style = "pep8"
column_limit = 100
Reference in a new issue Copy permalink