30af5fe2c9
Trailing window backed by SyncRecordStore dedup is strictly better than a single-day window — a single missed cron run with Last Business Day loses that day's activity permanently. SyncRecordStore is keyed by ibkr:trade:<tradeID> / ibkr:cash:<transactionID>, so overlapping pulls are no-ops. Caught during the brainstorming review. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
127 lines
5.5 KiB
Markdown
127 lines
5.5 KiB
Markdown
# Provider: Interactive Brokers (IBKR Flex Web Service)
|
||
|
||
Pulls a daily Activity Flex Query via the [`ibflex`](https://github.com/csingley/ibflex)
|
||
library, maps Trades + CashTransactions to broker-sync Activities, and
|
||
reconciles broker-side OpenPositions against WF-computed quantities.
|
||
|
||
## When this runs
|
||
|
||
- K8s CronJob `broker-sync-ibkr` in the `broker-sync` namespace, daily 02:00 UK.
|
||
- Manual trigger:
|
||
```bash
|
||
kubectl -n broker-sync create job --from=cronjob/broker-sync-ibkr broker-sync-ibkr-manual-$(date +%s)
|
||
```
|
||
|
||
## Vault secrets — `secret/broker-sync`
|
||
|
||
| Key | Description |
|
||
|---|---|
|
||
| `ibkr_flex_token` | Flex Web Service token (1-year validity, rotate via IBKR Client Portal). |
|
||
| `ibkr_flex_query_id` | Activity Flex Query ID (5–7 digit number). |
|
||
| `ibkr_account_id` | Wealthfolio account UUID for "Interactive Brokers (UK)". |
|
||
| `ibkr_account_id_upstream` | IBKR-side account number (e.g. `U12345678`) — guards against wrong-account ingestion. |
|
||
|
||
ExternalSecret `broker-sync-secrets` syncs all keys from `secret/broker-sync`
|
||
to a K8s secret of the same name. New keys take ~15 min to propagate.
|
||
|
||
## IBKR Flex Query design
|
||
|
||
In IBKR Client Portal → Reports → Flex Queries → Activity Flex Query, create
|
||
a new query named `broker-sync-activity` with:
|
||
|
||
| Section | Required fields |
|
||
|---|---|
|
||
| Account Information | accountId |
|
||
| Trades | tradeID, tradeDate, tradeTime, symbol, buySell, quantity, tradePrice, currency, ibCommission, assetCategory, exchange |
|
||
| Cash Transactions | transactionID, dateTime, type, amount, currency, description |
|
||
| Open Positions | symbol, position, markPrice, currency, assetCategory, exchange |
|
||
| Securities Information | symbol, description, conid |
|
||
|
||
**Date Format:** `yyyy-MM-dd`. **Time Format:** `HH:mm:ss` (no timezone
|
||
suffix — ibflex 1.1 rejects timezone abbreviations in the time field).
|
||
**Date Range:** `Last 90 Days` — trailing window so a missed cron run
|
||
doesn't lose data. SyncRecordStore (keyed by `external_id`) makes
|
||
overlapping pulls idempotent. For a one-off historical backfill, widen
|
||
temporarily to `Year to Date` or `Custom Date Range`, run once, then
|
||
switch back.
|
||
|
||
## Cash type mapping
|
||
|
||
| IBKR Flex `CashTransaction.type` | broker-sync `ActivityType` |
|
||
|---|---|
|
||
| Dividends | DIVIDEND |
|
||
| Withholding Tax | TAX |
|
||
| Broker Interest Received | INTEREST |
|
||
| Broker Interest Paid | FEE |
|
||
| Commission Adjustments | FEE |
|
||
| Other Fees | FEE |
|
||
| Deposits & Withdrawals | DEPOSIT (amount > 0) / WITHDRAWAL (amount < 0) |
|
||
| anything else | skipped + WARNING logged (refuse to guess) |
|
||
|
||
## Dedup keys
|
||
|
||
- Trades: `external_id = "ibkr:trade:" + tradeID`
|
||
- Cash: `external_id = "ibkr:cash:" + transactionID`
|
||
|
||
Both are stable across re-runs; `dedup.SyncRecordStore` rejects already-
|
||
synced IDs.
|
||
|
||
## Symbol canonicalisation
|
||
|
||
LSE-listed GBP instruments get a `.L` suffix (Wealthfolio convention).
|
||
US instruments and anything already suffixed pass through unchanged.
|
||
|
||
The heuristic: `exchange in {LSE, LSEETF, LSEIOB1}` OR
|
||
`(exchange is None AND currency == GBP)` → suffix with `.L`. Edge cases
|
||
not yet covered (Euronext, XETRA) — extend `canonical_symbol` when those
|
||
holdings exist.
|
||
|
||
## Position reconciliation
|
||
|
||
Each run pushes to Pushgateway under job `broker-sync-ibkr`:
|
||
- `ibkr_position_drift_shares{symbol, account="ibkr-uk"}` —
|
||
broker_qty − wf_qty per asset.
|
||
- `ibkr_sync_last_success_timestamp_seconds` — unix timestamp.
|
||
|
||
Alerts (TODO, will be added to the monitoring stack on first
|
||
non-zero drift):
|
||
- `IBKRPositionDrift{symbol}` — `|drift| > 0.01` for >24h, Slack `#security`.
|
||
- `IBKRSyncStale` — timestamp > 36h old.
|
||
- `IBKRFlexTokenExpired` — Loki rule on the "code 1003" log line.
|
||
|
||
## Account guard
|
||
|
||
Before yielding any activities, the provider checks
|
||
`flex.accountId == IBKR_ACCOUNT_ID_UPSTREAM`. Mismatch → raises
|
||
`IBKRAccountMismatchError` and writes nothing. Prevents wrong-account
|
||
ingestion from a misconfigured query (e.g., someone replaced the token
|
||
with another user's by mistake).
|
||
|
||
## Token rotation
|
||
|
||
Flex tokens expire after 1 year. When the cron starts failing with
|
||
`ResponseCodeError(code=1003)`:
|
||
|
||
1. Sign in to IBKR Client Portal → Reports → Settings → Flex Web Service
|
||
→ regenerate token.
|
||
2. `vault kv patch secret/broker-sync ibkr_flex_token='<new-token>'`
|
||
3. ExternalSecrets controller picks up the new value within ~15 min; no
|
||
manual pod restart needed.
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Likely cause | Fix |
|
||
|---|---|---|
|
||
| `IBKR_FLEX_TOKEN not provided` exit 2 | Vault has placeholder value or key missing | `vault kv patch secret/broker-sync ibkr_flex_token='<real-token>'` |
|
||
| `IBKRAccountMismatchError` | `ibkr_account_id_upstream` doesn't match the account in the Flex query | Re-check IBKR account number; fix the Vault value |
|
||
| `ResponseCodeError(code=1003)` | Flex token expired | See "Token rotation" above |
|
||
| `StatementGenerationTimeout` | IBKR side slow | Single retry built in; if it persists, try a smaller date range |
|
||
| `Can't convert '... TZ' to time` parser error | Flex query has Time Format with timezone suffix | Switch to `HH:mm:ss` (no TZ) in Flex query settings |
|
||
| `'ETF' is not a valid AssetClass` | ETF set in fixture not in ibflex enum | Use `STK` in fixtures (IBKR Flex categorises ETFs under STK) |
|
||
|
||
## References
|
||
|
||
- Spec: [`docs/specs/2026-05-26-ibkr-ingest-design.md`](../specs/2026-05-26-ibkr-ingest-design.md)
|
||
- Plan: [`docs/plans/2026-05-26-ibkr-flex-ingestion.md`](../plans/2026-05-26-ibkr-flex-ingestion.md)
|
||
- Library: <https://github.com/csingley/ibflex>
|
||
- IBKR Flex Web Service docs: <https://www.interactivebrokers.com/en/software/am/am/reports/flex_web_service.htm>
|