diff --git a/docs/plans/2026-02-22-trading-bot-implementation.md b/docs/plans/2026-02-22-trading-bot-implementation.md new file mode 100644 index 0000000..3e89ac2 --- /dev/null +++ b/docs/plans/2026-02-22-trading-bot-implementation.md @@ -0,0 +1,1149 @@ +# Trading Bot Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Build an automated stock trading bot that combines news sentiment analysis with technical strategies, learns from outcomes, and provides a real-time dashboard. + +**Architecture:** Event-driven Python microservices communicating via Redis Streams, with PostgreSQL+TimescaleDB for persistence, Alpaca for brokerage, and a React/TypeScript dashboard. See `docs/plans/2026-02-22-trading-bot-design.md` for full design. + +**Tech Stack:** Python 3.12, FastAPI, SQLAlchemy (async), Redis Streams, PostgreSQL+TimescaleDB, React 18, TypeScript, Tailwind CSS, alpaca-py, transformers (FinBERT), Ollama, OpenTelemetry, py-webauthn + +--- + +## Phase 1: Foundation + +### Task 1: Python Monorepo Setup + +**Files:** +- Create: `pyproject.toml` +- Create: `shared/__init__.py` +- Create: `shared/config.py` +- Create: `shared/redis_streams.py` +- Create: `shared/telemetry.py` +- Create: `tests/__init__.py` + +**Step 1: Create pyproject.toml** + +Single pyproject.toml at the repo root. Use `[project.optional-dependencies]` groups per service so each service only installs what it needs. Core deps shared by all: `sqlalchemy[asyncio]`, `asyncpg`, `redis`, `pydantic`, `pydantic-settings`, `opentelemetry-sdk`, `opentelemetry-exporter-prometheus`, `opentelemetry-api`. + +```toml +[project] +name = "trading-bot" +version = "0.1.0" +requires-python = ">=3.12" +dependencies = [ + "sqlalchemy[asyncio]>=2.0", + "asyncpg>=0.29", + "redis>=5.0", + "pydantic>=2.0", + "pydantic-settings>=2.0", + "opentelemetry-sdk>=1.20", + "opentelemetry-exporter-prometheus>=0.45b", + "opentelemetry-api>=1.20", + "alembic>=1.13", +] + +[project.optional-dependencies] +api = ["fastapi>=0.110", "uvicorn[standard]>=0.27", "websockets>=12.0", "py-webauthn>=2.0", "pyjwt[crypto]>=2.8"] +news = ["feedparser>=6.0", "praw>=7.7", "httpx>=0.27"] +sentiment = ["transformers>=4.38", "torch>=2.2", "ollama>=0.1"] +trading = ["alpaca-py>=0.21"] +backtester = ["numpy>=1.26", "pandas>=2.2"] +dev = ["pytest>=8.0", "pytest-asyncio>=0.23", "pytest-cov>=4.1", "ruff>=0.3", "mypy>=1.8"] + +[tool.pytest.ini_options] +asyncio_mode = "auto" +testpaths = ["tests"] + +[tool.ruff] +line-length = 120 +target-version = "py312" +``` + +**Step 2: Create shared config module** + +`shared/config.py` — Pydantic Settings class for all shared config (database URL, Redis URL, etc). Each service extends this with its own settings. + +```python +from pydantic_settings import BaseSettings + +class BaseConfig(BaseSettings): + database_url: str = "postgresql+asyncpg://trading:trading@localhost:5432/trading" + redis_url: str = "redis://localhost:6379/0" + log_level: str = "INFO" + otel_service_name: str = "trading-bot" + otel_metrics_port: int = 9090 + + model_config = {"env_prefix": "TRADING_"} +``` + +**Step 3: Create Redis Streams helper** + +`shared/redis_streams.py` — Thin wrapper around redis-py Streams for publishing and consuming. Consumer groups, auto-ack, deserialization. + +```python +import json +import logging +from typing import AsyncIterator +from redis.asyncio import Redis + +logger = logging.getLogger(__name__) + +class StreamPublisher: + def __init__(self, redis: Redis, stream: str): + self.redis = redis + self.stream = stream + + async def publish(self, data: dict) -> str: + msg_id = await self.redis.xadd(self.stream, {"data": json.dumps(data)}) + return msg_id + +class StreamConsumer: + def __init__(self, redis: Redis, stream: str, group: str, consumer: str): + self.redis = redis + self.stream = stream + self.group = group + self.consumer = consumer + + async def ensure_group(self) -> None: + try: + await self.redis.xgroup_create(self.stream, self.group, id="0", mkstream=True) + except Exception: + pass # group already exists + + async def consume(self, batch_size: int = 10, block_ms: int = 5000) -> AsyncIterator[tuple[str, dict]]: + await self.ensure_group() + while True: + messages = await self.redis.xreadgroup( + self.group, self.consumer, {self.stream: ">"}, count=batch_size, block=block_ms + ) + for _stream, entries in messages: + for msg_id, fields in entries: + data = json.loads(fields[b"data"]) + yield msg_id, data + await self.redis.xack(self.stream, self.group, msg_id) +``` + +**Step 4: Create OpenTelemetry helper** + +`shared/telemetry.py` — Sets up meter provider with Prometheus exporter, returns a meter. Each service calls `setup_telemetry("service-name")` at startup and starts a `/metrics` HTTP endpoint. + +```python +from opentelemetry import metrics +from opentelemetry.sdk.metrics import MeterProvider +from opentelemetry.exporter.prometheus import PrometheusMetricReader +from prometheus_client import start_http_server + +def setup_telemetry(service_name: str, metrics_port: int = 9090) -> metrics.Meter: + reader = PrometheusMetricReader() + provider = MeterProvider(metric_readers=[reader]) + metrics.set_meter_provider(provider) + start_http_server(metrics_port) + return metrics.get_meter(service_name) +``` + +**Step 5: Write tests for Redis Streams helper** + +```python +# tests/test_redis_streams.py +import pytest +from unittest.mock import AsyncMock, MagicMock +from shared.redis_streams import StreamPublisher, StreamConsumer + +@pytest.mark.asyncio +async def test_publisher_publishes_json(): + redis = AsyncMock() + redis.xadd = AsyncMock(return_value=b"1-0") + pub = StreamPublisher(redis, "test:stream") + msg_id = await pub.publish({"ticker": "AAPL", "score": 0.8}) + redis.xadd.assert_called_once() + assert msg_id == b"1-0" +``` + +Run: `python -m pytest tests/test_redis_streams.py -v` + +**Step 6: Commit** + +```bash +git add pyproject.toml shared/ tests/ +git commit -m "feat: project foundation — monorepo setup, shared config, redis streams, telemetry" +``` + +--- + +### Task 2: Docker Compose Infrastructure + +**Files:** +- Create: `docker-compose.yml` +- Create: `.env.example` +- Create: `.gitignore` + +**Step 1: Create docker-compose.yml with infrastructure services** + +Start with just the infrastructure containers (postgres+timescaledb, redis, ollama). Application services added later. + +```yaml +services: + postgres: + image: timescale/timescaledb:latest-pg16 + environment: + POSTGRES_USER: trading + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-trading} + POSTGRES_DB: trading + ports: + - "5432:5432" + volumes: + - pgdata:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U trading"] + interval: 5s + timeout: 5s + retries: 5 + + redis: + image: redis:7-alpine + ports: + - "6379:6379" + volumes: + - redisdata:/data + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 5s + timeout: 5s + retries: 5 + + ollama: + image: ollama/ollama:latest + ports: + - "11434:11434" + volumes: + - ollama_models:/root/.ollama + +volumes: + pgdata: + redisdata: + ollama_models: +``` + +**Step 2: Create .env.example and .gitignore** + +`.env.example`: document all env vars with safe defaults. +`.gitignore`: Python defaults + `.env`, `__pycache__`, `.venv`, node_modules, etc. + +**Step 3: Boot infrastructure and verify** + +```bash +docker compose up -d postgres redis +docker compose ps # verify healthy +``` + +**Step 4: Commit** + +```bash +git add docker-compose.yml .env.example .gitignore +git commit -m "feat: docker compose infrastructure — postgres+timescaledb, redis, ollama" +``` + +--- + +### Task 3: Database Models & Alembic Migrations + +**Files:** +- Create: `shared/models/__init__.py` +- Create: `shared/models/base.py` +- Create: `shared/models/trading.py` +- Create: `shared/models/news.py` +- Create: `shared/models/learning.py` +- Create: `shared/models/auth.py` +- Create: `shared/models/timeseries.py` +- Create: `shared/db.py` +- Create: `alembic.ini` +- Create: `alembic/env.py` +- Create: `tests/test_models.py` + +**Step 1: Create base model and DB session factory** + +`shared/models/base.py` — Declarative base with common columns (id, created_at, updated_at). +`shared/db.py` — Async engine + sessionmaker factory. + +```python +# shared/models/base.py +import uuid +from datetime import datetime, timezone +from sqlalchemy import DateTime, func +from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column +from sqlalchemy.dialects.postgresql import UUID + +class Base(DeclarativeBase): + pass + +class TimestampMixin: + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), server_default=func.now() + ) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), server_default=func.now(), onupdate=func.now() + ) +``` + +```python +# shared/db.py +from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker, AsyncSession +from shared.config import BaseConfig + +def create_db(config: BaseConfig) -> tuple: + engine = create_async_engine(config.database_url, echo=config.log_level == "DEBUG") + session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False) + return engine, session_factory +``` + +**Step 2: Create trading models** + +`shared/models/trading.py` — `Trade`, `Position`, `Signal`, `Strategy`, `StrategyWeightHistory` per design doc. + +Key columns per design doc: +- `trades`: ticker, side (enum: BUY/SELL), qty, price, timestamp, strategy_id (FK), signal_id (FK), status (enum: PENDING/FILLED/CANCELLED/REJECTED), pnl +- `positions`: ticker (unique), qty, avg_entry, unrealized_pnl, stop_loss, take_profit +- `signals`: ticker, direction (enum: LONG/SHORT/NEUTRAL), strength (float 0-1), strategy_sources (JSON), sentiment_score, acted_on (bool) +- `strategies`: name (unique), description, current_weight, active (bool) +- `strategy_weights_history`: strategy_id (FK), old_weight, new_weight, timestamp, reason + +**Step 3: Create news models** + +`shared/models/news.py` — `Article`, `ArticleSentiment` per design doc. + +**Step 4: Create learning models** + +`shared/models/learning.py` — `TradeOutcome`, `LearningAdjustment` per design doc. + +**Step 5: Create auth models** + +`shared/models/auth.py` — `User`, `UserCredential` per design doc. + +**Step 6: Create timeseries models** + +`shared/models/timeseries.py` — `MarketData`, `PortfolioSnapshot`, `StrategyMetric`. These will be TimescaleDB hypertables (created via migration with `SELECT create_hypertable(...)`). + +**Step 7: Set up Alembic** + +```bash +python -m alembic init alembic +``` + +Edit `alembic/env.py` to import all models and use async engine. Edit `alembic.ini` to read `sqlalchemy.url` from env. + +**Step 8: Generate and apply initial migration** + +```bash +python -m alembic revision --autogenerate -m "initial schema" +``` + +Manually edit migration to add TimescaleDB hypertable creation after table creates: +```python +op.execute("SELECT create_hypertable('market_data', 'timestamp')") +op.execute("SELECT create_hypertable('portfolio_snapshots', 'timestamp')") +op.execute("SELECT create_hypertable('strategy_metrics', 'timestamp')") +``` + +Apply: +```bash +docker compose up -d postgres +python -m alembic upgrade head +``` + +**Step 9: Write model tests** + +Test that models can be instantiated, relationships work, enums are correct. + +**Step 10: Commit** + +```bash +git add shared/models/ shared/db.py alembic/ alembic.ini tests/test_models.py +git commit -m "feat: database models and alembic migrations — all tables per design" +``` + +--- + +### Task 4: Pydantic Schemas + +**Files:** +- Create: `shared/schemas/__init__.py` +- Create: `shared/schemas/trading.py` +- Create: `shared/schemas/news.py` +- Create: `shared/schemas/learning.py` +- Create: `shared/schemas/auth.py` +- Create: `tests/test_schemas.py` + +**Step 1: Create schemas matching Redis Stream message formats** + +These are the Pydantic v2 models used for (de)serialization across services. Each Redis Stream message is one of these schemas serialized to JSON. + +Key schemas: +- `RawArticle` (published to `news:raw`): source, url, title, content, published_at, fetched_at +- `ScoredArticle` (published to `news:scored`): article fields + ticker, sentiment_score, confidence, model_used, entities +- `TradeSignal` (published to `signals:generated`): ticker, direction, strength, strategy_sources, sentiment_context, timestamp +- `TradeExecution` (published to `trades:executed`): trade_id, ticker, side, qty, price, status, signal_id, timestamp + +Also create request/response schemas for the API Gateway endpoints. + +**Step 2: Write schema validation tests** + +Test serialization round-trips, validation constraints (score range, required fields). + +**Step 3: Commit** + +```bash +git add shared/schemas/ tests/test_schemas.py +git commit -m "feat: pydantic schemas for all service message types" +``` + +--- + +## Phase 2: Brokerage & Market Data + +### Task 5: Brokerage Abstraction Layer + +**Files:** +- Create: `shared/broker/__init__.py` +- Create: `shared/broker/base.py` +- Create: `shared/broker/alpaca_broker.py` +- Create: `tests/test_broker.py` + +**Step 1: Define abstract broker interface** + +`shared/broker/base.py` — ABC with methods: `submit_order`, `cancel_order`, `get_positions`, `get_account`, `get_order_status`, `stream_market_data`. This is the seam that lets us swap Alpaca for another brokerage later. + +```python +from abc import ABC, abstractmethod +from shared.schemas.trading import OrderRequest, OrderResult, PositionInfo, AccountInfo + +class BaseBroker(ABC): + @abstractmethod + async def submit_order(self, order: OrderRequest) -> OrderResult: ... + + @abstractmethod + async def cancel_order(self, order_id: str) -> bool: ... + + @abstractmethod + async def get_positions(self) -> list[PositionInfo]: ... + + @abstractmethod + async def get_account(self) -> AccountInfo: ... + + @abstractmethod + async def get_order_status(self, order_id: str) -> OrderResult: ... +``` + +**Step 2: Implement Alpaca broker** + +`shared/broker/alpaca_broker.py` — Wraps `alpaca-py` SDK. Uses `TradingClient` for orders/positions/account, `StockDataStream` for real-time bars. Paper vs live controlled by config (API base URL). + +**Step 3: Write tests with mocked Alpaca client** + +Test order submission, position retrieval, error handling (rejected orders, network failures). + +**Step 4: Commit** + +```bash +git add shared/broker/ tests/test_broker.py +git commit -m "feat: brokerage abstraction layer with Alpaca implementation" +``` + +--- + +## Phase 3: News Pipeline + +### Task 6: News Fetcher Service + +**Files:** +- Create: `services/news-fetcher/__init__.py` +- Create: `services/news-fetcher/main.py` +- Create: `services/news-fetcher/sources/__init__.py` +- Create: `services/news-fetcher/sources/rss.py` +- Create: `services/news-fetcher/sources/reddit.py` +- Create: `services/news-fetcher/config.py` +- Create: `tests/services/test_news_fetcher.py` + +**Step 1: Create RSS source** + +Uses `feedparser` to poll configurable list of RSS feed URLs. Deduplicates by content_hash (SHA256 of URL+title). Converts feed entries to `RawArticle` schema. Configurable poll interval. + +Default feeds: Yahoo Finance, Reuters business, MarketWatch, SEC EDGAR RSS. + +**Step 2: Create Reddit source** + +Uses `praw` (async via `asyncpraw`) to poll r/wallstreetbets, r/stocks, r/investing. Fetches hot/new posts above score threshold. Converts to `RawArticle` schema. + +**Step 3: Create main service loop** + +`main.py` — Async service that: +1. Loads config (feed URLs, poll intervals, Reddit credentials) +2. Connects to Redis +3. Sets up telemetry (articles_fetched counter, fetch_errors counter, fetch_latency histogram) +4. Runs source pollers on configurable schedules via `asyncio.TaskGroup` +5. Publishes each `RawArticle` to `news:raw` stream + +**Step 4: Write tests** + +Test RSS parsing with fixture XML, Reddit post conversion, deduplication logic, stream publishing. + +**Step 5: Commit** + +```bash +git add services/news-fetcher/ tests/services/test_news_fetcher.py +git commit -m "feat: news fetcher service — RSS and Reddit sources" +``` + +--- + +### Task 7: Sentiment Analyzer Service + +**Files:** +- Create: `services/sentiment-analyzer/__init__.py` +- Create: `services/sentiment-analyzer/main.py` +- Create: `services/sentiment-analyzer/analyzers/__init__.py` +- Create: `services/sentiment-analyzer/analyzers/finbert.py` +- Create: `services/sentiment-analyzer/analyzers/ollama_analyzer.py` +- Create: `services/sentiment-analyzer/ticker_extractor.py` +- Create: `services/sentiment-analyzer/config.py` +- Create: `tests/services/test_sentiment_analyzer.py` + +**Step 1: Create FinBERT analyzer** + +Loads `ProsusAI/finbert` via HuggingFace transformers. Input: article title + first 512 tokens of content. Output: sentiment score (-1 to +1), confidence (0 to 1). Runs on CPU by default (GPU if available via torch.cuda). + +**Step 2: Create Ollama analyzer** + +Fallback for articles where FinBERT confidence < configurable threshold (default 0.6). Uses `ollama` Python client to query local Mistral/Llama 3. Structured prompt asking for JSON output with sentiment score, reasoning, and entity extraction. + +**Step 3: Create ticker extractor** + +Regex + lookup against a known ticker list (can be loaded from Alpaca's asset list). Extracts mentioned stock tickers from article text. Handles common patterns: $AAPL, "Apple Inc", NASDAQ:AAPL. + +**Step 4: Create main service loop** + +Consumes `news:raw`, routes through FinBERT → (if low confidence) → Ollama, extracts tickers, publishes `ScoredArticle` to `news:scored`. + +Telemetry: articles_scored counter, finbert_vs_ollama_ratio, inference_latency histogram. + +**Step 5: Write tests** + +Test FinBERT scoring (mock transformers pipeline), Ollama fallback routing, ticker extraction regex, end-to-end message flow. + +**Step 6: Commit** + +```bash +git add services/sentiment-analyzer/ tests/services/test_sentiment_analyzer.py +git commit -m "feat: sentiment analyzer — FinBERT + Ollama tiered analysis" +``` + +--- + +## Phase 4: Trading Core + +### Task 8: Strategy Implementations + +**Files:** +- Create: `shared/strategies/__init__.py` +- Create: `shared/strategies/base.py` +- Create: `shared/strategies/momentum.py` +- Create: `shared/strategies/mean_reversion.py` +- Create: `shared/strategies/news_driven.py` +- Create: `tests/test_strategies.py` + +**Step 1: Define strategy interface** + +```python +# shared/strategies/base.py +from abc import ABC, abstractmethod +from shared.schemas.trading import TradeSignal, MarketSnapshot, SentimentContext + +class BaseStrategy(ABC): + name: str + + @abstractmethod + async def evaluate( + self, ticker: str, market: MarketSnapshot, sentiment: SentimentContext | None = None + ) -> TradeSignal | None: + """Return a signal if this strategy has an opinion, None otherwise.""" + ... +``` + +`MarketSnapshot`: current price, OHLCV bars (recent history), volume profile, moving averages. +`SentimentContext`: recent sentiment scores for this ticker, article count, average confidence. + +**Step 2: Implement momentum strategy** + +Buy when price crosses above N-period SMA with increasing volume. Sell when crosses below. Signal strength proportional to distance from SMA. + +**Step 3: Implement mean reversion strategy** + +Buy when RSI < 30 (oversold), sell when RSI > 70 (overbought). Signal strength proportional to RSI extremity. + +**Step 4: Implement news-driven strategy** + +Buy on strong positive sentiment (score > 0.7, confidence > 0.6), sell on strong negative. Signal strength = sentiment_score * confidence. Decay factor for stale news (> 4 hours old). + +**Step 5: Write tests for each strategy** + +Test with known market data fixtures. Verify signal direction, strength ranges, edge cases (no data, flat market). + +**Step 6: Commit** + +```bash +git add shared/strategies/ tests/test_strategies.py +git commit -m "feat: trading strategies — momentum, mean reversion, news-driven" +``` + +--- + +### Task 9: Signal Generator Service + +**Files:** +- Create: `services/signal-generator/__init__.py` +- Create: `services/signal-generator/main.py` +- Create: `services/signal-generator/ensemble.py` +- Create: `services/signal-generator/market_data.py` +- Create: `services/signal-generator/config.py` +- Create: `tests/services/test_signal_generator.py` + +**Step 1: Create market data consumer** + +Connects to Alpaca WebSocket (`StockDataStream`) for real-time bars/quotes. Maintains in-memory rolling window of recent OHLCV bars per subscribed ticker. Builds `MarketSnapshot` objects. + +**Step 2: Create weighted ensemble** + +Loads strategy weights from DB (or Redis cache). Runs all active strategies, combines signals: +- For each ticker with signals: weighted average of direction * strength * weight +- Apply threshold: only emit signal if combined strength > configurable min (default 0.3) +- Tag output signal with contributing strategy sources and their individual strengths + +```python +async def combine_signals( + signals: list[tuple[BaseStrategy, TradeSignal]], + weights: dict[str, float], +) -> TradeSignal | None: + if not signals: + return None + weighted_sum = sum( + s.strength * (1 if s.direction == "LONG" else -1) * weights.get(strategy.name, 0.1) + for strategy, s in signals + ) + total_weight = sum(weights.get(strategy.name, 0.1) for strategy, _ in signals) + combined_strength = abs(weighted_sum / total_weight) if total_weight > 0 else 0 + # ... threshold check, build TradeSignal ... +``` + +**Step 3: Create main service loop** + +Consumes `news:scored` (builds sentiment context per ticker). Subscribes to Alpaca WebSocket for tickers mentioned in recent news + a watchlist. On each new bar or sentiment update, runs ensemble for affected tickers. Publishes qualifying signals to `signals:generated`. + +**Step 4: Write tests** + +Test ensemble weighting math, threshold filtering, sentiment context aggregation, market snapshot building. + +**Step 5: Commit** + +```bash +git add services/signal-generator/ tests/services/test_signal_generator.py +git commit -m "feat: signal generator — weighted ensemble with market data" +``` + +--- + +### Task 10: Trade Executor Service + +**Files:** +- Create: `services/trade-executor/__init__.py` +- Create: `services/trade-executor/main.py` +- Create: `services/trade-executor/risk_manager.py` +- Create: `services/trade-executor/config.py` +- Create: `tests/services/test_trade_executor.py` + +**Step 1: Create risk manager** + +Pre-trade risk checks: +- Position sizing: Kelly criterion or fixed fractional (configurable), max % of portfolio per position (default 5%) +- Max total exposure: sum of all position values < configurable % of portfolio (default 80%) +- Max positions: configurable cap (default 20) +- Stop-loss: auto-set at configurable % below entry (default 3%) +- Cooldown: no re-entry into same ticker within N minutes of exit (default 30) +- Market hours check: only trade during market hours (9:30 AM - 4:00 PM ET) + +**Step 2: Create main service loop** + +Consumes `signals:generated`. For each signal: +1. Run risk checks → reject if any fail +2. Calculate position size +3. Submit order via broker abstraction +4. Record trade in PostgreSQL (status: PENDING) +5. Poll/await fill confirmation +6. Update trade record (status: FILLED, actual price) +7. Update positions table +8. Publish `TradeExecution` to `trades:executed` + +Telemetry: trades_executed counter, order_fill_latency histogram, rejection_rate counter (by reason). + +**Step 3: Write tests** + +Test risk checks (position sizing math, exposure limits, cooldown), order flow with mocked broker, DB recording. + +**Step 4: Commit** + +```bash +git add services/trade-executor/ tests/services/test_trade_executor.py +git commit -m "feat: trade executor — risk management and order execution" +``` + +--- + +## Phase 5: Learning & Backtesting + +### Task 11: Learning Engine Service + +**Files:** +- Create: `services/learning-engine/__init__.py` +- Create: `services/learning-engine/main.py` +- Create: `services/learning-engine/evaluator.py` +- Create: `services/learning-engine/weight_adjuster.py` +- Create: `services/learning-engine/config.py` +- Create: `tests/services/test_learning_engine.py` + +**Step 1: Create trade evaluator** + +When a position is closed (detected via `trades:executed` with side opposite to open position): +- Compute realized P&L, ROI %, hold duration +- Compute risk-adjusted return (per-trade Sharpe approximation) +- Store `TradeOutcome` in DB +- Attribute credit/blame to contributing strategies proportionally to signal strength + +**Step 2: Create weight adjuster** + +Multi-armed bandit style (per design doc): +```python +new_weight = (1 - lr) * old_weight + lr * reward_signal +``` + +Guardrails (all configurable): +- Minimum 20 trades per strategy before any adjustment +- Max 10% weight shift per cycle +- Weight floor of 0.05 +- Normalize all weights to sum to 1.0 +- Exponential recency decay (recent trades weighted more) + +Store every adjustment in `learning_adjustments` table. Update strategy weights in `strategies` table and Redis cache. + +**Step 3: Create main service loop** + +Consumes `trades:executed`. On position close events, runs evaluator → weight adjuster. Periodically (configurable, default 1 hour) computes portfolio-level metrics and stores `PortfolioSnapshot` and `StrategyMetric` in TimescaleDB. + +**Step 4: Write tests** + +Test P&L calculation, credit attribution math, weight adjustment with guardrails (test each guardrail independently), normalization. + +**Step 5: Commit** + +```bash +git add services/learning-engine/ tests/services/test_learning_engine.py +git commit -m "feat: learning engine — multi-armed bandit strategy weight adjustment" +``` + +--- + +### Task 12: Backtesting Engine + +**Files:** +- Create: `backtester/__init__.py` +- Create: `backtester/engine.py` +- Create: `backtester/simulated_broker.py` +- Create: `backtester/data_loader.py` +- Create: `backtester/metrics.py` +- Create: `backtester/config.py` +- Create: `tests/test_backtester.py` + +**Step 1: Create simulated broker** + +Implements `BaseBroker` interface. Fills orders instantly at current bar's close price (or configurable slippage model). Tracks simulated positions and cash balance. Supports configurable commission model. + +**Step 2: Create data loader** + +Loads historical OHLCV bars from TimescaleDB `market_data` table. Loads historical sentiment from `article_sentiments` table. Aligns by timestamp. Returns an async iterator of `(timestamp, MarketSnapshot, SentimentContext | None)` tuples. + +**Step 3: Create backtest engine** + +Replays data loader output through the same strategy → ensemble → risk manager → simulated broker pipeline. Records all simulated trades. Uses the same code as live system (shared strategies, shared ensemble, shared risk manager). + +**Step 4: Create metrics calculator** + +From the simulated trade log, compute: +- Equity curve (portfolio value over time) +- Total return, annualized return +- Sharpe ratio, Sortino ratio +- Max drawdown (% and duration) +- Win rate, average win/loss ratio +- Per-strategy attribution (which strategies contributed most) +- Trade count, average hold duration + +**Step 5: Write tests** + +Test simulated broker fills, equity curve calculation, metrics math (known inputs → known outputs), data loader query. + +**Step 6: Commit** + +```bash +git add backtester/ tests/test_backtester.py +git commit -m "feat: backtesting engine — historical replay with shared strategies" +``` + +--- + +## Phase 6: API & Dashboard + +### Task 13: API Gateway — Auth + +**Files:** +- Create: `services/api-gateway/__init__.py` +- Create: `services/api-gateway/main.py` +- Create: `services/api-gateway/auth/__init__.py` +- Create: `services/api-gateway/auth/routes.py` +- Create: `services/api-gateway/auth/jwt.py` +- Create: `services/api-gateway/auth/middleware.py` +- Create: `services/api-gateway/config.py` +- Create: `tests/services/test_api_auth.py` + +**Step 1: Create FastAPI application shell** + +`main.py` — FastAPI app with CORS middleware (restricted origin), lifespan handler for DB/Redis connections, OpenTelemetry instrumentation (`opentelemetry-instrumentation-fastapi`). + +**Step 2: Implement passkey registration (sign up)** + +`POST /auth/register/begin` — Generate WebAuthn registration options (challenge, relying party info). Uses `py-webauthn`'s `generate_registration_options`. + +`POST /auth/register/complete` — Verify registration response, store credential public key in `user_credentials` table. Uses `verify_registration_response`. + +**Step 3: Implement passkey authentication (sign in)** + +`POST /auth/login/begin` — Generate authentication options with stored credential IDs. + +`POST /auth/login/complete` — Verify authentication response, update sign count, issue JWT (access token + refresh token). + +**Step 4: Create JWT helper** + +Issue short-lived access tokens (15 min) and longer refresh tokens (7 days). `POST /auth/refresh` to get new access token. + +**Step 5: Create auth middleware** + +FastAPI dependency that extracts and validates JWT from `Authorization: Bearer ` header. Skip for `/auth/*`, `/metrics`, `/health` routes. + +**Step 6: Write tests** + +Test registration flow, authentication flow (mock WebAuthn verification), JWT issuance/validation, middleware rejection of invalid tokens. + +**Step 7: Commit** + +```bash +git add services/api-gateway/ tests/services/test_api_auth.py +git commit -m "feat: API gateway with passkey (WebAuthn) authentication" +``` + +--- + +### Task 14: API Gateway — Trading Endpoints + +**Files:** +- Modify: `services/api-gateway/main.py` +- Create: `services/api-gateway/routes/__init__.py` +- Create: `services/api-gateway/routes/portfolio.py` +- Create: `services/api-gateway/routes/trades.py` +- Create: `services/api-gateway/routes/signals.py` +- Create: `services/api-gateway/routes/strategies.py` +- Create: `services/api-gateway/routes/news.py` +- Create: `services/api-gateway/routes/controls.py` +- Create: `services/api-gateway/routes/backtest.py` +- Create: `services/api-gateway/ws.py` +- Create: `tests/services/test_api_routes.py` + +**Step 1: Portfolio endpoints** + +- `GET /api/portfolio` — Current portfolio value, cash, buying power, daily P&L +- `GET /api/portfolio/positions` — All open positions with unrealized P&L +- `GET /api/portfolio/history` — Equity curve from `portfolio_snapshots` (query params: period) + +**Step 2: Trade endpoints** + +- `GET /api/trades` — Paginated trade history with filters (ticker, date range, strategy, profitable) +- `GET /api/trades/:id` — Single trade detail with linked signal, news context, outcome + +**Step 3: Signal & strategy endpoints** + +- `GET /api/signals` — Recent signals with filters +- `GET /api/strategies` — All strategies with current weights +- `GET /api/strategies/:id/history` — Weight history and adjustments log +- `GET /api/strategies/:id/metrics` — Performance metrics over time + +**Step 4: News endpoints** + +- `GET /api/news` — Recent scored articles with filters (ticker, source, score range) + +**Step 5: Control endpoints** + +- `POST /api/controls/pause` — Pause trading (sets flag in Redis checked by trade executor) +- `POST /api/controls/resume` — Resume trading +- `POST /api/controls/close-position` — Force close a position by ticker +- `GET /api/controls/status` — Current trading status (active/paused) + +**Step 6: Backtest endpoints** + +- `POST /api/backtest/run` — Start a backtest with config (date range, capital, strategies, weights). Runs in background task. Returns run_id. +- `GET /api/backtest/:run_id` — Get backtest results (status, metrics, trade log, equity curve) + +**Step 7: WebSocket endpoint** + +`/ws` — Authenticated WebSocket. Pushes real-time events: trade executions, new signals, portfolio value updates, news sentiment scores. Uses Redis pub/sub or polling to pick up events from other services. + +**Step 8: Write tests** + +Test each endpoint group with test client, mock DB queries, verify response schemas. + +**Step 9: Commit** + +```bash +git add services/api-gateway/ tests/services/test_api_routes.py +git commit -m "feat: API gateway trading endpoints, controls, backtest, WebSocket" +``` + +--- + +### Task 15: Dashboard — Project Setup & Auth + +**Files:** +- Create: `dashboard/package.json` +- Create: `dashboard/tsconfig.json` +- Create: `dashboard/vite.config.ts` +- Create: `dashboard/tailwind.config.ts` +- Create: `dashboard/postcss.config.js` +- Create: `dashboard/index.html` +- Create: `dashboard/src/main.tsx` +- Create: `dashboard/src/App.tsx` +- Create: `dashboard/src/api/client.ts` +- Create: `dashboard/src/api/auth.ts` +- Create: `dashboard/src/pages/Login.tsx` +- Create: `dashboard/src/pages/Register.tsx` +- Create: `dashboard/src/hooks/useAuth.ts` +- Create: `dashboard/src/components/ProtectedRoute.tsx` + +**Step 1: Scaffold React project** + +Vite + React 18 + TypeScript. Install: `@tanstack/react-query`, `react-router-dom`, `tailwindcss`, `@simplewebauthn/browser`, `lightweight-charts` (TradingView), `recharts`. + +**Step 2: Create API client** + +Axios or fetch wrapper with JWT interceptor (auto-attach token, auto-refresh on 401). + +**Step 3: Create auth pages** + +Registration page: username input → calls `/auth/register/begin` → triggers browser passkey creation → sends attestation to `/auth/register/complete`. + +Login page: username input → calls `/auth/login/begin` → triggers browser passkey assertion → sends to `/auth/login/complete` → stores JWT. + +**Step 4: Create protected route wrapper** + +Redirects to login if no valid token. Wraps all dashboard routes. + +**Step 5: Commit** + +```bash +git add dashboard/ +git commit -m "feat: dashboard setup with passkey authentication" +``` + +--- + +### Task 16: Dashboard — Trading Views + +**Files:** +- Create: `dashboard/src/pages/Portfolio.tsx` +- Create: `dashboard/src/pages/TradeLog.tsx` +- Create: `dashboard/src/pages/Strategies.tsx` +- Create: `dashboard/src/pages/NewsFeed.tsx` +- Create: `dashboard/src/pages/Backtest.tsx` +- Create: `dashboard/src/components/EquityCurve.tsx` +- Create: `dashboard/src/components/PositionsTable.tsx` +- Create: `dashboard/src/components/TradeDetail.tsx` +- Create: `dashboard/src/components/StrategyWeights.tsx` +- Create: `dashboard/src/components/SentimentCard.tsx` +- Create: `dashboard/src/components/Layout.tsx` +- Create: `dashboard/src/hooks/useWebSocket.ts` +- Create: `dashboard/src/hooks/usePortfolio.ts` + +**Step 1: Create layout and navigation** + +Sidebar nav with links to all 5 views. Top bar with portfolio value summary and trading status indicator (active/paused). + +**Step 2: Portfolio Overview page** + +- Portfolio value card with daily P&L (green/red) +- Equity curve chart (TradingView lightweight-charts) from `/api/portfolio/history` +- Open positions table with unrealized P&L, close button per position +- Key metrics row: ROI, Sharpe, win rate, max drawdown + +**Step 3: Trade Log page** + +- Paginated table from `/api/trades` +- Filters: ticker search, date range, strategy dropdown, profitable toggle +- Expandable rows showing linked news, signal details, strategy attribution + +**Step 4: Strategy Performance page** + +- Strategy cards showing name, current weight, win rate, total P&L +- Weight allocation donut chart (Recharts) +- Weight history line chart per strategy +- Adjustments log table + +**Step 5: News & Sentiment Feed page** + +- Live feed of scored articles from `/api/news` +- Ticker filter +- Color-coded sentiment badges (green/yellow/red) +- Sparkline of recent sentiment per ticker + +**Step 6: Backtesting page** + +- Config form: date range pickers, initial capital, strategy checkboxes with weight sliders, slippage/commission inputs +- Submit → shows progress → displays results dashboard +- Results: equity curve, metrics summary, trade log, per-strategy breakdown + +**Step 7: WebSocket hook for real-time updates** + +`useWebSocket` hook connects to `/ws`, dispatches events to TanStack Query cache invalidation. Toast notifications for trade executions. + +**Step 8: Commit** + +```bash +git add dashboard/src/ +git commit -m "feat: dashboard trading views — portfolio, trades, strategies, news, backtest" +``` + +--- + +## Phase 7: Containerization & Integration + +### Task 17: Dockerfiles & Full Docker Compose + +**Files:** +- Create: `docker/Dockerfile.service` (multi-stage, shared for all Python services) +- Create: `docker/Dockerfile.dashboard` +- Modify: `docker-compose.yml` + +**Step 1: Create Python service Dockerfile** + +Multi-stage build. Stage 1: install deps from pyproject.toml with appropriate extras. Stage 2: slim runtime image. Configurable via build arg which service to run and which extras to install. + +```dockerfile +FROM python:3.12-slim AS builder +WORKDIR /app +COPY pyproject.toml . +ARG EXTRAS="api" +RUN pip install --no-cache-dir ".[${EXTRAS}]" + +FROM python:3.12-slim +WORKDIR /app +COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages +COPY --from=builder /usr/local/bin /usr/local/bin +COPY shared/ shared/ +ARG SERVICE_DIR +COPY services/${SERVICE_DIR}/ service/ +CMD ["python", "-m", "service.main"] +``` + +**Step 2: Create dashboard Dockerfile** + +Multi-stage: Node build stage → nginx serving static files. + +**Step 3: Update docker-compose.yml with all services** + +Add all 6 Python services + dashboard container. Each service gets: +- Correct build args (EXTRAS, SERVICE_DIR) +- Depends-on for postgres, redis +- Environment variables from `.env` +- Health check +- Metrics port exposed for Prometheus scraping + +**Step 4: Write a smoke test script** + +`scripts/smoke-test.sh` — Boots full stack, waits for health checks, hits key endpoints, verifies non-error responses. + +**Step 5: Commit** + +```bash +git add docker/ docker-compose.yml scripts/ +git commit -m "feat: dockerfiles and full docker-compose orchestration" +``` + +--- + +### Task 18: Integration Testing & Seed Data + +**Files:** +- Create: `tests/integration/test_news_pipeline.py` +- Create: `tests/integration/test_trading_flow.py` +- Create: `scripts/seed_strategies.py` + +**Step 1: Seed default strategies** + +Script to insert the 3 strategies (momentum, mean_reversion, news_driven) with equal initial weights (0.333 each) into the `strategies` table. + +**Step 2: Write news pipeline integration test** + +Test the full flow: publish a mock RSS article → news fetcher picks it up → sentiment analyzer scores it → signal generator evaluates it. Uses real Redis + PostgreSQL (from docker-compose), mocked Alpaca and FinBERT. + +**Step 3: Write trading flow integration test** + +Test: inject a signal → trade executor processes it (mocked Alpaca broker) → trade recorded in DB → learning engine evaluates outcome. + +**Step 4: Commit** + +```bash +git add tests/integration/ scripts/seed_strategies.py +git commit -m "feat: integration tests and strategy seed data" +``` + +--- + +## Task Dependencies + +``` +Task 1 (foundation) ──┬── Task 2 (docker infra) + ├── Task 3 (models) ── Task 4 (schemas) + │ │ + │ ┌───────────┤ + │ │ │ + │ Task 5 (broker) │ + │ │ │ + │ ┌────────┤ Task 6 (news fetcher) + │ │ │ │ + │ │ Task 8 (strategies) Task 7 (sentiment) + │ │ │ │ + │ │ Task 9 (signal gen) ─┘ + │ │ │ + │ │ Task 10 (executor) + │ │ │ + │ │ Task 11 (learning) + │ │ │ + │ │ Task 12 (backtester) + │ │ + │ └── Task 13 (API auth) ── Task 14 (API routes) + │ │ + │ Task 15 (dashboard setup) + │ │ + │ Task 16 (dashboard views) + │ + └── Task 17 (docker) ── Task 18 (integration) +``` + +**Parallelizable pairs:** +- Task 2 + Task 3 (infra + models) +- Task 5 + Task 6 (broker + news fetcher) after Task 4 +- Task 7 + Task 8 (sentiment + strategies) after Task 4 +- Task 13 + Task 9 (API auth + signal gen) after their deps +- Task 15 + Task 12 (dashboard setup + backtester) after their deps