trading/.planning/codebase/INTEGRATIONS.md

# External Integrations

**Analysis Date:** 2025-02-23

## APIs & External Services

**Brokerage Trading:**
- Alpaca Markets - Paper and live trading
  - SDK/Client: `alpaca-py` (21.0+)
  - API Key: `TRADING_ALPACA_API_KEY` env var
  - Secret: `TRADING_ALPACA_SECRET_KEY` env var
  - Base URL: `TRADING_ALPACA_BASE_URL` (defaults to paper trading endpoint)
  - Paper trading: `TRADING_PAPER_TRADING=true` flag in config
  - Used by:
    - `shared/broker/alpaca_broker.py` - Order execution, position tracking, account info via `TradingClient`
    - `services/market_data/main.py` - Historical bars via `StockHistoricalDataClient` and live streaming
    - `services/api_gateway/tasks/portfolio_sync.py` - Portfolio synchronization background task
    - `services/trade_executor/main.py` - Order placement and status monitoring

**News & Content Sources:**
- Yahoo Finance RSS - Financial news feed
  - Feed URL: `https://finance.yahoo.com/news/rssindex`
  - Client: feedparser 6.0+
  - Polling: `TRADING_RSS_POLL_INTERVAL_SECONDS` (default 300s)
  - Configured in: `services/news_fetcher/config.py`

- Reuters RSS - Business news feed
  - Feed URL: `https://feeds.reuters.com/reuters/businessNews`
  - Client: feedparser 6.0+

- Dow Jones RSS - Market news feed
  - Feed URL: `https://feeds.content.dowjones.io/public/rss/mw_topstories`
  - Client: feedparser 6.0+

- Reddit - Wallstreetbets, stocks, investing communities
  - SDK/Client: praw 7.7+ (blocking) and asyncpraw 7.7+ (async)
  - Client ID: `TRADING_REDDIT_CLIENT_ID` env var
  - Client Secret: `TRADING_REDDIT_CLIENT_SECRET` env var
  - User Agent: `TRADING_REDDIT_USER_AGENT` (default: `trading-bot/0.1`)
  - Subreddits: `["wallstreetbets", "stocks", "investing"]` (configurable via `TRADING_REDDIT_SUBREDDITS`)
  - Min score filter: `TRADING_REDDIT_MIN_SCORE` (default 10)
  - Polling: `TRADING_REDDIT_POLL_INTERVAL_SECONDS` (default 600s)
  - Used by: `services/news_fetcher/main.py`

**Machine Learning Models:**
- FinBERT (Hugging Face) - Financial sentiment classification
  - Model: `ProsusAI/finbert` (transformer)
  - Confidence threshold: `TRADING_FINBERT_CONFIDENCE_THRESHOLD` (default 0.4)
  - Library: transformers 4.38+
  - Used by: `services/sentiment_analyzer/analyzers/finbert.py`
  - Location: Loaded from Hugging Face model hub

- Ollama - Local LLM for fallback sentiment analysis
  - Host: `TRADING_OLLAMA_HOST` (default `http://localhost:11434`)
  - Model: `TRADING_OLLAMA_MODEL` (default `gemma3`)
  - Used when: FinBERT confidence below threshold
  - Library: ollama-python 0.1+
  - Used by: `services/sentiment_analyzer/analyzers/ollama_analyzer.py`

## Data Storage

**Databases:**
- PostgreSQL 16 + TimescaleDB extension (Docker: `timescale/timescaledb:latest-pg16`)
  - Connection: Async via asyncpg driver
  - URL: `TRADING_DATABASE_URL` env var (default: `postgresql+asyncpg://trading:trading@localhost:5432/trading`)
  - Username: `trading` (env: `POSTGRES_USER`)
  - Password: `POSTGRES_PASSWORD` env var
  - Database: `trading`
  - ORM: SQLAlchemy 2.0+ with mapped_column style
  - Migrations: Alembic (auto-run via `python -m alembic upgrade head`)
  - Tables (14 models in `shared/models/`):
    - Auth: `users`, `webauthn_credentials`
    - News: `articles`, `article_sentiments`
    - Trading: `strategies`, `trade_signals`, `executed_trades`, `positions`
    - Learning: `strategy_weights`, `weight_history`
    - Market data: OHLCV timeseries via TimescaleDB hypertables
  - Used by: All services via `shared/db.py` async sessionmaker

**File Storage:**
- Local filesystem only (no cloud storage)
- Docker volumes for persistent data:
  - `pgdata` - PostgreSQL persistent storage
  - `redisdata` - Redis persistent storage

**Caching & Message Broker:**
- Redis 7-alpine (Docker image)
  - Connection: `TRADING_REDIS_URL` env var (default: `redis://localhost:6379/0`)
  - Default port: 6379
  - Persistent volume: `redisdata`
  - Features used:
    - Streams (consumer groups) for inter-service messaging
    - Key-value cache for rate limiting
  - Stream topics (producer/consumer contracts):
    - `news:raw` - Raw articles from news fetcher → consumed by sentiment analyzer
    - `news:scored` - Scored articles from sentiment analyzer → consumed by signal generator
    - `signals:generated` - Trade signals from signal generator → consumed by trade executor and learning engine
    - `trades:executed` - Executed trades from trade executor → consumed by learning engine
  - Client library: redis 5.0+ with async support
  - Consumer group per service (e.g., `sentiment-analyzer-group` on `news:raw` stream)

## Authentication & Identity

**Auth Provider:**
- Custom WebAuthn/Passkey implementation (no third-party OAuth provider)
  - Library: webauthn 2.0+ (Python backend)
  - Frontend: @simplewebauthn/browser 13.2.2 (JavaScript/TypeScript)
  - Flow: WebAuthn registration → credential storage → authentication challenge verification
  - Credentials storage: `webauthn_credentials` table in PostgreSQL
  - Implementation: `services/api_gateway/auth/routes.py` (register/login/verify endpoints)

**Session Management:**
- JWT tokens with HS256 (HMAC-SHA256)
  - Library: PyJWT 2.8+ with crypto support
  - Secret key: `TRADING_JWT_SECRET_KEY` env var (REQUIRED, must be 32+ bytes in production)
  - Algorithm: `TRADING_JWT_ALGORITHM` (default: HS256)
  - Access token expiry: `TRADING_ACCESS_TOKEN_EXPIRE_MINUTES` (default 15)
  - Refresh token expiry: `TRADING_REFRESH_TOKEN_EXPIRE_DAYS` (default 7)
  - Token verification: `services/api_gateway/auth/middleware.py` (Bearer token extraction and validation)
  - Implementation: `services/api_gateway/auth/jwt.py`

## Monitoring & Observability

**Error Tracking:**
- None detected - No Sentry or other APM integration
- Logging: Standard Python logging module with `TRADING_LOG_LEVEL` env var

**Logs:**
- Console logging to stdout/stderr
- Log level controlled by: `TRADING_LOG_LEVEL` env var (default: INFO)
- Each service logs to standard output (captured by Docker/container orchestration)

**Metrics & Telemetry:**
- OpenTelemetry 1.20+ - Metrics collection and export
- PrometheusMetricReader - Metrics exporter
- prometheus-client - HTTP `/metrics` endpoint
- Setup: `shared/telemetry.py` - `setup_telemetry()` called by each service
- Metrics port: `TRADING_OTEL_METRICS_PORT` env var (default 9090)
- Service name: `TRADING_OTEL_SERVICE_NAME` env var (default: `trading-bot`)
- Scrape endpoint: `http://<service>:9090/metrics` for each service
- Note: Prometheus/Grafana not deployed; metrics available for external monitoring

## CI/CD & Deployment

**Hosting:**
- Docker & Docker Compose (local development and self-hosted deployment)
- No managed hosting detected (e.g., AWS, Heroku, GCP)

**CI Pipeline:**
- None detected - No GitHub Actions, GitLab CI, or other CI/CD pipeline

**Docker Composition:**
- Services defined in `docker-compose.yml`:
  - `postgres` - Database
  - `redis` - Message broker
  - `migrations` - Database schema initialization
  - `news-fetcher` - Service
  - `sentiment-analyzer` - Service
  - `signal-generator` - Service
  - `trade-executor` - Service
  - `learning-engine` - Service
  - `market-data` - Service
  - `api-gateway` - FastAPI + WebSocket server (port 8000)
  - `dashboard` - Nginx serving React build (port 3000)

## Environment Configuration

**Required env vars:**
- `TRADING_JWT_SECRET_KEY` - **REQUIRED** for API Gateway JWT signing (must be set, generate with `python -c "import secrets; print(secrets.token_hex(32))"`)
- `TRADING_ALPACA_API_KEY` - Alpaca account API key
- `TRADING_ALPACA_SECRET_KEY` - Alpaca account secret
- `TRADING_REDDIT_CLIENT_ID` - Reddit app client ID
- `TRADING_REDDIT_CLIENT_SECRET` - Reddit app secret

**Optional env vars with defaults:**
- `TRADING_DATABASE_URL` - Default: `postgresql+asyncpg://trading:trading@localhost:5432/trading`
- `TRADING_REDIS_URL` - Default: `redis://localhost:6379/0`
- `TRADING_LOG_LEVEL` - Default: `INFO`
- `TRADING_ALPACA_BASE_URL` - Default: Paper trading endpoint
- `TRADING_PAPER_TRADING` - Default: `true`
- `TRADING_OLLAMA_HOST` - Default: `http://localhost:11434`
- `TRADING_OLLAMA_MODEL` - Default: `gemma3`
- `TRADING_FINBERT_CONFIDENCE_THRESHOLD` - Default: `0.4`
- `TRADING_RSS_POLL_INTERVAL_SECONDS` - Default: `300`
- `TRADING_REDDIT_POLL_INTERVAL_SECONDS` - Default: `600`
- `TRADING_RP_ID` - Default: `localhost`
- `TRADING_RP_NAME` - Default: `Trading Bot`
- `TRADING_RP_ORIGIN` - Default: `http://localhost:5173`
- `TRADING_CORS_ORIGINS` - Default: `["http://localhost:5173"]`
- `POSTGRES_PASSWORD` - Default: `trading`

**Secrets location:**
- `.env` file (git-ignored, loads via docker-compose `env_file` directive)
- Environment variables passed at container runtime
- No secrets management service (Vault, Secrets Manager) integrated

## Webhooks & Callbacks

**Incoming:**
- None detected - No webhook endpoints for external services

**Outgoing:**
- None detected - No outbound webhooks to external systems

**Real-time Communication:**
- WebSocket support via `websockets` library for API Gateway
- Used for: Real-time dashboard updates (TBD implementation in routes)
- Endpoint: `/ws` proxy configured in Vite dev server

## Service Dependencies (Internal Messaging)

**Message Flows:**
1. News Fetcher → (`news:raw` stream) → Sentiment Analyzer
2. Sentiment Analyzer → (`news:scored` stream) → Signal Generator
3. Signal Generator → (`signals:generated` stream) → Trade Executor + Learning Engine
4. Trade Executor → (`trades:executed` stream) → Learning Engine
5. Market Data → Alpaca API → Database (timeseries persisted)
6. API Gateway → Alpaca API → Portfolio sync background task

---

*Integration audit: 2025-02-23*