viktor/trading
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
trading/.planning/codebase/INTEGRATIONS.md
Viktor Barzin bc34c78072
docs: map existing codebase
2026-02-23 20:04:05 +00:00

9.5 KiB
Raw Blame History

External Integrations

Analysis Date: 2025-02-23

APIs & External Services

Brokerage Trading:

  • Alpaca Markets - Paper and live trading
    • SDK/Client: alpaca-py (21.0+)
    • API Key: TRADING_ALPACA_API_KEY env var
    • Secret: TRADING_ALPACA_SECRET_KEY env var
    • Base URL: TRADING_ALPACA_BASE_URL (defaults to paper trading endpoint)
    • Paper trading: TRADING_PAPER_TRADING=true flag in config
    • Used by:
      • shared/broker/alpaca_broker.py - Order execution, position tracking, account info via TradingClient
      • services/market_data/main.py - Historical bars via StockHistoricalDataClient and live streaming
      • services/api_gateway/tasks/portfolio_sync.py - Portfolio synchronization background task
      • services/trade_executor/main.py - Order placement and status monitoring

News & Content Sources:

  • Yahoo Finance RSS - Financial news feed

    • Feed URL: https://finance.yahoo.com/news/rssindex
    • Client: feedparser 6.0+
    • Polling: TRADING_RSS_POLL_INTERVAL_SECONDS (default 300s)
    • Configured in: services/news_fetcher/config.py

  • Reuters RSS - Business news feed

    • Feed URL: https://feeds.reuters.com/reuters/businessNews
    • Client: feedparser 6.0+

  • Dow Jones RSS - Market news feed

    • Feed URL: https://feeds.content.dowjones.io/public/rss/mw_topstories
    • Client: feedparser 6.0+

  • Reddit - Wallstreetbets, stocks, investing communities

    • SDK/Client: praw 7.7+ (blocking) and asyncpraw 7.7+ (async)
    • Client ID: TRADING_REDDIT_CLIENT_ID env var
    • Client Secret: TRADING_REDDIT_CLIENT_SECRET env var
    • User Agent: TRADING_REDDIT_USER_AGENT (default: trading-bot/0.1)
    • Subreddits: ["wallstreetbets", "stocks", "investing"] (configurable via TRADING_REDDIT_SUBREDDITS)
    • Min score filter: TRADING_REDDIT_MIN_SCORE (default 10)
    • Polling: TRADING_REDDIT_POLL_INTERVAL_SECONDS (default 600s)
    • Used by: services/news_fetcher/main.py

Machine Learning Models:

  • FinBERT (Hugging Face) - Financial sentiment classification

    • Model: ProsusAI/finbert (transformer)
    • Confidence threshold: TRADING_FINBERT_CONFIDENCE_THRESHOLD (default 0.4)
    • Library: transformers 4.38+
    • Used by: services/sentiment_analyzer/analyzers/finbert.py
    • Location: Loaded from Hugging Face model hub

  • Ollama - Local LLM for fallback sentiment analysis

    • Host: TRADING_OLLAMA_HOST (default http://localhost:11434)
    • Model: TRADING_OLLAMA_MODEL (default gemma3)
    • Used when: FinBERT confidence below threshold
    • Library: ollama-python 0.1+
    • Used by: services/sentiment_analyzer/analyzers/ollama_analyzer.py

Data Storage

Databases:

  • PostgreSQL 16 + TimescaleDB extension (Docker: timescale/timescaledb:latest-pg16)
    • Connection: Async via asyncpg driver
    • URL: TRADING_DATABASE_URL env var (default: postgresql+asyncpg://trading:trading@localhost:5432/trading)
    • Username: trading (env: POSTGRES_USER)
    • Password: POSTGRES_PASSWORD env var
    • Database: trading
    • ORM: SQLAlchemy 2.0+ with mapped_column style
    • Migrations: Alembic (auto-run via python -m alembic upgrade head)
    • Tables (14 models in shared/models/):
      • Auth: users, webauthn_credentials
      • News: articles, article_sentiments
      • Trading: strategies, trade_signals, executed_trades, positions
      • Learning: strategy_weights, weight_history
      • Market data: OHLCV timeseries via TimescaleDB hypertables
    • Used by: All services via shared/db.py async sessionmaker

File Storage:

  • Local filesystem only (no cloud storage)
  • Docker volumes for persistent data:
    • pgdata - PostgreSQL persistent storage
    • redisdata - Redis persistent storage

Caching & Message Broker:

  • Redis 7-alpine (Docker image)
    • Connection: TRADING_REDIS_URL env var (default: redis://localhost:6379/0)
    • Default port: 6379
    • Persistent volume: redisdata
    • Features used:
      • Streams (consumer groups) for inter-service messaging
      • Key-value cache for rate limiting
    • Stream topics (producer/consumer contracts):
      • news:raw - Raw articles from news fetcher → consumed by sentiment analyzer
      • news:scored - Scored articles from sentiment analyzer → consumed by signal generator
      • signals:generated - Trade signals from signal generator → consumed by trade executor and learning engine
      • trades:executed - Executed trades from trade executor → consumed by learning engine
    • Client library: redis 5.0+ with async support
    • Consumer group per service (e.g., sentiment-analyzer-group on news:raw stream)

Authentication & Identity

Auth Provider:

  • Custom WebAuthn/Passkey implementation (no third-party OAuth provider)
    • Library: webauthn 2.0+ (Python backend)
    • Frontend: @simplewebauthn/browser 13.2.2 (JavaScript/TypeScript)
    • Flow: WebAuthn registration → credential storage → authentication challenge verification
    • Credentials storage: webauthn_credentials table in PostgreSQL
    • Implementation: services/api_gateway/auth/routes.py (register/login/verify endpoints)

Session Management:

  • JWT tokens with HS256 (HMAC-SHA256)
    • Library: PyJWT 2.8+ with crypto support
    • Secret key: TRADING_JWT_SECRET_KEY env var (REQUIRED, must be 32+ bytes in production)
    • Algorithm: TRADING_JWT_ALGORITHM (default: HS256)
    • Access token expiry: TRADING_ACCESS_TOKEN_EXPIRE_MINUTES (default 15)
    • Refresh token expiry: TRADING_REFRESH_TOKEN_EXPIRE_DAYS (default 7)
    • Token verification: services/api_gateway/auth/middleware.py (Bearer token extraction and validation)
    • Implementation: services/api_gateway/auth/jwt.py

Monitoring & Observability

Error Tracking:

  • None detected - No Sentry or other APM integration
  • Logging: Standard Python logging module with TRADING_LOG_LEVEL env var

Logs:

  • Console logging to stdout/stderr
  • Log level controlled by: TRADING_LOG_LEVEL env var (default: INFO)
  • Each service logs to standard output (captured by Docker/container orchestration)

Metrics & Telemetry:

  • OpenTelemetry 1.20+ - Metrics collection and export
  • PrometheusMetricReader - Metrics exporter
  • prometheus-client - HTTP /metrics endpoint
  • Setup: shared/telemetry.py - setup_telemetry() called by each service
  • Metrics port: TRADING_OTEL_METRICS_PORT env var (default 9090)
  • Service name: TRADING_OTEL_SERVICE_NAME env var (default: trading-bot)
  • Scrape endpoint: http://<service>:9090/metrics for each service
  • Note: Prometheus/Grafana not deployed; metrics available for external monitoring

CI/CD & Deployment

Hosting:

  • Docker & Docker Compose (local development and self-hosted deployment)
  • No managed hosting detected (e.g., AWS, Heroku, GCP)

CI Pipeline:

  • None detected - No GitHub Actions, GitLab CI, or other CI/CD pipeline

Docker Composition:

  • Services defined in docker-compose.yml:
    • postgres - Database
    • redis - Message broker
    • migrations - Database schema initialization
    • news-fetcher - Service
    • sentiment-analyzer - Service
    • signal-generator - Service
    • trade-executor - Service
    • learning-engine - Service
    • market-data - Service
    • api-gateway - FastAPI + WebSocket server (port 8000)
    • dashboard - Nginx serving React build (port 3000)

Environment Configuration

Required env vars:

  • TRADING_JWT_SECRET_KEY - REQUIRED for API Gateway JWT signing (must be set, generate with python -c "import secrets; print(secrets.token_hex(32))")
  • TRADING_ALPACA_API_KEY - Alpaca account API key
  • TRADING_ALPACA_SECRET_KEY - Alpaca account secret
  • TRADING_REDDIT_CLIENT_ID - Reddit app client ID
  • TRADING_REDDIT_CLIENT_SECRET - Reddit app secret

Optional env vars with defaults:

  • TRADING_DATABASE_URL - Default: postgresql+asyncpg://trading:trading@localhost:5432/trading
  • TRADING_REDIS_URL - Default: redis://localhost:6379/0
  • TRADING_LOG_LEVEL - Default: INFO
  • TRADING_ALPACA_BASE_URL - Default: Paper trading endpoint
  • TRADING_PAPER_TRADING - Default: true
  • TRADING_OLLAMA_HOST - Default: http://localhost:11434
  • TRADING_OLLAMA_MODEL - Default: gemma3
  • TRADING_FINBERT_CONFIDENCE_THRESHOLD - Default: 0.4
  • TRADING_RSS_POLL_INTERVAL_SECONDS - Default: 300
  • TRADING_REDDIT_POLL_INTERVAL_SECONDS - Default: 600
  • TRADING_RP_ID - Default: localhost
  • TRADING_RP_NAME - Default: Trading Bot
  • TRADING_RP_ORIGIN - Default: http://localhost:5173
  • TRADING_CORS_ORIGINS - Default: ["http://localhost:5173"]
  • POSTGRES_PASSWORD - Default: trading

Secrets location:

  • .env file (git-ignored, loads via docker-compose env_file directive)
  • Environment variables passed at container runtime
  • No secrets management service (Vault, Secrets Manager) integrated

Webhooks & Callbacks

Incoming:

  • None detected - No webhook endpoints for external services

Outgoing:

  • None detected - No outbound webhooks to external systems

Real-time Communication:

  • WebSocket support via websockets library for API Gateway
  • Used for: Real-time dashboard updates (TBD implementation in routes)
  • Endpoint: /ws proxy configured in Vite dev server

Service Dependencies (Internal Messaging)

Message Flows:

  1. News Fetcher → (news:raw stream) → Sentiment Analyzer
  2. Sentiment Analyzer → (news:scored stream) → Signal Generator
  3. Signal Generator → (signals:generated stream) → Trade Executor + Learning Engine
  4. Trade Executor → (trades:executed stream) → Learning Engine
  5. Market Data → Alpaca API → Database (timeseries persisted)
  6. API Gateway → Alpaca API → Portfolio sync background task

Integration audit: 2025-02-23