docs: map existing codebase
This commit is contained in:
Viktor Barzin
parent
d36ae40df1
commit
bc34c78072
7 changed files with 2003 additions and 0 deletions
221
.planning/codebase/INTEGRATIONS.md
Normal file
221
.planning/codebase/INTEGRATIONS.md
Normal file
|
|
@ -0,0 +1,221 @@
|
|||
# External Integrations
|
||||
|
||||
**Analysis Date:** 2025-02-23
|
||||
|
||||
## APIs & External Services
|
||||
|
||||
**Brokerage Trading:**
|
||||
- Alpaca Markets - Paper and live trading
|
||||
- SDK/Client: `alpaca-py` (21.0+)
|
||||
- API Key: `TRADING_ALPACA_API_KEY` env var
|
||||
- Secret: `TRADING_ALPACA_SECRET_KEY` env var
|
||||
- Base URL: `TRADING_ALPACA_BASE_URL` (defaults to paper trading endpoint)
|
||||
- Paper trading: `TRADING_PAPER_TRADING=true` flag in config
|
||||
- Used by:
|
||||
- `shared/broker/alpaca_broker.py` - Order execution, position tracking, account info via `TradingClient`
|
||||
- `services/market_data/main.py` - Historical bars via `StockHistoricalDataClient` and live streaming
|
||||
- `services/api_gateway/tasks/portfolio_sync.py` - Portfolio synchronization background task
|
||||
- `services/trade_executor/main.py` - Order placement and status monitoring
|
||||
|
||||
**News & Content Sources:**
|
||||
- Yahoo Finance RSS - Financial news feed
|
||||
- Feed URL: `https://finance.yahoo.com/news/rssindex`
|
||||
- Client: feedparser 6.0+
|
||||
- Polling: `TRADING_RSS_POLL_INTERVAL_SECONDS` (default 300s)
|
||||
- Configured in: `services/news_fetcher/config.py`
|
||||
|
||||
- Reuters RSS - Business news feed
|
||||
- Feed URL: `https://feeds.reuters.com/reuters/businessNews`
|
||||
- Client: feedparser 6.0+
|
||||
|
||||
- Dow Jones RSS - Market news feed
|
||||
- Feed URL: `https://feeds.content.dowjones.io/public/rss/mw_topstories`
|
||||
- Client: feedparser 6.0+
|
||||
|
||||
- Reddit - Wallstreetbets, stocks, investing communities
|
||||
- SDK/Client: praw 7.7+ (blocking) and asyncpraw 7.7+ (async)
|
||||
- Client ID: `TRADING_REDDIT_CLIENT_ID` env var
|
||||
- Client Secret: `TRADING_REDDIT_CLIENT_SECRET` env var
|
||||
- User Agent: `TRADING_REDDIT_USER_AGENT` (default: `trading-bot/0.1`)
|
||||
- Subreddits: `["wallstreetbets", "stocks", "investing"]` (configurable via `TRADING_REDDIT_SUBREDDITS`)
|
||||
- Min score filter: `TRADING_REDDIT_MIN_SCORE` (default 10)
|
||||
- Polling: `TRADING_REDDIT_POLL_INTERVAL_SECONDS` (default 600s)
|
||||
- Used by: `services/news_fetcher/main.py`
|
||||
|
||||
**Machine Learning Models:**
|
||||
- FinBERT (Hugging Face) - Financial sentiment classification
|
||||
- Model: `ProsusAI/finbert` (transformer)
|
||||
- Confidence threshold: `TRADING_FINBERT_CONFIDENCE_THRESHOLD` (default 0.4)
|
||||
- Library: transformers 4.38+
|
||||
- Used by: `services/sentiment_analyzer/analyzers/finbert.py`
|
||||
- Location: Loaded from Hugging Face model hub
|
||||
|
||||
- Ollama - Local LLM for fallback sentiment analysis
|
||||
- Host: `TRADING_OLLAMA_HOST` (default `http://localhost:11434`)
|
||||
- Model: `TRADING_OLLAMA_MODEL` (default `gemma3`)
|
||||
- Used when: FinBERT confidence below threshold
|
||||
- Library: ollama-python 0.1+
|
||||
- Used by: `services/sentiment_analyzer/analyzers/ollama_analyzer.py`
|
||||
|
||||
## Data Storage
|
||||
|
||||
**Databases:**
|
||||
- PostgreSQL 16 + TimescaleDB extension (Docker: `timescale/timescaledb:latest-pg16`)
|
||||
- Connection: Async via asyncpg driver
|
||||
- URL: `TRADING_DATABASE_URL` env var (default: `postgresql+asyncpg://trading:trading@localhost:5432/trading`)
|
||||
- Username: `trading` (env: `POSTGRES_USER`)
|
||||
- Password: `POSTGRES_PASSWORD` env var
|
||||
- Database: `trading`
|
||||
- ORM: SQLAlchemy 2.0+ with mapped_column style
|
||||
- Migrations: Alembic (auto-run via `python -m alembic upgrade head`)
|
||||
- Tables (14 models in `shared/models/`):
|
||||
- Auth: `users`, `webauthn_credentials`
|
||||
- News: `articles`, `article_sentiments`
|
||||
- Trading: `strategies`, `trade_signals`, `executed_trades`, `positions`
|
||||
- Learning: `strategy_weights`, `weight_history`
|
||||
- Market data: OHLCV timeseries via TimescaleDB hypertables
|
||||
- Used by: All services via `shared/db.py` async sessionmaker
|
||||
|
||||
**File Storage:**
|
||||
- Local filesystem only (no cloud storage)
|
||||
- Docker volumes for persistent data:
|
||||
- `pgdata` - PostgreSQL persistent storage
|
||||
- `redisdata` - Redis persistent storage
|
||||
|
||||
**Caching & Message Broker:**
|
||||
- Redis 7-alpine (Docker image)
|
||||
- Connection: `TRADING_REDIS_URL` env var (default: `redis://localhost:6379/0`)
|
||||
- Default port: 6379
|
||||
- Persistent volume: `redisdata`
|
||||
- Features used:
|
||||
- Streams (consumer groups) for inter-service messaging
|
||||
- Key-value cache for rate limiting
|
||||
- Stream topics (producer/consumer contracts):
|
||||
- `news:raw` - Raw articles from news fetcher → consumed by sentiment analyzer
|
||||
- `news:scored` - Scored articles from sentiment analyzer → consumed by signal generator
|
||||
- `signals:generated` - Trade signals from signal generator → consumed by trade executor and learning engine
|
||||
- `trades:executed` - Executed trades from trade executor → consumed by learning engine
|
||||
- Client library: redis 5.0+ with async support
|
||||
- Consumer group per service (e.g., `sentiment-analyzer-group` on `news:raw` stream)
|
||||
|
||||
## Authentication & Identity
|
||||
|
||||
**Auth Provider:**
|
||||
- Custom WebAuthn/Passkey implementation (no third-party OAuth provider)
|
||||
- Library: webauthn 2.0+ (Python backend)
|
||||
- Frontend: @simplewebauthn/browser 13.2.2 (JavaScript/TypeScript)
|
||||
- Flow: WebAuthn registration → credential storage → authentication challenge verification
|
||||
- Credentials storage: `webauthn_credentials` table in PostgreSQL
|
||||
- Implementation: `services/api_gateway/auth/routes.py` (register/login/verify endpoints)
|
||||
|
||||
**Session Management:**
|
||||
- JWT tokens with HS256 (HMAC-SHA256)
|
||||
- Library: PyJWT 2.8+ with crypto support
|
||||
- Secret key: `TRADING_JWT_SECRET_KEY` env var (REQUIRED, must be 32+ bytes in production)
|
||||
- Algorithm: `TRADING_JWT_ALGORITHM` (default: HS256)
|
||||
- Access token expiry: `TRADING_ACCESS_TOKEN_EXPIRE_MINUTES` (default 15)
|
||||
- Refresh token expiry: `TRADING_REFRESH_TOKEN_EXPIRE_DAYS` (default 7)
|
||||
- Token verification: `services/api_gateway/auth/middleware.py` (Bearer token extraction and validation)
|
||||
- Implementation: `services/api_gateway/auth/jwt.py`
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
**Error Tracking:**
|
||||
- None detected - No Sentry or other APM integration
|
||||
- Logging: Standard Python logging module with `TRADING_LOG_LEVEL` env var
|
||||
|
||||
**Logs:**
|
||||
- Console logging to stdout/stderr
|
||||
- Log level controlled by: `TRADING_LOG_LEVEL` env var (default: INFO)
|
||||
- Each service logs to standard output (captured by Docker/container orchestration)
|
||||
|
||||
**Metrics & Telemetry:**
|
||||
- OpenTelemetry 1.20+ - Metrics collection and export
|
||||
- PrometheusMetricReader - Metrics exporter
|
||||
- prometheus-client - HTTP `/metrics` endpoint
|
||||
- Setup: `shared/telemetry.py` - `setup_telemetry()` called by each service
|
||||
- Metrics port: `TRADING_OTEL_METRICS_PORT` env var (default 9090)
|
||||
- Service name: `TRADING_OTEL_SERVICE_NAME` env var (default: `trading-bot`)
|
||||
- Scrape endpoint: `http://<service>:9090/metrics` for each service
|
||||
- Note: Prometheus/Grafana not deployed; metrics available for external monitoring
|
||||
|
||||
## CI/CD & Deployment
|
||||
|
||||
**Hosting:**
|
||||
- Docker & Docker Compose (local development and self-hosted deployment)
|
||||
- No managed hosting detected (e.g., AWS, Heroku, GCP)
|
||||
|
||||
**CI Pipeline:**
|
||||
- None detected - No GitHub Actions, GitLab CI, or other CI/CD pipeline
|
||||
|
||||
**Docker Composition:**
|
||||
- Services defined in `docker-compose.yml`:
|
||||
- `postgres` - Database
|
||||
- `redis` - Message broker
|
||||
- `migrations` - Database schema initialization
|
||||
- `news-fetcher` - Service
|
||||
- `sentiment-analyzer` - Service
|
||||
- `signal-generator` - Service
|
||||
- `trade-executor` - Service
|
||||
- `learning-engine` - Service
|
||||
- `market-data` - Service
|
||||
- `api-gateway` - FastAPI + WebSocket server (port 8000)
|
||||
- `dashboard` - Nginx serving React build (port 3000)
|
||||
|
||||
## Environment Configuration
|
||||
|
||||
**Required env vars:**
|
||||
- `TRADING_JWT_SECRET_KEY` - **REQUIRED** for API Gateway JWT signing (must be set, generate with `python -c "import secrets; print(secrets.token_hex(32))"`)
|
||||
- `TRADING_ALPACA_API_KEY` - Alpaca account API key
|
||||
- `TRADING_ALPACA_SECRET_KEY` - Alpaca account secret
|
||||
- `TRADING_REDDIT_CLIENT_ID` - Reddit app client ID
|
||||
- `TRADING_REDDIT_CLIENT_SECRET` - Reddit app secret
|
||||
|
||||
**Optional env vars with defaults:**
|
||||
- `TRADING_DATABASE_URL` - Default: `postgresql+asyncpg://trading:trading@localhost:5432/trading`
|
||||
- `TRADING_REDIS_URL` - Default: `redis://localhost:6379/0`
|
||||
- `TRADING_LOG_LEVEL` - Default: `INFO`
|
||||
- `TRADING_ALPACA_BASE_URL` - Default: Paper trading endpoint
|
||||
- `TRADING_PAPER_TRADING` - Default: `true`
|
||||
- `TRADING_OLLAMA_HOST` - Default: `http://localhost:11434`
|
||||
- `TRADING_OLLAMA_MODEL` - Default: `gemma3`
|
||||
- `TRADING_FINBERT_CONFIDENCE_THRESHOLD` - Default: `0.4`
|
||||
- `TRADING_RSS_POLL_INTERVAL_SECONDS` - Default: `300`
|
||||
- `TRADING_REDDIT_POLL_INTERVAL_SECONDS` - Default: `600`
|
||||
- `TRADING_RP_ID` - Default: `localhost`
|
||||
- `TRADING_RP_NAME` - Default: `Trading Bot`
|
||||
- `TRADING_RP_ORIGIN` - Default: `http://localhost:5173`
|
||||
- `TRADING_CORS_ORIGINS` - Default: `["http://localhost:5173"]`
|
||||
- `POSTGRES_PASSWORD` - Default: `trading`
|
||||
|
||||
**Secrets location:**
|
||||
- `.env` file (git-ignored, loads via docker-compose `env_file` directive)
|
||||
- Environment variables passed at container runtime
|
||||
- No secrets management service (Vault, Secrets Manager) integrated
|
||||
|
||||
## Webhooks & Callbacks
|
||||
|
||||
**Incoming:**
|
||||
- None detected - No webhook endpoints for external services
|
||||
|
||||
**Outgoing:**
|
||||
- None detected - No outbound webhooks to external systems
|
||||
|
||||
**Real-time Communication:**
|
||||
- WebSocket support via `websockets` library for API Gateway
|
||||
- Used for: Real-time dashboard updates (TBD implementation in routes)
|
||||
- Endpoint: `/ws` proxy configured in Vite dev server
|
||||
|
||||
## Service Dependencies (Internal Messaging)
|
||||
|
||||
**Message Flows:**
|
||||
1. News Fetcher → (`news:raw` stream) → Sentiment Analyzer
|
||||
2. Sentiment Analyzer → (`news:scored` stream) → Signal Generator
|
||||
3. Signal Generator → (`signals:generated` stream) → Trade Executor + Learning Engine
|
||||
4. Trade Executor → (`trades:executed` stream) → Learning Engine
|
||||
5. Market Data → Alpaca API → Database (timeseries persisted)
|
||||
6. API Gateway → Alpaca API → Portfolio sync background task
|
||||
|
||||
---
|
||||
|
||||
*Integration audit: 2025-02-23*
|
||||
Loading…
Add table
Add a link
Reference in a new issue