309 lines
17 KiB
Markdown
309 lines
17 KiB
Markdown
# Codebase Structure
|
|
|
|
**Analysis Date:** 2025-02-23
|
|
|
|
## Directory Layout
|
|
|
|
```
|
|
trading-bot/
|
|
├── alembic/ # Database migration versions and configuration
|
|
│ ├── env.py # Alembic runtime environment
|
|
│ ├── script.py.mako # Migration template
|
|
│ └── versions/ # Individual migration files
|
|
├── backtester/ # Historical replay engine
|
|
│ ├── __init__.py
|
|
│ ├── config.py # Backtester configuration
|
|
│ ├── data_loader.py # Load bars from database
|
|
│ ├── engine.py # Main replay loop with SimulatedBroker
|
|
│ ├── metrics.py # Equity curve, Sharpe, drawdown calculations
|
|
│ └── simulated_broker.py # Paper trading for backtests
|
|
├── dashboard/ # React/TypeScript frontend
|
|
│ ├── public/ # Static assets (favicon, manifest)
|
|
│ ├── src/
|
|
│ │ ├── api/ # API client (auth.ts, client.ts)
|
|
│ │ ├── assets/ # Images, fonts
|
|
│ │ ├── components/ # Reusable UI (PositionsTable, EquityCurve, etc.)
|
|
│ │ ├── hooks/ # Custom React hooks (useAuth, useWebSocket, usePortfolio)
|
|
│ │ ├── pages/ # Page components (Login, Portfolio, TradeLog, etc.)
|
|
│ │ ├── App.tsx # Router and layout
|
|
│ │ ├── main.tsx # Entry point
|
|
│ │ └── index.css # Tailwind styles
|
|
│ ├── index.html # HTML template
|
|
│ ├── package.json # npm dependencies (Vite, React, TypeScript)
|
|
│ └── vite.config.ts # Vite build config
|
|
├── docker/ # Dockerfiles and compose configs
|
|
│ ├── Dockerfile.service # Python service base image
|
|
│ ├── Dockerfile.dashboard # Node.js build + Nginx serve
|
|
│ └── nginx.conf # Nginx reverse proxy config
|
|
├── docs/ # Documentation and planning
|
|
│ └── plans/ # Implementation plans from GSD phases
|
|
├── scripts/ # Utility scripts
|
|
│ ├── seed_strategies.py # Initialize default strategies in DB
|
|
│ ├── smoke_test.sh # Health check / integration test script
|
|
│ └── __init__.py
|
|
├── services/ # Microservices (Python packages with underscores)
|
|
│ ├── api_gateway/ # FastAPI HTTP/WebSocket server
|
|
│ │ ├── main.py # App factory, lifespan, router registration
|
|
│ │ ├── config.py # ApiGatewayConfig (CORS, JWT secret, etc.)
|
|
│ │ ├── ws.py # WebSocket endpoint
|
|
│ │ ├── auth/
|
|
│ │ │ ├── routes.py # Register/login/logout endpoints
|
|
│ │ │ ├── jwt.py # JWT token encoding/decoding
|
|
│ │ │ └── middleware.py # get_current_user dependency
|
|
│ │ ├── routes/ # REST endpoints organized by domain
|
|
│ │ │ ├── news.py # GET /api/news
|
|
│ │ │ ├── signals.py # GET /api/signals
|
|
│ │ │ ├── trades.py # GET /api/trades
|
|
│ │ │ ├── portfolio.py # GET /api/portfolio (live account)
|
|
│ │ │ ├── strategies.py # GET/PUT /api/strategies
|
|
│ │ │ ├── controls.py # POST /api/controls/* (pause/resume/etc)
|
|
│ │ │ └── backtest.py # POST /api/backtest
|
|
│ │ └── tasks/
|
|
│ │ └── portfolio_sync.py # Background task: sync account data
|
|
│ ├── news_fetcher/ # Polls RSS + Reddit
|
|
│ │ ├── main.py # Entry point: fetch + deduplicate + publish
|
|
│ │ ├── config.py # NewsFetcherConfig (feed URLs, poll intervals)
|
|
│ │ └── sources/
|
|
│ │ ├── rss.py # RSSSource.fetch() via feedparser
|
|
│ │ └── reddit.py # RedditSource.fetch() via asyncpraw
|
|
│ ├── sentiment_analyzer/ # Score sentiment + extract tickers
|
|
│ │ ├── main.py # Entry point: consume raw → score → publish
|
|
│ │ ├── config.py # SentimentAnalyzerConfig (thresholds, model paths)
|
|
│ │ ├── ticker_extractor.py # Regex-based company name → ticker
|
|
│ │ └── analyzers/
|
|
│ │ ├── finbert.py # FinBERTAnalyzer (transformers model)
|
|
│ │ └── ollama_analyzer.py # OllamaAnalyzer (fallback LLM)
|
|
│ ├── signal_generator/ # Combine sentiment + market → signals
|
|
│ │ ├── main.py # Entry point: consume scored articles + bars → ensemble
|
|
│ │ ├── config.py # SignalGeneratorConfig (ensemble threshold)
|
|
│ │ ├── ensemble.py # WeightedEnsemble orchestration
|
|
│ │ └── market_data.py # MarketDataManager: SMA/RSI calculation per ticker
|
|
│ ├── trade_executor/ # Risk checks + order submission
|
|
│ │ ├── main.py # Entry point: consume signals → risk check → submit
|
|
│ │ ├── config.py # TradeExecutorConfig (risk limits, position size formula)
|
|
│ │ └── risk_manager.py # RiskManager: position limits, market hours, cooldowns
|
|
│ ├── learning_engine/ # Evaluate trades + adjust weights
|
|
│ │ ├── main.py # Entry point: consume executions → evaluate → adjust weights
|
|
│ │ ├── config.py # LearningEngineConfig (adjustment parameters)
|
|
│ │ ├── evaluator.py # TradeEvaluator: match opens to closes, calc P&L
|
|
│ │ └── weight_adjuster.py # WeightAdjuster: multi-armed bandit logic
|
|
│ ├── market_data/ # Fetch + stream OHLCV bars
|
|
│ │ ├── main.py # Entry point: fetch from Alpaca → publish bars
|
|
│ │ ├── config.py # MarketDataConfig (watchlist, timeframe, limits)
|
|
│ │ └── __main__.py # Allows `python -m services.market_data`
|
|
│ └── __init__.py
|
|
├── shared/ # Shared Python libraries (imported by all services)
|
|
│ ├── __init__.py
|
|
│ ├── config.py # BaseConfig: common settings (DB, Redis, logging)
|
|
│ ├── db.py # create_db(): async engine + sessionmaker
|
|
│ ├── redis_streams.py # StreamPublisher, StreamConsumer
|
|
│ ├── telemetry.py # setup_telemetry(): OpenTelemetry + Prometheus
|
|
│ ├── broker/ # Brokerage abstraction layer
|
|
│ │ ├── base.py # BaseBroker ABC (abstract interface)
|
|
│ │ └── alpaca_broker.py # AlpacaBroker: alpaca-py implementation
|
|
│ ├── models/ # SQLAlchemy ORM models
|
|
│ │ ├── __init__.py # Imports all models so Alembic discovers them
|
|
│ │ ├── base.py # Base (declarative), TimestampMixin
|
|
│ │ ├── auth.py # User, UserCredential
|
|
│ │ ├── news.py # Article, ArticleSentiment
|
|
│ │ ├── trading.py # Signal, Trade, Position, Strategy, StrategyWeightHistory
|
|
│ │ ├── learning.py # TradeOutcome, LearningAdjustment
|
|
│ │ └── timeseries.py # MarketData, PortfolioSnapshot, StrategyMetric
|
|
│ ├── schemas/ # Pydantic v2 message schemas
|
|
│ │ ├── __init__.py # Imports all schemas
|
|
│ │ ├── base.py # BaseSchema, TimestampSchema
|
|
│ │ ├── news.py # RawArticle, ScoredArticle
|
|
│ │ ├── trading.py # TradeSignal, TradeExecution, OrderRequest, etc.
|
|
│ │ └── learning.py # TradeOutcomeSchema, WeightAdjustment
|
|
│ └── strategies/ # Trading strategy implementations
|
|
│ ├── __init__.py # Exports BaseStrategy, concrete implementations
|
|
│ ├── base.py # BaseStrategy ABC
|
|
│ ├── momentum.py # MomentumStrategy
|
|
│ ├── mean_reversion.py # MeanReversionStrategy
|
|
│ └── news_driven.py # NewsDrivenStrategy
|
|
├── tests/ # Test suites
|
|
│ ├── __init__.py
|
|
│ ├── conftest.py # Pytest fixtures (fake Redis, DB, configs)
|
|
│ ├── test_*.py # Unit tests: models, schemas, broker, strategies, backtester
|
|
│ ├── services/ # Service-specific tests
|
|
│ │ ├── test_news_fetcher.py
|
|
│ │ ├── test_sentiment_analyzer.py
|
|
│ │ ├── test_signal_generator.py
|
|
│ │ ├── test_trade_executor.py
|
|
│ │ ├── test_learning_engine.py
|
|
│ │ ├── test_api_auth.py
|
|
│ │ ├── test_api_routes.py
|
|
│ │ ├── test_market_data.py
|
|
│ │ └── test_portfolio_sync.py
|
|
│ └── integration/ # Integration tests (require docker)
|
|
│ ├── test_news_pipeline.py
|
|
│ └── test_trading_flow.py
|
|
├── .claude/ # Project-specific Claude knowledge
|
|
│ └── CLAUDE.md # Architecture notes, patterns, known issues
|
|
├── .env.example # Example environment variables
|
|
├── alembic.ini # Alembic configuration
|
|
├── docker-compose.yml # Service orchestration (8 services + infra)
|
|
├── pyproject.toml # Python package config, dependencies, test config
|
|
└── README.md # Project overview
|
|
```
|
|
|
|
## Directory Purposes
|
|
|
|
**`alembic/`:**
|
|
- Purpose: Database schema migrations using Alembic (SQLAlchemy migration tool)
|
|
- Contains: Migration scripts (timestamped Python files in `versions/`)
|
|
- Workflow: `alembic revision -m "message"` creates migration, `alembic upgrade head` applies to DB
|
|
|
|
**`backtester/`:**
|
|
- Purpose: Historical replay engine for strategy testing
|
|
- Contains: Engine loop, simulated broker, metrics calculation
|
|
- Used by: API endpoint `/api/backtest` (triggered by dashboard)
|
|
- Workflow: Load historical bars from DB, replay trades with `SimulatedBroker`, output equity curve + metrics
|
|
|
|
**`dashboard/`:**
|
|
- Purpose: React TypeScript frontend for user interaction
|
|
- Contains: Pages (Login, Portfolio, TradeLog, News, etc.), components (charts, tables), API client, hooks
|
|
- Entry: `src/main.tsx` → `src/App.tsx`
|
|
- Build: `npm run build` outputs to `dist/`, served by Nginx in Docker
|
|
|
|
**`docker/`:**
|
|
- Purpose: Container images and deployment config
|
|
- Contains: `Dockerfile.service` (Python services base), `Dockerfile.dashboard` (Node build + Nginx)
|
|
- Used by: docker-compose.yml to build and run all services
|
|
|
|
**`scripts/`:**
|
|
- Purpose: Utility and setup scripts
|
|
- Key files:
|
|
- `seed_strategies.py`: Inserts default strategies into DB (Momentum, MeanReversion, NewsDriven)
|
|
- `smoke_test.sh`: Checks all service health endpoints
|
|
|
|
**`services/*/main.py`:**
|
|
- Purpose: Entry point for each microservice
|
|
- Pattern: Each service has own `main.py` that can be run as `python -m services.SERVICE_NAME`
|
|
- Lifecycle: Setup config → connect to Redis/DB → start polling/consuming → handle shutdown gracefully
|
|
|
|
**`services/*/config.py`:**
|
|
- Purpose: Service-specific configuration
|
|
- Pattern: Extends `BaseConfig`, adds service-specific settings
|
|
- Example: `NewsFetcherConfig` adds `rss_urls`, `reddit_subreddits`, `rss_poll_interval_seconds`
|
|
|
|
**`shared/models/`:**
|
|
- Purpose: SQLAlchemy ORM models (database schema)
|
|
- Key tables: `trades`, `signals`, `articles`, `article_sentiments`, `market_data`, `positions`, `users`, `strategies`, `strategy_weight_history`, `learning_adjustments`, `portfolio_snapshots`
|
|
- Pattern: Each model imports in `__init__.py` so Alembic can auto-discover for migrations
|
|
|
|
**`shared/schemas/`:**
|
|
- Purpose: Pydantic v2 message schemas for validation
|
|
- Used by: Services to validate messages on consume/publish via Redis Streams
|
|
- Naming: Schemas mirror the Redis Stream message types (RawArticle, ScoredArticle, TradeSignal, TradeExecution, etc.)
|
|
|
|
**`shared/strategies/`:**
|
|
- Purpose: Pluggable trading strategy implementations
|
|
- Pattern: All extend `BaseStrategy`, implement `async evaluate(ticker, market, sentiment) → TradeSignal | None`
|
|
- Used by: Signal generator via `WeightedEnsemble`
|
|
|
|
**`tests/`:**
|
|
- Purpose: Unit and integration tests
|
|
- Pattern: `test_*.py` files match modules in `shared/` and `services/`
|
|
- Marker: `@pytest.mark.integration` for tests requiring live services (redis, postgres)
|
|
- Run: `pytest tests/ -v -m "not integration"` for unit, or with docker running for integration
|
|
|
|
## Key File Locations
|
|
|
|
**Entry Points:**
|
|
- `services/news_fetcher/main.py`: Polls RSS/Reddit, publishes to `news:raw`
|
|
- `services/sentiment_analyzer/main.py`: Consumes `news:raw`, publishes to `news:scored`
|
|
- `services/signal_generator/main.py`: Consumes `news:scored` + `market:bars`, publishes to `signals:generated`
|
|
- `services/trade_executor/main.py`: Consumes `signals:generated`, submits orders, publishes to `trades:executed`
|
|
- `services/learning_engine/main.py`: Consumes `trades:executed`, adjusts strategy weights
|
|
- `services/market_data/main.py`: Fetches OHLCV bars, publishes to `market:bars`
|
|
- `services/api_gateway/main.py`: HTTP server listening on port 8000 (configurable)
|
|
- `dashboard/src/main.tsx`: React app entry, connects to API gateway
|
|
|
|
**Configuration:**
|
|
- `.env`: Runtime environment variables (database URL, Redis URL, API keys, etc.)
|
|
- `pyproject.toml`: Python project metadata, dependencies (split into optional groups per service)
|
|
- `docker-compose.yml`: Service orchestration, port mappings, environment variables
|
|
- `alembic.ini`: Database migration tool configuration
|
|
|
|
**Core Logic:**
|
|
- `services/signal_generator/ensemble.py`: Weighted ensemble combining strategies
|
|
- `services/trade_executor/risk_manager.py`: Risk checks before order submission
|
|
- `services/learning_engine/weight_adjuster.py`: Multi-armed bandit weight adjustment
|
|
- `backtester/engine.py`: Historical replay loop
|
|
- `shared/broker/alpaca_broker.py`: Alpaca order submission and account queries
|
|
|
|
**Testing:**
|
|
- `tests/conftest.py`: Pytest fixtures for mock Redis, DB, service configs
|
|
- `tests/test_models.py`: ORM model tests
|
|
- `tests/test_schemas.py`: Pydantic schema validation tests
|
|
- `tests/services/`: Service-specific unit tests
|
|
- `tests/integration/`: End-to-end pipeline tests (require docker)
|
|
|
|
## Naming Conventions
|
|
|
|
**Files:**
|
|
- Python service directories use underscores: `news_fetcher`, `api_gateway`, `signal_generator` (matches Docker Compose service names with hyphens replaced)
|
|
- Test files: `test_*.py` or `*_test.py` (pytest convention)
|
|
- Database migration files: `{timestamp}_{description}.py` (Alembic auto-generated)
|
|
- React components: PascalCase `.tsx` files (Login.tsx, Portfolio.tsx, PositionsTable.tsx)
|
|
- React hooks: `use*.ts` prefix (useAuth.ts, useWebSocket.ts, usePortfolio.ts)
|
|
- Config classes: `*Config` suffix extending `BaseConfig` (NewsFetcherConfig, ApiGatewayConfig)
|
|
|
|
**Directories:**
|
|
- Services under `services/`: lowercase with underscores (news_fetcher, api_gateway)
|
|
- Shared libraries under `shared/`: logical grouping (broker/, models/, schemas/, strategies/)
|
|
- Tests mirror source structure: `tests/services/test_*.py` for `services/*/` modules
|
|
- React pages: match route names (Portfolio.tsx for /portfolio, TradeLog.tsx for /trades)
|
|
|
|
## Where to Add New Code
|
|
|
|
**New Feature (End-to-End):**
|
|
- Service-specific code: `services/{SERVICE_NAME}/{module}.py`
|
|
- Shared models: `shared/models/{domain}.py` (if new entity type)
|
|
- Shared schemas: `shared/schemas/{domain}.py` (if new message type)
|
|
- Tests: `tests/services/test_{SERVICE_NAME}.py` and/or `tests/integration/`
|
|
|
|
**New Component/Module:**
|
|
- Service: Create `services/{service_name}/` directory with `__init__.py`, `main.py`, `config.py`
|
|
- Strategy: Extend `BaseStrategy` in `shared/strategies/{strategy_name}.py`
|
|
- Broker adapter: Extend `BaseBroker` in `shared/broker/{provider_name}_broker.py`
|
|
- API route: Add to `services/api_gateway/routes/{domain}.py`
|
|
- Dashboard page: Add `dashboard/src/pages/{PageName}.tsx` and route in `App.tsx`
|
|
|
|
**Utilities:**
|
|
- Shared helpers: `shared/{module}.py` (e.g., `shared/redis_streams.py`)
|
|
- Service-specific helpers: `services/{SERVICE_NAME}/{helper}.py`
|
|
- API utilities: `services/api_gateway/{utility}.py`
|
|
|
|
**Database:**
|
|
- New table: Add model in `shared/models/{domain}.py`
|
|
- Schema migration: Run `alembic revision -m "description"`, fill in migration file
|
|
- Persist data: Service calls `session.add(model_instance)` then `session.commit()`
|
|
|
|
**Tests:**
|
|
- Unit test: `tests/test_{module}.py` for shared, `tests/services/test_{service}.py` for services
|
|
- Integration test: `tests/integration/test_{flow}.py`
|
|
- Fixtures: Add to `tests/conftest.py` (database, Redis, mocked API responses)
|
|
|
|
## Special Directories
|
|
|
|
**`.planning/codebase/`:**
|
|
- Purpose: Generated codebase mapping documents (ARCHITECTURE.md, STRUCTURE.md, etc.)
|
|
- Generated by: `/gsd:map-codebase` command
|
|
- Committed: Yes, part of repo for CI/CD reference
|
|
|
|
**`.pytest_cache/`, `__pycache__/`:**
|
|
- Purpose: Pytest and Python runtime caches
|
|
- Generated: Yes (auto-created by pytest/Python)
|
|
- Committed: No (in .gitignore)
|
|
|
|
**`trading_bot.egg-info/`:**
|
|
- Purpose: Build metadata from `setuptools.build_meta`
|
|
- Generated: Yes (auto-created during install)
|
|
- Committed: No (in .gitignore)
|
|
|
|
**`dashboard/dist/`:**
|
|
- Purpose: Built React app (HTML, JS, CSS bundles)
|
|
- Generated: Yes (by `npm run build`)
|
|
- Committed: No (in .gitignore)
|