trading/.planning/codebase/STRUCTURE.md

Codebase Structure

Analysis Date: 2025-02-23

Directory Layout

trading-bot/
├── alembic/              # Database migration versions and configuration
│   ├── env.py            # Alembic runtime environment
│   ├── script.py.mako    # Migration template
│   └── versions/         # Individual migration files
├── backtester/           # Historical replay engine
│   ├── __init__.py
│   ├── config.py         # Backtester configuration
│   ├── data_loader.py    # Load bars from database
│   ├── engine.py         # Main replay loop with SimulatedBroker
│   ├── metrics.py        # Equity curve, Sharpe, drawdown calculations
│   └── simulated_broker.py  # Paper trading for backtests
├── dashboard/            # React/TypeScript frontend
│   ├── public/           # Static assets (favicon, manifest)
│   ├── src/
│   │   ├── api/          # API client (auth.ts, client.ts)
│   │   ├── assets/       # Images, fonts
│   │   ├── components/   # Reusable UI (PositionsTable, EquityCurve, etc.)
│   │   ├── hooks/        # Custom React hooks (useAuth, useWebSocket, usePortfolio)
│   │   ├── pages/        # Page components (Login, Portfolio, TradeLog, etc.)
│   │   ├── App.tsx       # Router and layout
│   │   ├── main.tsx      # Entry point
│   │   └── index.css     # Tailwind styles
│   ├── index.html        # HTML template
│   ├── package.json      # npm dependencies (Vite, React, TypeScript)
│   └── vite.config.ts    # Vite build config
├── docker/               # Dockerfiles and compose configs
│   ├── Dockerfile.service # Python service base image
│   ├── Dockerfile.dashboard # Node.js build + Nginx serve
│   └── nginx.conf        # Nginx reverse proxy config
├── docs/                 # Documentation and planning
│   └── plans/            # Implementation plans from GSD phases
├── scripts/              # Utility scripts
│   ├── seed_strategies.py  # Initialize default strategies in DB
│   ├── smoke_test.sh     # Health check / integration test script
│   └── __init__.py
├── services/             # Microservices (Python packages with underscores)
│   ├── api_gateway/      # FastAPI HTTP/WebSocket server
│   │   ├── main.py       # App factory, lifespan, router registration
│   │   ├── config.py     # ApiGatewayConfig (CORS, JWT secret, etc.)
│   │   ├── ws.py         # WebSocket endpoint
│   │   ├── auth/
│   │   │   ├── routes.py # Register/login/logout endpoints
│   │   │   ├── jwt.py    # JWT token encoding/decoding
│   │   │   └── middleware.py  # get_current_user dependency
│   │   ├── routes/       # REST endpoints organized by domain
│   │   │   ├── news.py   # GET /api/news
│   │   │   ├── signals.py  # GET /api/signals
│   │   │   ├── trades.py # GET /api/trades
│   │   │   ├── portfolio.py # GET /api/portfolio (live account)
│   │   │   ├── strategies.py # GET/PUT /api/strategies
│   │   │   ├── controls.py  # POST /api/controls/* (pause/resume/etc)
│   │   │   └── backtest.py  # POST /api/backtest
│   │   └── tasks/
│   │       └── portfolio_sync.py # Background task: sync account data
│   ├── news_fetcher/    # Polls RSS + Reddit
│   │   ├── main.py       # Entry point: fetch + deduplicate + publish
│   │   ├── config.py     # NewsFetcherConfig (feed URLs, poll intervals)
│   │   └── sources/
│   │       ├── rss.py    # RSSSource.fetch() via feedparser
│   │       └── reddit.py # RedditSource.fetch() via asyncpraw
│   ├── sentiment_analyzer/ # Score sentiment + extract tickers
│   │   ├── main.py       # Entry point: consume raw → score → publish
│   │   ├── config.py     # SentimentAnalyzerConfig (thresholds, model paths)
│   │   ├── ticker_extractor.py # Regex-based company name → ticker
│   │   └── analyzers/
│   │       ├── finbert.py # FinBERTAnalyzer (transformers model)
│   │       └── ollama_analyzer.py # OllamaAnalyzer (fallback LLM)
│   ├── signal_generator/  # Combine sentiment + market → signals
│   │   ├── main.py       # Entry point: consume scored articles + bars → ensemble
│   │   ├── config.py     # SignalGeneratorConfig (ensemble threshold)
│   │   ├── ensemble.py   # WeightedEnsemble orchestration
│   │   └── market_data.py # MarketDataManager: SMA/RSI calculation per ticker
│   ├── trade_executor/   # Risk checks + order submission
│   │   ├── main.py       # Entry point: consume signals → risk check → submit
│   │   ├── config.py     # TradeExecutorConfig (risk limits, position size formula)
│   │   └── risk_manager.py # RiskManager: position limits, market hours, cooldowns
│   ├── learning_engine/  # Evaluate trades + adjust weights
│   │   ├── main.py       # Entry point: consume executions → evaluate → adjust weights
│   │   ├── config.py     # LearningEngineConfig (adjustment parameters)
│   │   ├── evaluator.py  # TradeEvaluator: match opens to closes, calc P&L
│   │   └── weight_adjuster.py # WeightAdjuster: multi-armed bandit logic
│   ├── market_data/      # Fetch + stream OHLCV bars
│   │   ├── main.py       # Entry point: fetch from Alpaca → publish bars
│   │   ├── config.py     # MarketDataConfig (watchlist, timeframe, limits)
│   │   └── __main__.py   # Allows `python -m services.market_data`
│   └── __init__.py
├── shared/               # Shared Python libraries (imported by all services)
│   ├── __init__.py
│   ├── config.py         # BaseConfig: common settings (DB, Redis, logging)
│   ├── db.py             # create_db(): async engine + sessionmaker
│   ├── redis_streams.py  # StreamPublisher, StreamConsumer
│   ├── telemetry.py      # setup_telemetry(): OpenTelemetry + Prometheus
│   ├── broker/           # Brokerage abstraction layer
│   │   ├── base.py       # BaseBroker ABC (abstract interface)
│   │   └── alpaca_broker.py # AlpacaBroker: alpaca-py implementation
│   ├── models/           # SQLAlchemy ORM models
│   │   ├── __init__.py   # Imports all models so Alembic discovers them
│   │   ├── base.py       # Base (declarative), TimestampMixin
│   │   ├── auth.py       # User, UserCredential
│   │   ├── news.py       # Article, ArticleSentiment
│   │   ├── trading.py    # Signal, Trade, Position, Strategy, StrategyWeightHistory
│   │   ├── learning.py   # TradeOutcome, LearningAdjustment
│   │   └── timeseries.py # MarketData, PortfolioSnapshot, StrategyMetric
│   ├── schemas/          # Pydantic v2 message schemas
│   │   ├── __init__.py   # Imports all schemas
│   │   ├── base.py       # BaseSchema, TimestampSchema
│   │   ├── news.py       # RawArticle, ScoredArticle
│   │   ├── trading.py    # TradeSignal, TradeExecution, OrderRequest, etc.
│   │   └── learning.py   # TradeOutcomeSchema, WeightAdjustment
│   └── strategies/       # Trading strategy implementations
│       ├── __init__.py   # Exports BaseStrategy, concrete implementations
│       ├── base.py       # BaseStrategy ABC
│       ├── momentum.py    # MomentumStrategy
│       ├── mean_reversion.py # MeanReversionStrategy
│       └── news_driven.py # NewsDrivenStrategy
├── tests/                # Test suites
│   ├── __init__.py
│   ├── conftest.py       # Pytest fixtures (fake Redis, DB, configs)
│   ├── test_*.py         # Unit tests: models, schemas, broker, strategies, backtester
│   ├── services/         # Service-specific tests
│   │   ├── test_news_fetcher.py
│   │   ├── test_sentiment_analyzer.py
│   │   ├── test_signal_generator.py
│   │   ├── test_trade_executor.py
│   │   ├── test_learning_engine.py
│   │   ├── test_api_auth.py
│   │   ├── test_api_routes.py
│   │   ├── test_market_data.py
│   │   └── test_portfolio_sync.py
│   └── integration/      # Integration tests (require docker)
│       ├── test_news_pipeline.py
│       └── test_trading_flow.py
├── .claude/              # Project-specific Claude knowledge
│   └── CLAUDE.md         # Architecture notes, patterns, known issues
├── .env.example          # Example environment variables
├── alembic.ini           # Alembic configuration
├── docker-compose.yml    # Service orchestration (8 services + infra)
├── pyproject.toml        # Python package config, dependencies, test config
└── README.md             # Project overview

Directory Purposes

alembic/:

• Purpose: Database schema migrations using Alembic (SQLAlchemy migration tool)
• Contains: Migration scripts (timestamped Python files in versions/)
• Workflow: alembic revision -m "message" creates migration, alembic upgrade head applies to DB

backtester/:

• Purpose: Historical replay engine for strategy testing
• Contains: Engine loop, simulated broker, metrics calculation
• Used by: API endpoint /api/backtest (triggered by dashboard)
• Workflow: Load historical bars from DB, replay trades with SimulatedBroker, output equity curve + metrics

dashboard/:

• Purpose: React TypeScript frontend for user interaction
• Contains: Pages (Login, Portfolio, TradeLog, News, etc.), components (charts, tables), API client, hooks
• Entry: src/main.tsx → src/App.tsx
• Build: npm run build outputs to dist/, served by Nginx in Docker

docker/:

• Purpose: Container images and deployment config
• Contains: Dockerfile.service (Python services base), Dockerfile.dashboard (Node build + Nginx)
• Used by: docker-compose.yml to build and run all services

scripts/:

• Purpose: Utility and setup scripts
• Key files:
  • seed_strategies.py: Inserts default strategies into DB (Momentum, MeanReversion, NewsDriven)
  • smoke_test.sh: Checks all service health endpoints

services/*/main.py:

• Purpose: Entry point for each microservice
• Pattern: Each service has own main.py that can be run as python -m services.SERVICE_NAME
• Lifecycle: Setup config → connect to Redis/DB → start polling/consuming → handle shutdown gracefully

services/*/config.py:

• Purpose: Service-specific configuration
• Pattern: Extends BaseConfig, adds service-specific settings
• Example: NewsFetcherConfig adds rss_urls, reddit_subreddits, rss_poll_interval_seconds

shared/models/:

• Purpose: SQLAlchemy ORM models (database schema)
• Key tables: trades, signals, articles, article_sentiments, market_data, positions, users, strategies, strategy_weight_history, learning_adjustments, portfolio_snapshots
• Pattern: Each model imports in __init__.py so Alembic can auto-discover for migrations

shared/schemas/:

• Purpose: Pydantic v2 message schemas for validation
• Used by: Services to validate messages on consume/publish via Redis Streams
• Naming: Schemas mirror the Redis Stream message types (RawArticle, ScoredArticle, TradeSignal, TradeExecution, etc.)

shared/strategies/:

• Purpose: Pluggable trading strategy implementations
• Pattern: All extend BaseStrategy, implement async evaluate(ticker, market, sentiment) → TradeSignal | None
• Used by: Signal generator via WeightedEnsemble

tests/:

• Purpose: Unit and integration tests
• Pattern: test_*.py files match modules in shared/ and services/
• Marker: @pytest.mark.integration for tests requiring live services (redis, postgres)
• Run: pytest tests/ -v -m "not integration" for unit, or with docker running for integration

Key File Locations

Entry Points:

• services/news_fetcher/main.py: Polls RSS/Reddit, publishes to news:raw
• services/sentiment_analyzer/main.py: Consumes news:raw, publishes to news:scored
• services/signal_generator/main.py: Consumes news:scored + market:bars, publishes to signals:generated
• services/trade_executor/main.py: Consumes signals:generated, submits orders, publishes to trades:executed
• services/learning_engine/main.py: Consumes trades:executed, adjusts strategy weights
• services/market_data/main.py: Fetches OHLCV bars, publishes to market:bars
• services/api_gateway/main.py: HTTP server listening on port 8000 (configurable)
• dashboard/src/main.tsx: React app entry, connects to API gateway

Configuration:

• .env: Runtime environment variables (database URL, Redis URL, API keys, etc.)
• pyproject.toml: Python project metadata, dependencies (split into optional groups per service)
• docker-compose.yml: Service orchestration, port mappings, environment variables
• alembic.ini: Database migration tool configuration

Core Logic:

• services/signal_generator/ensemble.py: Weighted ensemble combining strategies
• services/trade_executor/risk_manager.py: Risk checks before order submission
• services/learning_engine/weight_adjuster.py: Multi-armed bandit weight adjustment
• backtester/engine.py: Historical replay loop
• shared/broker/alpaca_broker.py: Alpaca order submission and account queries

Testing:

• tests/conftest.py: Pytest fixtures for mock Redis, DB, service configs
• tests/test_models.py: ORM model tests
• tests/test_schemas.py: Pydantic schema validation tests
• tests/services/: Service-specific unit tests
• tests/integration/: End-to-end pipeline tests (require docker)

Naming Conventions

Files:

• Python service directories use underscores: news_fetcher, api_gateway, signal_generator (matches Docker Compose service names with hyphens replaced)
• Test files: test_*.py or *_test.py (pytest convention)
• Database migration files: {timestamp}_{description}.py (Alembic auto-generated)
• React components: PascalCase .tsx files (Login.tsx, Portfolio.tsx, PositionsTable.tsx)
• React hooks: use*.ts prefix (useAuth.ts, useWebSocket.ts, usePortfolio.ts)
• Config classes: *Config suffix extending BaseConfig (NewsFetcherConfig, ApiGatewayConfig)

Directories:

• Services under services/: lowercase with underscores (news_fetcher, api_gateway)
• Shared libraries under shared/: logical grouping (broker/, models/, schemas/, strategies/)
• Tests mirror source structure: tests/services/test_*.py for services/*/ modules
• React pages: match route names (Portfolio.tsx for /portfolio, TradeLog.tsx for /trades)

Where to Add New Code

New Feature (End-to-End):

• Service-specific code: services/{SERVICE_NAME}/{module}.py
• Shared models: shared/models/{domain}.py (if new entity type)
• Shared schemas: shared/schemas/{domain}.py (if new message type)
• Tests: tests/services/test_{SERVICE_NAME}.py and/or tests/integration/

New Component/Module:

• Service: Create services/{service_name}/ directory with __init__.py, main.py, config.py
• Strategy: Extend BaseStrategy in shared/strategies/{strategy_name}.py
• Broker adapter: Extend BaseBroker in shared/broker/{provider_name}_broker.py
• API route: Add to services/api_gateway/routes/{domain}.py
• Dashboard page: Add dashboard/src/pages/{PageName}.tsx and route in App.tsx

Utilities:

• Shared helpers: shared/{module}.py (e.g., shared/redis_streams.py)
• Service-specific helpers: services/{SERVICE_NAME}/{helper}.py
• API utilities: services/api_gateway/{utility}.py

Database:

• New table: Add model in shared/models/{domain}.py
• Schema migration: Run alembic revision -m "description", fill in migration file
• Persist data: Service calls session.add(model_instance) then session.commit()

Tests:

• Unit test: tests/test_{module}.py for shared, tests/services/test_{service}.py for services
• Integration test: tests/integration/test_{flow}.py
• Fixtures: Add to tests/conftest.py (database, Redis, mocked API responses)

Special Directories

.planning/codebase/:

• Purpose: Generated codebase mapping documents (ARCHITECTURE.md, STRUCTURE.md, etc.)
• Generated by: /gsd:map-codebase command
• Committed: Yes, part of repo for CI/CD reference

.pytest_cache/, __pycache__/:

• Purpose: Pytest and Python runtime caches
• Generated: Yes (auto-created by pytest/Python)
• Committed: No (in .gitignore)

trading_bot.egg-info/:

• Purpose: Build metadata from setuptools.build_meta
• Generated: Yes (auto-created during install)
• Committed: No (in .gitignore)

dashboard/dist/:

• Purpose: Built React app (HTML, JS, CSS bundles)
• Generated: Yes (by npm run build)
• Committed: No (in .gitignore)