150342bb9e
- Extract helpers to reduce function sizes (listing_tasks, app.py, query.py, listing_fetcher) - Replace nonlocal mutations with _PipelineState dataclass in listing_tasks - Fix bugs: isinstance→equality check in repository, verify_exp for OIDC tokens - Consolidate duplicate filter methods in listing_repository - Move hardcoded config to env vars with backward-compatible defaults - Simplify CLI decorator to auto-build QueryParameters - Add deprecation docstring to data_access.py - Test count: 158 → 387 (all passing)
144 lines
4.3 KiB
Markdown
144 lines
4.3 KiB
Markdown
---
|
|
name: dev-environment
|
|
description: |
|
|
Start, stop, rebuild, and manage the local Docker Compose development environment
|
|
for the realestate-crawler project. Use when: (1) user wants to start/stop the dev
|
|
environment, (2) needs to rebuild after code changes, (3) wants to check service
|
|
status or view logs, (4) needs to run database migrations.
|
|
author: Claude Code
|
|
version: 1.0.0
|
|
date: 2026-02-06
|
|
---
|
|
|
|
# Dev Environment Management
|
|
|
|
Docker Compose orchestrates the dev environment locally from `crawler/`. All project
|
|
commands (pytest, alembic, mypy, python, etc.) must run inside the `app` container
|
|
via `docker compose exec app <command>`. Only docker/kubectl commands run on the host.
|
|
|
|
## Starting the Dev Environment
|
|
|
|
```bash
|
|
# Start all services (Redis, MySQL, API, Celery worker, Celery beat)
|
|
cd crawler && docker compose up
|
|
|
|
# Start in detached mode (background)
|
|
cd crawler && docker compose up -d
|
|
|
|
# Rebuild images and start (after Dockerfile or dependency changes)
|
|
cd crawler && docker compose up --build
|
|
|
|
# Or use the start.sh helper
|
|
cd crawler && ./start.sh # foreground
|
|
cd crawler && ./start.sh --build # rebuild first
|
|
```
|
|
|
|
## Stopping the Dev Environment
|
|
|
|
```bash
|
|
cd crawler && docker compose down
|
|
|
|
# Also remove volumes (fresh database, fresh Redis)
|
|
cd crawler && docker compose down -v
|
|
|
|
# Or use the helper
|
|
cd crawler && ./start.sh --down
|
|
```
|
|
|
|
## Checking Status
|
|
|
|
```bash
|
|
# List running containers
|
|
cd crawler && docker compose ps
|
|
|
|
# Check health status
|
|
cd crawler && docker compose ps --format "table {{.Name}}\t{{.Status}}"
|
|
```
|
|
|
|
## Viewing Logs
|
|
|
|
```bash
|
|
# Follow all service logs
|
|
cd crawler && docker compose logs -f
|
|
|
|
# Follow specific service logs
|
|
cd crawler && docker compose logs -f app
|
|
cd crawler && docker compose logs -f celery
|
|
cd crawler && docker compose logs -f celery-beat
|
|
cd crawler && docker compose logs -f mysql
|
|
cd crawler && docker compose logs -f redis
|
|
|
|
# Or use the helper
|
|
cd crawler && ./start.sh --logs
|
|
```
|
|
|
|
## Restarting Individual Services
|
|
|
|
```bash
|
|
# Restart just the API (e.g., after config change)
|
|
cd crawler && docker compose restart app
|
|
|
|
# Restart Celery worker
|
|
cd crawler && docker compose restart celery
|
|
|
|
# Rebuild and restart a single service
|
|
cd crawler && docker compose up --build app
|
|
```
|
|
|
|
## Running Database Migrations
|
|
|
|
```bash
|
|
# Apply pending migrations
|
|
cd crawler && docker compose exec app alembic upgrade head
|
|
|
|
# Create a new migration
|
|
cd crawler && docker compose exec app alembic revision -m "description"
|
|
```
|
|
|
|
## Running Tests Inside Container
|
|
|
|
```bash
|
|
cd crawler && docker compose exec app pytest tests/ -v --cov=. --cov-report=term-missing
|
|
```
|
|
|
|
## Running Any Command Inside Container
|
|
|
|
All project commands must be run inside the `app` container:
|
|
|
|
```bash
|
|
# General pattern
|
|
cd crawler && docker compose exec app <command>
|
|
|
|
# Examples
|
|
cd crawler && docker compose exec app python main.py dump-listings --type rent
|
|
cd crawler && docker compose exec app mypy .
|
|
cd crawler && docker compose exec app ruff check .
|
|
cd crawler && docker compose exec app poetry install
|
|
cd crawler && docker compose exec app bash # interactive shell
|
|
```
|
|
|
|
## Services and Ports
|
|
|
|
| Service | Container | Port | Description |
|
|
|-------------|-----------------|-------|--------------------------------|
|
|
| redis | rec-redis | 6379 | Celery broker + GeoJSON cache |
|
|
| mysql | rec-mysql | 3306 | Primary database |
|
|
| app | rec-app | 5001 | FastAPI server (hot-reload) |
|
|
| celery | rec-celery | - | Background task worker |
|
|
| celery-beat | rec-celery-beat | - | Periodic task scheduler |
|
|
|
|
## Environment Variables
|
|
|
|
Key env vars are set in `docker-compose.yml`. To override locally, create a `.env` file
|
|
in `crawler/` (see `.env.sample`). Key overrides:
|
|
|
|
- `ROUTING_API_KEY` - Google Maps API key (passed from host env)
|
|
- `SCRAPE_SCHEDULES` - JSON array of periodic scrape configs (passed from host env)
|
|
|
|
## Notes
|
|
|
|
- The API server has hot-reload enabled for `api/`, `services/`, `repositories/`, and `models/` directories
|
|
- Source code is bind-mounted into containers, so local edits are reflected immediately
|
|
- Python virtualenv is stored in a named Docker volume (`app_venv`) shared across app, celery, and celery-beat
|
|
- MySQL data persists in the `mysql_data` volume; Redis data in `redis_data`
|
|
- Use `docker compose down -v` to reset all data (volumes)
|