Refactor codebase following Clean Code principles and add 229 tests
- Extract helpers to reduce function sizes (listing_tasks, app.py, query.py, listing_fetcher) - Replace nonlocal mutations with _PipelineState dataclass in listing_tasks - Fix bugs: isinstance→equality check in repository, verify_exp for OIDC tokens - Consolidate duplicate filter methods in listing_repository - Move hardcoded config to env vars with backward-compatible defaults - Simplify CLI decorator to auto-build QueryParameters - Add deprecation docstring to data_access.py - Test count: 158 → 387 (all passing)
This commit is contained in:
parent
7e05b3c971
commit
150342bb9e
48 changed files with 5029 additions and 990 deletions
144
.claude/skills/dev-environment.md
Normal file
144
.claude/skills/dev-environment.md
Normal file
|
|
@ -0,0 +1,144 @@
|
|||
---
|
||||
name: dev-environment
|
||||
description: |
|
||||
Start, stop, rebuild, and manage the local Docker Compose development environment
|
||||
for the realestate-crawler project. Use when: (1) user wants to start/stop the dev
|
||||
environment, (2) needs to rebuild after code changes, (3) wants to check service
|
||||
status or view logs, (4) needs to run database migrations.
|
||||
author: Claude Code
|
||||
version: 1.0.0
|
||||
date: 2026-02-06
|
||||
---
|
||||
|
||||
# Dev Environment Management
|
||||
|
||||
Docker Compose orchestrates the dev environment locally from `crawler/`. All project
|
||||
commands (pytest, alembic, mypy, python, etc.) must run inside the `app` container
|
||||
via `docker compose exec app <command>`. Only docker/kubectl commands run on the host.
|
||||
|
||||
## Starting the Dev Environment
|
||||
|
||||
```bash
|
||||
# Start all services (Redis, MySQL, API, Celery worker, Celery beat)
|
||||
cd crawler && docker compose up
|
||||
|
||||
# Start in detached mode (background)
|
||||
cd crawler && docker compose up -d
|
||||
|
||||
# Rebuild images and start (after Dockerfile or dependency changes)
|
||||
cd crawler && docker compose up --build
|
||||
|
||||
# Or use the start.sh helper
|
||||
cd crawler && ./start.sh # foreground
|
||||
cd crawler && ./start.sh --build # rebuild first
|
||||
```
|
||||
|
||||
## Stopping the Dev Environment
|
||||
|
||||
```bash
|
||||
cd crawler && docker compose down
|
||||
|
||||
# Also remove volumes (fresh database, fresh Redis)
|
||||
cd crawler && docker compose down -v
|
||||
|
||||
# Or use the helper
|
||||
cd crawler && ./start.sh --down
|
||||
```
|
||||
|
||||
## Checking Status
|
||||
|
||||
```bash
|
||||
# List running containers
|
||||
cd crawler && docker compose ps
|
||||
|
||||
# Check health status
|
||||
cd crawler && docker compose ps --format "table {{.Name}}\t{{.Status}}"
|
||||
```
|
||||
|
||||
## Viewing Logs
|
||||
|
||||
```bash
|
||||
# Follow all service logs
|
||||
cd crawler && docker compose logs -f
|
||||
|
||||
# Follow specific service logs
|
||||
cd crawler && docker compose logs -f app
|
||||
cd crawler && docker compose logs -f celery
|
||||
cd crawler && docker compose logs -f celery-beat
|
||||
cd crawler && docker compose logs -f mysql
|
||||
cd crawler && docker compose logs -f redis
|
||||
|
||||
# Or use the helper
|
||||
cd crawler && ./start.sh --logs
|
||||
```
|
||||
|
||||
## Restarting Individual Services
|
||||
|
||||
```bash
|
||||
# Restart just the API (e.g., after config change)
|
||||
cd crawler && docker compose restart app
|
||||
|
||||
# Restart Celery worker
|
||||
cd crawler && docker compose restart celery
|
||||
|
||||
# Rebuild and restart a single service
|
||||
cd crawler && docker compose up --build app
|
||||
```
|
||||
|
||||
## Running Database Migrations
|
||||
|
||||
```bash
|
||||
# Apply pending migrations
|
||||
cd crawler && docker compose exec app alembic upgrade head
|
||||
|
||||
# Create a new migration
|
||||
cd crawler && docker compose exec app alembic revision -m "description"
|
||||
```
|
||||
|
||||
## Running Tests Inside Container
|
||||
|
||||
```bash
|
||||
cd crawler && docker compose exec app pytest tests/ -v --cov=. --cov-report=term-missing
|
||||
```
|
||||
|
||||
## Running Any Command Inside Container
|
||||
|
||||
All project commands must be run inside the `app` container:
|
||||
|
||||
```bash
|
||||
# General pattern
|
||||
cd crawler && docker compose exec app <command>
|
||||
|
||||
# Examples
|
||||
cd crawler && docker compose exec app python main.py dump-listings --type rent
|
||||
cd crawler && docker compose exec app mypy .
|
||||
cd crawler && docker compose exec app ruff check .
|
||||
cd crawler && docker compose exec app poetry install
|
||||
cd crawler && docker compose exec app bash # interactive shell
|
||||
```
|
||||
|
||||
## Services and Ports
|
||||
|
||||
| Service | Container | Port | Description |
|
||||
|-------------|-----------------|-------|--------------------------------|
|
||||
| redis | rec-redis | 6379 | Celery broker + GeoJSON cache |
|
||||
| mysql | rec-mysql | 3306 | Primary database |
|
||||
| app | rec-app | 5001 | FastAPI server (hot-reload) |
|
||||
| celery | rec-celery | - | Background task worker |
|
||||
| celery-beat | rec-celery-beat | - | Periodic task scheduler |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Key env vars are set in `docker-compose.yml`. To override locally, create a `.env` file
|
||||
in `crawler/` (see `.env.sample`). Key overrides:
|
||||
|
||||
- `ROUTING_API_KEY` - Google Maps API key (passed from host env)
|
||||
- `SCRAPE_SCHEDULES` - JSON array of periodic scrape configs (passed from host env)
|
||||
|
||||
## Notes
|
||||
|
||||
- The API server has hot-reload enabled for `api/`, `services/`, `repositories/`, and `models/` directories
|
||||
- Source code is bind-mounted into containers, so local edits are reflected immediately
|
||||
- Python virtualenv is stored in a named Docker volume (`app_venv`) shared across app, celery, and celery-beat
|
||||
- MySQL data persists in the `mysql_data` volume; Redis data in `redis_data`
|
||||
- Use `docker compose down -v` to reset all data (volumes)
|
||||
Loading…
Add table
Add a link
Reference in a new issue