240 lines
8.8 KiB
Markdown
240 lines
8.8 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Project Overview
|
||
|
||
A real estate listing crawler and aggregator that scrapes property listings from Rightmove UK, extracts square meter data from floorplan images using OCR, calculates transit routes, and provides a web UI for browsing listings.
|
||
|
||
## Development Environment
|
||
|
||
**All project commands run inside Docker containers.** Start the dev environment with Docker Compose, then exec into containers:
|
||
|
||
- **Start dev environment**: `docker compose up -d` (locally)
|
||
- **Building/pushing images**: `docker build` / `docker push` (locally)
|
||
- **Deploying to K8s**: `kubectl` (locally, context: `kubernetes-admin@kubernetes`)
|
||
- **Running tests**: `docker compose exec app pytest tests/ -v`
|
||
- **CLI operations**: `docker compose exec app python main.py ...`
|
||
- **Migrations**: `docker compose exec app alembic upgrade head`
|
||
- **Type check**: `docker compose exec app mypy .`
|
||
- **Linting**: `docker compose exec app ruff check .`
|
||
|
||
Always ensure containers are running (`docker compose up -d`) before executing commands.
|
||
|
||
See `.claude/skills/` for detailed skills on dev environment, building, and deploying.
|
||
|
||
## Commands
|
||
|
||
### Setup and Run (Docker - Recommended)
|
||
|
||
```bash
|
||
# Start all services (Redis, MySQL, API, Celery) with Docker
|
||
./start.sh
|
||
|
||
# Rebuild images and start
|
||
./start.sh --build
|
||
|
||
# Stop all containers
|
||
./start.sh --down
|
||
|
||
# View logs
|
||
./start.sh --logs
|
||
```
|
||
|
||
### Setup and Run (Local with Poetry)
|
||
|
||
```bash
|
||
# Install dependencies
|
||
poetry install && cp .env.sample .env
|
||
|
||
# Start backend locally (requires Redis running)
|
||
./start.sh --local
|
||
|
||
# Start frontend (from frontend/ directory)
|
||
cd frontend && ./start.sh
|
||
```
|
||
|
||
### CLI Operations
|
||
|
||
The main CLI (`main.py`) uses Click with a `--data-dir` option (default: `data/rs/`):
|
||
|
||
```bash
|
||
# Dump listings from Rightmove API
|
||
python main.py dump-listings --type rent --min-price 2000 --max-price 4000 --min-bedrooms 2
|
||
|
||
# Download floorplan images
|
||
python main.py dump-images
|
||
|
||
# Extract square meters from floorplans using OCR
|
||
python main.py detect-floorplan
|
||
|
||
# Calculate transit routes (consumes Google Maps API calls)
|
||
python main.py routing --destination-address 'Address' -m transit -l 10
|
||
|
||
# Export to GeoJSON for visualization
|
||
python main.py export-immoweb -O output.js --type rent [filter options]
|
||
```
|
||
|
||
### Testing
|
||
|
||
```bash
|
||
# Run tests with coverage
|
||
pytest tests/ -v --cov=. --cov-report=term-missing
|
||
|
||
# Run type checker
|
||
mypy .
|
||
```
|
||
|
||
### Database Migrations
|
||
|
||
```bash
|
||
alembic upgrade head # Apply migrations
|
||
alembic revision -m "description" # Create new migration
|
||
```
|
||
|
||
### Code Formatting
|
||
|
||
```bash
|
||
yapf --style .style.yapf --recursive .
|
||
```
|
||
|
||
## Architecture
|
||
|
||
### Core Data Flow
|
||
|
||
1. **Scraping** (`rec/query.py`): Fetches listing IDs and details from Rightmove's Android API
|
||
2. **Processing** (`listing_processor.py`): Pipeline with steps for fetching details, downloading images, and OCR detection
|
||
3. **Storage**: SQLModel/SQLAlchemy with MySQL or SQLite, plus JSON files in `data/rs/<listing_id>/`
|
||
4. **API** (`api/app.py`): FastAPI endpoints authenticated via JWT from external Authentik service
|
||
5. **Background Tasks** (`tasks/listing_tasks.py`): Celery tasks for async listing processing with Redis broker
|
||
|
||
### Key Models
|
||
|
||
- `models/listing.py`: SQLModel entities (`RentListing`, `BuyListing`) with `QueryParameters` for filtering
|
||
- `data_access.py`: **DEPRECATED** - Legacy `Listing` dataclass for filesystem-based data access. Use `models.listing.RentListing` or `models.listing.BuyListing` instead.
|
||
|
||
### Services Layer (Unified CLI and API)
|
||
|
||
**IMPORTANT**: The `services/` directory contains unified handler functions that both the CLI and HTTP API use. This ensures consistency and code reuse.
|
||
|
||
#### High-level services (use these in CLI and API):
|
||
- **`listing_service.py`**: Listing operations
|
||
- `get_listings()` - Retrieve listings from database
|
||
- `refresh_listings()` - Fetch new listings from Rightmove (sync or async)
|
||
- `download_images()` - Download floorplan images
|
||
- `detect_floorplans()` - Run OCR on floorplans
|
||
- `calculate_routes()` - Calculate transit routes
|
||
|
||
- **`export_service.py`**: Export operations
|
||
- `export_to_csv()` - Export listings to CSV file
|
||
- `export_to_geojson()` - Export listings to GeoJSON (file or in-memory)
|
||
|
||
- **`district_service.py`**: District management
|
||
- `get_all_districts()` - Get district name → region ID mapping
|
||
- `get_district_names()` - Get list of district names
|
||
- `validate_districts()` - Validate district names
|
||
|
||
- **`task_service.py`**: Background task management
|
||
- `get_task_status()` - Get Celery task status
|
||
- `get_user_tasks()` - Get all tasks for a user
|
||
- `add_task_for_user()` - Associate task with user
|
||
|
||
#### Low-level services (internal implementation):
|
||
- `listing_fetcher.py`: Fetches listing data from Rightmove API
|
||
- `image_fetcher.py`: Downloads floorplan images
|
||
- `floorplan_detector.py`: OCR-based square meter detection
|
||
- `route_calculator.py`: Calculates transit routes using Google Maps API
|
||
- `query_splitter.py`: Intelligent query splitting to maximize data extraction
|
||
|
||
### Query Splitting System
|
||
|
||
Rightmove's API caps search results at ~1,500 listings per query. The query splitting system works around this limitation to fetch **all matching listings**.
|
||
|
||
#### How it works:
|
||
|
||
1. **Initial Split**: Queries are split by district and bedroom count
|
||
2. **Probe**: Each subquery is probed (minimal API request) to get `totalAvailableResults`
|
||
3. **Adaptive Split**: If results exceed threshold (1,200), the price range is binary-split
|
||
4. **Recursive Refinement**: Splitting continues until all subqueries are under threshold
|
||
5. **Full Fetch**: Each subquery fetches up to 60 pages (1,500 results max)
|
||
|
||
```
|
||
Original: 2BR, £1000-£5000 → 3,000 results (over cap!)
|
||
↓ split by price
|
||
£1000-£3000: 1,800 (still over!) | £3000-£5000: 1,200 ✓
|
||
↓ split again
|
||
£1000-£2000: 900 ✓ | £2000-£3000: 900 ✓
|
||
|
||
Final: 3 subqueries → 900 + 900 + 1,200 = 3,000 total results ✓
|
||
```
|
||
|
||
#### Key components:
|
||
- `config/scraper_config.py`: Configuration with env var loading
|
||
- `services/query_splitter.py`: `QuerySplitter` class with `SubQuery` dataclass
|
||
- `rec/query.py`: `probe_query()` for result count probing, `create_session()` for connection pooling
|
||
|
||
### Processing Pipeline
|
||
|
||
`ListingProcessor` runs sequential steps defined in `listing_processor.py`:
|
||
1. `FetchListingDetailsStep` - Get property details from API
|
||
2. `FetchImagesStep` - Download floorplan images
|
||
3. `DetectFloorplanStep` - OCR to extract square meters from floorplans
|
||
|
||
### Floorplan OCR
|
||
|
||
`rec/floorplan.py` uses pytesseract with image preprocessing (adaptive thresholding) to extract square meter values from floorplan images.
|
||
|
||
### Repository Pattern
|
||
|
||
`repositories/listing_repository.py` handles database operations with SQLModel sessions.
|
||
|
||
## Environment Variables
|
||
|
||
- `DB_CONNECTION_STRING`: Database URL (SQLite default: `sqlite:///data/wrongmove.db`)
|
||
- `CELERY_BROKER_URL` / `CELERY_RESULT_BACKEND`: Redis URLs
|
||
- `ROUTING_API_KEY`: Google Maps API key for transit routing
|
||
|
||
### Scraper Configuration
|
||
|
||
These control the query splitting behavior (see `.env.sample` for defaults):
|
||
|
||
| Variable | Default | Description |
|
||
|----------|---------|-------------|
|
||
| `RIGHTMOVE_MAX_CONCURRENT` | 5 | Max concurrent HTTP requests |
|
||
| `RIGHTMOVE_REQUEST_DELAY_MS` | 100 | Delay between requests (ms) |
|
||
| `RIGHTMOVE_SPLIT_THRESHOLD` | 1200 | Split query when results exceed this |
|
||
| `RIGHTMOVE_MIN_PRICE_BAND` | 100 | Minimum price band width (won't split below) |
|
||
| `RIGHTMOVE_MAX_PAGES` | 60 | Max pages per subquery (60 × 25 = 1500) |
|
||
| `RIGHTMOVE_PROXY_URL` | - | SOCKS proxy URL (e.g., `socks5://localhost:9050` for Tor) |
|
||
|
||
## Project Structure
|
||
|
||
- `main.py`: CLI entry point
|
||
- `api/`: FastAPI application with auth middleware
|
||
- `config/`: Configuration modules (scraper settings, scheduled tasks)
|
||
- `models/`: SQLModel database entities
|
||
- `repositories/`: Database access layer
|
||
- `rec/`: Core business logic (query, floorplan OCR, routing, districts)
|
||
- `services/`: Service layer modules (listing_fetcher, image_fetcher, floorplan_detector, route_calculator, query_splitter)
|
||
- `tasks/`: Celery background tasks
|
||
- `frontend/`: React/Vite frontend with Caddy proxy
|
||
- `alembic/`: Database migrations
|
||
- `tests/`: Test suite (unit and integration tests)
|
||
|
||
## Type Checking
|
||
|
||
The project uses strict mypy configuration with `disallow_untyped_defs=true`. Run `mypy .` to check types.
|
||
|
||
## Exploration Preferences
|
||
|
||
- Always ignore `node_modules` directory when exploring the codebase
|
||
|
||
## Git Workflow
|
||
|
||
**IMPORTANT**: After completing work items, always create separate commits for each logical change:
|
||
- Keep each commit focused on one feature/fix
|
||
- Do not include unrelated files
|
||
- Use descriptive commit messages
|
||
- Group related files together (e.g., tests with the code they test)
|
||
- **After each meaningful change, ask the user if they want to commit and push the changes**
|
||
|