viktor/wrongmove
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/

Browse source 
The crawler subdirectory was the only active project. Moving it to the
repo root simplifies paths and removes the unnecessary nesting. The
vqa/ and immoweb/ directories were legacy/unused and have been removed.

Updated .drone.yml, .gitignore, .claude/ docs, and skills to reflect
the new flat structure.
This commit is contained in:
Viktor Barzin 2026-02-07 23:01:20 +00:00
parent e2247be700
commit eafbc1ac52
No known key found for this signature in database
GPG key ID: 0EB088298288D958
221 changed files with 70 additions and 146140 deletions
Download patch file Download diff file Expand all files Collapse all files

101
crawler/.claude/skills/python-313-redis-generic-type/SKILL.md
View file

@ -1,101 +0,0 @@
---
name: python-313-redis-generic-type
description: |
Fix for "TypeError: <class 'redis.client.Redis'> is not a generic class" when using
redis-py with Python 3.13. Use when: (1) upgrading to Python 3.13 breaks redis type
annotations, (2) mypy passes but runtime fails with generic class error, (3) using
redis.Redis[str] or similar parameterized types. Covers redis-py generic type
compatibility with Python 3.13's stricter runtime generic checking.
author: Claude Code
version: 1.0.0
date: 2026-01-31
---
# Python 3.13 redis.Redis Generic Type Error
## Problem
Python 3.13 introduced stricter runtime checking for generic types. The redis-py library's
`Redis` class is not defined as a generic class at runtime, even though it works with type
checkers like mypy. This causes a `TypeError` when you use parameterized types like
`redis.Redis[str]` in type annotations that are evaluated at runtime.
## Context / Trigger Conditions
- Python 3.13 or later
- Using redis-py library
- Type annotation like `redis_client: redis.Redis[str]`
- Error message: `TypeError: <class 'redis.client.Redis'> is not a generic class`
- Works fine with mypy but fails at runtime
- Often appears when instantiating a class with this annotation
## Solution
### Option 1: Remove the type parameter (Recommended)
```python
# Before (breaks in Python 3.13)
redis_client: redis.Redis[str]
# After (works in all Python versions)
redis_client: redis.Redis # type: ignore[type-arg]
```
The `# type: ignore[type-arg]` comment silences mypy's warning about missing type arguments.
### Option 2: Use string annotation (deferred evaluation)
```python
from __future__ import annotations
redis_client: "redis.Redis[str]" # String annotation, not evaluated at runtime
```
### Option 3: Use TYPE_CHECKING guard
```python
from typing import TYPE_CHECKING
if TYPE_CHECKING:
RedisClient = redis.Redis[str]
else:
RedisClient = redis.Redis
redis_client: RedisClient
```
## Verification
1. Run your application with Python 3.13
2. The TypeError should no longer appear
3. Run mypy to ensure type checking still works (may need type: ignore comment)
## Example
### Before (Broken)
```python
import redis
class RedisRepository:
redis_client: redis.Redis[str] # TypeError at runtime in Python 3.13
def __init__(self):
self.redis_client = redis.Redis(host='localhost', decode_responses=True)
```
### After (Fixed)
```python
import redis
class RedisRepository:
redis_client: redis.Redis # type: ignore[type-arg]
def __init__(self):
self.redis_client = redis.Redis(host='localhost', decode_responses=True)
```
## Notes
- This is a breaking change in Python 3.13's handling of generic types
- The redis-py library may add proper generic support in future versions
- If using `decode_responses=True`, the client returns `str`; otherwise `bytes`
- The `type: ignore` comment is preferable to `Any` as it preserves some type safety
- This issue affects other libraries that aren't properly defined as Generic classes
## References
- [Python 3.13 Release Notes](https://docs.python.org/3.13/whatsnew/3.13.html)
- [redis-py GitHub Issues](https://github.com/redis/redis-py/issues)
- [PEP 585 - Type Hinting Generics In Standard Collections](https://peps.python.org/pep-0585/)

132
crawler/.claude/skills/python-parentheses-comparison-bug/SKILL.md
View file

@ -1,132 +0,0 @@
---
name: python-parentheses-comparison-bug
description: |
Debug Python comparison bug where parentheses around a variable cause unexpected behavior.
Use when: (1) condition always evaluates to False/True unexpectedly, (2) code like
"if (mylist) == 0" never triggers, (3) length check seems to not work, (4) comparison
with list/dict returns unexpected results. Common mistake where parentheses cause the
variable itself to be compared instead of its length.
author: Claude Code
version: 1.0.0
date: 2026-01-31
---
# Python Parentheses Comparison Bug
## Problem
A subtle Python bug where unnecessary parentheses around a variable in a comparison
cause the wrong value to be compared. The expression `(mylist) == 0` compares the list
itself to 0, not its length. Since a list is never equal to an integer, this always
returns False.
## Context / Trigger Conditions
- Condition that should sometimes be True is always False (or vice versa)
- Code pattern like `if (existing_items) == 0:` or `if (result) == expected:`
- The parentheses don't cause a syntax error but change semantics
- Often appears when copying/adapting code or during refactoring
- May pass code review because it "looks" correct
## Solution
### Identify the Bug Pattern
```python
# BUG: Compares list to 0, always False
if (existing_listings) == 0:
return True
# Also wrong: compares list to integer
if (items) == 5:
do_something()
```
### Fix: Use len() for Length Comparisons
```python
# CORRECT: Compares length to 0
if len(existing_listings) == 0:
return True
# Alternative: Use truthiness for empty check
if not existing_listings:
return True
# CORRECT: Compares length to integer
if len(items) == 5:
do_something()
```
## Verification
1. Add a debug print before the condition: `print(f"list={existing_listings}, len={len(existing_listings)}")`
2. Verify the condition now evaluates correctly
3. Write a unit test that exercises both branches of the condition
## Example
### Before (Broken)
```python
class FetchListingDetailsStep:
async def needs_processing(self, listing_id: int) -> bool:
existing_listings = await self.listing_repository.get_listings(
only_ids=[listing_id]
)
# BUG: This compares the list object to 0, which is always False
# The parentheses around existing_listings are misleading
if (existing_listings) == 0:
return True
return False
```
### After (Fixed)
```python
class FetchListingDetailsStep:
async def needs_processing(self, listing_id: int) -> bool:
existing_listings = await self.listing_repository.get_listings(
only_ids=[listing_id]
)
# CORRECT: Check if list is empty using len()
if len(existing_listings) == 0:
return True
return False
```
### Even Better (Pythonic)
```python
class FetchListingDetailsStep:
async def needs_processing(self, listing_id: int) -> bool:
existing_listings = await self.listing_repository.get_listings(
only_ids=[listing_id]
)
# Most Pythonic: Use truthiness
return not existing_listings
```
## Notes
- Python's truthiness: empty collections are falsy, non-empty are truthy
- This bug is particularly insidious because:
- It's syntactically valid
- It doesn't raise an exception
- The parentheses make it look intentional
- Code review may miss it
- Linters like pylint or flake8 won't catch this specific pattern
- Type checkers like mypy may warn about comparing incompatible types
- When debugging, add print statements to verify actual vs expected values
## Prevention
- Prefer `if not mylist:` over `if len(mylist) == 0:`
- Prefer `if mylist:` over `if len(mylist) > 0:`
- Remove unnecessary parentheses around single variables
- Enable mypy's strict mode which may catch type comparison issues
- Write unit tests that exercise both branches of conditions
## Related Patterns
```python
# These are all wrong (comparing object to number):
if (mydict) == 0: # Always False
if (mylist) > 0: # TypeError in Python 3
if (mystring) == 0: # Always False
# These are correct:
if len(mydict) == 0: # True if empty
if not mydict: # True if empty (preferred)
if len(mylist) > 0: # True if non-empty
if mylist: # True if non-empty (preferred)
```

46
crawler/.dockerignore
View file

@ -1,46 +0,0 @@
# Data directories
data/
*.db
*.sqlite
# Frontend (built separately)
frontend/
# Dependencies
.venv/
__pycache__/
*.pyc
*.pyo
# IDE
.idea/
.vscode/
*.swp
# Git
.git/
.gitignore
# Logs
*.log
recalculating.log
# Jupyter
*.ipynb
.ipynb_checkpoints/
# Tests
.pytest_cache/
.coverage
htmlcov/
# Claude
.claude/
# macOS
.DS_Store
._*
# Environment (keep .env.sample)
.env
.env.local

43
crawler/.env.sample
View file

@ -1,43 +0,0 @@
# Copy me to .env and source me
export ROUTING_API_KEY="<CHANGE ME>" # fetch from https://console.cloud.google.com/google/maps-apis/; prices - https://developers.google.com/maps/billing-and-pricing/pricing
# export DB_CONNECTION_STRING="mysql://wrongmove:wrongmove@localhost:3306/wrongmove" # example for mysql
export DB_CONNECTION_STRING="sqlite:///data/wrongmove.db" # by default use SQLite locally
export CELERY_BROKER_URL="redis://localhost:6379/0" # processing background tasks
export CELERY_RESULT_BACKEND="redis://localhost:6379/1"
# Rightmove scraper configuration
# These settings control query splitting to work around Rightmove's ~1500 result cap
RIGHTMOVE_MAX_CONCURRENT=5 # Max concurrent HTTP requests
RIGHTMOVE_REQUEST_DELAY_MS=100 # Delay between requests in milliseconds
RIGHTMOVE_SPLIT_THRESHOLD=1200 # Split query when results exceed this threshold
RIGHTMOVE_MIN_PRICE_BAND=100 # Minimum price band width (won't split below this)
RIGHTMOVE_MAX_PAGES=60 # Max pages per subquery (60 * 25 = 1500 max results)
RIGHTMOVE_PROXY_URL= # Optional SOCKS proxy URL (e.g., socks5://localhost:9050 for Tor)
# Throttling detection and circuit breaker
RIGHTMOVE_SLOW_RESPONSE_THRESHOLD=10.0 # Response time threshold in seconds
RIGHTMOVE_ENABLE_CIRCUIT_BREAKER=true # Enable circuit breaker protection
RIGHTMOVE_CIRCUIT_BREAKER_FAILURES=5 # Consecutive failures to open circuit
RIGHTMOVE_CIRCUIT_BREAKER_TIMEOUT=60.0 # Seconds to wait before recovery attempt
# Periodic scraping schedules (JSON array)
# Each schedule has: name, enabled, hour, minute, day_of_week, listing_type, min/max_bedrooms, min/max_price, district_names, furnish_types
# Cron fields: minute (0-59), hour (0-23), day_of_week (0-6, 0=Sunday)
# Example:
# SCRAPE_SCHEDULES='[{"name":"Daily RENT","listing_type":"RENT","hour":"2","min_bedrooms":2,"max_bedrooms":3,"min_price":2000,"max_price":4000}]'
# Multiple schedules:
# SCRAPE_SCHEDULES='[{"name":"RENT 2am","listing_type":"RENT","hour":"2"},{"name":"BUY 4am","listing_type":"BUY","hour":"4"}]'
SCRAPE_SCHEDULES=
# WebAuthn / Passkey configuration
WEBAUTHN_RP_ID=localhost # Relying Party ID (domain)
WEBAUTHN_RP_NAME=Wrongmove # Relying Party display name
WEBAUTHN_ORIGIN=https://localhost # Expected WebAuthn origin
# JWT configuration (for passkey-issued tokens)
JWT_SECRET=change-me-in-production # HMAC secret for HS256 signing
JWT_ALGORITHM=HS256 # JWT signing algorithm
JWT_EXPIRATION_HOURS=24 # Token expiry in hours
JWT_ISSUER=wrongmove # JWT issuer claim

21
crawler/.style.yapf
View file

@ -1,21 +0,0 @@
[style]
based_on_style = facebook
# Formatting options
column_limit = 88
indent_width = 4
use_tabs = false
continuation_indent_width = 4
# Spacing
spaces_around_power_operator = true
spaces_before_comment = 2
# Splitting rules
split_before_logical_operator = true
split_before_bitwise_operator = true
split_before_arithmetic_operator = true
split_before_expression_after_opening_paren = true
split_before_first_argument = false
split_complex_comprehension = true
allow_split_before_dict_value = true

240
crawler/CLAUDE.md
View file

@ -1,240 +0,0 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
A real estate listing crawler and aggregator that scrapes property listings from Rightmove UK, extracts square meter data from floorplan images using OCR, calculates transit routes, and provides a web UI for browsing listings.
## Development Environment
**All project commands run inside Docker containers.** Start the dev environment with Docker Compose, then exec into containers:
- **Start dev environment**: `docker compose up -d` (locally)
- **Building/pushing images**: `docker build` / `docker push` (locally)
- **Deploying to K8s**: `kubectl` (locally, context: `kubernetes-admin@kubernetes`)
- **Running tests**: `docker compose exec app pytest tests/ -v`
- **CLI operations**: `docker compose exec app python main.py ...`
- **Migrations**: `docker compose exec app alembic upgrade head`
- **Type check**: `docker compose exec app mypy .`
- **Linting**: `docker compose exec app ruff check .`
Always ensure containers are running (`docker compose up -d`) before executing commands.
See `.claude/skills/` for detailed skills on dev environment, building, and deploying.
## Commands
### Setup and Run (Docker - Recommended)
```bash
# Start all services (Redis, MySQL, API, Celery) with Docker
./start.sh
# Rebuild images and start
./start.sh --build
# Stop all containers
./start.sh --down
# View logs
./start.sh --logs
```
### Setup and Run (Local with Poetry)
```bash
# Install dependencies
poetry install && cp .env.sample .env
# Start backend locally (requires Redis running)
./start.sh --local
# Start frontend (from frontend/ directory)
cd frontend && ./start.sh
```
### CLI Operations
The main CLI (`main.py`) uses Click with a `--data-dir` option (default: `data/rs/`):
```bash
# Dump listings from Rightmove API
python main.py dump-listings --type rent --min-price 2000 --max-price 4000 --min-bedrooms 2
# Download floorplan images
python main.py dump-images
# Extract square meters from floorplans using OCR
python main.py detect-floorplan
# Calculate transit routes (consumes Google Maps API calls)
python main.py routing --destination-address 'Address' -m transit -l 10
# Export to GeoJSON for visualization
python main.py export-immoweb -O output.js --type rent [filter options]
```
### Testing
```bash
# Run tests with coverage
pytest tests/ -v --cov=. --cov-report=term-missing
# Run type checker
mypy .
```
### Database Migrations
```bash
alembic upgrade head # Apply migrations
alembic revision -m "description" # Create new migration
```
### Code Formatting
```bash
yapf --style .style.yapf --recursive .
```
## Architecture
### Core Data Flow
1. **Scraping** (`rec/query.py`): Fetches listing IDs and details from Rightmove's Android API
2. **Processing** (`listing_processor.py`): Pipeline with steps for fetching details, downloading images, and OCR detection
3. **Storage**: SQLModel/SQLAlchemy with MySQL or SQLite, plus JSON files in `data/rs/<listing_id>/`
4. **API** (`api/app.py`): FastAPI endpoints authenticated via JWT from external Authentik service
5. **Background Tasks** (`tasks/listing_tasks.py`): Celery tasks for async listing processing with Redis broker
### Key Models
- `models/listing.py`: SQLModel entities (`RentListing`, `BuyListing`) with `QueryParameters` for filtering
- `data_access.py`: **DEPRECATED** - Legacy `Listing` dataclass for filesystem-based data access. Use `models.listing.RentListing` or `models.listing.BuyListing` instead.
### Services Layer (Unified CLI and API)
**IMPORTANT**: The `services/` directory contains unified handler functions that both the CLI and HTTP API use. This ensures consistency and code reuse.
#### High-level services (use these in CLI and API):
- **`listing_service.py`**: Listing operations
- `get_listings()` - Retrieve listings from database
- `refresh_listings()` - Fetch new listings from Rightmove (sync or async)
- `download_images()` - Download floorplan images
- `detect_floorplans()` - Run OCR on floorplans
- `calculate_routes()` - Calculate transit routes
- **`export_service.py`**: Export operations
- `export_to_csv()` - Export listings to CSV file
- `export_to_geojson()` - Export listings to GeoJSON (file or in-memory)
- **`district_service.py`**: District management
- `get_all_districts()` - Get district name → region ID mapping
- `get_district_names()` - Get list of district names
- `validate_districts()` - Validate district names
- **`task_service.py`**: Background task management
- `get_task_status()` - Get Celery task status
- `get_user_tasks()` - Get all tasks for a user
- `add_task_for_user()` - Associate task with user
#### Low-level services (internal implementation):
- `listing_fetcher.py`: Fetches listing data from Rightmove API
- `image_fetcher.py`: Downloads floorplan images
- `floorplan_detector.py`: OCR-based square meter detection
- `route_calculator.py`: Calculates transit routes using Google Maps API
- `query_splitter.py`: Intelligent query splitting to maximize data extraction
### Query Splitting System
Rightmove's API caps search results at ~1,500 listings per query. The query splitting system works around this limitation to fetch **all matching listings**.
#### How it works:
1. **Initial Split**: Queries are split by district and bedroom count
2. **Probe**: Each subquery is probed (minimal API request) to get `totalAvailableResults`
3. **Adaptive Split**: If results exceed threshold (1,200), the price range is binary-split
4. **Recursive Refinement**: Splitting continues until all subqueries are under threshold
5. **Full Fetch**: Each subquery fetches up to 60 pages (1,500 results max)
```
Original: 2BR, £1000-£5000 → 3,000 results (over cap!)
↓ split by price
£1000-£3000: 1,800 (still over!) | £3000-£5000: 1,200 ✓
↓ split again
£1000-£2000: 900 ✓ | £2000-£3000: 900 ✓
Final: 3 subqueries → 900 + 900 + 1,200 = 3,000 total results ✓
```
#### Key components:
- `config/scraper_config.py`: Configuration with env var loading
- `services/query_splitter.py`: `QuerySplitter` class with `SubQuery` dataclass
- `rec/query.py`: `probe_query()` for result count probing, `create_session()` for connection pooling
### Processing Pipeline
`ListingProcessor` runs sequential steps defined in `listing_processor.py`:
1. `FetchListingDetailsStep` - Get property details from API
2. `FetchImagesStep` - Download floorplan images
3. `DetectFloorplanStep` - OCR to extract square meters from floorplans
### Floorplan OCR
`rec/floorplan.py` uses pytesseract with image preprocessing (adaptive thresholding) to extract square meter values from floorplan images.
### Repository Pattern
`repositories/listing_repository.py` handles database operations with SQLModel sessions.
## Environment Variables
- `DB_CONNECTION_STRING`: Database URL (SQLite default: `sqlite:///data/wrongmove.db`)
- `CELERY_BROKER_URL` / `CELERY_RESULT_BACKEND`: Redis URLs
- `ROUTING_API_KEY`: Google Maps API key for transit routing
### Scraper Configuration
These control the query splitting behavior (see `.env.sample` for defaults):
| Variable | Default | Description |
|----------|---------|-------------|
| `RIGHTMOVE_MAX_CONCURRENT` | 5 | Max concurrent HTTP requests |
| `RIGHTMOVE_REQUEST_DELAY_MS` | 100 | Delay between requests (ms) |
| `RIGHTMOVE_SPLIT_THRESHOLD` | 1200 | Split query when results exceed this |
| `RIGHTMOVE_MIN_PRICE_BAND` | 100 | Minimum price band width (won't split below) |
| `RIGHTMOVE_MAX_PAGES` | 60 | Max pages per subquery (60 × 25 = 1500) |
| `RIGHTMOVE_PROXY_URL` | - | SOCKS proxy URL (e.g., `socks5://localhost:9050` for Tor) |
## Project Structure
- `main.py`: CLI entry point
- `api/`: FastAPI application with auth middleware
- `config/`: Configuration modules (scraper settings, scheduled tasks)
- `models/`: SQLModel database entities
- `repositories/`: Database access layer
- `rec/`: Core business logic (query, floorplan OCR, routing, districts)
- `services/`: Service layer modules (listing_fetcher, image_fetcher, floorplan_detector, route_calculator, query_splitter)
- `tasks/`: Celery background tasks
- `frontend/`: React/Vite frontend with Caddy proxy
- `alembic/`: Database migrations
- `tests/`: Test suite (unit and integration tests)
## Type Checking
The project uses strict mypy configuration with `disallow_untyped_defs=true`. Run `mypy .` to check types.
## Exploration Preferences
- Always ignore `node_modules` directory when exploring the codebase
## Git Workflow
**IMPORTANT**: After completing work items, always create separate commits for each logical change:
- Keep each commit focused on one feature/fix
- Do not include unrelated files
- Use descriptive commit messages
- Group related files together (e.g., tests with the code they test)
- **After each meaningful change, ask the user if they want to commit and push the changes**

45
crawler/Dockerfile
View file

@ -1,45 +0,0 @@
# Stage 1: Install build tools and Python dependencies
FROM python:3.13-slim AS builder
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
gcc \
python3-dev \
libopencv-dev \
libmariadb-dev \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt ./
# Install dependencies into a venv using pip (no poetry needed)
RUN python -m venv /app/.venv && \
/app/.venv/bin/pip install --no-cache-dir -r requirements.txt
# Stage 2: Runtime system dependencies (runs in parallel with builder)
FROM python:3.13-slim AS runtime-base
RUN apt-get update && apt-get install -y --no-install-recommends \
libgl1 \
libglib2.0-0 \
tesseract-ocr \
tesseract-ocr-eng \
libmariadb3 \
&& rm -rf /var/lib/apt/lists/*
# Stage 3: Final image — combine venv from builder + runtime base
FROM runtime-base
WORKDIR /app
# Copy the venv from the builder stage
COPY --from=builder /app/.venv /app/.venv
ENV PATH="/app/.venv/bin:$PATH"
# Copy the application code
COPY . .
EXPOSE 5001
CMD ["uvicorn", "api.app:app", "--host", "0.0.0.0", "--port", "5001"]

47
crawler/GUIDE
View file

@ -1,47 +0,0 @@
package name: com.rightmove.android
frida --codeshare pcipolloni/universal-android-ssl-pinning-bypass-with-frida -f com.rightmove.android
1. install burp
2. Add listener 8282
3. Export certificate in the DER format
4. convert certificate with command
```
# converts from DER to PEM
openssl x509 -inform DER -in burp.der -out burp.pem
```
5. Copy cert to android with the proper name
```
# According to https://codeshare.frida.re/@pcipolloni/universal-android-ssl-pinning-bypass-with-frida/ the cert path is hardcoded
adb push burp.pem /data/local/tmp/cert-der.crt
```
6. Add the proxy in the android wifi settings
```
# find your own local network ip
ip addr
# Open the wifi you are connected to, edit and add port 8282 (from above) and the ip
```
192.168.0.211/24
1. Install frida server on android
4. run Frida
```
adb shell "/data/local/tmp/frida-server &"
```
5. Check if it runs with
```
frida-ps -U
```
6. pin rightmove
frida -U --codeshare pcipolloni/universal-android-ssl-pinning-bypass-with-frida -f com.rightmove.android

70
crawler/README.md
View file

@ -1,70 +0,0 @@
# Setup
1. Instal deps:
```bash
poetry install && cp .env.sample .env
```
2. Check `.env` if you want to customize settings for broker and db
3. run `./start.sh`
This starts the backend
To start the fronend:
```
cd frontend && cp .env.sample .env
```
Change the `DEV_HOST` to any name you want to use to access the web interface.
Next, setup the DNS record (e.g in your /etc/hosts) file.
This is important as auth is done via external [authentik] service that needs to redirect to a name.
Run `./start.sh`
This starts a Caddy proxy with correct certificates, and npm dev server.
All requests going to the frontend are forwarded to the npm server and the ones for the backed (that go to `/api/*`) are forwarded to the backend service.
Lastly, reachout to Viktor to allowlist your `DEV_HOST` so that authentik can authorize callbacks to your host.
# Formatting
```bash
yapf --style .style.yapf --recursive .
```
For VSCode - install yapf extension.
Enable formatting using yap and the style file in this repo (there may be an easier way; I put this in my user settings json):
```
{
"[python]": {
"editor.formatOnSaveMode": "file",
"editor.formatOnSave": true,
"editor.defaultFormatter": "eeyore.yapf",
"editor.formatOnType": false
},
"yapf.args": ["--style", "/home/wizard/code/realestate-crawler/crawler/.style.yapf"]
}
```
ADB commands (from `/Applications/BlueStacks.app/Contents/MacOS`):
Set proxy
```
./hd-adb shell settings put global http_proxy 192.168.9.110:8080
```
Disable proxy:
```
/hd-adb shell settings put global http_proxy :0
```
Connect adb
```
./hd-adb connect 127.0.0.1:5555
```
Disconnect adb
```
/hd-adb disconnect
```

39
crawler/TASKS.md
View file

@ -1,39 +0,0 @@
## Extra
- [ ] The routing is now expensive. I could simplify it by finding the walking distance to the nearest trainstations with overpass turbo and then have a routing map between stations.
- [ ] Partition query further as each query can listing query can only grab a 1000 entries at most. If the query is too broad, it will fail afterwards.
- District: City of London, totalAvailableResults: 60
- District: Greenwich, totalAvailableResults: 1371
- District: Hillingdon, totalAvailableResults: 1026
- District: Ealing, totalAvailableResults: 1736
- District: Richmond upon Thames, totalAvailableResults: 819
- District: Sutton, totalAvailableResults: 664
- District: Wandsworth, totalAvailableResults: 1824
- District: Camden, totalAvailableResults: 801
- District: Enfield, totalAvailableResults: 1056
- District: Croydon, totalAvailableResults: 1865
- District: Hackney, totalAvailableResults: 840
- District: Kingston upon Thames, totalAvailableResults: 685
- District: Kensington and Chelsea, totalAvailableResults: 658
- District: Bromley, totalAvailableResults: 1341
- District: Brent, totalAvailableResults: 1332
- District: Waltham Forest, totalAvailableResults: 763
- District: Southwark, totalAvailableResults: 1460
- District: Harrow, totalAvailableResults: 948
- District: Lewisham, totalAvailableResults: 1192
- District: Barnet, totalAvailableResults: 1683
- District: Islington, totalAvailableResults: 766
- District: Haringey, totalAvailableResults: 795
- District: Lambeth, totalAvailableResults: 1626
- District: Westminster, totalAvailableResults: 1130
- District: Tower Hamlets, totalAvailableResults: 2213
- District: Havering, totalAvailableResults: 863
- District: Barking and Dagenham, totalAvailableResults: 485
- District: Hammersmith and Fulham, totalAvailableResults: 1038
- District: Bexley, totalAvailableResults: 803
- District: Redbridge, totalAvailableResults: 720
- District: Newham, totalAvailableResults: 1306
- District: Merton, totalAvailableResults: 873
- District: Hounslow, totalAvailableResults: 1096

145
crawler/alembic.ini
View file

@ -1,145 +0,0 @@
# A generic, single database configuration.
[alembic]
# path to migration scripts.
# this is typically a path given in POSIX (e.g. forward slashes)
# format, relative to the token %(here)s which refers to the location of this
# ini file
script_location = %(here)s/alembic
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
# for all available tokens
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory. for multiple paths, the path separator
# is defined by "path_separator" below.
prepend_sys_path = .
# timezone to use when rendering the date within the migration file
# as well as the filename.
# If specified, requires the python>=3.9 or backports.zoneinfo library and tzdata library.
# Any required deps can installed by adding `alembic[tz]` to the pip requirements
# string value is passed to ZoneInfo()
# leave blank for localtime
# timezone =
# max length of characters to apply to the "slug" field
# truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; This defaults
# to <script_location>/versions. When using multiple version
# directories, initial revisions must be specified with --version-path.
# The path separator used here should be the separator specified by "path_separator"
# below.
# version_locations = %(here)s/bar:%(here)s/bat:%(here)s/alembic/versions
# path_separator; This indicates what character is used to split lists of file
# paths, including version_locations and prepend_sys_path within configparser
# files such as alembic.ini.
# The default rendered in new alembic.ini files is "os", which uses os.pathsep
# to provide os-dependent path splitting.
#
# Note that in order to support legacy alembic.ini files, this default does NOT
# take place if path_separator is not present in alembic.ini. If this
# option is omitted entirely, fallback logic is as follows:
#
# 1. Parsing of the version_locations option falls back to using the legacy
# "version_path_separator" key, which if absent then falls back to the legacy
# behavior of splitting on spaces and/or commas.
# 2. Parsing of the prepend_sys_path option falls back to the legacy
# behavior of splitting on spaces, commas, or colons.
#
# Valid values for path_separator are:
#
# path_separator = :
# path_separator = ;
# path_separator = space
# path_separator = newline
#
# Use os.pathsep. Default configuration used for new projects.
path_separator = os
# set to 'true' to search source files recursively
# in each "version_locations" directory
# new in Alembic version 1.10
# recursive_version_locations = false
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
# database URL. This is consumed by the user-maintained env.py script only.
# other means of configuring database URLs may be customized within the env.py
# file.
; sqlalchemy.url = driver://user:pass@localhost/dbname
; sqlalchemy.url = sqlite:///data/wrongmove.db
; sqlalchemy.url = mysql://wrongmove:wrongmove@localhost:3306/wrongmove
sqlalchemy.url = %(DB_CONNECTION_STRING)s
[post_write_hooks]
# post_write_hooks defines scripts or Python functions that are run
# on newly generated revision scripts. See the documentation for further
# detail and examples
# format using "black" - use the console_scripts runner, against the "black" entrypoint
# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME
# lint with attempts to fix using "ruff" - use the exec runner, execute a binary
# hooks = ruff
# ruff.type = exec
# ruff.executable = %(here)s/.venv/bin/ruff
# ruff.options = check --fix REVISION_SCRIPT_FILENAME
# Logging configuration. This is also consumed by the user-maintained
# env.py script only.
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARNING
handlers = console
qualname =
[logger_sqlalchemy]
level = WARNING
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

1
crawler/alembic/README
View file

@ -1 +0,0 @@
Generic single-database configuration.

77
crawler/alembic/env.py
View file

@ -1,77 +0,0 @@
from logging.config import fileConfig
from alembic import context
from database import engine
from sqlmodel import SQLModel
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Interpret the config file for Python logging.
# This line sets up loggers basically.
if config.config_file_name is not None:
fileConfig(config.config_file_name)
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
target_metadata = SQLModel.metadata
# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.
def run_migrations_offline() -> None:
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = config.get_main_option("sqlalchemy.url")
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online() -> None:
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
connectable = engine # Use the SQLModel engine directly
# connectable = engine_from_config(
# config.get_section(config.config_ini_section, {}),
# prefix="sqlalchemy.",
# poolclass=pool.NullPool,
# )
with connectable.connect() as connection:
context.configure(connection=connection, target_metadata=target_metadata)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()

28
crawler/alembic/script.py.mako
View file

@ -1,28 +0,0 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision: str = ${repr(up_revision)}
down_revision: Union[str, None] = ${repr(down_revision)}
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
def upgrade() -> None:
"""Upgrade schema."""
${upgrades if upgrades else "pass"}
def downgrade() -> None:
"""Downgrade schema."""
${downgrades if downgrades else "pass"}

82
crawler/alembic/versions/6363b18a22ca_initial.py
View file

@ -1,82 +0,0 @@
"""initial
Revision ID: 6363b18a22ca
Revises:
Create Date: 2025-06-22 15:27:23.187594
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
import sqlmodel
# revision identifiers, used by Alembic.
revision: str = '6363b18a22ca'
down_revision: Union[str, None] = None
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Upgrade schema."""
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('buylisting',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('price', sa.Float(), nullable=False),
sa.Column('number_of_bedrooms', sa.Integer(), nullable=False),
sa.Column('square_meters', sa.Float(), nullable=True),
sa.Column('agency', sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column('council_tax_band', sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column('longitude', sa.Float(), nullable=False),
sa.Column('latitude', sa.Float(), nullable=False),
sa.Column('price_history_json', sa.TEXT(), nullable=False),
sa.Column('listing_site', sa.Enum('RIGHTMOVE', name='listingsite'), nullable=False),
sa.Column('last_seen', sa.DateTime(), nullable=False),
sa.Column('photo_thumbnail', sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column('floorplan_image_paths', sa.JSON(), nullable=False),
sa.Column('additional_info', sa.JSON(), nullable=False),
sa.Column('routing_info_json', sa.TEXT(), nullable=True),
sa.Column('service_charge', sa.Float(), nullable=True),
sa.Column('lease_left', sa.Integer(), nullable=True),
sa.PrimaryKeyConstraint('id')
)
op.create_table('rentlisting',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('price', sa.Float(), nullable=False),
sa.Column('number_of_bedrooms', sa.Integer(), nullable=False),
sa.Column('square_meters', sa.Float(), nullable=True),
sa.Column('agency', sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column('council_tax_band', sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column('longitude', sa.Float(), nullable=False),
sa.Column('latitude', sa.Float(), nullable=False),
sa.Column('price_history_json', sa.TEXT(), nullable=False),
sa.Column('listing_site', sa.Enum('RIGHTMOVE', name='listingsite'), nullable=False),
sa.Column('last_seen', sa.DateTime(), nullable=False),
sa.Column('photo_thumbnail', sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column('floorplan_image_paths', sa.JSON(), nullable=False),
sa.Column('additional_info', sa.JSON(), nullable=False),
sa.Column('routing_info_json', sa.TEXT(), nullable=True),
sa.Column('available_from', sa.DateTime(), nullable=True),
sa.Column('furnish_type', sa.Enum('FURNISHED', 'UNFURNISHED', 'PART_FURNISHED', 'ASK_LANDLORD', 'UNKNOWN', name='furnishtype'), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_table('user',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('email', sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column('password', sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_user_email'), 'user', ['email'], unique=True)
# ### end Alembic commands ###
def downgrade() -> None:
"""Downgrade schema."""
# ### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_user_email'), table_name='user')
op.drop_table('user')
op.drop_table('rentlisting')
op.drop_table('buylisting')
# ### end Alembic commands ###

45
crawler/alembic/versions/8220f657bae5_add_indices_to_commonly_searched_.py
View file

@ -1,45 +0,0 @@
"""add indices to commonly searched numeric columns
Revision ID: 8220f657bae5
Revises: 6363b18a22ca
Create Date: 2025-06-30 22:54:11.706618
"""
from typing import Sequence, Union
from alembic import op
# revision identifiers, used by Alembic.
revision: str = '8220f657bae5'
down_revision: Union[str, None] = '6363b18a22ca'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Upgrade schema."""
# ### commands auto generated by Alembic - please adjust! ###
op.create_index(op.f('ix_buylisting_last_seen'), 'buylisting', ['last_seen'], unique=False)
op.create_index(op.f('ix_buylisting_number_of_bedrooms'), 'buylisting', ['number_of_bedrooms'], unique=False)
op.create_index(op.f('ix_buylisting_price'), 'buylisting', ['price'], unique=False)
op.create_index(op.f('ix_buylisting_square_meters'), 'buylisting', ['square_meters'], unique=False)
op.create_index(op.f('ix_rentlisting_last_seen'), 'rentlisting', ['last_seen'], unique=False)
op.create_index(op.f('ix_rentlisting_number_of_bedrooms'), 'rentlisting', ['number_of_bedrooms'], unique=False)
op.create_index(op.f('ix_rentlisting_price'), 'rentlisting', ['price'], unique=False)
op.create_index(op.f('ix_rentlisting_square_meters'), 'rentlisting', ['square_meters'], unique=False)
# ### end Alembic commands ###
def downgrade() -> None:
"""Downgrade schema."""
# ### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_rentlisting_square_meters'), table_name='rentlisting')
op.drop_index(op.f('ix_rentlisting_price'), table_name='rentlisting')
op.drop_index(op.f('ix_rentlisting_number_of_bedrooms'), table_name='rentlisting')
op.drop_index(op.f('ix_rentlisting_last_seen'), table_name='rentlisting')
op.drop_index(op.f('ix_buylisting_square_meters'), table_name='buylisting')
op.drop_index(op.f('ix_buylisting_price'), table_name='buylisting')
op.drop_index(op.f('ix_buylisting_number_of_bedrooms'), table_name='buylisting')
op.drop_index(op.f('ix_buylisting_last_seen'), table_name='buylisting')
# ### end Alembic commands ###

56
crawler/alembic/versions/a1b2c3d4e5f6_add_streaming_indexes.py
View file

@ -1,56 +0,0 @@
"""add streaming indexes for query optimization
Revision ID: a1b2c3d4e5f6
Revises: e5f1bc4e3323
Create Date: 2026-02-01 12:00:00.000000
"""
from typing import Sequence, Union
from alembic import op
# revision identifiers, used by Alembic.
revision: str = 'a1b2c3d4e5f6'
down_revision: Union[str, None] = 'e5f1bc4e3323'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add composite and single-column indexes for streaming query optimization."""
# Composite index for main query pattern (bedrooms, price, last_seen filtering)
op.create_index(
'ix_rentlisting_query_composite',
'rentlisting',
['number_of_bedrooms', 'price', 'last_seen'],
unique=False
)
op.create_index(
'ix_buylisting_query_composite',
'buylisting',
['number_of_bedrooms', 'price', 'last_seen'],
unique=False
)
# Missing single-column indexes for frequently filtered columns
op.create_index(
'ix_rentlisting_furnish_type',
'rentlisting',
['furnish_type'],
unique=False
)
op.create_index(
'ix_rentlisting_available_from',
'rentlisting',
['available_from'],
unique=False
)
def downgrade() -> None:
"""Remove streaming indexes."""
op.drop_index('ix_rentlisting_available_from', table_name='rentlisting')
op.drop_index('ix_rentlisting_furnish_type', table_name='rentlisting')
op.drop_index('ix_buylisting_query_composite', table_name='buylisting')
op.drop_index('ix_rentlisting_query_composite', table_name='rentlisting')

66
crawler/alembic/versions/b4c7d8e9f0a1_add_passkey_auth.py
View file

@ -1,66 +0,0 @@
"""add passkey auth
Revision ID: b4c7d8e9f0a1
Revises: a1b2c3d4e5f6
Create Date: 2025-07-15 10:00:00.000000
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
import sqlmodel
# revision identifiers, used by Alembic.
revision: str = 'b4c7d8e9f0a1'
down_revision: Union[str, None] = 'a1b2c3d4e5f6'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Upgrade schema."""
# Make user.password nullable
op.alter_column('user', 'password',
existing_type=sqlmodel.sql.sqltypes.AutoString(),
nullable=True)
# Add created_at to user table
op.add_column('user',
sa.Column('created_at', sa.DateTime(),
nullable=True,
server_default=sa.func.now()))
# Create passkeycredential table
op.create_table('passkeycredential',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('credential_id', sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column('public_key', sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column('sign_count', sa.Integer(), nullable=False),
sa.Column('transports', sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column('user_id', sa.Integer(), nullable=False),
sa.Column('created_at', sa.DateTime(), nullable=True,
server_default=sa.func.now()),
sa.ForeignKeyConstraint(['user_id'], ['user.id']),
sa.PrimaryKeyConstraint('id'),
)
op.create_index(op.f('ix_passkeycredential_credential_id'),
'passkeycredential', ['credential_id'], unique=True)
op.create_index(op.f('ix_passkeycredential_user_id'),
'passkeycredential', ['user_id'], unique=False)
def downgrade() -> None:
"""Downgrade schema."""
op.drop_index(op.f('ix_passkeycredential_user_id'),
table_name='passkeycredential')
op.drop_index(op.f('ix_passkeycredential_credential_id'),
table_name='passkeycredential')
op.drop_table('passkeycredential')
op.drop_column('user', 'created_at')
op.alter_column('user', 'password',
existing_type=sqlmodel.sql.sqltypes.AutoString(),
nullable=False)

30
crawler/alembic/versions/e5f1bc4e3323_fix_typo_in_logitude_column.py
View file

@ -1,30 +0,0 @@
"""fix typo in logitude column
Revision ID: e5f1bc4e3323
Revises: 8220f657bae5
Create Date: 2025-10-18 20:31:29.558034
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import mysql
# revision identifiers, used by Alembic.
revision: str = 'e5f1bc4e3323'
down_revision: Union[str, None] = '8220f657bae5'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Upgrade schema - this migration is now a no-op since tables already have correct column name."""
# The tables were created with 'longitude' (correct spelling) in the initial migration.
# This migration was incorrectly auto-generated and has been fixed to be a no-op.
pass
def downgrade() -> None:
"""Downgrade schema - no-op since upgrade is no-op."""
pass

318
crawler/api/app.py
View file

@ -1,318 +0,0 @@
"""FastAPI application for the Real Estate Crawler API."""
from datetime import datetime, timedelta
import json
import logging
import logging.config
from typing import Annotated, AsyncGenerator, Optional
from api.auth import get_current_user
from api.config import DEV_TIER_ORIGINS, PROD_TIER_ORIGINS
from api.passkey_routes import passkey_router
from dotenv import load_dotenv
from fastapi import Depends, FastAPI, Query
from fastapi.responses import StreamingResponse
from api.auth import User
from models.listing import QueryParameters, ListingType, FurnishType
from notifications import send_notification
from repositories.listing_repository import ListingRepository
from database import engine
from fastapi.middleware.cors import CORSMiddleware
from ui_exporter import convert_to_geojson_feature, convert_row_to_geojson
from services import listing_service, export_service, district_service, task_service
from services.listing_cache import (
get_cached_count,
get_cached_features,
cache_features_batch,
)
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from api.metrics import metrics_app
from opentelemetry.metrics import get_meter
load_dotenv()
logger = logging.getLogger("uvicorn")
DEFAULT_BATCH_SIZE = 50
def get_query_parameters(
listing_type: ListingType,
min_bedrooms: int = 1,
max_bedrooms: int = 999,
min_price: int = 0,
max_price: int = 10_000_000,
min_sqm: Optional[int] = None,
last_seen_days: Optional[int] = None,
let_date_available_from: Optional[datetime] = None,
furnish_types: Optional[str] = None, # comma-separated list
) -> QueryParameters:
"""Parse query parameters into QueryParameters model."""
parsed_furnish_types = None
if furnish_types:
parsed_furnish_types = [FurnishType(f.strip()) for f in furnish_types.split(",")]
return QueryParameters(
listing_type=listing_type,
min_bedrooms=min_bedrooms,
max_bedrooms=max_bedrooms,
min_price=min_price,
max_price=max_price,
min_sqm=min_sqm,
last_seen_days=last_seen_days,
let_date_available_from=let_date_available_from,
furnish_types=parsed_furnish_types,
)
app = FastAPI()
app.include_router(passkey_router)
app.mount("/metrics", metrics_app)
meter = get_meter(__name__)
request_counter = meter.create_counter(
name="custom_request_count",
description="Number of times /hello was called",
)
hist = meter.create_histogram(
name="custom_request_duration",
description="Duration of /hello requests in seconds",
)
# Allow CORS (for React frontend)
app.add_middleware(
CORSMiddleware,
allow_origins=[*DEV_TIER_ORIGINS, *PROD_TIER_ORIGINS],
allow_methods=["*"],
allow_headers=["*"],
)
@app.get("/api/status")
async def get_status() -> dict[str, str]:
request_counter.add(1, {"method": "GET", "path": "/status"})
hist.record(1.5, {"method": "GET", "path": "/status"})
return {"status": "OK"}
@app.get("/api/listing")
async def get_listing(
user: Annotated[User, Depends(get_current_user)],
limit: int = 5,
) -> dict[str, list]:
"""Get listings from the database."""
repository = ListingRepository(engine)
result = await listing_service.get_listings(repository, limit=limit)
logger.info(f"Fetched {result.total_count} listings for {user.email}")
return {"listings": result.listings}
@app.get("/api/listing_geojson")
async def get_listing_geojson(
user: Annotated[User, Depends(get_current_user)],
query_parameters: Annotated[QueryParameters, Depends(get_query_parameters)],
limit: int | None = None,
) -> dict:
"""Get listings as GeoJSON for map display."""
repository = ListingRepository(engine)
result = await export_service.export_to_geojson(
repository,
query_parameters=query_parameters,
limit=limit,
)
return result.data
async def _stream_from_cache(
query_parameters: QueryParameters,
batch_size: int,
limit: int | None,
) -> AsyncGenerator[str, None]:
"""Stream GeoJSON features from the Redis cache (cache-hit path)."""
cached_count = get_cached_count(query_parameters)
effective_total = min(limit, cached_count) if limit and cached_count else cached_count
yield json.dumps({
"type": "metadata",
"batch_size": batch_size,
"total_expected": effective_total,
"cached": True,
}) + "\n"
count = 0
for feature_batch in get_cached_features(query_parameters, batch_size=batch_size):
if limit and count + len(feature_batch) > limit:
feature_batch = feature_batch[:limit - count]
count += len(feature_batch)
yield json.dumps({"type": "batch", "features": feature_batch}) + "\n"
if limit and count >= limit:
break
yield json.dumps({"type": "complete", "total": count}) + "\n"
async def _stream_from_db(
query_parameters: QueryParameters,
batch_size: int,
limit: int | None,
) -> AsyncGenerator[str, None]:
"""Stream GeoJSON features from the database, populating the cache as we go."""
repository = ListingRepository(engine)
total = repository.count_listings(query_parameters)
effective_total = min(limit, total) if limit else total
yield json.dumps({
"type": "metadata",
"batch_size": batch_size,
"total_expected": effective_total,
"cached": False,
}) + "\n"
count = 0
batch: list[dict] = []
for row in repository.stream_listings_optimized(
query_parameters, limit=limit, page_size=batch_size
):
feature = convert_row_to_geojson(row, query_parameters.listing_type.value)
batch.append(feature)
count += 1
if len(batch) >= batch_size:
cache_features_batch(query_parameters, batch)
yield json.dumps({"type": "batch", "features": batch}) + "\n"
batch = []
if batch:
cache_features_batch(query_parameters, batch)
yield json.dumps({"type": "batch", "features": batch}) + "\n"
yield json.dumps({"type": "complete", "total": count}) + "\n"
@app.get("/api/listing_geojson/stream")
async def stream_listing_geojson(
user: Annotated[User, Depends(get_current_user)],
query_parameters: Annotated[QueryParameters, Depends(get_query_parameters)],
batch_size: int = DEFAULT_BATCH_SIZE,
limit: int | None = None,
) -> StreamingResponse:
"""Stream listings as NDJSON for progressive map loading.
Returns newline-delimited JSON with three message types:
- metadata: Initial message with batch_size and total_expected count
- batch: Array of GeoJSON features
- complete: Final message with total count
"""
cached_count = get_cached_count(query_parameters)
if cached_count is not None and cached_count > 0:
generator = _stream_from_cache(query_parameters, batch_size, limit)
else:
generator = _stream_from_db(query_parameters, batch_size, limit)
return StreamingResponse(
generator,
media_type="application/x-ndjson",
headers={
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no", # Disable nginx buffering
}
)
@app.post("/api/refresh_listings")
async def refresh_listings(
user: Annotated[User, Depends(get_current_user)],
query_parameters: Annotated[QueryParameters, Depends(get_query_parameters)],
) -> dict[str, str]:
"""Trigger a background task to refresh listings."""
await send_notification(
f"{user.email} refreshing listings with query parameters {query_parameters.model_dump_json()}"
)
repository = ListingRepository(engine)
result = await listing_service.refresh_listings(
repository,
query_parameters,
async_mode=True,
user_email=user.email,
)
# Track task for user
if result.task_id:
task_service.add_task_for_user(user.email, result.task_id)
return {"task_id": result.task_id or "", "message": result.message}
@app.get("/api/task_status")
async def get_task_status(
user: Annotated[User, Depends(get_current_user)],
task_id: str,
) -> dict[str, str | int | float | None]:
"""Get the status of a background task."""
status = task_service.get_task_status(task_id)
return {
"task_id": status.task_id,
"status": status.status,
"result": json.dumps(status.result) if status.result else None,
"progress": status.progress,
"processed": status.processed,
"total": status.total,
"message": status.message,
"error": status.error,
"traceback": status.traceback,
}
@app.get("/api/tasks_for_user")
async def get_tasks_for_user(
user: Annotated[User, Depends(get_current_user)],
) -> list[str]:
"""Get all task IDs for the current user."""
return task_service.get_user_tasks(user.email)
@app.post("/api/cancel_task")
async def cancel_task(
user: Annotated[User, Depends(get_current_user)],
task_id: str = Query(..., description="The task ID to cancel"),
) -> dict[str, str | bool]:
"""Cancel a running task and remove it from the user's task list."""
# Verify user owns this task
user_tasks = task_service.get_user_tasks(user.email)
if task_id not in user_tasks:
return {"success": False, "message": "Task not found or not owned by user"}
try:
task_service.cancel_task(task_id, user_email=user.email)
logger.info(f"Task {task_id} cancelled by {user.email}")
return {"success": True, "message": "Task cancelled"}
except Exception as e:
logger.error(f"Failed to cancel task {task_id}: {e}")
return {"success": False, "message": str(e)}
@app.post("/api/clear_all_tasks")
async def clear_all_tasks(
user: Annotated[User, Depends(get_current_user)],
) -> dict[str, str | int | bool]:
"""Clear all tasks for the current user."""
try:
count = task_service.clear_all_tasks(user.email)
logger.info(f"Cleared {count} tasks for {user.email}")
return {"success": True, "count": count, "message": f"Cleared {count} tasks"}
except Exception as e:
logger.error(f"Failed to clear tasks for {user.email}: {e}")
return {"success": False, "count": 0, "message": str(e)}
@app.get("/api/get_districts")
async def get_districts(
user: Annotated[User, Depends(get_current_user)],
) -> dict[str, str]:
"""Get all available districts."""
return district_service.get_all_districts()
FastAPIInstrumentor.instrument_app(app)

99
crawler/api/auth.py
View file

@ -1,99 +0,0 @@
from api.config import (
AUTHENTIK_URL,
OIDC_CACHE_TTL,
OIDC_CLIENT_ID,
OIDC_METADATA_URL,
JWT_SECRET,
JWT_ALGORITHM,
JWT_ISSUER,
)
from cachetools import TTLCache
from fastapi import Depends, HTTPException
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from httpx import AsyncClient
import jwt
from pydantic import BaseModel
# HTTPBearer scheme (provider-agnostic, works for both OIDC and passkey JWTs)
http_bearer = HTTPBearer()
JWKS_CACHE = TTLCache(maxsize=1, ttl=OIDC_CACHE_TTL)
OIDC_METADATA_CACHE = TTLCache(maxsize=1, ttl=OIDC_CACHE_TTL)
class User(BaseModel):
sub: str # User ID
email: str
name: str
async def get_oidc_metadata() -> dict: # type: ignore[type-arg]
if "oidc_metadata" not in OIDC_METADATA_CACHE:
async with AsyncClient() as client:
resp = await client.get(OIDC_METADATA_URL, follow_redirects=True)
OIDC_METADATA_CACHE["oidc_metadata"] = resp.json()
return OIDC_METADATA_CACHE["oidc_metadata"]
async def get_cached_jwks_client() -> jwt.PyJWKClient:
if "jwks_client" not in JWKS_CACHE:
metadata = await get_oidc_metadata()
jwks_url = metadata["jwks_uri"]
JWKS_CACHE["jwks_client"] = jwt.PyJWKClient(
jwks_url,
cache_keys=True, # PyJWT's built-in key caching
max_cached_keys=5,
)
return JWKS_CACHE["jwks_client"]
async def _verify_authentik_token(token: str) -> User:
"""Verify a token issued by Authentik (RS256 via JWKS)."""
metadata = await get_oidc_metadata()
signing_key = (await get_cached_jwks_client()).get_signing_key_from_jwt(token)
payload = jwt.decode(
token,
signing_key,
algorithms=["RS256"],
audience=OIDC_CLIENT_ID,
issuer=metadata["issuer"],
)
return User(**payload)
def _verify_passkey_token(token: str) -> User:
"""Verify a token issued by the passkey service (HS256)."""
payload = jwt.decode(
token,
JWT_SECRET,
algorithms=[JWT_ALGORITHM],
issuer=JWT_ISSUER,
)
return User(
sub=payload["sub"],
email=payload["email"],
name=payload.get("name", payload["email"]),
)
async def get_current_user(
credentials: HTTPAuthorizationCredentials = Depends(http_bearer),
) -> User:
token = credentials.credentials
try:
# Decode WITHOUT verification just to read the "iss" claim for routing.
# This is safe: we only use the issuer to decide which verified decode
# path to take next; the actual security check happens in the branch below.
unverified = jwt.decode(
token, options={"verify_signature": False, "verify_exp": False}
)
issuer = unverified.get("iss", "")
if issuer == JWT_ISSUER:
return _verify_passkey_token(token)
else:
return await _verify_authentik_token(token)
except jwt.PyJWTError as e:
raise HTTPException(status_code=401, detail=f"Invalid token: {e}")

33
crawler/api/config.py
View file

@ -1,33 +0,0 @@
from datetime import timedelta
import logging
import os
_logger = logging.getLogger(__name__)
# Authentik OIDC Configuration
AUTHENTIK_URL = os.getenv("AUTHENTIK_URL", "https://authentik.viktorbarzin.me")
OIDC_CLIENT_ID = os.getenv("OIDC_CLIENT_ID", "5AJKRgcdgVm1OyApBzFkadDFfStW9a555zwv2MOe")
OIDC_METADATA_URL = (
f"{AUTHENTIK_URL}/application/o/wrongmove/.well-known/openid-configuration"
)
OIDC_CACHE_TTL = timedelta(
hours=1
).total_seconds() # Cache to avoid spamming authentik with requests
DEV_TIER_ORIGINS = ["https://localhost/"]
PROD_TIER_ORIGINS = ["https://wrongmove.viktorbarzin.me/"]
# WebAuthn / Passkey Configuration
WEBAUTHN_RP_ID = os.getenv("WEBAUTHN_RP_ID", "localhost")
WEBAUTHN_RP_NAME = os.getenv("WEBAUTHN_RP_NAME", "Wrongmove")
WEBAUTHN_ORIGIN = os.getenv("WEBAUTHN_ORIGIN", "https://localhost")
# JWT Configuration (for passkey-issued tokens)
JWT_SECRET = os.getenv("JWT_SECRET", "change-me-in-production")
if JWT_SECRET == "change-me-in-production":
_logger.warning("JWT_SECRET is using the default value. Set JWT_SECRET env var in production.")
JWT_ALGORITHM = os.getenv("JWT_ALGORITHM", "HS256")
JWT_EXPIRATION_HOURS = int(os.getenv("JWT_EXPIRATION_HOURS", "24"))
JWT_ISSUER = os.getenv("JWT_ISSUER", "wrongmove")

17
crawler/api/metrics.py
View file

@ -1,17 +0,0 @@
# metrics.py
from opentelemetry.metrics import set_meter_provider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from prometheus_client import make_asgi_app
# Set up Prometheus reader and meter provider
reader = PrometheusMetricReader()
provider = MeterProvider(
metric_readers=[reader],
resource=Resource.create({SERVICE_NAME: "fastapi-metrics-app"}),
)
set_meter_provider(provider)
# Expose the Prometheus metrics endpoint
metrics_app = make_asgi_app() # Exposes /metrics

93
crawler/api/passkey_routes.py
View file

@ -1,93 +0,0 @@
import logging
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel, EmailStr
from database import engine
from repositories.user_repository import UserRepository
from services import passkey_service
logger = logging.getLogger("uvicorn")
passkey_router = APIRouter(prefix="/api/passkey", tags=["passkey"])
class RegisterBeginRequest(BaseModel):
email: EmailStr
class RegisterBeginResponse(BaseModel):
options: dict # type: ignore[type-arg]
session_id: str
class CeremonyCompleteRequest(BaseModel):
session_id: str
credential: dict # type: ignore[type-arg]
class AuthTokenResponse(BaseModel):
token: str
class LoginBeginResponse(BaseModel):
options: dict # type: ignore[type-arg]
session_id: str
@passkey_router.post("/register/begin", response_model=RegisterBeginResponse)
async def register_begin(body: RegisterBeginRequest) -> RegisterBeginResponse:
"""Start passkey registration ceremony."""
try:
user_repo = UserRepository(engine)
options, session_id = passkey_service.begin_registration(
body.email, user_repo
)
return RegisterBeginResponse(options=options, session_id=session_id)
except Exception as e:
logger.error(f"Registration begin failed: {e}")
raise HTTPException(status_code=400, detail=str(e))
@passkey_router.post("/register/complete", response_model=AuthTokenResponse)
async def register_complete(body: CeremonyCompleteRequest) -> AuthTokenResponse:
"""Complete passkey registration ceremony."""
try:
user_repo = UserRepository(engine)
token = passkey_service.complete_registration(
body.session_id, body.credential, user_repo
)
return AuthTokenResponse(token=token)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
except Exception as e:
logger.error(f"Registration complete failed: {e}")
raise HTTPException(status_code=400, detail=str(e))
@passkey_router.post("/login/begin", response_model=LoginBeginResponse)
async def login_begin() -> LoginBeginResponse:
"""Start passkey authentication ceremony."""
try:
user_repo = UserRepository(engine)
options, session_id = passkey_service.begin_authentication(user_repo)
return LoginBeginResponse(options=options, session_id=session_id)
except Exception as e:
logger.error(f"Login begin failed: {e}")
raise HTTPException(status_code=400, detail=str(e))
@passkey_router.post("/login/complete", response_model=AuthTokenResponse)
async def login_complete(body: CeremonyCompleteRequest) -> AuthTokenResponse:
"""Complete passkey authentication ceremony."""
try:
user_repo = UserRepository(engine)
token = passkey_service.complete_authentication(
body.session_id, body.credential, user_repo
)
return AuthTokenResponse(token=token)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
except Exception as e:
logger.error(f"Login complete failed: {e}")
raise HTTPException(status_code=400, detail=str(e))

31
crawler/celery_app.py
View file

@ -1,31 +0,0 @@
import sys
from celery import Celery
from dotenv import load_dotenv
import os
load_dotenv()
app = Celery(
"celery_app",
broker=os.getenv("CELERY_BROKER_URL", "redis://localhost:6379/0"),
backend=os.getenv("CELERY_RESULT_BACKEND", "redis://localhost:6379/1"),
include=["tasks.listing_tasks"],
)
app.conf.update(
task_serializer="json",
result_serializer="json",
accept_content=["json"],
timezone="UTC",
enable_utc=True,
)
if __name__ == "__main__":
try:
with app.connection() as conn:
conn.ensure_connection(max_retries=0)
print("Broker connection OK")
sys.exit(0)
except Exception as e:
print(f"Broker connection failed: {e}")
sys.exit(1)

5
crawler/config/__init__.py
View file

@ -1,5 +0,0 @@
"""Configuration modules."""
from config.schedule_config import ScheduleConfig, SchedulesConfig
from config.scraper_config import ScraperConfig
__all__ = ["ScheduleConfig", "SchedulesConfig", "ScraperConfig"]

122
crawler/config/schedule_config.py
View file

@ -1,122 +0,0 @@
"""Schedule configuration for periodic scraping tasks."""
from __future__ import annotations
import json
import logging
import os
import re
from typing import Self
from pydantic import BaseModel, field_validator
from models.listing import FurnishType, ListingType, QueryParameters
logger = logging.getLogger("uvicorn.error")
# Cron field validation patterns
CRON_MINUTE_PATTERN = re.compile(r"^(\*|([0-5]?\d)(,[0-5]?\d)*|\*/[1-9]\d*)$")
CRON_HOUR_PATTERN = re.compile(r"^(\*|(1?\d|2[0-3])(,(1?\d|2[0-3]))*|\*/[1-9]\d*)$")
CRON_DAY_OF_WEEK_PATTERN = re.compile(r"^(\*|[0-6](,[0-6])*|\*/[1-6])$")
class ScheduleConfig(BaseModel):
"""Configuration for a single periodic scrape schedule."""
name: str
enabled: bool = True
minute: str = "0"
hour: str = "2"
day_of_week: str = "*"
listing_type: ListingType
min_bedrooms: int = 1
max_bedrooms: int = 999
min_price: int = 0
max_price: int = 10_000_000
district_names: list[str] = []
furnish_types: list[str] | None = None
@field_validator("minute")
@classmethod
def validate_minute(cls, v: str) -> str:
"""Validate cron minute field (0-59, *, or */N)."""
if not CRON_MINUTE_PATTERN.match(v):
raise ValueError(
f"Invalid cron minute '{v}'. Must be 0-59, *, */N, or comma-separated values."
)
return v
@field_validator("hour")
@classmethod
def validate_hour(cls, v: str) -> str:
"""Validate cron hour field (0-23, *, or */N)."""
if not CRON_HOUR_PATTERN.match(v):
raise ValueError(
f"Invalid cron hour '{v}'. Must be 0-23, *, */N, or comma-separated values."
)
return v
@field_validator("day_of_week")
@classmethod
def validate_day_of_week(cls, v: str) -> str:
"""Validate cron day_of_week field (0-6, *, or */N)."""
if not CRON_DAY_OF_WEEK_PATTERN.match(v):
raise ValueError(
f"Invalid cron day_of_week '{v}'. Must be 0-6, *, */N, or comma-separated values."
)
return v
def to_query_parameters(self) -> QueryParameters:
"""Convert schedule config to QueryParameters for the scrape task."""
furnish_types_enum: list[FurnishType] | None = None
if self.furnish_types:
furnish_types_enum = [FurnishType(ft) for ft in self.furnish_types]
return QueryParameters(
listing_type=self.listing_type,
min_bedrooms=self.min_bedrooms,
max_bedrooms=self.max_bedrooms,
min_price=self.min_price,
max_price=self.max_price,
district_names=set(self.district_names),
furnish_types=furnish_types_enum,
)
class SchedulesConfig(BaseModel):
"""Container for multiple schedule configurations."""
schedules: list[ScheduleConfig] = []
@classmethod
def from_env(cls, env_var: str = "SCRAPE_SCHEDULES") -> Self:
"""Load schedules from environment variable.
Args:
env_var: Name of the environment variable containing JSON config.
Returns:
SchedulesConfig instance with parsed schedules.
Raises:
ValueError: If the JSON is invalid or schedule validation fails.
"""
raw_value = os.environ.get(env_var, "").strip()
if not raw_value:
logger.info(f"No {env_var} configured, no periodic scrapes will be scheduled")
return cls(schedules=[])
try:
parsed = json.loads(raw_value)
except json.JSONDecodeError as e:
raise ValueError(f"Invalid JSON in {env_var}: {e}") from e
if not isinstance(parsed, list):
raise ValueError(f"{env_var} must be a JSON array")
schedules = [ScheduleConfig.model_validate(item) for item in parsed]
return cls(schedules=schedules)
def get_enabled_schedules(self) -> list[ScheduleConfig]:
"""Return only enabled schedules."""
return [s for s in self.schedules if s.enabled]

89
crawler/config/scraper_config.py
View file

@ -1,89 +0,0 @@
"""Scraper configuration with environment variable loading."""
from __future__ import annotations
import os
from dataclasses import dataclass
from typing import Self
@dataclass(frozen=True)
class ScraperConfig:
"""Configuration for the Rightmove scraper.
Attributes:
max_concurrent_requests: Maximum number of concurrent HTTP requests.
request_delay_ms: Delay between requests in milliseconds.
result_cap: Maximum results Rightmove returns per query (their limit).
split_threshold: When results exceed this, split the query further.
min_price_band: Minimum width of a price band (won't split below this).
max_pages_per_query: Maximum pages to fetch per subquery (60 * 25 = 1500).
proxy_url: Optional SOCKS proxy URL (e.g., socks5://localhost:9050 for Tor).
slow_response_threshold: Response time threshold in seconds for throttle detection.
enable_circuit_breaker: Whether to enable circuit breaker protection.
circuit_breaker_failure_threshold: Number of consecutive failures to open circuit.
circuit_breaker_recovery_timeout: Seconds to wait before testing recovery.
"""
max_concurrent_requests: int = 5
request_delay_ms: int = 100
result_cap: int = 1500
split_threshold: int = 1200 # Split when approaching cap
min_price_band: int = 100 # Minimum band width in currency units
max_pages_per_query: int = 60 # 60 * 25 = 1500 results max
proxy_url: str | None = None
slow_response_threshold: float = 10.0 # seconds
enable_circuit_breaker: bool = True
circuit_breaker_failure_threshold: int = 5
circuit_breaker_recovery_timeout: float = 60.0
@classmethod
def from_env(cls) -> Self:
"""Load configuration from environment variables.
Environment variables:
RIGHTMOVE_MAX_CONCURRENT: Max concurrent requests (default: 5)
RIGHTMOVE_REQUEST_DELAY_MS: Request delay in ms (default: 100)
RIGHTMOVE_RESULT_CAP: Result cap per query (default: 1500)
RIGHTMOVE_SPLIT_THRESHOLD: Split threshold (default: 1200)
RIGHTMOVE_MIN_PRICE_BAND: Minimum price band width (default: 100)
RIGHTMOVE_MAX_PAGES: Max pages per query (default: 60)
RIGHTMOVE_PROXY_URL: SOCKS proxy URL (default: None)
RIGHTMOVE_SLOW_RESPONSE_THRESHOLD: Slow response threshold in seconds (default: 10.0)
RIGHTMOVE_ENABLE_CIRCUIT_BREAKER: Enable circuit breaker (default: True)
RIGHTMOVE_CIRCUIT_BREAKER_FAILURES: Failures to open circuit (default: 5)
RIGHTMOVE_CIRCUIT_BREAKER_TIMEOUT: Recovery timeout in seconds (default: 60.0)
Returns:
ScraperConfig instance with values from environment or defaults.
"""
return cls(
max_concurrent_requests=int(
os.environ.get("RIGHTMOVE_MAX_CONCURRENT", "5")
),
request_delay_ms=int(
os.environ.get("RIGHTMOVE_REQUEST_DELAY_MS", "100")
),
result_cap=int(os.environ.get("RIGHTMOVE_RESULT_CAP", "1500")),
split_threshold=int(
os.environ.get("RIGHTMOVE_SPLIT_THRESHOLD", "1200")
),
min_price_band=int(
os.environ.get("RIGHTMOVE_MIN_PRICE_BAND", "100")
),
max_pages_per_query=int(
os.environ.get("RIGHTMOVE_MAX_PAGES", "60")
),
proxy_url=os.environ.get("RIGHTMOVE_PROXY_URL") or None,
slow_response_threshold=float(
os.environ.get("RIGHTMOVE_SLOW_RESPONSE_THRESHOLD", "10.0")
),
enable_circuit_breaker=os.environ.get(
"RIGHTMOVE_ENABLE_CIRCUIT_BREAKER", "true"
).lower() in ("true", "1", "yes"),
circuit_breaker_failure_threshold=int(
os.environ.get("RIGHTMOVE_CIRCUIT_BREAKER_FAILURES", "5")
),
circuit_breaker_recovery_timeout=float(
os.environ.get("RIGHTMOVE_CIRCUIT_BREAKER_TIMEOUT", "60.0")
),
)

40
crawler/csv_exporter.py
View file

@ -1,40 +0,0 @@
from pathlib import Path
import pandas as pd
from models.listing import QueryParameters
from repositories.listing_repository import ListingRepository
async def export_to_csv(
repository: ListingRepository,
output_file: Path,
query_parameters: QueryParameters | None = None,
) -> None:
listings = await repository.get_listings(query_parameters=query_parameters)
ds = [listing.__dict__ for listing in listings]
df = pd.DataFrame(ds)
# read decisions on file
decisions_path = "data/decisions.json"
decisions = pd.read_json(decisions_path)
df.loc[:, "decision"] = df.id.apply(lambda x: decisions.get(x))
# remove _sa_instance_state column
drop_columns = ["_sa_instance_state", "additional_info"]
df = df.drop(columns=drop_columns)
# fill in gap values for service charge and lease left for Excel filters
if "service_charge" not in df.columns:
df.loc[:, "service_charge"] = -1
df.loc[:, "service_charge"] = df.service_charge.fillna(-1)
if "lease_left" not in df.columns:
df.loc[:, "lease_left"] = -1
df.loc[:, "lease_left"] = df.lease_left.fillna(-1)
if "square_meters" not in df.columns:
df.loc[:, "square_meters"] = -1
df.loc[:, "square_meters"] = df.square_meters.fillna(-1)
# Add price per sqm column
df.loc[:, "price_per_sqm"] = df.price / df.square_meters
df = df.sort_values(by=["price_per_sqm"], ascending=True)
df.to_csv(str(output_file), index=False)

469
crawler/data_access.py
View file

@ -1,469 +0,0 @@
"""Legacy filesystem-based data access.
.. deprecated::
This module is only used by the ``populate_db`` CLI command for migrating
old filesystem data into the database. Do not import from this module in
new code. Use ``models.listing.RentListing`` or ``models.listing.BuyListing``
and ``repositories.listing_repository.ListingRepository`` instead.
"""
import asyncio
from collections import defaultdict
from dataclasses import dataclass
import json
import pathlib
from typing import Any, List
import warnings
from models.listing import ListingSite, PriceHistoryItem
from rec import floorplan, routing
import re
import datetime
@dataclass()
class Listing:
"""Legacy Listing class for filesystem-based data access.
.. deprecated::
Use models.listing.RentListing or models.listing.BuyListing instead.
This class is kept for backwards compatibility with the populate_db command.
"""
identifier: int
_details_object: dict[str, Any] | None = None
_listing_object: dict[str, Any] | None = None
data_dir: pathlib.Path = pathlib.Path("data/rs/")
ALL_COLUMNS = [
"identifier",
"sqm_ocr",
"price",
"price_per_sqm",
"url",
"bedrooms",
"travel_time_fastest",
"travel_time_second",
"lease_left",
"service_charge",
"development",
"tenure_type",
"updated_days",
"status",
"last_seen",
"agency",
"council_tax_band",
]
def __post_init__(self) -> None:
warnings.warn(
"data_access.Listing is deprecated. Use models.listing.RentListing "
"or models.listing.BuyListing instead.",
DeprecationWarning,
stacklevel=3,
)
@staticmethod
def get_all_listings(
listing_paths: list[pathlib.Path],
seen_in_the_last_n_days: int = 30,
) -> List["Listing"]:
identifiers = []
for listing_path in listing_paths:
with open(listing_path) as f:
d = json.load(f)
# data_dir is the first directory before the listing_path
data_dir = pathlib.Path(listing_path)
while str(d["identifier"]) in str(data_dir.resolve().absolute()):
data_dir = data_dir.parent
listing = Listing(d["identifier"], data_dir=data_dir)
if (
listing.last_seen is not None
and listing.last_seen < seen_in_the_last_n_days
):
identifiers.append(listing)
return identifiers
def path_listing(self) -> pathlib.Path:
p = self.data_dir / str(self.identifier)
p.mkdir(parents=True, exist_ok=True)
return p
def path_listing_json(self) -> pathlib.Path:
return self.path_listing() / "listing.json"
def path_detail_json(self) -> pathlib.Path:
return self.path_listing() / "detail.json"
def path_routing_json(self) -> pathlib.Path:
return self.path_listing() / "routing.json"
def path_floorplan_model_json(self) -> pathlib.Path:
return self.path_listing() / "floorplan_model.json"
def path_floorplan_ocr_json(self) -> pathlib.Path:
return self.path_listing() / "floorplan_ocr.json"
def path_pic_folder(self) -> pathlib.Path:
return self.path_listing() / "pics"
def path_pic_file(self, order, name) -> pathlib.Path:
self.path_pic_folder().mkdir(parents=True, exist_ok=True)
return self.path_pic_folder() / f"{order}_{name}"
def path_floorplan_folder(self) -> pathlib.Path:
return self.path_listing() / "floorplans"
def path_floorplan_file(self, order, name) -> pathlib.Path:
self.path_floorplan_folder().mkdir(parents=True, exist_ok=True)
return self.path_floorplan_folder() / f"{order}_{name}"
def path_last_seen_listing(self) -> pathlib.Path:
return self.path_listing() / "last_seen.json"
def path_price_history(self) -> pathlib.Path:
return self.path_listing() / "price_history.json"
def dump_listing(self) -> None:
if self._listing_object is None:
raise ValueError("No listing data provided to dump.")
with open(self.path_listing_json(), "w") as f:
json.dump(self._listing_object, f)
with open(self.path_last_seen_listing(), "w") as f:
dt = datetime.datetime.now().isoformat()
json.dump(dt, f)
# some places list pw in price and others pcm
price = max(
self._listing_object["price"] or 0,
self._listing_object.get("monthlyRent", 0) or 0,
)
self.append_price_history(price)
def append_price_history(self, price: float) -> None:
"""Append the price history to the listing's price history file."""
existing_price_history = (
json.loads(self.path_price_history().read_text())
if self.path_price_history().exists()
else []
)
now = datetime.datetime.now().isoformat()
# if the last price is the same, just update the date
if len(existing_price_history) > 0:
last_price = existing_price_history[-1]["price"]
if last_price == price:
existing_price_history[-1]["last_seen"] = now
else:
existing_price_history.append(
{
"first_seen": now,
"last_seen": now,
"price": price,
}
)
with open(self.path_price_history(), "w") as f:
json.dump(existing_price_history, f, indent=4)
def list_floorplans(self):
images = list(self.path_floorplan_folder().glob("*"))
# todo add check if return is image
return images
async def calculate_sqm_ocr(self, recalculate=True):
objs = []
if self.path_floorplan_ocr_json().exists():
with open(self.path_floorplan_ocr_json()) as f:
objs = json.load(f)
if not recalculate and len(objs) > 0:
return
for floorplan_path in self.list_floorplans():
estimated_sqm, model_output = await asyncio.to_thread(
floorplan.calculate_ocr, floorplan_path
)
objs.append(
{
"floorplan_path": str(floorplan_path),
"estimated_sqm": estimated_sqm,
"text": model_output,
}
)
with open(self.path_floorplan_ocr_json(), "w") as f:
json.dump(objs, f)
async def sqm_ocr(self, recalculate=False) -> float | None:
if not self.path_floorplan_ocr_json().exists() or recalculate:
await self.calculate_sqm_ocr()
with open(self.path_floorplan_ocr_json()) as f:
objs = json.load(f)
sqms = [o["estimated_sqm"] for o in objs if o["estimated_sqm"] is not None]
if len(sqms) == 0:
return None
max_sqm = max(sqms)
return max_sqm
def calculate_route(
self, dest_address: str, travel_mode: routing.TravelMode, recalculate=False
) -> dict[str, Any]:
routing_cache = self.__get_routing_cache()
cache_key = self.__routing_cache_key(dest_address, travel_mode)
if (
route_cache := routing_cache.get(cache_key)
) is not None and not recalculate:
return {cache_key: route_cache}
result = routing.transit_route(
self.latitude,
self.longitude,
dest_address,
travel_mode,
)
if not result:
raise Exception(
(
f"Error calculating route from {self.identifier} "
f"to '{dest_address}' by {travel_mode}"
)
)
result = {**{cache_key: result}, **routing_cache}
with open(self.path_routing_json(), "w") as f:
json.dump(result, f)
return result
def travel_time(
self,
destination_address: str,
travel_mode: routing.TravelMode,
) -> list[dict[str, Any]]:
data = self.calculate_route(destination_address, travel_mode)
return self.__extract_travel_times(data, destination_address, travel_mode)
@property
def url(self):
return f"https://www.rightmove.co.uk/properties/{self.identifier}"
@property
def listingobject(self):
with open(self.path_listing_json()) as f:
return json.load(f)
@property
def detailobject(self) -> dict[str, Any]:
if self._details_object is not None:
return self._details_object
if (
self.path_detail_json().exists()
and json.load(self.path_detail_json().open()).get("property") is not None
):
with open(self.path_detail_json()) as f:
self._details_object = json.load(f)
return self._details_object # type: ignore
raise ValueError(f"Detail object for listing {self.identifier} not found.")
@property
def price(self) -> float:
return self.detailobject["property"]["price"]
@property
def tenure_type(self) -> str:
return self.detailobject["property"]["tenureType"]
async def price_per_sqm(self) -> float:
sqm_ocr = await self.sqm_ocr()
if sqm_ocr is None or sqm_ocr == 0:
return -1
return self.price / sqm_ocr
@property
def bedrooms(self) -> int:
return self.detailobject["property"]["bedrooms"]
@property
def latitude(self) -> float:
return self.detailobject["property"]["latitude"]
@property
def longitude(self) -> float:
return self.detailobject["property"]["longitude"]
@property
def leaseLeft(self) -> float | None:
ds = self.detailobject["property"].get("tenureInfo", {}).get("content", [])
for d in ds:
if d["type"] == "lengthOfLease":
matches = re.findall(r"(\d+\.?\d*)", d["value"])
if len(matches):
return float(matches[0])
return None
@property
def updateDaysAgo(self) -> int:
ts = self.detailobject["property"]["updateDate"] / 1000
now = datetime.datetime.now()
ds = datetime.datetime.fromtimestamp(ts)
return (now - ds).days
@property
def last_seen(self) -> int:
with open(self.path_last_seen_listing(), "r") as f:
datetime_str = json.load(f)
dt = datetime.datetime.fromisoformat(datetime_str)
return (datetime.datetime.now() - dt).days
@property
def serviceCharge(self) -> float | None:
ds = self.detailobject["property"].get("tenureInfo", {}).get("content", [])
for d in ds:
if d["type"] == "annualServiceCharge":
matches = re.findall(r"([\d,.]+)", d["value"])
if len(matches):
# remove separators (e.g. 6,395.76)
match = matches[0].replace(",", "")
return float(match)
return None
@property
def development(self) -> bool:
# aka new home
try:
return self.detailobject["property"]["development"]
except Exception:
return False
@property
def isRemoved(self) -> bool:
return not self.detailobject["property"]["visible"]
@property
def status(self) -> str:
if self.isRemoved:
return "removed"
status = self.detailobject["property"]["status"]
return status
@property
def agency(self) -> str:
return self.detailobject["property"]["branch"]["brandName"]
@property
def councilTaxBand(self) -> str:
return self.detailobject["property"]["councilTaxInfo"]["content"][0]["value"]
@property
def photoThumbnail(self) -> str | None:
# options are: 'url', 'thumbnailUrl', 'maxSizeUrl'
photos = self.detailobject["property"]["photos"]
if len(photos) > 0:
return photos[0]["url"]
return None
@property
def letDateAvailable(self) -> datetime.datetime | None:
# options are: 'url', 'thumbnailUrl', 'maxSizeUrl'
let_date_available: str | None = self.detailobject["property"][
"letDateAvailable"
] # Seems null for all assets?
if let_date_available is None:
return None
if let_date_available == "Now":
return datetime.datetime.now()
try:
return datetime.datetime.strptime(let_date_available, "%d/%m/%Y")
except ValueError:
# If the date format is not as expected, return None
return None
@property
def priceHistory(self) -> list[PriceHistoryItem]:
if not self.path_price_history().exists():
return []
with open(self.path_price_history(), "r") as f:
data = json.load(f)
return [
PriceHistoryItem(
first_seen=datetime.datetime.fromisoformat(item["first_seen"]),
last_seen=datetime.datetime.fromisoformat(item["last_seen"]),
price=item["price"],
)
for item in data
]
@property
def listing_site(self) -> ListingSite:
return ListingSite.RIGHTMOVE # this class supports only right move
def __routing_cache_key(
self,
dest_address: str,
travel_mode: routing.TravelMode,
) -> str:
return f"{dest_address} by {travel_mode}"
def __from_routing_cache_key(
self,
cache_key: str,
) -> tuple[str, routing.TravelMode]:
match = re.match(r"(.+) by (.+)", cache_key)
if not match:
raise ValueError(f"Invalid cache key: {cache_key}")
return match.group(1), routing.TravelMode[match.group(2)]
def __extract_travel_times(
self,
routing_data: dict[str, Any],
destination_address: str,
travel_mode: routing.TravelMode,
limit: int = 2,
) -> list[dict[str, Any]]:
res = []
cache_key = self.__routing_cache_key(destination_address, travel_mode)
for route in routing_data[cache_key]["routes"]:
distance = route["distanceMeters"]
duration = int(route["duration"].strip("s"))
duration_static = int(route["staticDuration"].strip("s"))
steps = route["legs"][0]["steps"]
initial_walk_duration = 0
used_transit = False
duration_per_transit = defaultdict(lambda: 0)
distance_per_transit = defaultdict(lambda: 0)
number_of_transit_stops = 0
for step in steps:
if not used_transit and step["travelMode"] == "WALK":
initial_walk_duration += int(step["staticDuration"].strip("s"))
else:
used_transit = True
duration_per_transit[step["travelMode"]] += int(
step["staticDuration"].strip("s")
)
distance_per_transit[step["travelMode"]] += step.get(
"distanceMeters", 0
)
if step["travelMode"] == "TRANSIT":
number_of_transit_stops += 1
res.append(
{
"duration": duration,
"distance": distance,
"duration_static": duration_static,
"initial_walk_duration": initial_walk_duration,
"duration_per_transit": dict(duration_per_transit),
"distance_per_transit": dict(distance_per_transit),
"number_of_transit_stops": number_of_transit_stops,
}
)
return res[:limit]
def __get_routing_cache(self) -> dict[str, Any]:
try:
with open(self.path_routing_json(), "x") as f:
json.dump({}, f)
return {}
except FileExistsError:
pass
with open(self.path_routing_json(), "r") as f:
return json.load(f)

23
crawler/database.py
View file

@ -1,23 +0,0 @@
import os
from sqlmodel import create_engine, SQLModel
from sqlalchemy.orm import sessionmaker
from dotenv import load_dotenv
load_dotenv()
# PostgreSQL example (or use "sqlite:///database.db" for SQLite)
# DATABASE_URL = "postgresql://user:password@localhost/db_name"
# DATABASE_URL = "sqlite:///data/wrongmove.db"
# DATABASE_URL = "mysql://wrongmove:wrongmove@localhost:3306/wrongmove"
DATABASE_URL = os.environ["DB_CONNECTION_STRING"]
debug = os.getenv("ENV", "dev") == "dev"
engine = create_engine(DATABASE_URL, echo=debug) # `echo=True` for debug logs
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
def init_db():
"""Create all tables (only for development; use migrations in production)."""
SQLModel.metadata.create_all(engine)

152
crawler/docker-compose.yml
View file

@ -1,152 +0,0 @@
services:
redis:
image: redis:8
container_name: rec-redis
ports:
- "6379:6379"
volumes:
- redis_data:/data
command: ["redis-server", "--appendonly", "yes"]
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 5
networks:
- rec-network
mysql:
image: mysql:9
container_name: rec-mysql
hostname: mysql
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: rootpass
MYSQL_DATABASE: wrongmove
MYSQL_USER: wrongmove
MYSQL_PASSWORD: wrongmove
volumes:
- mysql_data:/var/lib/mysql
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
networks:
- rec-network
app:
build:
context: .
dockerfile: Dockerfile
container_name: rec-app
ports:
- "5001:5001"
volumes:
# Bind mount source code for development
- .:/app
# Preserve virtual environment in container
- app_venv:/app/.venv
environment:
- ENV=dev
- DB_CONNECTION_STRING=mysql+mysqldb://wrongmove:wrongmove@mysql:3306/wrongmove
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/0
- ROUTING_API_KEY=${ROUTING_API_KEY:-}
depends_on:
redis:
condition: service_healthy
mysql:
condition: service_healthy
command: ["uvicorn", "api.app:app", "--host", "0.0.0.0", "--port", "5001", "--reload", "--reload-dir", "api", "--reload-dir", "services", "--reload-dir", "repositories", "--reload-dir", "models"]
networks:
- rec-network
celery:
build:
context: .
dockerfile: Dockerfile
container_name: rec-celery
volumes:
- .:/app
- app_venv:/app/.venv
environment:
- ENV=dev
- DB_CONNECTION_STRING=mysql+mysqldb://wrongmove:wrongmove@mysql:3306/wrongmove
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/0
- ROUTING_API_KEY=${ROUTING_API_KEY:-}
- SCRAPE_SCHEDULES=${SCRAPE_SCHEDULES:-}
depends_on:
redis:
condition: service_healthy
mysql:
condition: service_healthy
command: ["celery", "-A", "celery_app", "worker", "--loglevel=info"]
networks:
- rec-network
celery-beat:
build:
context: .
dockerfile: Dockerfile
container_name: rec-celery-beat
volumes:
- .:/app
- app_venv:/app/.venv
environment:
- ENV=dev
- DB_CONNECTION_STRING=mysql+mysqldb://wrongmove:wrongmove@mysql:3306/wrongmove
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/0
- SCRAPE_SCHEDULES=${SCRAPE_SCHEDULES:-}
depends_on:
- redis
- celery
command: ["celery", "-A", "celery_app", "beat", "--loglevel=info"]
networks:
- rec-network
frontend:
image: node:24-alpine
container_name: rec-frontend
working_dir: /app
ports:
- "5173:5173"
volumes:
- ./frontend:/app
- frontend_node_modules:/app/node_modules
environment:
- DEV_HOST=${DEV_HOST:-localhost}
command: sh -c "npm ci && npm run dev -- --host"
networks:
- rec-network
caddy:
image: caddy:alpine
container_name: rec-caddy
ports:
- "443:443"
volumes:
- ./frontend/Caddyfile.dev:/etc/caddy/Caddyfile
- caddy_data:/data
environment:
- DEV_HOST=${DEV_HOST:-localhost}
depends_on:
- frontend
- app
networks:
- rec-network
networks:
rec-network:
driver: bridge
volumes:
redis_data:
mysql_data:
app_venv:
frontend_node_modules:
caddy_data:

183
crawler/docs/BACKEND.md
View file

@ -1,183 +0,0 @@
# Real Estate Crawler - Backend Documentation
A property listing aggregator that scrapes Rightmove UK, extracts square meters via OCR, and calculates transit routes.
## Quick Start
```bash
# Docker (recommended) - starts Redis, MySQL, API, and Celery
./start.sh
# Or run locally with Poetry
poetry install
./start.sh --local
```
API available at `http://localhost:5001`
## Dependencies
| Dependency | Purpose |
|------------|---------|
| Python 3.11+ | Runtime |
| Redis | Celery message broker |
| MySQL/SQLite | Database |
| Tesseract OCR | Floorplan text extraction |
| Docker | Containerized deployment |
### Python Packages (key)
- `fastapi` + `uvicorn` - HTTP API
- `celery` - Background tasks
- `sqlmodel` - ORM
- `pytesseract` + `opencv` - OCR
- `aiohttp` - Async HTTP client
## API Endpoints
### Health Check
```bash
curl http://localhost:5001/api/status
# {"status": "OK"}
```
### Get Listings
```bash
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:5001/api/listing?limit=10"
```
### Get Listings as GeoJSON
```bash
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:5001/api/listing_geojson?listing_type=RENT&min_bedrooms=2&max_price=3000"
```
### Refresh Listings (async)
```bash
curl -X POST -H "Authorization: Bearer $TOKEN" \
"http://localhost:5001/api/refresh_listings?listing_type=RENT&min_bedrooms=2&max_bedrooms=3&min_price=2000&max_price=4000"
# {"task_id": "abc123", "message": "Task abc123 started"}
```
### Check Task Status
```bash
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:5001/api/task_status?task_id=abc123"
# {"task_id": "abc123", "status": "SUCCESS", "result": "..."}
```
### Get Districts
```bash
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:5001/api/get_districts"
# {"Westminster": "REGION^93965", "Camden": "REGION^93934", ...}
```
## CLI Commands
```bash
# Fetch listings from Rightmove
python main.py dump-listings -t rent --min-bedrooms 2 --max-price 4000
# Download floorplan images
python main.py dump-images
# Run OCR on floorplans
python main.py detect-floorplan
# Calculate transit routes
python main.py routing -d "10 Downing Street, London" -m TRANSIT -l 10
# Export to GeoJSON
python main.py export-immoweb -O output.geojson -t rent --min-bedrooms 2
# Export to CSV
python main.py export-csv -O output.csv -t rent
# List available districts
python main.py list-districts
```
## Query Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `listing_type` | RENT/BUY | Property type |
| `min_bedrooms` | int | Minimum bedrooms |
| `max_bedrooms` | int | Maximum bedrooms |
| `min_price` | int | Minimum price |
| `max_price` | int | Maximum price |
| `min_sqm` | int | Minimum square meters |
| `district` | string | District name (repeatable) |
| `furnish_types` | string | FURNISHED/UNFURNISHED/PART_FURNISHED |
| `last_seen_days` | int | Only listings seen in last N days |
## Architecture
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ CLI │ │ HTTP API │ │ Celery │
│ (main.py) │ │ (api/app.py)│ │ Worker │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
└───────────────────┼───────────────────┘
│
┌────────▼────────┐
│ Services │
│ (services/*.py) │
└────────┬────────┘
│
┌────────────┼────────────┐
│ │ │
┌──────▼──────┐ ┌───▼───┐ ┌──────▼──────┐
│ Repository │ │ Redis │ │ Rightmove │
│ (MySQL) │ │ │ │ API │
└─────────────┘ └───────┘ └─────────────┘
```
## Environment Variables
```bash
# Database
DB_CONNECTION_STRING=mysql://user:pass@localhost:3306/wrongmove
# Redis (Celery)
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0
# Google Maps (optional, for routing)
ROUTING_API_KEY=your_api_key
```
## Authentication
API endpoints (except `/api/status`) require JWT authentication via Authentik OIDC.
```bash
# Get token from Authentik, then:
curl -H "Authorization: Bearer $TOKEN" http://localhost:5001/api/listing
```
## Project Structure
```
├── main.py # CLI entry point
├── api/app.py # FastAPI application
├── services/ # Business logic (shared by CLI + API)
│ ├── listing_service.py
│ ├── export_service.py
│ ├── district_service.py
│ └── task_service.py
├── repositories/ # Database access
├── models/ # SQLModel entities
├── rec/ # Core logic (query, OCR, routing)
├── tasks/ # Celery background tasks
└── tests/ # Test suite
```
## Running Tests
```bash
pytest tests/ -v --cov=.
mypy .
```

4
crawler/frontend/.dockerignore
View file

@ -1,4 +0,0 @@
node_modules
.git
.env.local
*.md

7
crawler/frontend/.env.sample
View file

@ -1,7 +0,0 @@
# IMPORTANT: This is the URL you will use to access your app via https.
# Make sure it resolves to your dev server's IP address or hostname.
# Also it must be an authorized URL in your OIDC provider (reach out to viktor to configure that).
export DEV_HOST="<CHANGE ME>"
export FRONTEND_SERVICE="$DEV_HOST:5173" # react app server
export BACKEND_SERVICE="$DEV_HOST:5001" # where the backend api is running

24
crawler/frontend/.gitignore vendored
View file

@ -1,24 +0,0 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
*.local
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

11
crawler/frontend/Caddyfile.dev
View file

@ -1,11 +0,0 @@
{$DEV_HOST:localhost}:443 {
tls internal
handle /api/* {
reverse_proxy app:5001
}
handle {
reverse_proxy frontend:5173
}
}

29
crawler/frontend/Dockerfile
View file

@ -1,29 +0,0 @@
# Stage 1: Build the React app
FROM node:24-alpine AS builder
WORKDIR /app
# Copy package files first for better caching
COPY package.json package-lock.json* ./
# Install dependencies (prefers yarn if available)
RUN npm ci
# Copy all files and build
COPY . .
RUN npm run build
FROM nginx:alpine
# Remove default nginx static files
RUN rm -rf /usr/share/nginx/html/*
WORKDIR /app
# Copy only necessary files from the builder stage
COPY --from=builder /app/dist /usr/share/nginx/html
COPY --from=builder /app/nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

54
crawler/frontend/README.md
View file

@ -1,54 +0,0 @@
# React + TypeScript + Vite
This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.
Currently, two official plugins are available:
- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react) uses [Babel](https://babeljs.io/) for Fast Refresh
- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react-swc) uses [SWC](https://swc.rs/) for Fast Refresh
## Expanding the ESLint configuration
If you are developing a production application, we recommend updating the configuration to enable type-aware lint rules:
```js
export default tseslint.config({
extends: [
// Remove ...tseslint.configs.recommended and replace with this
...tseslint.configs.recommendedTypeChecked,
// Alternatively, use this for stricter rules
...tseslint.configs.strictTypeChecked,
// Optionally, add this for stylistic rules
...tseslint.configs.stylisticTypeChecked,
],
languageOptions: {
// other options...
parserOptions: {
project: ['./tsconfig.node.json', './tsconfig.app.json'],
tsconfigRootDir: import.meta.dirname,
},
},
})
```
You can also install [eslint-plugin-react-x](https://github.com/Rel1cx/eslint-react/tree/main/packages/plugins/eslint-plugin-react-x) and [eslint-plugin-react-dom](https://github.com/Rel1cx/eslint-react/tree/main/packages/plugins/eslint-plugin-react-dom) for React-specific lint rules:
```js
// eslint.config.js
import reactX from 'eslint-plugin-react-x'
import reactDom from 'eslint-plugin-react-dom'
export default tseslint.config({
plugins: {
// Add the react-x and react-dom plugins
'react-x': reactX,
'react-dom': reactDom,
},
rules: {
// other rules...
// Enable its recommended typescript rules
...reactX.configs['recommended-typescript'].rules,
...reactDom.configs.recommended.rules,
},
})
```

21
crawler/frontend/components.json
View file

@ -1,21 +0,0 @@
{
"$schema": "https://ui.shadcn.com/schema.json",
"style": "new-york",
"rsc": false,
"tsx": true,
"tailwind": {
"config": "",
"css": "src/index.css",
"baseColor": "neutral",
"cssVariables": true,
"prefix": ""
},
"aliases": {
"components": "@/components",
"utils": "@/lib/utils",
"ui": "@/components/ui",
"lib": "@/lib",
"hooks": "@/hooks"
},
"iconLibrary": "lucide"
}

28
crawler/frontend/eslint.config.js
View file

@ -1,28 +0,0 @@
import js from '@eslint/js'
import globals from 'globals'
import reactHooks from 'eslint-plugin-react-hooks'
import reactRefresh from 'eslint-plugin-react-refresh'
import tseslint from 'typescript-eslint'
export default tseslint.config(
{ ignores: ['dist'] },
{
extends: [js.configs.recommended, ...tseslint.configs.recommended],
files: ['**/*.{ts,tsx}'],
languageOptions: {
ecmaVersion: 2020,
globals: globals.browser,
},
plugins: {
'react-hooks': reactHooks,
'react-refresh': reactRefresh,
},
rules: {
...reactHooks.configs.recommended.rules,
'react-refresh/only-export-components': [
'warn',
{ allowConstantExport: true },
],
},
},
)

17
crawler/frontend/index.html
View file

@ -1,17 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/vite.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Wrongmove</title>
<script src="/HexgridHeatmap.js"> </script>
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>

23
crawler/frontend/nginx.conf
View file

@ -1,23 +0,0 @@
server {
listen 80;
server_name _;
# Root directory for static files (must match Docker COPY path)
root /usr/share/nginx/html;
index index.html;
# Serve static files
location / {
try_files $uri $uri/ /index.html;
}
# Enable gzip compression
gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
# Cache static assets
# location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff2)$ {
# expires 1y;
# add_header Cache-Control "public, immutable";
# }
}

6291
crawler/frontend/package-lock.json generated
View file

File diff suppressed because it is too large Load diff

70
crawler/frontend/package.json
View file

@ -1,70 +0,0 @@
{
"name": "wrongmove",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "tsc -b && vite build",
"lint": "eslint .",
"preview": "vite preview"
},
"dependencies": {
"@hookform/resolvers": "^5.1.1",
"@radix-ui/react-accordion": "^1.2.12",
"@radix-ui/react-alert-dialog": "^1.1.14",
"@radix-ui/react-checkbox": "^1.3.3",
"@radix-ui/react-dialog": "^1.1.14",
"@radix-ui/react-hover-card": "^1.1.14",
"@radix-ui/react-label": "^2.1.7",
"@radix-ui/react-popover": "^1.1.14",
"@radix-ui/react-progress": "^1.1.7",
"@radix-ui/react-scroll-area": "^1.2.9",
"@radix-ui/react-select": "^2.2.5",
"@radix-ui/react-separator": "^1.1.7",
"@radix-ui/react-slider": "^1.3.6",
"@radix-ui/react-slot": "^1.2.3",
"@radix-ui/react-tabs": "^1.1.13",
"@radix-ui/react-tooltip": "^1.2.7",
"@simplewebauthn/browser": "^10.0.0",
"@tabler/icons-react": "^3.34.0",
"@tailwindcss/vite": "^4.1.10",
"@types/crossfilter": "^0.0.38",
"@types/d3": "^7.4.3",
"@types/mapbox-gl": "^3.4.1",
"@types/turf": "^3.5.32",
"chrono-node": "^2.8.3",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"crossfilter2": "^1.5.4",
"d3": "^7.9.0",
"date-fns": "^4.1.0",
"lucide-react": "^0.515.0",
"mapbox-gl": "^3.12.0",
"oidc-client-ts": "^3.2.1",
"react": "^19.1.0",
"react-day-picker": "^9.7.0",
"react-dom": "^19.1.0",
"react-hook-form": "^7.58.1",
"react-oidc-context": "^3.3.0",
"react-virtuoso": "^4.18.1",
"tailwind-merge": "^3.3.1",
"tailwindcss": "^4.1.10",
"zod": "^3.25.67"
},
"devDependencies": {
"@eslint/js": "^9.25.0",
"@types/node": "^24.0.1",
"@types/react": "^19.1.2",
"@types/react-dom": "^19.1.2",
"@vitejs/plugin-react-swc": "^3.9.0",
"eslint": "^9.25.0",
"eslint-plugin-react-hooks": "^5.2.0",
"eslint-plugin-react-refresh": "^0.4.19",
"globals": "^16.0.0",
"tw-animate-css": "^1.3.4",
"typescript": "~5.8.3",
"typescript-eslint": "^8.30.1",
"vite": "^6.3.5"
}
}

2491
crawler/frontend/public/HexgridHeatmap.js
View file

File diff suppressed because it is too large Load diff

4
crawler/frontend/public/robots.txt
View file

@ -1,4 +0,0 @@
# Block ALL bots from ALL pages
User-agent: *
Disallow: /

1
crawler/frontend/public/vite.svg
View file

@ -1 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" class="iconify iconify--logos" width="31.88" height="32" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 257"><defs><linearGradient id="IconifyId1813088fe1fbc01fb466" x1="-.828%" x2="57.636%" y1="7.652%" y2="78.411%"><stop offset="0%" stop-color="#41D1FF"></stop><stop offset="100%" stop-color="#BD34FE"></stop></linearGradient><linearGradient id="IconifyId1813088fe1fbc01fb467" x1="43.376%" x2="50.316%" y1="2.242%" y2="89.03%"><stop offset="0%" stop-color="#FFEA83"></stop><stop offset="8.333%" stop-color="#FFDD35"></stop><stop offset="100%" stop-color="#FFA800"></stop></linearGradient></defs><path fill="url(#IconifyId1813088fe1fbc01fb466)" d="M255.153 37.938L134.897 252.976c-2.483 4.44-8.862 4.466-11.382.048L.875 37.958c-2.746-4.814 1.371-10.646 6.827-9.67l120.385 21.517a6.537 6.537 0 0 0 2.322-.004l117.867-21.483c5.438-.991 9.574 4.796 6.877 9.62Z"></path><path fill="url(#IconifyId1813088fe1fbc01fb467)" d="M185.432.063L96.44 17.501a3.268 3.268 0 0 0-2.634 3.014l-5.474 92.456a3.268 3.268 0 0 0 3.997 3.378l24.777-5.718c2.318-.535 4.413 1.507 3.936 3.838l-7.361 36.047c-.495 2.426 1.782 4.5 4.151 3.78l15.304-4.649c2.372-.72 4.652 1.36 4.15 3.788l-11.698 56.621c-.732 3.542 3.979 5.473 5.943 2.437l1.313-2.028l72.516-144.72c1.215-2.423-.88-5.186-3.54-4.672l-25.505 4.922c-2.396.462-4.435-1.77-3.759-4.114l16.646-57.705c.677-2.35-1.37-4.583-3.769-4.113Z"></path></svg>
Side by side

Before

Width:  |  Height:  |  Size: 1.5 KiB

14
crawler/frontend/src/App.css
View file

@ -1,14 +0,0 @@
#root {
margin: 0;
padding: 0;
height: 100%;
overflow: hidden;
}
html,
body {
overflow: hidden;
height: 100%;
margin: 0;
padding: 0;
}

317
crawler/frontend/src/App.tsx
View file

@ -1,317 +0,0 @@
import { useEffect, useState, useRef, useCallback } from 'react';
import './App.css';
import { getUser } from './auth/authService';
import { getStoredPasskeyUser } from './auth/passkeyService';
import { fromOidcUser, type AuthUser } from './auth/types';
import AlertError from './components/AlertError';
import LoginModal from './components/LoginModal';
import AuthCallback from './components/AuthCallback';
import { Map } from './components/Map';
import { FilterPanel, type ParameterValues, DEFAULT_FILTER_VALUES, Metric } from './components/FilterPanel';
import { Header } from './components/Header';
import { StatsBar, type ViewMode } from './components/StatsBar';
import { ListView } from './components/ListView';
import { StreamingProgressBar } from './components/StreamingProgressBar';
import { Sheet, SheetContent, SheetTrigger } from './components/ui/sheet';
import { Button } from './components/ui/button';
import { Filter } from 'lucide-react';
import type { GeoJSONFeatureCollection, PropertyProperties, PropertyFeature } from '@/types';
import { refreshListings, fetchTasksForUser, streamListingGeoJSON, type StreamingProgress } from '@/services';
function App() {
const [listingData, setListingData] = useState<GeoJSONFeatureCollection | null>(null);
const [taskID, setTaskID] = useState<string | null>(null);
const [user, setUser] = useState<AuthUser | null>(null);
const [queryParameters, setQueryParameters] = useState<ParameterValues | null>(null);
const [submitError, setSubmitError] = useState<string | null>(null);
const [alertDialogIsOpen, setAlertDialogIsOpen] = useState(false);
const [isLoading, setIsLoading] = useState(false);
const [viewMode, setViewMode] = useState<ViewMode>('map');
const [mobileFilterOpen, setMobileFilterOpen] = useState(false);
const [highlightedProperty, setHighlightedProperty] = useState<string | null>(null);
const [streamingProgress, setStreamingProgress] = useState<StreamingProgress | null>(null);
// Ref to track accumulated features during streaming
const accumulatedFeaturesRef = useRef<PropertyFeature[]>([]);
// Ref to track if initial load has been triggered
const initialLoadTriggeredRef = useRef(false);
// Check if this is the callback route - render dedicated component
if (window.location.pathname === '/callback') {
return <AuthCallback />;
}
useEffect(() => {
// Check passkey user first, then fall back to OIDC
const passkeyUser = getStoredPasskeyUser();
if (passkeyUser) {
setUser(passkeyUser);
} else {
getUser().then((oidcUser) => {
if (oidcUser) {
setUser(fromOidcUser(oidcUser));
}
});
}
}, []);
const handlePasskeyLogin = (passkeyUser: AuthUser) => {
setUser(passkeyUser);
};
useEffect(() => {
if (!user) {
return;
}
fetchTasksForUser(user).then((tasks) => {
if (tasks && tasks.length > 0) {
setTaskID(tasks[0]);
}
});
}, [user, taskID]);
// Load listings function - used by both auto-load and manual submit
const loadListings = useCallback(async (parameters: ParameterValues) => {
if (!user) return;
setQueryParameters(parameters);
setMobileFilterOpen(false);
setIsLoading(true);
accumulatedFeaturesRef.current = [];
setStreamingProgress({ count: 0 });
setListingData(null);
let updateScheduled = false;
const flushUpdate = () => {
updateScheduled = false;
setListingData({
type: 'FeatureCollection',
features: [...accumulatedFeaturesRef.current]
});
};
const scheduleUpdate = () => {
if (!updateScheduled) {
updateScheduled = true;
requestAnimationFrame(flushUpdate);
}
};
try {
for await (const batch of streamListingGeoJSON(user, parameters, (progress) => {
setStreamingProgress(progress);
})) {
accumulatedFeaturesRef.current.push(...batch);
scheduleUpdate();
}
// Final flush to ensure all data is rendered
flushUpdate();
} catch (error) {
if (error instanceof Error) {
setSubmitError(error.message);
} else {
setSubmitError(String(error));
}
setAlertDialogIsOpen(true);
} finally {
setIsLoading(false);
setStreamingProgress(null);
}
}, [user]);
// Auto-load data with default filters when user is authenticated
useEffect(() => {
if (!user || initialLoadTriggeredRef.current) {
return;
}
initialLoadTriggeredRef.current = true;
const defaultParams: ParameterValues = {
...DEFAULT_FILTER_VALUES,
available_from: new Date(),
};
loadListings(defaultParams);
}, [user, loadListings]);
if (!user) {
return <LoginModal isOpen={user === null} onPasskeyLogin={handlePasskeyLogin} />;
}
const onSubmit = async (action: 'fetch-data' | 'visualize', parameters: ParameterValues) => {
if (action === 'visualize') {
loadListings(parameters);
} else if (action === 'fetch-data') {
setQueryParameters(parameters);
setMobileFilterOpen(false);
setIsLoading(true);
try {
const data = await refreshListings(user!, parameters);
setTaskID(data.task_id);
} catch (error) {
if (error instanceof Error) {
setSubmitError(error.message);
} else {
setSubmitError(String(error));
}
setAlertDialogIsOpen(true);
} finally {
setIsLoading(false);
}
}
};
const handleMetricChange = (metric: Metric) => {
setQueryParameters(prev => prev ? { ...prev, metric } : null);
};
const handlePropertyClick = (property: PropertyProperties, _coordinates: [number, number]) => {
setHighlightedProperty(property.url);
// Optionally: pan map to coordinates
};
const renderMainContent = () => {
if (!listingData) {
return (
<div className="flex-1 flex items-center justify-center bg-muted/20">
<div className="text-center p-8 max-w-md">
{isLoading ? (
<>
<div className="text-6xl mb-4 animate-pulse">🏠</div>
<h2 className="text-xl font-semibold mb-2">Loading Properties...</h2>
<p className="text-muted-foreground mb-4">
Fetching listings with default filters. You can adjust filters on the left.
</p>
</>
) : (
<>
<div className="text-6xl mb-4">🏠</div>
<h2 className="text-xl font-semibold mb-2">Welcome to Property Explorer</h2>
<p className="text-muted-foreground mb-4">
Use the filters on the left to find properties. Apply filters to visualize existing data or refresh to fetch new listings.
</p>
</>
)}
</div>
</div>
);
}
if (listingData.features.length === 0) {
return (
<div className="flex-1 flex items-center justify-center">
<div className="text-center p-8">
<div className="text-6xl mb-4">🔍</div>
<h2 className="text-xl font-semibold mb-2">No listings found</h2>
<p className="text-muted-foreground">
Try adjusting the filters or run a data refresh to fetch new listings.
</p>
</div>
</div>
);
}
return (
<>
{/* Map View */}
{(viewMode === 'map' || viewMode === 'split') && (
<div className={`relative ${viewMode === 'split' ? 'w-1/2' : 'flex-1'}`} style={{ minHeight: 0 }}>
<Map
listingData={listingData}
queryParameters={queryParameters}
onPropertyClick={handlePropertyClick}
/>
</div>
)}
{/* List View */}
{(viewMode === 'list' || viewMode === 'split') && (
<div className={`${viewMode === 'split' ? 'w-1/2 border-l' : 'flex-1'}`}>
<ListView
listingData={listingData}
onPropertyClick={handlePropertyClick}
highlightedPropertyUrl={highlightedProperty}
/>
</div>
)}
</>
);
};
const handleTaskCancelled = () => {
setTaskID(null);
};
return (
<div className="h-screen flex flex-col overflow-hidden">
{/* Header */}
<Header
user={user}
taskID={taskID}
onTaskCancelled={handleTaskCancelled}
/>
{/* Main content area */}
<div className="flex-1 flex overflow-hidden min-h-0">
{/* Filter Panel - Desktop (fixed sidebar) */}
<div className="hidden md:block w-64 shrink-0 h-full overflow-hidden">
<FilterPanel
onSubmit={onSubmit}
onMetricChange={handleMetricChange}
isLoading={isLoading}
listingCount={listingData?.features.length}
/>
</div>
{/* Filter Panel - Mobile (sheet) */}
<div className="md:hidden fixed bottom-4 right-4 z-50">
<Sheet open={mobileFilterOpen} onOpenChange={setMobileFilterOpen}>
<SheetTrigger asChild>
<Button size="lg" className="rounded-full shadow-lg h-14 w-14">
<Filter className="h-6 w-6" />
</Button>
</SheetTrigger>
<SheetContent side="left" className="w-80 p-0">
<FilterPanel
onSubmit={onSubmit}
onMetricChange={handleMetricChange}
isLoading={isLoading}
listingCount={listingData?.features.length}
/>
</SheetContent>
</Sheet>
</div>
{/* Main View Area */}
<div className="flex-1 flex flex-col overflow-hidden min-h-0">
{/* Streaming Progress Bar */}
<div className="relative shrink-0">
<StreamingProgressBar progress={streamingProgress} isLoading={isLoading} />
</div>
{/* Map/List Container */}
<div className="flex-1 flex overflow-hidden min-h-0">
{renderMainContent()}
</div>
{/* Stats Bar */}
{listingData && listingData.features.length > 0 && (
<div className="shrink-0">
<StatsBar
listingData={listingData}
viewMode={viewMode}
onViewModeChange={setViewMode}
/>
</div>
)}
</div>
</div>
{/* Error Dialog */}
<AlertError message={submitError} open={alertDialogIsOpen} setIsOpen={setAlertDialogIsOpen} />
</div>
);
}
export default App;

89
crawler/frontend/src/AppSidebar.tsx
View file

@ -1,89 +0,0 @@
import {
Sidebar,
SidebarContent,
SidebarGroup,
SidebarGroupContent,
SidebarGroupLabel,
SidebarHeader,
SidebarMenu,
SidebarMenuButton,
SidebarMenuItem,
SidebarRail,
} from "@/components/ui/sidebar"
import * as React from "react"
const data = {
navMain: [
{
title: "Property Explorer",
url: "#",
items: [
{
title: "Map View",
url: "#",
isActive: true,
},
{
title: "List View",
url: "#",
},
],
},
{
title: "Data Management",
url: "#",
items: [
{
title: "Refresh Listings",
url: "#",
},
{
title: "Active Tasks",
url: "#",
},
],
},
{
title: "Settings",
url: "#",
items: [
{
title: "Preferences",
url: "#",
},
{
title: "Account",
url: "#",
},
],
},
],
}
export function AppSidebar({ ...props }: React.ComponentProps<typeof Sidebar>) {
return (
<Sidebar {...props}>
<SidebarHeader>
</SidebarHeader>
<SidebarContent>
{data.navMain.map((item) => (
<SidebarGroup key={item.title}>
<SidebarGroupLabel>{item.title}</SidebarGroupLabel>
<SidebarGroupContent>
<SidebarMenu>
{item.items.map((subItem) => (
<SidebarMenuItem key={subItem.title}>
<SidebarMenuButton asChild isActive={subItem.isActive}>
<a href={subItem.url}>{subItem.title}</a>
</SidebarMenuButton>
</SidebarMenuItem>
))}
</SidebarMenu>
</SidebarGroupContent>
</SidebarGroup>
))}
</SidebarContent>
<SidebarRail />
</Sidebar>
)
}

80
crawler/frontend/src/assets/Map.css
View file

@ -1,80 +0,0 @@
#map-container {
position: absolute;
top: 0;
bottom: 0;
right: 0;
left: 0;
width: 100%;
height: 100%;
}
#legend {
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.15);
line-height: 18px;
height: auto;
min-height: 300px;
width: 90px;
padding: 12px;
position: absolute;
top: 10px;
right: 10px;
background: rgba(255, 255, 255, 0.95);
border-radius: 8px;
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
}
#legend .axis path,
#legend .axis line {
stroke: #e5e7eb;
}
#legend .axis text {
fill: #6b7280;
}
.propertyListingPopupItem {
display: flex;
box-sizing: border-box;
justify-content: center;
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
padding: 6px;
width: 50%;
}
/* Mapbox popup styling improvements */
.mapboxgl-popup-content {
padding: 0 !important;
border-radius: 8px !important;
overflow: hidden;
box-shadow: 0 4px 20px rgba(0, 0, 0, 0.15) !important;
}
.mapboxgl-popup-close-button {
font-size: 20px;
padding: 4px 8px;
color: #6b7280;
z-index: 10;
}
.mapboxgl-popup-close-button:hover {
background-color: #f3f4f6;
color: #111827;
}
/* Improve marker visibility */
.mapboxgl-marker {
cursor: pointer;
}
/* Mobile adjustments */
@media (max-width: 768px) {
#legend {
width: 75px;
padding: 8px;
min-height: 250px;
}
.mapboxgl-popup-content {
max-width: 90vw !important;
}
}

1
crawler/frontend/src/assets/react.svg
View file

@ -1 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" class="iconify iconify--logos" width="35.93" height="32" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 228"><path fill="#00D8FF" d="M210.483 73.824a171.49 171.49 0 0 0-8.24-2.597c.465-1.9.893-3.777 1.273-5.621c6.238-30.281 2.16-54.676-11.769-62.708c-13.355-7.7-35.196.329-57.254 19.526a171.23 171.23 0 0 0-6.375 5.848a155.866 155.866 0 0 0-4.241-3.917C100.759 3.829 77.587-4.822 63.673 3.233C50.33 10.957 46.379 33.89 51.995 62.588a170.974 170.974 0 0 0 1.892 8.48c-3.28.932-6.445 1.924-9.474 2.98C17.309 83.498 0 98.307 0 113.668c0 15.865 18.582 31.778 46.812 41.427a145.52 145.52 0 0 0 6.921 2.165a167.467 167.467 0 0 0-2.01 9.138c-5.354 28.2-1.173 50.591 12.134 58.266c13.744 7.926 36.812-.22 59.273-19.855a145.567 145.567 0 0 0 5.342-4.923a168.064 168.064 0 0 0 6.92 6.314c21.758 18.722 43.246 26.282 56.54 18.586c13.731-7.949 18.194-32.003 12.4-61.268a145.016 145.016 0 0 0-1.535-6.842c1.62-.48 3.21-.974 4.76-1.488c29.348-9.723 48.443-25.443 48.443-41.52c0-15.417-17.868-30.326-45.517-39.844Zm-6.365 70.984c-1.4.463-2.836.91-4.3 1.345c-3.24-10.257-7.612-21.163-12.963-32.432c5.106-11 9.31-21.767 12.459-31.957c2.619.758 5.16 1.557 7.61 2.4c23.69 8.156 38.14 20.213 38.14 29.504c0 9.896-15.606 22.743-40.946 31.14Zm-10.514 20.834c2.562 12.94 2.927 24.64 1.23 33.787c-1.524 8.219-4.59 13.698-8.382 15.893c-8.067 4.67-25.32-1.4-43.927-17.412a156.726 156.726 0 0 1-6.437-5.87c7.214-7.889 14.423-17.06 21.459-27.246c12.376-1.098 24.068-2.894 34.671-5.345a134.17 134.17 0 0 1 1.386 6.193ZM87.276 214.515c-7.882 2.783-14.16 2.863-17.955.675c-8.075-4.657-11.432-22.636-6.853-46.752a156.923 156.923 0 0 1 1.869-8.499c10.486 2.32 22.093 3.988 34.498 4.994c7.084 9.967 14.501 19.128 21.976 27.15a134.668 134.668 0 0 1-4.877 4.492c-9.933 8.682-19.886 14.842-28.658 17.94ZM50.35 144.747c-12.483-4.267-22.792-9.812-29.858-15.863c-6.35-5.437-9.555-10.836-9.555-15.216c0-9.322 13.897-21.212 37.076-29.293c2.813-.98 5.757-1.905 8.812-2.773c3.204 10.42 7.406 21.315 12.477 32.332c-5.137 11.18-9.399 22.249-12.634 32.792a134.718 134.718 0 0 1-6.318-1.979Zm12.378-84.26c-4.811-24.587-1.616-43.134 6.425-47.789c8.564-4.958 27.502 2.111 47.463 19.835a144.318 144.318 0 0 1 3.841 3.545c-7.438 7.987-14.787 17.08-21.808 26.988c-12.04 1.116-23.565 2.908-34.161 5.309a160.342 160.342 0 0 1-1.76-7.887Zm110.427 27.268a347.8 347.8 0 0 0-7.785-12.803c8.168 1.033 15.994 2.404 23.343 4.08c-2.206 7.072-4.956 14.465-8.193 22.045a381.151 381.151 0 0 0-7.365-13.322Zm-45.032-43.861c5.044 5.465 10.096 11.566 15.065 18.186a322.04 322.04 0 0 0-30.257-.006c4.974-6.559 10.069-12.652 15.192-18.18ZM82.802 87.83a323.167 323.167 0 0 0-7.227 13.238c-3.184-7.553-5.909-14.98-8.134-22.152c7.304-1.634 15.093-2.97 23.209-3.984a321.524 321.524 0 0 0-7.848 12.897Zm8.081 65.352c-8.385-.936-16.291-2.203-23.593-3.793c2.26-7.3 5.045-14.885 8.298-22.6a321.187 321.187 0 0 0 7.257 13.246c2.594 4.48 5.28 8.868 8.038 13.147Zm37.542 31.03c-5.184-5.592-10.354-11.779-15.403-18.433c4.902.192 9.899.29 14.978.29c5.218 0 10.376-.117 15.453-.343c-4.985 6.774-10.018 12.97-15.028 18.486Zm52.198-57.817c3.422 7.8 6.306 15.345 8.596 22.52c-7.422 1.694-15.436 3.058-23.88 4.071a382.417 382.417 0 0 0 7.859-13.026a347.403 347.403 0 0 0 7.425-13.565Zm-16.898 8.101a358.557 358.557 0 0 1-12.281 19.815a329.4 329.4 0 0 1-23.444.823c-7.967 0-15.716-.248-23.178-.732a310.202 310.202 0 0 1-12.513-19.846h.001a307.41 307.41 0 0 1-10.923-20.627a310.278 310.278 0 0 1 10.89-20.637l-.001.001a307.318 307.318 0 0 1 12.413-19.761c7.613-.576 15.42-.876 23.31-.876H128c7.926 0 15.743.303 23.354.883a329.357 329.357 0 0 1 12.335 19.695a358.489 358.489 0 0 1 11.036 20.54a329.472 329.472 0 0 1-11 20.722Zm22.56-122.124c8.572 4.944 11.906 24.881 6.52 51.026c-.344 1.668-.73 3.367-1.15 5.09c-10.622-2.452-22.155-4.275-34.23-5.408c-7.034-10.017-14.323-19.124-21.64-27.008a160.789 160.789 0 0 1 5.888-5.4c18.9-16.447 36.564-22.941 44.612-18.3ZM128 90.808c12.625 0 22.86 10.235 22.86 22.86s-10.235 22.86-22.86 22.86s-22.86-10.235-22.86-22.86s10.235-22.86 22.86-22.86Z"></path></svg>
Side by side

Before

Width:  |  Height:  |  Size: 4 KiB

45
crawler/frontend/src/auth/authService.ts
View file

@ -1,45 +0,0 @@
import { User, UserManager } from 'oidc-client-ts';
import { oidcConfig } from './config';
import { parseOidcError, type AuthError } from './errors';
const userManager = new UserManager(oidcConfig);
export const login = async (): Promise<void> => {
try {
await userManager.signinRedirect();
} catch (error) {
console.error('Login redirect failed:', error);
throw parseOidcError(error);
}
};
export const logout = async (): Promise<void> => {
try {
await userManager.signoutRedirect();
} catch (error) {
console.error('Logout redirect failed:', error);
throw parseOidcError(error);
}
};
export const handleCallback = async (): Promise<User> => {
try {
const user = await userManager.signinRedirectCallback();
return user;
} catch (error) {
console.error('Callback handling failed:', error);
throw parseOidcError(error);
}
};
export const getUser = async (): Promise<User | null> => {
try {
const user = await userManager.getUser();
return user;
} catch (error) {
console.error('Error fetching user:', error);
return null;
}
};
export type { AuthError };

14
crawler/frontend/src/auth/config.ts
View file

@ -1,14 +0,0 @@
import { WebStorageStateStore } from "oidc-client-ts";
export const oidcConfig = {
authority: "https://authentik.viktorbarzin.me/application/o/wrongmove/",
client_id: "5AJKRgcdgVm1OyApBzFkadDFfStW9a555zwv2MOe",
redirect_uri: import.meta.env.MODE === 'development' ? "https://localhost/callback" : "https://wrongmove.viktorbarzin.me/callback",
post_logout_redirect_uri: import.meta.env.MODE === 'development' ? "https://localhost/" : "https://wrongmove.viktorbarzin.me/",
userStore: new WebStorageStateStore({ store: window.localStorage }),
response_type: 'code', // PKCE flow (recommended for SPAs)
scope: 'openid profile email', // Requested scopes
automaticSilentRenew: true, // Renew tokens silently
filterProtocolClaims: true,
loadUserInfo: true,
};

60
crawler/frontend/src/auth/errors.ts
View file

@ -1,60 +0,0 @@
export enum AuthErrorType {
REDIRECT_FAILED = 'REDIRECT_FAILED',
CALLBACK_FAILED = 'CALLBACK_FAILED',
NETWORK_ERROR = 'NETWORK_ERROR',
USER_CANCELLED = 'USER_CANCELLED',
}
export interface AuthError {
type: AuthErrorType;
message: string;
retryable: boolean;
}
export function parseOidcError(error: unknown): AuthError {
const errorMessage = error instanceof Error ? error.message : String(error);
const errorString = errorMessage.toLowerCase();
// Check for popup/redirect blocked errors
if (errorString.includes('popup') || errorString.includes('blocked') || errorString.includes('window')) {
return {
type: AuthErrorType.REDIRECT_FAILED,
message: 'Unable to redirect. Please check if popups are blocked.',
retryable: true,
};
}
// Check for user cancellation
if (errorString.includes('cancel') || errorString.includes('closed') || errorString.includes('denied')) {
return {
type: AuthErrorType.USER_CANCELLED,
message: 'Sign in was cancelled.',
retryable: true,
};
}
// Check for network errors
if (errorString.includes('network') || errorString.includes('fetch') || errorString.includes('timeout') || errorString.includes('failed to fetch')) {
return {
type: AuthErrorType.NETWORK_ERROR,
message: 'Unable to reach authentication server. Please check your connection.',
retryable: true,
};
}
// Check for callback/state errors
if (errorString.includes('state') || errorString.includes('invalid') || errorString.includes('mismatch') || errorString.includes('no matching state')) {
return {
type: AuthErrorType.CALLBACK_FAILED,
message: 'Login verification failed. Please try again.',
retryable: true,
};
}
// Default error
return {
type: AuthErrorType.CALLBACK_FAILED,
message: errorMessage || 'An unexpected error occurred during sign in.',
retryable: true,
};
}

155
crawler/frontend/src/auth/passkeyService.ts
View file

@ -1,155 +0,0 @@
import { startRegistration, startAuthentication } from '@simplewebauthn/browser';
import type { AuthUser } from './types';
const PASSKEY_USER_KEY = 'passkey_user';
interface RegisterBeginResponse {
options: PublicKeyCredentialCreationOptionsJSON;
session_id: string;
}
interface LoginBeginResponse {
options: PublicKeyCredentialRequestOptionsJSON;
session_id: string;
}
interface AuthTokenResponse {
token: string;
}
// WebAuthn JSON types from the spec
type PublicKeyCredentialCreationOptionsJSON = Parameters<typeof startRegistration>[0];
type PublicKeyCredentialRequestOptionsJSON = Parameters<typeof startAuthentication>[0];
function parseJwt(token: string): Record<string, unknown> {
const base64Url = token.split('.')[1];
const base64 = base64Url.replace(/-/g, '+').replace(/_/g, '/');
const jsonPayload = decodeURIComponent(
atob(base64)
.split('')
.map((c) => '%' + ('00' + c.charCodeAt(0).toString(16)).slice(-2))
.join('')
);
return JSON.parse(jsonPayload);
}
export async function registerPasskey(email: string): Promise<AuthUser> {
// Step 1: Begin registration
const beginRes = await fetch('/api/passkey/register/begin', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ email }),
});
if (!beginRes.ok) {
const err = await beginRes.json();
throw new Error(err.detail || 'Failed to start registration');
}
const beginData: RegisterBeginResponse = await beginRes.json();
// Step 2: Browser WebAuthn ceremony
const attResp = await startRegistration(beginData.options);
// Step 3: Complete registration
const completeRes = await fetch('/api/passkey/register/complete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
session_id: beginData.session_id,
credential: attResp,
}),
});
if (!completeRes.ok) {
const err = await completeRes.json();
throw new Error(err.detail || 'Failed to complete registration');
}
const { token }: AuthTokenResponse = await completeRes.json();
const claims = parseJwt(token);
const user: AuthUser = {
sub: claims.sub as string,
email: claims.email as string,
name: (claims.name as string) || (claims.email as string),
accessToken: token,
provider: 'passkey',
};
localStorage.setItem(PASSKEY_USER_KEY, JSON.stringify(user));
return user;
}
export async function loginWithPasskey(): Promise<AuthUser> {
// Step 1: Begin authentication
const beginRes = await fetch('/api/passkey/login/begin', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
});
if (!beginRes.ok) {
const err = await beginRes.json();
throw new Error(err.detail || 'Failed to start login');
}
const beginData: LoginBeginResponse = await beginRes.json();
// Step 2: Browser WebAuthn ceremony
const assertionResp = await startAuthentication(beginData.options);
// Step 3: Complete authentication
const completeRes = await fetch('/api/passkey/login/complete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
session_id: beginData.session_id,
credential: assertionResp,
}),
});
if (!completeRes.ok) {
const err = await completeRes.json();
throw new Error(err.detail || 'Failed to complete login');
}
const { token }: AuthTokenResponse = await completeRes.json();
const claims = parseJwt(token);
const user: AuthUser = {
sub: claims.sub as string,
email: claims.email as string,
name: (claims.name as string) || (claims.email as string),
accessToken: token,
provider: 'passkey',
};
localStorage.setItem(PASSKEY_USER_KEY, JSON.stringify(user));
return user;
}
export function getStoredPasskeyUser(): AuthUser | null {
const stored = localStorage.getItem(PASSKEY_USER_KEY);
if (!stored) return null;
try {
const user: AuthUser = JSON.parse(stored);
// Check JWT expiration
const claims = parseJwt(user.accessToken);
const exp = claims.exp as number | undefined;
if (exp && exp * 1000 < Date.now()) {
localStorage.removeItem(PASSKEY_USER_KEY);
return null;
}
return user;
} catch {
localStorage.removeItem(PASSKEY_USER_KEY);
return null;
}
}
export function clearPasskeyUser(): void {
localStorage.removeItem(PASSKEY_USER_KEY);
}

19
crawler/frontend/src/auth/types.ts
View file

@ -1,19 +0,0 @@
import type { User } from 'oidc-client-ts';
export interface AuthUser {
sub: string;
email: string;
name: string;
accessToken: string;
provider: 'oidc' | 'passkey';
}
export function fromOidcUser(user: User): AuthUser {
return {
sub: user.profile.sub,
email: user.profile.email ?? '',
name: user.profile.name ?? user.profile.email ?? '',
accessToken: user.access_token,
provider: 'oidc',
};
}

159
crawler/frontend/src/components/ActiveQuery.tsx
View file

@ -1,159 +0,0 @@
import { getUser } from '@/auth/authService';
import { getStoredPasskeyUser } from '@/auth/passkeyService';
import { fromOidcUser, type AuthUser } from '@/auth/types';
import { POLLING_INTERVALS } from '@/constants';
import { fetchTaskStatus, cancelTask } from '@/services';
import { TaskStatus, type TaskResult } from '@/types';
import React, { useEffect, useState } from 'react';
import AlertError from './AlertError';
import { Spinner } from './Spinner';
import { HoverCard, HoverCardContent, HoverCardTrigger } from './ui/hover-card';
import { Progress } from './ui/progress';
import { Button } from './ui/button';
import { X } from 'lucide-react';
interface ActiveQueryProps {
taskID: string | null;
onTaskCancelled?: () => void;
}
const ActiveQuery: React.FC<ActiveQueryProps> = ({ taskID, onTaskCancelled }) => {
const [user, setUser] = useState<AuthUser | null>(null);
useEffect(() => {
const passkeyUser = getStoredPasskeyUser();
if (passkeyUser) {
setUser(passkeyUser);
} else {
getUser().then((oidcUser) => {
if (oidcUser) setUser(fromOidcUser(oidcUser));
});
}
}, []);
const [progressPercentage, setProgressPercentage] = useState<number>(0);
const [taskStatus, setTaskStatus] = useState<TaskStatus | null>(TaskStatus.PENDING);
const [lastUpdateTime, setLastUpdateTime] = useState<Date>(new Date());
const [fetchStatusError, setFetchStatusError] = useState<string | null>(null);
const [alertDialogIsOpen, setAlertDialogIsOpen] = useState(false);
const [isCancelling, setIsCancelling] = useState(false);
const handleCancelTask = async () => {
if (!user || !taskID || isCancelling) return;
setIsCancelling(true);
try {
const result = await cancelTask(user, taskID);
if (result.success) {
setTaskStatus(TaskStatus.REVOKED);
onTaskCancelled?.();
} else {
setFetchStatusError(result.message);
setAlertDialogIsOpen(true);
}
} catch (error) {
setFetchStatusError(error instanceof Error ? error.message : 'Failed to cancel task');
setAlertDialogIsOpen(true);
} finally {
setIsCancelling(false);
}
};
const pollTaskStatus = async (interval: NodeJS.Timeout) => {
if (!user || !taskID) {
return;
}
try {
const data = await fetchTaskStatus(user, taskID);
setLastUpdateTime(new Date());
const status = data.status as TaskStatus;
setTaskStatus(status);
if (status === TaskStatus.FAILURE || status === TaskStatus.REVOKED) {
clearInterval(interval);
setFetchStatusError('Task failed with status: ' + status);
setAlertDialogIsOpen(true);
return;
}
if (status === TaskStatus.SUCCESS) {
clearInterval(interval);
setProgressPercentage(100);
return;
}
// Only parse result for in-progress tasks
if (data.result) {
try {
const parsedResult: TaskResult = JSON.parse(data.result);
setProgressPercentage(parsedResult.progress * 100);
} catch {
// Result parsing failed, but task is still running - ignore
}
}
} catch (error) {
clearInterval(interval);
setTaskStatus(TaskStatus.FAILURE);
setAlertDialogIsOpen(true);
if (error instanceof Error) {
setFetchStatusError(error.message);
} else {
setFetchStatusError('Failed to update task status: ' + String(error));
}
}
};
useEffect(() => {
const interval = setInterval(
() => pollTaskStatus(interval),
POLLING_INTERVALS.TASK_STATUS_MS
);
return () => clearInterval(interval);
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [taskID, user]);
if (!taskID) {
return null;
}
const isInProgress = taskStatus &&
taskStatus !== TaskStatus.SUCCESS &&
taskStatus !== TaskStatus.FAILURE &&
taskStatus !== TaskStatus.REVOKED;
return (
<>
<div className="flex items-center gap-2 p-2 border-t bg-muted/50">
<HoverCard>
<HoverCardTrigger className="flex-1">
<div className="flex items-center gap-2">
{taskStatus && <span className="text-sm">Task: {taskStatus}</span>}
{isInProgress && <Spinner />}
</div>
<Progress value={progressPercentage} className="mt-1" />
</HoverCardTrigger>
<HoverCardContent>
Task ID: {taskID}
<br />
Last updated: {lastUpdateTime.toLocaleString()}
</HoverCardContent>
</HoverCard>
{isInProgress && (
<Button
variant="ghost"
size="sm"
onClick={handleCancelTask}
disabled={isCancelling}
className="h-8 px-2 text-destructive hover:text-destructive hover:bg-destructive/10"
>
<X className="h-4 w-4" />
<span className="ml-1 hidden sm:inline">Cancel</span>
</Button>
)}
</div>
<AlertError message={fetchStatusError} open={alertDialogIsOpen} setIsOpen={setAlertDialogIsOpen} />
</>
);
};
export default ActiveQuery;

26
crawler/frontend/src/components/AlertError.tsx
View file

@ -1,26 +0,0 @@
import { AlertDialog, AlertDialogAction, AlertDialogContent, AlertDialogDescription, AlertDialogFooter, AlertDialogHeader, AlertDialogTitle } from "./ui/alert-dialog";
export default function AlertError(
props: {
message: string | null;
open: boolean;
setIsOpen: (isOpen: boolean) => void;
}
) {
return (
<AlertDialog open={props.open} onOpenChange={props.setIsOpen}>
{/* <AlertDialogTrigger>Open</AlertDialogTrigger> */}
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Error</AlertDialogTitle>
<AlertDialogDescription>
{props.message}
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogAction onClick={() => props.setIsOpen(false)}>Close</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
)
}

111
crawler/frontend/src/components/AuthCallback.tsx
View file

@ -1,111 +0,0 @@
import React, { useEffect, useState } from 'react';
import { handleCallback, login, type AuthError } from '@/auth/authService';
import { Loader2, CheckCircle, AlertCircle, Home } from 'lucide-react';
import { Button } from './ui/button';
type CallbackState = 'processing' | 'success' | 'error';
const AuthCallback: React.FC = () => {
const [state, setState] = useState<CallbackState>('processing');
const [error, setError] = useState<AuthError | null>(null);
useEffect(() => {
const processCallback = async () => {
try {
await handleCallback();
setState('success');
// Auto-redirect after success
setTimeout(() => {
window.location.href = '/';
}, 1500);
} catch (err) {
setError(err as AuthError);
setState('error');
}
};
processCallback();
}, []);
const handleRetry = async () => {
setState('processing');
setError(null);
try {
await login();
} catch (err) {
setError(err as AuthError);
setState('error');
}
};
const handleGoHome = () => {
window.location.href = '/';
};
return (
<div className="min-h-screen flex items-center justify-center bg-background p-4">
<div className="w-full max-w-md">
<div className="bg-card border rounded-xl shadow-lg p-8">
{state === 'processing' && (
<div className="text-center space-y-4">
<div className="flex justify-center">
<div className="p-4 bg-primary/10 rounded-full">
<Loader2 className="h-8 w-8 text-primary animate-spin" />
</div>
</div>
<div className="space-y-2">
<h1 className="text-xl font-semibold">Completing Sign In</h1>
<p className="text-muted-foreground">
Please wait while we verify your credentials...
</p>
</div>
</div>
)}
{state === 'success' && (
<div className="text-center space-y-4">
<div className="flex justify-center">
<div className="p-4 bg-green-500/10 rounded-full">
<CheckCircle className="h-8 w-8 text-green-500" />
</div>
</div>
<div className="space-y-2">
<h1 className="text-xl font-semibold">Welcome Back!</h1>
<p className="text-muted-foreground">
Redirecting you to the dashboard...
</p>
</div>
</div>
)}
{state === 'error' && (
<div className="text-center space-y-6">
<div className="flex justify-center">
<div className="p-4 bg-destructive/10 rounded-full">
<AlertCircle className="h-8 w-8 text-destructive" />
</div>
</div>
<div className="space-y-2">
<h1 className="text-xl font-semibold">Sign In Failed</h1>
<p className="text-muted-foreground">
{error?.message || 'An unexpected error occurred.'}
</p>
</div>
<div className="flex flex-col sm:flex-row gap-3 justify-center">
<Button onClick={handleRetry} className="gap-2">
Try Again
</Button>
<Button variant="outline" onClick={handleGoHome} className="gap-2">
<Home className="h-4 w-4" />
Go Home
</Button>
</div>
</div>
)}
</div>
</div>
</div>
);
};
export default AuthCallback;

569
crawler/frontend/src/components/FilterPanel.tsx
View file

@ -1,569 +0,0 @@
import { zodResolver } from "@hookform/resolvers/zod";
import { useState, useEffect } from "react";
import { useForm } from "react-hook-form";
import { z } from "zod";
import { Button } from "./ui/button";
import { Calendar29 } from "./ui/DatePicker";
import { Form, FormControl, FormDescription, FormField, FormItem, FormLabel, FormMessage } from "./ui/form";
import { Input } from "./ui/input";
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "./ui/select";
import { Accordion, AccordionContent, AccordionItem, AccordionTrigger } from "./ui/accordion";
import { Loader2, Filter, RefreshCw } from "lucide-react";
import { ScrollArea } from "./ui/scroll-area";
export enum Metric {
qmprice = 'qmprice',
rooms = 'rooms',
qm = 'qm',
price = 'total_price',
}
export enum ListingType {
RENT = 'RENT',
BUY = 'BUY'
}
export enum FurnishType {
FURNISHED = 'furnished',
PART_FURNISHED = 'partFurnished',
UNFURNISHED = 'unfurnished',
}
// Default filter values - exported so App.tsx can use them for initial load
export const DEFAULT_FILTER_VALUES = {
metric: Metric.qmprice,
listing_type: ListingType.RENT,
min_bedrooms: 1,
max_bedrooms: 3,
max_price: 3000,
min_price: 2000,
min_sqm: 50,
max_sqm: undefined,
min_price_per_sqm: undefined,
max_price_per_sqm: undefined,
last_seen_days: 28,
available_from: new Date(),
district: '',
furnish_types: [] as FurnishType[],
} as const;
export interface ParameterValues {
metric: Metric
listing_type: ListingType
min_bedrooms?: number
max_bedrooms?: number
min_price?: number
max_price?: number
min_sqm?: number
max_sqm?: number
min_price_per_sqm?: number
max_price_per_sqm?: number
last_seen_days?: number
available_from?: Date
district: string
furnish_types?: FurnishType[]
}
interface FilterPanelProps {
onSubmit: (action: 'fetch-data' | 'visualize', fromValues: ParameterValues) => void;
onMetricChange?: (metric: Metric) => void;
isLoading?: boolean;
listingCount?: number;
}
const formSchema = z.object({
metric: z.nativeEnum(Metric, { required_error: "Metric is required" }),
listing_type: z.nativeEnum(ListingType, { required_error: "Listing Type is required" }),
min_bedrooms: z.number().min(0).max(10).optional(),
max_bedrooms: z.number().min(0).max(10).optional(),
max_price: z.number().optional(),
min_price: z.number().min(0).optional(),
min_sqm: z.number().optional(),
max_sqm: z.number().optional(),
min_price_per_sqm: z.number().optional(),
max_price_per_sqm: z.number().optional(),
last_seen_days: z.number().min(0).optional(),
available_from: z.date(),
district: z.string(),
furnish_types: z.array(z.nativeEnum(FurnishType)).optional(),
});
type FormValues = z.infer<typeof formSchema>;
export function FilterPanel({ onSubmit, onMetricChange, isLoading, listingCount }: FilterPanelProps) {
const [availableFromRawInput, setAvailableFromRawInput] = useState("now");
const [selectedFurnishTypes, setSelectedFurnishTypes] = useState<FurnishType[]>([]);
const form = useForm<FormValues>({
resolver: zodResolver(formSchema),
defaultValues: {
...DEFAULT_FILTER_VALUES,
available_from: new Date(), // Fresh date on each render
},
});
// Watch listing_type to make filters type-aware
const watchedListingType = form.watch('listing_type');
// Update price defaults when listing type changes
useEffect(() => {
if (watchedListingType === ListingType.BUY) {
form.setValue('min_price', 300000);
form.setValue('max_price', 600000);
} else {
form.setValue('min_price', 2000);
form.setValue('max_price', 3000);
}
// Clear furnish types when switching to BUY
if (watchedListingType === ListingType.BUY) {
setSelectedFurnishTypes([]);
}
}, [watchedListingType, form]);
const handleFormSubmit = (action: 'fetch-data' | 'visualize') => {
return form.handleSubmit((values) => {
const params: ParameterValues = {
...values,
furnish_types: selectedFurnishTypes.length > 0 ? selectedFurnishTypes : undefined,
};
onSubmit(action, params);
})();
};
const toggleFurnishType = (type: FurnishType) => {
setSelectedFurnishTypes(prev =>
prev.includes(type)
? prev.filter(t => t !== type)
: [...prev, type]
);
};
// Count active filters
const countActiveFilters = (): number => {
const values = form.getValues();
let count = 0;
if (values.min_bedrooms && values.min_bedrooms > 0) count++;
if (values.max_bedrooms && values.max_bedrooms < 10) count++;
if (values.min_price && values.min_price > 0) count++;
if (values.max_price) count++;
if (values.min_sqm && values.min_sqm > 0) count++;
if (values.max_sqm) count++;
if (values.min_price_per_sqm) count++;
if (values.max_price_per_sqm) count++;
if (values.district && values.district.length > 0) count++;
if (selectedFurnishTypes.length > 0) count++;
if (values.last_seen_days && values.last_seen_days < 365) count++;
return count;
};
const [activeFilterCount, setActiveFilterCount] = useState(0);
useEffect(() => {
const subscription = form.watch(() => {
setActiveFilterCount(countActiveFilters());
});
return () => subscription.unsubscribe();
}, [form, selectedFurnishTypes]);
return (
<div className="h-full flex flex-col bg-background border-r overflow-hidden">
{/* Header */}
<div className="p-4 border-b shrink-0">
<div className="flex items-center gap-2">
<Filter className="h-5 w-5" />
<h2 className="font-semibold text-lg">Filters</h2>
{activeFilterCount > 0 && (
<span className="ml-auto bg-primary text-primary-foreground text-xs px-2 py-0.5 rounded-full">
{activeFilterCount}
</span>
)}
</div>
{listingCount !== undefined && (
<p className="text-sm text-muted-foreground mt-1">
{listingCount.toLocaleString()} listings
</p>
)}
</div>
{/* Filters */}
<ScrollArea className="flex-1 min-h-0">
<Form {...form}>
<form className="p-4 space-y-4">
<Accordion type="multiple" defaultValue={["visualization", "price-size", "features"]} className="w-full">
{/* Visualization Options */}
<AccordionItem value="visualization">
<AccordionTrigger className="py-2 text-sm font-medium">
Visualization
</AccordionTrigger>
<AccordionContent>
<div className="space-y-4">
<FormField
control={form.control}
name="metric"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">Color by</FormLabel>
<Select onValueChange={(value) => {
field.onChange(value);
onMetricChange?.(value as Metric);
}} defaultValue={field.value}>
<FormControl>
<SelectTrigger className="h-8 text-sm">
<SelectValue placeholder="Metric" />
</SelectTrigger>
</FormControl>
<SelectContent>
<SelectItem value={Metric.qmprice}>Price per m²</SelectItem>
<SelectItem value={Metric.rooms}>Bedrooms</SelectItem>
<SelectItem value={Metric.qm}>Size (m²)</SelectItem>
<SelectItem value={Metric.price}>Total Price</SelectItem>
</SelectContent>
</Select>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="listing_type"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">Type</FormLabel>
<Select onValueChange={field.onChange} defaultValue={field.value}>
<FormControl>
<SelectTrigger className="h-8 text-sm">
<SelectValue placeholder="Type" />
</SelectTrigger>
</FormControl>
<SelectContent>
<SelectItem value={ListingType.RENT}>For Rent</SelectItem>
<SelectItem value={ListingType.BUY}>For Sale</SelectItem>
</SelectContent>
</Select>
<FormMessage />
</FormItem>
)}
/>
</div>
</AccordionContent>
</AccordionItem>
{/* Price & Size */}
<AccordionItem value="price-size">
<AccordionTrigger className="py-2 text-sm font-medium">
Price & Size
</AccordionTrigger>
<AccordionContent>
<div className="space-y-4">
<div className="grid grid-cols-2 gap-2">
<FormField
control={form.control}
name="min_price"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">Min Price (£)</FormLabel>
<FormControl>
<Input
type="number"
placeholder="0"
className="h-8 text-sm"
{...field}
onChange={(e) => field.onChange(e.target.value ? Number(e.target.value) : undefined)}
/>
</FormControl>
</FormItem>
)}
/>
<FormField
control={form.control}
name="max_price"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">Max Price (£)</FormLabel>
<FormControl>
<Input
type="number"
placeholder="Any"
className="h-8 text-sm"
{...field}
onChange={(e) => field.onChange(e.target.value ? Number(e.target.value) : undefined)}
/>
</FormControl>
</FormItem>
)}
/>
</div>
<div className="grid grid-cols-2 gap-2">
<FormField
control={form.control}
name="min_sqm"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">Min Size (m²)</FormLabel>
<FormControl>
<Input
type="number"
placeholder="0"
className="h-8 text-sm"
{...field}
onChange={(e) => field.onChange(e.target.value ? Number(e.target.value) : undefined)}
/>
</FormControl>
</FormItem>
)}
/>
<FormField
control={form.control}
name="max_sqm"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">Max Size (m²)</FormLabel>
<FormControl>
<Input
type="number"
placeholder="Any"
className="h-8 text-sm"
{...field}
onChange={(e) => field.onChange(e.target.value ? Number(e.target.value) : undefined)}
/>
</FormControl>
</FormItem>
)}
/>
</div>
<div className="grid grid-cols-2 gap-2">
<FormField
control={form.control}
name="min_price_per_sqm"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">Min £/m²</FormLabel>
<FormControl>
<Input
type="number"
placeholder="0"
className="h-8 text-sm"
{...field}
onChange={(e) => field.onChange(e.target.value ? Number(e.target.value) : undefined)}
/>
</FormControl>
</FormItem>
)}
/>
<FormField
control={form.control}
name="max_price_per_sqm"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">Max £/m²</FormLabel>
<FormControl>
<Input
type="number"
placeholder="Any"
className="h-8 text-sm"
{...field}
onChange={(e) => field.onChange(e.target.value ? Number(e.target.value) : undefined)}
/>
</FormControl>
</FormItem>
)}
/>
</div>
</div>
</AccordionContent>
</AccordionItem>
{/* Features */}
<AccordionItem value="features">
<AccordionTrigger className="py-2 text-sm font-medium">
Features
</AccordionTrigger>
<AccordionContent>
<div className="space-y-4">
<div className="grid grid-cols-2 gap-2">
<FormField
control={form.control}
name="min_bedrooms"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">Min Beds</FormLabel>
<FormControl>
<Input
type="number"
placeholder="0"
className="h-8 text-sm"
min={0}
max={10}
{...field}
onChange={(e) => field.onChange(e.target.value ? Number(e.target.value) : undefined)}
/>
</FormControl>
</FormItem>
)}
/>
<FormField
control={form.control}
name="max_bedrooms"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">Max Beds</FormLabel>
<FormControl>
<Input
type="number"
placeholder="Any"
className="h-8 text-sm"
min={0}
max={10}
{...field}
onChange={(e) => field.onChange(e.target.value ? Number(e.target.value) : undefined)}
/>
</FormControl>
</FormItem>
)}
/>
</div>
{watchedListingType === ListingType.RENT && (
<div>
<FormLabel className="text-xs">Furnishing</FormLabel>
<div className="flex flex-wrap gap-2 mt-2">
{[
{ value: FurnishType.FURNISHED, label: 'Furnished' },
{ value: FurnishType.PART_FURNISHED, label: 'Part' },
{ value: FurnishType.UNFURNISHED, label: 'Unfurn.' },
].map((option) => (
<button
key={option.value}
type="button"
onClick={() => toggleFurnishType(option.value)}
className={`px-2 py-1 text-xs rounded-md border transition-colors ${
selectedFurnishTypes.includes(option.value)
? 'bg-primary text-primary-foreground border-primary'
: 'bg-background hover:bg-muted border-input'
}`}
>
{option.label}
</button>
))}
</div>
</div>
)}
</div>
</AccordionContent>
</AccordionItem>
{/* Location */}
<AccordionItem value="location">
<AccordionTrigger className="py-2 text-sm font-medium">
Location
</AccordionTrigger>
<AccordionContent>
<FormField
control={form.control}
name="district"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">District</FormLabel>
<FormControl>
<Input
type="text"
placeholder="e.g., Westminster, Camden"
className="h-8 text-sm"
{...field}
/>
</FormControl>
<FormDescription className="text-xs">
Comma-separated list of districts
</FormDescription>
</FormItem>
)}
/>
</AccordionContent>
</AccordionItem>
{/* Availability / Recency */}
<AccordionItem value="availability">
<AccordionTrigger className="py-2 text-sm font-medium">
{watchedListingType === ListingType.RENT ? 'Availability' : 'Recency'}
</AccordionTrigger>
<AccordionContent>
<div className="space-y-4">
{watchedListingType === ListingType.RENT && (
<FormField
control={form.control}
name="available_from"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">Available From</FormLabel>
<FormControl>
<Calendar29
onSelect={field.onChange}
selected={field.value}
rawInputValue={availableFromRawInput}
onChangeRawInputValue={setAvailableFromRawInput}
/>
</FormControl>
</FormItem>
)}
/>
)}
<FormField
control={form.control}
name="last_seen_days"
render={({ field }) => (
<FormItem>
<FormLabel className="text-xs">Last Seen (days)</FormLabel>
<FormControl>
<Input
type="number"
placeholder="28"
className="h-8 text-sm"
{...field}
onChange={(e) => field.onChange(e.target.value ? Number(e.target.value) : undefined)}
/>
</FormControl>
<FormDescription className="text-xs">
Show listings seen in last N days
</FormDescription>
</FormItem>
)}
/>
</div>
</AccordionContent>
</AccordionItem>
</Accordion>
</form>
</Form>
</ScrollArea>
{/* Action Buttons */}
<div className="p-4 border-t space-y-2 shrink-0">
<Button
className="w-full"
onClick={() => handleFormSubmit('visualize')}
disabled={isLoading}
>
{isLoading ? (
<>
<Loader2 className="mr-2 h-4 w-4 animate-spin" />
Loading...
</>
) : (
<>
<Filter className="mr-2 h-4 w-4" />
Apply Filters
</>
)}
</Button>
<Button
variant="outline"
className="w-full"
onClick={() => handleFormSubmit('fetch-data')}
disabled={isLoading}
>
<RefreshCw className="mr-2 h-4 w-4" />
Refresh Data
</Button>
</div>
</div>
);
}

90
crawler/frontend/src/components/Header.tsx
View file

@ -1,90 +0,0 @@
import type { AuthUser } from '@/auth/types';
import { Button } from './ui/button';
import { Separator } from './ui/separator';
import { LogOut, Home, Filter } from 'lucide-react';
import { logout } from '@/auth/authService';
import { clearPasskeyUser } from '@/auth/passkeyService';
import { HealthIndicator } from './HealthIndicator';
import { TaskIndicator } from './TaskIndicator';
interface HeaderProps {
user: AuthUser;
activeFilterCount?: number;
taskID?: string | null;
isLoading?: boolean;
onToggleFilters?: () => void;
showFilterToggle?: boolean;
onTaskCancelled?: () => void;
}
export function Header({
user,
activeFilterCount = 0,
taskID,
onToggleFilters,
showFilterToggle = false,
onTaskCancelled,
}: HeaderProps) {
const handleLogout = async () => {
if (user.provider === 'passkey') {
clearPasskeyUser();
window.location.reload();
} else {
await logout();
}
};
return (
<header className="flex h-14 shrink-0 items-center gap-3 border-b bg-background px-4">
{/* Logo / Brand */}
<div className="flex items-center gap-2">
<Home className="h-5 w-5 text-primary" />
<span className="font-semibold text-lg hidden sm:inline">Wrongmove</span>
</div>
<Separator orientation="vertical" className="h-6" />
{/* Health Indicator */}
<HealthIndicator />
{/* Task Indicator */}
<TaskIndicator taskID={taskID ?? null} onTaskCancelled={onTaskCancelled} />
{/* Filter Toggle (mobile) */}
{showFilterToggle && (
<Button
variant="outline"
size="sm"
className="sm:hidden"
onClick={onToggleFilters}
>
<Filter className="h-4 w-4" />
{activeFilterCount > 0 && (
<span className="ml-1 bg-primary text-primary-foreground text-xs px-1.5 py-0.5 rounded-full">
{activeFilterCount}
</span>
)}
</Button>
)}
{/* Spacer */}
<div className="flex-1" />
{/* User Menu */}
<div className="flex items-center gap-3">
<span className="text-sm text-muted-foreground hidden md:inline">
{user.email}
</span>
<Button
variant="ghost"
size="sm"
onClick={handleLogout}
className="gap-2"
>
<LogOut className="h-4 w-4" />
<span className="hidden sm:inline">Logout</span>
</Button>
</div>
</header>
);
}

83
crawler/frontend/src/components/HealthIndicator.tsx
View file

@ -1,83 +0,0 @@
import { useEffect, useState } from 'react';
import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger } from './ui/tooltip';
import { checkBackendHealth, type HealthStatus, type HealthCheckResult } from '@/services';
import { Circle, Loader2 } from 'lucide-react';
interface HealthIndicatorProps {
/** How often to check health in milliseconds (default: 30000 = 30s) */
interval?: number;
}
export function HealthIndicator({ interval = 30000 }: HealthIndicatorProps) {
const [health, setHealth] = useState<HealthCheckResult>({ status: 'checking' });
useEffect(() => {
// Initial check
checkBackendHealth().then(setHealth);
// Periodic checks
const intervalId = setInterval(() => {
checkBackendHealth().then(setHealth);
}, interval);
return () => clearInterval(intervalId);
}, [interval]);
const getStatusColor = (status: HealthStatus) => {
switch (status) {
case 'healthy':
return 'text-green-500';
case 'unhealthy':
return 'text-red-500';
case 'checking':
return 'text-muted-foreground';
}
};
const getStatusLabel = (status: HealthStatus) => {
switch (status) {
case 'healthy':
return 'Connected';
case 'unhealthy':
return 'Disconnected';
case 'checking':
return 'Checking...';
}
};
const getTooltipContent = () => {
if (health.status === 'checking') {
return 'Checking backend connection...';
}
if (health.status === 'healthy') {
return `Backend connected (${health.latencyMs}ms)`;
}
return `Backend unavailable: ${health.error || 'Unknown error'}`;
};
return (
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<div className="flex items-center gap-1.5 cursor-default">
{health.status === 'checking' ? (
<Loader2 className="h-3 w-3 animate-spin text-muted-foreground" />
) : (
<Circle
className={`h-2.5 w-2.5 fill-current ${getStatusColor(health.status)}`}
/>
)}
<span className={`text-xs ${getStatusColor(health.status)} hidden sm:inline`}>
{getStatusLabel(health.status)}
</span>
</div>
</TooltipTrigger>
<TooltipContent side="bottom">
<p>{getTooltipContent()}</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
);
}

151
crawler/frontend/src/components/ListView.tsx
View file

@ -1,151 +0,0 @@
import { useState, useMemo, useCallback } from 'react';
import { ArrowUpDown, ArrowUp, ArrowDown } from 'lucide-react';
import { Virtuoso } from 'react-virtuoso';
import { Button } from './ui/button';
import { PropertyCard } from './PropertyCard';
import type { GeoJSONFeatureCollection, PropertyFeature, PropertyProperties } from '@/types';
type SortField = 'total_price' | 'qmprice' | 'qm' | 'rooms' | 'last_seen';
type SortOrder = 'asc' | 'desc';
interface ListViewProps {
listingData: GeoJSONFeatureCollection;
onPropertyClick?: (property: PropertyProperties, coordinates: [number, number]) => void;
highlightedPropertyUrl?: string | null;
}
interface SortConfig {
field: SortField;
order: SortOrder;
}
const SORT_OPTIONS: { field: SortField; label: string }[] = [
{ field: 'total_price', label: 'Price' },
{ field: 'qmprice', label: '£/m²' },
{ field: 'qm', label: 'Size' },
{ field: 'rooms', label: 'Beds' },
{ field: 'last_seen', label: 'Last Seen' },
];
export function ListView({ listingData, onPropertyClick, highlightedPropertyUrl }: ListViewProps) {
const [sortConfig, setSortConfig] = useState<SortConfig>({ field: 'qmprice', order: 'asc' });
// Calculate average price per sqm for "good deal" indicator
const avgPricePerSqm = useMemo(() => {
const validPrices = listingData.features
.map((f) => f.properties.qmprice)
.filter((p): p is number => typeof p === 'number' && p > 0);
return validPrices.length > 0
? validPrices.reduce((a, b) => a + b, 0) / validPrices.length
: 0;
}, [listingData]);
// Sort features
const sortedFeatures = useMemo(() => {
const features = [...listingData.features];
features.sort((a, b) => {
let aValue: number | string;
let bValue: number | string;
switch (sortConfig.field) {
case 'total_price':
aValue = a.properties.total_price || 0;
bValue = b.properties.total_price || 0;
break;
case 'qmprice':
aValue = a.properties.qmprice || 0;
bValue = b.properties.qmprice || 0;
break;
case 'qm':
aValue = a.properties.qm || 0;
bValue = b.properties.qm || 0;
break;
case 'rooms':
aValue = a.properties.rooms || 0;
bValue = b.properties.rooms || 0;
break;
case 'last_seen':
aValue = new Date(a.properties.last_seen).getTime();
bValue = new Date(b.properties.last_seen).getTime();
break;
default:
return 0;
}
if (typeof aValue === 'number' && typeof bValue === 'number') {
return sortConfig.order === 'asc' ? aValue - bValue : bValue - aValue;
}
return 0;
});
return features;
}, [listingData.features, sortConfig]);
const handleSort = (field: SortField) => {
setSortConfig((prev) => ({
field,
order: prev.field === field && prev.order === 'asc' ? 'desc' : 'asc',
}));
};
const handlePropertyClick = useCallback((feature: PropertyFeature) => {
if (onPropertyClick) {
onPropertyClick(feature.properties, feature.geometry.coordinates);
}
}, [onPropertyClick]);
const SortIcon = ({ field }: { field: SortField }) => {
if (sortConfig.field !== field) {
return <ArrowUpDown className="h-3.5 w-3.5" />;
}
return sortConfig.order === 'asc'
? <ArrowUp className="h-3.5 w-3.5" />
: <ArrowDown className="h-3.5 w-3.5" />;
};
return (
<div className="h-full flex flex-col bg-background">
{/* Sort controls */}
<div className="flex items-center gap-1 p-2 border-b overflow-x-auto">
<span className="text-xs text-muted-foreground mr-1 shrink-0">Sort:</span>
{SORT_OPTIONS.map((option) => (
<Button
key={option.field}
variant={sortConfig.field === option.field ? 'secondary' : 'ghost'}
size="sm"
className="h-7 px-2 text-xs shrink-0"
onClick={() => handleSort(option.field)}
>
{option.label}
<SortIcon field={option.field} />
</Button>
))}
</div>
{/* Listing count */}
<div className="px-3 py-2 text-sm text-muted-foreground border-b">
Showing {sortedFeatures.length.toLocaleString()} properties
</div>
{/* Property list */}
<Virtuoso
className="flex-1"
data={sortedFeatures}
overscan={200}
itemContent={(_index, feature) => (
<div className="px-3 pb-2 first:pt-3">
<PropertyCard
key={feature.properties.url}
property={feature.properties}
variant="compact"
avgPricePerSqm={avgPricePerSqm}
isHighlighted={feature.properties.url === highlightedPropertyUrl}
onClick={() => handlePropertyClick(feature)}
/>
</div>
)}
/>
</div>
);
}

189
crawler/frontend/src/components/LoginModal.tsx
View file

@ -1,189 +0,0 @@
import { login, type AuthError } from '@/auth/authService';
import { registerPasskey, loginWithPasskey } from '@/auth/passkeyService';
import type { AuthUser } from '@/auth/types';
import { Button } from "@/components/ui/button";
import { DialogDescription } from '@radix-ui/react-dialog';
import React, { useState } from 'react';
import { Dialog, DialogContent, DialogHeader, DialogTitle } from './ui/dialog';
import { Tabs, TabsContent, TabsList, TabsTrigger } from './ui/tabs';
import { Home, LogIn, AlertCircle, Loader2, Fingerprint, Mail } from 'lucide-react';
interface LoginModalProps {
isOpen: boolean;
onPasskeyLogin: (user: AuthUser) => void;
}
const LoginModal: React.FC<LoginModalProps> = ({ isOpen, onPasskeyLogin }) => {
const [isLoading, setIsLoading] = useState(false);
const [error, setError] = useState<string | null>(null);
const [email, setEmail] = useState('');
if (!isOpen) return null;
const handleSSOLogin = async () => {
setIsLoading(true);
setError(null);
try {
await login();
} catch (err) {
setError((err as AuthError).message);
setIsLoading(false);
}
};
const handlePasskeyLogin = async () => {
setIsLoading(true);
setError(null);
try {
const user = await loginWithPasskey();
onPasskeyLogin(user);
} catch (err) {
setError(err instanceof Error ? err.message : String(err));
} finally {
setIsLoading(false);
}
};
const handlePasskeyRegister = async () => {
if (!email.trim()) {
setError('Please enter your email address');
return;
}
setIsLoading(true);
setError(null);
try {
const user = await registerPasskey(email.trim());
onPasskeyLogin(user);
} catch (err) {
setError(err instanceof Error ? err.message : String(err));
} finally {
setIsLoading(false);
}
};
const clearError = () => setError(null);
return (
<Dialog open={isOpen}>
<DialogContent className="sm:max-w-[425px]" showCloseButton={false}>
<DialogHeader className="space-y-4">
<div className="flex items-center gap-3">
<div className="p-2 bg-primary/10 rounded-lg">
<Home className="h-6 w-6 text-primary" />
</div>
<div>
<DialogTitle className="text-xl">Wrongmove</DialogTitle>
<DialogDescription className="text-sm text-muted-foreground">
Your smart property search companion
</DialogDescription>
</div>
</div>
</DialogHeader>
<div className="py-2">
{/* Error State */}
{error && (
<div className="bg-destructive/10 border border-destructive/30 rounded-lg p-4 flex items-start gap-3 mb-4">
<AlertCircle className="h-5 w-5 text-destructive shrink-0 mt-0.5" />
<div className="flex-1 space-y-2">
<p className="text-sm text-destructive">{error}</p>
<Button
size="sm"
variant="ghost"
onClick={clearError}
>
Dismiss
</Button>
</div>
</div>
)}
<Tabs defaultValue="signin">
<TabsList>
<TabsTrigger value="signin">Sign In</TabsTrigger>
<TabsTrigger value="signup">Sign Up</TabsTrigger>
</TabsList>
<TabsContent value="signin" className="space-y-4 pt-2">
<Button
onClick={handlePasskeyLogin}
disabled={isLoading}
className="w-full gap-2"
size="lg"
>
{isLoading ? (
<Loader2 className="h-4 w-4 animate-spin" />
) : (
<Fingerprint className="h-4 w-4" />
)}
Sign in with Passkey
</Button>
<div className="relative">
<div className="absolute inset-0 flex items-center">
<span className="w-full border-t" />
</div>
<div className="relative flex justify-center text-xs uppercase">
<span className="bg-background px-2 text-muted-foreground">or</span>
</div>
</div>
<Button
onClick={handleSSOLogin}
disabled={isLoading}
variant="outline"
className="w-full gap-2"
size="lg"
>
{isLoading ? (
<Loader2 className="h-4 w-4 animate-spin" />
) : (
<LogIn className="h-4 w-4" />
)}
Sign in with SSO
</Button>
</TabsContent>
<TabsContent value="signup" className="space-y-4 pt-2">
<div className="space-y-2">
<div className="relative">
<Mail className="absolute left-3 top-1/2 -translate-y-1/2 h-4 w-4 text-muted-foreground" />
<input
type="email"
placeholder="Email address"
value={email}
onChange={(e) => setEmail(e.target.value)}
onKeyDown={(e) => {
if (e.key === 'Enter') handlePasskeyRegister();
}}
className="flex h-10 w-full rounded-md border border-input bg-background px-3 py-2 pl-10 text-sm ring-offset-background placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2"
/>
</div>
</div>
<Button
onClick={handlePasskeyRegister}
disabled={isLoading || !email.trim()}
className="w-full gap-2"
size="lg"
>
{isLoading ? (
<Loader2 className="h-4 w-4 animate-spin" />
) : (
<Fingerprint className="h-4 w-4" />
)}
Create account with Passkey
</Button>
<p className="text-xs text-muted-foreground text-center">
A passkey uses your device's biometrics or security key for secure, passwordless authentication.
</p>
</TabsContent>
</Tabs>
</div>
</DialogContent>
</Dialog>
);
};
export default LoginModal;

366
crawler/frontend/src/components/Map.tsx
View file

@ -1,366 +0,0 @@
import crossfilter from "crossfilter2";
import * as d3 from "d3";
import mapboxgl from "mapbox-gl";
import 'mapbox-gl/dist/mapbox-gl.css';
import { useEffect, useRef, useMemo, useCallback } from "react";
import { renderToString } from 'react-dom/server';
import "../assets/Map.css";
import { Metric, type ParameterValues } from "./Parameters";
import { PropertyCard } from "./PropertyCard";
import { ScrollArea } from "./ui/scroll-area";
import type { GeoJSONFeatureCollection, PropertyFeature, PropertyProperties } from "@/types";
import { MAP_CONFIG, HEATMAP_CONFIG, PERCENTILE_CONFIG } from "@/constants";
import { getColorSchemeForMetric, getMetricInterpretation } from "@/constants/colorSchemes";
import { clone, percentile, calculateColorStops } from "@/utils/mapUtils";
// Type declaration for the external HexgridHeatmap library
declare class HexgridHeatmap {
_tree: {
search: (bounds: { minX: number; maxX: number; minY: number; maxY: number }) => PropertyWithCoords[];
};
constructor(map: mapboxgl.Map, id: string, beforeLayer: string);
setIntensity(value: number): void;
setSpread(value: number): void;
setCellDensity(value: number): void;
setPropertyName(name: string): void;
setData(data: GeoJSONFeatureCollection): void;
setColorStops(stops: [number, string][]): void;
update(): void;
}
interface PropertyWithCoords {
properties: PropertyProperties;
}
interface CrossfilterRecord extends PropertyProperties {
index: number;
}
interface MapProps {
listingData: GeoJSONFeatureCollection;
queryParameters: ParameterValues | null;
onPropertyClick?: (property: PropertyProperties, coordinates: [number, number]) => void;
}
interface FilterState {
city: string;
country: string | null;
mode: string;
count?: number;
}
export function Map(props: MapProps) {
const data = props.listingData;
const mapRef = useRef<mapboxgl.Map | null>(null);
const mapContainerRef = useRef<HTMLDivElement | null>(null);
const heatmapRef = useRef<HexgridHeatmap | null>(null);
const updateTimeoutRef = useRef<number | null>(null);
const isMapLoadedRef = useRef<boolean>(false);
const lastDataLengthRef = useRef<number>(0);
const filter: FilterState = { city: 'London', country: null, mode: Metric.qmprice };
if (props.queryParameters) {
filter.mode = props.queryParameters.metric;
}
// Get appropriate color scheme based on metric
const colorScheme = useMemo(() => {
return getColorSchemeForMetric(filter.mode);
}, [filter.mode]);
const metricInfo = useMemo(() => {
return getMetricInterpretation(filter.mode);
}, [filter.mode]);
// Calculate average price per sqm for property cards
const avgPricePerSqm = useMemo(() => {
const validPrices = data.features
.map((f) => f.properties.qmprice)
.filter((p): p is number => typeof p === 'number' && p > 0);
return validPrices.length > 0
? validPrices.reduce((a, b) => a + b, 0) / validPrices.length
: 0;
}, [data]);
// Build crossfilter data
const buildCrossfilterData = useCallback(() => {
return data.features.map(function (d: PropertyFeature, i: number) {
const propsCopy = clone(d.properties) as CrossfilterRecord;
propsCopy.index = i;
return propsCopy;
});
}, [data]);
const updateHeatmap = useCallback(() => {
if (!mapRef.current || !isMapLoadedRef.current) return;
const crossData = buildCrossfilterData();
const cf = crossfilter(crossData);
const qmDim = cf.dimension(function (d: CrossfilterRecord) { return d.qm; });
const cityDim = cf.dimension(function (d: CrossfilterRecord) { return d.city; });
const countryDim = cf.dimension(function (d: CrossfilterRecord) { return d.country; });
const indexDim = cf.dimension(function (d: CrossfilterRecord) { return d.index; });
// Create heatmap if it doesn't exist
if (!heatmapRef.current) {
heatmapRef.current = new HexgridHeatmap(mapRef.current, "hexgrid-heatmap", "waterway-label");
heatmapRef.current.setIntensity(HEATMAP_CONFIG.INTENSITY);
heatmapRef.current.setSpread(HEATMAP_CONFIG.SPREAD);
heatmapRef.current.setCellDensity(HEATMAP_CONFIG.CELL_DENSITY);
}
const heatmap = heatmapRef.current;
heatmap.setPropertyName(filter.mode);
if (filter.mode === Metric.qmprice) {
qmDim.filter((d) => (d as number) > 0);
}
if (filter.city) {
cityDim.filterExact(filter.city);
} else if (filter.country) {
countryDim.filterExact(filter.country);
}
const subset: GeoJSONFeatureCollection = { type: "FeatureCollection", features: [] };
indexDim.top(Infinity).forEach(function (i: CrossfilterRecord) {
subset.features.push(data.features[i.index]);
});
// Update heatmap data
heatmap.setData(subset);
let values = subset.features.map(function (d: PropertyFeature) {
return d.properties[filter.mode as keyof PropertyProperties] as number;
});
values = values.sort(function (a: number, b: number) { return a - b; });
const minIndex = Math.round(values.length * PERCENTILE_CONFIG.MIN_BOUND);
const maxIndex = Math.round(values.length * PERCENTILE_CONFIG.MAX_BOUND);
const min = values[minIndex];
const max = values[maxIndex];
makeLegend(colorScheme, min, max);
const colorStopsValue = calculateColorStops(colorScheme, min, max);
heatmap.setColorStops(colorStopsValue);
heatmap.update();
// Fit bounds only on first load or significant data change
if (lastDataLengthRef.current === 0 && subset.features.length > 0) {
const longitudes = subset.features.map(function (d: PropertyFeature) { return d.geometry.coordinates[0]; }).sort(function (a: number, b: number) { return a - b; });
const latitudes = subset.features.map(function (d: PropertyFeature) { return d.geometry.coordinates[1]; }).sort(function (a: number, b: number) { return a - b; });
const minlng = percentile(longitudes, PERCENTILE_CONFIG.BOUNDS_CLIP_MIN);
const maxlng = percentile(longitudes, PERCENTILE_CONFIG.BOUNDS_CLIP_MAX);
const minlat = percentile(latitudes, PERCENTILE_CONFIG.BOUNDS_CLIP_MIN);
const maxlat = percentile(latitudes, PERCENTILE_CONFIG.BOUNDS_CLIP_MAX);
mapRef.current?.fitBounds([
[minlng, minlat],
[maxlng, maxlat]
], { duration: 0 });
}
lastDataLengthRef.current = subset.features.length;
}, [data, filter.mode, filter.city, filter.country, colorScheme, buildCrossfilterData]);
// Initialize map
useEffect(() => {
if (!mapContainerRef.current) return;
mapboxgl.accessToken = MAP_CONFIG.MAPBOX_TOKEN;
mapRef.current = new mapboxgl.Map({
container: mapContainerRef.current,
style: MAP_CONFIG.STYLE,
center: MAP_CONFIG.DEFAULT_CENTER,
zoom: MAP_CONFIG.DEFAULT_ZOOM
});
mapRef.current.on('load', function () {
isMapLoadedRef.current = true;
lastDataLengthRef.current = 0;
updateHeatmap();
});
mapRef.current.on('click', function (e: mapboxgl.MapMouseEvent) {
if (!mapRef.current) return;
const layers = ['hexgrid-heatmap', 'hexgrid-heatmap-back']
.filter(l => mapRef.current!.getLayer(l));
if (layers.length === 0) return;
const features = mapRef.current.queryRenderedFeatures(e.point, { layers });
if (features.length > 0) {
const props = features[0].properties;
if (props && props.searchMinX !== undefined) {
openListingsDialog(e.lngLat.lng, e.lngLat.lat, {
minX: props.searchMinX,
minY: props.searchMinY,
maxX: props.searchMaxX,
maxY: props.searchMaxY
});
}
}
});
mapRef.current.on('mousemove', function (e: mapboxgl.MapMouseEvent) {
if (!mapRef.current) return;
const layers = ['hexgrid-heatmap', 'hexgrid-heatmap-back']
.filter(l => mapRef.current!.getLayer(l));
if (layers.length === 0) return;
const features = mapRef.current.queryRenderedFeatures(e.point, { layers });
mapRef.current.getCanvas().style.cursor = features.length > 0 ? 'pointer' : '';
});
return () => {
if (updateTimeoutRef.current) {
clearTimeout(updateTimeoutRef.current);
}
heatmapRef.current = null;
isMapLoadedRef.current = false;
mapRef.current?.remove();
};
// eslint-disable-next-line react-hooks/exhaustive-deps
}, []);
// Debounced update effect - only update after 200ms of no changes
useEffect(() => {
if (!isMapLoadedRef.current) return;
// Clear any pending update
if (updateTimeoutRef.current) {
clearTimeout(updateTimeoutRef.current);
}
// Schedule new update after 200ms of no changes
updateTimeoutRef.current = window.setTimeout(() => {
updateHeatmap();
}, 200);
return () => {
if (updateTimeoutRef.current) {
clearTimeout(updateTimeoutRef.current);
}
};
}, [data, updateHeatmap]);
function makeLegend(colorstops: [number, string][], minValue: number, maxValue: number) {
const svg_height = 280, svg_width = 80;
d3.select('#svg').selectAll('*').remove();
const svg = d3.select('#svg');
svg
.attr('height', svg_height)
.attr('width', svg_width);
// Add metric name at top
svg.append("text")
.attr("x", svg_width / 2)
.attr("y", 12)
.attr("text-anchor", "middle")
.attr("font-size", "11px")
.attr("font-weight", "600")
.attr("fill", "#374151")
.text(metricInfo.name);
const gradientTop = 30;
const gradientHeight = svg_height - 70;
const linearGradient = svg.append("defs")
.append("linearGradient")
.attr("id", "linear-gradient");
linearGradient
.attr("x1", "0%")
.attr("y1", "100%")
.attr("x2", "0%")
.attr("y2", "0%");
svg.append("rect")
.attr("x", 0)
.attr("y", gradientTop)
.attr("width", svg_width * 0.35)
.attr("height", gradientHeight)
.attr('rx', 4)
.style("fill", "url(#linear-gradient)");
colorstops.forEach(function (d: [number, string]) {
linearGradient.append("stop")
.attr("offset", d[0] + "%")
.attr("stop-color", d[1]);
});
const xScale = d3.scaleLinear().range([gradientHeight - 10, 0]).domain([minValue, maxValue]);
const xAxis = d3.axisRight(xScale).ticks(5).tickFormat((d) => {
const num = d as number;
if (num >= 1000) {
return `${(num / 1000).toFixed(1)}k`;
}
return String(Math.round(num));
});
svg.append("g")
.attr("class", "axis")
.attr("transform", "translate(" + (svg_width * 0.38) + "," + (gradientTop + 5) + ")")
.call(xAxis)
.selectAll("text")
.attr("font-size", "10px");
// Add interpretation labels at bottom
svg.append("text")
.attr("x", svg_width / 2)
.attr("y", svg_height - 25)
.attr("text-anchor", "middle")
.attr("font-size", "9px")
.attr("fill", "#22c55e")
.text(metricInfo.low);
svg.append("text")
.attr("x", svg_width / 2)
.attr("y", svg_height - 10)
.attr("text-anchor", "middle")
.attr("font-size", "9px")
.attr("fill", "#ef4444")
.text(metricInfo.high);
}
function openListingsDialog(longitude: number, latitude: number, searchBounds: { minX: number; minY: number; maxX: number; maxY: number }) {
if (!heatmapRef.current || !mapRef.current) return;
const properties = heatmapRef.current._tree.search(searchBounds);
if (properties.length > 0) {
const listingDialogPopup = getListingDialog(properties);
new mapboxgl.Popup()
.setLngLat([longitude, latitude])
.setHTML(renderToString(listingDialogPopup))
.setMaxWidth("450px")
.addTo(mapRef.current);
}
}
function getListingDialog(properties: PropertyWithCoords[]) {
return (
<ScrollArea className="rounded-md">
<div className="overflow-y-auto max-h-[500px] w-[420px]">
<div className="px-3 py-2 text-sm font-medium border-b bg-muted/50">
<strong>{properties.length}</strong> properties in this area
</div>
<div className="p-2 space-y-2">
{properties.map((property) => (
<PropertyCard
key={property.properties.url}
property={property.properties}
variant="full"
avgPricePerSqm={avgPricePerSqm}
/>
))}
</div>
</div>
</ScrollArea>
);
}
return (
<div className="relative w-full h-full">
<div id='map-container' ref={mapContainerRef}></div>
<div id="legend">
<svg id="svg"></svg>
</div>
</div>
);
}
// Re-export types for backwards compatibility
export { Metric, type ParameterValues } from "./Parameters";

263
crawler/frontend/src/components/Parameters.tsx
View file

@ -1,263 +0,0 @@
import { zodResolver } from "@hookform/resolvers/zod";
import { DialogTitle } from "@radix-ui/react-dialog";
import { useState } from "react";
import { useForm } from "react-hook-form";
import { z } from "zod";
import { Button } from "./ui/button";
import { Calendar29 } from "./ui/DatePicker";
import { Dialog, DialogContent, DialogTrigger } from "./ui/dialog";
import { Form, FormControl, FormDescription, FormField, FormItem, FormLabel, FormMessage } from "./ui/form";
import { Input } from "./ui/input";
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "./ui/select";
export enum Metric {
qmprice = 'qmprice',
rooms = 'rooms',
qm = 'qm',
price = 'total_price',
}
export enum ListingType {
RENT = 'RENT',
BUY = 'BUY'
}
export enum FurnishType {
FURNISHED = 'furnished',
PART_FURNISHED = 'partFurnished',
UNFURNISHED = 'unfurnished',
}
export interface ParameterValues {
metric: Metric
listing_type: ListingType
min_bedrooms?: number
max_bedrooms?: number
min_price?: number
max_price?: number
min_sqm?: number
max_sqm?: number
min_price_per_sqm?: number
max_price_per_sqm?: number
last_seen_days?: number
available_from?: Date
district: string
furnish_types?: FurnishType[]
}
export function Parameters(
props: {
isOpen: boolean,
onSubmit: (action: 'fetch-data' | 'visualize', fromValues: ParameterValues) => void,
setIsOpen: (isOpen: boolean) => void
}
) {
const {
register,
} = useForm<ParameterValues>()
const [action, setAction] = useState<'fetch-data' | 'visualize' | null>(null)
const [availableFromRawInput, setAvailableFromRawInput] = useState("now");
const formSchema = z.object({
metric: z.nativeEnum(Metric, { required_error: "Metric is required" }),
listing_type: z.nativeEnum(ListingType, { required_error: "Listing Type is required" }),
min_bedrooms: z.number().min(1).max(10).optional(),
max_bedrooms: z.number().min(1).max(10).optional(),
max_price: z.number().optional(),
min_price: z.number().min(0).optional(),
min_sqm: z.number().optional(),
last_seen_days: z.number().min(0).optional(),
available_from: z.date(),
district: z.string()
})
const form = useForm<z.infer<typeof formSchema>>({
resolver: zodResolver(formSchema),
defaultValues: {
metric: Metric.qmprice,
listing_type: ListingType.RENT,
min_bedrooms: 1,
max_bedrooms: 3,
max_price: 3000,
min_price: 2000,
min_sqm: 50,
last_seen_days: 28,
available_from: new Date(),
district: ''
},
})
// 2. Define a submit handler.
function onSubmit(values: z.infer<typeof formSchema>) {
// Do something with the form values.
// ✅ This will be type-safe and validated.
console.log(values)
if (action) {
props.onSubmit(action, values)
}
}
return <>
<Dialog open={props.isOpen} onOpenChange={props.setIsOpen} >
{/* <Dialog > */}
<DialogTrigger asChild>
<Button variant="outline" onClick={() => props.setIsOpen(true)}>Open Parameters</Button>
</DialogTrigger>
<DialogContent className="sm:max-w-[425px]">
<DialogTitle>
Visualization Parameters
</DialogTitle>
<Form {...form}>
<form onSubmit={form.handleSubmit(onSubmit)} className="space-y-4">
<FormField
control={form.control}
name="metric"
render={({ field }) => (
<FormItem className="flex flex-row items-center gap-4">
<FormLabel>Metric to visualize</FormLabel>
<Select onValueChange={field.onChange} defaultValue={field.value}>
<FormControl>
<SelectTrigger className="w-[180px]">
<SelectValue placeholder="Metric to Visualize" />
</SelectTrigger>
</FormControl>
<SelectContent {...register('metric')} >
<SelectItem value={Metric.qmprice}>Price per squaremeter</SelectItem>
<SelectItem value={Metric.rooms}>Number of rooms</SelectItem>
<SelectItem value={Metric.qm}>Area</SelectItem>
<SelectItem value={Metric.price}>Price</SelectItem>
</SelectContent>
</Select>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="listing_type"
render={({ field }) => (
<FormItem className="flex flex-row items-center gap-4">
<FormLabel>Listing Type</FormLabel>
<FormControl >
<Select onValueChange={field.onChange} defaultValue={field.value}>
<FormControl>
<SelectTrigger className="w-[180px]">
<SelectValue placeholder="Metric to Visualize" />
</SelectTrigger>
</FormControl>
<SelectContent >
<SelectItem value={ListingType.BUY}>To buy</SelectItem>
<SelectItem value={ListingType.RENT}>To rent</SelectItem>
</SelectContent>
</Select>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="min_sqm"
render={({ field }) => (
<FormItem className="flex flex-row items-center gap-4">
<FormLabel>Min square meters</FormLabel>
<FormControl >
<Input type="number" placeholder={"m²"} {...field} onChange={(e) => field.onChange(Number(e.target.value))} />
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="min_bedrooms"
render={({ field }) => (
<FormItem className="flex flex-row items-center gap-4">
<FormLabel>Minimum number of bedrooms</FormLabel>
<FormControl >
<Input type="number" placeholder={"# bedrooms"} {...field} onChange={(e) => field.onChange(Number(e.target.value))} />
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="max_bedrooms"
render={({ field }) => (
<FormItem className="flex flex-row items-center gap-4">
<FormLabel>Maximum number of bedrooms</FormLabel>
<FormControl >
<Input type="number" placeholder={"# bedrooms"} {...field} onChange={(e) => field.onChange(Number(e.target.value))} />
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="min_price"
render={({ field }) => (
<FormItem className="flex flex-row items-center gap-4">
<FormLabel>Min price</FormLabel>
<FormControl >
<Input type="number" placeholder={"£"} {...field} onChange={(e) => field.onChange(Number(e.target.value))} />
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="max_price"
render={({ field }) => (
<FormItem className="flex flex-row items-center gap-4">
<FormLabel>Max price</FormLabel>
<FormControl >
<Input type="number" placeholder={"£"} {...field} onChange={(e) => field.onChange(Number(e.target.value))} />
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="available_from"
render={({ field }) => (
<FormItem className="flex flex-col">
<FormLabel>Available from</FormLabel>
<FormControl>
<Calendar29 onSelect={field.onChange} selected={field.value} rawInputValue={availableFromRawInput} onChangeRawInputValue={setAvailableFromRawInput} />
</FormControl>
<FormDescription>
Applicable for renting listings only
</FormDescription>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="last_seen_days"
render={({ field }) => (
<FormItem className="flex flex-row items-center gap-4">
<FormLabel>Last seen days</FormLabel>
<FormControl >
<Input type="number" placeholder={"days"} {...field} onChange={(e) => field.onChange(Number(e.target.value))} />
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<Button type="submit" value={"visualize"} onClick={() => setAction("visualize")}>Visualize existing data</Button>
<Button type="submit" value={"fetch-data"} onClick={() => setAction("fetch-data")}>Submit data refresh job</Button>
</form>
</Form>
</DialogContent>
</Dialog>
</>
}

200
crawler/frontend/src/components/PropertyCard.tsx
View file

@ -1,200 +0,0 @@
import { ExternalLink, Bed, Maximize2, PoundSterling, Clock, Building } from 'lucide-react';
import { Button } from './ui/button';
import type { PropertyProperties } from '@/types';
interface PropertyCardProps {
property: PropertyProperties;
variant?: 'compact' | 'full';
isHighlighted?: boolean;
avgPricePerSqm?: number;
onClick?: () => void;
}
export function PropertyCard({
property,
variant = 'compact',
isHighlighted = false,
avgPricePerSqm,
onClick,
}: PropertyCardProps) {
const lastSeenDate = property.last_seen.split('T')[0];
const lastSeenDays = Math.round((Date.now() - new Date(lastSeenDate).getTime()) / (1000 * 60 * 60 * 24));
// Determine if this is a good deal
const isGoodDeal = avgPricePerSqm && property.qmprice > 0 && property.qmprice < avgPricePerSqm * 0.9;
const isExpensive = avgPricePerSqm && property.qmprice > avgPricePerSqm * 1.1;
const priceIndicator = isGoodDeal
? { color: 'text-green-600 bg-green-50', label: 'Good deal' }
: isExpensive
? { color: 'text-red-600 bg-red-50', label: 'Above avg' }
: null;
const handleClick = () => {
window.open(property.url, '_blank', 'noopener,noreferrer');
onClick?.();
};
if (variant === 'compact') {
return (
<div
className={`flex gap-3 p-3 rounded-lg border transition-colors cursor-pointer hover:bg-muted/50 ${
isHighlighted ? 'ring-2 ring-primary bg-primary/5' : ''
}`}
onClick={handleClick}
>
{/* Thumbnail */}
<div className="w-20 h-20 rounded-md overflow-hidden flex-shrink-0 bg-muted">
{property.photo_thumbnail && (
<img
src={property.photo_thumbnail}
alt="Property"
className="w-full h-full object-cover"
/>
)}
</div>
{/* Details */}
<div className="flex-1 min-w-0">
<div className="flex items-start justify-between gap-2">
<div className="font-semibold text-base truncate">
£{property.total_price.toLocaleString()}
{property.listing_type !== 'BUY' && (
<span className="text-muted-foreground font-normal text-sm">/mo</span>
)}
</div>
{priceIndicator && (
<span className={`text-xs px-1.5 py-0.5 rounded ${priceIndicator.color}`}>
{priceIndicator.label}
</span>
)}
</div>
<div className="flex items-center gap-3 mt-1 text-sm text-muted-foreground">
<span className="flex items-center gap-1">
<Bed className="h-3.5 w-3.5" />
{property.rooms}
</span>
<span className="flex items-center gap-1">
<Maximize2 className="h-3.5 w-3.5" />
{property.qm} m²
</span>
<span className="flex items-center gap-1">
£{property.qmprice}/m²
</span>
</div>
<div className="flex items-center gap-2 mt-1 text-xs text-muted-foreground">
<span className="flex items-center gap-1">
<Clock className="h-3 w-3" />
{lastSeenDays}d ago
</span>
<span className="truncate">{property.agency}</span>
</div>
</div>
</div>
);
}
// Full variant (for popup/detail view)
return (
<div className={`p-4 border rounded-lg ${isHighlighted ? 'ring-2 ring-primary' : ''}`}>
{/* Header with image and price */}
<div className="flex gap-4">
<a
href={property.url}
target="_blank"
rel="noopener noreferrer"
className="block w-32 h-24 rounded-md overflow-hidden flex-shrink-0 bg-muted hover:opacity-90 transition-opacity"
>
{property.photo_thumbnail && (
<img
src={property.photo_thumbnail}
alt="Property"
className="w-full h-full object-cover"
/>
)}
</a>
<div className="flex-1">
<div className="flex items-start justify-between">
<div>
<div className="font-semibold text-xl">
£{property.total_price.toLocaleString()}
{property.listing_type !== 'BUY' && (
<span className="text-muted-foreground font-normal text-sm">/mo</span>
)}
</div>
{priceIndicator && (
<span className={`inline-block mt-1 text-xs px-2 py-0.5 rounded ${priceIndicator.color}`}>
{priceIndicator.label}
</span>
)}
</div>
</div>
</div>
</div>
{/* Stats grid */}
<div className="grid grid-cols-2 gap-3 mt-4">
<div className="flex items-center gap-2 text-sm">
<Bed className="h-4 w-4 text-muted-foreground" />
<span><strong>{property.rooms}</strong> bedrooms</span>
</div>
<div className="flex items-center gap-2 text-sm">
<Maximize2 className="h-4 w-4 text-muted-foreground" />
<span><strong>{property.qm}</strong> m²</span>
</div>
<div className="flex items-center gap-2 text-sm">
<PoundSterling className="h-4 w-4 text-muted-foreground" />
<span><strong>£{property.qmprice}</strong>/m²</span>
</div>
{property.listing_type !== 'BUY' && property.available_from && (
<div className="flex items-center gap-2 text-sm">
<Clock className="h-4 w-4 text-muted-foreground" />
<span>Available <strong>{property.available_from}</strong></span>
</div>
)}
{property.listing_type === 'BUY' && (
<div className="flex items-center gap-2 text-sm">
<Clock className="h-4 w-4 text-muted-foreground" />
<span>Seen <strong>{lastSeenDays}d</strong> ago</span>
</div>
)}
</div>
{/* Agency and last seen */}
<div className="flex items-center gap-2 mt-3 text-sm text-muted-foreground">
<Building className="h-4 w-4" />
<span>{property.agency}</span>
<span className="mx-1">•</span>
<span>Seen {lastSeenDays} days ago</span>
</div>
{/* Price history */}
{property.price_history.length > 1 && (
<div className="mt-3 pt-3 border-t">
<div className="text-xs font-medium text-muted-foreground mb-1">Price history</div>
<div className="space-y-0.5">
{property.price_history.slice(0, 5).map((entry) => (
<div key={entry.id} className="text-sm flex justify-between">
<span className="text-muted-foreground">{entry.last_seen.split('T')[0]}</span>
<span>£{entry.price.toLocaleString()}</span>
</div>
))}
</div>
</div>
)}
{/* Actions */}
<div className="mt-4">
<Button asChild className="w-full">
<a href={property.url} target="_blank" rel="noopener noreferrer">
View Listing
<ExternalLink className="ml-2 h-4 w-4" />
</a>
</Button>
</div>
</div>
);
}

46
crawler/frontend/src/components/Spinner.tsx
View file

@ -1,46 +0,0 @@
import { cn } from '@/lib/utils';
import { type VariantProps, cva } from 'class-variance-authority';
import { Loader2 } from 'lucide-react';
import React from 'react';
const spinnerVariants = cva('flex-col items-center justify-center', {
variants: {
show: {
true: 'flex',
false: 'hidden',
},
},
defaultVariants: {
show: true,
},
});
const loaderVariants = cva('animate-spin text-primary', {
variants: {
size: {
small: 'size-6',
medium: 'size-8',
large: 'size-12',
},
},
defaultVariants: {
size: 'medium',
},
});
interface SpinnerContentProps
extends VariantProps<typeof spinnerVariants>,
VariantProps<typeof loaderVariants> {
className?: string;
children?: React.ReactNode;
}
export function Spinner({ size, show, children, className }: SpinnerContentProps) {
return (
<span className={spinnerVariants({ show })}>
<Loader2 className={cn(loaderVariants({ size }), className)} />
{/* <span className="text-red-400">Loading with custom style</span> */}
{children}
</span>
);
}

128
crawler/frontend/src/components/StatsBar.tsx
View file

@ -1,128 +0,0 @@
import { BarChart3, MapPin, PoundSterling, Maximize2, List, Map as MapIcon } from 'lucide-react';
import { Button } from './ui/button';
import type { GeoJSONFeatureCollection, PropertyFeature } from '@/types';
export type ViewMode = 'map' | 'list' | 'split';
interface StatsBarProps {
listingData: GeoJSONFeatureCollection | null;
viewMode: ViewMode;
onViewModeChange: (mode: ViewMode) => void;
}
interface ListingStats {
count: number;
avgPrice: number;
avgPricePerSqm: number;
avgSize: number;
}
function calculateStats(data: GeoJSONFeatureCollection | null): ListingStats {
if (!data || data.features.length === 0) {
return { count: 0, avgPrice: 0, avgPricePerSqm: 0, avgSize: 0 };
}
const features = data.features;
const count = features.length;
const validPrices = features
.map((f: PropertyFeature) => f.properties.total_price)
.filter((p): p is number => typeof p === 'number' && p > 0);
const validPricesPerSqm = features
.map((f: PropertyFeature) => f.properties.qmprice)
.filter((p): p is number => typeof p === 'number' && p > 0);
const validSizes = features
.map((f: PropertyFeature) => f.properties.qm)
.filter((s): s is number => typeof s === 'number' && s > 0);
const avgPrice = validPrices.length > 0
? validPrices.reduce((a, b) => a + b, 0) / validPrices.length
: 0;
const avgPricePerSqm = validPricesPerSqm.length > 0
? validPricesPerSqm.reduce((a, b) => a + b, 0) / validPricesPerSqm.length
: 0;
const avgSize = validSizes.length > 0
? validSizes.reduce((a, b) => a + b, 0) / validSizes.length
: 0;
return { count, avgPrice, avgPricePerSqm, avgSize };
}
function formatCurrency(value: number): string {
if (value >= 1000) {
return `£${(value / 1000).toFixed(1)}k`;
}
return `£${Math.round(value)}`;
}
export function StatsBar({ listingData, viewMode, onViewModeChange }: StatsBarProps) {
const stats = calculateStats(listingData);
return (
<div className="flex items-center justify-between px-4 py-2 bg-muted/50 border-t text-sm">
{/* Stats */}
<div className="flex items-center gap-4 text-muted-foreground">
<div className="flex items-center gap-1.5">
<MapPin className="h-4 w-4" />
<span className="font-medium text-foreground">{stats.count.toLocaleString()}</span>
<span className="hidden sm:inline">listings</span>
</div>
{stats.avgPrice > 0 && (
<>
<div className="hidden md:flex items-center gap-1.5">
<PoundSterling className="h-4 w-4" />
<span>Avg: <span className="font-medium text-foreground">{formatCurrency(stats.avgPrice)}</span></span>
</div>
<div className="hidden lg:flex items-center gap-1.5">
<BarChart3 className="h-4 w-4" />
<span>Avg £/m²: <span className="font-medium text-foreground">{formatCurrency(stats.avgPricePerSqm)}</span></span>
</div>
<div className="hidden lg:flex items-center gap-1.5">
<Maximize2 className="h-4 w-4" />
<span>Avg: <span className="font-medium text-foreground">{Math.round(stats.avgSize)} m²</span></span>
</div>
</>
)}
</div>
{/* View Mode Toggle */}
<div className="flex items-center gap-1 bg-background rounded-md border p-0.5">
<Button
variant={viewMode === 'map' ? 'secondary' : 'ghost'}
size="sm"
className="h-7 px-2"
onClick={() => onViewModeChange('map')}
>
<MapIcon className="h-4 w-4" />
<span className="hidden sm:inline ml-1">Map</span>
</Button>
<Button
variant={viewMode === 'list' ? 'secondary' : 'ghost'}
size="sm"
className="h-7 px-2"
onClick={() => onViewModeChange('list')}
>
<List className="h-4 w-4" />
<span className="hidden sm:inline ml-1">List</span>
</Button>
<Button
variant={viewMode === 'split' ? 'secondary' : 'ghost'}
size="sm"
className="h-7 px-2 hidden md:flex"
onClick={() => onViewModeChange('split')}
>
<div className="flex gap-0.5">
<div className="w-2 h-4 bg-current rounded-sm opacity-60" />
<div className="w-2 h-4 border border-current rounded-sm" />
</div>
<span className="hidden sm:inline ml-1">Split</span>
</Button>
</div>
</div>
);
}

47
crawler/frontend/src/components/StreamingProgressBar.tsx
View file

@ -1,47 +0,0 @@
import { Loader2 } from 'lucide-react';
import type { StreamingProgress } from '@/services';
interface StreamingProgressBarProps {
progress: StreamingProgress | null;
isLoading: boolean;
}
export function StreamingProgressBar({ progress, isLoading }: StreamingProgressBarProps) {
if (!isLoading) return null;
return (
<div className="absolute top-0 left-0 right-0 z-10 bg-background/95 backdrop-blur-sm border-b px-4 py-2">
<div className="flex items-center gap-3">
<Loader2 className="h-4 w-4 animate-spin text-primary" />
<div className="flex-1">
<div className="flex items-center justify-between text-sm">
<span className="font-medium">
{progress
? `Loading listings...`
: 'Loading...'}
</span>
{progress && (
<span className="text-muted-foreground">
{progress.count.toLocaleString()}
{progress.total ? ` / ${progress.total.toLocaleString()}` : ''} loaded
</span>
)}
</div>
{progress && (
<div className="mt-1 h-1.5 w-full bg-primary/20 rounded-full overflow-hidden">
<div
className="h-full bg-primary transition-all duration-300 ease-out rounded-full"
style={{
width: progress.total
? `${Math.min((progress.count / progress.total) * 100, 100)}%`
: '100%',
animation: progress.total ? undefined : 'pulse 1.5s ease-in-out infinite',
}}
/>
</div>
)}
</div>
</div>
</div>
);
}

294
crawler/frontend/src/components/TaskIndicator.tsx
View file

@ -1,294 +0,0 @@
import { getUser } from '@/auth/authService';
import { getStoredPasskeyUser } from '@/auth/passkeyService';
import { fromOidcUser, type AuthUser } from '@/auth/types';
import { POLLING_INTERVALS } from '@/constants';
import { fetchTaskStatus, cancelTask, clearAllTasks } from '@/services';
import { TaskStatus, type TaskResult } from '@/types';
import { useEffect, useState } from 'react';
import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger } from './ui/tooltip';
import { Button } from './ui/button';
import { Loader2, CheckCircle2, XCircle, X, Trash2 } from 'lucide-react';
import { TaskProgressDrawer } from './TaskProgressDrawer';
interface TaskIndicatorProps {
taskID: string | null;
onTaskCancelled?: () => void;
}
export function TaskIndicator({ taskID, onTaskCancelled }: TaskIndicatorProps) {
const [user, setUser] = useState<AuthUser | null>(null);
const [progressPercentage, setProgressPercentage] = useState<number>(0);
const [processed, setProcessed] = useState<number | null>(null);
const [total, setTotal] = useState<number | null>(null);
const [taskStatus, setTaskStatus] = useState<TaskStatus | null>(null);
const [taskResult, setTaskResult] = useState<TaskResult | null>(null);
const [isCancelling, setIsCancelling] = useState(false);
const [isClearing, setIsClearing] = useState(false);
const [drawerOpen, setDrawerOpen] = useState(false);
useEffect(() => {
const passkeyUser = getStoredPasskeyUser();
if (passkeyUser) {
setUser(passkeyUser);
} else {
getUser().then((oidcUser) => {
if (oidcUser) setUser(fromOidcUser(oidcUser));
});
}
}, []);
useEffect(() => {
if (!user || !taskID) {
setTaskStatus(null);
setTaskResult(null);
return;
}
// Reset state for new task
setTaskStatus(TaskStatus.PENDING);
setProgressPercentage(0);
setProcessed(null);
setTotal(null);
setTaskResult(null);
const pollTaskStatus = async () => {
try {
const data = await fetchTaskStatus(user, taskID);
const status = data.status as TaskStatus;
setTaskStatus(status);
if (status === TaskStatus.SUCCESS) {
setProgressPercentage(100);
// Parse final result for the drawer to show completed state.
// Only update taskResult if the new result has phase info;
// otherwise keep the last in-progress result which has richer data
// than the bare SUCCESS return value.
if (data.result) {
try {
const parsedResult: TaskResult = JSON.parse(data.result);
if (parsedResult.phase) {
setTaskResult(parsedResult);
}
} catch {
// Ignore parsing errors
}
}
return true; // Stop polling
}
if (status === TaskStatus.FAILURE || status === TaskStatus.REVOKED) {
return true; // Stop polling
}
// Parse progress for in-progress tasks
if (data.result) {
try {
const parsedResult: TaskResult = JSON.parse(data.result);
// Only update taskResult if the parsed data has a phase field.
// This prevents blanking the drawer when the backend sends a
// state update without phase info (e.g. during brief transitions).
if (parsedResult.phase) {
setTaskResult(parsedResult);
}
// Only update progress/processed/total when the fields are
// actually present — otherwise keep the previous values so
// the UI doesn't flash back to 0 during phase transitions.
if (parsedResult.progress !== undefined) {
setProgressPercentage(parsedResult.progress * 100);
}
if (parsedResult.processed !== undefined) {
setProcessed(parsedResult.processed);
}
if (parsedResult.total !== undefined) {
setTotal(parsedResult.total);
}
} catch {
// Ignore parsing errors
}
}
return false; // Continue polling
} catch {
setTaskStatus(TaskStatus.FAILURE);
return true; // Stop polling on error
}
};
// Initial poll
pollTaskStatus();
const interval = setInterval(async () => {
const shouldStop = await pollTaskStatus();
if (shouldStop) {
clearInterval(interval);
}
}, POLLING_INTERVALS.TASK_STATUS_MS);
return () => clearInterval(interval);
}, [taskID, user]);
const handleCancel = async () => {
if (!user || !taskID || isCancelling) return;
setIsCancelling(true);
try {
const result = await cancelTask(user, taskID);
if (result.success) {
setTaskStatus(TaskStatus.REVOKED);
onTaskCancelled?.();
}
} catch {
// Ignore cancel errors
} finally {
setIsCancelling(false);
}
};
const handleClearAll = async () => {
if (!user || isClearing) return;
setIsClearing(true);
try {
const result = await clearAllTasks(user);
if (result.success) {
setTaskStatus(null);
setTaskResult(null);
onTaskCancelled?.();
}
} catch {
// Ignore clear errors
} finally {
setIsClearing(false);
}
};
if (!taskID || !taskStatus) {
return null;
}
const isInProgress = taskStatus !== TaskStatus.SUCCESS &&
taskStatus !== TaskStatus.FAILURE &&
taskStatus !== TaskStatus.REVOKED;
const getStatusIcon = () => {
if (isInProgress) {
return <Loader2 className="h-3.5 w-3.5 animate-spin text-blue-500" />;
}
if (taskStatus === TaskStatus.SUCCESS) {
return <CheckCircle2 className="h-3.5 w-3.5 text-green-500" />;
}
return <XCircle className="h-3.5 w-3.5 text-red-500" />;
};
const getProgressText = () => {
if (processed !== null && total !== null && total > 0) {
return `${processed} / ${total}`;
}
if (taskResult?.phase && taskResult.phase !== 'processing') {
const phaseLabels: Record<string, string> = {
splitting: 'Splitting',
splitting_complete: 'Split done',
fetching: 'Fetching',
};
return phaseLabels[taskResult.phase] ?? `${Math.round(progressPercentage)}%`;
}
return `${Math.round(progressPercentage)}%`;
};
const getTooltipContent = () => {
if (isInProgress) {
if (processed !== null && total !== null && total > 0) {
return `Processing: ${processed} / ${total} listings (${Math.round(progressPercentage)}%) — click for details`;
}
return `Task running: ${getProgressText()} — click for details`;
}
if (taskStatus === TaskStatus.SUCCESS) {
return 'Task completed successfully — click for details';
}
if (taskStatus === TaskStatus.REVOKED) {
return 'Task was cancelled';
}
return 'Task failed';
};
return (
<TooltipProvider>
<div className="flex items-center gap-2">
<Tooltip>
<TooltipTrigger asChild>
<div
className="flex items-center gap-2 cursor-pointer"
onClick={() => setDrawerOpen(true)}
>
{getStatusIcon()}
{isInProgress && (
<div className="flex items-center gap-2">
<div className="w-24 h-1.5 bg-primary/20 rounded-full overflow-hidden hidden sm:block">
<div
className="h-full bg-primary transition-all duration-300 ease-out rounded-full"
style={{ width: `${Math.min(progressPercentage, 100)}%` }}
/>
</div>
<span className="text-xs text-muted-foreground hidden sm:inline min-w-[60px]">
{getProgressText()}
</span>
</div>
)}
{!isInProgress && (
<span className="text-xs text-muted-foreground hidden sm:inline">
{taskStatus}
</span>
)}
</div>
</TooltipTrigger>
<TooltipContent side="bottom">
<p>{getTooltipContent()}</p>
<p className="text-xs text-muted-foreground mt-1">ID: {taskID.slice(0, 8)}...</p>
</TooltipContent>
</Tooltip>
{isInProgress && (
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="ghost"
size="icon"
onClick={handleCancel}
disabled={isCancelling}
className="h-6 w-6 text-muted-foreground hover:text-destructive"
>
<X className="h-3 w-3" />
</Button>
</TooltipTrigger>
<TooltipContent side="bottom">
<p>Cancel task</p>
</TooltipContent>
</Tooltip>
)}
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="ghost"
size="icon"
onClick={handleClearAll}
disabled={isClearing}
className="h-6 w-6 text-muted-foreground hover:text-destructive"
>
<Trash2 className="h-3 w-3" />
</Button>
</TooltipTrigger>
<TooltipContent side="bottom">
<p>Clear all tasks</p>
</TooltipContent>
</Tooltip>
</div>
<TaskProgressDrawer
open={drawerOpen}
onOpenChange={setDrawerOpen}
taskResult={taskResult}
taskStatus={taskStatus}
taskID={taskID}
onCancel={handleCancel}
isCancelling={isCancelling}
/>
</TooltipProvider>
);
}

367
crawler/frontend/src/components/TaskProgressDrawer.tsx
View file

@ -1,367 +0,0 @@
import { TaskStatus, type TaskPhase, type TaskResult } from '@/types';
import {
Sheet,
SheetContent,
SheetHeader,
SheetTitle,
SheetDescription,
SheetFooter,
} from './ui/sheet';
import { Button } from './ui/button';
import { CheckCircle2, Circle, Loader2, XCircle } from 'lucide-react';
import { useEffect, useRef } from 'react';
interface TaskProgressDrawerProps {
open: boolean;
onOpenChange: (open: boolean) => void;
taskResult: TaskResult | null;
taskStatus: TaskStatus | null;
taskID: string | null;
onCancel: () => void;
isCancelling: boolean;
}
const PHASES: { key: TaskPhase; label: string }[] = [
{ key: 'splitting', label: 'Splitting queries' },
{ key: 'fetching', label: 'Fetching & processing' },
{ key: 'processing', label: 'Processing remaining' },
];
function getPhaseIndex(phase: TaskPhase | undefined): number {
if (!phase) return -1;
if (phase === 'splitting_complete') return 1; // splitting done, fetching is next
if (phase === 'completed') return PHASES.length;
return PHASES.findIndex((p) => p.key === phase);
}
function formatEta(seconds: number | undefined): string {
if (seconds === undefined || seconds <= 0) return '';
const mins = Math.floor(seconds / 60);
const secs = Math.round(seconds % 60);
if (mins > 0) {
return `~${mins}m ${secs}s remaining`;
}
return `~${secs}s remaining`;
}
function StatusBadge({ status }: { status: TaskStatus | null }) {
if (!status) return null;
const isInProgress =
status !== TaskStatus.SUCCESS &&
status !== TaskStatus.FAILURE &&
status !== TaskStatus.REVOKED;
if (isInProgress) {
return (
<span className="inline-flex items-center gap-1 rounded-full bg-blue-100 px-2 py-0.5 text-xs font-medium text-blue-700">
<Loader2 className="h-3 w-3 animate-spin" />
Running
</span>
);
}
if (status === TaskStatus.SUCCESS) {
return (
<span className="inline-flex items-center gap-1 rounded-full bg-green-100 px-2 py-0.5 text-xs font-medium text-green-700">
<CheckCircle2 className="h-3 w-3" />
Complete
</span>
);
}
if (status === TaskStatus.REVOKED) {
return (
<span className="inline-flex items-center gap-1 rounded-full bg-yellow-100 px-2 py-0.5 text-xs font-medium text-yellow-700">
<XCircle className="h-3 w-3" />
Cancelled
</span>
);
}
return (
<span className="inline-flex items-center gap-1 rounded-full bg-red-100 px-2 py-0.5 text-xs font-medium text-red-700">
<XCircle className="h-3 w-3" />
Failed
</span>
);
}
function PhaseTimeline({
currentPhase,
taskStatus,
}: {
currentPhase: TaskPhase | undefined;
taskStatus: TaskStatus | null;
}) {
const isTerminal =
taskStatus === TaskStatus.SUCCESS ||
taskStatus === TaskStatus.FAILURE ||
taskStatus === TaskStatus.REVOKED;
const activeIdx = isTerminal ? PHASES.length : getPhaseIndex(currentPhase);
return (
<div className="flex flex-col gap-1">
{PHASES.map((phase, idx) => {
const isCompleted = idx < activeIdx;
const isActive = idx === activeIdx && !isTerminal;
const isFuture = idx > activeIdx;
return (
<div key={phase.key} className="flex items-center gap-2">
{isCompleted && (
<CheckCircle2 className="h-4 w-4 text-green-500 shrink-0" />
)}
{isActive && (
<Loader2 className="h-4 w-4 animate-spin text-blue-500 shrink-0" />
)}
{isFuture && (
<Circle className="h-4 w-4 text-muted-foreground/40 shrink-0" />
)}
<span
className={
isActive
? 'text-sm font-medium text-foreground'
: isCompleted
? 'text-sm text-muted-foreground'
: 'text-sm text-muted-foreground/40'
}
>
{phase.label}
</span>
</div>
);
})}
</div>
);
}
function CounterRow({ label, value, total }: { label: string; value?: number; total?: number }) {
if (value === undefined) return null;
return (
<div className="flex justify-between text-sm">
<span className="text-muted-foreground">{label}</span>
<span className="font-mono tabular-nums">
{value}
{total !== undefined && ` / ${total}`}
</span>
</div>
);
}
function PhaseDetails({ result }: { result: TaskResult }) {
const phase = result.phase;
if (phase === 'splitting' || phase === 'splitting_complete') {
return (
<div className="rounded-md border p-3 space-y-1">
<p className="text-xs font-medium text-muted-foreground uppercase tracking-wide mb-2">
Splitting
</p>
<CounterRow
label="Subqueries probed"
value={result.subqueries_probed}
total={result.subqueries_initial}
/>
{result.subqueries_total !== undefined && (
<CounterRow label="Final subqueries" value={result.subqueries_total} />
)}
{result.estimated_results !== undefined && (
<CounterRow label="Estimated results" value={result.estimated_results} />
)}
</div>
);
}
if (phase === 'fetching') {
return (
<div className="rounded-md border p-3 space-y-1">
<p className="text-xs font-medium text-muted-foreground uppercase tracking-wide mb-2">
{result.fetching_done ? 'Fetching complete' : 'Fetching & processing'}
</p>
<CounterRow
label="Subqueries completed"
value={result.subqueries_completed}
total={result.subqueries_total}
/>
<CounterRow label="IDs collected" value={result.ids_collected} />
<CounterRow label="Pages fetched" value={result.pages_fetched} />
{(result.details_fetched !== undefined && result.details_fetched > 0) && (
<>
<div className="border-t my-2" />
<CounterRow
label="Details fetched"
value={result.details_fetched}
total={result.total}
/>
<CounterRow label="Images downloaded" value={result.images_downloaded} />
<CounterRow label="OCR completed" value={result.ocr_completed} />
{(result.failed ?? 0) > 0 && (
<div className="flex justify-between text-sm">
<span className="text-red-500">Failed</span>
<span className="font-mono tabular-nums text-red-500">{result.failed}</span>
</div>
)}
</>
)}
</div>
);
}
if (phase === 'processing') {
return (
<div className="rounded-md border p-3 space-y-1">
<p className="text-xs font-medium text-muted-foreground uppercase tracking-wide mb-2">
Processing
</p>
<CounterRow
label="Details fetched"
value={result.details_fetched}
total={result.total}
/>
<CounterRow label="Images downloaded" value={result.images_downloaded} />
<CounterRow label="OCR completed" value={result.ocr_completed} />
{(result.failed ?? 0) > 0 && (
<div className="flex justify-between text-sm">
<span className="text-red-500">Failed</span>
<span className="font-mono tabular-nums text-red-500">{result.failed}</span>
</div>
)}
</div>
);
}
return null;
}
function LogViewer({ logs }: { logs: string[] }) {
const scrollRef = useRef<HTMLDivElement>(null);
const isAutoScrolling = useRef(true);
const handleScroll = () => {
const el = scrollRef.current;
if (!el) return;
const atBottom = el.scrollHeight - el.scrollTop - el.clientHeight < 30;
isAutoScrolling.current = atBottom;
};
useEffect(() => {
if (isAutoScrolling.current && scrollRef.current) {
scrollRef.current.scrollTop = scrollRef.current.scrollHeight;
}
}, [logs]);
return (
<div
ref={scrollRef}
onScroll={handleScroll}
className="rounded-md bg-zinc-950 p-3 overflow-y-auto font-mono text-[11px] leading-4 text-zinc-300 min-h-[100px] h-full"
>
{logs.length === 0 ? (
<span className="text-zinc-500 italic">Waiting for logs...</span>
) : (
logs.map((line, i) => (
<div key={i} className="whitespace-pre-wrap break-all">
{line}
</div>
))
)}
</div>
);
}
export function TaskProgressDrawer({
open,
onOpenChange,
taskResult,
taskStatus,
taskID,
onCancel,
isCancelling,
}: TaskProgressDrawerProps) {
const isInProgress =
taskStatus !== null &&
taskStatus !== TaskStatus.SUCCESS &&
taskStatus !== TaskStatus.FAILURE &&
taskStatus !== TaskStatus.REVOKED;
const progressPercent = taskResult
? Math.min((taskResult.progress ?? 0) * 100, 100)
: 0;
return (
<Sheet open={open} onOpenChange={onOpenChange}>
<SheetContent side="right" className="flex flex-col w-full sm:!max-w-lg">
<SheetHeader>
<div className="flex items-center justify-between pr-6">
<SheetTitle>Crawl Job Progress</SheetTitle>
<StatusBadge status={taskStatus} />
</div>
{taskID && (
<SheetDescription>
Task ID: {taskID.slice(0, 8)}...
</SheetDescription>
)}
</SheetHeader>
{/* Fixed top section: timeline + counters + progress */}
<div className="space-y-3 px-4 shrink-0">
<PhaseTimeline
currentPhase={taskResult?.phase}
taskStatus={taskStatus}
/>
{taskResult && <PhaseDetails result={taskResult} />}
{taskResult && (taskResult.phase === 'processing' || taskResult.phase === 'fetching') && (taskResult.total ?? 0) > 0 && (
<div className="space-y-1">
<div className="w-full h-2 bg-primary/20 rounded-full overflow-hidden">
<div
className="h-full bg-primary transition-all duration-300 ease-out rounded-full"
style={{ width: `${progressPercent}%` }}
/>
</div>
<div className="flex justify-between text-xs text-muted-foreground">
<span>
{taskResult.processed ?? 0} / {taskResult.total ?? '?'}
</span>
<span>{formatEta(taskResult.eta_seconds)}</span>
</div>
</div>
)}
{taskResult?.message && (
<p className="text-sm text-muted-foreground">{taskResult.message}</p>
)}
</div>
{/* Log viewer fills remaining space */}
<div className="flex-1 min-h-0 flex flex-col gap-1 px-4 pb-2">
<p className="text-xs font-medium text-muted-foreground uppercase tracking-wide shrink-0">
Worker Logs
</p>
<div className="flex-1 min-h-0">
<LogViewer logs={taskResult?.logs ?? []} />
</div>
</div>
{isInProgress && (
<SheetFooter>
<Button
variant="destructive"
onClick={onCancel}
disabled={isCancelling}
className="w-full"
>
{isCancelling ? (
<>
<Loader2 className="mr-2 h-4 w-4 animate-spin" />
Cancelling...
</>
) : (
'Cancel Job'
)}
</Button>
</SheetFooter>
)}
</SheetContent>
</Sheet>
);
}

111
crawler/frontend/src/components/ui/DatePicker.tsx
View file

@ -1,111 +0,0 @@
"use client"
import { parseDate } from "chrono-node"
import { CalendarIcon } from "lucide-react"
import * as React from "react"
import { Button } from "@/components/ui/button"
import { Calendar } from "@/components/ui/calendar"
import { Input } from "@/components/ui/input"
import {
Popover,
PopoverContent,
PopoverTrigger,
} from "@/components/ui/popover"
function formatDate(date: Date | undefined) {
if (!date) {
return ""
}
return date.toLocaleDateString("en-US", {
day: "2-digit",
month: "long",
year: "numeric",
})
}
export function Calendar29(
props: {
onSelect?: (date?: Date) => void,
selected?: Date | undefined,
rawInputValue?: string | undefined
onChangeRawInputValue?: (rawInputValue: string) => void
}
) {
const [open, setOpen] = React.useState(false)
const [value, setValue] = React.useState(props.rawInputValue ?? "now")
const [date, setDate] = React.useState<Date | undefined>(
parseDate(value) || undefined
)
const [month, setMonth] = React.useState<Date | undefined>(date)
return (
<div className="flex flex-col gap-3">
<div className="relative flex gap-2">
<Input
id="date"
value={props.rawInputValue !== undefined ? props.rawInputValue : value}
placeholder="Tomorrow or next week"
className="bg-background pr-10"
onChange={(e) => {
setValue(e.target.value)
if (props.onChangeRawInputValue) {
props.onChangeRawInputValue(e.target.value)
}
const date = parseDate(e.target.value)
if (date) {
setDate(date)
setMonth(date)
if (props.onSelect) {
props.onSelect(date)
}
}
}}
onKeyDown={(e) => {
if (e.key === "ArrowDown") {
e.preventDefault()
setOpen(true)
}
}}
/>
<Popover open={open} onOpenChange={setOpen}>
<PopoverTrigger asChild>
<Button
id="date-picker"
variant="ghost"
className="absolute top-1/2 right-2 size-6 -translate-y-1/2"
>
<CalendarIcon className="size-3.5" />
<span className="sr-only">Select date</span>
</Button>
</PopoverTrigger>
<PopoverContent className="w-auto overflow-hidden p-0" align="end">
<Calendar
mode="single"
selected={date}
captionLayout="dropdown"
month={month}
onMonthChange={setMonth}
onSelect={(date) => {
setDate(date)
setValue(formatDate(date))
if (props.onChangeRawInputValue) {
props.onChangeRawInputValue(formatDate(date))
}
setOpen(false)
if (props.onSelect) {
props.onSelect(date)
}
}}
/>
</PopoverContent>
</Popover>
</div>
<div className="text-muted-foreground px-1 text-sm">
<span className="font-medium">{formatDate(date)}</span>.
</div>
</div>
)
}

56
crawler/frontend/src/components/ui/accordion.tsx
View file

@ -1,56 +0,0 @@
"use client"
import * as React from "react"
import * as AccordionPrimitive from "@radix-ui/react-accordion"
import { ChevronDown } from "lucide-react"
import { cn } from "@/lib/utils"
const Accordion = AccordionPrimitive.Root
const AccordionItem = React.forwardRef<
React.ComponentRef<typeof AccordionPrimitive.Item>,
React.ComponentPropsWithoutRef<typeof AccordionPrimitive.Item>
>(({ className, ...props }, ref) => (
<AccordionPrimitive.Item
ref={ref}
className={cn("border-b", className)}
{...props}
/>
))
AccordionItem.displayName = "AccordionItem"
const AccordionTrigger = React.forwardRef<
React.ComponentRef<typeof AccordionPrimitive.Trigger>,
React.ComponentPropsWithoutRef<typeof AccordionPrimitive.Trigger>
>(({ className, children, ...props }, ref) => (
<AccordionPrimitive.Header className="flex">
<AccordionPrimitive.Trigger
ref={ref}
className={cn(
"flex flex-1 items-center justify-between py-4 text-sm font-medium transition-all hover:underline text-left [&[data-state=open]>svg]:rotate-180",
className
)}
{...props}
>
{children}
<ChevronDown className="h-4 w-4 shrink-0 text-muted-foreground transition-transform duration-200" />
</AccordionPrimitive.Trigger>
</AccordionPrimitive.Header>
))
AccordionTrigger.displayName = AccordionPrimitive.Trigger.displayName
const AccordionContent = React.forwardRef<
React.ComponentRef<typeof AccordionPrimitive.Content>,
React.ComponentPropsWithoutRef<typeof AccordionPrimitive.Content>
>(({ className, children, ...props }, ref) => (
<AccordionPrimitive.Content
ref={ref}
className="overflow-hidden text-sm data-[state=closed]:animate-accordion-up data-[state=open]:animate-accordion-down"
{...props}
>
<div className={cn("pb-4 pt-0", className)}>{children}</div>
</AccordionPrimitive.Content>
))
AccordionContent.displayName = AccordionPrimitive.Content.displayName
export { Accordion, AccordionItem, AccordionTrigger, AccordionContent }

155
crawler/frontend/src/components/ui/alert-dialog.tsx
View file

@ -1,155 +0,0 @@
import * as React from "react"
import * as AlertDialogPrimitive from "@radix-ui/react-alert-dialog"
import { cn } from "@/lib/utils"
import { buttonVariants } from "@/components/ui/button"
function AlertDialog({
...props
}: React.ComponentProps<typeof AlertDialogPrimitive.Root>) {
return <AlertDialogPrimitive.Root data-slot="alert-dialog" {...props} />
}
function AlertDialogTrigger({
...props
}: React.ComponentProps<typeof AlertDialogPrimitive.Trigger>) {
return (
<AlertDialogPrimitive.Trigger data-slot="alert-dialog-trigger" {...props} />
)
}
function AlertDialogPortal({
...props
}: React.ComponentProps<typeof AlertDialogPrimitive.Portal>) {
return (
<AlertDialogPrimitive.Portal data-slot="alert-dialog-portal" {...props} />
)
}
function AlertDialogOverlay({
className,
...props
}: React.ComponentProps<typeof AlertDialogPrimitive.Overlay>) {
return (
<AlertDialogPrimitive.Overlay
data-slot="alert-dialog-overlay"
className={cn(
"data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 fixed inset-0 z-50 bg-black/50",
className
)}
{...props}
/>
)
}
function AlertDialogContent({
className,
...props
}: React.ComponentProps<typeof AlertDialogPrimitive.Content>) {
return (
<AlertDialogPortal>
<AlertDialogOverlay />
<AlertDialogPrimitive.Content
data-slot="alert-dialog-content"
className={cn(
"bg-background data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 fixed top-[50%] left-[50%] z-50 grid w-full max-w-[calc(100%-2rem)] translate-x-[-50%] translate-y-[-50%] gap-4 rounded-lg border p-6 shadow-lg duration-200 sm:max-w-lg",
className
)}
{...props}
/>
</AlertDialogPortal>
)
}
function AlertDialogHeader({
className,
...props
}: React.ComponentProps<"div">) {
return (
<div
data-slot="alert-dialog-header"
className={cn("flex flex-col gap-2 text-center sm:text-left", className)}
{...props}
/>
)
}
function AlertDialogFooter({
className,
...props
}: React.ComponentProps<"div">) {
return (
<div
data-slot="alert-dialog-footer"
className={cn(
"flex flex-col-reverse gap-2 sm:flex-row sm:justify-end",
className
)}
{...props}
/>
)
}
function AlertDialogTitle({
className,
...props
}: React.ComponentProps<typeof AlertDialogPrimitive.Title>) {
return (
<AlertDialogPrimitive.Title
data-slot="alert-dialog-title"
className={cn("text-lg font-semibold", className)}
{...props}
/>
)
}
function AlertDialogDescription({
className,
...props
}: React.ComponentProps<typeof AlertDialogPrimitive.Description>) {
return (
<AlertDialogPrimitive.Description
data-slot="alert-dialog-description"
className={cn("text-muted-foreground text-sm", className)}
{...props}
/>
)
}
function AlertDialogAction({
className,
...props
}: React.ComponentProps<typeof AlertDialogPrimitive.Action>) {
return (
<AlertDialogPrimitive.Action
className={cn(buttonVariants(), className)}
{...props}
/>
)
}
function AlertDialogCancel({
className,
...props
}: React.ComponentProps<typeof AlertDialogPrimitive.Cancel>) {
return (
<AlertDialogPrimitive.Cancel
className={cn(buttonVariants({ variant: "outline" }), className)}
{...props}
/>
)
}
export {
AlertDialog,
AlertDialogPortal,
AlertDialogOverlay,
AlertDialogTrigger,
AlertDialogContent,
AlertDialogHeader,
AlertDialogFooter,
AlertDialogTitle,
AlertDialogDescription,
AlertDialogAction,
AlertDialogCancel,
}

90
crawler/frontend/src/components/ui/breadcrumb.tsx
View file

@ -1,90 +0,0 @@
import * as React from "react"
import { Slot } from "@radix-ui/react-slot"
import { ChevronRight } from "lucide-react"
import { cn } from "@/lib/utils"
function Breadcrumb({ ...props }: React.ComponentProps<"nav">) {
return <nav aria-label="breadcrumb" data-slot="breadcrumb" {...props} />
}
function BreadcrumbList({ className, ...props }: React.ComponentProps<"ol">) {
return (
<ol
data-slot="breadcrumb-list"
className={cn(
"text-muted-foreground flex flex-wrap items-center gap-1.5 text-sm break-words sm:gap-2.5",
className
)}
{...props}
/>
)
}
function BreadcrumbItem({ className, ...props }: React.ComponentProps<"li">) {
return (
<li
data-slot="breadcrumb-item"
className={cn("inline-flex items-center gap-1.5", className)}
{...props}
/>
)
}
function BreadcrumbLink({
asChild,
className,
...props
}: React.ComponentProps<"a"> & {
asChild?: boolean
}) {
const Comp = asChild ? Slot : "a"
return (
<Comp
data-slot="breadcrumb-link"
className={cn("hover:text-foreground transition-colors", className)}
{...props}
/>
)
}
function BreadcrumbPage({ className, ...props }: React.ComponentProps<"span">) {
return (
<span
data-slot="breadcrumb-page"
role="link"
aria-disabled="true"
aria-current="page"
className={cn("text-foreground font-normal", className)}
{...props}
/>
)
}
function BreadcrumbSeparator({
children,
className,
...props
}: React.ComponentProps<"li">) {
return (
<li
data-slot="breadcrumb-separator"
role="presentation"
aria-hidden="true"
className={cn("[&>svg]:size-3.5", className)}
{...props}
>
{children ?? <ChevronRight />}
</li>
)
}
export {
Breadcrumb,
BreadcrumbList,
BreadcrumbItem,
BreadcrumbLink,
BreadcrumbPage,
BreadcrumbSeparator,
}

59
crawler/frontend/src/components/ui/button.tsx
View file

@ -1,59 +0,0 @@
import * as React from "react"
import { Slot } from "@radix-ui/react-slot"
import { cva, type VariantProps } from "class-variance-authority"
import { cn } from "@/lib/utils"
const buttonVariants = cva(
"inline-flex items-center justify-center gap-2 whitespace-nowrap rounded-md text-sm font-medium transition-all disabled:pointer-events-none disabled:opacity-50 [&_svg]:pointer-events-none [&_svg:not([class*='size-'])]:size-4 shrink-0 [&_svg]:shrink-0 outline-none focus-visible:border-ring focus-visible:ring-ring/50 focus-visible:ring-[3px] aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40 aria-invalid:border-destructive",
{
variants: {
variant: {
default:
"bg-primary text-primary-foreground shadow-xs hover:bg-primary/90",
destructive:
"bg-destructive text-white shadow-xs hover:bg-destructive/90 focus-visible:ring-destructive/20 dark:focus-visible:ring-destructive/40 dark:bg-destructive/60",
outline:
"border bg-background shadow-xs hover:bg-accent hover:text-accent-foreground dark:bg-input/30 dark:border-input dark:hover:bg-input/50",
secondary:
"bg-secondary text-secondary-foreground shadow-xs hover:bg-secondary/80",
ghost:
"hover:bg-accent hover:text-accent-foreground dark:hover:bg-accent/50",
link: "text-primary underline-offset-4 hover:underline",
},
size: {
default: "h-9 px-4 py-2 has-[>svg]:px-3",
sm: "h-8 rounded-md gap-1.5 px-3 has-[>svg]:px-2.5",
lg: "h-10 rounded-md px-6 has-[>svg]:px-4",
icon: "size-9",
},
},
defaultVariants: {
variant: "default",
size: "default",
},
}
)
function Button({
className,
variant,
size,
asChild = false,
...props
}: React.ComponentProps<"button"> &
VariantProps<typeof buttonVariants> & {
asChild?: boolean
}) {
const Comp = asChild ? Slot : "button"
return (
<Comp
data-slot="button"
className={cn(buttonVariants({ variant, size, className }))}
{...props}
/>
)
}
export { Button, buttonVariants }

208
crawler/frontend/src/components/ui/calendar.tsx
View file

@ -1,208 +0,0 @@
import * as React from "react"
import {
ChevronDownIcon,
ChevronLeftIcon,
ChevronRightIcon,
} from "lucide-react"
import { DayButton, DayPicker, getDefaultClassNames } from "react-day-picker"
import { cn } from "@/lib/utils"
import { Button, buttonVariants } from "@/components/ui/button"
function Calendar({
className,
classNames,
showOutsideDays = true,
captionLayout = "label",
buttonVariant = "ghost",
formatters,
components,
...props
}: React.ComponentProps<typeof DayPicker> & {
buttonVariant?: React.ComponentProps<typeof Button>["variant"]
}) {
const defaultClassNames = getDefaultClassNames()
return (
<DayPicker
showOutsideDays={showOutsideDays}
className={cn(
"bg-background group/calendar p-3 [--cell-size:--spacing(8)] [[data-slot=card-content]_&]:bg-transparent [[data-slot=popover-content]_&]:bg-transparent",
String.raw`rtl:**:[.rdp-button\_next>svg]:rotate-180`,
String.raw`rtl:**:[.rdp-button\_previous>svg]:rotate-180`,
className
)}
captionLayout={captionLayout}
formatters={{
formatMonthDropdown: (date) =>
date.toLocaleString("default", { month: "short" }),
...formatters,
}}
classNames={{
root: cn("w-fit", defaultClassNames.root),
months: cn(
"flex gap-4 flex-col md:flex-row relative",
defaultClassNames.months
),
month: cn("flex flex-col w-full gap-4", defaultClassNames.month),
nav: cn(
"flex items-center gap-1 w-full absolute top-0 inset-x-0 justify-between",
defaultClassNames.nav
),
button_previous: cn(
buttonVariants({ variant: buttonVariant }),
"size-(--cell-size) aria-disabled:opacity-50 p-0 select-none",
defaultClassNames.button_previous
),
button_next: cn(
buttonVariants({ variant: buttonVariant }),
"size-(--cell-size) aria-disabled:opacity-50 p-0 select-none",
defaultClassNames.button_next
),
month_caption: cn(
"flex items-center justify-center h-(--cell-size) w-full px-(--cell-size)",
defaultClassNames.month_caption
),
dropdowns: cn(
"w-full flex items-center text-sm font-medium justify-center h-(--cell-size) gap-1.5",
defaultClassNames.dropdowns
),
dropdown_root: cn(
"relative has-focus:border-ring border border-input shadow-xs has-focus:ring-ring/50 has-focus:ring-[3px] rounded-md",
defaultClassNames.dropdown_root
),
dropdown: cn("absolute inset-0 opacity-0", defaultClassNames.dropdown),
caption_label: cn(
"select-none font-medium",
captionLayout === "label"
? "text-sm"
: "rounded-md pl-2 pr-1 flex items-center gap-1 text-sm h-8 [&>svg]:text-muted-foreground [&>svg]:size-3.5",
defaultClassNames.caption_label
),
table: "w-full border-collapse",
weekdays: cn("flex", defaultClassNames.weekdays),
weekday: cn(
"text-muted-foreground rounded-md flex-1 font-normal text-[0.8rem] select-none",
defaultClassNames.weekday
),
week: cn("flex w-full mt-2", defaultClassNames.week),
week_number_header: cn(
"select-none w-(--cell-size)",
defaultClassNames.week_number_header
),
week_number: cn(
"text-[0.8rem] select-none text-muted-foreground",
defaultClassNames.week_number
),
day: cn(
"relative w-full h-full p-0 text-center [&:first-child[data-selected=true]_button]:rounded-l-md [&:last-child[data-selected=true]_button]:rounded-r-md group/day aspect-square select-none",
defaultClassNames.day
),
range_start: cn(
"rounded-l-md bg-accent",
defaultClassNames.range_start
),
range_middle: cn("rounded-none", defaultClassNames.range_middle),
range_end: cn("rounded-r-md bg-accent", defaultClassNames.range_end),
today: cn(
"bg-accent text-accent-foreground rounded-md data-[selected=true]:rounded-none",
defaultClassNames.today
),
outside: cn(
"text-muted-foreground aria-selected:text-muted-foreground",
defaultClassNames.outside
),
disabled: cn(
"text-muted-foreground opacity-50",
defaultClassNames.disabled
),
hidden: cn("invisible", defaultClassNames.hidden),
...classNames,
}}
components={{
Root: ({ className, rootRef, ...props }) => {
return (
<div
data-slot="calendar"
ref={rootRef}
className={cn(className)}
{...props}
/>
)
},
Chevron: ({ className, orientation, ...props }) => {
if (orientation === "left") {
return (
<ChevronLeftIcon className={cn("size-4", className)} {...props} />
)
}
if (orientation === "right") {
return (
<ChevronRightIcon
className={cn("size-4", className)}
{...props}
/>
)
}
return (
<ChevronDownIcon className={cn("size-4", className)} {...props} />
)
},
DayButton: CalendarDayButton,
WeekNumber: ({ children, ...props }) => {
return (
<td {...props}>
<div className="flex size-(--cell-size) items-center justify-center text-center">
{children}
</div>
</td>
)
},
...components,
}}
{...props}
/>
)
}
function CalendarDayButton({
className,
day,
modifiers,
...props
}: React.ComponentProps<typeof DayButton>) {
const defaultClassNames = getDefaultClassNames()
const ref = React.useRef<HTMLButtonElement>(null)
React.useEffect(() => {
if (modifiers.focused) ref.current?.focus()
}, [modifiers.focused])
return (
<Button
ref={ref}
variant="ghost"
size="icon"
data-day={day.date.toLocaleDateString()}
data-selected-single={
modifiers.selected &&
!modifiers.range_start &&
!modifiers.range_end &&
!modifiers.range_middle
}
data-range-start={modifiers.range_start}
data-range-end={modifiers.range_end}
data-range-middle={modifiers.range_middle}
className={cn(
"data-[selected-single=true]:bg-primary data-[selected-single=true]:text-primary-foreground data-[range-middle=true]:bg-accent data-[range-middle=true]:text-accent-foreground data-[range-start=true]:bg-primary data-[range-start=true]:text-primary-foreground data-[range-end=true]:bg-primary data-[range-end=true]:text-primary-foreground group-data-[focused=true]/day:border-ring group-data-[focused=true]/day:ring-ring/50 dark:hover:text-accent-foreground flex aspect-square size-auto w-full min-w-(--cell-size) flex-col gap-1 leading-none font-normal group-data-[focused=true]/day:relative group-data-[focused=true]/day:z-10 group-data-[focused=true]/day:ring-[3px] data-[range-end=true]:rounded-md data-[range-end=true]:rounded-r-md data-[range-middle=true]:rounded-none data-[range-start=true]:rounded-md data-[range-start=true]:rounded-l-md [&>span]:text-xs [&>span]:opacity-70",
defaultClassNames.day,
className
)}
{...props}
/>
)
}
export { Calendar, CalendarDayButton }

29
crawler/frontend/src/components/ui/checkbox.tsx
View file

@ -1,29 +0,0 @@
"use client"
import * as React from "react"
import * as CheckboxPrimitive from "@radix-ui/react-checkbox"
import { Check } from "lucide-react"
import { cn } from "@/lib/utils"
const Checkbox = React.forwardRef<
React.ComponentRef<typeof CheckboxPrimitive.Root>,
React.ComponentPropsWithoutRef<typeof CheckboxPrimitive.Root>
>(({ className, ...props }, ref) => (
<CheckboxPrimitive.Root
ref={ref}
className={cn(
"peer h-4 w-4 shrink-0 rounded-sm border border-primary shadow focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring disabled:cursor-not-allowed disabled:opacity-50 data-[state=checked]:bg-primary data-[state=checked]:text-primary-foreground",
className
)}
{...props}
>
<CheckboxPrimitive.Indicator
className={cn("flex items-center justify-center text-current")}
>
<Check className="h-4 w-4" />
</CheckboxPrimitive.Indicator>
</CheckboxPrimitive.Root>
))
Checkbox.displayName = CheckboxPrimitive.Root.displayName
export { Checkbox }

141
crawler/frontend/src/components/ui/dialog.tsx
View file

@ -1,141 +0,0 @@
import * as React from "react"
import * as DialogPrimitive from "@radix-ui/react-dialog"
import { XIcon } from "lucide-react"
import { cn } from "@/lib/utils"
function Dialog({
...props
}: React.ComponentProps<typeof DialogPrimitive.Root>) {
return <DialogPrimitive.Root data-slot="dialog" {...props} />
}
function DialogTrigger({
...props
}: React.ComponentProps<typeof DialogPrimitive.Trigger>) {
return <DialogPrimitive.Trigger data-slot="dialog-trigger" {...props} />
}
function DialogPortal({
...props
}: React.ComponentProps<typeof DialogPrimitive.Portal>) {
return <DialogPrimitive.Portal data-slot="dialog-portal" {...props} />
}
function DialogClose({
...props
}: React.ComponentProps<typeof DialogPrimitive.Close>) {
return <DialogPrimitive.Close data-slot="dialog-close" {...props} />
}
function DialogOverlay({
className,
...props
}: React.ComponentProps<typeof DialogPrimitive.Overlay>) {
return (
<DialogPrimitive.Overlay
data-slot="dialog-overlay"
className={cn(
"data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 fixed inset-0 z-50 bg-black/50",
className
)}
{...props}
/>
)
}
function DialogContent({
className,
children,
showCloseButton = true,
...props
}: React.ComponentProps<typeof DialogPrimitive.Content> & {
showCloseButton?: boolean
}) {
return (
<DialogPortal data-slot="dialog-portal">
<DialogOverlay />
<DialogPrimitive.Content
data-slot="dialog-content"
className={cn(
"bg-background data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 fixed top-[50%] left-[50%] z-50 grid w-full max-w-[calc(100%-2rem)] translate-x-[-50%] translate-y-[-50%] gap-4 rounded-lg border p-6 shadow-lg duration-200 sm:max-w-lg",
className
)}
{...props}
>
{children}
{showCloseButton && (
<DialogPrimitive.Close
data-slot="dialog-close"
className="ring-offset-background focus:ring-ring data-[state=open]:bg-accent data-[state=open]:text-muted-foreground absolute top-4 right-4 rounded-xs opacity-70 transition-opacity hover:opacity-100 focus:ring-2 focus:ring-offset-2 focus:outline-hidden disabled:pointer-events-none [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4"
>
<XIcon />
<span className="sr-only">Close</span>
</DialogPrimitive.Close>
)}
</DialogPrimitive.Content>
</DialogPortal>
)
}
function DialogHeader({ className, ...props }: React.ComponentProps<"div">) {
return (
<div
data-slot="dialog-header"
className={cn("flex flex-col gap-2 text-center sm:text-left", className)}
{...props}
/>
)
}
function DialogFooter({ className, ...props }: React.ComponentProps<"div">) {
return (
<div
data-slot="dialog-footer"
className={cn(
"flex flex-col-reverse gap-2 sm:flex-row sm:justify-end",
className
)}
{...props}
/>
)
}
function DialogTitle({
className,
...props
}: React.ComponentProps<typeof DialogPrimitive.Title>) {
return (
<DialogPrimitive.Title
data-slot="dialog-title"
className={cn("text-lg leading-none font-semibold", className)}
{...props}
/>
)
}
function DialogDescription({
className,
...props
}: React.ComponentProps<typeof DialogPrimitive.Description>) {
return (
<DialogPrimitive.Description
data-slot="dialog-description"
className={cn("text-muted-foreground text-sm", className)}
{...props}
/>
)
}
export {
Dialog,
DialogClose,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogOverlay,
DialogPortal,
DialogTitle,
DialogTrigger,
}

165
crawler/frontend/src/components/ui/form.tsx
View file

@ -1,165 +0,0 @@
import * as React from "react"
import * as LabelPrimitive from "@radix-ui/react-label"
import { Slot } from "@radix-ui/react-slot"
import {
Controller,
FormProvider,
useFormContext,
useFormState,
type ControllerProps,
type FieldPath,
type FieldValues,
} from "react-hook-form"
import { cn } from "@/lib/utils"
import { Label } from "@/components/ui/label"
const Form = FormProvider
type FormFieldContextValue<
TFieldValues extends FieldValues = FieldValues,
TName extends FieldPath<TFieldValues> = FieldPath<TFieldValues>,
> = {
name: TName
}
const FormFieldContext = React.createContext<FormFieldContextValue>(
{} as FormFieldContextValue
)
const FormField = <
TFieldValues extends FieldValues = FieldValues,
TName extends FieldPath<TFieldValues> = FieldPath<TFieldValues>,
>({
...props
}: ControllerProps<TFieldValues, TName>) => {
return (
<FormFieldContext.Provider value={{ name: props.name }}>
<Controller {...props} />
</FormFieldContext.Provider>
)
}
const useFormField = () => {
const fieldContext = React.useContext(FormFieldContext)
const itemContext = React.useContext(FormItemContext)
const { getFieldState } = useFormContext()
const formState = useFormState({ name: fieldContext.name })
const fieldState = getFieldState(fieldContext.name, formState)
if (!fieldContext) {
throw new Error("useFormField should be used within <FormField>")
}
const { id } = itemContext
return {
id,
name: fieldContext.name,
formItemId: `${id}-form-item`,
formDescriptionId: `${id}-form-item-description`,
formMessageId: `${id}-form-item-message`,
...fieldState,
}
}
type FormItemContextValue = {
id: string
}
const FormItemContext = React.createContext<FormItemContextValue>(
{} as FormItemContextValue
)
function FormItem({ className, ...props }: React.ComponentProps<"div">) {
const id = React.useId()
return (
<FormItemContext.Provider value={{ id }}>
<div
data-slot="form-item"
className={cn("grid gap-2", className)}
{...props}
/>
</FormItemContext.Provider>
)
}
function FormLabel({
className,
...props
}: React.ComponentProps<typeof LabelPrimitive.Root>) {
const { error, formItemId } = useFormField()
return (
<Label
data-slot="form-label"
data-error={!!error}
className={cn("data-[error=true]:text-destructive", className)}
htmlFor={formItemId}
{...props}
/>
)
}
function FormControl({ ...props }: React.ComponentProps<typeof Slot>) {
const { error, formItemId, formDescriptionId, formMessageId } = useFormField()
return (
<Slot
data-slot="form-control"
id={formItemId}
aria-describedby={
!error
? `${formDescriptionId}`
: `${formDescriptionId} ${formMessageId}`
}
aria-invalid={!!error}
{...props}
/>
)
}
function FormDescription({ className, ...props }: React.ComponentProps<"p">) {
const { formDescriptionId } = useFormField()
return (
<p
data-slot="form-description"
id={formDescriptionId}
className={cn("text-muted-foreground text-sm", className)}
{...props}
/>
)
}
function FormMessage({ className, ...props }: React.ComponentProps<"p">) {
const { error, formMessageId } = useFormField()
const body = error ? String(error?.message ?? "") : props.children
if (!body) {
return null
}
return (
<p
data-slot="form-message"
id={formMessageId}
className={cn("text-destructive text-sm", className)}
{...props}
>
{body}
</p>
)
}
export {
useFormField,
Form,
FormItem,
FormLabel,
FormControl,
FormDescription,
FormMessage,
FormField,
}

42
crawler/frontend/src/components/ui/hover-card.tsx
View file

@ -1,42 +0,0 @@
import * as React from "react"
import * as HoverCardPrimitive from "@radix-ui/react-hover-card"
import { cn } from "@/lib/utils"
function HoverCard({
...props
}: React.ComponentProps<typeof HoverCardPrimitive.Root>) {
return <HoverCardPrimitive.Root data-slot="hover-card" {...props} />
}
function HoverCardTrigger({
...props
}: React.ComponentProps<typeof HoverCardPrimitive.Trigger>) {
return (
<HoverCardPrimitive.Trigger data-slot="hover-card-trigger" {...props} />
)
}
function HoverCardContent({
className,
align = "center",
sideOffset = 4,
...props
}: React.ComponentProps<typeof HoverCardPrimitive.Content>) {
return (
<HoverCardPrimitive.Portal data-slot="hover-card-portal">
<HoverCardPrimitive.Content
data-slot="hover-card-content"
align={align}
sideOffset={sideOffset}
className={cn(
"bg-popover text-popover-foreground data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 data-[side=bottom]:slide-in-from-top-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 z-50 w-64 origin-(--radix-hover-card-content-transform-origin) rounded-md border p-4 shadow-md outline-hidden",
className
)}
{...props}
/>
</HoverCardPrimitive.Portal>
)
}
export { HoverCard, HoverCardTrigger, HoverCardContent }

21
crawler/frontend/src/components/ui/input.tsx
View file

@ -1,21 +0,0 @@
import * as React from "react"
import { cn } from "@/lib/utils"
function Input({ className, type, ...props }: React.ComponentProps<"input">) {
return (
<input
type={type}
data-slot="input"
className={cn(
"file:text-foreground placeholder:text-muted-foreground selection:bg-primary selection:text-primary-foreground dark:bg-input/30 border-input flex h-9 w-full min-w-0 rounded-md border bg-transparent px-3 py-1 text-base shadow-xs transition-[color,box-shadow] outline-none file:inline-flex file:h-7 file:border-0 file:bg-transparent file:text-sm file:font-medium disabled:pointer-events-none disabled:cursor-not-allowed disabled:opacity-50 md:text-sm",
"focus-visible:border-ring focus-visible:ring-ring/50 focus-visible:ring-[3px]",
"aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40 aria-invalid:border-destructive",
className
)}
{...props}
/>
)
}
export { Input }

22
crawler/frontend/src/components/ui/label.tsx
View file

@ -1,22 +0,0 @@
import * as React from "react"
import * as LabelPrimitive from "@radix-ui/react-label"
import { cn } from "@/lib/utils"
function Label({
className,
...props
}: React.ComponentProps<typeof LabelPrimitive.Root>) {
return (
<LabelPrimitive.Root
data-slot="label"
className={cn(
"flex items-center gap-2 text-sm leading-none font-medium select-none group-data-[disabled=true]:pointer-events-none group-data-[disabled=true]:opacity-50 peer-disabled:cursor-not-allowed peer-disabled:opacity-50",
className
)}
{...props}
/>
)
}
export { Label }

46
crawler/frontend/src/components/ui/popover.tsx
View file

@ -1,46 +0,0 @@
import * as React from "react"
import * as PopoverPrimitive from "@radix-ui/react-popover"
import { cn } from "@/lib/utils"
function Popover({
...props
}: React.ComponentProps<typeof PopoverPrimitive.Root>) {
return <PopoverPrimitive.Root data-slot="popover" {...props} />
}
function PopoverTrigger({
...props
}: React.ComponentProps<typeof PopoverPrimitive.Trigger>) {
return <PopoverPrimitive.Trigger data-slot="popover-trigger" {...props} />
}
function PopoverContent({
className,
align = "center",
sideOffset = 4,
...props
}: React.ComponentProps<typeof PopoverPrimitive.Content>) {
return (
<PopoverPrimitive.Portal>
<PopoverPrimitive.Content
data-slot="popover-content"
align={align}
sideOffset={sideOffset}
className={cn(
"bg-popover text-popover-foreground data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 data-[side=bottom]:slide-in-from-top-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 z-50 w-72 origin-(--radix-popover-content-transform-origin) rounded-md border p-4 shadow-md outline-hidden",
className
)}
{...props}
/>
</PopoverPrimitive.Portal>
)
}
function PopoverAnchor({
...props
}: React.ComponentProps<typeof PopoverPrimitive.Anchor>) {
return <PopoverPrimitive.Anchor data-slot="popover-anchor" {...props} />
}
export { Popover, PopoverTrigger, PopoverContent, PopoverAnchor }

29
crawler/frontend/src/components/ui/progress.tsx
View file

@ -1,29 +0,0 @@
import * as React from "react"
import * as ProgressPrimitive from "@radix-ui/react-progress"
import { cn } from "@/lib/utils"
function Progress({
className,
value,
...props
}: React.ComponentProps<typeof ProgressPrimitive.Root>) {
return (
<ProgressPrimitive.Root
data-slot="progress"
className={cn(
"bg-primary/20 relative h-2 w-full overflow-hidden rounded-full",
className
)}
{...props}
>
<ProgressPrimitive.Indicator
data-slot="progress-indicator"
className="bg-primary h-full w-full flex-1 transition-all"
style={{ transform: `translateX(-${100 - (value || 0)}%)` }}
/>
</ProgressPrimitive.Root>
)
}
export { Progress }

56
crawler/frontend/src/components/ui/scroll-area.tsx
View file

@ -1,56 +0,0 @@
import * as React from "react"
import * as ScrollAreaPrimitive from "@radix-ui/react-scroll-area"
import { cn } from "@/lib/utils"
function ScrollArea({
className,
children,
...props
}: React.ComponentProps<typeof ScrollAreaPrimitive.Root>) {
return (
<ScrollAreaPrimitive.Root
data-slot="scroll-area"
className={cn("relative", className)}
{...props}
>
<ScrollAreaPrimitive.Viewport
data-slot="scroll-area-viewport"
className="focus-visible:ring-ring/50 size-full rounded-[inherit] transition-[color,box-shadow] outline-none focus-visible:ring-[3px] focus-visible:outline-1"
>
{children}
</ScrollAreaPrimitive.Viewport>
<ScrollBar />
<ScrollAreaPrimitive.Corner />
</ScrollAreaPrimitive.Root>
)
}
function ScrollBar({
className,
orientation = "vertical",
...props
}: React.ComponentProps<typeof ScrollAreaPrimitive.ScrollAreaScrollbar>) {
return (
<ScrollAreaPrimitive.ScrollAreaScrollbar
data-slot="scroll-area-scrollbar"
orientation={orientation}
className={cn(
"flex touch-none p-px transition-colors select-none",
orientation === "vertical" &&
"h-full w-2.5 border-l border-l-transparent",
orientation === "horizontal" &&
"h-2.5 flex-col border-t border-t-transparent",
className
)}
{...props}
>
<ScrollAreaPrimitive.ScrollAreaThumb
data-slot="scroll-area-thumb"
className="bg-border relative flex-1 rounded-full"
/>
</ScrollAreaPrimitive.ScrollAreaScrollbar>
)
}
export { ScrollArea, ScrollBar }

183
crawler/frontend/src/components/ui/select.tsx
View file

@ -1,183 +0,0 @@
import * as React from "react"
import * as SelectPrimitive from "@radix-ui/react-select"
import { CheckIcon, ChevronDownIcon, ChevronUpIcon } from "lucide-react"
import { cn } from "@/lib/utils"
function Select({
...props
}: React.ComponentProps<typeof SelectPrimitive.Root>) {
return <SelectPrimitive.Root data-slot="select" {...props} />
}
function SelectGroup({
...props
}: React.ComponentProps<typeof SelectPrimitive.Group>) {
return <SelectPrimitive.Group data-slot="select-group" {...props} />
}
function SelectValue({
...props
}: React.ComponentProps<typeof SelectPrimitive.Value>) {
return <SelectPrimitive.Value data-slot="select-value" {...props} />
}
function SelectTrigger({
className,
size = "default",
children,
...props
}: React.ComponentProps<typeof SelectPrimitive.Trigger> & {
size?: "sm" | "default"
}) {
return (
<SelectPrimitive.Trigger
data-slot="select-trigger"
data-size={size}
className={cn(
"border-input data-[placeholder]:text-muted-foreground [&_svg:not([class*='text-'])]:text-muted-foreground focus-visible:border-ring focus-visible:ring-ring/50 aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40 aria-invalid:border-destructive dark:bg-input/30 dark:hover:bg-input/50 flex w-fit items-center justify-between gap-2 rounded-md border bg-transparent px-3 py-2 text-sm whitespace-nowrap shadow-xs transition-[color,box-shadow] outline-none focus-visible:ring-[3px] disabled:cursor-not-allowed disabled:opacity-50 data-[size=default]:h-9 data-[size=sm]:h-8 *:data-[slot=select-value]:line-clamp-1 *:data-[slot=select-value]:flex *:data-[slot=select-value]:items-center *:data-[slot=select-value]:gap-2 [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4",
className
)}
{...props}
>
{children}
<SelectPrimitive.Icon asChild>
<ChevronDownIcon className="size-4 opacity-50" />
</SelectPrimitive.Icon>
</SelectPrimitive.Trigger>
)
}
function SelectContent({
className,
children,
position = "popper",
...props
}: React.ComponentProps<typeof SelectPrimitive.Content>) {
return (
<SelectPrimitive.Portal>
<SelectPrimitive.Content
data-slot="select-content"
className={cn(
"bg-popover text-popover-foreground data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 data-[side=bottom]:slide-in-from-top-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 relative z-50 max-h-(--radix-select-content-available-height) min-w-[8rem] origin-(--radix-select-content-transform-origin) overflow-x-hidden overflow-y-auto rounded-md border shadow-md",
position === "popper" &&
"data-[side=bottom]:translate-y-1 data-[side=left]:-translate-x-1 data-[side=right]:translate-x-1 data-[side=top]:-translate-y-1",
className
)}
position={position}
{...props}
>
<SelectScrollUpButton />
<SelectPrimitive.Viewport
className={cn(
"p-1",
position === "popper" &&
"h-[var(--radix-select-trigger-height)] w-full min-w-[var(--radix-select-trigger-width)] scroll-my-1"
)}
>
{children}
</SelectPrimitive.Viewport>
<SelectScrollDownButton />
</SelectPrimitive.Content>
</SelectPrimitive.Portal>
)
}
function SelectLabel({
className,
...props
}: React.ComponentProps<typeof SelectPrimitive.Label>) {
return (
<SelectPrimitive.Label
data-slot="select-label"
className={cn("text-muted-foreground px-2 py-1.5 text-xs", className)}
{...props}
/>
)
}
function SelectItem({
className,
children,
...props
}: React.ComponentProps<typeof SelectPrimitive.Item>) {
return (
<SelectPrimitive.Item
data-slot="select-item"
className={cn(
"focus:bg-accent focus:text-accent-foreground [&_svg:not([class*='text-'])]:text-muted-foreground relative flex w-full cursor-default items-center gap-2 rounded-sm py-1.5 pr-8 pl-2 text-sm outline-hidden select-none data-[disabled]:pointer-events-none data-[disabled]:opacity-50 [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4 *:[span]:last:flex *:[span]:last:items-center *:[span]:last:gap-2",
className
)}
{...props}
>
<span className="absolute right-2 flex size-3.5 items-center justify-center">
<SelectPrimitive.ItemIndicator>
<CheckIcon className="size-4" />
</SelectPrimitive.ItemIndicator>
</span>
<SelectPrimitive.ItemText>{children}</SelectPrimitive.ItemText>
</SelectPrimitive.Item>
)
}
function SelectSeparator({
className,
...props
}: React.ComponentProps<typeof SelectPrimitive.Separator>) {
return (
<SelectPrimitive.Separator
data-slot="select-separator"
className={cn("bg-border pointer-events-none -mx-1 my-1 h-px", className)}
{...props}
/>
)
}
function SelectScrollUpButton({
className,
...props
}: React.ComponentProps<typeof SelectPrimitive.ScrollUpButton>) {
return (
<SelectPrimitive.ScrollUpButton
data-slot="select-scroll-up-button"
className={cn(
"flex cursor-default items-center justify-center py-1",
className
)}
{...props}
>
<ChevronUpIcon className="size-4" />
</SelectPrimitive.ScrollUpButton>
)
}
function SelectScrollDownButton({
className,
...props
}: React.ComponentProps<typeof SelectPrimitive.ScrollDownButton>) {
return (
<SelectPrimitive.ScrollDownButton
data-slot="select-scroll-down-button"
className={cn(
"flex cursor-default items-center justify-center py-1",
className
)}
{...props}
>
<ChevronDownIcon className="size-4" />
</SelectPrimitive.ScrollDownButton>
)
}
export {
Select,
SelectContent,
SelectGroup,
SelectItem,
SelectLabel,
SelectScrollDownButton,
SelectScrollUpButton,
SelectSeparator,
SelectTrigger,
SelectValue,
}

28
crawler/frontend/src/components/ui/separator.tsx
View file

@ -1,28 +0,0 @@
"use client"
import * as React from "react"
import * as SeparatorPrimitive from "@radix-ui/react-separator"
import { cn } from "@/lib/utils"
function Separator({
className,
orientation = "horizontal",
decorative = true,
...props
}: React.ComponentProps<typeof SeparatorPrimitive.Root>) {
return (
<SeparatorPrimitive.Root
data-slot="separator"
decorative={decorative}
orientation={orientation}
className={cn(
"bg-border shrink-0 data-[orientation=horizontal]:h-px data-[orientation=horizontal]:w-full data-[orientation=vertical]:h-full data-[orientation=vertical]:w-px",
className
)}
{...props}
/>
)
}
export { Separator }

137
crawler/frontend/src/components/ui/sheet.tsx
View file

@ -1,137 +0,0 @@
import * as React from "react"
import * as SheetPrimitive from "@radix-ui/react-dialog"
import { XIcon } from "lucide-react"
import { cn } from "@/lib/utils"
function Sheet({ ...props }: React.ComponentProps<typeof SheetPrimitive.Root>) {
return <SheetPrimitive.Root data-slot="sheet" {...props} />
}
function SheetTrigger({
...props
}: React.ComponentProps<typeof SheetPrimitive.Trigger>) {
return <SheetPrimitive.Trigger data-slot="sheet-trigger" {...props} />
}
function SheetClose({
...props
}: React.ComponentProps<typeof SheetPrimitive.Close>) {
return <SheetPrimitive.Close data-slot="sheet-close" {...props} />
}
function SheetPortal({
...props
}: React.ComponentProps<typeof SheetPrimitive.Portal>) {
return <SheetPrimitive.Portal data-slot="sheet-portal" {...props} />
}
function SheetOverlay({
className,
...props
}: React.ComponentProps<typeof SheetPrimitive.Overlay>) {
return (
<SheetPrimitive.Overlay
data-slot="sheet-overlay"
className={cn(
"data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 fixed inset-0 z-50 bg-black/50",
className
)}
{...props}
/>
)
}
function SheetContent({
className,
children,
side = "right",
...props
}: React.ComponentProps<typeof SheetPrimitive.Content> & {
side?: "top" | "right" | "bottom" | "left"
}) {
return (
<SheetPortal>
<SheetOverlay />
<SheetPrimitive.Content
data-slot="sheet-content"
className={cn(
"bg-background data-[state=open]:animate-in data-[state=closed]:animate-out fixed z-50 flex flex-col gap-4 shadow-lg transition ease-in-out data-[state=closed]:duration-300 data-[state=open]:duration-500",
side === "right" &&
"data-[state=closed]:slide-out-to-right data-[state=open]:slide-in-from-right inset-y-0 right-0 h-full w-3/4 border-l sm:max-w-sm",
side === "left" &&
"data-[state=closed]:slide-out-to-left data-[state=open]:slide-in-from-left inset-y-0 left-0 h-full w-3/4 border-r sm:max-w-sm",
side === "top" &&
"data-[state=closed]:slide-out-to-top data-[state=open]:slide-in-from-top inset-x-0 top-0 h-auto border-b",
side === "bottom" &&
"data-[state=closed]:slide-out-to-bottom data-[state=open]:slide-in-from-bottom inset-x-0 bottom-0 h-auto border-t",
className
)}
{...props}
>
{children}
<SheetPrimitive.Close className="ring-offset-background focus:ring-ring data-[state=open]:bg-secondary absolute top-4 right-4 rounded-xs opacity-70 transition-opacity hover:opacity-100 focus:ring-2 focus:ring-offset-2 focus:outline-hidden disabled:pointer-events-none">
<XIcon className="size-4" />
<span className="sr-only">Close</span>
</SheetPrimitive.Close>
</SheetPrimitive.Content>
</SheetPortal>
)
}
function SheetHeader({ className, ...props }: React.ComponentProps<"div">) {
return (
<div
data-slot="sheet-header"
className={cn("flex flex-col gap-1.5 p-4", className)}
{...props}
/>
)
}
function SheetFooter({ className, ...props }: React.ComponentProps<"div">) {
return (
<div
data-slot="sheet-footer"
className={cn("mt-auto flex flex-col gap-2 p-4", className)}
{...props}
/>
)
}
function SheetTitle({
className,
...props
}: React.ComponentProps<typeof SheetPrimitive.Title>) {
return (
<SheetPrimitive.Title
data-slot="sheet-title"
className={cn("text-foreground font-semibold", className)}
{...props}
/>
)
}
function SheetDescription({
className,
...props
}: React.ComponentProps<typeof SheetPrimitive.Description>) {
return (
<SheetPrimitive.Description
data-slot="sheet-description"
className={cn("text-muted-foreground text-sm", className)}
{...props}
/>
)
}
export {
Sheet,
SheetTrigger,
SheetClose,
SheetContent,
SheetHeader,
SheetFooter,
SheetTitle,
SheetDescription,
}

725
crawler/frontend/src/components/ui/sidebar.tsx
View file

@ -1,725 +0,0 @@
import { Slot } from "@radix-ui/react-slot"
import { cva, type VariantProps } from "class-variance-authority"
import { PanelLeftIcon } from "lucide-react"
import * as React from "react"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
import { Separator } from "@/components/ui/separator"
import {
Sheet,
SheetContent,
SheetDescription,
SheetHeader,
SheetTitle,
} from "@/components/ui/sheet"
import { Skeleton } from "@/components/ui/skeleton"
import {
Tooltip,
TooltipContent,
TooltipProvider,
TooltipTrigger,
} from "@/components/ui/tooltip"
import { useIsMobile } from "@/hooks/use-mobile"
import { cn } from "@/lib/utils"
const SIDEBAR_COOKIE_NAME = "sidebar_state"
const SIDEBAR_COOKIE_MAX_AGE = 60 * 60 * 24 * 7
const SIDEBAR_WIDTH = "16rem"
const SIDEBAR_WIDTH_MOBILE = "18rem"
const SIDEBAR_WIDTH_ICON = "3rem"
const SIDEBAR_KEYBOARD_SHORTCUT = "b"
type SidebarContextProps = {
state: "expanded" | "collapsed"
open: boolean
setOpen: (open: boolean) => void
openMobile: boolean
setOpenMobile: (open: boolean) => void
isMobile: boolean
toggleSidebar: () => void
}
const SidebarContext = React.createContext<SidebarContextProps | null>(null)
function useSidebar() {
const context = React.useContext(SidebarContext)
if (!context) {
throw new Error("useSidebar must be used within a SidebarProvider.")
}
return context
}
function SidebarProvider({
defaultOpen = true,
open: openProp,
onOpenChange: setOpenProp,
className,
style,
children,
...props
}: React.ComponentProps<"div"> & {
defaultOpen?: boolean
open?: boolean
onOpenChange?: (open: boolean) => void
}) {
const isMobile = useIsMobile()
const [openMobile, setOpenMobile] = React.useState(false)
// This is the internal state of the sidebar.
// We use openProp and setOpenProp for control from outside the component.
const [_open, _setOpen] = React.useState(defaultOpen)
const open = openProp ?? _open
const setOpen = React.useCallback(
(value: boolean | ((value: boolean) => boolean)) => {
const openState = typeof value === "function" ? value(open) : value
if (setOpenProp) {
setOpenProp(openState)
} else {
_setOpen(openState)
}
// This sets the cookie to keep the sidebar state.
document.cookie = `${SIDEBAR_COOKIE_NAME}=${openState}; path=/; max-age=${SIDEBAR_COOKIE_MAX_AGE}`
},
[setOpenProp, open]
)
// Helper to toggle the sidebar.
const toggleSidebar = React.useCallback(() => {
return isMobile ? setOpenMobile((open) => !open) : setOpen((open) => !open)
}, [isMobile, setOpen, setOpenMobile])
// Adds a keyboard shortcut to toggle the sidebar.
React.useEffect(() => {
const handleKeyDown = (event: KeyboardEvent) => {
if (
event.key === SIDEBAR_KEYBOARD_SHORTCUT &&
(event.metaKey || event.ctrlKey)
) {
event.preventDefault()
toggleSidebar()
}
}
window.addEventListener("keydown", handleKeyDown)
return () => window.removeEventListener("keydown", handleKeyDown)
}, [toggleSidebar])
// We add a state so that we can do data-state="expanded" or "collapsed".
// This makes it easier to style the sidebar with Tailwind classes.
const state = open ? "expanded" : "collapsed"
const contextValue = React.useMemo<SidebarContextProps>(
() => ({
state,
open,
setOpen,
isMobile,
openMobile,
setOpenMobile,
toggleSidebar,
}),
[state, open, setOpen, isMobile, openMobile, setOpenMobile, toggleSidebar]
)
return (
<SidebarContext.Provider value={contextValue}>
<TooltipProvider delayDuration={0}>
<div
data-slot="sidebar-wrapper"
style={
{
"--sidebar-width": SIDEBAR_WIDTH,
"--sidebar-width-icon": SIDEBAR_WIDTH_ICON,
...style,
} as React.CSSProperties
}
className={cn(
"group/sidebar-wrapper has-data-[variant=inset]:bg-sidebar flex min-h-svh w-full",
className
)}
{...props}
>
{children}
</div>
</TooltipProvider>
</SidebarContext.Provider>
)
}
function Sidebar({
side = "left",
variant = "sidebar",
collapsible = "offcanvas",
className,
children,
...props
}: React.ComponentProps<"div"> & {
side?: "left" | "right"
variant?: "sidebar" | "floating" | "inset"
collapsible?: "offcanvas" | "icon" | "none"
}) {
const { isMobile, state, openMobile, setOpenMobile } = useSidebar()
if (collapsible === "none") {
return (
<div
data-slot="sidebar"
className={cn(
"bg-sidebar text-sidebar-foreground flex h-full w-(--sidebar-width) flex-col",
className
)}
{...props}
>
{children}
</div>
)
}
if (isMobile) {
return (
<Sheet open={openMobile} onOpenChange={setOpenMobile} {...props}>
<SheetContent
data-sidebar="sidebar"
data-slot="sidebar"
data-mobile="true"
className="bg-sidebar text-sidebar-foreground w-(--sidebar-width) p-0 [&>button]:hidden"
style={
{
"--sidebar-width": SIDEBAR_WIDTH_MOBILE,
} as React.CSSProperties
}
side={side}
>
<SheetHeader className="sr-only">
<SheetTitle>Sidebar</SheetTitle>
<SheetDescription>Displays the mobile sidebar.</SheetDescription>
</SheetHeader>
<div className="flex h-full w-full flex-col">{children}</div>
</SheetContent>
</Sheet>
)
}
return (
<div
className="group peer text-sidebar-foreground hidden md:block"
data-state={state}
data-collapsible={state === "collapsed" ? collapsible : ""}
data-variant={variant}
data-side={side}
data-slot="sidebar"
>
{/* This is what handles the sidebar gap on desktop */}
<div
data-slot="sidebar-gap"
className={cn(
"relative w-(--sidebar-width) bg-transparent transition-[width] duration-200 ease-linear",
"group-data-[collapsible=offcanvas]:w-0",
"group-data-[side=right]:rotate-180",
variant === "floating" || variant === "inset"
? "group-data-[collapsible=icon]:w-[calc(var(--sidebar-width-icon)+(--spacing(4)))]"
: "group-data-[collapsible=icon]:w-(--sidebar-width-icon)"
)}
/>
<div
data-slot="sidebar-container"
className={cn(
"fixed inset-y-0 z-10 hidden h-svh w-(--sidebar-width) transition-[left,right,width] duration-200 ease-linear md:flex",
side === "left"
? "left-0 group-data-[collapsible=offcanvas]:left-[calc(var(--sidebar-width)*-1)]"
: "right-0 group-data-[collapsible=offcanvas]:right-[calc(var(--sidebar-width)*-1)]",
// Adjust the padding for floating and inset variants.
variant === "floating" || variant === "inset"
? "p-2 group-data-[collapsible=icon]:w-[calc(var(--sidebar-width-icon)+(--spacing(4))+2px)]"
: "group-data-[collapsible=icon]:w-(--sidebar-width-icon) group-data-[side=left]:border-r group-data-[side=right]:border-l",
className
)}
{...props}
>
<div
data-sidebar="sidebar"
data-slot="sidebar-inner"
className="bg-sidebar group-data-[variant=floating]:border-sidebar-border flex h-full w-full flex-col group-data-[variant=floating]:rounded-lg group-data-[variant=floating]:border group-data-[variant=floating]:shadow-sm"
>
{children}
</div>
</div>
</div>
)
}
function SidebarTrigger({
className,
onClick,
...props
}: React.ComponentProps<typeof Button>) {
const { toggleSidebar } = useSidebar()
return (
<Button
data-sidebar="trigger"
data-slot="sidebar-trigger"
variant="ghost"
size="icon"
className={cn("size-7", className)}
onClick={(event) => {
onClick?.(event)
toggleSidebar()
}}
{...props}
>
<PanelLeftIcon />
<span className="sr-only">Toggle Sidebar</span>
</Button>
)
}
function SidebarRail({ className, ...props }: React.ComponentProps<"button">) {
const { toggleSidebar } = useSidebar()
return (
<button
data-sidebar="rail"
data-slot="sidebar-rail"
aria-label="Toggle Sidebar"
tabIndex={-1}
onClick={toggleSidebar}
title="Toggle Sidebar"
className={cn(
"hover:after:bg-sidebar-border absolute inset-y-0 z-20 hidden w-4 -translate-x-1/2 transition-all ease-linear group-data-[side=left]:-right-4 group-data-[side=right]:left-0 after:absolute after:inset-y-0 after:left-1/2 after:w-[2px] sm:flex",
"in-data-[side=left]:cursor-w-resize in-data-[side=right]:cursor-e-resize",
"[[data-side=left][data-state=collapsed]_&]:cursor-e-resize [[data-side=right][data-state=collapsed]_&]:cursor-w-resize",
"hover:group-data-[collapsible=offcanvas]:bg-sidebar group-data-[collapsible=offcanvas]:translate-x-0 group-data-[collapsible=offcanvas]:after:left-full",
"[[data-side=left][data-collapsible=offcanvas]_&]:-right-2",
"[[data-side=right][data-collapsible=offcanvas]_&]:-left-2",
className
)}
{...props}
/>
)
}
function SidebarInset({ className, ...props }: React.ComponentProps<"main">) {
return (
<main
data-slot="sidebar-inset"
className={cn(
"bg-background relative flex w-full flex-1 flex-col",
"md:peer-data-[variant=inset]:m-2 md:peer-data-[variant=inset]:ml-0 md:peer-data-[variant=inset]:rounded-xl md:peer-data-[variant=inset]:shadow-sm md:peer-data-[variant=inset]:peer-data-[state=collapsed]:ml-2",
className
)}
{...props}
/>
)
}
function SidebarInput({
className,
...props
}: React.ComponentProps<typeof Input>) {
return (
<Input
data-slot="sidebar-input"
data-sidebar="input"
className={cn("bg-background h-8 w-full shadow-none", className)}
{...props}
/>
)
}
function SidebarHeader({ className, ...props }: React.ComponentProps<"div">) {
return (
<div
data-slot="sidebar-header"
data-sidebar="header"
className={cn("flex flex-col gap-2 p-2", className)}
{...props}
/>
)
}
function SidebarFooter({ className, ...props }: React.ComponentProps<"div">) {
return (
<div
data-slot="sidebar-footer"
data-sidebar="footer"
className={cn("flex flex-col gap-2 p-2", className)}
{...props}
/>
)
}
function SidebarSeparator({
className,
...props
}: React.ComponentProps<typeof Separator>) {
return (
<Separator
data-slot="sidebar-separator"
data-sidebar="separator"
className={cn("bg-sidebar-border mx-2 w-auto", className)}
{...props}
/>
)
}
function SidebarContent({ className, ...props }: React.ComponentProps<"div">) {
return (
<div
data-slot="sidebar-content"
data-sidebar="content"
className={cn(
"flex min-h-0 flex-1 flex-col gap-2 overflow-auto group-data-[collapsible=icon]:overflow-hidden",
className
)}
{...props}
/>
)
}
function SidebarGroup({ className, ...props }: React.ComponentProps<"div">) {
return (
<div
data-slot="sidebar-group"
data-sidebar="group"
className={cn("relative flex w-full min-w-0 flex-col p-2", className)}
{...props}
/>
)
}
function SidebarGroupLabel({
className,
asChild = false,
...props
}: React.ComponentProps<"div"> & { asChild?: boolean }) {
const Comp = asChild ? Slot : "div"
return (
<Comp
data-slot="sidebar-group-label"
data-sidebar="group-label"
className={cn(
"text-sidebar-foreground/70 ring-sidebar-ring flex h-8 shrink-0 items-center rounded-md px-2 text-xs font-medium outline-hidden transition-[margin,opacity] duration-200 ease-linear focus-visible:ring-2 [&>svg]:size-4 [&>svg]:shrink-0",
"group-data-[collapsible=icon]:-mt-8 group-data-[collapsible=icon]:opacity-0",
className
)}
{...props}
/>
)
}
function SidebarGroupAction({
className,
asChild = false,
...props
}: React.ComponentProps<"button"> & { asChild?: boolean }) {
const Comp = asChild ? Slot : "button"
return (
<Comp
data-slot="sidebar-group-action"
data-sidebar="group-action"
className={cn(
"text-sidebar-foreground ring-sidebar-ring hover:bg-sidebar-accent hover:text-sidebar-accent-foreground absolute top-3.5 right-3 flex aspect-square w-5 items-center justify-center rounded-md p-0 outline-hidden transition-transform focus-visible:ring-2 [&>svg]:size-4 [&>svg]:shrink-0",
// Increases the hit area of the button on mobile.
"after:absolute after:-inset-2 md:after:hidden",
"group-data-[collapsible=icon]:hidden",
className
)}
{...props}
/>
)
}
function SidebarGroupContent({
className,
...props
}: React.ComponentProps<"div">) {
return (
<div
data-slot="sidebar-group-content"
data-sidebar="group-content"
className={cn("w-full text-sm", className)}
{...props}
/>
)
}
function SidebarMenu({ className, ...props }: React.ComponentProps<"ul">) {
return (
<ul
data-slot="sidebar-menu"
data-sidebar="menu"
className={cn("flex w-full min-w-0 flex-col gap-1", className)}
{...props}
/>
)
}
function SidebarMenuItem({ className, ...props }: React.ComponentProps<"li">) {
return (
<li
data-slot="sidebar-menu-item"
data-sidebar="menu-item"
className={cn("group/menu-item relative", className)}
{...props}
/>
)
}
const sidebarMenuButtonVariants = cva(
"peer/menu-button flex w-full items-center gap-2 overflow-hidden rounded-md p-2 text-left text-sm outline-hidden ring-sidebar-ring transition-[width,height,padding] hover:bg-sidebar-accent hover:text-sidebar-accent-foreground focus-visible:ring-2 active:bg-sidebar-accent active:text-sidebar-accent-foreground disabled:pointer-events-none disabled:opacity-50 group-has-data-[sidebar=menu-action]/menu-item:pr-8 aria-disabled:pointer-events-none aria-disabled:opacity-50 data-[active=true]:bg-sidebar-accent data-[active=true]:font-medium data-[active=true]:text-sidebar-accent-foreground data-[state=open]:hover:bg-sidebar-accent data-[state=open]:hover:text-sidebar-accent-foreground group-data-[collapsible=icon]:size-8! group-data-[collapsible=icon]:p-2! [&>span:last-child]:truncate [&>svg]:size-4 [&>svg]:shrink-0",
{
variants: {
variant: {
default: "hover:bg-sidebar-accent hover:text-sidebar-accent-foreground",
outline:
"bg-background shadow-[0_0_0_1px_hsl(var(--sidebar-border))] hover:bg-sidebar-accent hover:text-sidebar-accent-foreground hover:shadow-[0_0_0_1px_hsl(var(--sidebar-accent))]",
},
size: {
default: "h-8 text-sm",
sm: "h-7 text-xs",
lg: "h-12 text-sm group-data-[collapsible=icon]:p-0!",
},
},
defaultVariants: {
variant: "default",
size: "default",
},
}
)
function SidebarMenuButton({
asChild = false,
isActive = false,
variant = "default",
size = "default",
tooltip,
className,
...props
}: React.ComponentProps<"button"> & {
asChild?: boolean
isActive?: boolean
tooltip?: string | React.ComponentProps<typeof TooltipContent>
} & VariantProps<typeof sidebarMenuButtonVariants>) {
const Comp = asChild ? Slot : "button"
const { isMobile, state } = useSidebar()
const button = (
<Comp
data-slot="sidebar-menu-button"
data-sidebar="menu-button"
data-size={size}
data-active={isActive}
className={cn(sidebarMenuButtonVariants({ variant, size }), className)}
{...props}
/>
)
if (!tooltip) {
return button
}
if (typeof tooltip === "string") {
tooltip = {
children: tooltip,
}
}
return (
<Tooltip>
<TooltipTrigger asChild>{button}</TooltipTrigger>
<TooltipContent
side="right"
align="center"
hidden={state !== "collapsed" || isMobile}
{...tooltip}
/>
</Tooltip>
)
}
function SidebarMenuAction({
className,
asChild = false,
showOnHover = false,
...props
}: React.ComponentProps<"button"> & {
asChild?: boolean
showOnHover?: boolean
}) {
const Comp = asChild ? Slot : "button"
return (
<Comp
data-slot="sidebar-menu-action"
data-sidebar="menu-action"
className={cn(
"text-sidebar-foreground ring-sidebar-ring hover:bg-sidebar-accent hover:text-sidebar-accent-foreground peer-hover/menu-button:text-sidebar-accent-foreground absolute top-1.5 right-1 flex aspect-square w-5 items-center justify-center rounded-md p-0 outline-hidden transition-transform focus-visible:ring-2 [&>svg]:size-4 [&>svg]:shrink-0",
// Increases the hit area of the button on mobile.
"after:absolute after:-inset-2 md:after:hidden",
"peer-data-[size=sm]/menu-button:top-1",
"peer-data-[size=default]/menu-button:top-1.5",
"peer-data-[size=lg]/menu-button:top-2.5",
"group-data-[collapsible=icon]:hidden",
showOnHover &&
"peer-data-[active=true]/menu-button:text-sidebar-accent-foreground group-focus-within/menu-item:opacity-100 group-hover/menu-item:opacity-100 data-[state=open]:opacity-100 md:opacity-0",
className
)}
{...props}
/>
)
}
function SidebarMenuBadge({
className,
...props
}: React.ComponentProps<"div">) {
return (
<div
data-slot="sidebar-menu-badge"
data-sidebar="menu-badge"
className={cn(
"text-sidebar-foreground pointer-events-none absolute right-1 flex h-5 min-w-5 items-center justify-center rounded-md px-1 text-xs font-medium tabular-nums select-none",
"peer-hover/menu-button:text-sidebar-accent-foreground peer-data-[active=true]/menu-button:text-sidebar-accent-foreground",
"peer-data-[size=sm]/menu-button:top-1",
"peer-data-[size=default]/menu-button:top-1.5",
"peer-data-[size=lg]/menu-button:top-2.5",
"group-data-[collapsible=icon]:hidden",
className
)}
{...props}
/>
)
}
function SidebarMenuSkeleton({
className,
showIcon = false,
...props
}: React.ComponentProps<"div"> & {
showIcon?: boolean
}) {
// Random width between 50 to 90%.
const width = React.useMemo(() => {
return `${Math.floor(Math.random() * 40) + 50}%`
}, [])
return (
<div
data-slot="sidebar-menu-skeleton"
data-sidebar="menu-skeleton"
className={cn("flex h-8 items-center gap-2 rounded-md px-2", className)}
{...props}
>
{showIcon && (
<Skeleton
className="size-4 rounded-md"
data-sidebar="menu-skeleton-icon"
/>
)}
<Skeleton
className="h-4 max-w-(--skeleton-width) flex-1"
data-sidebar="menu-skeleton-text"
style={
{
"--skeleton-width": width,
} as React.CSSProperties
}
/>
</div>
)
}
function SidebarMenuSub({ className, ...props }: React.ComponentProps<"ul">) {
return (
<ul
data-slot="sidebar-menu-sub"
data-sidebar="menu-sub"
className={cn(
"border-sidebar-border mx-3.5 flex min-w-0 translate-x-px flex-col gap-1 border-l px-2.5 py-0.5",
"group-data-[collapsible=icon]:hidden",
className
)}
{...props}
/>
)
}
function SidebarMenuSubItem({
className,
...props
}: React.ComponentProps<"li">) {
return (
<li
data-slot="sidebar-menu-sub-item"
data-sidebar="menu-sub-item"
className={cn("group/menu-sub-item relative", className)}
{...props}
/>
)
}
function SidebarMenuSubButton({
asChild = false,
size = "md",
isActive = false,
className,
...props
}: React.ComponentProps<"a"> & {
asChild?: boolean
size?: "sm" | "md"
isActive?: boolean
}) {
const Comp = asChild ? Slot : "a"
return (
<Comp
data-slot="sidebar-menu-sub-button"
data-sidebar="menu-sub-button"
data-size={size}
data-active={isActive}
className={cn(
"text-sidebar-foreground ring-sidebar-ring hover:bg-sidebar-accent hover:text-sidebar-accent-foreground active:bg-sidebar-accent active:text-sidebar-accent-foreground [&>svg]:text-sidebar-accent-foreground flex h-7 min-w-0 -translate-x-px items-center gap-2 overflow-hidden rounded-md px-2 outline-hidden focus-visible:ring-2 disabled:pointer-events-none disabled:opacity-50 aria-disabled:pointer-events-none aria-disabled:opacity-50 [&>span:last-child]:truncate [&>svg]:size-4 [&>svg]:shrink-0",
"data-[active=true]:bg-sidebar-accent data-[active=true]:text-sidebar-accent-foreground",
size === "sm" && "text-xs",
size === "md" && "text-sm",
"group-data-[collapsible=icon]:hidden",
className
)}
{...props}
/>
)
}
export {
Sidebar,
SidebarContent,
SidebarFooter,
SidebarGroup,
SidebarGroupAction,
SidebarGroupContent,
SidebarGroupLabel,
SidebarHeader,
SidebarInput,
SidebarInset,
SidebarMenu,
SidebarMenuAction,
SidebarMenuBadge,
SidebarMenuButton,
SidebarMenuItem,
SidebarMenuSkeleton,
SidebarMenuSub,
SidebarMenuSubButton,
SidebarMenuSubItem,
SidebarProvider,
SidebarRail,
SidebarSeparator,
SidebarTrigger,
useSidebar
}

13
crawler/frontend/src/components/ui/skeleton.tsx
View file

@ -1,13 +0,0 @@
import { cn } from "@/lib/utils"
function Skeleton({ className, ...props }: React.ComponentProps<"div">) {
return (
<div
data-slot="skeleton"
className={cn("bg-accent animate-pulse rounded-md", className)}
{...props}
/>
)
}
export { Skeleton }

34
crawler/frontend/src/components/ui/slider.tsx
View file

@ -1,34 +0,0 @@
"use client"
import * as React from "react"
import * as SliderPrimitive from "@radix-ui/react-slider"
import { cn } from "@/lib/utils"
const Slider = React.forwardRef<
React.ComponentRef<typeof SliderPrimitive.Root>,
React.ComponentPropsWithoutRef<typeof SliderPrimitive.Root>
>(({ className, ...props }, ref) => (
<SliderPrimitive.Root
ref={ref}
className={cn(
"relative flex w-full touch-none select-none items-center",
className
)}
{...props}
>
<SliderPrimitive.Track className="relative h-1.5 w-full grow overflow-hidden rounded-full bg-primary/20">
<SliderPrimitive.Range className="absolute h-full bg-primary" />
</SliderPrimitive.Track>
{props.defaultValue?.map((_, index) => (
<SliderPrimitive.Thumb
key={index}
className="block h-4 w-4 rounded-full border border-primary/50 bg-background shadow transition-colors focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring disabled:pointer-events-none disabled:opacity-50"
/>
)) ?? (
<SliderPrimitive.Thumb className="block h-4 w-4 rounded-full border border-primary/50 bg-background shadow transition-colors focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring disabled:pointer-events-none disabled:opacity-50" />
)}
</SliderPrimitive.Root>
))
Slider.displayName = SliderPrimitive.Root.displayName
export { Slider }

60
crawler/frontend/src/components/ui/tabs.tsx
View file

@ -1,60 +0,0 @@
import * as React from "react"
import * as TabsPrimitive from "@radix-ui/react-tabs"
import { cn } from "@/lib/utils"
function Tabs({
...props
}: React.ComponentProps<typeof TabsPrimitive.Root>) {
return <TabsPrimitive.Root data-slot="tabs" {...props} />
}
function TabsList({
className,
...props
}: React.ComponentProps<typeof TabsPrimitive.List>) {
return (
<TabsPrimitive.List
data-slot="tabs-list"
className={cn(
"inline-flex h-10 items-center justify-center rounded-md bg-muted p-1 text-muted-foreground w-full",
className
)}
{...props}
/>
)
}
function TabsTrigger({
className,
...props
}: React.ComponentProps<typeof TabsPrimitive.Trigger>) {
return (
<TabsPrimitive.Trigger
data-slot="tabs-trigger"
className={cn(
"inline-flex items-center justify-center whitespace-nowrap rounded-sm px-3 py-1.5 text-sm font-medium ring-offset-background transition-all focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50 data-[state=active]:bg-background data-[state=active]:text-foreground data-[state=active]:shadow-sm flex-1",
className
)}
{...props}
/>
)
}
function TabsContent({
className,
...props
}: React.ComponentProps<typeof TabsPrimitive.Content>) {
return (
<TabsPrimitive.Content
data-slot="tabs-content"
className={cn(
"mt-2 ring-offset-background focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2",
className
)}
{...props}
/>
)
}
export { Tabs, TabsList, TabsTrigger, TabsContent }

61
crawler/frontend/src/components/ui/tooltip.tsx
View file

@ -1,61 +0,0 @@
"use client"
import * as React from "react"
import * as TooltipPrimitive from "@radix-ui/react-tooltip"
import { cn } from "@/lib/utils"
function TooltipProvider({
delayDuration = 0,
...props
}: React.ComponentProps<typeof TooltipPrimitive.Provider>) {
return (
<TooltipPrimitive.Provider
data-slot="tooltip-provider"
delayDuration={delayDuration}
{...props}
/>
)
}
function Tooltip({
...props
}: React.ComponentProps<typeof TooltipPrimitive.Root>) {
return (
<TooltipProvider>
<TooltipPrimitive.Root data-slot="tooltip" {...props} />
</TooltipProvider>
)
}
function TooltipTrigger({
...props
}: React.ComponentProps<typeof TooltipPrimitive.Trigger>) {
return <TooltipPrimitive.Trigger data-slot="tooltip-trigger" {...props} />
}
function TooltipContent({
className,
sideOffset = 0,
children,
...props
}: React.ComponentProps<typeof TooltipPrimitive.Content>) {
return (
<TooltipPrimitive.Portal>
<TooltipPrimitive.Content
data-slot="tooltip-content"
sideOffset={sideOffset}
className={cn(
"bg-primary text-primary-foreground animate-in fade-in-0 zoom-in-95 data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=closed]:zoom-out-95 data-[side=bottom]:slide-in-from-top-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 z-50 w-fit origin-(--radix-tooltip-content-transform-origin) rounded-md px-3 py-1.5 text-xs text-balance",
className
)}
{...props}
>
{children}
<TooltipPrimitive.Arrow className="bg-primary fill-primary z-50 size-2.5 translate-y-[calc(-50%_-_2px)] rotate-45 rounded-[2px]" />
</TooltipPrimitive.Content>
</TooltipPrimitive.Portal>
)
}
export { Tooltip, TooltipTrigger, TooltipContent, TooltipProvider }

83
crawler/frontend/src/constants/colorSchemes.ts
View file

@ -1,83 +0,0 @@
// Color schemes for map visualization
// Different color schemes for different metrics to improve clarity
import { Metric } from "@/components/Parameters";
// For metrics where LOW is GOOD (price, price per sqm): Green → Yellow → Red
export const LOW_IS_GOOD_COLOR_STOPS: [number, string][] = [
[0, 'rgba(34, 197, 94, 0.7)'], // Green - good deal
[25, 'rgba(132, 204, 22, 0.7)'], // Lime
[50, 'rgba(250, 204, 21, 0.7)'], // Yellow - neutral
[75, 'rgba(249, 115, 22, 0.7)'], // Orange
[100, 'rgba(239, 68, 68, 0.7)'], // Red - expensive
];
// For metrics where HIGH is GOOD (size, rooms): Red → Yellow → Green
export const HIGH_IS_GOOD_COLOR_STOPS: [number, string][] = [
[0, 'rgba(239, 68, 68, 0.7)'], // Red - small
[25, 'rgba(249, 115, 22, 0.7)'], // Orange
[50, 'rgba(250, 204, 21, 0.7)'], // Yellow - medium
[75, 'rgba(132, 204, 22, 0.7)'], // Lime
[100, 'rgba(34, 197, 94, 0.7)'], // Green - large
];
// Legacy color stops (for backwards compatibility)
export const LEGACY_COLOR_STOPS: [number, string][] = [
[0, 'rgba(0,185,243,0)'],
[25, 'rgba(0,185,243,0.24)'],
[60, 'rgba(255,223,0,0.3)'],
[100, 'rgba(255,105,0,0.3)'],
];
// Get the appropriate color scheme based on metric type
export function getColorSchemeForMetric(metric: Metric | string): [number, string][] {
switch (metric) {
case Metric.qmprice:
case Metric.price:
case 'qmprice':
case 'total_price':
// Lower price is better → Green for low, Red for high
return LOW_IS_GOOD_COLOR_STOPS;
case Metric.qm:
case Metric.rooms:
case 'qm':
case 'rooms':
// Higher value is better → Green for high, Red for low
return HIGH_IS_GOOD_COLOR_STOPS;
default:
return LOW_IS_GOOD_COLOR_STOPS;
}
}
// Get interpretation text for legend
export function getMetricInterpretation(metric: Metric | string): { low: string; high: string; name: string } {
switch (metric) {
case Metric.qmprice:
case 'qmprice':
return { low: 'Good deal', high: 'Expensive', name: 'Price per m²' };
case Metric.price:
case 'total_price':
return { low: 'Good deal', high: 'Expensive', name: 'Total Price' };
case Metric.qm:
case 'qm':
return { low: 'Small', high: 'Large', name: 'Size (m²)' };
case Metric.rooms:
case 'rooms':
return { low: 'Few rooms', high: 'Many rooms', name: 'Bedrooms' };
default:
return { low: 'Low', high: 'High', name: 'Value' };
}
}
// Color scheme names for display
export const COLOR_SCHEME_NAMES = {
LOW_IS_GOOD: 'Green → Red (low is good)',
HIGH_IS_GOOD: 'Red → Green (high is good)',
LEGACY: 'Classic (blue → orange)',
} as const;

62
crawler/frontend/src/constants/index.ts
View file

@ -1,62 +0,0 @@
// Application constants and configuration
// Re-export color schemes
export * from './colorSchemes';
// API endpoints
export const API_ENDPOINTS = {
LISTING_GEOJSON: '/api/listing_geojson',
LISTING_GEOJSON_STREAM: '/api/listing_geojson/stream',
REFRESH_LISTINGS: '/api/refresh_listings',
TASK_STATUS: '/api/task_status',
TASKS_FOR_USER: '/api/tasks_for_user',
CANCEL_TASK: '/api/cancel_task',
CLEAR_ALL_TASKS: '/api/clear_all_tasks',
GET_DISTRICTS: '/api/get_districts',
} as const;
// Map configuration
export const MAP_CONFIG = {
MAPBOX_TOKEN: import.meta.env.VITE_MAPBOX_TOKEN || 'pk.eyJ1IjoiZGktdG8iLCJhIjoiY2o0bnBoYXcxMW1mNzJ3bDhmc2xiNWttaiJ9.ZccatVk_4shzoAsEUXXecA',
DEFAULT_CENTER: [13.38032, 49.994210] as [number, number],
DEFAULT_ZOOM: 5,
STYLE: 'mapbox://styles/mapbox/light-v9',
} as const;
// Heatmap configuration
export const HEATMAP_CONFIG = {
INTENSITY: 9,
SPREAD: 0.05,
CELL_DENSITY: 0.5, // Smaller value = bigger hexagons
} as const;
// Percentile configuration for data visualization
export const PERCENTILE_CONFIG = {
MIN_BOUND: 0.05, // 5th percentile for color scale minimum
MAX_BOUND: 0.95, // 95th percentile for color scale maximum
BOUNDS_CLIP_MIN: 0.01, // 1st percentile for bounding box
BOUNDS_CLIP_MAX: 0.99, // 99th percentile for bounding box
} as const;
// Heatmap color gradient stops
export const HEATMAP_COLOR_STOPS: [number, string][] = [
[0, 'rgba(0,185,243,0)'],
[25, 'rgba(0,185,243,0.24)'],
[60, 'rgba(255,223,0,0.3)'],
[100, 'rgba(255,105,0,0.3)'],
];
// Default form values
export const DEFAULT_FORM_VALUES = {
min_bedrooms: 1,
max_bedrooms: 3,
max_price: 3000,
min_price: 2000,
min_sqm: 50,
last_seen_days: 28,
} as const;
// Polling intervals
export const POLLING_INTERVALS = {
TASK_STATUS_MS: 5000, // 5 seconds
} as const;

19
crawler/frontend/src/hooks/use-mobile.ts
View file

@ -1,19 +0,0 @@
import * as React from "react"
const MOBILE_BREAKPOINT = 768
export function useIsMobile() {
const [isMobile, setIsMobile] = React.useState<boolean | undefined>(undefined)
React.useEffect(() => {
const mql = window.matchMedia(`(max-width: ${MOBILE_BREAKPOINT - 1}px)`)
const onChange = () => {
setIsMobile(window.innerWidth < MOBILE_BREAKPOINT)
}
mql.addEventListener("change", onChange)
setIsMobile(window.innerWidth < MOBILE_BREAKPOINT)
return () => mql.removeEventListener("change", onChange)
}, [])
return !!isMobile
}

Some files were not shown because too many files have changed in this diff Show more