No description
Find a file
Viktor Barzin 8d22c97320
Add comprehensive test suite: 219 new tests across backend and frontend
Backend (103 tests):
- Unit tests for listing_service, export_service, district_service
- Regression tests for API response contracts and query parameter validation
- Integration tests for API workflows, Redis listing cache, listing processor pipeline, and repository advanced queries
- E2E tests for streaming with filters, batching, caching, and task management

Frontend (116 tests):
- Service tests for apiClient, streamingService, taskService, listingService, healthService
- Hook tests for useTaskProgress (WebSocket + polling)
- Component tests for PropertyCard, FilterPanel, Header, ListView, TaskProgressDrawer, TaskIndicator, StreamingProgressBar, HealthIndicator
- E2E tests for filter-stream-display flow

Infrastructure:
- Add pytest-xdist and test markers (regression, integration, e2e)
- Add conftest fixtures: fake_redis, rent_listing_factory, seeded_repository
- Add vitest + testing-library + MSW for frontend testing
2026-02-10 21:59:45 +00:00
.claude Add Claude agent definitions [ci skip] 2026-02-09 23:02:30 +00:00
.github/workflows adding ruff auto check for pull requests as well as fixing all ruff errors (#1) 2025-09-14 19:40:18 +01:00
alembic Fix alembic migration to rename buylisting.longtitude to longitude 2026-02-09 20:47:30 +00:00
api Switch task progress to throttled event-driven updates 2026-02-10 21:24:33 +00:00
config Add self-hosted routing clients and distance calculator 2026-02-08 13:14:37 +00:00
docs Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
frontend Add comprehensive test suite: 219 new tests across backend and frontend 2026-02-10 21:59:45 +00:00
k8s Add Kubernetes manifests for routing engines 2026-02-08 13:16:38 +00:00
models Harden backend security: IDOR fix, error sanitization, rate limiter fallback, security headers 2026-02-08 19:42:30 +00:00
rec Fix POI distance calculation reliability for remote/Celery execution 2026-02-08 20:11:12 +00:00
repositories Fix POI distance calculation reliability for remote/Celery execution 2026-02-08 20:11:12 +00:00
scripts Fix OSRM setup: update Geofabrik URL and use bind mount for data 2026-02-08 14:51:52 +00:00
services Fix real-time task progress by closing WS on pubsub exit and keeping polling active 2026-02-09 22:48:57 +00:00
tasks Switch task progress to throttled event-driven updates 2026-02-10 21:24:33 +00:00
tests Add comprehensive test suite: 219 new tests across backend and frontend 2026-02-10 21:59:45 +00:00
utils Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
.dockerignore Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
.drone.yml Fix Drone CI variable interpolation in API verify-deploy step 2026-02-10 21:32:11 +00:00
.env.sample Add OSRM and OTP Docker services with setup scripts 2026-02-08 13:16:10 +00:00
.gitignore Fix OSRM setup: update Geofabrik URL and use bind mount for data 2026-02-08 14:51:52 +00:00
.style.yapf Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
alembic.ini Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
celery_app.py Add POI API routes and Celery task 2026-02-08 13:14:47 +00:00
CLAUDE.md Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
csv_exporter.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
data_access.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
database.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
docker-compose.yml Reformat docker-compose command arrays and update tsconfig buildinfo 2026-02-09 20:52:44 +00:00
Dockerfile Add API anti-abuse hardening: disable docs in prod, origin validator, exception handler 2026-02-08 20:06:46 +00:00
GUIDE Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
listing_processor.py Fix buy listing support: thread ListingType through processing pipeline 2026-02-07 23:34:08 +00:00
main.py Add POI CLI commands: add-poi, list-pois, calculate-poi 2026-02-08 13:15:24 +00:00
notifications.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
podman-compose.yml Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
poetry.lock Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
pyproject.toml Add comprehensive test suite: 219 new tests across backend and frontend 2026-02-10 21:59:45 +00:00
README.md Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
redis_repository.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
requirements.txt Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
runall.sh Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
start.sh Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
TASKS.md Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
ui_exporter.py Fix buy listing support: thread ListingType through processing pipeline 2026-02-07 23:34:08 +00:00

Setup

  1. Instal deps:
poetry install && cp .env.sample .env
  1. Check .env if you want to customize settings for broker and db
  2. run ./start.sh

This starts the backend

To start the fronend:

cd frontend && cp .env.sample .env

Change the DEV_HOST to any name you want to use to access the web interface.

Next, setup the DNS record (e.g in your /etc/hosts) file. This is important as auth is done via external [authentik] service that needs to redirect to a name.

Run ./start.sh

This starts a Caddy proxy with correct certificates, and npm dev server. All requests going to the frontend are forwarded to the npm server and the ones for the backed (that go to /api/*) are forwarded to the backend service.

Lastly, reachout to Viktor to allowlist your DEV_HOST so that authentik can authorize callbacks to your host.

Formatting

yapf --style .style.yapf --recursive .

For VSCode - install yapf extension. Enable formatting using yap and the style file in this repo (there may be an easier way; I put this in my user settings json):

{
    "[python]": {
        "editor.formatOnSaveMode": "file",
        "editor.formatOnSave": true,
        "editor.defaultFormatter": "eeyore.yapf",
        "editor.formatOnType": false
      },
      "yapf.args": ["--style", "/home/wizard/code/realestate-crawler/crawler/.style.yapf"]
}

ADB commands (from /Applications/BlueStacks.app/Contents/MacOS):

Set proxy

./hd-adb shell settings put global http_proxy 192.168.9.110:8080

Disable proxy:

/hd-adb shell settings put global http_proxy :0

Connect adb

./hd-adb connect 127.0.0.1:5555

Disconnect adb

/hd-adb disconnect