No description
Find a file
Viktor Barzin e5ce8c1201
Fix buy listing support: thread ListingType through processing pipeline
The listing processor was hardcoded to create RentListing objects and
query only the rentlisting table. Buy listings fetched from Rightmove
were stored in the wrong table with missing fields. This threads
ListingType through ListingProcessor and all Step subclasses so the
correct model (RentListing/BuyListing) is created, the correct table
is queried, and buy-specific fields (service_charge, lease_left) are
parsed from the API response and included in GeoJSON streaming output.
2026-02-07 23:34:08 +00:00
.claude Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
.github/workflows adding ruff auto check for pull requests as well as fixing all ruff errors (#1) 2025-09-14 19:40:18 +01:00
alembic Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
api Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
config Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
crawler Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
docs Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
frontend Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
models Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
rec Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
repositories Fix buy listing support: thread ListingType through processing pipeline 2026-02-07 23:34:08 +00:00
services Fix buy listing support: thread ListingType through processing pipeline 2026-02-07 23:34:08 +00:00
tasks Fix buy listing support: thread ListingType through processing pipeline 2026-02-07 23:34:08 +00:00
tests Fix buy listing support: thread ListingType through processing pipeline 2026-02-07 23:34:08 +00:00
utils Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
.dockerignore Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
.drone.yml Pin deployment image to build number tag instead of restarting latest 2026-02-07 23:07:16 +00:00
.env.sample Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
.gitignore Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
.style.yapf Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
alembic.ini Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
celery_app.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
CLAUDE.md Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
csv_exporter.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
data_access.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
database.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
docker-compose.yml Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
Dockerfile Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
GUIDE Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
listing_processor.py Fix buy listing support: thread ListingType through processing pipeline 2026-02-07 23:34:08 +00:00
main.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
notifications.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
podman-compose.yml Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
poetry.lock Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
pyproject.toml Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
README.md Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
redis_repository.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
requirements.txt Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
runall.sh Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
start.sh Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
TASKS.md Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
ui_exporter.py Fix buy listing support: thread ListingType through processing pipeline 2026-02-07 23:34:08 +00:00

Setup

  1. Instal deps:
poetry install && cp .env.sample .env
  1. Check .env if you want to customize settings for broker and db
  2. run ./start.sh

This starts the backend

To start the fronend:

cd frontend && cp .env.sample .env

Change the DEV_HOST to any name you want to use to access the web interface.

Next, setup the DNS record (e.g in your /etc/hosts) file. This is important as auth is done via external [authentik] service that needs to redirect to a name.

Run ./start.sh

This starts a Caddy proxy with correct certificates, and npm dev server. All requests going to the frontend are forwarded to the npm server and the ones for the backed (that go to /api/*) are forwarded to the backend service.

Lastly, reachout to Viktor to allowlist your DEV_HOST so that authentik can authorize callbacks to your host.

Formatting

yapf --style .style.yapf --recursive .

For VSCode - install yapf extension. Enable formatting using yap and the style file in this repo (there may be an easier way; I put this in my user settings json):

{
    "[python]": {
        "editor.formatOnSaveMode": "file",
        "editor.formatOnSave": true,
        "editor.defaultFormatter": "eeyore.yapf",
        "editor.formatOnType": false
      },
      "yapf.args": ["--style", "/home/wizard/code/realestate-crawler/crawler/.style.yapf"]
}

ADB commands (from /Applications/BlueStacks.app/Contents/MacOS):

Set proxy

./hd-adb shell settings put global http_proxy 192.168.9.110:8080

Disable proxy:

/hd-adb shell settings put global http_proxy :0

Connect adb

./hd-adb connect 127.0.0.1:5555

Disconnect adb

/hd-adb disconnect