The listing processor was hardcoded to create RentListing objects and query only the rentlisting table. Buy listings fetched from Rightmove were stored in the wrong table with missing fields. This threads ListingType through ListingProcessor and all Step subclasses so the correct model (RentListing/BuyListing) is created, the correct table is queried, and buy-specific fields (service_charge, lease_left) are parsed from the API response and included in GeoJSON streaming output. |
||
|---|---|---|
| .claude | ||
| .github/workflows | ||
| alembic | ||
| api | ||
| config | ||
| crawler | ||
| docs | ||
| frontend | ||
| models | ||
| rec | ||
| repositories | ||
| services | ||
| tasks | ||
| tests | ||
| utils | ||
| .dockerignore | ||
| .drone.yml | ||
| .env.sample | ||
| .gitignore | ||
| .style.yapf | ||
| alembic.ini | ||
| celery_app.py | ||
| CLAUDE.md | ||
| csv_exporter.py | ||
| data_access.py | ||
| database.py | ||
| docker-compose.yml | ||
| Dockerfile | ||
| GUIDE | ||
| listing_processor.py | ||
| main.py | ||
| notifications.py | ||
| podman-compose.yml | ||
| poetry.lock | ||
| pyproject.toml | ||
| README.md | ||
| redis_repository.py | ||
| requirements.txt | ||
| runall.sh | ||
| start.sh | ||
| TASKS.md | ||
| ui_exporter.py | ||
Setup
- Instal deps:
poetry install && cp .env.sample .env
- Check
.envif you want to customize settings for broker and db - run
./start.sh
This starts the backend
To start the fronend:
cd frontend && cp .env.sample .env
Change the DEV_HOST to any name you want to use to access the web interface.
Next, setup the DNS record (e.g in your /etc/hosts) file. This is important as auth is done via external [authentik] service that needs to redirect to a name.
Run ./start.sh
This starts a Caddy proxy with correct certificates, and npm dev server.
All requests going to the frontend are forwarded to the npm server and the ones for the backed (that go to /api/*) are forwarded to the backend service.
Lastly, reachout to Viktor to allowlist your DEV_HOST so that authentik can authorize callbacks to your host.
Formatting
yapf --style .style.yapf --recursive .
For VSCode - install yapf extension. Enable formatting using yap and the style file in this repo (there may be an easier way; I put this in my user settings json):
{
"[python]": {
"editor.formatOnSaveMode": "file",
"editor.formatOnSave": true,
"editor.defaultFormatter": "eeyore.yapf",
"editor.formatOnType": false
},
"yapf.args": ["--style", "/home/wizard/code/realestate-crawler/crawler/.style.yapf"]
}
ADB commands (from /Applications/BlueStacks.app/Contents/MacOS):
Set proxy
./hd-adb shell settings put global http_proxy 192.168.9.110:8080
Disable proxy:
/hd-adb shell settings put global http_proxy :0
Connect adb
./hd-adb connect 127.0.0.1:5555
Disconnect adb
/hd-adb disconnect