Structured logging via JsonFormatter replaces uvicorn's default format so Loki can parse timestamps and fields. 14 business metrics (scrape stats, throttle events, circuit breaker state, cache hit rate, OCR success rate, Celery task lifecycle) are defined in a shared metrics module and instrumented across the scraper pipeline, API, and workers. Celery workers expose a Prometheus HTTP endpoint on configurable ports. |
||
|---|---|---|
| .claude | ||
| .github/workflows | ||
| alembic | ||
| api | ||
| config | ||
| docs | ||
| frontend | ||
| grafana | ||
| k8s | ||
| models | ||
| rec | ||
| repositories | ||
| scripts | ||
| services | ||
| tasks | ||
| tests | ||
| utils | ||
| .dockerignore | ||
| .drone.yml | ||
| .env.sample | ||
| .gitignore | ||
| .style.yapf | ||
| alembic.ini | ||
| celery_app.py | ||
| CLAUDE.md | ||
| csv_exporter.py | ||
| data_access.py | ||
| database.py | ||
| docker-compose.yml | ||
| Dockerfile | ||
| GUIDE | ||
| listing_processor.py | ||
| logging_config.py | ||
| main.py | ||
| notifications.py | ||
| podman-compose.yml | ||
| poetry.lock | ||
| pyproject.toml | ||
| README.md | ||
| redis_repository.py | ||
| requirements.txt | ||
| runall.sh | ||
| start.sh | ||
| TASKS.md | ||
| ui_exporter.py | ||
Setup
- Instal deps:
poetry install && cp .env.sample .env
- Check
.envif you want to customize settings for broker and db - run
./start.sh
This starts the backend
To start the fronend:
cd frontend && cp .env.sample .env
Change the DEV_HOST to any name you want to use to access the web interface.
Next, setup the DNS record (e.g in your /etc/hosts) file. This is important as auth is done via external [authentik] service that needs to redirect to a name.
Run ./start.sh
This starts a Caddy proxy with correct certificates, and npm dev server.
All requests going to the frontend are forwarded to the npm server and the ones for the backed (that go to /api/*) are forwarded to the backend service.
Lastly, reachout to Viktor to allowlist your DEV_HOST so that authentik can authorize callbacks to your host.
Formatting
yapf --style .style.yapf --recursive .
For VSCode - install yapf extension. Enable formatting using yap and the style file in this repo (there may be an easier way; I put this in my user settings json):
{
"[python]": {
"editor.formatOnSaveMode": "file",
"editor.formatOnSave": true,
"editor.defaultFormatter": "eeyore.yapf",
"editor.formatOnType": false
},
"yapf.args": ["--style", "/home/wizard/code/realestate-crawler/crawler/.style.yapf"]
}
ADB commands (from /Applications/BlueStacks.app/Contents/MacOS):
Set proxy
./hd-adb shell settings put global http_proxy 192.168.9.110:8080
Disable proxy:
/hd-adb shell settings put global http_proxy :0
Connect adb
./hd-adb connect 127.0.0.1:5555
Disconnect adb
/hd-adb disconnect