No description
Find a file
Viktor Barzin d6edb747d2
Add structured JSON logging, OTel business metrics, and Grafana dashboard
Structured logging via JsonFormatter replaces uvicorn's default format so
Loki can parse timestamps and fields.  14 business metrics (scrape stats,
throttle events, circuit breaker state, cache hit rate, OCR success rate,
Celery task lifecycle) are defined in a shared metrics module and
instrumented across the scraper pipeline, API, and workers.  Celery
workers expose a Prometheus HTTP endpoint on configurable ports.
2026-02-14 10:59:12 +00:00
.claude Add Claude agent definitions [ci skip] 2026-02-09 23:02:30 +00:00
.github/workflows adding ruff auto check for pull requests as well as fixing all ruff errors (#1) 2025-09-14 19:40:18 +01:00
alembic Fix alembic migration to rename buylisting.longtitude to longitude 2026-02-09 20:47:30 +00:00
api Add structured JSON logging, OTel business metrics, and Grafana dashboard 2026-02-14 10:59:12 +00:00
config Fix 7 bugs: security, memory leak, stale state, error handling 2026-02-13 19:36:43 +00:00
docs Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
frontend Auto-redirect to login on 401 API responses 2026-02-13 21:16:53 +00:00
grafana Add structured JSON logging, OTel business metrics, and Grafana dashboard 2026-02-14 10:59:12 +00:00
k8s Add Kubernetes manifests for routing engines 2026-02-08 13:16:38 +00:00
models Harden backend security: IDOR fix, error sanitization, rate limiter fallback, security headers 2026-02-08 19:42:30 +00:00
rec Add structured JSON logging, OTel business metrics, and Grafana dashboard 2026-02-14 10:59:12 +00:00
repositories Fix POI distance calculation reliability for remote/Celery execution 2026-02-08 20:11:12 +00:00
scripts Fix OSRM setup: update Geofabrik URL and use bind mount for data 2026-02-08 14:51:52 +00:00
services Refactor backend for cleaner error handling, DRY, and type safety 2026-02-10 22:19:24 +00:00
tasks Add structured JSON logging, OTel business metrics, and Grafana dashboard 2026-02-14 10:59:12 +00:00
tests Fix 7 bugs: security, memory leak, stale state, error handling 2026-02-13 19:36:43 +00:00
utils Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
.dockerignore Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
.drone.yml Switch frontend build to kaniko to avoid ephemeral-storage eviction 2026-02-13 20:12:15 +00:00
.env.sample Add OSRM and OTP Docker services with setup scripts 2026-02-08 13:16:10 +00:00
.gitignore Fix OSRM setup: update Geofabrik URL and use bind mount for data 2026-02-08 14:51:52 +00:00
.style.yapf Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
alembic.ini Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
celery_app.py Add structured JSON logging, OTel business metrics, and Grafana dashboard 2026-02-14 10:59:12 +00:00
CLAUDE.md Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
csv_exporter.py Refactor backend for cleaner error handling, DRY, and type safety 2026-02-10 22:19:24 +00:00
data_access.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
database.py Fall back to SQLite default when DB_CONNECTION_STRING is unset 2026-02-10 22:38:52 +00:00
docker-compose.yml Add structured JSON logging, OTel business metrics, and Grafana dashboard 2026-02-14 10:59:12 +00:00
Dockerfile Cache test image to speed up CI backend test step 2026-02-10 22:57:42 +00:00
GUIDE Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
listing_processor.py Add structured JSON logging, OTel business metrics, and Grafana dashboard 2026-02-14 10:59:12 +00:00
logging_config.py Add structured JSON logging, OTel business metrics, and Grafana dashboard 2026-02-14 10:59:12 +00:00
main.py Add POI CLI commands: add-poi, list-pois, calculate-poi 2026-02-08 13:15:24 +00:00
notifications.py Refactor backend for cleaner error handling, DRY, and type safety 2026-02-10 22:19:24 +00:00
podman-compose.yml Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
poetry.lock Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
pyproject.toml Add comprehensive test suite: 219 new tests across backend and frontend 2026-02-10 21:59:45 +00:00
README.md Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
redis_repository.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
requirements.txt Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
runall.sh Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
start.sh Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
TASKS.md Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
ui_exporter.py Refactor backend for cleaner error handling, DRY, and type safety 2026-02-10 22:19:24 +00:00

Setup

  1. Instal deps:
poetry install && cp .env.sample .env
  1. Check .env if you want to customize settings for broker and db
  2. run ./start.sh

This starts the backend

To start the fronend:

cd frontend && cp .env.sample .env

Change the DEV_HOST to any name you want to use to access the web interface.

Next, setup the DNS record (e.g in your /etc/hosts) file. This is important as auth is done via external [authentik] service that needs to redirect to a name.

Run ./start.sh

This starts a Caddy proxy with correct certificates, and npm dev server. All requests going to the frontend are forwarded to the npm server and the ones for the backed (that go to /api/*) are forwarded to the backend service.

Lastly, reachout to Viktor to allowlist your DEV_HOST so that authentik can authorize callbacks to your host.

Formatting

yapf --style .style.yapf --recursive .

For VSCode - install yapf extension. Enable formatting using yap and the style file in this repo (there may be an easier way; I put this in my user settings json):

{
    "[python]": {
        "editor.formatOnSaveMode": "file",
        "editor.formatOnSave": true,
        "editor.defaultFormatter": "eeyore.yapf",
        "editor.formatOnType": false
      },
      "yapf.args": ["--style", "/home/wizard/code/realestate-crawler/crawler/.style.yapf"]
}

ADB commands (from /Applications/BlueStacks.app/Contents/MacOS):

Set proxy

./hd-adb shell settings put global http_proxy 192.168.9.110:8080

Disable proxy:

/hd-adb shell settings put global http_proxy :0

Connect adb

./hd-adb connect 127.0.0.1:5555

Disconnect adb

/hd-adb disconnect