Commit graph

5 commits

Author SHA1 Message Date
Viktor Barzin
f6f0aa8d14
Fix metrics not initialized in Celery prefork worker processes
worker_ready fires in the main process after pool workers are forked,
so init_metrics() never ran in the child processes. Add a
worker_process_init handler to initialize OTel metrics in each worker.
2026-02-15 12:58:59 +00:00
Viktor Barzin
25912eac0c
Fix metric imports: use module-level access instead of name imports
Type-annotated metric variables (e.g. `geojson_cache_operations: Counter`)
don't exist as importable names until init_metrics() runs.  Switch all
`from api.metrics import <metric>` to `import api.metrics as m` and
access instruments as attributes at runtime to avoid ImportError.
2026-02-14 11:21:49 +00:00
Viktor Barzin
d6edb747d2
Add structured JSON logging, OTel business metrics, and Grafana dashboard
Structured logging via JsonFormatter replaces uvicorn's default format so
Loki can parse timestamps and fields.  14 business metrics (scrape stats,
throttle events, circuit breaker state, cache hit rate, OCR success rate,
Celery task lifecycle) are defined in a shared metrics module and
instrumented across the scraper pipeline, API, and workers.  Celery
workers expose a Prometheus HTTP endpoint on configurable ports.
2026-02-14 10:59:12 +00:00
Viktor Barzin
bd788df9aa
Add POI API routes and Celery task
FastAPI router with CRUD endpoints for POIs, distance calculation
trigger, and distance queries. Streaming GeoJSON endpoint now accepts
include_poi_distances=true to inject travel times into features.
Celery task wraps the distance calculator with progress reporting.
2026-02-08 13:14:47 +00:00
Viktor Barzin
eafbc1ac52
Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/
The crawler subdirectory was the only active project. Moving it to the
repo root simplifies paths and removes the unnecessary nesting. The
vqa/ and immoweb/ directories were legacy/unused and have been removed.

Updated .drone.yml, .gitignore, .claude/ docs, and skills to reflect
the new flat structure.
2026-02-07 23:01:20 +00:00
Renamed from crawler/celery_app.py (Browse further)