wrongmove/crawler/services
Viktor Barzin b9f576ae2b
Stream-process listings as IDs arrive via asyncio.Queue
Replace the sequential fetch-all-then-process pipeline with a streaming
architecture where listing processing starts as soon as IDs become
available from each subquery. A producer task fetches pages and enqueues
new IDs (filtered inline against DB), while 20 consumer workers process
listings concurrently from the queue.

- Add ListingRepository.get_listing_ids() for fast ID-only projection
- Refactor listing_tasks.py: remove get_ids_to_process/dump_listings_and_monitor,
  replace with unified producer/worker/monitor pipeline
- Apply same pattern to CLI path in listing_fetcher.py
- Remove 'filtering' phase from frontend, show combined fetch+process metrics
- Add fetching_done flag to TaskResult for phase transition tracking
2026-02-06 23:43:54 +00:00
..
__init__.py Add services layer, tests, streaming UI, and cleanup legacy code 2026-02-06 20:55:10 +00:00
district_service.py Add services layer, tests, streaming UI, and cleanup legacy code 2026-02-06 20:55:10 +00:00
export_service.py Add services layer, tests, streaming UI, and cleanup legacy code 2026-02-06 20:55:10 +00:00
floorplan_detector.py Add services layer, tests, streaming UI, and cleanup legacy code 2026-02-06 20:55:10 +00:00
image_fetcher.py Add services layer, tests, streaming UI, and cleanup legacy code 2026-02-06 20:55:10 +00:00
listing_cache.py Remove 1000-result limit, add Redis caching and virtual scrolling 2026-02-06 20:47:36 +00:00
listing_fetcher.py Stream-process listings as IDs arrive via asyncio.Queue 2026-02-06 23:43:54 +00:00
listing_service.py Add services layer, tests, streaming UI, and cleanup legacy code 2026-02-06 20:55:10 +00:00
query_splitter.py Add crawl job progress drawer with phase tracking and live logs 2026-02-06 22:37:53 +00:00
route_calculator.py Add services layer, tests, streaming UI, and cleanup legacy code 2026-02-06 20:55:10 +00:00
task_service.py Add services layer, tests, streaming UI, and cleanup legacy code 2026-02-06 20:55:10 +00:00