Stream-process listings as IDs arrive via asyncio.Queue

Replace the sequential fetch-all-then-process pipeline with a streaming
architecture where listing processing starts as soon as IDs become
available from each subquery. A producer task fetches pages and enqueues
new IDs (filtered inline against DB), while 20 consumer workers process
listings concurrently from the queue.

- Add ListingRepository.get_listing_ids() for fast ID-only projection
- Refactor listing_tasks.py: remove get_ids_to_process/dump_listings_and_monitor,
  replace with unified producer/worker/monitor pipeline
- Apply same pattern to CLI path in listing_fetcher.py
- Remove 'filtering' phase from frontend, show combined fetch+process metrics
- Add fetching_done flag to TaskResult for phase transition tracking
This commit is contained in:
Viktor Barzin 2026-02-06 23:43:54 +00:00
parent 7e8f1f0339
commit b9f576ae2b
No known key found for this signature in database
GPG key ID: 0EB088298288D958
6 changed files with 372 additions and 420 deletions

View file

@ -180,7 +180,6 @@ export function TaskIndicator({ taskID, onTaskCancelled }: TaskIndicatorProps) {
splitting: 'Splitting',
splitting_complete: 'Split done',
fetching: 'Fetching',
filtering: 'Filtering',
};
return phaseLabels[taskResult.phase] ?? `${Math.round(progressPercentage)}%`;
}