Commit graph

11 commits

Author SHA1 Message Date
Viktor Barzin
e8293c6042 Add intelligent query splitting to maximize Rightmove data extraction 2026-02-06 20:47:36 +00:00
Viktor Barzin
ceb943f198 Fix refresh listings returning immediate success with no progress
The get_ids_to_process function was using set union instead of set
difference, causing it to return all existing listing IDs along with
new ones. This meant:

1. When there were no new listings, the task would iterate through all
   existing listings, find nothing to process for each, and complete
   almost instantly

2. The task showed no progress because processing was too fast

Fixed by:
- Changed `all_listing_ids.union(identifiers)` to `identifiers - all_listing_ids`
  to only return IDs that are NOT already in the database
- Added explicit check for empty set with informative task state
  "No new listings found" so users understand why the task completed quickly
2026-02-01 22:30:37 +00:00
Viktor Barzin
0ca955796e Show processed/total counts in task progress indicator 2026-02-01 19:19:59 +00:00
Viktor Barzin
c7ac448f15 Add configurable scheduling, UI health/task indicators, and auto-load map with default filters 2026-02-01 17:28:37 +00:00
Viktor Barzin
d1cef99c5a
make task processing a bit better. still doing 1 query to check if needs processing; will fix later 2025-07-27 20:09:41 +00:00
Viktor Barzin
87efe0694c
format progress to 2 digits and add status updates before starting 2025-07-27 18:47:09 +00:00
Viktor Barzin
91a0436f7f
migrate processing to a pipeline approach where each listing is processed in a pipeline in parallel and status reported back to track progress 2025-07-27 18:33:39 +00:00
Viktor Barzin
206471cee8
fix argument error in tasks 2025-07-26 10:38:51 +00:00
Viktor Barzin
272d54d014
add daily scrape of interesting rent listings 2025-07-25 22:14:45 +00:00
Viktor Barzin
d4b22deda0
save user queries in redis so that user can refresh the page and still come back to their latest task 2025-07-06 12:02:25 +00:00
Viktor Barzin
93129333e6
migrate background tasks to celery 2025-06-22 21:18:52 +00:00