- Increase Redis cache TTL from 30 minutes to 24 hours
- Add stale-while-revalidate: serve stale cache (>4h) immediately while
repopulating in background with SETNX lock to prevent concurrent rebuilds
- Add in-memory frontend LRU cache (5 entries) so repeat filter visits
are instant without network requests
- Invalidate frontend cache on listing refresh and task completion
- Add unit tests for get_cache_age, is_cache_stale, acquire_repopulation_lock
Three-pronged fix for duplicate listings appearing in the UI:
1. Backend: Replace direct rpush cache writes with staged population
(write to temp key, then atomic RENAME to live key). Skip cache
writes entirely for POI-enriched requests. Clean staging keys on
invalidation.
2. Frontend: Add AbortController to cancel in-flight streaming requests
when loadListings is called again, preventing data mixing.
3. Frontend: Deduplicate features by URL during stream accumulation as
a safety net against any remaining server-side duplicates.
The crawler subdirectory was the only active project. Moving it to the
repo root simplifies paths and removes the unnecessary nesting. The
vqa/ and immoweb/ directories were legacy/unused and have been removed.
Updated .drone.yml, .gitignore, .claude/ docs, and skills to reflect
the new flat structure.
2026-02-07 23:01:20 +00:00
Renamed from crawler/services/listing_cache.py (Browse further)
