viktor/wrongmove
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

migrate processing to a pipeline approach where each listing is processed in a pipeline in parallel and status reported back to track progress

Browse source
This commit is contained in:
Viktor Barzin 2025-07-27 18:33:39 +00:00
parent 4fa09e31c8
commit 91a0436f7f
No known key found for this signature in database
GPG key ID: 4056458DBDBF8863
6 changed files with 347 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

2
crawler/api/app.py
View file

@ -91,6 +91,8 @@ async def refresh_listings(
await send_notification(
f"{user.email} refreshing listings with query parameters {query_parameters.model_dump_json()}"
)
# await listing_tasks.async_dump_listings_task(query_parameters.model_dump_json()) # Use this for local debugging - run task in sync
# return {}
# TODO: rate limit
expiry_time = datetime.now() + timedelta(minutes=10)
task = listing_tasks.dump_listings_task.apply_async(