Viktor Barzin
f880664a98
Add throttling detection and circuit breaker for Rightmove scraper
2026-02-06 20:47:36 +00:00
Viktor Barzin
e8293c6042
Add intelligent query splitting to maximize Rightmove data extraction
2026-02-06 20:47:36 +00:00
Viktor Barzin
29ba739063
Improve login UI with error handling and callback page
2026-02-02 20:33:40 +00:00
Viktor Barzin
ceb943f198
Fix refresh listings returning immediate success with no progress
...
The get_ids_to_process function was using set union instead of set
difference, causing it to return all existing listing IDs along with
new ones. This meant:
1. When there were no new listings, the task would iterate through all
existing listings, find nothing to process for each, and complete
almost instantly
2. The task showed no progress because processing was too fast
Fixed by:
- Changed `all_listing_ids.union(identifiers)` to `identifiers - all_listing_ids`
to only return IDs that are NOT already in the database
- Added explicit check for empty set with informative task state
"No new listings found" so users understand why the task completed quickly
2026-02-01 22:30:37 +00:00
Viktor Barzin
4b1e971edf
delete detect floorplan as it was renamed
2026-02-01 21:58:16 +00:00
Viktor Barzin
22aa9c86a7
update drone.yaml to build on push
2026-02-01 21:53:18 +00:00
Viktor Barzin
f88bba032f
Test without secret
2026-02-01 20:58:20 +00:00
Viktor Barzin
dc1186601a
Final webhook test
2026-02-01 20:54:53 +00:00
Viktor Barzin
fe01df0e7e
delete dump images as it is renamed
2026-02-01 20:53:40 +00:00
Viktor Barzin
4969df8745
Test new webhook
2026-02-01 20:51:13 +00:00
Viktor Barzin
ccef50b371
Trigger webhook
2026-02-01 20:47:50 +00:00
Viktor Barzin
1680cda7b7
Test webhook with secret
2026-02-01 20:45:52 +00:00
Viktor Barzin
835a2a9d53
Fix stuck Celery tasks and add purge all tasks functionality
2026-02-01 20:40:07 +00:00
Viktor Barzin
93f7f57de3
Trigger build test
2026-02-01 20:29:56 +00:00
Viktor Barzin
6d17d2ccad
Fix Drone CI trigger to allow push events
...
Remove cron filter that was blocking push events from triggering builds.
The cron filter was applying to all events, not just cron jobs.
2026-02-01 20:25:32 +00:00
Viktor Barzin
61e1832bc1
delete 1 dump listings as it is renamed
2026-02-01 20:23:18 +00:00
Viktor Barzin
4d7ca7b920
update dockerignore
2026-02-01 20:22:03 +00:00
Viktor Barzin
997084d0f2
Trigger CI build test
2026-02-01 20:18:35 +00:00
Viktor Barzin
3f427b062c
Test push trigger
2026-02-01 20:03:53 +00:00
Viktor Barzin
aad9bdd950
Add push trigger to Drone CI pipeline
...
Build on every push to master branch in addition to nightly cron builds.
2026-02-01 19:56:00 +00:00
Viktor Barzin
218781e038
Add visual progress bar to task indicator
2026-02-01 19:23:27 +00:00
Viktor Barzin
0ca955796e
Show processed/total counts in task progress indicator
2026-02-01 19:19:59 +00:00
Viktor Barzin
6d8f69610f
Add proper buy listing support with type-aware UI filters and display
2026-02-01 19:13:29 +00:00
Viktor Barzin
c7ac448f15
Add configurable scheduling, UI health/task indicators, and auto-load map with default filters
2026-02-01 17:28:37 +00:00
Viktor Barzin
1c8c3e4657
add alembic mutation for logitute name
2026-01-28 21:00:46 +00:00
Viktor Barzin
ced9a153bd
replace pymysql with mysqlclient as it is "better"; also fix an issue in the ui exporter that had wrong imports
2025-10-18 09:58:55 +00:00
Kadir
0801aaf200
More ruff fixes ( #2 )
...
* adding ruff auto check for pull requests as well as fixing all ruff errors
* More ruff fixes: forgot half of the ruff checks
Forgot to do a git add all :D
---------
Co-authored-by: Kadir <git@k8n.dev>
2025-09-14 19:44:03 +01:00
Kadir
4c23acdb55
adding ruff auto check for pull requests as well as fixing all ruff errors ( #1 )
...
Co-authored-by: Kadir <git@k8n.dev>
2025-09-14 19:40:18 +01:00
Kadir
b1e0a414cf
Used ruff to cleanup
...
I hope it just works right as I cannot test things if they work
2025-09-14 19:02:30 +01:00
Kadir
8d11e4a81c
Fix deps and move to a better local environment
...
- Cleaned up some deps and moved them to the dev section
- Moved from mysqlclient to pymysql which is a python native one which does not require the OS to have the correct mysql lib
- Added a podman compose file so we can have all dependencies in one place easily without the need to install redis or a database locally
For podman install
- podman
- podman-compose (do a poetry sync I think?)
- podman-compose up to start the containers
2025-09-14 19:00:26 +01:00
Viktor Barzin
520286aaee
update readme with instructions on how to run everything
2025-08-28 21:39:20 +00:00
Viktor Barzin
62329a2eb4
add redis container in start.sh in case it is not running in dev mode
2025-08-28 20:48:42 +00:00
Viktor Barzin
ff57117054
do not watch files with uvicorn as the datadir is quite big and monitor is very very slow
2025-08-23 22:36:57 +00:00
Viktor Barzin
526f4fc0c3
update last seen property when processing listings to refresh data
2025-08-23 22:36:37 +00:00
Viktor Barzin
480957dc72
add timeout when fetching details and use new entrypoint for task processing
2025-08-23 22:20:42 +00:00
Viktor Barzin
e5c68f6bb7
add opentelemetry
2025-08-02 17:25:56 +00:00
Viktor Barzin
d1cef99c5a
make task processing a bit better. still doing 1 query to check if needs processing; will fix later
2025-07-27 20:09:41 +00:00
Viktor Barzin
87efe0694c
format progress to 2 digits and add status updates before starting
2025-07-27 18:47:09 +00:00
Viktor Barzin
91a0436f7f
migrate processing to a pipeline approach where each listing is processed in a pipeline in parallel and status reported back to track progress
2025-07-27 18:33:39 +00:00
Viktor Barzin
4fa09e31c8
make property image clickable
2025-07-27 11:11:04 +00:00
Viktor Barzin
42ed20b833
read slack notification webhook url from env
2025-07-26 13:15:21 +00:00
Viktor Barzin
4e7734d327
add healtcheck api endpoint
2025-07-26 13:06:28 +00:00
Viktor Barzin
206471cee8
fix argument error in tasks
2025-07-26 10:38:51 +00:00
Viktor Barzin
272d54d014
add daily scrape of interesting rent listings
2025-07-25 22:14:45 +00:00
Viktor Barzin
762408e054
add apprise and send notification when refreshing listings
2025-07-25 21:32:06 +00:00
Viktor Barzin
ce386e748d
default to last seen 28 days from 7
2025-07-15 18:52:25 +00:00
Viktor Barzin
c98f6ff97f
hack - add available from parameter only for rent listings
2025-07-06 12:18:32 +00:00
Viktor Barzin
d4b22deda0
save user queries in redis so that user can refresh the page and still come back to their latest task
2025-07-06 12:02:25 +00:00
Viktor Barzin
a055c92dea
split listing query into multiple subqueries to avoid hittign the 1.5k rightmove limit
2025-07-06 12:00:12 +00:00
Viktor Barzin
241b8d1842
bugfix fetching prices for buy listings
2025-07-06 11:48:36 +00:00
