Viktor Barzin
3d024277fb
add uvicorn logger when dumping listings
2025-06-22 21:15:35 +00:00
Viktor Barzin
744fa7b8a7
use the uvicorn logger; also update dockerfile to run migrations as parto f startup instead of when starting uvicorn
2025-06-22 14:00:47 +00:00
Viktor Barzin
9eae30b0d6
bugfix - pages start from 1
2025-06-22 13:04:10 +00:00
Viktor Barzin
03d24aff99
refactor dump listings to extract all pages with data
2025-06-22 12:43:24 +00:00
Viktor Barzin
ec2e0dded8
add method to dump all data for listings e2e
2025-06-21 12:04:31 +00:00
Viktor Barzin
296a4e7603
bugfix fetching transactions to make use of db cache
2025-06-09 21:45:29 +00:00
Viktor Barzin
289206afc0
some cleanups
2025-06-08 20:58:28 +00:00
Viktor Barzin
831c45e1f5
use cached data when fetching details
2025-06-08 18:23:25 +00:00
Viktor Barzin
ba87d07cd2
migrate dump images command to use model listings
2025-06-07 13:56:00 +00:00
Viktor Barzin
4f5a934fa9
refactor dump listings to start using model instead of the data_access object
2025-06-07 12:46:53 +00:00
Viktor Barzin
842f7cefbe
merge dump listings and dump details commands - fetch both details and listings in the same command
2025-06-07 12:00:23 +00:00
Viktor Barzin
29213f3d26
refactor the semaphore when dumping listings
2025-06-06 20:09:16 +00:00
Viktor Barzin
11315359d2
reuse query params when exporting to immoweb and allow filtering from available date
2025-06-01 15:17:14 +00:00
Viktor Barzin
9735db72a0
limit the number of concurrenct requests when dumping listings as right move blocks us
2025-06-01 00:27:12 +00:00
Viktor Barzin
0b9d50af47
reformat with black; looks better
2025-05-31 23:50:43 +00:00
Viktor Barzin
102c20ac42
[bugfix] if no district is passed, fetch data for all
2025-05-26 19:31:33 +00:00
Viktor Barzin
9f3e466b23
add filter for furnished/unfurnished type for rented listings
2025-05-18 17:22:48 +00:00
Viktor Barzin
b873eaf203
fix types and format
2025-05-18 12:30:49 +00:00
Viktor Barzin
ad879f2d4f
convert listings dump to asyncio
2025-05-17 21:55:42 +00:00
Viktor Barzin
df24c2c1b7
add cli param for querying properties to rent
...
example:
python main.py --data-dir data/rs2 dump-listings --max-price 3500 --min-bedrooms 2 --max-bedrooms 4 --district islington -t rent
2025-05-17 21:22:39 +00:00
Viktor Barzin
bb9afc76fe
expose rightmove query parameters as cli options
2025-05-17 20:40:44 +00:00
Viktor Barzin
48d379567b
parameterize data path when fetching listings
2025-05-14 20:19:08 +00:00
Viktor Barzin
ea56555884
refactor main.py click to use click commands to allow passing parameters to commands and enable fetching districts by district name
2025-05-14 19:42:08 +00:00
Viktor Barzin
90b531f5d9
[1/n] click-ify - add entrypoint for click script and add
...
1_dump_listings command
run via:
poetry run python main.py --step dump_listings
2025-05-11 18:59:41 +00:00
Kadir
29c8d1960b
adding last seen date into the listing
2025-01-26 21:41:18 +00:00
Kadir
4b6b8628c2
add runall script, update parameters to 4 bed etc and allow incremental updating
2024-11-23 22:57:22 +00:00
Kadir
b16fee0648
changing to 500 listings per query
2024-04-01 20:28:15 +02:00
Kadir
5720e68547
Rewriting 1_dump to enabling crawling of more real estate
2024-03-30 19:23:19 +01:00
Kadir
d777558b34
ruff format
2024-03-25 20:48:48 +00:00
Kadir
4dea766a12
fixing floorplan detection and adding recalculation method
2024-03-18 00:56:39 +00:00
Kadir
335adc0856
add routing, incremental crawling, travel time, lease and development
2024-03-13 16:24:57 +00:00
Kadir
36258d877f
crawling for 3 and refactoring to allow incremental crawls
2024-03-11 14:43:53 +00:00
Kadir
508aa02812
Real crawling scripts and floorplan detection
...
1. get all listings
2. get all detail jsons
3. get all images
4. get all floorplans
5. detecting floorplans
Also updating dependencies for huggingface etc.
2024-03-10 18:49:39 +00:00
