Viktor Barzin
d205d15c74
Add services layer, tests, streaming UI, and cleanup legacy code
2026-02-06 20:55:10 +00:00
Kadir
0801aaf200
More ruff fixes ( #2 )
...
* adding ruff auto check for pull requests as well as fixing all ruff errors
* More ruff fixes: forgot half of the ruff checks
Forgot to do a git add all :D
---------
Co-authored-by: Kadir <git@k8n.dev>
2025-09-14 19:44:03 +01:00
Kadir
b1e0a414cf
Used ruff to cleanup
...
I hope it just works right as I cannot test things if they work
2025-09-14 19:02:30 +01:00
Viktor Barzin
91a0436f7f
migrate processing to a pipeline approach where each listing is processed in a pipeline in parallel and status reported back to track progress
2025-07-27 18:33:39 +00:00
Viktor Barzin
d4b22deda0
save user queries in redis so that user can refresh the page and still come back to their latest task
2025-07-06 12:02:25 +00:00
Viktor Barzin
a7e0773c0a
add worker api to refresh data in the background
2025-06-21 12:49:04 +00:00
Viktor Barzin
289206afc0
some cleanups
2025-06-08 20:58:28 +00:00
Viktor Barzin
3785d01009
migrate immoweb exporter to use models
2025-06-08 18:18:38 +00:00
Viktor Barzin
e317d2ec54
use query params to filter out models; also make csv exporter work with models
2025-06-08 17:01:33 +00:00
Viktor Barzin
80c335ba04
migrate routing command to use the models and store data there
2025-06-08 11:45:05 +00:00
Viktor Barzin
325823e631
refactor detect floorplan to use model listings
2025-06-07 14:30:32 +00:00
Viktor Barzin
ba87d07cd2
migrate dump images command to use model listings
2025-06-07 13:56:00 +00:00
Viktor Barzin
4f5a934fa9
refactor dump listings to start using model instead of the data_access object
2025-06-07 12:46:53 +00:00
Viktor Barzin
842f7cefbe
merge dump listings and dump details commands - fetch both details and listings in the same command
2025-06-07 12:00:23 +00:00
Viktor Barzin
b7a2ea75aa
add json field to store any additional blob of data that may be missing; also populate db when dumping listings
2025-06-06 19:57:50 +00:00
Viktor Barzin
8b2025e700
add command to dump existing listing from fs to db
2025-06-04 22:03:55 +00:00
Viktor Barzin
0bdfeec195
make the csv exporter use the filtering params to allow exporting customizable reports
2025-06-01 20:11:00 +00:00
Viktor Barzin
0acd417d34
add option to filter for min sqm per listing
2025-06-01 19:26:24 +00:00
Viktor Barzin
8b90ecde11
add filter for last seen days
2025-06-01 15:26:38 +00:00
Viktor Barzin
11315359d2
reuse query params when exporting to immoweb and allow filtering from available date
2025-06-01 15:17:14 +00:00
Viktor Barzin
a23a5ae192
extract common listing filtering options into a decorator to enable reuse between commands
2025-06-01 12:11:15 +00:00
Viktor Barzin
0b9d50af47
reformat with black; looks better
2025-05-31 23:50:43 +00:00
Viktor Barzin
7e8c79d3d1
add command to export the data in a way that the ui (immoweb) can consume
2025-05-26 19:37:06 +00:00
Viktor Barzin
10ae25e0d3
allow caching routing by destination and travel mode; also export all travel methods to csv column
2025-05-20 22:12:56 +00:00
Viktor Barzin
d9fe5421fc
fix type hint in paths vs strings
2025-05-18 21:31:51 +00:00
Viktor Barzin
482fff689b
parameterize routing logic - extract api key as env var; allow searching dest by address; limit the number of listings to process to prevent accidental api key usage
2025-05-18 21:13:50 +00:00
Viktor Barzin
9f3e466b23
add filter for furnished/unfurnished type for rented listings
2025-05-18 17:22:48 +00:00
Viktor Barzin
b1e0ed170b
detect floorplan using asyncio
2025-05-17 22:58:35 +00:00
Viktor Barzin
01ac24b4b7
use aiohttp to fetch details concurrently
2025-05-17 22:34:27 +00:00
Viktor Barzin
3e7a144fb4
make dumping details async
2025-05-17 22:11:33 +00:00
Viktor Barzin
ad879f2d4f
convert listings dump to asyncio
2025-05-17 21:55:42 +00:00
Viktor Barzin
df24c2c1b7
add cli param for querying properties to rent
...
example:
python main.py --data-dir data/rs2 dump-listings --max-price 3500 --min-bedrooms 2 --max-bedrooms 4 --district islington -t rent
2025-05-17 21:22:39 +00:00
Viktor Barzin
bb9afc76fe
expose rightmove query parameters as cli options
2025-05-17 20:40:44 +00:00
Viktor Barzin
96562c0895
add csv exporter command
2025-05-17 20:13:28 +00:00
Viktor Barzin
ca5619976f
parameterize routing step to work with custom data paths
2025-05-14 21:08:03 +00:00
Viktor Barzin
e3abf4b373
parameterize detect floorplan step to work with custom data paths
2025-05-14 21:05:59 +00:00
Viktor Barzin
07fef7fbab
parameterize dump images step to work with custom data paths
2025-05-14 21:02:32 +00:00
Viktor Barzin
e424361ed9
parameterize dump_detail to use a custom data dir and also move data dir param as part of the click context
2025-05-14 20:32:37 +00:00
Viktor Barzin
48d379567b
parameterize data path when fetching listings
2025-05-14 20:19:08 +00:00
Viktor Barzin
ea56555884
refactor main.py click to use click commands to allow passing parameters to commands and enable fetching districts by district name
2025-05-14 19:42:08 +00:00
Viktor Barzin
962c9a2f38
print all steps that we are running with
2025-05-11 19:13:19 +00:00
Viktor Barzin
9134145e02
[5/n] click-ify add routing command
...
run with:
poetry run python main.py --step routing
2025-05-11 19:11:23 +00:00
Viktor Barzin
48f694e002
[4/n] click-ify add detect floorplan command
...
run with
poetry run python main.py --step detect_floorplan
2025-05-11 19:06:08 +00:00
Viktor Barzin
70e8ef9f95
[3/n] click-ify add dump images command
...
run with
poetry run python main.py --step dump_images
2025-05-11 19:04:19 +00:00
Viktor Barzin
c2196c15c1
[2/n] click-ify - add 2_dump_detail command
...
run with
poetry run python main.py --step dump_detail
2025-05-11 19:02:23 +00:00
Viktor Barzin
90b531f5d9
[1/n] click-ify - add entrypoint for click script and add
...
1_dump_listings command
run via:
poetry run python main.py --step dump_listings
2025-05-11 18:59:41 +00:00
