Viktor Barzin
4f5a934fa9
refactor dump listings to start using model instead of the data_access object
2025-06-07 12:46:53 +00:00
Viktor Barzin
842f7cefbe
merge dump listings and dump details commands - fetch both details and listings in the same command
2025-06-07 12:00:23 +00:00
Viktor Barzin
8b2025e700
add command to dump existing listing from fs to db
2025-06-04 22:03:55 +00:00
Viktor Barzin
0acd417d34
add option to filter for min sqm per listing
2025-06-01 19:26:24 +00:00
Viktor Barzin
2e3e900ec3
store price history when fetching details. this will be helpful later on as we start periodically scraping to store prices over time
2025-06-01 18:09:27 +00:00
Viktor Barzin
11315359d2
reuse query params when exporting to immoweb and allow filtering from available date
2025-06-01 15:17:14 +00:00
Viktor Barzin
c543be7ff6
add let date available from parameter
2025-06-01 00:51:44 +00:00
Viktor Barzin
0b9d50af47
reformat with black; looks better
2025-05-31 23:50:43 +00:00
Viktor Barzin
f9a7620bf0
recalculate floorplans when theres only an empty object
2025-05-31 23:31:06 +00:00
Viktor Barzin
18b0e39495
add photo thumbnail to dict_nicely to allow visualizing in the ui
2025-05-31 21:57:03 +00:00
Viktor Barzin
10ae25e0d3
allow caching routing by destination and travel mode; also export all travel methods to csv column
2025-05-20 22:12:56 +00:00
Viktor Barzin
482fff689b
parameterize routing logic - extract api key as env var; allow searching dest by address; limit the number of listings to process to prevent accidental api key usage
2025-05-18 21:13:50 +00:00
Viktor Barzin
9067d45327
add council tax band when exporting
2025-05-18 18:02:19 +00:00
Viktor Barzin
5192b1955c
export agency
2025-05-18 17:41:50 +00:00
Viktor Barzin
b873eaf203
fix types and format
2025-05-18 12:30:49 +00:00
Viktor Barzin
b1e0ed170b
detect floorplan using asyncio
2025-05-17 22:58:35 +00:00
Viktor Barzin
96562c0895
add csv exporter command
2025-05-17 20:13:28 +00:00
Viktor Barzin
07fef7fbab
parameterize dump images step to work with custom data paths
2025-05-14 21:02:32 +00:00
Viktor Barzin
e424361ed9
parameterize dump_detail to use a custom data dir and also move data dir param as part of the click context
2025-05-14 20:32:37 +00:00
Viktor Barzin
48d379567b
parameterize data path when fetching listings
2025-05-14 20:19:08 +00:00
Viktor Barzin
835494d29f
reformat most things
2025-05-07 21:25:40 +00:00
Kadir
244de72877
reverting isRemoved function to fix
2025-02-16 04:44:42 +00:00
Kadir
302ca95cfb
fixing full detail dumping
2025-02-16 03:02:21 +00:00
Kadir
bb765a9312
making last_seen to a relative date
2025-02-14 21:21:50 +00:00
Kadir
29c8d1960b
adding last seen date into the listing
2025-01-26 21:41:18 +00:00
Kadir
f66007bc85
adding status and other fields
2024-09-22 11:31:32 +01:00
Kadir
6d343e52e7
adding days updated
2024-08-11 19:36:25 +01:00
Kadir
f98cd02696
adding better exception messages and interrupt free crawling
2024-05-06 18:54:55 +01:00
Kadir
69a5bba65e
adding service charge to excel
2024-04-05 11:38:31 +01:00
Kadir
966b9007a0
adding service charge as the info into the sheet
2024-04-05 11:37:03 +01:00
Kadir
d777558b34
ruff format
2024-03-25 20:48:48 +00:00
Kadir
47f7b2b672
Adding initial walking time and identifier to the information
2024-03-25 20:47:31 +00:00
Kadir
e79e24ff98
fix: remove debug statement
2024-03-18 01:02:06 +00:00
Kadir
4dea766a12
fixing floorplan detection and adding recalculation method
2024-03-18 00:56:39 +00:00
Kadir
335adc0856
add routing, incremental crawling, travel time, lease and development
2024-03-13 16:24:57 +00:00
Kadir
36258d877f
crawling for 3 and refactoring to allow incremental crawls
2024-03-11 14:43:53 +00:00
Kadir
de2639f9c3
fixing bugs, adding properties for querying and analysis§
2024-03-11 09:44:37 +00:00
Kadir
d108bf11ee
adding tesseract OCR for floorplan detection
2024-03-10 22:32:34 +00:00
Kadir
508aa02812
Real crawling scripts and floorplan detection
...
1. get all listings
2. get all detail jsons
3. get all images
4. get all floorplans
5. detecting floorplans
Also updating dependencies for huggingface etc.
2024-03-10 18:49:39 +00:00
