Commit graph

  • 0bdfeec195
    make the csv exporter use the filtering params to allow exporting customizable reports Viktor Barzin 2025-06-01 20:11:00 +00:00
  • 0acd417d34
    add option to filter for min sqm per listing Viktor Barzin 2025-06-01 19:26:24 +00:00
  • 1c77c78518
    add price history in the ui if theres more than 1 price history Viktor Barzin 2025-06-01 18:13:29 +00:00
  • ab5c4ce0a6
    refactor runall script to share filter options when dumping listings and when exporting to immoweb Viktor Barzin 2025-06-01 18:10:09 +00:00
  • 2e3e900ec3
    store price history when fetching details. this will be helpful later on as we start periodically scraping to store prices over time Viktor Barzin 2025-06-01 18:09:27 +00:00
  • 8b90ecde11
    add filter for last seen days Viktor Barzin 2025-06-01 15:26:38 +00:00
  • 11315359d2
    reuse query params when exporting to immoweb and allow filtering from available date Viktor Barzin 2025-06-01 15:17:14 +00:00
  • a23a5ae192
    extract common listing filtering options into a decorator to enable reuse between commands Viktor Barzin 2025-06-01 12:11:15 +00:00
  • c543be7ff6
    add let date available from parameter Viktor Barzin 2025-06-01 00:51:44 +00:00
  • 9735db72a0
    limit the number of concurrenct requests when dumping listings as right move blocks us Viktor Barzin 2025-06-01 00:27:12 +00:00
  • 24bf44caf9
    trust env when fetching details and listings. this allows setting proxy env vars. I use this in conjunction with a tor proxy to avoid ip-bans from rightmove Viktor Barzin 2025-06-01 00:12:31 +00:00
  • 0b9d50af47
    reformat with black; looks better Viktor Barzin 2025-05-31 23:50:43 +00:00
  • 1122f5a96f
    use aiohttp for dumping listing query Viktor Barzin 2025-05-31 23:48:45 +00:00
  • de1416de7e
    add last seen in immoweb Viktor Barzin 2025-05-31 23:32:59 +00:00
  • b312a53526
    remove exporting to csv in the runall script - we now use immoweb as a frontend Viktor Barzin 2025-05-31 23:31:41 +00:00
  • f9a7620bf0
    recalculate floorplans when theres only an empty object Viktor Barzin 2025-05-31 23:31:06 +00:00
  • abf06be0a7
    do not show listing where sqm is not available when we vizualive sqm-based metrics Viktor Barzin 2025-05-31 22:12:11 +00:00
  • 20ab7fde95
    add all of dict_nicely to the export for immoweb to allow using any of the available listing data Viktor Barzin 2025-05-31 21:57:44 +00:00
  • 18b0e39495
    add photo thumbnail to dict_nicely to allow visualizing in the ui Viktor Barzin 2025-05-31 21:57:03 +00:00
  • c7a2b23967
    upgrade immoweb mapboxgl to 3.12.0 and add on click events for listings Viktor Barzin 2025-05-31 21:55:52 +00:00
  • 151da16c27
    add immoweb ui Viktor Barzin 2025-05-26 19:41:36 +00:00
  • 7e8c79d3d1
    add command to export the data in a way that the ui (immoweb) can consume Viktor Barzin 2025-05-26 19:36:54 +00:00
  • 102c20ac42
    [bugfix] if no district is passed, fetch data for all Viktor Barzin 2025-05-26 19:31:33 +00:00
  • 56c203189f
    dockerize Viktor Barzin 2025-05-21 21:30:00 +00:00
  • 1e0f302178
    defer transformers and pytesseract imports to when used. this shortens startup time of all other commands quite a bit Viktor Barzin 2025-05-21 21:24:57 +00:00
  • 10ae25e0d3
    allow caching routing by destination and travel mode; also export all travel methods to csv column Viktor Barzin 2025-05-20 21:58:08 +00:00
  • f2118c9bc4
    update runall script to use new command for sourcing routes Viktor Barzin 2025-05-18 21:32:16 +00:00
  • d9fe5421fc
    fix type hint in paths vs strings Viktor Barzin 2025-05-18 21:24:04 +00:00
  • 482fff689b
    parameterize routing logic - extract api key as env var; allow searching dest by address; limit the number of listings to process to prevent accidental api key usage Viktor Barzin 2025-05-18 21:13:50 +00:00
  • 57f477c54d
    add .env.sample env with api key for routing service exported Viktor Barzin 2025-05-18 19:50:18 +00:00
  • ea24847cae
    add stratford region Viktor Barzin 2025-05-18 18:02:32 +00:00
  • 9067d45327
    add council tax band when exporting Viktor Barzin 2025-05-18 18:02:19 +00:00
  • 5192b1955c
    export agency Viktor Barzin 2025-05-18 17:41:50 +00:00
  • 9f3e466b23
    add filter for furnished/unfurnished type for rented listings Viktor Barzin 2025-05-18 17:22:48 +00:00
  • b873eaf203
    fix types and format Viktor Barzin 2025-05-18 12:27:26 +00:00
  • 91d3237516
    remove some empty fields in exploration notebook Viktor Barzin 2025-05-17 23:24:40 +00:00
  • 0b3c644ea3
    set the min price param in runall Viktor Barzin 2025-05-17 23:18:27 +00:00
  • 2741a77560
    sort output dataframe by price per sqm Viktor Barzin 2025-05-17 23:14:18 +00:00
  • 47347543d2
    update runall script to use the click entrypoint Viktor Barzin 2025-05-17 23:14:00 +00:00
  • b1e0ed170b
    detect floorplan using asyncio Viktor Barzin 2025-05-17 22:58:35 +00:00
  • 68cc70bd11
    dump images using aiohttp and concurrently Viktor Barzin 2025-05-17 22:42:37 +00:00
  • 01ac24b4b7
    use aiohttp to fetch details concurrently Viktor Barzin 2025-05-17 22:34:27 +00:00
  • 61b8c82592
    add ipdb to the dev deps Viktor Barzin 2025-05-17 22:11:51 +00:00
  • 3e7a144fb4
    make dumping details async Viktor Barzin 2025-05-17 22:11:33 +00:00
  • ad879f2d4f
    convert listings dump to asyncio Viktor Barzin 2025-05-17 21:55:42 +00:00
  • df24c2c1b7
    add cli param for querying properties to rent Viktor Barzin 2025-05-17 21:22:39 +00:00
  • bb9afc76fe
    expose rightmove query parameters as cli options Viktor Barzin 2025-05-17 20:40:44 +00:00
  • 96562c0895
    add csv exporter command Viktor Barzin 2025-05-17 20:13:28 +00:00
  • ca5619976f
    parameterize routing step to work with custom data paths Viktor Barzin 2025-05-14 21:08:03 +00:00
  • e3abf4b373
    parameterize detect floorplan step to work with custom data paths Viktor Barzin 2025-05-14 21:05:59 +00:00
  • 07fef7fbab
    parameterize dump images step to work with custom data paths Viktor Barzin 2025-05-14 21:01:58 +00:00
  • e424361ed9
    parameterize dump_detail to use a custom data dir and also move data dir param as part of the click context Viktor Barzin 2025-05-14 20:32:37 +00:00
  • 48d379567b
    parameterize data path when fetching listings Viktor Barzin 2025-05-14 20:19:08 +00:00
  • ea56555884 refactor main.py click to use click commands to allow passing parameters to commands and enable fetching districts by district name Viktor Barzin 2025-05-14 19:41:13 +00:00
  • bb2488a63b Removing sqlalchemy and the db part as it was never used Kadir 2025-05-12 01:17:45 +01:00
  • 3f4be8b7ff decisions + logger Kadir 2025-05-12 01:01:19 +01:00
  • 962c9a2f38
    print all steps that we are running with Viktor Barzin 2025-05-11 19:13:19 +00:00
  • 9134145e02
    [5/n] click-ify add routing command Viktor Barzin 2025-05-11 19:11:23 +00:00
  • 48f694e002
    [4/n] click-ify add detect floorplan command Viktor Barzin 2025-05-11 19:06:08 +00:00
  • 70e8ef9f95
    [3/n] click-ify add dump images command Viktor Barzin 2025-05-11 19:04:19 +00:00
  • c2196c15c1
    [2/n] click-ify - add 2_dump_detail command Viktor Barzin 2025-05-11 19:02:23 +00:00
  • 90b531f5d9
    [1/n] click-ify - add entrypoint for click script and add 1_dump_listings command Viktor Barzin 2025-05-11 18:59:41 +00:00
  • 0a66efa48a
    update pyproject.toml to use latest version of dev dependency section Viktor Barzin 2025-05-07 21:56:12 +00:00
  • 835494d29f
    reformat most things Viktor Barzin 2025-05-07 21:25:40 +00:00
  • bd7c781adb
    remove requirements.txt as we are using poetry...im blind.. Viktor Barzin 2025-05-07 21:06:43 +00:00
  • 028eef63a8
    add readme and .style.yapf for formatting Viktor Barzin 2025-05-07 20:58:22 +00:00
  • d24b667e73
    add sqlalchemy to requirements.txt Viktor Barzin 2025-05-07 20:57:28 +00:00
  • dd2de477bc
    Add requirements.txt Viktor Barzin 2025-05-07 17:31:29 +00:00
  • 8f8956818e refactor routing executor to: - be nicer - include checks on floorplan - include checks on if listing disabled Kadir 2025-03-31 03:02:54 +01:00
  • cd21bd0bb6 notebooks Kadir 2025-03-30 23:42:29 +01:00
  • 2c2adcfa7c improving OCR Kadir 2025-03-30 23:41:52 +01:00
  • 1b69fd4305 adding new dependencies Kadir 2025-03-02 17:11:48 +00:00
  • 244de72877 reverting isRemoved function to fix Kadir 2025-02-16 04:44:42 +00:00
  • 302ca95cfb fixing full detail dumping Kadir 2025-02-16 03:02:21 +00:00
  • e0e7853c8c updating districts of relevance to me Kadir 2025-02-14 21:35:44 +00:00
  • bb765a9312 making last_seen to a relative date Kadir 2025-02-14 21:21:50 +00:00
  • 29c8d1960b adding last seen date into the listing Kadir 2025-01-26 21:39:51 +00:00
  • 4b6b8628c2 add runall script, update parameters to 4 bed etc and allow incremental updating Kadir 2024-11-23 22:57:22 +00:00
  • dbf72e42e3 removing districts which are far away from the default crawl Kadir 2024-09-23 01:43:20 +01:00
  • f66007bc85 adding status and other fields Kadir 2024-09-22 11:31:32 +01:00
  • 4dfbcc64c1 floorplan rec: fix regex rule + filter out extreme values Kadir 2024-08-25 12:11:55 +02:00
  • 2e94bda7fa refac Kadir 2024-08-25 09:50:41 +02:00
  • 6d343e52e7 adding days updated Kadir 2024-08-11 19:36:25 +01:00
  • f98cd02696 adding better exception messages and interrupt free crawling Kadir 2024-05-06 18:54:55 +01:00
  • 874fed3e8f format and committing notebook Kadir 2024-04-05 11:38:55 +01:00
  • 69a5bba65e adding service charge to excel Kadir 2024-04-05 11:38:31 +01:00
  • b5e316d1df making sure the routing is only applied to listings which are not done already Kadir 2024-04-05 11:38:09 +01:00
  • 966b9007a0 adding service charge as the info into the sheet Kadir 2024-04-05 11:37:03 +01:00
  • 5305451fe8 changing detail downloads to prefiltering first. Making the progress bar more accurate and frontloading Kadir 2024-04-01 20:28:37 +02:00
  • b16fee0648 changing to 500 listings per query Kadir 2024-04-01 20:28:15 +02:00
  • 7ce54a01bd removing download of the normal images.. I dont use them Kadir 2024-04-01 15:26:34 +02:00
  • c8c53b8696 Reduced need for routing by limiting to a radius of 7 miles Kadir 2024-04-01 15:21:31 +02:00
  • 6a43a7f485 adding tasks and updating exploration notebook Kadir 2024-04-01 15:09:13 +02:00
  • a7a4f88a39 sort districts Kadir 2024-03-30 19:24:35 +01:00
  • acc67192c9 add counter to 2_dump_details to understand how many new are crawled Kadir 2024-03-30 19:24:03 +01:00
  • 5720e68547 Rewriting 1_dump to enabling crawling of more real estate Kadir 2024-03-30 19:23:19 +01:00
  • 40285245d5 remove debug print statements Kadir 2024-03-30 18:35:32 +01:00
  • ec86b572b3 adding districts and location identifier to search Kadir 2024-03-30 18:31:49 +01:00
  • 4c40462bb8 adding property type Kadir 2024-03-25 20:58:35 +00:00
  • d777558b34 ruff format Kadir 2024-03-25 20:48:48 +00:00