viktor/wrongmove
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor runall script to share filter options when dumping listings and when exporting to immoweb

Browse source
This commit is contained in:
Viktor Barzin 2025-06-01 18:10:09 +00:00
parent 2e3e900ec3
commit ab5c4ce0a6
No known key found for this signature in database
GPG key ID: 4056458DBDBF8863
1 changed files with 4 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

7
crawler/runall.sh
View file

@ -3,11 +3,12 @@
set -euxo pipefail
DATA_DIR="data/rs"
LISTING_FILTER_OPTIONS="--min-price 2000 --max-price 4000 --min-bedrooms 2 --max-bedrooms 4 -t rent --available-from $(date +%Y-%m-%d) --last-seen-days 7"
python main.py --data-dir $DATA_DIR dump-listings --min-price 2000 --max-price 4000 --min-bedrooms 2 --max-bedrooms 4 -t rent
python main.py --data-dir $DATA_DIR dump-listings $LISTING_FILTER_OPTIONS
python main.py --data-dir $DATA_DIR dump-details
python main.py --data-dir $DATA_DIR dump-images
python main.py --data-dir $DATA_DIR detect-floorplan
#python main.py --data-dir $DATA_DIR routing --destination-address 'Meta Brock Street' -m transit # NOTE: THIS CONSUMES API CALLS; USE CAREFULLY; add -l to limit number of entries
# python main.py --data-dir $DATA_DIR export-csv -O data/listings.csv # not used anymore - immoweb exporter is used to visualize data
python main.py --data-dir $DATA_DIR export-immoweb -O ../immoweb/data/london_geojs.js
# python main.py --data-dir $DATA_DIR export-csv -O data/listings.csv # not used anymore - immoweb exporter is used to visualize data instead of csv
python main.py --data-dir $DATA_DIR export-immoweb -O ../immoweb/data/london_geojs.js $LISTING_FILTER_OPTIONS