viktor/wrongmove
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Rewriting 1_dump to enabling crawling of more real estate

Browse source
This commit is contained in:
Kadir 2024-03-30 19:23:19 +01:00
parent 40285245d5
commit 5720e68547
2 changed files with 63 additions and 30 deletions
Download patch file Download diff file Expand all files Collapse all files

52
crawler/TASKS
View file

@ -1,19 +1,35 @@
- [ ] Partition query further as each query can listing query can only grab a 1000 entries at most. If the query is too broad, it will fail afterwards.
- [ ] Update capability of the database
- [ ] Check if the entry already exists in the database
- [ ] Crawl single entry
- [ ] Download pictures and map to database entry
# Distance measuring
- [ ] route api to find shortest path apartments
- [ ] Switch to distance matrix api. Evaluate if its the right one.
# sqm measure
- [ ] OCR to detect the area of the apartment
- District: City of London, totalAvailableResults: 60
- District: Greenwich, totalAvailableResults: 1371
- District: Hillingdon, totalAvailableResults: 1026
- District: Ealing, totalAvailableResults: 1736
- District: Richmond upon Thames, totalAvailableResults: 819
- District: Sutton, totalAvailableResults: 664
- District: Wandsworth, totalAvailableResults: 1824
- District: Camden, totalAvailableResults: 801
- District: Enfield, totalAvailableResults: 1056
- District: Croydon, totalAvailableResults: 1865
- District: Hackney, totalAvailableResults: 840
- District: Kingston upon Thames, totalAvailableResults: 685
- District: Kensington and Chelsea, totalAvailableResults: 658
- District: Bromley, totalAvailableResults: 1341
- District: Brent, totalAvailableResults: 1332
- District: Waltham Forest, totalAvailableResults: 763
- District: Southwark, totalAvailableResults: 1460
- District: Harrow, totalAvailableResults: 948
- District: Lewisham, totalAvailableResults: 1192
- District: Barnet, totalAvailableResults: 1683
- District: Islington, totalAvailableResults: 766
- District: Haringey, totalAvailableResults: 795
- District: Lambeth, totalAvailableResults: 1626
- District: Westminster, totalAvailableResults: 1130
- District: Tower Hamlets, totalAvailableResults: 2213
- District: Havering, totalAvailableResults: 863
- District: Barking and Dagenham, totalAvailableResults: 485
- District: Hammersmith and Fulham, totalAvailableResults: 1038
- District: Bexley, totalAvailableResults: 803
- District: Redbridge, totalAvailableResults: 720
- District: Newham, totalAvailableResults: 1306
- District: Merton, totalAvailableResults: 873
- District: Hounslow, totalAvailableResults: 1096