Commit graph

19 commits

Author SHA1 Message Date
Viktor Barzin
a42944a756 wrongmove: round-3 fix sweep — scrape pipeline, BUY tab, filter URL state, render hygiene, map polish
Coordinated fix across 31 bugs found in a parallel QA pass. Findings docs at /tmp/wrongmove-bugs/qa-round-3/qa{1,2,3,4}-*.md.

## Backend / scrape (Fix-1) — 8 bugs

- B1 [P0] Scrape totally broken on prod: pod UID 100 vs NFS dir 1000:1000 mode 775 → PermissionError on every never-seen listing. Switched Dockerfile to explicit `useradd --uid 1000 --gid 1000`; added securityContext + chown initContainer to k8s/{api,celery-beat}-deployment.yaml. Celery worker manifest lives outside this repo — Dockerfile UID change is the load-bearing fix.
- B4 [P1] Celery broker reaped every ~30s by Redis HAProxy idle timeout. Added `broker_transport_options` / `result_backend_transport_options` with `socket_keepalive=True, health_check_interval=25` in celery_app.py + same kwargs on every redis.from_url/Redis call across services/, utils/redis_lock.py, redis_repository.py.
- B5 [P1] dump_listings_task never published terminal FAILURE to the task_progress pub/sub channel — UI polled forever. Wrap body in try/except that publishes FAILURE before re-raising.
- B6 [P1] _process_worker had no per-listing exception handler — one bad listing killed the whole scrape via asyncio.gather. Wrap loop body in try/except Exception (re-raises CancelledError).
- B20 [P2] dump_listings_task gained time_limit=3600, soft_time_limit=3500, acks_late=True.
- B21 [P2] RedisRepository moved off shared db0 (was alongside paperless-ngx) to db3 via REDIS_USER_DB env var; keys prefixed `wrongmove:user:`.
- B32 [P3] redis_lock now uses uuid4() owner token + Lua compare-and-delete.
- B33 [P3] Slack notify in refresh_listings → asyncio.create_task (fire-and-forget).

## Frontend filter system (Fix-2) — 7 bugs

- B2 [P0] BUY tab click triggered "Maximum update depth exceeded" → ErrorBoundary. Replaced the three mutually-triggering useEffects in FilterBar with a single one-way controlled-value flow (URL → parent state → form), guarded by previousListingTypeRef so price-defaults fires once per real transition.
- B3 [P0] Filter values never reached the URL. Wired useFilterParams.setFilterValues into FilterBar/FilterPanel onSubmit + handleRemoveChip + new handleResetAllFilters; fed parsed filterValues into both forms' defaultValues; added URL→form sync via form.reset on browser back/forward.
- B8 [P1] Chip removal now resets form state via new FilterBar onFormReady callback — More badge no longer sticks.
- B12 [P2] Desktop swipe-review FAB added next to header (mobile FAB unchanged).
- B17 [P2] "Reset all" affordance on chip strip.
- B22 [P2] formatPrice precision: 1500 → £1.5k, 2500 → £2.5k (no longer collides with £2k/£3k defaults).
- B30 [P3] last_seen_days input gained min={0}.

## Frontend render hygiene + data integrity (Fix-3) — 8 bugs

- B7 [P1] streamingService bails on first non-NDJSON chunk (HTML response = backend down) and throws StreamParseError so the existing AlertError dialog surfaces a single user-visible error instead of 18× console.error spam.
- B9 [P1] formatDuration widened to (null|undefined|number): returns "—" for non-finite or negative, caps implausibly large values.
- B10 [P1] PropertyCard / PropertyCardCompact / SwipeCard JSX leaves render "—" for null total_price/qm/qmprice (was "£0/0 m²/£0/m²" — looked like free listings).
- B13 [P2] hexgrid worker reduceAverage uses Number.isFinite filter instead of !isNaN (which incorrectly accepted null → 0, biasing per-hex averages low).
- B14 [P2] ListingDetail Overview wraps agency in "Listed by" labelled block so it can't collapse to a bare agency name.
- B15 [P2] Compact POIDistanceBadges iterates all three travel modes with "—" for missing, matching the detail-sheet Travel table.
- B24 [P3] Drawer.Description (sr-only) added to ListingDetailSheet + MobileBottomSheet to silence Radix a11y warning.
- B25 [P3] lastSeenDays clamped to ≥0 so future timestamps don't render as "-7d ago".

## Frontend map / carousel / tasks polish (Fix-4) — 8 bugs

- B11 [P2] HexgridHeatmapClient destroy race: Map.tsx adds .catch() + ref guard so post-destroy promise rejections are silent no-ops. Verified by browser smoke (24 rapid Map↔List toggles → 0 pageErrors).
- B16 [P2] PhotoCarousel + inner CardCarousel gained keyboard nav (Arrow keys).
- B18 [P2] Default map center moved from Czech Republic to London (zoom 10).
- B19+B29 [P2/P3] Mapbox token: no longer hard-coded fallback; reads env-only and shows a clear "Map unavailable — set VITE_MAPBOX_TOKEN" banner when missing.
- B23 [P3] PhotoCarousel suppresses "1/1" counter for single-photo listings; added onError fallback for broken URLs.
- B26 [P3] PhotoCarousel only enables loop when photos.length > 1.
- B27 [P3] TaskIndicator cancel/clear-all buttons gained aria-label + data-testid.
- B28 [P3] useTaskProgress strips terminal-local task IDs from the polling union — no more forever-poll on completed tasks.

## Tests

74 new vitest tests + 18 new pytest tests. Local: tsc clean, 201 vitest tests pass, 633 pytest tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-10 22:27:29 +00:00
Viktor Barzin
1ae00b7cbf
Add multi-layer caching: 24h Redis TTL, stale-while-revalidate, frontend LRU cache
- Increase Redis cache TTL from 30 minutes to 24 hours
- Add stale-while-revalidate: serve stale cache (>4h) immediately while
  repopulating in background with SETNX lock to prevent concurrent rebuilds
- Add in-memory frontend LRU cache (5 entries) so repeat filter visits
  are instant without network requests
- Invalidate frontend cache on listing refresh and task completion
- Add unit tests for get_cache_age, is_cache_stale, acquire_repopulation_lock
2026-02-23 20:09:36 +00:00
Viktor Barzin
34f9e933c0
Add debug CLI decisions, POIs, and tasks subcommands 2026-02-22 15:15:30 +00:00
Viktor Barzin
2196e256f4
Register debug group in main.py with districts subcommand and tests 2026-02-22 15:12:05 +00:00
Viktor Barzin
1536e98c8b
fix: skip frontend config tests when frontend/ dir absent (Docker build) 2026-02-21 15:52:43 +00:00
Viktor Barzin
43f9d210fb
Fix tests to match decision service API and add filtering to non-streaming endpoint
- Update test mocks from _get_disliked_ids to _get_user_id_safe
- Fix decision service test method names (clear_decision -> remove_decision, etc.)
- Fix positional vs keyword arg assertion in set_decision test
- Add decision_filter param to non-streaming listing_geojson endpoint
2026-02-21 15:52:31 +00:00
Viktor Barzin
49280d9679
fix: QA fixes for property decisions feature
- Replace deprecated datetime.utcnow() with datetime.now(UTC) in model
  and repository
- Add listing_type validation to decision_service (RENT/BUY only)
- Fix decision filtering tests failing due to rate limiting by patching
  _match_endpoint
- Add SwipeCard component test suite (11 tests covering rendering,
  interactions, and POI distances)
- Add test for invalid listing_type validation
2026-02-21 14:04:34 +00:00
Viktor Barzin
8452f65d25
feat: filter disliked listings from GeoJSON endpoints
Both /api/listing_geojson and /api/listing_geojson/stream now exclude
disliked listings by default. A decision_filter='everything' param
bypasses filtering. 2 integration tests verify the behavior.
2026-02-21 13:57:43 +00:00
Viktor Barzin
341de89004
feat: add decision API routes with tests
PUT /api/decisions/{listing_id} to set decision,
GET /api/decisions to list all user decisions,
DELETE /api/decisions/{listing_id} to remove a decision.
All 6 API route tests pass.
2026-02-21 13:54:38 +00:00
Viktor Barzin
4877a5fc9f
feat: add decision_service with unit tests
Service layer provides validation, delegation to repository, and
get_disliked_listing_ids for filtering. All 7 unit tests pass.
2026-02-21 13:52:54 +00:00
Viktor Barzin
91c8c884d2
feat: add DecisionRepository with unit tests
Repository provides upsert, get, delete, and disliked-ID-set queries
for listing decisions. All 6 unit tests pass.
2026-02-21 13:52:20 +00:00
Viktor Barzin
f833309297
Refactor backend for cleaner error handling, DRY, and type safety
- Extract rate limiter DRY: consolidate 3 duplicated check/respond paths
  into _check_counter and _enforce_limit helpers, add proper type annotations
- Replace bare Exception raises with FloorplanDownloadError and
  RightmoveApiError; narrow catch clauses to specific exception types;
  fix Step base class to inherit from ABC
- Consolidate MAX_OCR_WORKERS into config/scraper_config.py; extract
  _find_tenure_value helper to deduplicate tenure parsing
- Extract _build_poi_distances_lookup from stream endpoint to reduce nesting
- Fix csv_exporter: optional decisions.json, NaN instead of -1 sentinels,
  guard against division by zero on missing square meters
- Fix notifications.py broken list[Surface]() constructor, database.py
  stale comments and missing type annotation, auth.py type:ignore,
  ui_exporter.py stale TODO
- Fix 3 pre-existing test failures: mock cache layer in streaming tests,
  bypass rate limiter for test isolation, fix cache invalidation test to
  account for two-pattern scan loop
2026-02-10 22:19:24 +00:00
Viktor Barzin
8d22c97320
Add comprehensive test suite: 219 new tests across backend and frontend
Backend (103 tests):
- Unit tests for listing_service, export_service, district_service
- Regression tests for API response contracts and query parameter validation
- Integration tests for API workflows, Redis listing cache, listing processor pipeline, and repository advanced queries
- E2E tests for streaming with filters, batching, caching, and task management

Frontend (116 tests):
- Service tests for apiClient, streamingService, taskService, listingService, healthService
- Hook tests for useTaskProgress (WebSocket + polling)
- Component tests for PropertyCard, FilterPanel, Header, ListView, TaskProgressDrawer, TaskIndicator, StreamingProgressBar, HealthIndicator
- E2E tests for filter-stream-display flow

Infrastructure:
- Add pytest-xdist and test markers (regression, integration, e2e)
- Add conftest fixtures: fake_redis, rent_listing_factory, seeded_repository
- Add vitest + testing-library + MSW for frontend testing
2026-02-10 21:59:45 +00:00
Viktor Barzin
7319f77f1d
Remove JS obfuscator that broke Mapbox GL map rendering
vite-plugin-obfuscator processes ALL output chunks including vendor
libraries, corrupting Mapbox GL's WebGL shader string literals via
base64 encoding and string splitting. This caused the map to render
as a blank screen in production.

Vite's built-in esbuild minification already mangles identifiers and
removes whitespace, providing sufficient code protection.

Adds regression tests to prevent re-introducing obfuscation plugins.
2026-02-08 22:47:01 +00:00
Viktor Barzin
1ace45353a
Add API anti-abuse hardening: disable docs in prod, origin validator, exception handler
- Disable OpenAPI docs/redoc/openapi.json when APP_ENV=production
- Strip uvicorn Server header with --no-server-header in Dockerfile and docker-compose.yml
- Add OriginValidatorMiddleware to reject state-changing requests from disallowed origins
- Add global exception handler to prevent stack trace leakage on unhandled errors
- Add tests for all new security features (OpenAPI, origin validation, exception handler, server header)
2026-02-08 20:06:46 +00:00
Viktor Barzin
492921424e
Add security regression tests for all hardening fixes
- New: test_security_headers.py — verify all headers present, HSTS conditional on HTTPS
- New: test_passkey_error_handling.py — generic vs user-facing error messages
- New: test_poi_validation.py — field length and coordinate range constraints
- Extend test_rate_limiter.py — client IP depth selection, in-memory fallback enforcement
- Extend test_models.py — sqm range validation
- Extend test_task_service.py — IDOR 404, ownership 200, traceback suppression in production
2026-02-08 19:42:53 +00:00
Viktor Barzin
87b5bd8676
Add API rate limiting, metrics guard, and audit middleware
Per-user rate limits via Redis sliding window, IP-restricted /metrics
endpoint, audit logging of all requests, CORS tightening, and export
caps on listing/geojson endpoints.
2026-02-08 00:45:43 +00:00
Viktor Barzin
e5ce8c1201
Fix buy listing support: thread ListingType through processing pipeline
The listing processor was hardcoded to create RentListing objects and
query only the rentlisting table. Buy listings fetched from Rightmove
were stored in the wrong table with missing fields. This threads
ListingType through ListingProcessor and all Step subclasses so the
correct model (RentListing/BuyListing) is created, the correct table
is queried, and buy-specific fields (service_charge, lease_left) are
parsed from the API response and included in GeoJSON streaming output.
2026-02-07 23:34:08 +00:00
Viktor Barzin
eafbc1ac52
Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/
The crawler subdirectory was the only active project. Moving it to the
repo root simplifies paths and removes the unnecessary nesting. The
vqa/ and immoweb/ directories were legacy/unused and have been removed.

Updated .drone.yml, .gitignore, .claude/ docs, and skills to reflect
the new flat structure.
2026-02-07 23:01:20 +00:00