Commit graph

16 commits

Author SHA1 Message Date
Viktor Barzin
c2e08fe46e wrongmove: add "seen" soft-hide decision with price-aware resurfacing
Opening a listing's detail sheet for >3s now passively marks it `seen`
and snapshots its current `total_price`. Seen listings are hidden from
the main list by default but automatically resurface when the price
changes (any direction). Distinct from `disliked` — explicit
like/dislike always overrides the passive seen state.

New "Hidden (N)" toggle on the FilterBar appears whenever at least one
listing is currently being hidden by the seen filter. Toggling it
reveals those rows in-place (without unmarking them) so the user can
review or explicitly clear via the existing Like/Dislike/Clear flow.

## Backend
- Alembic f7a8b9c0d1e2: `ALTER TABLE listingdecision ADD COLUMN
  price_at_decision FLOAT NULL`.
- `models/decision.py`: ListingDecision gains nullable
  `price_at_decision: float | None`.
- `services/decision_service.py`: adds `seen` to VALID_DECISIONS;
  set_decision accepts an optional `price_at_decision`; it's only
  forwarded to the repo for decision='seen' (other types null-out the
  column to keep semantics clean).
- `repositories/decision_repository.py`: upsert_decision now carries
  price_at_decision through the MySQL + SQLite upsert paths.
- `api/decision_routes.py`: SetDecisionRequest + DecisionResponse
  expose the new field.

## Frontend
- `types/index.ts`: DecisionType = 'liked' | 'disliked' | 'seen';
  ListingDecision gains `price_at_decision?: number | null`.
- `services/decisionService.ts`: setDecision sends the price only for
  decision='seen' (and only when it's a finite number).
- `hooks/useDecisions.ts`: rewritten to store `Map<key, DecisionEntry>`
  (decision + price snapshot). New `markSeen(id, price, type)` short-
  circuits on existing liked/disliked. New `getDecisionEntry`,
  `seenCount`.
- `App.tsx`: 3s `setTimeout` dwell timer fires markSeen when the
  detail sheet stays open. Filter logic in `processedListingData`
  hides `seen` rows whose `total_price === price_at_decision`, with
  `showHidden` bypass. Computes `hiddenCount` to drive the toggle.
- `components/FilterBar.tsx`: new conditional "Hidden (N)" / "Showing
  hidden (N)" toggle button (Eye / EyeOff lucide icons), surfaces only
  when hiddenCount > 0.

## Tests
- pytest: 2 new (test_set_seen_carries_price, test_liked_drops_price_
  even_if_supplied) + 1 updated to assert the new 5-arg repo
  signature. 24 passed.
- vitest: 6 new for useDecisions (markSeen liked/disliked skip, price
  snapshot, re-mark, null price, seenCount) + 5 new for decisionService
  payload shape. 221 total passed, tsc clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 11:07:44 +00:00
Viktor Barzin
a42944a756 wrongmove: round-3 fix sweep — scrape pipeline, BUY tab, filter URL state, render hygiene, map polish
Coordinated fix across 31 bugs found in a parallel QA pass. Findings docs at /tmp/wrongmove-bugs/qa-round-3/qa{1,2,3,4}-*.md.

## Backend / scrape (Fix-1) — 8 bugs

- B1 [P0] Scrape totally broken on prod: pod UID 100 vs NFS dir 1000:1000 mode 775 → PermissionError on every never-seen listing. Switched Dockerfile to explicit `useradd --uid 1000 --gid 1000`; added securityContext + chown initContainer to k8s/{api,celery-beat}-deployment.yaml. Celery worker manifest lives outside this repo — Dockerfile UID change is the load-bearing fix.
- B4 [P1] Celery broker reaped every ~30s by Redis HAProxy idle timeout. Added `broker_transport_options` / `result_backend_transport_options` with `socket_keepalive=True, health_check_interval=25` in celery_app.py + same kwargs on every redis.from_url/Redis call across services/, utils/redis_lock.py, redis_repository.py.
- B5 [P1] dump_listings_task never published terminal FAILURE to the task_progress pub/sub channel — UI polled forever. Wrap body in try/except that publishes FAILURE before re-raising.
- B6 [P1] _process_worker had no per-listing exception handler — one bad listing killed the whole scrape via asyncio.gather. Wrap loop body in try/except Exception (re-raises CancelledError).
- B20 [P2] dump_listings_task gained time_limit=3600, soft_time_limit=3500, acks_late=True.
- B21 [P2] RedisRepository moved off shared db0 (was alongside paperless-ngx) to db3 via REDIS_USER_DB env var; keys prefixed `wrongmove:user:`.
- B32 [P3] redis_lock now uses uuid4() owner token + Lua compare-and-delete.
- B33 [P3] Slack notify in refresh_listings → asyncio.create_task (fire-and-forget).

## Frontend filter system (Fix-2) — 7 bugs

- B2 [P0] BUY tab click triggered "Maximum update depth exceeded" → ErrorBoundary. Replaced the three mutually-triggering useEffects in FilterBar with a single one-way controlled-value flow (URL → parent state → form), guarded by previousListingTypeRef so price-defaults fires once per real transition.
- B3 [P0] Filter values never reached the URL. Wired useFilterParams.setFilterValues into FilterBar/FilterPanel onSubmit + handleRemoveChip + new handleResetAllFilters; fed parsed filterValues into both forms' defaultValues; added URL→form sync via form.reset on browser back/forward.
- B8 [P1] Chip removal now resets form state via new FilterBar onFormReady callback — More badge no longer sticks.
- B12 [P2] Desktop swipe-review FAB added next to header (mobile FAB unchanged).
- B17 [P2] "Reset all" affordance on chip strip.
- B22 [P2] formatPrice precision: 1500 → £1.5k, 2500 → £2.5k (no longer collides with £2k/£3k defaults).
- B30 [P3] last_seen_days input gained min={0}.

## Frontend render hygiene + data integrity (Fix-3) — 8 bugs

- B7 [P1] streamingService bails on first non-NDJSON chunk (HTML response = backend down) and throws StreamParseError so the existing AlertError dialog surfaces a single user-visible error instead of 18× console.error spam.
- B9 [P1] formatDuration widened to (null|undefined|number): returns "—" for non-finite or negative, caps implausibly large values.
- B10 [P1] PropertyCard / PropertyCardCompact / SwipeCard JSX leaves render "—" for null total_price/qm/qmprice (was "£0/0 m²/£0/m²" — looked like free listings).
- B13 [P2] hexgrid worker reduceAverage uses Number.isFinite filter instead of !isNaN (which incorrectly accepted null → 0, biasing per-hex averages low).
- B14 [P2] ListingDetail Overview wraps agency in "Listed by" labelled block so it can't collapse to a bare agency name.
- B15 [P2] Compact POIDistanceBadges iterates all three travel modes with "—" for missing, matching the detail-sheet Travel table.
- B24 [P3] Drawer.Description (sr-only) added to ListingDetailSheet + MobileBottomSheet to silence Radix a11y warning.
- B25 [P3] lastSeenDays clamped to ≥0 so future timestamps don't render as "-7d ago".

## Frontend map / carousel / tasks polish (Fix-4) — 8 bugs

- B11 [P2] HexgridHeatmapClient destroy race: Map.tsx adds .catch() + ref guard so post-destroy promise rejections are silent no-ops. Verified by browser smoke (24 rapid Map↔List toggles → 0 pageErrors).
- B16 [P2] PhotoCarousel + inner CardCarousel gained keyboard nav (Arrow keys).
- B18 [P2] Default map center moved from Czech Republic to London (zoom 10).
- B19+B29 [P2/P3] Mapbox token: no longer hard-coded fallback; reads env-only and shows a clear "Map unavailable — set VITE_MAPBOX_TOKEN" banner when missing.
- B23 [P3] PhotoCarousel suppresses "1/1" counter for single-photo listings; added onError fallback for broken URLs.
- B26 [P3] PhotoCarousel only enables loop when photos.length > 1.
- B27 [P3] TaskIndicator cancel/clear-all buttons gained aria-label + data-testid.
- B28 [P3] useTaskProgress strips terminal-local task IDs from the polling union — no more forever-poll on completed tasks.

## Tests

74 new vitest tests + 18 new pytest tests. Local: tsc clean, 201 vitest tests pass, 633 pytest tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-10 22:27:29 +00:00
Viktor Barzin
35f1987ac1
Add navigation & usage metrics for end-user experience visibility
Instrument DB query timing (11 operations across 3 repositories),
streaming lifecycle (TTFB, duration, feature count), cache operation
latency, listing detail step breakdown, and frontend page load /
time-to-first-listing / stream download / detail load metrics.

Adds 16 new OTel instruments, extends the perf ingestion endpoint
with 4 new frontend metrics, and adds ~20 Grafana dashboard panels
across 4 new rows (DB Query Performance, Streaming Performance,
Listing Detail Breakdown, Cache Performance, Frontend Navigation).
2026-02-23 20:28:42 +00:00
Viktor Barzin
1ae00b7cbf
Add multi-layer caching: 24h Redis TTL, stale-while-revalidate, frontend LRU cache
- Increase Redis cache TTL from 30 minutes to 24 hours
- Add stale-while-revalidate: serve stale cache (>4h) immediately while
  repopulating in background with SETNX lock to prevent concurrent rebuilds
- Add in-memory frontend LRU cache (5 entries) so repeat filter visits
  are instant without network requests
- Invalidate frontend cache on listing refresh and task completion
- Add unit tests for get_cache_age, is_cache_stale, acquire_repopulation_lock
2026-02-23 20:09:36 +00:00
Viktor Barzin
d90fa38776
Add frontend performance metrics pipeline to Prometheus
Collect browser-side worker round-trips, computation times, main-thread
operations, and feature counts, batch them client-side, and expose as
Prometheus histograms via a new POST /api/perf endpoint.
2026-02-22 17:30:29 +00:00
Viktor Barzin
8ef6868881
Eliminate frontend POI waterfall for faster initial load
Listing stream fires immediately on auth without waiting for POI
fetch. POI distances are not needed for initial rendering and are
only computed when user selects POI metric or sets travel filters.
This saves ~200-500ms on initial load and keeps the stream on the
cached Redis path.
2026-02-22 13:31:10 +00:00
Viktor Barzin
3885fd52fe
Add bulk POI distances endpoint for decoupled loading
New GET /api/poi/distances/bulk returns all POI distances keyed by
listing ID, allowing the frontend to fetch distances separately
from the listing stream and keep the stream on the cached path.
2026-02-22 13:29:35 +00:00
Viktor Barzin
a2745c1478
Add tappable cards, detail bottom sheet, swipe gestures, and favorites view
- Decision types, services (decisionService, listingDetailService), and index exports
- useDecisions hook with optimistic updates and Map-based state
- useListingDetail hook with session-level caching
- PhotoCarousel component using embla-carousel-react
- ListingDetail component with full property info, like/dislike buttons
- ListingDetailSheet using vaul Drawer (slide-up bottom sheet)
- SwipeablePropertyCard with @use-gesture/react and @react-spring/web
- SwipeReviewMode for mobile full-screen swipe review
- FavoritesView with virtualized liked listings and remove button
- App.tsx integration: decision state, client-side disliked filtering, detail sheet, swipe handlers
- ListView conditionally renders SwipeablePropertyCard when handlers provided
- StatsBar adds 'saved' view mode with heart icon
- Header adds liked count indicator
- New deps: vaul, embla-carousel-react, @use-gesture/react, @react-spring/web
2026-02-21 15:49:15 +00:00
Viktor Barzin
813f048e46
feat: add frontend decision service and types 2026-02-21 13:51:12 +00:00
Viktor Barzin
2d6726dcd7
Show last listing update time next to connection status in header
Add last_updated timestamp to /api/status endpoint by querying
MAX(last_seen) across both listing tables. Display it in the
HealthIndicator as relative time (e.g. "2h ago") with full
date/time in the tooltip on hover.
2026-02-17 19:54:15 +00:00
Viktor Barzin
a1829957c1
Auto-redirect to login on 401 API responses
When a session token expires, API calls return 401 but nothing caught
it — errors were shown as generic dialogs or swallowed. Now both
apiClient and streamingService detect 401 responses and clear auth
state, which causes App.tsx to render the login modal automatically.
2026-02-13 21:16:53 +00:00
Viktor Barzin
8d22c97320
Add comprehensive test suite: 219 new tests across backend and frontend
Backend (103 tests):
- Unit tests for listing_service, export_service, district_service
- Regression tests for API response contracts and query parameter validation
- Integration tests for API workflows, Redis listing cache, listing processor pipeline, and repository advanced queries
- E2E tests for streaming with filters, batching, caching, and task management

Frontend (116 tests):
- Service tests for apiClient, streamingService, taskService, listingService, healthService
- Hook tests for useTaskProgress (WebSocket + polling)
- Component tests for PropertyCard, FilterPanel, Header, ListView, TaskProgressDrawer, TaskIndicator, StreamingProgressBar, HealthIndicator
- E2E tests for filter-stream-display flow

Infrastructure:
- Add pytest-xdist and test markers (regression, integration, e2e)
- Add conftest fixtures: fake_redis, rent_listing_factory, seeded_repository
- Add vitest + testing-library + MSW for frontend testing
2026-02-10 21:59:45 +00:00
Viktor Barzin
73d19e29d5
Fix duplicate listings via staged Redis cache and frontend stream cancellation
Three-pronged fix for duplicate listings appearing in the UI:

1. Backend: Replace direct rpush cache writes with staged population
   (write to temp key, then atomic RENAME to live key). Skip cache
   writes entirely for POI-enriched requests. Clean staging keys on
   invalidation.

2. Frontend: Add AbortController to cancel in-flight streaming requests
   when loadListings is called again, preventing data mixing.

3. Frontend: Deduplicate features by URL during stream accumulation as
   a safety net against any remaining server-side duplicates.
2026-02-09 21:17:30 +00:00
Viktor Barzin
54bdcac14a
Run alembic migrations on startup, fix User model, add POI travel sorting and streaming options 2026-02-08 18:50:13 +00:00
Viktor Barzin
8509a0326f
Add frontend POI management and travel time display
POIManager component in FilterPanel for creating/deleting POIs and
triggering distance calculations. PropertyCard shows travel time badges
(walk/cycle/transit) per POI. Map renders POI locations as red markers.
API client extended with POST body support for POI endpoints.
2026-02-08 13:16:32 +00:00
Viktor Barzin
eafbc1ac52
Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/
The crawler subdirectory was the only active project. Moving it to the
repo root simplifies paths and removes the unnecessary nesting. The
vqa/ and immoweb/ directories were legacy/unused and have been removed.

Updated .drone.yml, .gitignore, .claude/ docs, and skills to reflect
the new flat structure.
2026-02-07 23:01:20 +00:00