CI test-unit failed on pipeline #49 because the three TestSetupPeriodicTasks
cases asserted exact call counts on `sender.add_periodic_task` and the
new unconditional `daily-market-aggregator` registration bumped each by
one. Fix:
- `tasks/listing_tasks.py`: lifted the market-aggregator registration
out of the SchedulesConfig try-block — it's now independent of the
user's SCRAPE_SCHEDULES (a malformed scrape config no longer takes
the aggregator down with it).
- `tests/unit/test_listing_tasks.py`: updated the four cases to account
for the +1 unconditional aggregator call. `test_handles_config_error_
gracefully` now asserts the aggregator still registers when
SchedulesConfig.from_env raises (regression coverage for the
independence guarantee).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two surfaces wired up so the user can "get a vibe of the market":
**Per-listing** — each PropertyCard now shows a small pill next to the
price when the listing's total_price moved >=1% over a 14-day lookback
(e.g. "↓ £200 (-4%) in 14d"). Drops render green, rises render red.
Computed from `price_history_json` by the daily aggregator and
denormalised onto the listing row so the streaming endpoint just
passes it through.
**Macro** — new always-visible inline strip above the chip strip
showing today's median total price, median £/m², and listing count
for the current filter's bedroom band, each with a 30-day % delta:
"Rent · 1-2 bed · 30d: Median £2,500 ↓ -4% · £/m² £50 ↓ -2% · Listings 4,200 ↑ +5%".
Both data sources are populated daily at 04:00 UTC by a new Celery
beat task that fires 1h after the 03:00 RENT scrape and feeds two
sinks: a per-listing update pass and an upsert to a new
`dailylistingaggregate` table keyed on
(snapshot_date, listing_type, min_bedrooms, max_bedrooms).
## Backend
- `models/listing.py`: Listing parent gains `price_14d_ago` + `price_
change_pct_14d` nullable floats (inherited by RentListing/BuyListing).
New `DailyListingAggregate` table model with unique constraint on
(date, type, min_bed, max_bed).
- Alembic `a8b9c0d1e2f3`: adds the two columns to both listing tables
and creates the aggregate table + date index.
- `services/market_aggregator.py` (new): `compute_trend_for_listing`,
`update_per_listing_trend` (batched, idempotent), `_stats` (median
+ mean filtered to positive finite values), `compute_aggregate_
snapshot` (dialect-aware MySQL / SQLite upsert), `fetch_trend_
series` (range query for the API).
- `tasks/market_tasks.py` (new): `compute_daily_market_aggregates_task`
Celery task wrapping both stages.
- `tasks/listing_tasks.py:setup_periodic_tasks`: registers the daily
task at 04:00 UTC alongside the existing scrape schedules.
- `celery_app.py`: includes the new tasks module.
- `api/app.py`: new `GET /api/market_trend?listing_type=&min_bedrooms=&
max_bedrooms=&days=` endpoint returning the daily series.
- `ui_exporter.py`: GeoJSON feature properties now carry
`price_14d_ago` and `price_change_pct_14d` so the frontend can
render the badge without an extra round-trip.
## Frontend
- `types/index.ts`: new `MarketTrendPoint`; `PropertyProperties` gains
the two optional trend fields.
- `components/PropertyCard.tsx`: derived `trendBadge` (>=1% threshold,
null-safe) rendered as a small pill on both card variants.
- `hooks/useMarketTrend.ts` (new): fetches the trend series, derives
current-vs-oldest deltas per metric (% change rounded to 1dp).
- `components/MarketTrendStrip.tsx` (new): compact inline strip with
three metric cells. Hidden when the aggregator hasn't produced any
rows yet (graceful start during the first week post-launch).
- `App.tsx`: renders the strip above the chip strip whenever the
active queryParameters are known.
## Tests
- pytest: 10 new (trend math edge cases including null history,
malformed JSON, only-recent entries, drops, rises, zero current
price; _stats empty / nonpositive filtering; upsert idempotency on
an in-memory SQLite seed). 34 decision + aggregator tests pass.
- vitest: 8 new (useMarketTrend fetch URL, two-point delta,
single-point null delta, empty series; PropertyCard trend badge
arrow direction + sign for drops/rises, noise threshold, null
guard). 229 tests pass total, tsc clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Opening a listing's detail sheet for >3s now passively marks it `seen`
and snapshots its current `total_price`. Seen listings are hidden from
the main list by default but automatically resurface when the price
changes (any direction). Distinct from `disliked` — explicit
like/dislike always overrides the passive seen state.
New "Hidden (N)" toggle on the FilterBar appears whenever at least one
listing is currently being hidden by the seen filter. Toggling it
reveals those rows in-place (without unmarking them) so the user can
review or explicitly clear via the existing Like/Dislike/Clear flow.
## Backend
- Alembic f7a8b9c0d1e2: `ALTER TABLE listingdecision ADD COLUMN
price_at_decision FLOAT NULL`.
- `models/decision.py`: ListingDecision gains nullable
`price_at_decision: float | None`.
- `services/decision_service.py`: adds `seen` to VALID_DECISIONS;
set_decision accepts an optional `price_at_decision`; it's only
forwarded to the repo for decision='seen' (other types null-out the
column to keep semantics clean).
- `repositories/decision_repository.py`: upsert_decision now carries
price_at_decision through the MySQL + SQLite upsert paths.
- `api/decision_routes.py`: SetDecisionRequest + DecisionResponse
expose the new field.
## Frontend
- `types/index.ts`: DecisionType = 'liked' | 'disliked' | 'seen';
ListingDecision gains `price_at_decision?: number | null`.
- `services/decisionService.ts`: setDecision sends the price only for
decision='seen' (and only when it's a finite number).
- `hooks/useDecisions.ts`: rewritten to store `Map<key, DecisionEntry>`
(decision + price snapshot). New `markSeen(id, price, type)` short-
circuits on existing liked/disliked. New `getDecisionEntry`,
`seenCount`.
- `App.tsx`: 3s `setTimeout` dwell timer fires markSeen when the
detail sheet stays open. Filter logic in `processedListingData`
hides `seen` rows whose `total_price === price_at_decision`, with
`showHidden` bypass. Computes `hiddenCount` to drive the toggle.
- `components/FilterBar.tsx`: new conditional "Hidden (N)" / "Showing
hidden (N)" toggle button (Eye / EyeOff lucide icons), surfaces only
when hiddenCount > 0.
## Tests
- pytest: 2 new (test_set_seen_carries_price, test_liked_drops_price_
even_if_supplied) + 1 updated to assert the new 5-arg repo
signature. 24 passed.
- vitest: 6 new for useDecisions (markSeen liked/disliked skip, price
snapshot, re-mark, null price, seenCount) + 5 new for decisionService
payload shape. 221 total passed, tsc clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Increase Redis cache TTL from 30 minutes to 24 hours
- Add stale-while-revalidate: serve stale cache (>4h) immediately while
repopulating in background with SETNX lock to prevent concurrent rebuilds
- Add in-memory frontend LRU cache (5 entries) so repeat filter visits
are instant without network requests
- Invalidate frontend cache on listing refresh and task completion
- Add unit tests for get_cache_age, is_cache_stale, acquire_repopulation_lock
Decision IDs are now loaded inside the streaming generators after
the metadata message is yielded, eliminating a blocking DB query
from the pre-stream path (~100-200ms improvement to TTFB).
Subsequent batches use the normal batch_size (default 50). This
reduces server-side time-to-first-property by ~10x since only 5
features need to be serialized before the first yield.
- Update test mocks from _get_disliked_ids to _get_user_id_safe
- Fix decision service test method names (clear_decision -> remove_decision, etc.)
- Fix positional vs keyword arg assertion in set_decision test
- Add decision_filter param to non-streaming listing_geojson endpoint
Tests were failing because the new decision filtering code in api/app.py
tries to query the database for disliked IDs, but test fixtures that mock
the ListingRepository didn't also mock _get_disliked_ids. Additionally,
rate limiter was not bypassed in TestListingGeoJsonEndpoint client fixture,
causing 429s when tests run in sequence.
- Replace deprecated datetime.utcnow() with datetime.now(UTC) in model
and repository
- Add listing_type validation to decision_service (RENT/BUY only)
- Fix decision filtering tests failing due to rate limiting by patching
_match_endpoint
- Add SwipeCard component test suite (11 tests covering rendering,
interactions, and POI distances)
- Add test for invalid listing_type validation
Both /api/listing_geojson and /api/listing_geojson/stream now exclude
disliked listings by default. A decision_filter='everything' param
bypasses filtering. 2 integration tests verify the behavior.
PUT /api/decisions/{listing_id} to set decision,
GET /api/decisions to list all user decisions,
DELETE /api/decisions/{listing_id} to remove a decision.
All 6 API route tests pass.
Add last_updated timestamp to /api/status endpoint by querying
MAX(last_seen) across both listing tables. Display it in the
HealthIndicator as relative time (e.g. "2h ago") with full
date/time in the tooltip on hover.
- WebSocket: verify task ownership before allowing subscribe (security)
- POI routes: replace assert with HTTPException for production safety
- cancel_task: return HTTP 404 instead of 200 for missing tasks
- routing_config: add descriptive ValueError for invalid env vars
- POIManager: show error feedback instead of silently swallowing failures
- VisualizationCard: reset POI/travel mode state on metric switch
- Map: clean up heatmap layers/sources on unmount to prevent memory leak
- Update test to expect 404 from cancel_task ownership check
The fixture accepted in_memory_engine but never actually patched
database.engine or api.app.engine, causing tests to hit the real
SQLite path which fails in CI where data/ doesn't exist.
- Extract rate limiter DRY: consolidate 3 duplicated check/respond paths
into _check_counter and _enforce_limit helpers, add proper type annotations
- Replace bare Exception raises with FloorplanDownloadError and
RightmoveApiError; narrow catch clauses to specific exception types;
fix Step base class to inherit from ABC
- Consolidate MAX_OCR_WORKERS into config/scraper_config.py; extract
_find_tenure_value helper to deduplicate tenure parsing
- Extract _build_poi_distances_lookup from stream endpoint to reduce nesting
- Fix csv_exporter: optional decisions.json, NaN instead of -1 sentinels,
guard against division by zero on missing square meters
- Fix notifications.py broken list[Surface]() constructor, database.py
stale comments and missing type annotation, auth.py type:ignore,
ui_exporter.py stale TODO
- Fix 3 pre-existing test failures: mock cache layer in streaming tests,
bypass rate limiter for test isolation, fix cache invalidation test to
account for two-pattern scan loop
Three-pronged fix for duplicate listings appearing in the UI:
1. Backend: Replace direct rpush cache writes with staged population
(write to temp key, then atomic RENAME to live key). Skip cache
writes entirely for POI-enriched requests. Clean staging keys on
invalidation.
2. Frontend: Add AbortController to cancel in-flight streaming requests
when loadListings is called again, preventing data mixing.
3. Frontend: Deduplicate features by URL during stream accumulation as
a safety net against any remaining server-side duplicates.
vite-plugin-obfuscator processes ALL output chunks including vendor
libraries, corrupting Mapbox GL's WebGL shader string literals via
base64 encoding and string splitting. This caused the map to render
as a blank screen in production.
Vite's built-in esbuild minification already mangles identifiers and
removes whitespace, providing sufficient code protection.
Adds regression tests to prevent re-introducing obfuscation plugins.
- Disable OpenAPI docs/redoc/openapi.json when APP_ENV=production
- Strip uvicorn Server header with --no-server-header in Dockerfile and docker-compose.yml
- Add OriginValidatorMiddleware to reject state-changing requests from disallowed origins
- Add global exception handler to prevent stack trace leakage on unhandled errors
- Add tests for all new security features (OpenAPI, origin validation, exception handler, server header)
Per-user rate limits via Redis sliding window, IP-restricted /metrics
endpoint, audit logging of all requests, CORS tightening, and export
caps on listing/geojson endpoints.
The listing processor was hardcoded to create RentListing objects and
query only the rentlisting table. Buy listings fetched from Rightmove
were stored in the wrong table with missing fields. This threads
ListingType through ListingProcessor and all Step subclasses so the
correct model (RentListing/BuyListing) is created, the correct table
is queried, and buy-specific fields (service_charge, lease_left) are
parsed from the API response and included in GeoJSON streaming output.
The crawler subdirectory was the only active project. Moving it to the
repo root simplifies paths and removes the unnecessary nesting. The
vqa/ and immoweb/ directories were legacy/unused and have been removed.
Updated .drone.yml, .gitignore, .claude/ docs, and skills to reflect
the new flat structure.
