CI test-unit failed on pipeline #49 because the three TestSetupPeriodicTasks
cases asserted exact call counts on `sender.add_periodic_task` and the
new unconditional `daily-market-aggregator` registration bumped each by
one. Fix:
- `tasks/listing_tasks.py`: lifted the market-aggregator registration
out of the SchedulesConfig try-block — it's now independent of the
user's SCRAPE_SCHEDULES (a malformed scrape config no longer takes
the aggregator down with it).
- `tests/unit/test_listing_tasks.py`: updated the four cases to account
for the +1 unconditional aggregator call. `test_handles_config_error_
gracefully` now asserts the aggregator still registers when
SchedulesConfig.from_env raises (regression coverage for the
independence guarantee).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two surfaces wired up so the user can "get a vibe of the market":
**Per-listing** — each PropertyCard now shows a small pill next to the
price when the listing's total_price moved >=1% over a 14-day lookback
(e.g. "↓ £200 (-4%) in 14d"). Drops render green, rises render red.
Computed from `price_history_json` by the daily aggregator and
denormalised onto the listing row so the streaming endpoint just
passes it through.
**Macro** — new always-visible inline strip above the chip strip
showing today's median total price, median £/m², and listing count
for the current filter's bedroom band, each with a 30-day % delta:
"Rent · 1-2 bed · 30d: Median £2,500 ↓ -4% · £/m² £50 ↓ -2% · Listings 4,200 ↑ +5%".
Both data sources are populated daily at 04:00 UTC by a new Celery
beat task that fires 1h after the 03:00 RENT scrape and feeds two
sinks: a per-listing update pass and an upsert to a new
`dailylistingaggregate` table keyed on
(snapshot_date, listing_type, min_bedrooms, max_bedrooms).
## Backend
- `models/listing.py`: Listing parent gains `price_14d_ago` + `price_
change_pct_14d` nullable floats (inherited by RentListing/BuyListing).
New `DailyListingAggregate` table model with unique constraint on
(date, type, min_bed, max_bed).
- Alembic `a8b9c0d1e2f3`: adds the two columns to both listing tables
and creates the aggregate table + date index.
- `services/market_aggregator.py` (new): `compute_trend_for_listing`,
`update_per_listing_trend` (batched, idempotent), `_stats` (median
+ mean filtered to positive finite values), `compute_aggregate_
snapshot` (dialect-aware MySQL / SQLite upsert), `fetch_trend_
series` (range query for the API).
- `tasks/market_tasks.py` (new): `compute_daily_market_aggregates_task`
Celery task wrapping both stages.
- `tasks/listing_tasks.py:setup_periodic_tasks`: registers the daily
task at 04:00 UTC alongside the existing scrape schedules.
- `celery_app.py`: includes the new tasks module.
- `api/app.py`: new `GET /api/market_trend?listing_type=&min_bedrooms=&
max_bedrooms=&days=` endpoint returning the daily series.
- `ui_exporter.py`: GeoJSON feature properties now carry
`price_14d_ago` and `price_change_pct_14d` so the frontend can
render the badge without an extra round-trip.
## Frontend
- `types/index.ts`: new `MarketTrendPoint`; `PropertyProperties` gains
the two optional trend fields.
- `components/PropertyCard.tsx`: derived `trendBadge` (>=1% threshold,
null-safe) rendered as a small pill on both card variants.
- `hooks/useMarketTrend.ts` (new): fetches the trend series, derives
current-vs-oldest deltas per metric (% change rounded to 1dp).
- `components/MarketTrendStrip.tsx` (new): compact inline strip with
three metric cells. Hidden when the aggregator hasn't produced any
rows yet (graceful start during the first week post-launch).
- `App.tsx`: renders the strip above the chip strip whenever the
active queryParameters are known.
## Tests
- pytest: 10 new (trend math edge cases including null history,
malformed JSON, only-recent entries, drops, rises, zero current
price; _stats empty / nonpositive filtering; upsert idempotency on
an in-memory SQLite seed). 34 decision + aggregator tests pass.
- vitest: 8 new (useMarketTrend fetch URL, two-point delta,
single-point null delta, empty series; PropertyCard trend badge
arrow direction + sign for drops/rises, noise threshold, null
guard). 229 tests pass total, tsc clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Opening a listing's detail sheet for >3s now passively marks it `seen`
and snapshots its current `total_price`. Seen listings are hidden from
the main list by default but automatically resurface when the price
changes (any direction). Distinct from `disliked` — explicit
like/dislike always overrides the passive seen state.
New "Hidden (N)" toggle on the FilterBar appears whenever at least one
listing is currently being hidden by the seen filter. Toggling it
reveals those rows in-place (without unmarking them) so the user can
review or explicitly clear via the existing Like/Dislike/Clear flow.
## Backend
- Alembic f7a8b9c0d1e2: `ALTER TABLE listingdecision ADD COLUMN
price_at_decision FLOAT NULL`.
- `models/decision.py`: ListingDecision gains nullable
`price_at_decision: float | None`.
- `services/decision_service.py`: adds `seen` to VALID_DECISIONS;
set_decision accepts an optional `price_at_decision`; it's only
forwarded to the repo for decision='seen' (other types null-out the
column to keep semantics clean).
- `repositories/decision_repository.py`: upsert_decision now carries
price_at_decision through the MySQL + SQLite upsert paths.
- `api/decision_routes.py`: SetDecisionRequest + DecisionResponse
expose the new field.
## Frontend
- `types/index.ts`: DecisionType = 'liked' | 'disliked' | 'seen';
ListingDecision gains `price_at_decision?: number | null`.
- `services/decisionService.ts`: setDecision sends the price only for
decision='seen' (and only when it's a finite number).
- `hooks/useDecisions.ts`: rewritten to store `Map<key, DecisionEntry>`
(decision + price snapshot). New `markSeen(id, price, type)` short-
circuits on existing liked/disliked. New `getDecisionEntry`,
`seenCount`.
- `App.tsx`: 3s `setTimeout` dwell timer fires markSeen when the
detail sheet stays open. Filter logic in `processedListingData`
hides `seen` rows whose `total_price === price_at_decision`, with
`showHidden` bypass. Computes `hiddenCount` to drive the toggle.
- `components/FilterBar.tsx`: new conditional "Hidden (N)" / "Showing
hidden (N)" toggle button (Eye / EyeOff lucide icons), surfaces only
when hiddenCount > 0.
## Tests
- pytest: 2 new (test_set_seen_carries_price, test_liked_drops_price_
even_if_supplied) + 1 updated to assert the new 5-arg repo
signature. 24 passed.
- vitest: 6 new for useDecisions (markSeen liked/disliked skip, price
snapshot, re-mark, null price, seenCount) + 5 new for decisionService
payload shape. 221 total passed, tsc clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Increase Redis cache TTL from 30 minutes to 24 hours
- Add stale-while-revalidate: serve stale cache (>4h) immediately while
repopulating in background with SETNX lock to prevent concurrent rebuilds
- Add in-memory frontend LRU cache (5 entries) so repeat filter visits
are instant without network requests
- Invalidate frontend cache on listing refresh and task completion
- Add unit tests for get_cache_age, is_cache_stale, acquire_repopulation_lock
- Update test mocks from _get_disliked_ids to _get_user_id_safe
- Fix decision service test method names (clear_decision -> remove_decision, etc.)
- Fix positional vs keyword arg assertion in set_decision test
- Add decision_filter param to non-streaming listing_geojson endpoint
- Replace deprecated datetime.utcnow() with datetime.now(UTC) in model
and repository
- Add listing_type validation to decision_service (RENT/BUY only)
- Fix decision filtering tests failing due to rate limiting by patching
_match_endpoint
- Add SwipeCard component test suite (11 tests covering rendering,
interactions, and POI distances)
- Add test for invalid listing_type validation
Both /api/listing_geojson and /api/listing_geojson/stream now exclude
disliked listings by default. A decision_filter='everything' param
bypasses filtering. 2 integration tests verify the behavior.
PUT /api/decisions/{listing_id} to set decision,
GET /api/decisions to list all user decisions,
DELETE /api/decisions/{listing_id} to remove a decision.
All 6 API route tests pass.
- Extract rate limiter DRY: consolidate 3 duplicated check/respond paths
into _check_counter and _enforce_limit helpers, add proper type annotations
- Replace bare Exception raises with FloorplanDownloadError and
RightmoveApiError; narrow catch clauses to specific exception types;
fix Step base class to inherit from ABC
- Consolidate MAX_OCR_WORKERS into config/scraper_config.py; extract
_find_tenure_value helper to deduplicate tenure parsing
- Extract _build_poi_distances_lookup from stream endpoint to reduce nesting
- Fix csv_exporter: optional decisions.json, NaN instead of -1 sentinels,
guard against division by zero on missing square meters
- Fix notifications.py broken list[Surface]() constructor, database.py
stale comments and missing type annotation, auth.py type:ignore,
ui_exporter.py stale TODO
- Fix 3 pre-existing test failures: mock cache layer in streaming tests,
bypass rate limiter for test isolation, fix cache invalidation test to
account for two-pattern scan loop
vite-plugin-obfuscator processes ALL output chunks including vendor
libraries, corrupting Mapbox GL's WebGL shader string literals via
base64 encoding and string splitting. This caused the map to render
as a blank screen in production.
Vite's built-in esbuild minification already mangles identifiers and
removes whitespace, providing sufficient code protection.
Adds regression tests to prevent re-introducing obfuscation plugins.
- Disable OpenAPI docs/redoc/openapi.json when APP_ENV=production
- Strip uvicorn Server header with --no-server-header in Dockerfile and docker-compose.yml
- Add OriginValidatorMiddleware to reject state-changing requests from disallowed origins
- Add global exception handler to prevent stack trace leakage on unhandled errors
- Add tests for all new security features (OpenAPI, origin validation, exception handler, server header)
Per-user rate limits via Redis sliding window, IP-restricted /metrics
endpoint, audit logging of all requests, CORS tightening, and export
caps on listing/geojson endpoints.
The listing processor was hardcoded to create RentListing objects and
query only the rentlisting table. Buy listings fetched from Rightmove
were stored in the wrong table with missing fields. This threads
ListingType through ListingProcessor and all Step subclasses so the
correct model (RentListing/BuyListing) is created, the correct table
is queried, and buy-specific fields (service_charge, lease_left) are
parsed from the API response and included in GeoJSON streaming output.
The crawler subdirectory was the only active project. Moving it to the
repo root simplifies paths and removes the unnecessary nesting. The
vqa/ and immoweb/ directories were legacy/unused and have been removed.
Updated .drone.yml, .gitignore, .claude/ docs, and skills to reflect
the new flat structure.
