npm ci can also OOM during dependency installation. Move the heap
limit before npm ci so it applies to all Node processes. Bump Drone
pod limits to 4Gi (requests 2Gi) to cover Docker-in-Docker overhead.
- Replace `npm run build` (tsc -b && vite build) with `npx vite build`
in Dockerfile since Vite transpiles via SWC independently of tsc.
Type-checking is already done in the test step.
- Set Node heap to 1024MB (was 384MB which OOMed even for Vite)
- Bump Drone pod memory: requests 1.5Gi, limits 3Gi to cover
plugins/docker overhead
- Set NODE_OPTIONS=--max-old-space-size=512 in Dockerfile to cap tsc
heap usage within constrained CI pods
- Add resource requests (1Gi) and limits (2Gi) to frontend Docker
build steps in Drone pipeline
Test files use Vitest globals (vi, describe, it, expect) which aren't
available to tsc during production builds. Exclude __tests__ dirs and
*.test.* / *.spec.* files from tsconfig.app.json so tsc -b succeeds.
Add a 'test' stage to Dockerfile that extends runtime-base with the
venv and test dependencies (pytest, fakeredis, etc.) pre-installed.
Drone CI now builds and caches this image as :test, then uses it
directly for running tests — eliminating apt-get and pip install
on every build.
The fixture accepted in_memory_engine but never actually patched
database.engine or api.app.engine, causing tests to hit the real
SQLite path which fails in CI where data/ doesn't exist.
- Extract rate limiter DRY: consolidate 3 duplicated check/respond paths
into _check_counter and _enforce_limit helpers, add proper type annotations
- Replace bare Exception raises with FloorplanDownloadError and
RightmoveApiError; narrow catch clauses to specific exception types;
fix Step base class to inherit from ABC
- Consolidate MAX_OCR_WORKERS into config/scraper_config.py; extract
_find_tenure_value helper to deduplicate tenure parsing
- Extract _build_poi_distances_lookup from stream endpoint to reduce nesting
- Fix csv_exporter: optional decisions.json, NaN instead of -1 sentinels,
guard against division by zero on missing square meters
- Fix notifications.py broken list[Surface]() constructor, database.py
stale comments and missing type annotation, auth.py type:ignore,
ui_exporter.py stale TODO
- Fix 3 pre-existing test failures: mock cache layer in streaming tests,
bypass rate limiter for test isolation, fix cache invalidation test to
account for two-pattern scan loop
Frontend pipeline: runs vitest via node:24-alpine before building.
API pipeline: installs deps and runs pytest via python:3.13-slim before building.
Both steps fail-fast (-x) so broken tests block deployment.
Drone expands ${VAR} as its own variables before the shell runs, so
${BASE_API} and ${DEPLOY} were replaced with empty strings. Use $VAR
(no braces) so the shell handles them instead. Also add fallback for
empty jq output to prevent "sh: out of range" errors.
Replace timer-based _monitor_progress (1s sleep loop) with a
ProgressReporter class that publishes on actual state changes,
throttled to at most 1 publish per 250ms. A background flush
every 2s keeps ETA/elapsed current during quiet periods.
Switch WebSocket forwarder from get_message() polling (1s timeout)
to async pubsub.listen() for instant Redis-to-WebSocket delivery.
Combined latency improvement: ~1.5s average → ~250ms.
Replace WebSocket-only useTaskWebSocket with useTaskProgress that
provides a unified task state interface. TaskIndicator no longer
manages its own polling or auth — it receives task state from the
parent via props. Rename wsTasks prop to tasks throughout.
With 8+ active tasks, polling every 5s generates ~96 task_status
requests/min, exceeding the 60/60s rate limit. Two fixes:
- Adaptive polling: 30s when WebSocket is connected (safety net),
5s only when WebSocket is down (primary source)
- Raise task_status rate limit to 200/60s and tasks_for_user to
60/60s to handle burst scenarios (page reloads, WS reconnects)
Three interconnected bugs prevented progress updates from reaching the frontend:
1. _forward_pubsub could exit silently while _handle_client_messages kept
the WebSocket alive (responding to pings), so the client never detected
the broken forwarding path. Replace asyncio.gather with asyncio.wait
(FIRST_COMPLETED) so both coroutines are cancelled together.
2. Polling was stopped on WS connect with no fallback if forwarding broke.
Now polling runs always alongside WebSocket as a safety net.
3. Redis publish failures in task_progress_publisher were logged at DEBUG
and the broken client was reused forever. Log at WARNING and reset the
client so the next call reconnects.
Polling was disabled when wsConnected was true, but if the WS connected
while workers hadn't been redeployed (no pub/sub messages flowing), the
UI received no updates at all. Polling now always runs at 5s as the
baseline. WebSocket provides faster real-time updates on top when
available — the two coexist, last writer wins.
Replace 5s HTTP polling with WebSocket-based real-time updates for task
progress. Celery workers publish progress to Redis pub/sub channels;
a FastAPI WebSocket endpoint subscribes and forwards to the browser.
Polling is kept as a 30s fallback when WebSocket is unavailable.
The task progress drawer now supports multiple concurrent jobs with a
tab bar for switching between scrape and POI distance tasks.
Backend:
- Add services/task_progress_publisher.py (Redis pub/sub bridge)
- Add api/ws_routes.py (WebSocket endpoint with JWT auth)
- Publish progress from listing_tasks and poi_tasks
- Publish REVOKED via pub/sub on cancel/clear to fix stuck UI
Frontend:
- Add useTaskWebSocket hook with reconnection and keepalive
- Add TaskState and WS message types
- TaskIndicator: WS-driven updates with polling fallback
- TaskProgressDrawer: multi-job tabs, POI phase timeline
- Guard against WS overwriting local cancel state
Three-pronged fix for duplicate listings appearing in the UI:
1. Backend: Replace direct rpush cache writes with staged population
(write to temp key, then atomic RENAME to live key). Skip cache
writes entirely for POI-enriched requests. Clean staging keys on
invalidation.
2. Frontend: Add AbortController to cancel in-flight streaming requests
when loadListings is called again, preventing data mixing.
3. Frontend: Deduplicate features by URL during stream accumulation as
a safety net against any remaining server-side duplicates.
index.html is served with Cache-Control: no-cache so the browser always
fetches the latest version with updated asset hashes. Hashed assets under
/assets/ are cached indefinitely since their filenames change on rebuild.
This prevents browsers from serving old cached JS bundles (including the
broken obfuscated build) after a new deployment.
vite-plugin-obfuscator processes ALL output chunks including vendor
libraries, corrupting Mapbox GL's WebGL shader string literals via
base64 encoding and string splitting. This caused the map to render
as a blank screen in production.
Vite's built-in esbuild minification already mangles identifiers and
removes whitespace, providing sufficient code protection.
Adds regression tests to prevent re-introducing obfuscation plugins.
Both frontend and API pipelines now wait for K8s deployments to fully
roll out before marking the build as successful. Polls the K8s API
every 5s for up to 300s, checking observedGeneration, updatedReplicas,
and readyReplicas to confirm the new image is live in production.
- Fix silent log loss: replace hardcoded "uvicorn.error" logger with __name__
in osrm_client, otp_client, poi_distance_calculator, and poi_tasks (uvicorn
logger has no handlers in Celery worker, so all errors were silently dropped)
- Add Celery retry: autoretry_for=(Exception,), max_retries=3, retry_backoff
- Add top-level exception handling in task with full traceback logging
- Fix upsert_distances: replace session.merge() (PK-based) with proper
dialect-aware INSERT ON DUPLICATE KEY UPDATE / ON CONFLICT DO UPDATE
- Filter out listings with null/zero coordinates before routing
- Raise OSError when all routing engines fail with 0 results computed,
distinguishing "nothing to compute" from "all engines unreachable"
- Disable OpenAPI docs/redoc/openapi.json when APP_ENV=production
- Strip uvicorn Server header with --no-server-header in Dockerfile and docker-compose.yml
- Add OriginValidatorMiddleware to reject state-changing requests from disallowed origins
- Add global exception handler to prevent stack trace leakage on unhandled errors
- Add tests for all new security features (OpenAPI, origin validation, exception handler, server header)
- Disable source maps in production builds (vite.config.ts: sourcemap: false)
- Add vite-plugin-obfuscator for JS obfuscation (hex identifiers, base64 string encoding)
- Move OIDC config behind VITE_* env vars with dev fallbacks (auth/config.ts)
- Add server_tokens off to nginx.conf to stop advertising nginx version
- Add type declaration for vite-plugin-obfuscator
- POI popup: use DOM API with textContent (auto-escapes) instead of template literal in setHTML
- Listing popup: replace renderToString + setHTML with createRoot + setDOMContent for proper React lifecycle
- Fix task status IDOR by adding ownership check; suppress traceback/error in production
- Passkey routes: return generic error messages for internal exceptions, keep ValueError for user-facing
- JWT_SECRET and OIDC_CLIENT_ID: raise RuntimeError in production when using defaults
- Rate limiter: add in-memory fallback counter when Redis is unavailable
- Fix X-Forwarded-For IP spoofing with trusted_proxy_depth (rightmost-N selection)
- Add SecurityHeadersMiddleware (X-Content-Type-Options, X-Frame-Options, CSP, conditional HSTS)
- CORS: add PUT/DELETE methods for POI routes
- POI input validation: field length and coordinate range constraints
- QueryParameters: add min_sqm <= max_sqm validation
The distance calculator always queried the rentlisting table regardless of
listing type because get_listings() defaulted to RentListing when called
without query_parameters. Added a listing_type parameter to get_listings()
and _get_model_for_query() so callers can select the correct table directly.
Simplify the filter UI to show only essential filters (type toggle, price/bedroom
range sliders, min size) by default, with advanced filters collapsed. Extract
visualization controls (color-by metric, POI travel mode) into a separate
VisualizationCard component. Wire up previously ignored backend filters: max_sqm,
min/max_price_per_sqm, and district_names now work end-to-end.
Math.round(values.length * 0.95) produces an out-of-bounds index when
the dataset has fewer than ~20 features (e.g. after tight travel time
filtering). values[outOfBounds] returns undefined, cascading to NaN
color stops which crash Mapbox's expression evaluator. Clamp both
min and max indices to values.length - 1.
Replace the single global max travel time filter with per-POI filters.
Each POI gets its own travel mode selector and max minutes input in the
filter panel. Listings must satisfy ALL active filters (AND logic).
Fix Mapbox "Input is not a number" error by ensuring color stops are
always strictly monotonic (guard min === max) and always set (even when
no valid metric values exist). Also filter Infinity values from the
color scale computation. Widen the filter panel from w-64 to w-80.
Thread onTaskCompleted callback from TaskIndicator through Header to App.tsx
so listings auto-refresh when a background task (e.g. POI distance calculation)
completes. Add AllPOIDistances component to PropertyCard that shows all user
POIs with travel times or — placeholder for missing modes.
- Update Geofabrik download URL from great-britain to united-kingdom
(old path returns 302 redirect to homepage).
- Switch OSRM Docker volumes from named volume to bind mount
(./osrm-data:/data) so osrm-setup.sh output is used directly.
- Add osrm-data/ to .gitignore (large binaries, regenerated by script).
After creating a POI, automatically trigger WALK and BICYCLE distance
calculations (cheap OSRM batch API). TRANSIT is excluded since it uses
the expensive OTP backend — users trigger it manually via the calculator
button. Failure is non-fatal: the POI is still created and calculation
can be retried manually.
- Fix OSRM client to use semicolons (not commas) for source/destination
indices in /table API requests. Commas caused "Query string malformed"
errors for any batch with more than one origin.
- Add error handling in poi_distance_calculator for unreachable routing
engines (OSRM/OTP). Connection failures now log an error and skip the
mode instead of crashing the entire Celery task.
Deployments and Services for osrm-foot (256-512MB), osrm-bicycle
(256-512MB), and OTP (1-2GB). Includes PVCs for data storage and an
init Job to download and pre-process Greater London OSM data.
POIManager component in FilterPanel for creating/deleting POIs and
triggering distance calculations. PropertyCard shows travel time badges
(walk/cycle/transit) per POI. Map renders POI locations as red markers.
API client extended with POST body support for POI endpoints.
Adds osrm-foot, osrm-bicycle, and otp services to Docker Compose under
a 'routing' profile (opt-in). Setup scripts download Greater London OSM
data and pre-process for OSRM foot/bicycle profiles, plus TfL GTFS for
OTP transit. Routing engine env vars added to .env.sample.
