Commit graph

487 commits

Author SHA1 Message Date
Viktor Barzin
1ae00b7cbf
Add multi-layer caching: 24h Redis TTL, stale-while-revalidate, frontend LRU cache
- Increase Redis cache TTL from 30 minutes to 24 hours
- Add stale-while-revalidate: serve stale cache (>4h) immediately while
  repopulating in background with SETNX lock to prevent concurrent rebuilds
- Add in-memory frontend LRU cache (5 entries) so repeat filter visits
  are instant without network requests
- Invalidate frontend cache on listing refresh and task completion
- Add unit tests for get_cache_age, is_cache_stale, acquire_repopulation_lock
2026-02-23 20:09:36 +00:00
Viktor Barzin
04bda8c127
Fix FilterPanel test: button text changed to 'Show Matching Listings'
The FilterPanel component's submit button was renamed from 'Apply Filters'
to 'Show Matching Listings' but the test was not updated to match.
2026-02-23 19:40:06 +00:00
Viktor Barzin
9b3d35669f
Fix API crash: status_code=204 with response body assertion error
FastAPI rejects status_code=204 on routes with a return type annotation.
Return an explicit Response(status_code=204) instead.
2026-02-23 19:22:10 +00:00
Viktor Barzin
7dc8dc736f Retrigger CI build 2026-02-22 23:59:21 +00:00
Viktor Barzin
6dcb0d8787 fix clone backoff duration format 2026-02-22 23:40:50 +00:00
Viktor Barzin
af29295793 fix clone backoff duration format 2026-02-22 23:40:49 +00:00
Viktor Barzin
f02e27e9ba test clone retry 2026-02-22 23:37:46 +00:00
Viktor Barzin
b2904c03c2 [ci skip] Add clone retry settings for DNS resilience 2026-02-22 23:37:21 +00:00
Viktor Barzin
c68841e499 [ci skip] Add clone retry settings for DNS resilience 2026-02-22 23:37:19 +00:00
Viktor Barzin
432ceb575d Remove custom clone override, use built-in Woodpecker clone 2026-02-22 23:30:38 +00:00
Viktor Barzin
62fe594be2 Remove custom clone override, use built-in Woodpecker clone 2026-02-22 23:30:38 +00:00
Viktor Barzin
e3d8939d17 test CI secrets reload 2026-02-22 23:22:51 +00:00
Viktor Barzin
0666411570 test CI with fixed secrets 2026-02-22 23:20:05 +00:00
Viktor Barzin
553fa55102 trigger CI rebuild 2026-02-22 23:16:35 +00:00
Viktor Barzin
3efe7e251a Fix clone: no-op clone override + step-level auth 2026-02-22 23:02:54 +00:00
Viktor Barzin
2000ea5403 Fix clone: no-op clone override + step-level auth 2026-02-22 23:02:53 +00:00
Viktor Barzin
d3b8cb1f84 Standardize clone: skip_clone + step-level retry 2026-02-22 22:55:12 +00:00
Viktor Barzin
9e702a3f5f Standardize clone: skip_clone + step-level retry 2026-02-22 22:54:43 +00:00
Viktor Barzin
4e881780d9 Fix clone: use GitHub token for auth + DNS retry 2026-02-22 22:49:03 +00:00
Viktor Barzin
3a875c70a7 Fix clone: use GitHub token for auth + DNS retry 2026-02-22 22:48:31 +00:00
Viktor Barzin
813a0d7379 Fix clone: add netrc auth + cleanup between retries 2026-02-22 22:28:49 +00:00
Viktor Barzin
8747db4aaf Fix clone: add netrc auth + cleanup between retries 2026-02-22 22:28:16 +00:00
Viktor Barzin
32b38c432b Add clone retry logic for DNS resilience 2026-02-22 22:13:53 +00:00
Viktor Barzin
71f1d324a1 Add clone retry logic for DNS resilience 2026-02-22 22:13:22 +00:00
Viktor Barzin
23b544f73a Add Woodpecker CI API pipeline 2026-02-22 21:43:26 +00:00
Viktor Barzin
7c857cab75 Add Woodpecker CI frontend pipeline 2026-02-22 21:43:00 +00:00
Viktor Barzin
692d68c270
Unpause deployments and add ReplicaSet/status debug output
Add spec.paused: null to the strategic merge patch to clear any
paused state that might be blocking rollouts. Also add detailed
deployment status debug output (conditions, replica counts,
paused state) and list ReplicaSets to diagnose why new pods
aren't being created despite spec changes.
2026-02-22 19:40:44 +00:00
Viktor Barzin
0a2b207bee
Use strategic merge patch with restart annotation in deploy step
The JSON Patch was updating the deployment spec image but K8s wasn't
creating new pods — the rollout was stuck from previous failed builds.
Switch to strategic-merge-patch that also sets restartedAt annotation,
equivalent to 'kubectl rollout restart', which forces K8s to create
a new ReplicaSet regardless of stalled rollout state.
2026-02-22 19:21:56 +00:00
Viktor Barzin
339e2cf2ab
Improve frontend UI/UX: accessibility, discoverability, and cleanup
- Add prefers-reduced-motion support (global CSS + SwipeCard spring config)
- Add dismissible map click hint overlay with localStorage persistence
- Replace generic image alt text with descriptive property info across 4 components
- Rename filter buttons to clarify intent (Show Matching Listings / Scrape New from Rightmove)
- Fix mobile FAB overlap with bottom sheet via dynamic snap-aware positioning
- Add swipe review onboarding overlay with gesture explanations and button labels
- Delete unused components: AppSidebar, ActiveQuery, SavedView
2026-02-22 18:47:09 +00:00
Viktor Barzin
268f4fd272
Add detailed debug output to API verify-deploy
Show deployment spec image to confirm patch worked, dump all pod
details (image, ready, phase, restarts, reason) on attempts 1/10/30
and on failure. This will reveal whether pods are crashlooping, stuck
in Pending, or if the deployment patch didn't take effect.
2026-02-22 18:35:20 +00:00
Viktor Barzin
67d4ab3821
Fix celery metrics not reaching Prometheus and update Grafana dashboard
Init OTel metrics at module level in celery_app.py so prefork child
processes inherit the MeterProvider and PrometheusMetricReader from
the parent.  Previously, worker_process_init created a separate
MeterProvider in each child, disconnected from the HTTP server in
the main process — so all scrape/celery/OCR metrics were silently
lost.

Update Grafana dashboard with API Performance and Frontend Performance
sections, synced from the live cluster dashboard.
2026-02-22 17:58:20 +00:00
Viktor Barzin
bfee06525b
Fix verify-deploy image comparison to handle docker.io prefix
Kubernetes may normalize image references to include docker.io/ prefix
(e.g., docker.io/viktorbarzin/realestatecrawler:356), causing exact
string match to fail. Use endswith() instead. Also add debug output
on first attempt to show actual pod images and readiness state.
2026-02-22 17:52:32 +00:00
Viktor Barzin
d90fa38776
Add frontend performance metrics pipeline to Prometheus
Collect browser-side worker round-trips, computation times, main-thread
operations, and feature counts, batch them client-side, and expose as
Prometheus histograms via a new POST /api/perf endpoint.
2026-02-22 17:30:29 +00:00
Viktor Barzin
c24c3a545c
Fix API verify-deploy timeout by removing pod age filter
The jq filter required pods to be younger than 60 seconds AND ready,
but pods often take longer than 60s to become ready (alembic migrations,
image pull, etc.), causing the check to never match. The image tag check
is sufficient to confirm the new version is running. This aligns the API
verify-deploy with the frontend verify-deploy which has no age filter.
2026-02-22 17:20:19 +00:00
Viktor Barzin
34743127fb
Add debug CLI listings subcommand (list, detail, stream, refresh) 2026-02-22 15:17:56 +00:00
Viktor Barzin
34f9e933c0
Add debug CLI decisions, POIs, and tasks subcommands 2026-02-22 15:15:30 +00:00
Viktor Barzin
2196e256f4
Register debug group in main.py with districts subcommand and tests 2026-02-22 15:12:05 +00:00
Viktor Barzin
7a35489f35
Add debug CLI districts subcommand 2026-02-22 15:12:02 +00:00
Viktor Barzin
df0fa41586
Add cli/ package with shared debug context 2026-02-22 15:11:58 +00:00
Viktor Barzin
9f38b1ea9c
Add debug CLI implementation plan 2026-02-22 15:08:34 +00:00
Viktor Barzin
01e928151a
Add debug CLI design doc 2026-02-22 15:05:18 +00:00
Viktor Barzin
81ff9d9e41
Move hexgrid heatmap computation to Web Worker
R-tree building, hex grid generation, and percentile sorting now run
off the main thread, eliminating 20s+ UI freezes on large datasets.
The old bundled HexgridHeatmap.js is replaced by a typed worker +
main-thread client with dual R-trees (worker for grid gen, main
thread for synchronous click queries).
2026-02-22 15:04:37 +00:00
Viktor Barzin
c6f7b47446
Fix e2e test for adaptive first batch size
test_large_batch_streaming expected 4 batches (200/50) but the
adaptive first batch splits the first batch into 5+45, producing
5 batches total.
2026-02-22 13:48:36 +00:00
Viktor Barzin
8ef6868881
Eliminate frontend POI waterfall for faster initial load
Listing stream fires immediately on auth without waiting for POI
fetch. POI distances are not needed for initial rendering and are
only computed when user selects POI metric or sets travel filters.
This saves ~200-500ms on initial load and keeps the stream on the
cached Redis path.
2026-02-22 13:31:10 +00:00
Viktor Barzin
3885fd52fe
Add bulk POI distances endpoint for decoupled loading
New GET /api/poi/distances/bulk returns all POI distances keyed by
listing ID, allowing the frontend to fetch distances separately
from the listing stream and keep the stream on the cached path.
2026-02-22 13:29:35 +00:00
Viktor Barzin
2f3d2dc480
Add Server-Timing header to streaming endpoint
Reports cache_check latency in the response header, visible in
browser DevTools Network tab for ongoing performance monitoring.
2026-02-22 13:27:11 +00:00
Viktor Barzin
e53b1e120a
Defer decision ID fetch to after metadata message
Decision IDs are now loaded inside the streaming generators after
the metadata message is yielded, eliminating a blocking DB query
from the pre-stream path (~100-200ms improvement to TTFB).
2026-02-22 13:26:17 +00:00
Viktor Barzin
e99006e2f9
Send smaller first batch (5 features) for faster first paint
Subsequent batches use the normal batch_size (default 50). This
reduces server-side time-to-first-property by ~10x since only 5
features need to be serialized before the first yield.
2026-02-22 13:24:01 +00:00
Viktor Barzin
9179456bf7
Set explicit resource limits for API and celery-beat pods
Overrides LimitRange defaults (500m CPU) which caused kernel CPU
throttling during streaming requests. API gets 2000m CPU limit,
celery-beat gets 200m.
2026-02-22 13:20:56 +00:00
Viktor Barzin
8db7b60493
Add time-to-first-property implementation plan
7 tasks: K8s resource limits, adaptive first batch, deferred
decision fetch, Server-Timing headers, bulk POI distances endpoint,
frontend waterfall elimination, and end-to-end verification.
2026-02-22 13:16:42 +00:00