Commit graph

475 commits

Author SHA1 Message Date
Viktor Barzin
0666411570 test CI with fixed secrets 2026-02-22 23:20:05 +00:00
Viktor Barzin
553fa55102 trigger CI rebuild 2026-02-22 23:16:35 +00:00
Viktor Barzin
3efe7e251a Fix clone: no-op clone override + step-level auth 2026-02-22 23:02:54 +00:00
Viktor Barzin
2000ea5403 Fix clone: no-op clone override + step-level auth 2026-02-22 23:02:53 +00:00
Viktor Barzin
d3b8cb1f84 Standardize clone: skip_clone + step-level retry 2026-02-22 22:55:12 +00:00
Viktor Barzin
9e702a3f5f Standardize clone: skip_clone + step-level retry 2026-02-22 22:54:43 +00:00
Viktor Barzin
4e881780d9 Fix clone: use GitHub token for auth + DNS retry 2026-02-22 22:49:03 +00:00
Viktor Barzin
3a875c70a7 Fix clone: use GitHub token for auth + DNS retry 2026-02-22 22:48:31 +00:00
Viktor Barzin
813a0d7379 Fix clone: add netrc auth + cleanup between retries 2026-02-22 22:28:49 +00:00
Viktor Barzin
8747db4aaf Fix clone: add netrc auth + cleanup between retries 2026-02-22 22:28:16 +00:00
Viktor Barzin
32b38c432b Add clone retry logic for DNS resilience 2026-02-22 22:13:53 +00:00
Viktor Barzin
71f1d324a1 Add clone retry logic for DNS resilience 2026-02-22 22:13:22 +00:00
Viktor Barzin
23b544f73a Add Woodpecker CI API pipeline 2026-02-22 21:43:26 +00:00
Viktor Barzin
7c857cab75 Add Woodpecker CI frontend pipeline 2026-02-22 21:43:00 +00:00
Viktor Barzin
692d68c270
Unpause deployments and add ReplicaSet/status debug output
Add spec.paused: null to the strategic merge patch to clear any
paused state that might be blocking rollouts. Also add detailed
deployment status debug output (conditions, replica counts,
paused state) and list ReplicaSets to diagnose why new pods
aren't being created despite spec changes.
2026-02-22 19:40:44 +00:00
Viktor Barzin
0a2b207bee
Use strategic merge patch with restart annotation in deploy step
The JSON Patch was updating the deployment spec image but K8s wasn't
creating new pods — the rollout was stuck from previous failed builds.
Switch to strategic-merge-patch that also sets restartedAt annotation,
equivalent to 'kubectl rollout restart', which forces K8s to create
a new ReplicaSet regardless of stalled rollout state.
2026-02-22 19:21:56 +00:00
Viktor Barzin
339e2cf2ab
Improve frontend UI/UX: accessibility, discoverability, and cleanup
- Add prefers-reduced-motion support (global CSS + SwipeCard spring config)
- Add dismissible map click hint overlay with localStorage persistence
- Replace generic image alt text with descriptive property info across 4 components
- Rename filter buttons to clarify intent (Show Matching Listings / Scrape New from Rightmove)
- Fix mobile FAB overlap with bottom sheet via dynamic snap-aware positioning
- Add swipe review onboarding overlay with gesture explanations and button labels
- Delete unused components: AppSidebar, ActiveQuery, SavedView
2026-02-22 18:47:09 +00:00
Viktor Barzin
268f4fd272
Add detailed debug output to API verify-deploy
Show deployment spec image to confirm patch worked, dump all pod
details (image, ready, phase, restarts, reason) on attempts 1/10/30
and on failure. This will reveal whether pods are crashlooping, stuck
in Pending, or if the deployment patch didn't take effect.
2026-02-22 18:35:20 +00:00
Viktor Barzin
67d4ab3821
Fix celery metrics not reaching Prometheus and update Grafana dashboard
Init OTel metrics at module level in celery_app.py so prefork child
processes inherit the MeterProvider and PrometheusMetricReader from
the parent.  Previously, worker_process_init created a separate
MeterProvider in each child, disconnected from the HTTP server in
the main process — so all scrape/celery/OCR metrics were silently
lost.

Update Grafana dashboard with API Performance and Frontend Performance
sections, synced from the live cluster dashboard.
2026-02-22 17:58:20 +00:00
Viktor Barzin
bfee06525b
Fix verify-deploy image comparison to handle docker.io prefix
Kubernetes may normalize image references to include docker.io/ prefix
(e.g., docker.io/viktorbarzin/realestatecrawler:356), causing exact
string match to fail. Use endswith() instead. Also add debug output
on first attempt to show actual pod images and readiness state.
2026-02-22 17:52:32 +00:00
Viktor Barzin
d90fa38776
Add frontend performance metrics pipeline to Prometheus
Collect browser-side worker round-trips, computation times, main-thread
operations, and feature counts, batch them client-side, and expose as
Prometheus histograms via a new POST /api/perf endpoint.
2026-02-22 17:30:29 +00:00
Viktor Barzin
c24c3a545c
Fix API verify-deploy timeout by removing pod age filter
The jq filter required pods to be younger than 60 seconds AND ready,
but pods often take longer than 60s to become ready (alembic migrations,
image pull, etc.), causing the check to never match. The image tag check
is sufficient to confirm the new version is running. This aligns the API
verify-deploy with the frontend verify-deploy which has no age filter.
2026-02-22 17:20:19 +00:00
Viktor Barzin
34743127fb
Add debug CLI listings subcommand (list, detail, stream, refresh) 2026-02-22 15:17:56 +00:00
Viktor Barzin
34f9e933c0
Add debug CLI decisions, POIs, and tasks subcommands 2026-02-22 15:15:30 +00:00
Viktor Barzin
2196e256f4
Register debug group in main.py with districts subcommand and tests 2026-02-22 15:12:05 +00:00
Viktor Barzin
7a35489f35
Add debug CLI districts subcommand 2026-02-22 15:12:02 +00:00
Viktor Barzin
df0fa41586
Add cli/ package with shared debug context 2026-02-22 15:11:58 +00:00
Viktor Barzin
9f38b1ea9c
Add debug CLI implementation plan 2026-02-22 15:08:34 +00:00
Viktor Barzin
01e928151a
Add debug CLI design doc 2026-02-22 15:05:18 +00:00
Viktor Barzin
81ff9d9e41
Move hexgrid heatmap computation to Web Worker
R-tree building, hex grid generation, and percentile sorting now run
off the main thread, eliminating 20s+ UI freezes on large datasets.
The old bundled HexgridHeatmap.js is replaced by a typed worker +
main-thread client with dual R-trees (worker for grid gen, main
thread for synchronous click queries).
2026-02-22 15:04:37 +00:00
Viktor Barzin
c6f7b47446
Fix e2e test for adaptive first batch size
test_large_batch_streaming expected 4 batches (200/50) but the
adaptive first batch splits the first batch into 5+45, producing
5 batches total.
2026-02-22 13:48:36 +00:00
Viktor Barzin
8ef6868881
Eliminate frontend POI waterfall for faster initial load
Listing stream fires immediately on auth without waiting for POI
fetch. POI distances are not needed for initial rendering and are
only computed when user selects POI metric or sets travel filters.
This saves ~200-500ms on initial load and keeps the stream on the
cached Redis path.
2026-02-22 13:31:10 +00:00
Viktor Barzin
3885fd52fe
Add bulk POI distances endpoint for decoupled loading
New GET /api/poi/distances/bulk returns all POI distances keyed by
listing ID, allowing the frontend to fetch distances separately
from the listing stream and keep the stream on the cached path.
2026-02-22 13:29:35 +00:00
Viktor Barzin
2f3d2dc480
Add Server-Timing header to streaming endpoint
Reports cache_check latency in the response header, visible in
browser DevTools Network tab for ongoing performance monitoring.
2026-02-22 13:27:11 +00:00
Viktor Barzin
e53b1e120a
Defer decision ID fetch to after metadata message
Decision IDs are now loaded inside the streaming generators after
the metadata message is yielded, eliminating a blocking DB query
from the pre-stream path (~100-200ms improvement to TTFB).
2026-02-22 13:26:17 +00:00
Viktor Barzin
e99006e2f9
Send smaller first batch (5 features) for faster first paint
Subsequent batches use the normal batch_size (default 50). This
reduces server-side time-to-first-property by ~10x since only 5
features need to be serialized before the first yield.
2026-02-22 13:24:01 +00:00
Viktor Barzin
9179456bf7
Set explicit resource limits for API and celery-beat pods
Overrides LimitRange defaults (500m CPU) which caused kernel CPU
throttling during streaming requests. API gets 2000m CPU limit,
celery-beat gets 200m.
2026-02-22 13:20:56 +00:00
Viktor Barzin
8db7b60493
Add time-to-first-property implementation plan
7 tasks: K8s resource limits, adaptive first batch, deferred
decision fetch, Server-Timing headers, bulk POI distances endpoint,
frontend waterfall elimination, and end-to-end verification.
2026-02-22 13:16:42 +00:00
Viktor Barzin
6d653dba63
Add time-to-first-property performance optimization design
Measured baseline: 877ms TTFP cold, 334ms warm, with CPU throttling
confirmed (500m limit, 10-12 throttle events per streaming request).
Plan covers: CPU limit bump, frontend waterfall elimination, adaptive
first batch, deferred decision filtering, POI decoupling, and
Server-Timing headers.
2026-02-22 13:14:01 +00:00
Viktor Barzin
9dc011754b
Add cancel button to streaming progress bar
The X button aborts the in-flight fetch via AbortController,
which was already wired up but had no UI trigger. Works for
both desktop and mobile views.
2026-02-22 02:04:56 +00:00
Viktor Barzin
301579c255
Separate photo swipe from card swipe gestures
Move drag handler from outer card to details section only.
Swiping on photos now navigates the carousel, swiping on
the details area below triggers like/dislike/skip actions.
2026-02-22 02:02:10 +00:00
Viktor Barzin
a2c1f81644
Fix photo extraction: look for both 'photos' and 'images' keys
Rightmove API stores photos under the 'photos' key in the response,
but the GeoJSON export and detail API were only checking 'images'.
This key mismatch caused all listings to fall back to the single
photo_thumbnail. Now checks both keys with fallback.
2026-02-22 01:21:50 +00:00
Viktor Barzin
d50d1c07f6
Use high-res images and return all photos in GeoJSON
- Use maxSizeUrl instead of url for photo URLs (highest available
  resolution from Rightmove)
- Remove 5-photo cap in GeoJSON export — return all available photos
- Apply same fix to both streaming and model-based export paths,
  and to the listing detail API endpoint
2026-02-22 00:54:58 +00:00
Viktor Barzin
e2c22f025f
Expand swipe card to 50/50 photo/details split with all info
- Card now fills available height with photo carousel in top half
  and property details in bottom half
- Details section shows: price, beds/sqm/price-per-sqm, agency,
  available date, all POI distances, and price history summary
- Fix DialogTitle accessibility warning in ListingDetailSheet and
  MobileBottomSheet (add sr-only Drawer.Title)
2026-02-22 00:49:32 +00:00
Viktor Barzin
611449d328
Add photo carousel to swipe cards
SwipeCard now shows all available photos (up to 5) using
embla-carousel instead of just the thumbnail. Includes prev/next
arrow buttons and dot indicators. The photo area uses touch-action:
pan-x so carousel swipes don't trigger card swipes.
2026-02-21 23:57:26 +00:00
Viktor Barzin
a742f9bb65
Fix frontend verify-deploy: remove stale 60s age filter
The age filter rejected pods older than 60s, but by the time the
verify step runs after build+publish+deploy, the new pods are often
already past that window. The image tag check is sufficient to
identify the correct deployment.
2026-02-21 23:50:20 +00:00
Viktor Barzin
0bb7e4c723
Parallelize API tests and build, gate publish on test pass
Same pattern as the frontend pipeline: build pushes to a staging
tag (build-N) in parallel with tests. Tests split into unit and
integration shards using a shared venv from the workspace. Publish
step uses skopeo to copy the manifest to final tags (:N, :latest,
:builder) only after all tests and the build succeed. Named the
final Dockerfile stage 'production' so target skips the test stage.
2026-02-21 23:46:16 +00:00
Viktor Barzin
68e7f2a334
Separate frontend image build from publish, gate publish on tests
Build step now pushes to a staging tag (build-N) in parallel with
test shards. A new publish step uses skopeo to copy the manifest to
the final tags (:N and :latest) only after all tests pass. This is
a server-side manifest copy with no layer re-upload.
2026-02-21 21:16:37 +00:00
Viktor Barzin
eacdf24621
Add tap-to-detail on swipe cards and fix color overlay alignment
- Add onTap callback to SwipeCard using useDrag's tap detection
- Wire through SwipeReviewMode to open ListingDetailSheet on tap
- Fix color overlay misalignment: add relative to card container so
  the absolute overlay positions within the rounded card, not the
  full-width outer wrapper
2026-02-21 21:13:32 +00:00
Viktor Barzin
9c954c0e43
Add prev/next arrow buttons to photo carousel 2026-02-21 21:07:17 +00:00