Commit graph

8 commits

Author SHA1 Message Date
Viktor Barzin
150342bb9e
Refactor codebase following Clean Code principles and add 229 tests
- Extract helpers to reduce function sizes (listing_tasks, app.py, query.py, listing_fetcher)
  - Replace nonlocal mutations with _PipelineState dataclass in listing_tasks
  - Fix bugs: isinstance→equality check in repository, verify_exp for OIDC tokens
  - Consolidate duplicate filter methods in listing_repository
  - Move hardcoded config to env vars with backward-compatible defaults
  - Simplify CLI decorator to auto-build QueryParameters
  - Add deprecation docstring to data_access.py
  - Test count: 158 → 387 (all passing)
2026-02-07 20:19:57 +00:00
Viktor Barzin
b4837e1603
Add crawl job progress drawer with phase tracking and live logs
- Add phase-aware progress reporting across all crawl phases (splitting,
  fetching, filtering, processing) with per-step counters
- Add TaskProgressDrawer component with phase timeline stepper, detail
  counters, progress bar with ETA, and live worker log viewer
- Add on_step_complete callback to ListingProcessor for granular tracking
  of details/images/OCR steps
- Extend QuerySplitter on_progress callback with structured counter data
- Capture celery worker logs via ring buffer handler and inject into task
  state updates for frontend display
- Guard taskResult updates with phase presence check to prevent drawer
  from blanking during state transitions
2026-02-06 22:37:53 +00:00
Viktor Barzin
c4b11ccfe9 Add comprehensive logging to Celery tasks and listing processor 2026-02-06 20:47:36 +00:00
Kadir
0801aaf200
More ruff fixes (#2)
* adding ruff auto check for pull requests as well as fixing all ruff errors

* More ruff fixes: forgot half of the ruff checks

Forgot to do a git add all :D

---------

Co-authored-by: Kadir <git@k8n.dev>
2025-09-14 19:44:03 +01:00
Viktor Barzin
526f4fc0c3
update last seen property when processing listings to refresh data 2025-08-23 22:36:37 +00:00
Viktor Barzin
480957dc72
add timeout when fetching details and use new entrypoint for task processing 2025-08-23 22:20:42 +00:00
Viktor Barzin
d1cef99c5a
make task processing a bit better. still doing 1 query to check if needs processing; will fix later 2025-07-27 20:09:41 +00:00
Viktor Barzin
91a0436f7f
migrate processing to a pipeline approach where each listing is processed in a pipeline in parallel and status reported back to track progress 2025-07-27 18:33:39 +00:00