No description
Find a file
Viktor Barzin c2acbf5d2e CI: migrate Docker build/push from Woodpecker to GitHub Actions
Was: Woodpecker built+pushed to DockerHub, then `kubectl set image` patched
the four Deployments to a pinned numeric tag. With Deployments pinned to
:51 (immutable tag), Keel polled forever and never saw a digest bump — and
no DockerHub pull-secret meant Keel hit 401 on the private repo at every
poll. The 4-Deployment setup also had a latent ImagePullBackOff risk: if a
node was replaced, fresh pulls would fail.

Now: GHA builds+pushes (.github/workflows/build-{api,frontend}.yml) on push
to master. Cluster Deployments reference :latest with an imagePullSecret
sourced from Vault via ESO (codified in infra/stacks/real-estate-crawler/
main.tf, separate commit). Keel polls :latest, sees the new digest after
each GHA build, and rolls all four Deployments.

- .github/workflows/build-api.yml: pytest (unit + integration/regression/
  e2e/test_listing_geojson) + buildx push viktorbarzin/realestatecrawler
  to {<8-char-sha>, latest}.
- .github/workflows/build-frontend.yml: vitest (all 4 ex-shards in one
  run) + Vite build with VITE_MAPBOX_TOKEN from GHA secret + buildx push
  viktorbarzin/immoweb to {<8-char-sha>, latest}.
- .woodpecker/{api,frontend}.yml renamed to
  .woodpecker/build-fallback-{api,frontend}.yml with `event: deployment`
  so they no longer fire on push — kept as manual-only fallback if GHA
  is down (CLAUDE.md convention from the 10 already-migrated projects).
- .claude/CLAUDE.md: Git Workflow section updated to reflect GHA as
  primary + the dockerhub-pull-secret wiring.

GHA repo secrets DOCKERHUB_TOKEN and MAPBOX_TOKEN populated from Vault
fields viktor.dockerhub_registry_password and ci/global.wrongmove-mapbox-token
respectively (DOCKERHUB_USERNAME=viktorbarzin was already set).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 19:11:31 +00:00
.claude CI: migrate Docker build/push from Woodpecker to GitHub Actions 2026-05-18 19:11:31 +00:00
.github/workflows CI: migrate Docker build/push from Woodpecker to GitHub Actions 2026-05-18 19:11:31 +00:00
.woodpecker CI: migrate Docker build/push from Woodpecker to GitHub Actions 2026-05-18 19:11:31 +00:00
alembic wrongmove: daily price-trend monitoring (per-listing badge + macro strip) 2026-05-16 12:02:25 +00:00
api wrongmove: daily price-trend monitoring (per-listing badge + macro strip) 2026-05-16 12:02:25 +00:00
cli Add debug CLI listings subcommand (list, detail, stream, refresh) 2026-02-22 15:17:56 +00:00
config Add configurable request timeout and retry on TimeoutError 2026-02-21 17:50:36 +00:00
docs docs: add UI/UX redesign implementation plan 2026-02-28 15:46:30 +00:00
frontend wrongmove: daily price-trend monitoring (per-listing badge + macro strip) 2026-05-16 12:02:25 +00:00
grafana Add Server-Timing headers to all API endpoints for per-request latency breakdown 2026-02-23 21:30:51 +00:00
k8s wrongmove: round-3 fix sweep — scrape pipeline, BUY tab, filter URL state, render hygiene, map polish 2026-05-10 22:27:29 +00:00
models wrongmove: daily price-trend monitoring (per-listing badge + macro strip) 2026-05-16 12:02:25 +00:00
rec Add configurable request timeout and retry on TimeoutError 2026-02-21 17:50:36 +00:00
repositories wrongmove: add "seen" soft-hide decision with price-aware resurfacing 2026-05-16 11:07:44 +00:00
scripts Add setup-test-venv.sh for running tests without Docker [ci skip] 2026-02-14 11:44:49 +00:00
services wrongmove: daily price-trend monitoring (per-listing badge + macro strip) 2026-05-16 12:02:25 +00:00
tasks wrongmove: fix DetachedInstanceError in daily market aggregator task 2026-05-16 12:53:26 +00:00
tests wrongmove: stabilise setup_periodic_tasks tests after market-aggregator add 2026-05-16 12:07:50 +00:00
utils wrongmove: round-3 fix sweep — scrape pipeline, BUY tab, filter URL state, render hygiene, map polish 2026-05-10 22:27:29 +00:00
.dockerignore Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
.env.sample Add OSRM and OTP Docker services with setup scripts 2026-02-08 13:16:10 +00:00
.gitignore Add .worktrees/ to .gitignore for git worktree isolation 2026-02-21 15:48:34 +00:00
.style.yapf Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
alembic.ini Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
celery_app.py wrongmove: daily price-trend monitoring (per-listing badge + macro strip) 2026-05-16 12:02:25 +00:00
CLAUDE.md Add setup-test-venv.sh for running tests without Docker [ci skip] 2026-02-14 11:44:49 +00:00
csv_exporter.py Replace pandas with stdlib csv, apprise with direct Slack webhook, switch to opencv-headless 2026-02-21 19:47:10 +00:00
data_access.py Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
database.py Fall back to SQLite default when DB_CONNECTION_STRING is unset 2026-02-10 22:38:52 +00:00
docker-compose.yml Add structured JSON logging, OTel business metrics, and Grafana dashboard 2026-02-14 10:59:12 +00:00
Dockerfile wrongmove: round-3 fix sweep — scrape pipeline, BUY tab, filter URL state, render hygiene, map polish 2026-05-10 22:27:29 +00:00
GUIDE Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
listing_processor.py Fix metric imports: use module-level access instead of name imports 2026-02-14 11:21:49 +00:00
logging_config.py Add structured JSON logging, OTel business metrics, and Grafana dashboard 2026-02-14 10:59:12 +00:00
main.py Add debug CLI listings subcommand (list, detail, stream, refresh) 2026-02-22 15:17:56 +00:00
notifications.py Replace pandas with stdlib csv, apprise with direct Slack webhook, switch to opencv-headless 2026-02-21 19:47:10 +00:00
podman-compose.yml Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
poetry.lock Regenerate requirements.txt after dependency cleanup 2026-02-21 19:47:15 +00:00
pyproject.toml Replace pandas with stdlib csv, apprise with direct Slack webhook, switch to opencv-headless 2026-02-21 19:47:10 +00:00
README.md Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
redis_repository.py wrongmove: round-3 fix sweep — scrape pipeline, BUY tab, filter URL state, render hygiene, map polish 2026-05-10 22:27:29 +00:00
requirements.txt Regenerate requirements.txt after dependency cleanup 2026-02-21 19:47:15 +00:00
runall.sh Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
start.sh Add shared pre-push hook to run tests before pushing 2026-02-14 11:36:26 +00:00
TASKS.md Flatten repo structure: move crawler/ to root, remove vqa/ and immoweb/ 2026-02-07 23:01:20 +00:00
ui_exporter.py wrongmove: daily price-trend monitoring (per-listing badge + macro strip) 2026-05-16 12:02:25 +00:00

Setup

  1. Instal deps:
poetry install && cp .env.sample .env
  1. Check .env if you want to customize settings for broker and db
  2. run ./start.sh

This starts the backend

To start the fronend:

cd frontend && cp .env.sample .env

Change the DEV_HOST to any name you want to use to access the web interface.

Next, setup the DNS record (e.g in your /etc/hosts) file. This is important as auth is done via external [authentik] service that needs to redirect to a name.

Run ./start.sh

This starts a Caddy proxy with correct certificates, and npm dev server. All requests going to the frontend are forwarded to the npm server and the ones for the backed (that go to /api/*) are forwarded to the backend service.

Lastly, reachout to Viktor to allowlist your DEV_HOST so that authentik can authorize callbacks to your host.

Formatting

yapf --style .style.yapf --recursive .

For VSCode - install yapf extension. Enable formatting using yap and the style file in this repo (there may be an easier way; I put this in my user settings json):

{
    "[python]": {
        "editor.formatOnSaveMode": "file",
        "editor.formatOnSave": true,
        "editor.defaultFormatter": "eeyore.yapf",
        "editor.formatOnType": false
      },
      "yapf.args": ["--style", "/home/wizard/code/realestate-crawler/crawler/.style.yapf"]
}

ADB commands (from /Applications/BlueStacks.app/Contents/MacOS):

Set proxy

./hd-adb shell settings put global http_proxy 192.168.9.110:8080

Disable proxy:

/hd-adb shell settings put global http_proxy :0

Connect adb

./hd-adb connect 127.0.0.1:5555

Disconnect adb

/hd-adb disconnect