CI: migrate Docker build/push from Woodpecker to GitHub Actions
Was: Woodpecker built+pushed to DockerHub, then `kubectl set image` patched
the four Deployments to a pinned numeric tag. With Deployments pinned to
:51 (immutable tag), Keel polled forever and never saw a digest bump — and
no DockerHub pull-secret meant Keel hit 401 on the private repo at every
poll. The 4-Deployment setup also had a latent ImagePullBackOff risk: if a
node was replaced, fresh pulls would fail.
Now: GHA builds+pushes (.github/workflows/build-{api,frontend}.yml) on push
to master. Cluster Deployments reference :latest with an imagePullSecret
sourced from Vault via ESO (codified in infra/stacks/real-estate-crawler/
main.tf, separate commit). Keel polls :latest, sees the new digest after
each GHA build, and rolls all four Deployments.
- .github/workflows/build-api.yml: pytest (unit + integration/regression/
e2e/test_listing_geojson) + buildx push viktorbarzin/realestatecrawler
to {<8-char-sha>, latest}.
- .github/workflows/build-frontend.yml: vitest (all 4 ex-shards in one
run) + Vite build with VITE_MAPBOX_TOKEN from GHA secret + buildx push
viktorbarzin/immoweb to {<8-char-sha>, latest}.
- .woodpecker/{api,frontend}.yml renamed to
.woodpecker/build-fallback-{api,frontend}.yml with `event: deployment`
so they no longer fire on push — kept as manual-only fallback if GHA
is down (CLAUDE.md convention from the 10 already-migrated projects).
- .claude/CLAUDE.md: Git Workflow section updated to reflect GHA as
primary + the dockerhub-pull-secret wiring.
GHA repo secrets DOCKERHUB_TOKEN and MAPBOX_TOKEN populated from Vault
fields viktor.dockerhub_registry_password and ci/global.wrongmove-mapbox-token
respectively (DOCKERHUB_USERNAME=viktorbarzin was already set).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 19:11:31 +00:00
|
|
|
# Fallback pipeline — only fires on manual `deployment` event after GHA
|
|
|
|
|
# (.github/workflows/build-api.yml) became the primary builder on
|
|
|
|
|
# 2026-05-18. Kept around in case GHA is down and a build needs to ship
|
|
|
|
|
# from inside the cluster. `event: deployment` means it never auto-fires
|
|
|
|
|
# on push.
|
2026-02-22 21:43:26 +00:00
|
|
|
when:
|
CI: migrate Docker build/push from Woodpecker to GitHub Actions
Was: Woodpecker built+pushed to DockerHub, then `kubectl set image` patched
the four Deployments to a pinned numeric tag. With Deployments pinned to
:51 (immutable tag), Keel polled forever and never saw a digest bump — and
no DockerHub pull-secret meant Keel hit 401 on the private repo at every
poll. The 4-Deployment setup also had a latent ImagePullBackOff risk: if a
node was replaced, fresh pulls would fail.
Now: GHA builds+pushes (.github/workflows/build-{api,frontend}.yml) on push
to master. Cluster Deployments reference :latest with an imagePullSecret
sourced from Vault via ESO (codified in infra/stacks/real-estate-crawler/
main.tf, separate commit). Keel polls :latest, sees the new digest after
each GHA build, and rolls all four Deployments.
- .github/workflows/build-api.yml: pytest (unit + integration/regression/
e2e/test_listing_geojson) + buildx push viktorbarzin/realestatecrawler
to {<8-char-sha>, latest}.
- .github/workflows/build-frontend.yml: vitest (all 4 ex-shards in one
run) + Vite build with VITE_MAPBOX_TOKEN from GHA secret + buildx push
viktorbarzin/immoweb to {<8-char-sha>, latest}.
- .woodpecker/{api,frontend}.yml renamed to
.woodpecker/build-fallback-{api,frontend}.yml with `event: deployment`
so they no longer fire on push — kept as manual-only fallback if GHA
is down (CLAUDE.md convention from the 10 already-migrated projects).
- .claude/CLAUDE.md: Git Workflow section updated to reflect GHA as
primary + the dockerhub-pull-secret wiring.
GHA repo secrets DOCKERHUB_TOKEN and MAPBOX_TOKEN populated from Vault
fields viktor.dockerhub_registry_password and ci/global.wrongmove-mapbox-token
respectively (DOCKERHUB_USERNAME=viktorbarzin was already set).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 19:11:31 +00:00
|
|
|
- event: deployment
|
2026-02-22 21:43:26 +00:00
|
|
|
|
2026-02-22 23:37:21 +00:00
|
|
|
clone:
|
|
|
|
|
git:
|
|
|
|
|
image: woodpeckerci/plugin-git
|
|
|
|
|
settings:
|
|
|
|
|
attempts: 5
|
2026-02-22 23:40:50 +00:00
|
|
|
backoff: 10s
|
2026-02-22 23:37:21 +00:00
|
|
|
|
2026-02-22 22:55:12 +00:00
|
|
|
steps:
|
2026-02-22 21:43:26 +00:00
|
|
|
- name: install-api-deps
|
|
|
|
|
image: python:3.13-slim
|
2026-03-15 23:47:30 +00:00
|
|
|
backend_options:
|
|
|
|
|
kubernetes:
|
|
|
|
|
resources:
|
|
|
|
|
requests:
|
|
|
|
|
memory: 512Mi
|
|
|
|
|
limits:
|
|
|
|
|
memory: 1Gi
|
2026-02-22 21:43:26 +00:00
|
|
|
commands:
|
|
|
|
|
- apt-get update && apt-get install -y --no-install-recommends libglib2.0-0
|
|
|
|
|
- python -m venv .venv
|
|
|
|
|
- .venv/bin/pip install --quiet --upgrade pip
|
|
|
|
|
- >-
|
|
|
|
|
.venv/bin/pip install --quiet
|
|
|
|
|
pytest pytest-asyncio pytest-cov httpx fakeredis aioresponses
|
|
|
|
|
fastapi uvicorn sqlmodel sqlalchemy alembic pyjwt cryptography
|
|
|
|
|
celery redis click aiohttp aiohttp-socks pillow numpy pytesseract
|
|
|
|
|
opentelemetry-api opentelemetry-sdk opentelemetry-exporter-prometheus
|
|
|
|
|
opentelemetry-instrumentation-fastapi opentelemetry-instrumentation-sqlalchemy
|
|
|
|
|
python-dotenv webauthn apprise tenacity prometheus-client
|
|
|
|
|
email-validator opencv-python-headless tqdm pandas cachetools watchdog
|
|
|
|
|
|
|
|
|
|
- name: test-unit
|
|
|
|
|
image: python:3.13-slim
|
2026-02-23 22:00:09 +00:00
|
|
|
depends_on:
|
|
|
|
|
- install-api-deps
|
2026-02-22 21:43:26 +00:00
|
|
|
commands:
|
|
|
|
|
- apt-get update && apt-get install -y --no-install-recommends libglib2.0-0
|
|
|
|
|
- .venv/bin/pytest tests/unit/ -v --tb=short
|
|
|
|
|
|
|
|
|
|
- name: test-integration
|
|
|
|
|
image: python:3.13-slim
|
2026-02-23 22:00:09 +00:00
|
|
|
depends_on:
|
|
|
|
|
- install-api-deps
|
2026-02-22 21:43:26 +00:00
|
|
|
commands:
|
|
|
|
|
- apt-get update && apt-get install -y --no-install-recommends libglib2.0-0
|
|
|
|
|
- .venv/bin/pytest tests/integration/ tests/regression/ tests/e2e/ tests/test_listing_geojson.py -v --tb=short
|
|
|
|
|
|
2026-03-15 22:52:01 +00:00
|
|
|
- name: build-and-push-api
|
|
|
|
|
image: woodpeckerci/plugin-docker-buildx
|
|
|
|
|
depends_on:
|
|
|
|
|
- test-unit
|
|
|
|
|
- test-integration
|
2026-02-22 21:43:26 +00:00
|
|
|
settings:
|
|
|
|
|
username: viktorbarzin
|
|
|
|
|
password:
|
|
|
|
|
from_secret: dockerhub-token
|
|
|
|
|
repo: viktorbarzin/realestatecrawler
|
|
|
|
|
dockerfile: Dockerfile
|
|
|
|
|
context: .
|
|
|
|
|
target: production
|
2026-03-15 22:52:01 +00:00
|
|
|
platforms:
|
|
|
|
|
- linux/amd64
|
|
|
|
|
tag: ["${CI_PIPELINE_NUMBER}", "latest"]
|
|
|
|
|
cache_from: "viktorbarzin/realestatecrawler:latest"
|
|
|
|
|
cache_to: "type=inline"
|
2026-02-22 21:43:26 +00:00
|
|
|
|
|
|
|
|
- name: update-deployment
|
|
|
|
|
image: alpine
|
2026-02-23 22:00:09 +00:00
|
|
|
depends_on:
|
2026-03-15 22:52:01 +00:00
|
|
|
- build-and-push-api
|
2026-02-22 21:43:26 +00:00
|
|
|
commands:
|
|
|
|
|
- apk add --no-cache curl jq
|
|
|
|
|
- |
|
|
|
|
|
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
|
|
|
|
|
IMAGE="viktorbarzin/realestatecrawler:${CI_PIPELINE_NUMBER}"
|
|
|
|
|
RESTART_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
2026-02-23 22:00:09 +00:00
|
|
|
API="https://kubernetes:6443/apis/apps/v1/namespaces/realestate-crawler/deployments"
|
|
|
|
|
|
2026-02-22 21:43:26 +00:00
|
|
|
for DEPLOY in realestate-crawler-api realestate-crawler-celery realestate-crawler-celery-beat; do
|
2026-02-23 22:00:09 +00:00
|
|
|
STATUS=$(curl -sfk "$API/$DEPLOY" \
|
|
|
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
|
|
|
-H "Accept: application/json")
|
|
|
|
|
CONTAINER=$(echo "$STATUS" | jq -r '.spec.template.spec.containers[0].name')
|
|
|
|
|
PAUSED=$(echo "$STATUS" | jq -r '.spec.paused // false')
|
|
|
|
|
echo "Patching $DEPLOY (container=$CONTAINER, paused=$PAUSED) to image $IMAGE..."
|
|
|
|
|
|
|
|
|
|
curl -sf -X PATCH "$API/$DEPLOY" \
|
|
|
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
|
|
|
-H "Content-Type: application/strategic-merge-patch+json" \
|
|
|
|
|
-k -d "{\"spec\":{\"paused\":null,\"template\":{\"metadata\":{\"annotations\":{\"kubectl.kubernetes.io/restartedAt\":\"$RESTART_AT\"}},\"spec\":{\"containers\":[{\"name\":\"$CONTAINER\",\"image\":\"$IMAGE\"}]}}}}" \
|
|
|
|
|
| jq '{name: .metadata.name, generation: .metadata.generation, image: .spec.template.spec.containers[0].image, paused: .spec.paused, restartedAt: .spec.template.metadata.annotations["kubectl.kubernetes.io/restartedAt"]}'
|
2026-02-22 21:43:26 +00:00
|
|
|
done
|
|
|
|
|
|
|
|
|
|
- name: verify-deploy
|
|
|
|
|
image: alpine
|
2026-02-23 22:00:09 +00:00
|
|
|
depends_on:
|
|
|
|
|
- update-deployment
|
2026-02-22 21:43:26 +00:00
|
|
|
commands:
|
|
|
|
|
- apk add --no-cache curl jq
|
|
|
|
|
- |
|
|
|
|
|
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
|
|
|
|
|
EXPECTED_IMAGE="viktorbarzin/realestatecrawler:${CI_PIPELINE_NUMBER}"
|
2026-02-23 22:00:09 +00:00
|
|
|
BASE_API="https://kubernetes:6443/api/v1/namespaces/realestate-crawler/pods"
|
|
|
|
|
DEPLOY_API="https://kubernetes:6443/apis/apps/v1/namespaces/realestate-crawler/deployments"
|
|
|
|
|
|
2026-02-22 21:43:26 +00:00
|
|
|
for DEPLOY in realestate-crawler-api realestate-crawler-celery realestate-crawler-celery-beat; do
|
|
|
|
|
echo "Verifying $DEPLOY..."
|
|
|
|
|
PODS_API="$BASE_API?labelSelector=app%3D$DEPLOY"
|
2026-02-23 22:00:09 +00:00
|
|
|
RS_API="https://kubernetes:6443/apis/apps/v1/namespaces/realestate-crawler/replicasets?labelSelector=app%3D$DEPLOY"
|
|
|
|
|
|
|
|
|
|
DEPLOY_STATUS=$(curl -sfk "$DEPLOY_API/$DEPLOY" \
|
|
|
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
|
|
|
-H "Accept: application/json")
|
|
|
|
|
echo " Deployment spec image: $(echo "$DEPLOY_STATUS" | jq -r '.spec.template.spec.containers[0].image')"
|
|
|
|
|
echo " Deployment paused: $(echo "$DEPLOY_STATUS" | jq -r '.spec.paused // false')"
|
|
|
|
|
echo " Deployment status: replicas=$(echo "$DEPLOY_STATUS" | jq -r '.status.replicas // 0') updated=$(echo "$DEPLOY_STATUS" | jq -r '.status.updatedReplicas // 0') ready=$(echo "$DEPLOY_STATUS" | jq -r '.status.readyReplicas // 0') unavailable=$(echo "$DEPLOY_STATUS" | jq -r '.status.unavailableReplicas // 0')"
|
|
|
|
|
echo " Conditions:"
|
|
|
|
|
echo "$DEPLOY_STATUS" | jq -r '.status.conditions[]? | " \(.type): \(.status) (\(.reason // "unknown"))"' 2>/dev/null || echo " (none)"
|
|
|
|
|
|
|
|
|
|
echo " ReplicaSets:"
|
|
|
|
|
curl -sfk "$RS_API" \
|
|
|
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
|
|
|
-H "Accept: application/json" | \
|
|
|
|
|
jq -r '.items[] | " \(.metadata.name) desired=\(.spec.replicas) ready=\(.status.readyReplicas // 0) image=\(.spec.template.spec.containers[0].image)"' 2>/dev/null || echo " (none)"
|
|
|
|
|
|
2026-02-22 21:43:26 +00:00
|
|
|
FOUND=0
|
|
|
|
|
for i in $(seq 1 60); do
|
2026-02-23 22:00:09 +00:00
|
|
|
RAW=$(curl -sfk "$PODS_API" \
|
|
|
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
|
|
|
-H "Accept: application/json")
|
|
|
|
|
|
|
|
|
|
if [ "$i" -eq 1 ] || [ "$i" -eq 10 ] || [ "$i" -eq 30 ]; then
|
|
|
|
|
echo " DEBUG (attempt $i): All pods for $DEPLOY:"
|
|
|
|
|
echo "$RAW" | jq -r '[.items[] | {
|
|
|
|
|
name: .metadata.name,
|
|
|
|
|
image: .spec.containers[0].image,
|
|
|
|
|
ready: ([.status.containerStatuses[]? | .ready] | first // "unknown"),
|
|
|
|
|
phase: .status.phase,
|
|
|
|
|
restarts: ([.status.containerStatuses[]? | .restartCount] | first // 0),
|
|
|
|
|
reason: ([.status.containerStatuses[]? | .state | to_entries[] | .value.reason // empty] | first // "running")
|
|
|
|
|
}] | .[] | " \(.name) image=\(.image) ready=\(.ready) phase=\(.phase) restarts=\(.restarts) reason=\(.reason)"' 2>/dev/null || echo " (no pods or parse error)"
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
RESULT=$(echo "$RAW" | \
|
|
|
|
|
jq --arg img "$EXPECTED_IMAGE" '[.items[] | select(
|
|
|
|
|
(.status.containerStatuses[]? | .ready == true) and
|
|
|
|
|
(.spec.containers[]? | .image | endswith($img))
|
|
|
|
|
) | {name: .metadata.name, image: .spec.containers[0].image, started: .status.startTime}]')
|
|
|
|
|
|
|
|
|
|
COUNT=$(echo "$RESULT" | jq 'length' 2>/dev/null || echo 0)
|
|
|
|
|
echo " Attempt $i/60: $COUNT pod(s) ready with image matching $EXPECTED_IMAGE"
|
|
|
|
|
|
|
|
|
|
if [ "$COUNT" -gt 0 ] 2>/dev/null; then
|
|
|
|
|
echo "$RESULT" | jq -r '.[] | " \(.name) image=\(.image) started=\(.started)"'
|
|
|
|
|
echo "$DEPLOY is live!"
|
|
|
|
|
FOUND=1
|
|
|
|
|
break
|
|
|
|
|
fi
|
|
|
|
|
|
2026-02-22 21:43:26 +00:00
|
|
|
sleep 5
|
|
|
|
|
done
|
2026-02-23 22:00:09 +00:00
|
|
|
|
|
|
|
|
if [ "$FOUND" -ne 1 ]; then
|
|
|
|
|
echo " FINAL DEBUG: All pods for $DEPLOY:"
|
|
|
|
|
echo "$RAW" | jq -r '[.items[] | {
|
|
|
|
|
name: .metadata.name,
|
|
|
|
|
image: .spec.containers[0].image,
|
|
|
|
|
ready: ([.status.containerStatuses[]? | .ready] | first // "unknown"),
|
|
|
|
|
phase: .status.phase,
|
|
|
|
|
restarts: ([.status.containerStatuses[]? | .restartCount] | first // 0),
|
|
|
|
|
reason: ([.status.containerStatuses[]? | .state | to_entries[] | .value.reason // empty] | first // "running")
|
|
|
|
|
}] | .[] | " \(.name) image=\(.image) ready=\(.ready) phase=\(.phase) restarts=\(.restarts) reason=\(.reason)"' 2>/dev/null || echo " (no pods or parse error)"
|
|
|
|
|
echo "ERROR: No new ready pod for $DEPLOY with image $EXPECTED_IMAGE appeared within 5 minutes"
|
|
|
|
|
exit 1
|
|
|
|
|
fi
|
2026-02-22 21:43:26 +00:00
|
|
|
done
|