wrongmove/docs/plans/2026-02-21-ci-pipeline-optimization-design.md

# CI Pipeline Optimization — Design Document

Date: 2026-02-21

## Problem

The Woodpecker CI pipeline is slow. Both the frontend and backend pipelines install dependencies multiple times per build and pull Docker cache layers from Docker Hub instead of the local registry at `10.0.20.10:5000`.

## Root Causes

1. **Frontend `npm ci` runs twice**: once in the test step (bare node image), once inside the Kaniko Dockerfile build. 560MB of node_modules downloaded each time.
2. **Backend builds 2 Docker images**: one for the `:test` target, one for the final image. Both execute the `builder` stage which runs `pip install -r requirements.txt` (1874 lines, hash-pinned).
3. **Docker cache pulled from Docker Hub**: `cache_from` and `cache_repo` point to Docker Hub. Local registry at `10.0.20.10:5000` already exists and has all images but is not used for caching.

## Solution

### 1. Local Registry for Docker Layer Caching

Switch all `cache_from` and `cache_repo` references from Docker Hub to `10.0.20.10:5000`. The local registry already has `viktorbarzin/realestatecrawler` and `viktorbarzin/immoweb` with `builder`, `test`, and `latest` tags. Local network pulls are near-instant vs Docker Hub.

- Docker Hub remains the "production" registry (k8s pulls from there)
- Local registry becomes the "CI cache" registry

### 2. Merge Tests into Dockerfile (Frontend)

Replace the separate "Run frontend tests" Woodpecker step + Docker build with a single multi-stage Dockerfile build:

- `deps` stage: `npm ci` (cached if `package-lock.json` unchanged)
- `test` stage: runs `vitest run` (fails the build if tests fail)
- `builder` stage: runs `vite build`
- Final stage: nginx serving built assets

This eliminates the double `npm ci`.

### 3. Merge Tests into Dockerfile (Backend)

Replace the separate "Cache test image" + "Run backend tests" + "Build API image" steps with a single Docker build:

- `builder` stage: `pip install -r requirements.txt` (cached if requirements unchanged)
- `runtime-base` stage: system dependencies
- `test` stage: runs `pytest tests/ -x -q` (fails the build if tests fail)
- Final stage: production image with app code

This eliminates the double pip install and double image build.

### 4. Simplified Pipeline Steps

**Frontend: 5 steps to 4**

Before: clone, npm ci + vitest, Kaniko build (npm ci again), deploy, verify
After: clone, Kaniko build (npm ci + vitest + vite build), deploy, verify

**Backend: 5 steps to 4**

Before: clone, build :test image (pip install), pytest, build final image (pip install again), deploy + verify
After: clone, build image (pip install + pytest + final), deploy, verify

## Expected Impact

- `npm ci` runs once instead of twice per frontend build
- `pip install` runs once instead of twice per backend build
- Docker layer cache from local network instead of Docker Hub
- On cache hit (deps unchanged): dependency install steps are completely skipped (cached layers)

## Constraints

- Tests must always run (gate deployment)
- Final images must still be pushed to Docker Hub for k8s to pull
- Local registry is HTTP-only at `10.0.20.10:5000`