infra/docs/plans/2026-05-07-forgejo-registry-consolidation-plan.md

# Forgejo Registry Consolidation — Plan

**Date**: 2026-05-07
**Status**: Approved — execution in progress (Phase 0)
**Design**: `2026-05-07-forgejo-registry-consolidation-design.md`

This is the implementation roadmap for migrating off `registry-private`
onto Forgejo's OCI registry. See the design doc for problem
statement and rationale. Execution spans 5 phases over ≥3 weeks.

## Phase 0 — Prepare Forgejo (1 PR, no cutover risk)

| Task | File / artifact |
|---|---|
| Bump Forgejo memory request+limit 384Mi → 1Gi | `infra/stacks/forgejo/main.tf` |
| Add `FORGEJO__packages__ENABLED=true` and `FORGEJO__packages__CHUNKED_UPLOAD_PATH=/data/tmp/package-upload` env vars (defensive — already default in v11) | `infra/stacks/forgejo/main.tf` |
| Bump Forgejo PVC 5Gi → 15Gi, auto-resize cap 20Gi → 50Gi | `infra/stacks/forgejo/main.tf` |
| Bump ingress `max_body_size = "5g"` (wired into ingress_factory as a Buffering middleware) | `infra/stacks/forgejo/main.tf`, `infra/modules/kubernetes/ingress_factory/main.tf` |
| Create `cluster-puller` (read:package), `ci-pusher` (write:package), and a third `cleanup` PAT on `ci-pusher`; store PATs in Vault | runbook: `docs/runbooks/forgejo-registry-setup.md` |
| Extend `registry-credentials` Secret with 4th `auths` entry for `forgejo.viktorbarzin.me` | `infra/stacks/kyverno/modules/kyverno/registry-credentials.tf` |
| Add containerd `hosts.toml` entry redirecting `forgejo.viktorbarzin.me` → in-cluster Traefik LB `10.0.20.200` | `infra/stacks/infra/main.tf` cloud-init + new `infra/scripts/setup-forgejo-containerd-mirror.sh` for existing nodes |
| Forgejo retention CronJob (`0 4 * * *`, dry-run for first 7 days) | new `infra/stacks/forgejo/cleanup.tf` + `infra/stacks/forgejo/files/cleanup.sh` |
| Forgejo integrity probe CronJob (`*/15 * * * *`) | `infra/stacks/monitoring/modules/monitoring/main.tf` |
| Make existing alerts instance-aware so they cover both registries | `infra/stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl` |

**Smoke test (must pass before declaring Phase 0 done):**

- `docker login forgejo.viktorbarzin.me` succeeds.
- Push a hello-world image to `forgejo.viktorbarzin.me/viktor/smoketest:1` succeeds.
- `crictl pull forgejo.viktorbarzin.me/viktor/smoketest:1` from a k8s
  node succeeds, using the auto-synced `registry-credentials` Secret.
- A fresh namespace gets the cloned Secret with 4 `auths` entries.
- Delete the smoketest package via API.
- Forgejo integrity probe completes once and pushes metrics.

## Phase 1 — Source migration (parallel-safe, no production impact)

For each project the recipe is identical:

1. `git init` + push to `forgejo.viktorbarzin.me/viktor/<name>` —
   register in Woodpecker via OAuth.
2. Add `.woodpecker.yml` based on `payslip-ingest/.woodpecker.yml`.
   Push step uses `woodpeckerci/plugin-docker-buildx` with TWO
   `repo:` entries (dual-push).
3. Confirm first build pushes to BOTH registries.

Projects (bake clock starts at "all dual-push"):

| Project | Action |
|---|---|
| `claude-agent-service` | Extract from monorepo to Forgejo. New `.woodpecker.yml`. |
| `fire-planner` | Extract from monorepo to Forgejo. New `.woodpecker.yml`. |
| `wealthfolio-sync` | Extract from monorepo to Forgejo. New `.woodpecker.yml`. |
| `hmrc-sync` | Extract from monorepo to Forgejo. New `.woodpecker.yml`. |
| `freedify` | Push from monorepo to Forgejo. New `.woodpecker.yml`. (Upstream is gone.) |
| `payslip-ingest` | Already on Forgejo. Add second `repo:` entry to `.woodpecker.yml`. |
| `job-hunter` | Already on Forgejo. Add second `repo:` entry. |
| `beadboard` | Push to Forgejo. New `.woodpecker.yml`. Disable GHA workflow. **Don't archive GitHub yet** (deferred to Phase 3). |
| `claude-memory-mcp` | Push to Forgejo. New `.woodpecker.yml`. |
| `infra-ci` | Edit `.woodpecker/build-ci-image.yml` to dual-push. ALSO `docker save | gzip` to `/opt/registry/data/private/_breakglass/` on VM AND `/srv/nfs/forgejo-breakglass/` on NAS. Pin a `latest` symlink. |

Break-glass runbook (`docs/runbooks/forgejo-registry-breakglass.md`)
documents the recovery path.

## Phase 2 — Bake (≥14 days)

- No `image=` lines change. Pods still pull from
  `registry.viktorbarzin.me`.
- **Daily smoke check**: pull a recent image from Forgejo as
  `cluster-puller`, verify integrity (HEAD on manifest + each blob).
- **Bake exit criteria**:
  - Zero `RegistryManifestIntegrityFailure` alerts on Forgejo.
  - Zero `ContainerNearOOM` for the forgejo pod.
  - Retention CronJob has run ≥14 times successfully.
  - At least one full Sunday GC cycle has elapsed.
  - Switch retention CronJob to `DRY_RUN=false` on day 7, observe
    until day 14.

## Phase 3 — Cutover (one PR per project, single session)

Order = lowest blast radius first. Each step:
`image=` flip → `kubectl rollout restart` → verify pull from Forgejo.

1. `payslip-ingest` (`infra/stacks/payslip-ingest/main.tf`)
2. `job-hunter` (`infra/stacks/job-hunter/main.tf`)
3. `claude-agent-service` (`infra/stacks/claude-agent-service/main.tf`)
4. `fire-planner` (`infra/stacks/fire-planner/main.tf`)
5. `wealthfolio-sync` (`infra/stacks/wealthfolio/main.tf`)
6. `freedify` (`infra/stacks/freedify/factory/main.tf`)
7. `chrome-service` (`infra/stacks/chrome-service/main.tf`)
8. `beads-server` / `beadboard` (`infra/stacks/beads-server/main.tf`).
   Then `gh repo archive ViktorBarzin/beadboard`.
9. `infra-ci` — flip `image:` references in 4 `.woodpecker/*.yml`
   files in the infra repo. Verify next push to master applies cleanly.
10. `claude-memory-mcp` — update `CLAUDE.md` install instruction from
    `claude plugins install github:ViktorBarzin/claude-memory-mcp` to
    `claude plugins install https://forgejo.viktorbarzin.me/viktor/claude-memory-mcp.git`.
    `gh repo archive ViktorBarzin/claude-memory-mcp`.

## Phase 4 — Decommission

| Step | File / location |
|---|---|
| Stop `registry-private` container on VM (10.0.20.10): edit `/opt/registry/docker-compose.yml`, comment out service, `docker compose up -d --remove-orphans`. (Manual SSH — cloud-init won't redeploy on TF apply per memory id=1078.) | live VM |
| Update cloud-init template to match the new compose file | `infra/stacks/infra/main.tf:288` |
| Delete `auths` entries for `registry.viktorbarzin.me` / `:5050` / `10.0.20.10:5050` from the dockerconfigjson | `infra/stacks/kyverno/modules/kyverno/registry-credentials.tf` |
| Drop `registry.viktorbarzin.me` and `10.0.20.10:5050` `hosts.toml` entries on each node + cloud-init template | `infra/stacks/infra/main.tf` cloud-init + ad-hoc script |
| After 1 week of no incidents, delete `/opt/registry/data/private/` blob storage on the VM (~2.6GB freed) | manual SSH |

## Phase 5 — Docs

In the same commit as the Phase 4 closing:

| Doc | Update |
|---|---|
| `docs/runbooks/registry-vm.md` | Note `registry-private` is gone; pull-through caches and break-glass tarballs only |
| `docs/runbooks/registry-rebuild-image.md` | Replaced by NEW `forgejo-registry-rebuild-image.md` |
| `docs/runbooks/forgejo-registry-rebuild-image.md` (NEW) | Forgejo PVC restore procedure |
| `docs/runbooks/forgejo-registry-breakglass.md` (NEW) | infra-ci tarball recovery |
| `docs/architecture/ci-cd.md` | Image registry section flips to Forgejo |
| `docs/architecture/monitoring.md` | Integrity probe target updated |
| `infra/.claude/CLAUDE.md` | Registry references updated |
| `CLAUDE.md` (monorepo root) | claude-memory-mcp install URL updated |
| `infra/.claude/reference/service-catalog.md` | Cross-reference checked |

## Critical files modified

| File | Phase | What |
|---|---|---|
| `infra/stacks/forgejo/main.tf` | 0 | Memory bump, packages env vars, PVC bump, ingress max_body_size |
| `infra/stacks/forgejo/cleanup.tf` (NEW) | 0 | Retention CronJob |
| `infra/stacks/forgejo/files/cleanup.sh` (NEW) | 0 | Retention script (mounted via ConfigMap) |
| `infra/modules/kubernetes/ingress_factory/main.tf` | 0 | Wire `max_body_size` into a Traefik Buffering middleware |
| `infra/stacks/kyverno/modules/kyverno/registry-credentials.tf` | 0 | Add 4th `auths` entry |
| `infra/stacks/infra/main.tf` | 0 + 4 | Containerd hosts.toml block (add Forgejo, later remove registry-private); compose template update |
| `infra/scripts/setup-forgejo-containerd-mirror.sh` (NEW) | 0 | One-shot rollout for existing nodes |
| `infra/stacks/monitoring/modules/monitoring/main.tf` | 0 | Forgejo integrity probe CronJob |
| `infra/stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl` | 0 | Make alerts instance-aware |
| `infra/stacks/monitoring/main.tf` | 0 | Plumb `forgejo_pull_token` into module |
| `infra/.woodpecker/build-ci-image.yml` | 1 | Dual-push to add Forgejo target + tarball break-glass |
| `<each-project>/.woodpecker.yml` | 1 | Dual-push (NEW for fire-planner, wealthfolio-sync, hmrc-sync, freedify, beadboard, claude-memory-mcp; EDIT for payslip-ingest, job-hunter, claude-agent-service) |
| `infra/.woodpecker/{default,drift-detection,build-cli}.yml` | 3 | Flip `image:` to Forgejo for infra-ci |
| `infra/stacks/{beads-server,chrome-service,claude-agent-service,fire-planner,freedify/factory,job-hunter,payslip-ingest,wealthfolio}/main.tf` | 3 | Flip `image =` to Forgejo |

## Verification

- **Push** (Phase 0/1): `docker push forgejo.viktorbarzin.me/viktor/<name>` visible in Forgejo Web UI under viktor/.
- **Pull** (Phase 0): `crictl pull forgejo.viktorbarzin.me/viktor/smoketest:1` succeeds with auto-synced Secret.
- **Dual-push** (Phase 1): every Woodpecker pipeline run pushes to BOTH endpoints — confirmed via HEAD checks on `<reg>:<sha>` for both.
- **Bake** (Phase 2): existing daily Forgejo `/api/healthz` external monitor stays green; integrity probe stays green; no `ContainerNearOOM` for forgejo pod.
- **Cutover** (Phase 3): `kubectl rollout status deploy/<svc> -n <ns>` succeeds. `kubectl describe pod` shows the image was pulled from `forgejo.viktorbarzin.me`.
- **Decommission** (Phase 4): `docker ps` on registry VM no longer shows `registry-private`. Brand-new namespace gets the Secret with only the Forgejo `auths` entry. Pull still works.