infra/docs/post-mortems/2026-04-19-registry-orphan-index.md

# Post-Mortem: Private Registry Orphan OCI-Index — Repeat Incident

| Field | Value |
|-------|-------|
| **Date** | 2026-04-19 (first occurrence 2026-04-13) |
| **Duration** | ~40 min of blocked CI each time; only detected via pipeline failures |
| **Severity** | SEV2 — all infra CI pipelines using `infra-ci:latest` failed (P366 → P376 all exit 126 "image can't be pulled") |
| **Affected Services** | Every Woodpecker pipeline that starts with `image: registry.viktorbarzin.me:5050/infra-ci:latest` — `default.yml`, `build-cli.yml`, `renew-tls.yml`, `drift-detection.yml`, `provision-user.yml`, `k8s-portal.yml`, `postmortem-todos.yml`, `issue-automation.yml`, `pve-nfs-exports-sync.yml` |
| **Status** | Hot fix green (three commits: `a05d63ee`, `6371e75e`, `c113be4d` — URL fix + rebuild). This doc captures the permanent fix landed in the same branch. |

## Summary

On 2026-04-19 ~09:00 UTC, every infra CI pipeline started failing at the
`clone` step with "image can't be pulled". The image in question — the CI
toolchain image `registry.viktorbarzin.me:5050/infra-ci:latest` — resolved
to an OCI image index whose `linux/amd64` platform manifest
(`sha256:98f718c8…`) and its in-toto attestation
(`sha256:27d5ab83…`) returned **HTTP 404** from the private registry.
The index record itself still existed — it's the children that had been
garbage-collected out from under it.

This is the **second identical incident**: the same failure mode occurred
on 2026-04-13 against a different image. Both times the immediate fix was
to rebuild the image from scratch; both times the root cause was left
unaddressed.

## Impact

- **User-facing**: all CI pipelines failed. No automated Terraform applies,
  no TLS renewal, no drift detection. Manual workflows (Woodpecker UI
  reruns) all failed with the same error.
- **Blast radius**: every pipeline that pulls `infra-ci`. Does NOT affect
  k8s workloads (those pull via containerd, which goes through the
  pull-through proxy on :5000/:5010 — a completely different code path).
- **Duration on 2026-04-19**: from first P366 failure to the hot-fix
  commit `c113be4d` — roughly 40 min. Pipelines that had already been
  triggered queued up until the rebuild restored `:latest`.
- **Data loss**: none. The registry has the index object; the child
  manifests are re-producible by rebuilding the source image.
- **Monitoring gap**: nothing alerted. The only signal was the individual
  pipeline failures from Woodpecker. No Prometheus alert fires on "the
  registry served a 404 for a tag that exists".

## Timeline (UTC, 2026-04-19)

| Time | Event |
|------|-------|
| ~09:00 | P366 (`default.yml` on master) fails with exit 126. |
| 09:00–11:00 | P367, P368, … P376 all fail with the same error. Nobody pages — there's no alert configured. |
| 11:15 | User notices and investigates: `skopeo inspect` reveals the missing platform manifest. |
| 11:20 | Hot fix phase begins: `a05d63ee` fixes a push-URL misalignment, `6371e75e` and `c113be4d` trigger a full rebuild. |
| 11:40 | Rebuild completes; `infra-ci:latest` resolves to a fresh, complete index. Pipelines green from P377 onward. |
| 11:45 | User requests a proper root-cause fix: "this is the second time — what's actually broken?" |
| 12:00 | Investigation begins (this document's work). |

## Root Cause Chain

```
[1] cleanup-tags.sh runs daily at 02:00 on the registry VM
 └─> For each repository, keeps the last 10 tags by mtime, rmtrees the rest.
     This walks `_manifests/tags/<tag>` directly, bypassing the registry API.
         │
         ├─> [2] Subtle on-disk asymmetry: a registry:2 tag rmtree removes
         │    BOTH the `_manifests/tags/<tag>/` dir AND — on 2.8.x — the
         │    per-repo revision-link files under
         │    `<repo>/_manifests/revisions/sha256/<child-digest>/link` for
         │    every child referenced by that tag's index. The raw blob data
         │    under `/var/lib/registry/docker/registry/v2/blobs/sha256/<.>/data`
         │    is NOT touched — GC owns that, and GC only runs Sunday.
         │
         ├─> [3] If ANOTHER tag's index still references one of those same
         │    children (common — successive rebuilds share layers), the child
         │    blob survives. But the revision-link is gone, so the registry
         │    API can no longer map `<repo>/manifests/sha256:<child>` back
         │    to the blob. HEAD → 404, even though the bytes are on disk.
         │    distribution/distribution#3324 is the upstream class of this bug.
         │
         └─> [4] Result: the surviving index (e.g. `infra-ci:5319f03e`) is
              intact on disk, its children's blob data files are intact on
              disk, but HEAD `/v2/infra-ci/manifests/sha256:98f718c8…`
              returns 404. The registry has the bytes, but cannot find them
              through the API because the per-repo link bridge is gone.

[pull] containerd resolves `infra-ci:latest`
         │
         ├─> GET /v2/infra-ci/manifests/latest → 200 OK, returns the index
         │
         └─> GET /v2/infra-ci/manifests/sha256:98f718c8… → 404 Not Found
              └─> containerd fails the pull with "manifest unknown"
                    └─> woodpecker exit 126
```

> **Detection-gotcha** uncovered 2026-04-19 while implementing
> `fix-broken-blobs.sh`: a scan that checks `/blobs/sha256/<child>/data` for
> presence is NOT equivalent to "can the registry serve this child?" The
> authoritative check is whether
> `<repo>/_manifests/revisions/sha256/<child>/link` exists. The script
> was rewritten to check the per-repo link file after the HTTP probe
> caught 38 real orphans the filesystem scan had reported clean.

## Why Existing Remediation Missed It

1. **`fix-broken-blobs.sh` only scans layer links.** The existing cron
   walks `_layers/sha256/` and removes link files whose blob `data` is
   missing. It does NOT inspect `_manifests/revisions/sha256/` to see
   whether an image-index's referenced children still exist. That's
   exactly the class of orphan this incident represents.
2. **`registry:2` image tag was floating.** `docker-compose.yml` pinned
   only to `registry:2`. Whatever Docker Inc. last rebuilt as
   "v2-current" was running, with no version pin. Any regression in
   the upstream walker would silently swap in.
3. **No integrity monitoring.** Prometheus alerted on cache hit rate
   and registry-down, but nothing probes "are the manifests the registry
   advertises actually fetchable?"
4. **CI pipeline didn't verify its own push.** `buildx --push` returns
   success as soon as it uploads. If a child blob upload 0-byted or
   the client disconnected mid-push (distinct from the GC mode but the
   same on-disk symptom), nothing would notice until the next pull.

## Permanent Fix — Three Phases

### Phase 1 — Detection (ship today)

1. **Post-push integrity check** in `.woodpecker/build-ci-image.yml`.
   After `build-and-push`, a new step walks the just-pushed manifest
   (and every child of an image index) and HEADs every referenced blob.
   Any non-200 fails the pipeline immediately, catching broken pushes at
   the source rather than leaking them to consumers.
2. **Prometheus alert `RegistryManifestIntegrityFailure`.** A new
   CronJob (`registry-integrity-probe`, every 15m, in the `monitoring`
   namespace) walks the private registry's catalog, HEADs every tag's
   manifest, follows each image index's children, and pushes
   `registry_manifest_integrity_failures` to Pushgateway. Accompanying
   alerts: `RegistryIntegrityProbeStale`, `RegistryCatalogInaccessible`.
3. **Post-mortem** — this document. Linked from
   `.claude/reference/service-catalog.md` via the new runbook.

### Phase 2 — Prevention

4. **Pin `registry:2` → `registry:2.8.3`** in
   `modules/docker-registry/docker-compose.yml` (all six registry
   services). Removes the floating-tag footgun.
5. **Extend `fix-broken-blobs.sh`** to scan every
   `_manifests/revisions/sha256/<digest>` that is an image index and
   flag children whose blob `data` file is missing. The script prints a
   loud WARNING per orphan; it does not auto-delete the index, because
   deleting a published image is a conscious decision, not an automated
   repair.

### Phase 3 — Recovery tooling

6. **Manual event trigger** on `build-ci-image.yml`. Rebuilds no longer
   need a cosmetic Dockerfile edit — POST to the Woodpecker API or
   click "Run manually" in the UI.
7. **Runbook** `docs/runbooks/registry-rebuild-image.md` — exact
   command sequence for the next time this happens, plus fallback paths.

## Out of Scope

- **Pull-through caches.** The DockerHub / GHCR mirrors on
  `:5000` / `:5010` are healthy (74.5% cache hit rate, no 404s). The
  orphan problem is private-registry-only. No changes to nginx or
  containerd `hosts.toml`.
- **Registry HA / replication.** Single-VM SPOF is a known
  architectural choice. Harbor or a replicated registry would solve
  more than this incident requires, at multi-day cost. Synology offsite
  snapshots already give RPO < 1 day.
- **Disabling `cleanup-tags.sh`.** Keeping storage bounded is still
  necessary; the fix is detection + rebuild, not "stop cleaning up".

## Lessons

- **Repeat incidents deserve root-cause work, not a third hot-fix.** The
  2026-04-13 incident was closed when CI turned green. Without a probe
  and without a scan for orphan indexes, the next incident was
  inevitable — and it happened six days later against a different image.
- **"No alert fired, so it wasn't detected" is a monitoring gap, not an
  outage feature.** The registry was serving 404s for 2+ hours before
  anyone noticed, because our only signal was "pipeline failures" and
  our eyes were elsewhere. The new probe closes that gap.
- **CI pipelines should verify their own output.** The `buildx --push`
  "success" exit code is not a guarantee of pulled-back integrity — as
  this incident proves. A 30-second post-push HEAD walk is cheap
  insurance.

## Related

- **Prior incident (same failure mode, different image)**: memory `709`
  / `710` — 2026-04-13.
- **Runbook**: `docs/runbooks/registry-rebuild-image.md` (new).
- **Hot-fix commits**: `a05d63ee`, `6371e75e`, `c113be4d`.
- **Upstream bug class**: `distribution/distribution#3324`.

## 2026-04-19 — Bulk cleanup sweep (beads code-8hk + code-jh3c)

Same failure class, broader scope. The `registry-integrity-probe`
surfaced 38 broken manifest references persisting after the 04-19
infra-ci fix. `beads-dispatcher` + `beads-reaper` CronJobs were stuck
`ImagePullBackOff` on `claude-agent-service:0c24c9b6` for >6h. All 34
affected `repo:tag` pairs were OCI indexes whose `linux/amd64` child
manifests were absent from blob storage (same orphan pattern).

**Action taken**:
1. Bumped `beads-server/main.tf` var default `claude_agent_service_image_tag`
   from `0c24c9b6` → `2fd7670d` (the canonical tag in
   `claude-agent-service/main.tf`), reused — same image already healthy
   on the registry. `scripts/tg apply` on `beads-server`. Deleted the
   stuck Jobs so new CronJob ticks could fire.
2. Enumerated 34 broken `(repo, tag, parent_digest)` triples via HTTP
   probe using `registry-probe-credentials` K8s Secret. Deleted each
   via `DELETE /v2/<repo>/manifests/<digest>` (33× 202, 1× 404 —
   claude-agent-service:latest pointed at an already-deleted digest).
3. Ran `docker exec registry-private /bin/registry garbage-collect
   /etc/docker/registry/config.yml` — reclaimed ~3GB of orphan blob
   storage.
4. Rebuilt the 3 in-use broken tags (all 3 OCI-index parents pointed
   at missing children, so no cached copies would survive pod
   reschedule):
   - `freedify:latest` / `freedify:c803de02` — built on registry VM
     directly (no CI pipeline exists for this image; python FastAPI).
   - `beadboard:17a38e43` / `beadboard:latest` — GHA
     `workflow_dispatch` failed at registry login (missing
     `REGISTRY_USERNAME`/`REGISTRY_PASSWORD` GH secrets). Built on
     registry VM directly as the fallback. GitHub secret gap is a
     follow-up — beads `code-8hk` notes it.
   - `priority-pass-backend:ae1420a0` / `priority-pass-frontend:ae1420a0`
     — Woodpecker pipeline #8 on repo 81. Pipeline `kubectl set image`'d
     the Deployment to `ae1420a0` (drift vs TF `v5`/`v8` defaults, but
     that drift is pre-existing, not introduced by this cleanup).
   - `wealthfolio-sync:latest` — **not rebuilt**. Monthly CronJob (next
     run 2026-05-01), no source tree or CI pipeline available in the
     monorepo; deferred for separate follow-up.

**Post-cleanup state**:
- Probe: 39 tags, 0 failures. `registry_manifest_integrity_failures{} = 0`.
- Alert `RegistryManifestIntegrityFailure` cleared (was firing for
  5h 32m).
- No `ImagePullBackOff` pods anywhere in the cluster.
- 28 of 34 deleted manifests were **dangling tags not referenced by any
  workload** — old `382d6b1*`, `v2`-`v7`, `yt-fallback`, etc. Safe
  deletes, no rebuilds needed.

**Permanent fix still in flight**: Phase 2/3 of this post-mortem
(post-push verification in CI, atomic `cleanup-tags.sh`) — not
addressed by this cleanup. The probe continues to be the
authoritative detector.