diff --git a/docs/post-mortems/2026-04-19-registry-orphan-index.md b/docs/post-mortems/2026-04-19-registry-orphan-index.md index db918e5b..da883760 100644 --- a/docs/post-mortems/2026-04-19-registry-orphan-index.md +++ b/docs/post-mortems/2026-04-19-registry-orphan-index.md @@ -60,17 +60,26 @@ unaddressed. └─> For each repository, keeps the last 10 tags by mtime, rmtrees the rest. This walks `_manifests/tags/` directly, bypassing the registry API. │ - ├─> [2] registry:2 garbage-collect runs weekly (Sun 03:25 for the - │ private registry). Walks live manifests through refcounts, but - │ distribution/distribution#3324 showed this walker has historical - │ bugs with OCI image-index children — it can decrement a shared - │ child's refcount below 1 and delete the blob even while the - │ index that references it is still referenced. + ├─> [2] Subtle on-disk asymmetry: a registry:2 tag rmtree removes + │ BOTH the `_manifests/tags//` dir AND — on 2.8.x — the + │ per-repo revision-link files under + │ `/_manifests/revisions/sha256//link` for + │ every child referenced by that tag's index. The raw blob data + │ under `/var/lib/registry/docker/registry/v2/blobs/sha256/<.>/data` + │ is NOT touched — GC owns that, and GC only runs Sunday. │ - └─> [3] Result: the `infra-ci:latest` index is intact - (`_manifests/revisions/sha256//data` present on disk), but - its `.manifests[0].digest` — the `linux/amd64` child — points - to a `blobs/sha256/98/98f718c8…/` whose `data` file is gone. + ├─> [3] If ANOTHER tag's index still references one of those same + │ children (common — successive rebuilds share layers), the child + │ blob survives. But the revision-link is gone, so the registry + │ API can no longer map `/manifests/sha256:` back + │ to the blob. HEAD → 404, even though the bytes are on disk. + │ distribution/distribution#3324 is the upstream class of this bug. + │ + └─> [4] Result: the surviving index (e.g. `infra-ci:5319f03e`) is + intact on disk, its children's blob data files are intact on + disk, but HEAD `/v2/infra-ci/manifests/sha256:98f718c8…` + returns 404. The registry has the bytes, but cannot find them + through the API because the per-repo link bridge is gone. [pull] containerd resolves `infra-ci:latest` │ @@ -81,6 +90,14 @@ unaddressed. └─> woodpecker exit 126 ``` +> **Detection-gotcha** uncovered 2026-04-19 while implementing +> `fix-broken-blobs.sh`: a scan that checks `/blobs/sha256//data` for +> presence is NOT equivalent to "can the registry serve this child?" The +> authoritative check is whether +> `/_manifests/revisions/sha256//link` exists. The script +> was rewritten to check the per-repo link file after the HTTP probe +> caught 38 real orphans the filesystem scan had reported clean. + ## Why Existing Remediation Missed It 1. **`fix-broken-blobs.sh` only scans layer links.** The existing cron diff --git a/modules/docker-registry/fix-broken-blobs.sh b/modules/docker-registry/fix-broken-blobs.sh index ba7a0191..dfc21a22 100644 --- a/modules/docker-registry/fix-broken-blobs.sh +++ b/modules/docker-registry/fix-broken-blobs.sh @@ -118,22 +118,33 @@ for registry_name in sorted(os.listdir(BASE)): total_index_scanned += 1 + # Per-repo revision links — serving a child manifest via the API + # requires /_manifests/revisions/sha256//link + # to exist. The blob data alone is not enough: cleanup-tags.sh + # rmtrees tag dirs (which on 2.8.x also orphans the per-repo + # revision links for index children), while the upstream blob + # data survives in /blobs/. That's exactly the 2026-04-19 + # failure mode — the probe sees 404 even though the blob file + # is still on disk. + revisions_root = os.path.dirname(root) # …/_manifests/revisions for child in manifest.get("manifests", []): child_digest = child.get("digest", "") if not child_digest.startswith("sha256:"): continue child_hex = child_digest[len("sha256:"):] - child_blob = os.path.join(blobs_root, child_hex[:2], child_hex, "data") - if os.path.isfile(child_blob): + child_link = os.path.join(revisions_root, "sha256", child_hex, "link") + if os.path.isfile(child_link): continue platform = child.get("platform", {}) arch = platform.get("architecture", "?") os_ = platform.get("os", "?") + child_blob = os.path.join(blobs_root, child_hex[:2], child_hex, "data") + blob_state = "blob-data-present" if os.path.isfile(child_blob) else "blob-data-gone" print( f"WARNING [{registry_name}/{repo}] ORPHAN INDEX: " f"{digest_dir[:12]} references missing child {child_hex[:12]} " - f"({arch}/{os_}) — rebuild required, will not auto-repair" + f"({arch}/{os_}, {blob_state}) — registry returns 404, rebuild required" ) total_index_orphans += 1