viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

add registry blob integrity checker to self-heal corrupted cache

Browse source 
The cleanup-tags.sh + garbage-collect cycle can delete blob data while
leaving _layers/ link files intact. The registry then returns HTTP 200
with 0 bytes for those layers, causing "unexpected EOF" on image pulls.

fix-broken-blobs.sh walks all repositories, checks each layer link
against actual blob data, and removes orphaned links so the registry
re-fetches from upstream on next pull.

Schedule: daily at 2:30am (after tag cleanup) and Sunday 3:30am
(after garbage collection). First run found 2335/2556 (91%) of
layer links were orphaned.
This commit is contained in:
Viktor Barzin 2026-03-29 22:31:39 +03:00
parent facf959ecf
commit dd461beb33
2 changed files with 66 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

7
stacks/infra/main.tf
View file

@ -270,6 +270,10 @@ module "docker-registry-template" {
format("echo %s | base64 -d > /opt/registry/cleanup-tags.sh && chmod +x /opt/registry/cleanup-tags.sh",
base64encode(file("${path.root}/../../modules/docker-registry/cleanup-tags.sh"))
),
# Write blob integrity checker
format("echo %s | base64 -d > /opt/registry/fix-broken-blobs.sh && chmod +x /opt/registry/fix-broken-blobs.sh",
base64encode(file("${path.root}/../../modules/docker-registry/fix-broken-blobs.sh"))
),
# Create systemd unit for docker compose
format("echo %s | base64 -d > /etc/systemd/system/docker-compose-registry.service",
base64encode(<<-UNIT
@ -304,6 +308,9 @@ UNIT
"( crontab -l 2>/dev/null; echo '25 3 * * 0 /usr/bin/docker exec registry-private registry garbage-collect -m /etc/docker/registry/config.yml >> /var/log/registry-gc.log 2>&1' ) | crontab -",
# Cron: tag cleanup (daily 2am, keep last 10 tags per image)
"( crontab -l 2>/dev/null; echo '0 2 * * * python3 /opt/registry/cleanup-tags.sh 10 >> /var/log/registry-cleanup.log 2>&1' ) | crontab -",
# Cron: blob integrity check (after GC on Sunday, and daily 2:30am after tag cleanup)
"( crontab -l 2>/dev/null; echo '30 3 * * 0 python3 /opt/registry/fix-broken-blobs.sh >> /var/log/registry-fix-blobs.log 2>&1' ) | crontab -",
"( crontab -l 2>/dev/null; echo '30 2 * * 1-6 python3 /opt/registry/fix-broken-blobs.sh >> /var/log/registry-fix-blobs.log 2>&1' ) | crontab -",
]
}