infra/stacks/kyverno/modules/kyverno/registry-credentials.tf
Viktor Barzin 5d22b449f9 [forgejo] Phase 0 of registry consolidation: prepare Forgejo OCI registry
Stage 1 of moving private images off the registry:2 container at
registry.viktorbarzin.me:5050 (which has hit distribution#3324 corruption
3x in 3 weeks) onto Forgejo's built-in OCI registry. No cutover risk —
pods still pull from the existing registry until Phase 3.

What changes:
* Forgejo deployment: memory 384Mi→1Gi, PVC 5Gi→15Gi (cap 50Gi).
  Explicit FORGEJO__packages__ENABLED + CHUNKED_UPLOAD_PATH (defensive,
  v11 default-on).
* ingress_factory: max_body_size variable was declared but never wired
  in after the nginx→Traefik migration. Now creates a per-ingress
  Buffering middleware when set; default null = no limit (preserves
  existing behavior). Forgejo ingress sets max_body_size=5g to allow
  multi-GB layer pushes.
* Cluster-wide registry-credentials Secret: 4th auths entry for
  forgejo.viktorbarzin.me, populated from Vault secret/viktor/
  forgejo_pull_token (cluster-puller PAT, read:package). Existing
  Kyverno ClusterPolicy syncs cluster-wide — no policy edits.
* Containerd hosts.toml redirect: forgejo.viktorbarzin.me → in-cluster
  Traefik LB 10.0.20.200 (avoids hairpin NAT for in-cluster pulls).
  Cloud-init for new VMs + scripts/setup-forgejo-containerd-mirror.sh
  for existing nodes.
* Forgejo retention CronJob (0 4 * * *): keeps newest 10 versions per
  package + always :latest. First 7 days dry-run (DRY_RUN=true);
  flip the local in cleanup.tf after log review.
* Forgejo integrity probe CronJob (*/15): same algorithm as the
  existing registry-integrity-probe. Existing Prometheus alerts
  (RegistryManifestIntegrityFailure et al) made instance-aware so
  they cover both registries during the bake.
* Docs: design+plan in docs/plans/, setup runbook in docs/runbooks/.

Operational note — the apply order is non-trivial because the new
Vault keys (forgejo_pull_token, forgejo_cleanup_token,
secret/ci/global/forgejo_*) must exist BEFORE terragrunt apply in the
kyverno + monitoring + forgejo stacks. The setup runbook documents
the bootstrap sequence.

Phase 1 (per-project dual-push pipelines) follows in subsequent
commits. Bake clock starts when the last project goes dual-push.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 15:51:34 +00:00

141 lines
4.3 KiB
HCL

# =============================================================================
# Private Docker Registry Credentials — Auto-sync to all namespaces
# =============================================================================
# Source secret in kyverno namespace, cloned by ClusterPolicy into every NS.
# Pods use imagePullSecrets: [{name: registry-credentials}] to pull from
# registry.viktorbarzin.me (or 10.0.20.10:5050 internally).
data "vault_kv_secret_v2" "viktor" {
mount = "secret"
name = "viktor"
}
resource "kubernetes_secret" "registry_credentials" {
metadata {
name = "registry-credentials"
namespace = kubernetes_namespace.kyverno.metadata[0].name
}
type = "kubernetes.io/dockerconfigjson"
data = {
".dockerconfigjson" = jsonencode({
auths = {
"registry.viktorbarzin.me" = {
auth = base64encode("${data.vault_kv_secret_v2.viktor.data["registry_user"]}:${data.vault_kv_secret_v2.viktor.data["registry_password"]}")
}
"registry.viktorbarzin.me:5050" = {
auth = base64encode("${data.vault_kv_secret_v2.viktor.data["registry_user"]}:${data.vault_kv_secret_v2.viktor.data["registry_password"]}")
}
"10.0.20.10:5050" = {
auth = base64encode("${data.vault_kv_secret_v2.viktor.data["registry_user"]}:${data.vault_kv_secret_v2.viktor.data["registry_password"]}")
}
# Forgejo OCI registry — read-only PAT for the cluster-puller service
# account user. Pushes go through ci-pusher (separate PAT in Vault
# secret/ci/global, surfaced to Woodpecker).
"forgejo.viktorbarzin.me" = {
auth = base64encode("cluster-puller:${data.vault_kv_secret_v2.viktor.data["forgejo_pull_token"]}")
}
}
})
}
}
# Grant Kyverno controllers permission to manage Secrets (needed for generate clone rules)
resource "kubernetes_cluster_role" "kyverno_secret_manager" {
metadata {
name = "kyverno:secret-manager"
labels = {
"app.kubernetes.io/instance" = "kyverno"
}
}
rule {
api_groups = [""]
resources = ["secrets"]
verbs = ["get", "list", "watch", "create", "update", "patch", "delete"]
}
}
resource "kubernetes_cluster_role_binding" "kyverno_admission_secret_manager" {
metadata {
name = "kyverno:admission-controller:secret-manager"
}
role_ref {
api_group = "rbac.authorization.k8s.io"
kind = "ClusterRole"
name = kubernetes_cluster_role.kyverno_secret_manager.metadata[0].name
}
subject {
kind = "ServiceAccount"
name = "kyverno-admission-controller"
namespace = "kyverno"
}
}
resource "kubernetes_cluster_role_binding" "kyverno_background_secret_manager" {
metadata {
name = "kyverno:background-controller:secret-manager"
}
role_ref {
api_group = "rbac.authorization.k8s.io"
kind = "ClusterRole"
name = kubernetes_cluster_role.kyverno_secret_manager.metadata[0].name
}
subject {
kind = "ServiceAccount"
name = "kyverno-background-controller"
namespace = "kyverno"
}
}
resource "kubernetes_manifest" "sync_registry_credentials" {
manifest = {
apiVersion = "kyverno.io/v1"
kind = "ClusterPolicy"
metadata = {
name = "sync-registry-credentials"
}
spec = {
rules = [
{
name = "sync-registry-secret"
match = {
any = [
{
resources = {
kinds = ["Namespace"]
}
}
]
}
exclude = {
any = [
{
resources = {
namespaces = ["kube-system", "kube-public", "kube-node-lease"]
}
}
]
}
generate = {
apiVersion = "v1"
kind = "Secret"
name = "registry-credentials"
namespace = "{{request.object.metadata.name}}"
synchronize = true
clone = {
namespace = "kyverno"
name = "registry-credentials"
}
}
}
]
}
}
depends_on = [
helm_release.kyverno,
kubernetes_secret.registry_credentials,
kubernetes_cluster_role_binding.kyverno_admission_secret_manager,
kubernetes_cluster_role_binding.kyverno_background_secret_manager,
]
}