upgrades: fix hourly gotenberg error + cap update notifications at weekly
All checks were successful
ci/woodpecker/push/default Pipeline was successful

Viktor was getting upgrade-error Slack messages every hour and wants
update notifications at most weekly. Root cause of the errors: Keel kept
trying to roll gotenberg 8.25->8.25.1 in paperless-ngx but kyverno's
require-trusted-registries denied it — gotenberg/* (and apache/*, which
tika will hit next) were never allowlisted, and Keel's Slack notifier at
info level re-posted the identical failure to #general on every hourly
poll since Jun 28.

Changes: allowlist gotenberg/* + apache/* so the patch applies cleanly;
disable Keel's direct Slack notifier and replace failure visibility with
a KeelUpdateFailing Loki-ruler alert (alert-on-change: one notification
plus the daily digest, never an hourly drip); remove diun's Slack
notifier whose default message @channel-pinged #image-updates for every
new upstream tag every 6h (the n8n upgrade-agent webhook feed is
untouched). The k8s upgrade report is already weekly (Mon 06:07 UTC).
Paperless-ngx itself stays paused (keel policy=never, user-managed) while
the ingest runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-07-02 07:16:50 +00:00
parent 5d5d9752cb
commit a64d2ba2b9
5 changed files with 39 additions and 25 deletions

View file

@ -38,7 +38,7 @@ resource "helm_release" "keel" {
chart = "keel"
# Latest stable per `helm search repo keel/keel -l` 2026-05-16
# (app version 0.21.1). 1.0.6 doesn't exist verify before bumping.
version = "1.2.0"
version = "1.2.0"
# Atomic mitigates partial-deploy state. Keel itself is exempt from
# auto-update (Kyverno mutate excludes the keel namespace), so it only
@ -80,16 +80,15 @@ resource "helm_release" "keel" {
persistence = {
enabled = false
}
# Slack notifications: post every rollout to the configured channel.
# Bot token from Vault (secret/viktor -> slack_bot_token). The Keel
# chart sets SLACK_BOT_TOKEN, SLACK_CHANNELS, etc. on the deployment
# from these values.
# Direct Slack notifications DISABLED (2026-07-02): at notificationLevel
# info Keel posted every rollout event to #general, and a stuck update
# (gotenberg blocked by require-trusted-registries) re-posted the same
# failure EVERY HOURLY POLL for days. Failure visibility now comes from
# the KeelUpdateFailing Loki-ruler alert (stacks/monitoring loki.tf),
# which rides the alert-on-change routing: one Slack notification plus
# the daily digest never an hourly drip.
slack = {
enabled = true
botToken = data.vault_kv_secret_v2.viktor.data["slack_bot_token"]
channel = "general"
# No approval flow opt-out-pure means everything auto-rolls.
# If we ever introduce gated rollouts, set approvalsChannel here.
enabled = false
}
# Keel uses each watched Deployment's own imagePullSecrets to query
# its registry. Forgejo creds (`registry-credentials`) are auto-synced