monitoring: consolidate all Slack alerting to #alerts, abandon #security
Some checks are pending
ci/woodpecker/push/default Pipeline is running

The dedicated #security Slack channel was unreachable: the shared incoming
webhook (Vault secret/viktor -> alertmanager_slack_api_url) belongs to a
Slack app that isn't a member of #security, so any channel override on it
returns HTTP 404 channel_not_found. The goldmane-edges-digest was silently
failing for that reason.

Per request ("dump the security channel, post in an existing one"), route
everything to #alerts instead:
- alertmanager slack-security receiver -> #alerts (keeps its [SECURITY/<sev>]
  title styling so security-lane alerts still stand out in the shared channel)
- goldmane-edges-digest CronJob SLACK_CHANNEL -> #alerts (comment only; value
  was already switched and applied last change)
- AggregatorDown / DigestFailing alert summaries reworded to say #alerts
- docs swept (security.md, monitoring.md, ADR-0014, goldmane runbook,
  .claude/CLAUDE.md, service-catalog, CONTEXT.md) to drop the
  "invite the app / flip back to #security" caveats and state the
  #security abandonment + #alerts consolidation as the current routing.

Monitoring stack applied (alertmanager rolled, live config verified:
slack-security channel is now #alerts).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-06-26 13:29:44 +00:00
parent 196d0db4bd
commit fd33d1a447
9 changed files with 32 additions and 28 deletions

View file

@ -456,14 +456,11 @@ resource "kubernetes_cron_job_v1" "digest" {
}
env {
name = "SLACK_CHANNEL"
# The shared alertmanager_slack_api_url incoming webhook's Slack
# app is NOT a member of #security, so overriding the channel to
# it returns HTTP 404 channel_not_found (verified 2026-06-25).
# alertmanager's own slack-security receiver shares this webhook
# and almost certainly hits the same wall. Post to #alerts (the
# webhook's working channel, same as alert-digest) until the app
# is invited to #security, then flip this back. See
# docs/runbooks/goldmane-flow-trail.md.
# Posts to #alerts. The dedicated #security channel was abandoned
# 2026-06-25 the shared alertmanager_slack_api_url webhook's
# Slack app isn't a member of it (channel override 404s), so all
# Slack (incl. alertmanager's security-lane alerts) consolidated
# to #alerts. See docs/runbooks/goldmane-flow-trail.md.
value = "#alerts"
}