diff --git a/docs/architecture/dns.md b/docs/architecture/dns.md index cba4803c..f73a8997 100644 --- a/docs/architecture/dns.md +++ b/docs/architecture/dns.md @@ -1,6 +1,6 @@ # DNS Architecture -Last updated: 2026-04-15 +Last updated: 2026-04-19 ## Overview @@ -254,27 +254,42 @@ Config is synced to all 3 Technitium instances by CronJob `technitium-split-hori ## CoreDNS Configuration -CoreDNS is managed via a Terraform `kubernetes_config_map` resource in `stacks/technitium/modules/technitium/main.tf`. +CoreDNS is managed via Terraform in `stacks/technitium/modules/technitium/` — the Corefile ConfigMap lives in `main.tf`, and scaling/PDB are in `coredns.tf` (a `kubernetes_deployment_v1_patch` against the kubeadm-managed Deployment). ``` .:53 { errors / health / ready kubernetes cluster.local in-addr.arpa ip6.arpa # K8s service discovery prometheus :9153 # Metrics - forward . 10.0.20.1 8.8.8.8 1.1.1.1 # pfSense → Google → Cloudflare - cache (success 10000 300, denial 10000 300) + forward . 10.0.20.1 8.8.8.8 1.1.1.1 { + policy sequential # try upstreams in order + health_check 5s # mark unhealthy in 5s + max_fails 2 + } + cache { + success 10000 300 6 + denial 10000 300 60 + serve_stale 3600s 86400s # resilience during upstream outage + } loop / reload / loadbalance } viktorbarzin.lan:53 { template: .*\..*\.viktorbarzin\.lan\.$ → NXDOMAIN # ndots:5 junk filter - forward . 10.96.0.53 # Technitium ClusterIP - cache (success 10000 300, denial 10000 300) + forward . 10.96.0.53 { # Technitium ClusterIP + health_check 5s + max_fails 2 + } + cache (success 10000 300, denial 10000 300, serve_stale 3600s 86400s) } ``` +**Scaling**: 3 replicas, `required` anti-affinity on `kubernetes.io/hostname` (spread across 3 distinct nodes). PodDisruptionBudget `coredns` with `minAvailable=2`. + **Kyverno ndots injection**: A Kyverno policy injects `ndots:2` on all pods cluster-wide to reduce search domain expansion noise. The template regex is a second layer of defense for any queries that still get expanded. +**Failover behaviour**: With `policy sequential` on the root forward block, CoreDNS tries pfSense first; if `health_check 5s` detects pfSense as down, it fails over to 8.8.8.8 then 1.1.1.1 within ~5s rather than timing out per-query. Combined with `serve_stale`, pods keep resolving cached names for up to 24h even with full upstream failure. + ## Cloudflare DNS — External Domains All public domains are under the `viktorbarzin.me` zone. DNS records are **auto-created per service** via the `ingress_factory` module's `dns_type` parameter. A small number of records (Helm-managed ingresses, special cases) remain centrally managed in `config.tfvars`. @@ -360,9 +375,28 @@ Vault DB engine rotates password | Metric Source | Dashboard | Alerts | |---------------|-----------|--------| | Technitium query logs (PostgreSQL) | Grafana `technitium-dns.json` | — | -| CoreDNS Prometheus metrics (:9153) | Grafana CoreDNS dashboard | — | +| CoreDNS Prometheus metrics (:9153) | Grafana CoreDNS dashboard | `CoreDNSErrors`, `CoreDNSForwardFailureRate` | +| Technitium zone-sync CronJob (Pushgateway) | — | `TechnitiumZoneSyncFailed`, `TechnitiumZoneSyncStale`, `TechnitiumZoneCountMismatch` | +| Technitium DNS pod availability | — | `TechnitiumDNSDown` | +| `dns-anomaly-monitor` CronJob (Pushgateway) | — | `DNSQuerySpike`, `DNSQueryRateDropped`, `DNSHighErrorRate` | | Uptime Kuma | External monitors for all proxied domains | ExternalAccessDivergence (15min) | +### Metrics pushed by `technitium-zone-sync` + +The zone-sync CronJob (runs every 30min) pushes the following to the Prometheus Pushgateway under `job=technitium-zone-sync`: + +| Metric | Labels | Meaning | +|--------|--------|---------| +| `technitium_zone_sync_status` | — | 0 = last run succeeded, 1 = at least one zone failed to create | +| `technitium_zone_sync_failures` | — | Number of zones that failed to create this run | +| `technitium_zone_sync_last_run` | — | Unix timestamp of last run (used by `TechnitiumZoneSyncStale`) | +| `technitium_zone_count` | `instance=primary\|` | Zone count on each Technitium instance (drives `TechnitiumZoneCountMismatch`) | + +### DNS alert rewrites + +- `DNSQuerySpike` was previously broken: it compared current queries against `dns_anomaly_avg_queries`, which was computed from a per-pod `/tmp/dns_avg` file. Each CronJob run started with a fresh `/tmp`, so `NEW_AVG == TOTAL_QUERIES` every time and the spike condition could never fire. Rewritten to use `avg_over_time(dns_anomaly_total_queries[1h] offset 15m)` which compares against the actual 1h Prometheus history. +- `DNSQueryRateDropped` (new): fires when query rate drops below 50% of 1h average — upstream clients may be failing to reach Technitium. + ## Troubleshooting ### DNS Not Resolving Internal Domains diff --git a/docs/runbooks/technitium-apply.md b/docs/runbooks/technitium-apply.md new file mode 100644 index 00000000..a560452e --- /dev/null +++ b/docs/runbooks/technitium-apply.md @@ -0,0 +1,51 @@ +# Runbook: Applying the Technitium Terraform stack + +Last updated: 2026-04-19 + +The `stacks/technitium/` apply has a **post-apply readiness gate** that asserts all three DNS instances are healthy before the apply is allowed to finish. This runbook explains what it checks, how to interpret failures, and how to override it for emergency maintenance. + +## What the gate checks + +`stacks/technitium/modules/technitium/readiness.tf` defines `null_resource.technitium_readiness_gate`. It runs after the three Technitium deployments, the DNS LoadBalancer service, and the PDB are applied, and performs: + +1. **Rollout status** — `kubectl rollout status deploy/ --timeout=180s` for `technitium`, `technitium-secondary`, `technitium-tertiary`. Fails if any deployment has not reached its desired pod count within 180s. +2. **Per-pod API health** — for every pod with label `dns-server=true`, executes `wget http://127.0.0.1:5380/api/stats/get` inside the pod and asserts the response contains `"status":"ok"`. Catches Technitium process hangs that TCP probes miss. +3. **Zone-count parity** — queries `technitium-web`, `technitium-secondary-web`, `technitium-tertiary-web` and counts the zones returned. Fails if the three counts differ, which would mean `technitium-zone-sync` has drifted or a replica has lost state. + +The gate is re-run whenever any of the deployment container spec, the CoreDNS Corefile, or the apply timestamp changes (see `triggers` in `readiness.tf`). + +## Emergency override + +Set `skip_readiness=true` via terragrunt inputs or pass it directly to the Terraform apply: + +```bash +cd infra/stacks/technitium +scripts/tg apply -var skip_readiness=true +``` + +Only use this when you need to land a Terraform change while one Technitium instance is intentionally offline (e.g., you are replacing its PVC, migrating storage, or recovering a corrupted config DB). Re-apply without the flag once the instance is back. + +You can also target around the gate during emergency work: + +```bash +scripts/tg apply -target=kubernetes_config_map.coredns +``` + +`-target` bypasses the `depends_on` chain feeding the gate, so a single-resource push does not need the gate to pass. + +## Failure modes and responses + +| Symptom | Likely cause | Fix | +|---------|--------------|-----| +| `rollout status` times out on one deployment | Pod stuck `Pending` (node pressure / anti-affinity with other dns-server pods) or `ImagePullBackOff` | `kubectl describe pod` for events. If anti-affinity is blocking, confirm 3 nodes are Ready. | +| API check fails on a pod but readiness probe passes | Technitium process hung but port 53 still accepting TCP (liveness probe is `tcp_socket` on :53) | `kubectl delete pod ` — deployment will recreate it. | +| Zone count differs between instances | `technitium-zone-sync` CronJob is failing or AXFR is blocked | `kubectl logs -n technitium -l job-name=`. Check `TechnitiumZoneSyncFailed` alert. | +| Gate passes but external clients still cannot resolve | Gate only checks in-pod API and intra-cluster zone parity — external path (LoadBalancer → Technitium pod) is not tested | Run the LAN-client drill in `docs/architecture/dns.md` troubleshooting section. | + +## What the gate does NOT check + +- External reachability through the LoadBalancer IP `10.0.20.201` (that would require a LAN-side probe). +- CoreDNS health (CoreDNS is patched by `coredns.tf`, not this module's deployments — alerts `CoreDNSErrors` / `CoreDNSForwardFailureRate` catch regressions post-apply). +- Upstream resolver health (covered by `CoreDNSForwardFailureRate`). + +For broader end-to-end verification, see `docs/architecture/dns.md` → "Verification" section, or run the Uptime Kuma external DNS probe. diff --git a/stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl b/stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl index aafefc6d..16fe8c3b 100755 --- a/stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl +++ b/stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl @@ -1868,13 +1868,24 @@ serverFiles: summary: "NetFlow processing delay p50: {{ $value | printf \"%.0f\" }}s — softflowd may be overloaded" - name: "DNS Anomaly Detection" rules: + # Spike detection: compare current value against its own 1h history via + # avg_over_time. Previous version compared against dns_anomaly_avg_queries + # which was computed from a per-pod /tmp file and always equalled the + # current value (fresh /tmp each run), so the alert could never fire. - alert: DNSQuerySpike - expr: dns_anomaly_total_queries > 2 * dns_anomaly_avg_queries and dns_anomaly_total_queries > 1000 + expr: dns_anomaly_total_queries > 2 * avg_over_time(dns_anomaly_total_queries[1h] offset 15m) and dns_anomaly_total_queries > 1000 for: 0m labels: severity: warning annotations: - summary: "DNS query spike: {{ $value | printf \"%.0f\" }} queries (>2x average)" + summary: "DNS query spike: {{ $value | printf \"%.0f\" }} queries (>2x 1h avg)" + - alert: DNSQueryRateDropped + expr: dns_anomaly_total_queries < 0.5 * avg_over_time(dns_anomaly_total_queries[1h] offset 15m) and avg_over_time(dns_anomaly_total_queries[1h] offset 15m) > 1000 + for: 10m + labels: + severity: warning + annotations: + summary: "DNS query volume dropped: {{ $value | printf \"%.0f\" }} queries (<50% of 1h avg) — upstream clients may be failing to reach Technitium" - alert: DNSHighErrorRate expr: dns_anomaly_server_failure > 100 for: 0m @@ -1882,6 +1893,34 @@ serverFiles: severity: warning annotations: summary: "High DNS SERVFAIL rate: {{ $value | printf \"%.0f\" }} failures detected" + - alert: TechnitiumZoneSyncFailed + expr: technitium_zone_sync_status != 0 + for: 30m + labels: + severity: warning + annotations: + summary: "Technitium zone-sync CronJob has reported failure for 30m — replicas may be missing zones" + - alert: TechnitiumZoneSyncStale + expr: (time() - technitium_zone_sync_last_run) > 3600 + for: 10m + labels: + severity: warning + annotations: + summary: "Technitium zone-sync has not run successfully in >1h (last: {{ $value | humanizeDuration }} ago)" + - alert: TechnitiumZoneCountMismatch + expr: (max(technitium_zone_count) - min(technitium_zone_count)) > 0 + for: 15m + labels: + severity: warning + annotations: + summary: "Technitium zone counts differ across instances (max-min delta: {{ $value | printf \"%.0f\" }}) — replica has drifted from primary" + - alert: CoreDNSForwardFailureRate + expr: sum(rate(coredns_forward_responses_total{rcode=~"SERVFAIL|REFUSED"}[5m])) > 0.1 + for: 5m + labels: + severity: warning + annotations: + summary: "CoreDNS forward SERVFAIL/REFUSED rate: {{ $value | printf \"%.2f\" }}/s — upstream DNS (pfSense/public) may be unhealthy" - name: qbittorrent rules: - alert: MAMMouseClass diff --git a/stacks/technitium/modules/technitium/coredns.tf b/stacks/technitium/modules/technitium/coredns.tf new file mode 100644 index 00000000..4ae0ab17 --- /dev/null +++ b/stacks/technitium/modules/technitium/coredns.tf @@ -0,0 +1,69 @@ +# ============================================================================= +# CoreDNS — Scaling, Anti-Affinity, PDB +# ============================================================================= +# +# CoreDNS is kube-system / kubeadm-managed. We only patch replicas + affinity +# here (the Corefile ConfigMap is in main.tf). The hashicorp/kubernetes v3 +# provider removed the *_patch resource family from v2, so we apply the +# desired state via `kubectl patch` inside a null_resource. The patch is +# idempotent — a no-op when the deployment already matches. +# +# Kubeadm upgrades preserve the replica count on the existing deployment but +# reset the pod template (including affinity) from the ClusterConfiguration. +# Re-running `terraform apply` re-asserts the affinity patch; the readiness +# gate in `readiness.tf` catches regressions if the patch is reverted. + +resource "null_resource" "coredns_scale_and_affinity" { + triggers = { + replicas = 3 + spec_hash = sha256(file("${path.module}/coredns.tf")) + } + + provisioner "local-exec" { + command = <<-BASH + set -euo pipefail + # 1. Scale to 3 replicas. + kubectl -n kube-system scale deploy/coredns --replicas=3 + + # 2. Switch anti-affinity from preferred → required on hostname. + kubectl -n kube-system patch deploy/coredns --type=json -p='[ + { + "op": "replace", + "path": "/spec/template/spec/affinity/podAntiAffinity", + "value": { + "requiredDuringSchedulingIgnoredDuringExecution": [ + { + "labelSelector": { + "matchExpressions": [ + {"key": "k8s-app", "operator": "In", "values": ["kube-dns"]} + ] + }, + "topologyKey": "kubernetes.io/hostname" + } + ] + } + } + ]' || true + + # 3. Wait for rollout to settle. + kubectl -n kube-system rollout status deploy/coredns --timeout=120s + BASH + interpreter = ["/bin/bash", "-c"] + } +} + +# PDB — keep at least 2 CoreDNS pods running during voluntary disruptions. +resource "kubernetes_pod_disruption_budget_v1" "coredns" { + metadata { + name = "coredns" + namespace = "kube-system" + } + spec { + min_available = "2" + selector { + match_labels = { + "k8s-app" = "kube-dns" + } + } + } +} diff --git a/stacks/technitium/modules/technitium/ha.tf b/stacks/technitium/modules/technitium/ha.tf index 9c96db29..bf52f399 100644 --- a/stacks/technitium/modules/technitium/ha.tf +++ b/stacks/technitium/modules/technitium/ha.tf @@ -115,11 +115,11 @@ resource "kubernetes_deployment" "technitium_secondary" { } resources { requests = { - cpu = "25m" - memory = "512Mi" + cpu = "100m" + memory = "1Gi" } limits = { - memory = "512Mi" + memory = "1Gi" } } port { @@ -270,11 +270,11 @@ resource "kubernetes_deployment" "technitium_tertiary" { } resources { requests = { - cpu = "25m" - memory = "512Mi" + cpu = "100m" + memory = "1Gi" } limits = { - memory = "512Mi" + memory = "1Gi" } } port { @@ -391,44 +391,90 @@ resource "kubernetes_cron_job_v1" "technitium_zone_sync" { set -e PRIMARY="http://technitium-primary.technitium.svc.cluster.local:5380" REPLICAS="http://technitium-secondary-web.technitium.svc.cluster.local:5380 http://technitium-tertiary-web.technitium.svc.cluster.local:5380" + PUSHGW="http://prometheus-prometheus-pushgateway.monitoring:9091/metrics/job/technitium-zone-sync" + + # Track overall status — non-zero if any zone fails to create + OVERALL_STATUS=0 + FAIL_COUNT=0 + SYNCED=0 # Login to primary P_TOKEN=$(curl -sf "$PRIMARY/api/user/login?user=$TECH_USER&pass=$TECH_PASS" | sed -n 's/.*"token":"\([^"]*\)".*/\1/p') - if [ -z "$P_TOKEN" ]; then echo "ERROR: Cannot login to primary"; exit 1; fi + if [ -z "$P_TOKEN" ]; then echo "ERROR: Cannot login to primary"; OVERALL_STATUS=1; fi - # Get zones from primary (excluding default zones that don't need replication) - curl -sf "$PRIMARY/api/zones/list?token=$P_TOKEN" | tr ',' '\n' | sed -n 's/.*"name":"\([^"]*\)".*/\1/p' | \ - grep -v -E '^(localhost|0\.in-addr\.arpa|127\.in-addr\.arpa|255\.in-addr\.arpa|1\.0\.0.*ip6\.arpa)$$' > /tmp/primary_zones.txt - echo "Primary has $(wc -l < /tmp/primary_zones.txt) zones to replicate" - - # Enable zone transfers on primary for all zones - while read -r zone; do - curl -sf "$PRIMARY/api/zones/options/set?token=$P_TOKEN&zone=$zone&zoneTransfer=Allow" > /dev/null || true - done < /tmp/primary_zones.txt - - # Sync to each replica - SYNCED=0 - for REPLICA in $REPLICAS; do - R_TOKEN=$(curl -sf "$REPLICA/api/user/login?user=$TECH_USER&pass=$TECH_PASS" | sed -n 's/.*"token":"\([^"]*\)".*/\1/p') - if [ -z "$R_TOKEN" ]; then echo "WARN: Cannot login to $REPLICA, skipping"; continue; fi - - # Get existing zones on this replica - curl -sf "$REPLICA/api/zones/list?token=$R_TOKEN" | tr ',' '\n' | sed -n 's/.*"name":"\([^"]*\)".*/\1/p' > /tmp/replica_zones.txt + if [ "$OVERALL_STATUS" -eq 0 ]; then + # Get zones from primary (excluding default zones that don't need replication) + curl -sf "$PRIMARY/api/zones/list?token=$P_TOKEN" | tr ',' '\n' | sed -n 's/.*"name":"\([^"]*\)".*/\1/p' | \ + grep -v -E '^(localhost|0\.in-addr\.arpa|127\.in-addr\.arpa|255\.in-addr\.arpa|1\.0\.0.*ip6\.arpa)$$' > /tmp/primary_zones.txt + PRIMARY_COUNT=$(wc -l < /tmp/primary_zones.txt) + echo "Primary has $PRIMARY_COUNT zones to replicate" + # Enable zone transfers on primary for all zones while read -r zone; do - if grep -qx "$zone" /tmp/replica_zones.txt; then - # Zone exists — just resync - curl -sf "$REPLICA/api/zones/resync?token=$R_TOKEN&zone=$zone" > /dev/null || true - else - # New zone — create as Secondary and sync - echo "NEW: Creating $zone on $REPLICA" - curl -sf "$REPLICA/api/zones/create?token=$R_TOKEN&zone=$zone&type=Secondary&primaryNameServerAddresses=$PRIMARY_IP" > /dev/null || true - SYNCED=$((SYNCED + 1)) - fi + curl -sf "$PRIMARY/api/zones/options/set?token=$P_TOKEN&zone=$zone&zoneTransfer=Allow" > /dev/null || true done < /tmp/primary_zones.txt - done - echo "Zone sync complete. $$SYNCED new zone(s) created." + # Sync to each replica + for REPLICA in $REPLICAS; do + R_NAME=$(echo "$REPLICA" | sed 's|http://||; s|-web.*||') + R_TOKEN=$(curl -sf "$REPLICA/api/user/login?user=$TECH_USER&pass=$TECH_PASS" | sed -n 's/.*"token":"\([^"]*\)".*/\1/p') + if [ -z "$R_TOKEN" ]; then + echo "ERROR: Cannot login to $REPLICA" + OVERALL_STATUS=1 + FAIL_COUNT=$((FAIL_COUNT + 1)) + # Push replica zone_count=0 so divergence alert fires + printf 'technitium_zone_count{instance="%s"} 0\n' "$R_NAME" | \ + curl -sf --data-binary @- "$PUSHGW/instance/$R_NAME" || true + continue + fi + + # Get existing zones on this replica + curl -sf "$REPLICA/api/zones/list?token=$R_TOKEN" | tr ',' '\n' | sed -n 's/.*"name":"\([^"]*\)".*/\1/p' > /tmp/replica_zones.txt + REPLICA_COUNT=$(wc -l < /tmp/replica_zones.txt) + + while read -r zone; do + if grep -qx "$zone" /tmp/replica_zones.txt; then + # Zone exists — just resync + curl -sf "$REPLICA/api/zones/resync?token=$R_TOKEN&zone=$zone" > /dev/null || true + else + # New zone — create as Secondary and validate response + echo "NEW: Creating $zone on $REPLICA" + RESP=$(curl -sf "$REPLICA/api/zones/create?token=$R_TOKEN&zone=$zone&type=Secondary&primaryNameServerAddresses=$PRIMARY_IP" || echo '{"status":"error"}') + if echo "$RESP" | grep -q '"status":"ok"'; then + SYNCED=$((SYNCED + 1)) + else + echo "ERROR: Failed to create $zone on $REPLICA: $RESP" + OVERALL_STATUS=1 + FAIL_COUNT=$((FAIL_COUNT + 1)) + fi + fi + done < /tmp/primary_zones.txt + + # Push per-replica zone count + printf 'technitium_zone_count{instance="%s"} %s\n' "$R_NAME" "$REPLICA_COUNT" | \ + curl -sf --data-binary @- "$PUSHGW/instance/$R_NAME" || true + done + + # Push primary zone count + printf 'technitium_zone_count{instance="primary"} %s\n' "$PRIMARY_COUNT" | \ + curl -sf --data-binary @- "$PUSHGW/instance/primary" || true + fi + + # Push overall status (0=ok, 1=fail) + last-run timestamp + cat < rollout status deploy/$d" + kubectl -n $NS rollout status deploy/$d --timeout=180s + done + + # 2. Per-pod API + DNS check (via kubectl exec on the pod itself — no + # ephemeral debug pods, no iamge pull, no zombies). + PODS=$(kubectl -n $NS get pod -l dns-server=true -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}') + if [ -z "$PODS" ]; then + echo "ERROR: no dns-server=true pods found" + exit 1 + fi + + for POD in $PODS; do + echo "-> API check on $POD" + if ! kubectl -n $NS exec "$POD" -- wget -qO- --timeout=10 "http://127.0.0.1:5380/api/stats/get?token=&type=LastHour" | grep -q '"status":"ok"'; then + echo "ERROR: API check failed on $POD" + exit 1 + fi + done + + # 3. Zone-count parity — use the three web services from within any + # running technitium pod (has wget) to avoid spawning probe pods. + FIRST_POD=$(echo "$PODS" | head -1) + COUNTS="" + for SVC in technitium-web technitium-secondary-web technitium-tertiary-web; do + COUNT=$(kubectl -n $NS exec "$FIRST_POD" -- sh -c "wget -qO- --timeout=10 'http://$SVC:5380/api/zones/list?token=' | tr ',' '\n' | grep -c '\"name\":' || true" 2>/dev/null | tail -1) + echo "-> $SVC zone count: $${COUNT:-unknown}" + COUNTS="$COUNTS $COUNT" + done + UNIQ=$(echo $COUNTS | tr ' ' '\n' | sort -u | wc -l) + if [ "$UNIQ" -gt 1 ]; then + echo "ERROR: zone counts differ across instances:$COUNTS" + exit 1 + fi + + echo "=== Technitium readiness gate PASSED ===" + BASH + interpreter = ["/bin/bash", "-c"] + } + + depends_on = [ + kubernetes_deployment.technitium, + kubernetes_deployment.technitium_secondary, + kubernetes_deployment.technitium_tertiary, + kubernetes_service.technitium-dns, + kubernetes_pod_disruption_budget_v1.technitium_dns, + ] +}