Adds per-node DNS cache that transparently intercepts pod queries on
10.96.0.10 (kube-dns ClusterIP) AND 169.254.20.10 (link-local) via
hostNetwork + NET_ADMIN iptables NOTRACK rules. Pods keep using their
existing /etc/resolv.conf (nameserver 10.96.0.10) unchanged — no kubelet
rollout needed for transparent mode.
Layout mirrors existing stacks (technitium, descheduler, kured):
stacks/nodelocal-dns/
main.tf # module wiring + IP params
modules/nodelocal-dns/main.tf # SA, Services, ConfigMap, DS
Key decisions:
- Image: registry.k8s.io/dns/k8s-dns-node-cache:1.23.1
- Co-listens on 169.254.20.10 + 10.96.0.10 (transparent interception)
- Upstream path: kube-dns-upstream (new headless svc) → CoreDNS pods
(separate ClusterIP avoids cache looping back through itself)
- viktorbarzin.lan zone forwards directly to Technitium ClusterIP
(10.96.0.53), bypassing CoreDNS for internal names
- priorityClassName: system-node-critical
- tolerations: operator=Exists (runs on master + all tainted nodes)
- No CPU limit (cluster-wide policy); mem requests=32Mi, limit=128Mi
- Kyverno dns_config drift suppressed on the DaemonSet
- Kubelet clusterDNS NOT changed — transparent mode is sufficient;
rolling 5 nodes just to switch to 169.254.20.10 has no additional
benefit and expanding blast radius for no reason.
Verified:
- DaemonSet 5/5 Ready across k8s-master + 4 workers
- dig @169.254.20.10 idrac.viktorbarzin.lan -> 192.168.1.4
- dig @169.254.20.10 github.com -> 140.82.121.3
- Deleted all 3 CoreDNS pods; cached queries still resolved via
NodeLocal DNSCache (resilience confirmed)
Docs: architecture/dns.md — adds NodeLocal DNSCache to Components table,
graph diagram, stacks table; rewrites pod DNS resolution paths to show
the cache layer; adds troubleshooting entry.
Closes: code-2k6
25 KiB
DNS Architecture
Last updated: 2026-04-19 (NodeLocal DNSCache deployed — Workstream C)
Overview
DNS is served by a split architecture: Technitium DNS handles internal resolution (.viktorbarzin.lan) and recursive lookups, while Cloudflare DNS manages all public domains (.viktorbarzin.me). Kubernetes pods use CoreDNS which forwards to Technitium for internal zones. All three Technitium instances run on encrypted block storage with zone replication via AXFR every 30 minutes. A NodeLocal DNSCache DaemonSet runs on every node and transparently intercepts pod DNS traffic, caching responses locally so pods keep resolving even during CoreDNS, Technitium, or pfSense disruptions.
Architecture Diagram
graph TB
subgraph "External"
Internet[Internet Clients]
CF[Cloudflare DNS<br/>~50 domains<br/>viktorbarzin.me]
CFTunnel[Cloudflared Tunnel<br/>3 replicas]
end
subgraph "LAN (192.168.1.0/24)"
LAN[LAN Clients<br/>WiFi / Wired]
TPLINK[TP-Link AP<br/>Dumb AP only]
end
subgraph "pfSense (10.0.20.1)"
pf_dnsmasq[dnsmasq<br/>Forwarder]
pf_kea[Kea DHCP4<br/>3 subnets, 53 reservations]
pf_ddns[Kea DHCP-DDNS<br/>RFC 2136]
pf_nat[NAT rdr<br/>UDP 53 → Technitium]
end
subgraph "Kubernetes Cluster"
NodeLocalDNS[NodeLocal DNSCache<br/>DaemonSet, 5 nodes<br/>169.254.20.10 + 10.96.0.10]
CoreDNS[CoreDNS<br/>kube-system<br/>.:53 + viktorbarzin.lan:53]
KubeDNSUpstream[kube-dns-upstream<br/>ClusterIP, selects CoreDNS pods]
subgraph "Technitium HA (namespace: technitium)"
Primary[Primary<br/>technitium]
Secondary[Secondary<br/>technitium-secondary]
Tertiary[Tertiary<br/>technitium-tertiary]
end
LB_DNS[LoadBalancer<br/>10.0.20.201<br/>ETP=Local]
ClusterIP[ClusterIP<br/>10.96.0.53<br/>pinned]
subgraph "Automation CronJobs"
ZoneSync[zone-sync<br/>every 30min]
SplitHorizon[split-horizon-sync<br/>every 6h]
DNSOpt[dns-optimization<br/>every 6h]
PassSync[password-sync<br/>every 6h]
DNSSync[phpipam-dns-sync<br/>every 15min]
end
end
Internet -->|DNS query| CF
CF -->|CNAME to tunnel| CFTunnel
LAN -->|DNS query UDP 53| pf_nat
pf_nat -->|forward| LB_DNS
pf_kea -->|lease event| pf_ddns
pf_ddns -->|A + PTR| LB_DNS
pf_dnsmasq -->|.viktorbarzin.lan| LB_DNS
pf_dnsmasq -->|public queries| CF
NodeLocalDNS -->|cache miss| KubeDNSUpstream
KubeDNSUpstream --> CoreDNS
CoreDNS -->|.viktorbarzin.lan| ClusterIP
CoreDNS -->|public queries| pf_dnsmasq
LB_DNS --> Primary
LB_DNS --> Secondary
LB_DNS --> Tertiary
ClusterIP --> Primary
ClusterIP --> Secondary
ClusterIP --> Tertiary
ZoneSync -->|AXFR| Primary
ZoneSync -->|replicate| Secondary
ZoneSync -->|replicate| Tertiary
Components
| Component | Location | Version | Purpose |
|---|---|---|---|
| Technitium DNS | K8s namespace technitium |
14.3.0 | Primary internal DNS + recursive resolver |
| CoreDNS | K8s kube-system |
Cluster default | K8s service discovery + forwarding to Technitium |
| NodeLocal DNSCache | K8s kube-system (DaemonSet) |
k8s-dns-node-cache:1.23.1 |
Per-node DNS cache, transparent interception on 10.96.0.10 + 169.254.20.10. Insulates pods from CoreDNS/Technitium/pfSense disruption. |
| Cloudflare DNS | SaaS | N/A | Public domain management (~50 domains) |
| pfSense dnsmasq | 10.0.20.1 | pfSense 2.7.x | DNS forwarder for management VLAN |
| Kea DHCP-DDNS | 10.0.20.1 | pfSense 2.7.x | Automatic DNS registration on DHCP lease |
| phpIPAM | K8s namespace phpipam |
v1.7.0 | IPAM ↔ DNS bidirectional sync |
Terraform Stacks
| Stack | Path | DNS Resources |
|---|---|---|
| Technitium | stacks/technitium/ |
3 deployments, services, PVCs, 4 CronJobs, CoreDNS ConfigMap |
| NodeLocal DNSCache | stacks/nodelocal-dns/ |
DaemonSet (5 pods), ConfigMap, kube-dns-upstream Service, headless metrics Service |
| Cloudflared | stacks/cloudflared/ |
Cloudflare DNS records (A, AAAA, CNAME, MX, TXT), tunnel config |
| phpIPAM | stacks/phpipam/ |
dns-sync CronJob, pfsense-import CronJob |
| pfSense | stacks/pfsense/ |
VM config (DNS config is via pfSense web UI) |
DNS Resolution Paths
K8s Pod → Internal Domain (.viktorbarzin.lan)
Pod → NodeLocal DNSCache (intercepts on kube-dns:10.96.0.10)
→ cache hit: serve locally (TTL 30s / stale up to 86400s via CoreDNS upstream)
→ cache miss: forward to kube-dns-upstream (selects CoreDNS pods directly)
→ CoreDNS: template matches 2+ labels before .viktorbarzin.lan → NXDOMAIN
→ CoreDNS: forward to Technitium ClusterIP (10.96.0.53)
→ Technitium resolves from viktorbarzin.lan zone
The ndots:5 template in CoreDNS short-circuits queries like www.cloudflare.com.viktorbarzin.lan (caused by K8s search domain expansion) by returning NXDOMAIN for any query with 2+ labels before .viktorbarzin.lan. Only single-label queries (e.g., idrac.viktorbarzin.lan) reach Technitium.
K8s Pod → Public Domain
Pod → NodeLocal DNSCache (intercepts on kube-dns:10.96.0.10)
→ cache hit: serve locally
→ cache miss: forward to kube-dns-upstream (selects CoreDNS pods directly)
→ CoreDNS: forward to pfSense (10.0.20.1), fallback 8.8.8.8, 1.1.1.1
→ pfSense dnsmasq → Cloudflare (1.1.1.1)
LAN Client (192.168.1.x) → Any Domain
Client gets DNS=192.168.1.2 (pfSense WAN) from DHCP
→ pfSense NAT rdr on WAN interface → Technitium LB (10.0.20.201)
→ Technitium resolves:
- .viktorbarzin.lan → local zone
- .viktorbarzin.me (non-proxied) → recursive, then Split Horizon translates
176.12.22.76 → 10.0.20.200 for 192.168.1.0/24 clients
- other → recursive to Cloudflare DoH (1.1.1.1)
Client source IPs are preserved (no SNAT on 192.168.1.x → 10.0.20.x path) — Technitium logs show real per-device IPs.
Management VLAN (10.0.10.x) → Any Domain
Client gets DNS from Kea DHCP → pfSense (10.0.10.1)
→ pfSense dnsmasq:
- .viktorbarzin.lan → forward to Technitium (10.0.20.201)
- other → forward to Cloudflare (1.1.1.1)
K8s VLAN (10.0.20.x) → Any Domain
Client gets DNS from Kea DHCP → pfSense (10.0.20.1)
→ pfSense dnsmasq:
- .viktorbarzin.lan → forward to Technitium (10.0.20.201)
- other → forward to Cloudflare (1.1.1.1)
Technitium DNS — Internal DNS Server
Deployment Topology
Three independent Technitium instances, each with its own encrypted block storage PVC (proxmox-lvm-encrypted, 2Gi each):
| Instance | Deployment | PVC | Web Service | Role |
|---|---|---|---|---|
| Primary | technitium |
technitium-primary-config-encrypted |
technitium-web:5380 |
Authoritative primary, zone edits happen here |
| Secondary | technitium-secondary |
technitium-secondary-config-encrypted |
technitium-secondary-web:5380 |
AXFR replica |
| Tertiary | technitium-tertiary |
technitium-tertiary-config-encrypted |
technitium-tertiary-web:5380 |
AXFR replica |
All three pods share the dns-server=true label, so the DNS LoadBalancer (10.0.20.201) and ClusterIP (10.96.0.53) route queries to any healthy instance.
High Availability
- Pod anti-affinity:
requiredonkubernetes.io/hostname— all 3 pods run on different nodes - PodDisruptionBudget:
minAvailable=2— at least 2 DNS pods survive voluntary disruptions - Recreate strategy: Each deployment uses
Recreate(RWO block storage) - Zone sync CronJob (
technitium-zone-sync, every 30min): Replicates all primary zones to secondary/tertiary via AXFR. Idempotent — skips existing zones, creates missing ones as Secondary type.
Services
| Service | Type | IP | Selector | Purpose |
|---|---|---|---|---|
technitium-dns |
LoadBalancer | 10.0.20.201 | dns-server=true |
External LAN access, externalTrafficPolicy: Local |
technitium-dns-internal |
ClusterIP | 10.96.0.53 (pinned) | dns-server=true |
CoreDNS forwarding, survives Service recreation |
technitium-primary |
ClusterIP | auto | app=technitium |
Zone transfers (AXFR) + API access to primary only |
technitium-web |
ClusterIP | auto | app=technitium |
Web UI (port 5380) + DoH (port 80) |
technitium-secondary-web |
ClusterIP | auto | app=technitium-secondary |
Secondary API access |
technitium-tertiary-web |
ClusterIP | auto | app=technitium-tertiary |
Tertiary API access |
Zones
Primary zones (managed on primary, replicated to secondary/tertiary):
| Zone | Type | Records | Notes |
|---|---|---|---|
viktorbarzin.lan |
Primary | 30+ A/CNAME | Internal hosts (idrac, grafana, proxmox, vaultwarden, etc.) |
10.0.10.in-addr.arpa |
Primary | PTR | Reverse DNS for management VLAN |
20.0.10.in-addr.arpa |
Primary | PTR | Reverse DNS for K8s VLAN |
1.168.192.in-addr.arpa |
Primary | PTR | Reverse DNS for LAN |
2.3.10.in-addr.arpa |
Primary | PTR | Reverse DNS for VPN |
0.168.192.in-addr.arpa |
Primary | PTR | Reverse DNS for Valchedrym site |
emrsn.org |
Primary (stub) | — | Returns NXDOMAIN locally (avoids 27K+ daily corporate query floods) |
Dynamic updates: Enabled via UseSpecifiedNetworkACL from pfSense IPs (10.0.20.1, 10.0.10.1, 192.168.1.2) for Kea DDNS RFC 2136 updates.
Resolver Settings
| Setting | Value | Rationale |
|---|---|---|
| Forwarders | Cloudflare DoH (1.1.1.1, 1.0.0.1) | Encrypted upstream DNS |
| Cache max entries | 100K | Ample for homelab |
| Cache min TTL | 60s | Reduces re-queries for short-TTL domains (e.g., headscale: 18s) |
| Cache max TTL | 7 days | Long cache for stable records |
| Serve stale | Enabled (3 days) | Resilience during upstream failures |
Ad Blocking
Technitium runs built-in DNS blocking with:
- OISD Big List (~486K domains)
- StevenBlack hosts list
Blocking is enabled on all three instances (DNS_SERVER_ENABLE_BLOCKING=true on secondary/tertiary).
Query Logging
| Backend | Status | Retention | Purpose |
|---|---|---|---|
MySQL (technitium DB) |
Disabled | — | Legacy, disabled by password-sync CronJob |
PostgreSQL (technitium DB on CNPG) |
Enabled | 90 days | Primary query log store |
Grafana dashboard (grafana-technitium-dashboard ConfigMap) visualizes query logs from the MySQL datasource. A Grafana datasource is auto-provisioned via sidecar.
Web UI & Ingress
- Web UI:
technitium.viktorbarzin.me(Authentik-protected viaingress_factory) - DNS-over-HTTPS:
dns.viktorbarzin.me(separate ingress, port 80) - Homepage widget: Technitium widget showing totalQueries, totalCached, totalBlocked, totalRecursive
Split Horizon (Hairpin NAT Fix)
Problem
The TP-Link AP (dumb AP on 192.168.1.x) does not support hairpin NAT. LAN clients resolving non-proxied *.viktorbarzin.me domains get the public IP 176.12.22.76, but can't reach it because the TP-Link won't route back to the local network.
Solution
Technitium's Split Horizon AddressTranslation app post-processes DNS responses for 192.168.1.0/24 clients, translating the public IP to the internal Traefik LB IP:
176.12.22.76 → 10.0.20.200
DNS Rebinding Protection has viktorbarzin.me in privateDomains to allow the translated private IP without being stripped as a rebinding attack.
Scope
- Affected: Non-proxied domains (ha-sofia, immich, headscale, calibre, vaultwarden, etc.) for 192.168.1.x clients
- Not affected: Cloudflare-proxied domains (resolve to Cloudflare edge IPs, no translation needed)
- Not affected: 10.0.x.x and K8s clients (reach public IP via pfSense outbound NAT normally)
Config is synced to all 3 Technitium instances by CronJob technitium-split-horizon-sync (every 6h).
NodeLocal DNSCache
A DaemonSet in kube-system (node-local-dns, image registry.k8s.io/dns/k8s-dns-node-cache:1.23.1) runs on every node including the control plane. Each pod uses hostNetwork: true + NET_ADMIN and installs iptables NOTRACK rules so it transparently serves DNS on both:
- 169.254.20.10 — the canonical link-local IP from the upstream docs
- 10.96.0.10 — the
kube-dnsClusterIP, so existing pods (which already use this as their nameserver) hit the on-node cache with no kubelet change
Cache misses go to a separate kube-dns-upstream ClusterIP service (not kube-dns, to avoid looping back to ourselves) that selects the CoreDNS pods directly via k8s-app=kube-dns.
Priority class is system-node-critical; tolerations are permissive (operator: Exists) so the DaemonSet runs on tainted master and other reserved nodes. Kyverno dns_config drift is suppressed via ignore_changes on the DaemonSet.
Caching: cluster.local:53 caches 9984 success / 9984 denial entries with 30s/5s TTLs. Other zones cache 30s. If CoreDNS is killed, nodes keep answering cached names — verified on 2026-04-19 by deleting all three CoreDNS pods and running dig @169.254.20.10 idrac.viktorbarzin.lan + dig @169.254.20.10 github.com from a pod (both returned answers).
Kubelet clusterDNS: Unchanged — still 10.96.0.10. NodeLocal DNSCache co-listens on that IP so traffic interception is transparent; switching kubelet to 169.254.20.10 would require a rolling reconfigure of every node and provides no additional cache benefit over transparent mode.
Metrics: A headless Service node-local-dns (ClusterIP None) exposes each pod on port 9253 for Prometheus scraping (annotated prometheus.io/scrape=true).
CoreDNS Configuration
CoreDNS is managed via Terraform in stacks/technitium/modules/technitium/ — the Corefile ConfigMap lives in main.tf, and scaling/PDB are in coredns.tf (a kubernetes_deployment_v1_patch against the kubeadm-managed Deployment).
.:53 {
errors / health / ready
kubernetes cluster.local in-addr.arpa ip6.arpa # K8s service discovery
prometheus :9153 # Metrics
forward . 10.0.20.1 8.8.8.8 1.1.1.1 {
policy sequential # try upstreams in order
health_check 5s # mark unhealthy in 5s
max_fails 2
}
cache {
success 10000 300 6
denial 10000 300 60
serve_stale 86400s # resilience during upstream outage
}
loop / reload / loadbalance
}
viktorbarzin.lan:53 {
template: .*\..*\.viktorbarzin\.lan\.$ → NXDOMAIN # ndots:5 junk filter
forward . 10.96.0.53 { # Technitium ClusterIP
health_check 5s
max_fails 2
}
cache (success 10000 300, denial 10000 300, serve_stale 86400s)
}
Scaling: 3 replicas, required anti-affinity on kubernetes.io/hostname (spread across 3 distinct nodes). PodDisruptionBudget coredns with minAvailable=2.
Kyverno ndots injection: A Kyverno policy injects ndots:2 on all pods cluster-wide to reduce search domain expansion noise. The template regex is a second layer of defense for any queries that still get expanded.
Failover behaviour: With policy sequential on the root forward block, CoreDNS tries pfSense first; if health_check 5s detects pfSense as down, it fails over to 8.8.8.8 then 1.1.1.1 within ~5s rather than timing out per-query. Combined with serve_stale, pods keep resolving cached names for up to 24h even with full upstream failure.
Cloudflare DNS — External Domains
All public domains are under the viktorbarzin.me zone. DNS records are auto-created per service via the ingress_factory module's dns_type parameter. A small number of records (Helm-managed ingresses, special cases) remain centrally managed in config.tfvars.
How DNS Records Are Created
stacks/<service>/main.tf
module "ingress" {
source = ingress_factory
dns_type = "proxied" # ← auto-creates Cloudflare DNS record
}
dns_type = "proxied": Creates CNAME →{tunnel_id}.cfargotunnel.com(Cloudflare CDN)dns_type = "non-proxied": Creates A → public IP + AAAA → IPv6dns_type = "none"(default): No DNS record
The Cloudflare tunnel uses a wildcard rule (*.viktorbarzin.me → Traefik) — no per-hostname tunnel config needed. Traefik handles host-based routing via K8s Ingress resources.
Record Types
| Type | Records | Target | Example |
|---|---|---|---|
| Proxied CNAME | ~100 domains | {tunnel_id}.cfargotunnel.com |
blog, hackmd, homepage, ntfy |
| Non-proxied A | ~35 domains | 176.12.22.76 (public IP) |
mail, headscale, immich |
| Non-proxied AAAA | ~35 domains | IPv6 (HE tunnel) | Same as non-proxied A |
| MX | 1 | mail.viktorbarzin.me |
Inbound email |
| TXT (SPF) | 1 | v=spf1 include:mailgun.org -all |
Email authentication |
| TXT (DKIM) | 4 | RSA keys (s1, mail, brevo1, brevo2) | Email signing |
| TXT (DMARC) | 1 | v=DMARC1; p=quarantine; pct=100 |
Email policy |
| TXT (MTA-STS) | 1 | v=STSv1; id=20260412 |
TLS enforcement |
| TXT (TLSRPT) | 1 | v=TLSRPTv1; rua=mailto:postmaster@... |
TLS reporting |
| A (keyserver) | 1 | 130.162.165.220 (Oracle VPS) |
PGP keyserver |
Proxied vs Non-Proxied
- Proxied (orange cloud): Traffic routes through Cloudflare CDN → Cloudflared tunnel → Traefik. Benefits: DDoS protection, caching, no public IP exposure.
- Non-proxied (grey cloud): DNS resolves directly to public IP. Required for services needing direct connections (mail, VPN, WebSocket-heavy apps).
Zone Settings
- HTTP/3 (QUIC): Enabled globally via
cloudflare_zone_settings_override
DHCP → DNS Auto-Registration
Devices get automatic DNS registration without manual intervention. See networking.md § IPAM & DNS Auto-Registration for the full data flow diagram.
Summary:
- Kea DHCP on pfSense assigns IP (53 reservations across 3 subnets)
- Kea DDNS sends RFC 2136 dynamic update to Technitium (A + PTR records) — immediate
- phpipam-pfsense-import CronJob (5min) pulls Kea leases + ARP table into phpIPAM
- phpipam-dns-sync CronJob (15min) pushes named phpIPAM hosts → Technitium A + PTR, pulls Technitium PTR → phpIPAM hostnames
Automation CronJobs
| CronJob | Schedule | Namespace | Purpose |
|---|---|---|---|
technitium-zone-sync |
*/30 * * * * |
technitium | AXFR replication to secondary/tertiary |
technitium-password-sync |
0 */6 * * * |
technitium | Vault-rotated MySQL password → Technitium config, configure PG logging |
technitium-split-horizon-sync |
15 */6 * * * |
technitium | Split Horizon + DNS Rebinding Protection on all 3 instances |
technitium-dns-optimization |
30 */6 * * * |
technitium | Min cache TTL 60s, emrsn.org stub zone |
phpipam-dns-sync |
*/15 * * * * |
phpipam | Bidirectional phpIPAM ↔ Technitium DNS sync |
phpipam-pfsense-import |
*/5 * * * * |
phpipam | Import Kea DHCP leases + ARP from pfSense |
Password Rotation Flow
Vault's database engine rotates the Technitium MySQL password every 7 days. The flow:
Vault DB engine rotates password
→ ExternalSecret (refreshInterval=15m) pulls from static-creds/mysql-technitium
→ K8s Secret technitium-db-creds updated
→ CronJob technitium-password-sync (every 6h):
1. Logs into Technitium API
2. Disables MySQL query logging (migrated to PG)
3. Checks PG plugin is loaded (warns if missing)
4. Configures PG query logging (90-day retention)
Monitoring
| Metric Source | Dashboard | Alerts |
|---|---|---|
| Technitium query logs (PostgreSQL) | Grafana technitium-dns.json |
— |
| CoreDNS Prometheus metrics (:9153) | Grafana CoreDNS dashboard | CoreDNSErrors, CoreDNSForwardFailureRate |
| Technitium zone-sync CronJob (Pushgateway) | — | TechnitiumZoneSyncFailed, TechnitiumZoneSyncStale, TechnitiumZoneCountMismatch |
| Technitium DNS pod availability | — | TechnitiumDNSDown |
dns-anomaly-monitor CronJob (Pushgateway) |
— | DNSQuerySpike, DNSQueryRateDropped, DNSHighErrorRate |
| Uptime Kuma | External monitors for all proxied domains | ExternalAccessDivergence (15min) |
Metrics pushed by technitium-zone-sync
The zone-sync CronJob (runs every 30min) pushes the following to the Prometheus Pushgateway under job=technitium-zone-sync:
| Metric | Labels | Meaning |
|---|---|---|
technitium_zone_sync_status |
— | 0 = last run succeeded, 1 = at least one zone failed to create |
technitium_zone_sync_failures |
— | Number of zones that failed to create this run |
technitium_zone_sync_last_run |
— | Unix timestamp of last run (used by TechnitiumZoneSyncStale) |
technitium_zone_count |
instance=primary|<replica-host> |
Zone count on each Technitium instance (drives TechnitiumZoneCountMismatch) |
DNS alert rewrites
DNSQuerySpikewas previously broken: it compared current queries againstdns_anomaly_avg_queries, which was computed from a per-pod/tmp/dns_avgfile. Each CronJob run started with a fresh/tmp, soNEW_AVG == TOTAL_QUERIESevery time and the spike condition could never fire. Rewritten to useavg_over_time(dns_anomaly_total_queries[1h] offset 15m)which compares against the actual 1h Prometheus history.DNSQueryRateDropped(new): fires when query rate drops below 50% of 1h average — upstream clients may be failing to reach Technitium.
Troubleshooting
DNS Not Resolving Internal Domains
- Check NodeLocal DNSCache pods first — pod queries go through these:
kubectl -n kube-system get pod -l k8s-app=node-local-dns -o wide - Check Technitium pods:
kubectl get pod -n technitium - Check all 3 are healthy:
kubectl get pod -n technitium -l dns-server=true - Test via NodeLocal DNSCache from a pod:
kubectl exec -it <pod> -- dig @169.254.20.10 idrac.viktorbarzin.lan - Bypass NodeLocal DNSCache (test CoreDNS directly):
kubectl exec -it <pod> -- dig @<kube-dns-upstream-ClusterIP> idrac.viktorbarzin.lan(kubectl get svc -n kube-system kube-dns-upstream) - Check CoreDNS logs:
kubectl logs -n kube-system -l k8s-app=kube-dns - Verify ClusterIP service:
kubectl get svc -n technitium technitium-dns-internal
LAN Clients Can't Resolve
- Verify pfSense NAT rule redirects UDP 53 on WAN to 10.0.20.201
- Check Technitium LB service:
kubectl get svc -n technitium technitium-dns - Test from LAN:
dig @192.168.1.2 idrac.viktorbarzin.lan - Check
externalTrafficPolicy: Local— if no Technitium pod runs on the node receiving traffic, it drops
Hairpin NAT Not Working (LAN → *.viktorbarzin.me Fails)
- Verify Split Horizon app is installed on all instances
- Check CronJob status:
kubectl get cronjob -n technitium technitium-split-horizon-sync - Run the job manually:
kubectl create job --from=cronjob/technitium-split-horizon-sync test-sh -n technitium - Test:
dig @10.0.20.201 immich.viktorbarzin.me— should return 10.0.20.200 for 192.168.1.x source
Zone Not Replicating to Secondary/Tertiary
- Check zone-sync CronJob:
kubectl get cronjob -n technitium technitium-zone-sync - Check recent jobs:
kubectl get jobs -n technitium | grep zone-sync - Verify AXFR is enabled on primary: Check zone options → Zone Transfer = Allow
- Run sync manually:
kubectl create job --from=cronjob/technitium-zone-sync test-sync -n technitium
High NXDOMAIN Rate in Logs
Common causes:
- ndots:5 expansion: Pods query
host.search.domain.viktorbarzin.lan— mitigated by CoreDNS template + Kyverno ndots:2 - Corporate domains (emrsn.org): 27K+ daily queries — mitigated by stub zone returning NXDOMAIN locally
- Ad blocking: Expected for blocked domains
Adding a New DNS Record
For internal .viktorbarzin.lan records:
- Add host in phpIPAM web UI (
phpipam.viktorbarzin.me) with hostname - Wait 15 minutes for
phpipam-dns-syncto push to Technitium - Or add directly in Technitium web UI (
technitium.viktorbarzin.me)
For external .viktorbarzin.me records:
- Add
dns_type = "proxied"(or"non-proxied") to theingress_factorymodule call in the service stack - Run
scripts/tg applyon the service stack — DNS record is auto-created - For non-standard records (MX, TXT), add a
cloudflare_recordresource instacks/cloudflared/modules/cloudflared/cloudflare.tf
Incident History
- 2026-04-14 (SEV1): NFS
fsid=0caused Technitium primary data loss on restart. Fixed by migrating all 3 instances toproxmox-lvm-encrypted, adding zone-sync CronJob (30min AXFR). See post-mortem.
Related
- Networking Architecture — VLAN topology, IPAM auto-registration, ingress flow, MetalLB
- Mailserver Architecture — DNS records for email (MX, SPF, DKIM, DMARC)
- Security Architecture — Kyverno ndots policy
- Monitoring Architecture — CoreDNS metrics, Uptime Kuma external monitors
- Runbook:
docs/runbooks/add-dns-record.md(referenced but not yet created)