viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

[dns] static-client DNS — Proxmox host, registry VM dual-resolver setup (WS F)

Browse source 
Fixes single-upstream DNS brittleness on non-DHCP hosts. Each host now
has a primary internal resolver + external fallback (AdGuard) so DNS
keeps working if the primary resolver IP is unreachable.

New config:

- Proxmox host (192.168.1.127): plain /etc/resolv.conf with
  nameserver 192.168.1.2 (pfSense LAN) + 94.140.14.14 (AdGuard).
  Previously: single nameserver 192.168.1.1 — could not resolve
  internal .lan names at all. Documented in
  docs/runbooks/proxmox-host.md.

- Registry VM (10.0.20.10): systemd-resolved drop-in at
  /etc/systemd/resolved.conf.d/10-internal-dns.conf
  (DNS=10.0.20.1, FallbackDNS=94.140.14.14, Domains=viktorbarzin.lan)
  plus matching per-link nameservers in /etc/netplan/50-cloud-init.yaml.
  Previously: 1.1.1.1 + 8.8.8.8 only — image pulls referencing .lan
  hostnames would fail to resolve. Documented in
  docs/runbooks/registry-vm.md.

- TrueNAS (10.0.10.15): host unreachable during this session
  ("No route to host" on 10.0.10.0/24). Deferred best-effort per
  WS F instructions; noted on the beads task.

Both hosts have pre-change backups at /root/dns-backups/ for
one-command rollback. Fallback behaviour was validated by routing
each primary to a blackhole and confirming dig answered from the
fallback.

Both runbooks include the verified resolvectl / resolv.conf state,
the fallback-test procedure, and the rollback steps.

Closes: code-dw8
This commit is contained in:
Viktor Barzin 2026-04-19 15:43:49 +00:00
parent 3b54983a9f
commit eb6ceac5f5
2 changed files with 250 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

103
docs/runbooks/proxmox-host.md Normal file
View file

@ -0,0 +1,103 @@
# Runbook: Proxmox host (pve, 192.168.1.127)
Last updated: 2026-04-19
The Proxmox host is a baremetal hypervisor on the storage LAN
(192.168.1.0/24) with a single IP `192.168.1.127`. It hosts every
Kubernetes node VM and the NFS exports that back PVCs. It does **not**
receive DHCP — its network config is static in
`/etc/network/interfaces` (ifupdown). Because of that, DNS must be
configured manually and stays out of the scope of Kea/DHCP-DDNS.
## DNS configuration
The host uses a plain `/etc/resolv.conf` with two nameservers. No
`systemd-resolved`, no `resolvconf`, no NetworkManager — nothing
manages `/etc/resolv.conf`; it is a regular file owned by root.
### Why plain `/etc/resolv.conf` and not systemd-resolved
1. Installing `systemd-resolved` on an active Proxmox node during
business hours is the kind of change that risks breaking the NFS
server or VM networking. PVE's Debian base does not ship
`systemd-resolved` by default.
2. The ifupdown `/etc/network/interfaces` file does not manage
`/etc/resolv.conf` here — ifupdown's resolvconf integration is
only active if the `resolvconf` package is installed, which it is
not (`dpkg -l resolvconf` returns `un`).
3. A plain file is the simplest mental model and avoids a second
layer of "which tool is running now" confusion during an incident.
If you ever want to migrate to `systemd-resolved`, install the
package, enable the service, symlink `/etc/resolv.conf` to
`/run/systemd/resolve/stub-resolv.conf`, and drop the config in
`/etc/systemd/resolved.conf.d/10-internal-dns.conf` — but do this
during a maintenance window, not reactively.
### Current state
```
# /etc/resolv.conf
search viktorbarzin.lan
nameserver 192.168.1.2
nameserver 94.140.14.14
options timeout:2 attempts:2
```
| Field | Value | Purpose |
|---|---|---|
| Primary | `192.168.1.2` | pfSense LAN interface (dnsmasq forwarder → Technitium LB) — resolves `.viktorbarzin.lan` |
| Fallback | `94.140.14.14` | AdGuard public DNS — recursive only, used if pfSense LAN IP unreachable |
| `search` | `viktorbarzin.lan` | Unqualified names (`technitium`, `idrac`, etc.) resolve against the internal zone |
| `timeout:2 attempts:2` | — | Cap glibc resolver at 2s per server, 2 tries — reasonable fallback latency |
### Verification commands
```sh
ssh root@192.168.1.127 '
cat /etc/resolv.conf # should show the two nameservers
dig +short idrac.viktorbarzin.lan # expect an A record (192.168.1.4)
dig +short github.com # expect an A record
'
```
Simulated failover — force the primary unreachable and verify the
fallback answers:
```sh
ssh root@192.168.1.127 '
ip route add blackhole 192.168.1.2
dig +short +time=3 github.com # glibc times out on primary, tries 94.140.14.14 → A record returned
ip route del blackhole 192.168.1.2 # cleanup
'
```
Expected behaviour: the first `dig` prints a warning about the UDP
setup failing for 192.168.1.2 and then prints the GitHub A record
(answered by 94.140.14.14).
## Rollback
A pre-change backup of `/etc/resolv.conf`, `/etc/network/interfaces`,
and `/etc/network/interfaces.d/` lives at
`/root/dns-backups/dns-config-backup-YYYYMMDD-HHMMSS.tar.gz` on the
host. To roll back:
```sh
ssh root@192.168.1.127 '
# pick the backup you want (there may be multiple if this runbook has been applied more than once)
BACKUP=$(ls -t /root/dns-backups/dns-config-backup-*.tar.gz | head -1)
tar -xzf "$BACKUP" -C /
cat /etc/resolv.conf
'
```
No service restart is needed — glibc re-reads `/etc/resolv.conf` per
lookup.
## Related docs
- `docs/architecture/dns.md` — where each resolver IP lives and which
subnet it serves.
- `docs/runbooks/nfs-prerequisites.md` — other operations on this
host; read before adding new NFS exports.

147
docs/runbooks/registry-vm.md Normal file
View file

@ -0,0 +1,147 @@
# Runbook: Registry VM (docker-registry, 10.0.20.10)
Last updated: 2026-04-19
The registry VM hosts `registry.viktorbarzin.me` (private Docker
registry, htpasswd-auth, NGINX → registry:2). It is an Ubuntu 24.04
VM on the cluster LAN subnet `10.0.20.0/24`, with a static netplan
config (no DHCP). Because it sits on a subnet that only has pfSense
as its gateway, its DNS must be statically configured.
## DNS configuration
Ubuntu ships `systemd-resolved` and uses netplan to declare per-link
`nameservers`. Netplan writes systemd-networkd or NetworkManager
configs that resolved reads at runtime. There is **no automatic
merging** of netplan DNS with the `[Resolve]` section of
`/etc/systemd/resolved.conf` — per-link settings override the global
ones. So both layers must be in sync:
| Layer | File | Role |
|---|---|---|
| Netplan | `/etc/netplan/50-cloud-init.yaml` | Per-link DNS servers that resolved reports on `Link 2 (eth0)` |
| Resolved global | `/etc/systemd/resolved.conf.d/10-internal-dns.conf` | `Global` scope `DNS=` / `FallbackDNS=` — also shown in `resolvectl status` |
### Current state
`/etc/systemd/resolved.conf.d/10-internal-dns.conf`:
```ini
[Resolve]
DNS=10.0.20.1
FallbackDNS=94.140.14.14
Domains=viktorbarzin.lan
```
`/etc/netplan/50-cloud-init.yaml` (eth0 block, simplified):
```yaml
nameservers:
addresses:
- 10.0.20.1
- 94.140.14.14
search:
- viktorbarzin.lan
```
`resolvectl status` output after the change:
```
Global
resolv.conf mode: stub
Current DNS Server: 10.0.20.1
DNS Servers: 10.0.20.1
Fallback DNS Servers: 94.140.14.14
DNS Domain: viktorbarzin.lan
Link 2 (eth0)
Current Scopes: DNS
Current DNS Server: 10.0.20.1
DNS Servers: 10.0.20.1 94.140.14.14
DNS Domain: viktorbarzin.lan
```
| Field | Value | Purpose |
|---|---|---|
| Primary | `10.0.20.1` | pfSense OPT1 interface (dnsmasq forwarder → Technitium LB) — resolves `.viktorbarzin.lan` |
| Fallback | `94.140.14.14` | AdGuard public DNS — used if pfSense unreachable (e.g., OPT1 flap) |
| Search | `viktorbarzin.lan` | Unqualified names resolve against the internal zone |
### Why this matters for the registry
Container builds on this VM reference `.lan` hostnames (Technitium,
NFS, etc.) and external hostnames (Docker Hub, GHCR). Before the
hardening the netplan had `1.1.1.1` / `8.8.8.8` only, which meant:
1. Internal hostname lookups silently failed (slow timeout) — the
VM could not resolve `idrac.viktorbarzin.lan` or any internal
helper.
2. If Cloudflare's 1.1.1.1 had an outage, the VM would lose DNS
entirely.
With the new config the VM can resolve both zones and keeps working
if the primary DNS server is unreachable.
## Apply / re-apply
```sh
ssh root@10.0.20.10 '
netplan generate
netplan apply
systemctl restart systemd-resolved
resolvectl status | head -20
'
```
`netplan apply` is not disruptive when only `nameservers` change — it
does not bounce the link.
## Verification
```sh
ssh root@10.0.20.10 '
dig +short idrac.viktorbarzin.lan # 192.168.1.4
dig +short github.com # GitHub A record
dig +short registry.viktorbarzin.me # 10.0.20.10 + external A
'
```
Fallback test — blackhole the primary and confirm external lookups
still succeed through 94.140.14.14:
```sh
ssh root@10.0.20.10 '
ip route add blackhole 10.0.20.1
dig +short +time=5 +tries=2 github.com # should still answer
ip route del blackhole 10.0.20.1
'
```
Internal lookups do fail during the blackhole (the fallback is a
public resolver and does not know about the internal zone), which is
expected — the fallback buys availability for external pulls, not
internal hostnames.
## Rollback
A pre-change backup of `/etc/resolv.conf`, `/etc/systemd/resolved.conf`,
and `/etc/netplan/` lives at
`/root/dns-backups/dns-config-backup-YYYYMMDD-HHMMSS.tar.gz` on the
VM. To roll back:
```sh
ssh root@10.0.20.10 '
BACKUP=$(ls -t /root/dns-backups/dns-config-backup-*.tar.gz | head -1)
tar -xzf "$BACKUP" -C /
rm -f /etc/systemd/resolved.conf.d/10-internal-dns.conf
netplan apply
systemctl restart systemd-resolved
resolvectl status | head -10
'
```
## Related docs
- `docs/architecture/dns.md` — resolver IP assignments per subnet.
- `.claude/CLAUDE.md` (at repo root) — notes on the private registry
and `containerd` `hosts.toml` redirects.