break-glass SSH: drop port-knock for exposed key-only :52222; version host config

Viktor got locked out of the break-glass path (forgot the port-knock setup) and
deleted the edge-router forwards, then asked to review and redesign it from
scratch.

Root cause of the lockout: the knock added no real security (key-only SSH is
already brute-force-proof) and its only benefit — hiding the port — came at the
cost of a circular dependency. The knock sequence lived only in in-cluster
Vault, which is unreachable in the exact away/cold scenario break-glass exists
for. So the unlock secret was unavailable precisely when needed.

New model (self-contained, nothing to remember): plain key-only SSH on the
Proxmox host's :52222, openly reachable. The edge router forwards WAN tcp/52222
-> 192.168.1.127:52222 (external port MUST equal internal on the TP-Link AX6000
- it rejects remaps; port 22 itself is reserved). The exposed port trusts only a
dedicated break-glass key via `Match LocalPort` (a leak of any other root key
does not grant internet access), rate-limited (iptables hashlimit) + fail2ban.

- Removed knockd (package + config) and the legacy Synology SSH forward
  (ext 3333 -> .13:22, a needless WAN exposure the original plan wanted gone).
- Fixed the fail2ban jail for Debian 13 (auth logs under sshd-session, not sshd
  - the stock journalmatch silently never banned).
- Versioned the host config in scripts/ (it was applied ad-hoc, never committed)
  and recorded the deliberate Wave-1 "no public-IP" exception in security.md +
  .claude/CLAUDE.md. Superseded the 2026-05-30 port-knock design docs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-06-11 18:23:39 +00:00
parent e2788d1b2d
commit df332b59e6
9 changed files with 989 additions and 1 deletions

View file

@ -0,0 +1,26 @@
#!/usr/bin/env bash
set -euo pipefail
# Break-glass base firewall (redesigned 2026-06-11; replaced the port-knock gate).
#
# Source of truth. Deploy to the PVE host with:
# scp scripts/breakglass-firewall.sh root@192.168.1.127:/usr/local/sbin/breakglass-firewall.sh
# ssh root@192.168.1.127 'chmod 0755 /usr/local/sbin/breakglass-firewall.sh && systemctl restart breakglass-firewall.service'
# The breakglass-firewall.service oneshot runs this at boot (RemainAfterExit).
#
# Model: key-only SSH break-glass on :52222, openly reachable from the WAN, NO
# port-knock. The SSH key is the gate (brute-force-proof); the rate-limit below
# only trims scanner noise / slows a hypothetical sshd 0-day.
# :22 -> LAN admin (all of root's keys), always allowed.
# :52222 -> WAN break-glass. LAN/VLAN sources bypass the limit; external NEW
# connections are rate-limited per source IP, then accepted.
iptables -N BREAKGLASS 2>/dev/null || iptables -F BREAKGLASS
iptables -C INPUT -j BREAKGLASS 2>/dev/null || iptables -I INPUT 1 -j BREAKGLASS
iptables -A BREAKGLASS -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A BREAKGLASS -p tcp --dport 22 -j ACCEPT
iptables -A BREAKGLASS -p tcp --dport 52222 -s 192.168.1.0/24 -j ACCEPT
iptables -A BREAKGLASS -p tcp --dport 52222 -s 10.0.0.0/8 -j ACCEPT
iptables -A BREAKGLASS -p tcp --dport 52222 -m conntrack --ctstate NEW \
-m hashlimit --hashlimit-name bg_ssh --hashlimit-mode srcip \
--hashlimit-above 6/min --hashlimit-burst 3 -j DROP
iptables -A BREAKGLASS -p tcp --dport 52222 -j ACCEPT

View file

@ -0,0 +1,18 @@
# Break-glass SSH fail2ban jail (redesigned 2026-06-11). Source of truth.
# Deploy to the PVE host with:
# scp scripts/fail2ban-breakglass-sshd.local root@192.168.1.127:/etc/fail2ban/jail.d/breakglass-sshd.local
# ssh root@192.168.1.127 'systemctl restart fail2ban'
#
# GOTCHA (Debian 13 / OpenSSH 9.x): auth lines are logged under
# _COMM=sshd-session, NOT _COMM=sshd. The stock Debian jail keys journalmatch on
# `_SYSTEMD_UNIT=ssh.service + _COMM=sshd` and therefore silently NEVER bans.
# Match by unit only so both sshd and sshd-session lines are seen. Ban on both
# SSH ports (the WAN break-glass listener is :52222).
[sshd]
enabled = true
backend = systemd
journalmatch = _SYSTEMD_UNIT=ssh.service
port = ssh,52222
maxretry = 4
findtime = 10m
bantime = 1h

View file

@ -0,0 +1,31 @@
# Break-glass SSH drop-in (redesigned 2026-06-11). Source of truth.
# Deploy to the PVE host with:
# scp scripts/sshd-10-breakglass.conf root@192.168.1.127:/etc/ssh/sshd_config.d/10-breakglass.conf
# ssh root@192.168.1.127 'sshd -t && systemctl reload ssh'
#
# :22 = LAN admin, all of root's keys (default AuthorizedKeysFile).
# :52222 = WAN-exposed break-glass. The edge router forwards WAN tcp/52222 ->
# 192.168.1.127:52222 (external port MUST equal internal port on the
# TP-Link AX6000 — it rejects remaps; port 22 itself is reserved).
# The Match LocalPort block trusts ONLY the dedicated break-glass key
# (authorized_keys.breakglass), so a leak of any other root key does
# NOT grant internet access. Rate-limited by the BREAKGLASS iptables
# chain + fail2ban. No port-knock.
#
# NOTE: the trailing `Match all` is REQUIRED. /etc/ssh/sshd_config has
# `Include sshd_config.d/*.conf` near the top but a global `PermitRootLogin`
# further down; without `Match all` resetting context, that later global
# directive would be swallowed into the `Match LocalPort 52222` condition.
Port 22
Port 52222
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
PermitRootLogin prohibit-password
MaxAuthTries 3
LoginGraceTime 20
Match LocalPort 52222
AuthorizedKeysFile /root/.ssh/authorized_keys.breakglass
PermitRootLogin prohibit-password
Match all