pve-host: ship journal to Loki (snoopy command audit + sshd-pve) for emo's root SSH

Emo's Claude agent was given root SSH to the Proxmox host (`ssh pve`, dedicated
shared-root key emo-pve-agent@devvm) so he can manage the host — e.g. the R730
fan daemon — through his agent. To keep an audit trail of what that agent does,
and to feed the long-pending Wave-1 S1 security rule, the PVE host now ships its
systemd journal to cluster Loki:

- snoopy logs every execve() to journald (identifier=snoopy), enabled via
  /etc/ld.so.preload; config scripts/pve-snoopy.ini.
- promtail v3.5.1 (amd64) ships /var/log/journal to Loki as {job="pve-journal"}
  (full host journal; filter identifier="snoopy" for the command audit), and
  relabels sshd auth to {job="sshd-pve"} — which ACTIVATES S1 (it was PENDING
  only for lack of this shipper). Config/unit: scripts/pve-promtail.{yaml,service}.

S1 won't false-fire on legitimate access: the devvm SNATs through pfSense to
192.168.1.2, which is already in the S1 source-IP allowlist.

Loki is reached via an /etc/hosts pin (10.0.20.203 loki.viktorbarzin.lan);
follow-up noted to register a Technitium CNAME so it auto-tracks LB renumbers.

Host pieces are hand-managed (not Terraform), like fan-control and the rpi-sofia
promtail — these files are the source of truth. Docs updated: security.md
(S1 LIVE) and monitoring.md ("External host: pve").

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

scripts/pve-promtail.service

@@ -0,0 +1,17 @@
+# systemd unit for promtail on the PVE host (192.168.1.127). Install to
+# /etc/systemd/system/promtail.service. See scripts/pve-promtail.yaml for the full deploy.
+[Unit]
+Description=Promtail (ships PVE host journal -> cluster Loki)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/config.yml
+Restart=on-failure
+RestartSec=5
+User=root
+Group=root
+
+[Install]
+WantedBy=multi-user.target

scripts/pve-promtail.yaml

@@ -0,0 +1,53 @@
+# Promtail config for the PVE host (192.168.1.127) — ships the systemd journal to cluster Loki.
+#
+# NOT Terraform-managed (the PVE host is the hypervisor, outside k8s). Deployed by hand,
+# same pattern as scripts/fan-control.* and the rpi-sofia promtail. This file is source-of-truth.
+#
+# Deploy:
+#   scp scripts/pve-promtail.yaml    root@192.168.1.127:/etc/promtail/config.yml
+#   scp scripts/pve-promtail.service root@192.168.1.127:/etc/systemd/system/promtail.service
+#   ssh root@192.168.1.127 'mkdir -p /var/lib/promtail && systemctl daemon-reload && systemctl enable --now promtail'
+#   # Binary: grafana/loki v3.5.1 promtail-linux-amd64 -> /usr/local/bin/promtail (chmod 0755).
+#   # Loki reach: /etc/hosts pin "10.0.20.203 loki.viktorbarzin.lan" (Traefik LB, ETP-Local).
+#   #   FOLLOW-UP: replace the pin with a Technitium CNAME loki.viktorbarzin.lan -> ingress.viktorbarzin.lan
+#   #   so it auto-tracks Traefik LB renumbers (also fixes the rpi-sofia pin — see docs/architecture/monitoring.md).
+#
+# Streams produced:
+#   {job="pve-journal"}                    — full host journal (filter identifier="snoopy" for the command audit)
+#   {job="sshd-pve"}                       — sshd auth lines; feeds the Loki S1 security rule (docs/architecture/security.md)
+#   {job="pve-journal", identifier="snoopy"} — snoopy command audit (every execve on the host; see scripts/pve-snoopy.ini)
+server:
+  http_listen_port: 9080
+  grpc_listen_port: 0
+  log_level: warn
+
+positions:
+  filename: /var/lib/promtail/positions.yaml
+
+clients:
+  - url: https://loki.viktorbarzin.lan/loki/api/v1/push
+    tls_config:
+      insecure_skip_verify: true
+
+scrape_configs:
+  - job_name: journal
+    journal:
+      max_age: 12h
+      json: false
+      path: /var/log/journal
+      labels:
+        host: pve
+        job: pve-journal
+    relabel_configs:
+      - source_labels: ['__journal__systemd_unit']
+        target_label: unit
+      - source_labels: ['__journal_priority_keyword']
+        target_label: level
+      - source_labels: ['__journal_syslog_identifier']
+        target_label: identifier
+      # sshd auth lines (identifier sshd / sshd-session) -> job=sshd-pve so the Loki S1
+      # security rule ({job="sshd-pve"}) matches. snoopy command lines stay job=pve-journal.
+      - source_labels: ['__journal_syslog_identifier']
+        regex: 'sshd.*'
+        target_label: job
+        replacement: 'sshd-pve'

scripts/pve-snoopy.ini

@@ -0,0 +1,21 @@
+; snoopy config for the PVE host (192.168.1.127) — logs every execve() to journald.
+;
+; Install to /etc/snoopy.ini. Enable globally by adding the lib to /etc/ld.so.preload:
+;   apt-get install -y snoopy
+;   echo /usr/lib/x86_64-linux-gnu/libsnoopy.so > /etc/ld.so.preload   # enable (no snoopy-enable in the Debian pkg)
+;   # disable/rollback: truncate -s 0 /etc/ld.so.preload   (or remove the line)
+;
+; output=devlog writes directly to /dev/log -> journald (identifier "snoopy").
+; DO NOT use output=syslog on a systemd host — snoopy's own docs warn it can hang the system on boot.
+;
+; Shipped to Loki by promtail as {job="pve-journal", identifier="snoopy"} (scripts/pve-promtail.yaml).
+; Attribution note: all sessions run as root (shared root key), so uid/login are always root;
+; correlate a command's sid/time with the matching {job="sshd-pve"} "Accepted publickey ... SHA256:<fp>"
+; line to attribute it to a person (e.g. emo's agent key fp SHA256:Wd+m0EABlm4RDDykDh85PIYSqe0Al8Hr9AZ+7Ksy4HQ).
+[snoopy]
+output = devlog
+message_format = "snoopy uid=%{uid} login=%{login} tty=%{tty} sid=%{sid} cwd=%{cwd} : %{cmdline}"
+syslog_ident = snoopy
+syslog_facility = LOG_AUTHPRIV
+syslog_level = LOG_INFO
+filter_chain = ""