From ceae4d5f062e9b6998ba823ec4c9c0ac7cebac46 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Sun, 21 Jun 2026 13:39:26 +0000 Subject: [PATCH] docs: rewrite CrowdSec enforcement architecture (firewall-bouncer + CF WAF; Yaegi plugin removed) The Traefik Yaegi CrowdSec bouncer plugin was dead on Traefik 3.7.5 (handler never invoked) and has been removed. Document the replacement: in-kernel nftables drop via cs-firewall-bouncer on direct hosts, and a Cloudflare IP-List + zone WAF block rule (fed by a LAPI->CF-list sync CronJob) on proxied hosts. Both add zero per-request latency and fail open. Co-Authored-By: Claude Opus 4.8 --- .claude/CLAUDE.md | 4 +- docs/architecture/networking.md | 85 +++++++++----- docs/architecture/security.md | 198 +++++++++++++++++++------------- 3 files changed, 176 insertions(+), 111 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index cc4abddf..03d4de95 100755 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -202,7 +202,7 @@ the workflow's built-in `GITHUB_TOKEN` (`packages: write`). - **Critical path services scaled to 3**: Traefik, Authentik, CrowdSec LAPI, PgBouncer, Cloudflared. - **PDBs**: minAvailable=2 on Traefik and Authentik. - **Fallback proxies**: basicAuth when Authentik is down, fail-open when poison-fountain is down. -- **CrowdSec bouncer**: graceful degradation mode (fail-open on error). +- **CrowdSec enforcement is out-of-band** (no Traefik plugin/middleware — the dead Yaegi `crowdsec-bouncer-traefik-plugin` was removed on Traefik 3.7.5): banned IPs are dropped **in-kernel via nftables** by the `cs-firewall-bouncer` DaemonSet on **direct** hosts (drops in BOTH the `input` and `forward` hooks — Traefik is ETP=Local so client traffic is DNAT'd to the pod via `forward`; pulls ALL decisions incl. the ~31k CAPI blocklist), and **blocked at the Cloudflare edge** for **proxied** hosts (one `crowdsec_ban` Rules List + a zone WAF block rule, fed by the `crowdsec-cf-sync` CronJob in `rybbit` ns every 2 min — excludes CAPI). Zero per-request latency; **fails open** (LAPI down → no new bans, existing drops persist, legit traffic never blocked). Whitelist covers RFC1918 + tailnet + internal CIDRs. Full as-built: `docs/architecture/security.md`. - **Rate limiting**: Return 429 (not 503). Per-service tuning via dedicated middleware + `skip_default_rate_limit` (default 10/s burst 50): Immich 1000/20000, ActualBudget 50/300 (app boot = ~70 parallel revalidations). - **Retry middleware**: 2 attempts, 100ms — in default ingress chain. - **Entrypoint transport timeouts** (`websecure` `respondingTimeouts`): `writeTimeout=0` (unlimited download duration), `readTimeout=3600s` (uploads ≤1h), `idleTimeout=600s`. These are **HARD total-duration caps**, not nginx-style per-read idle timeouts — a finite `writeTimeout` truncates *any* large download at that wall-clock mark (a prior `writeTimeout=60s` silently cut Immich videos at 60s). **Do NOT re-tighten `writeTimeout`**; keep `readTimeout` finite (slow-loris backstop) but ≥ longest expected upload. Full rationale: `docs/architecture/networking.md` → "Entrypoint Transport Timeouts". @@ -216,7 +216,7 @@ the workflow's built-in `GITHUB_TOKEN` (`packages: write`). |---------|--------------------------| | Nextcloud | MaxRequestWorkers=150, needs 8Gi limit (Apache transient memory spikes, see commit eb94144), very generous startup probe | | Immich | ML on SSD (CUDA), disable ModSecurity (breaks streaming), frequent upgrades. **`immich-machine-learning` MUST run with `MACHINE_LEARNING_MODEL_TTL > 0`** (set to `600` in `stacks/immich/main.tf`, env on the `immich-machine-learning` deployment). At `0`, no model ever unloads and onnxruntime's CUDA arena (OCR's dynamic input shapes inflate it to ~10 GB) is held forever on the **time-sliced T4 it shares with llama-swap/frigate/immich-server** — which has no VRAM isolation, so immich-ml starved llama-swap (qwen3-8b) and silently broke recruiter-responder triage for ~5 h on 2026-06-02 (post-mortem `docs/post-mortems/2026-06-02-immich-ml-ttl-gpu-oom-recruiter.md`). TTL>0 lets idle models (OCR, face — AND CLIP) free VRAM. The TTL is a single GLOBAL knob (no per-model pin), so CLIP would also unload after 600s idle; the `clip-keepalive` CronJob (`*/5 * * * *`, same stack) pings the CLIP textual encoder so smart-search stays warm without pinning the ad-hoc models. **Smart search has a SECOND warmth layer in Postgres** (don't conflate it with the ML model): the ~665MB vchord `clip_index` must stay resident in PG `shared_buffers`, else an ANN probe that lands on an evicted list pays a ~1.8s cold storage read vs ~4ms warm. The `postStart` hook prewarms it ONCE at pod start and `pg_prewarm.autoprewarm` only re-warms at *startup*, so the index decays out of cache over days under job buffer-pressure (observed ~33% resident after 9d uptime → slow context search, easily misattributed to the ML model). The `clip-index-prewarm` CronJob (`*/5`, same stack) re-runs `pg_prewarm('clip_index')` to pin it hot; `immich-search-probe` (`*/5`) measures live latency + residency → Pushgateway gauges (`immich_smart_search_db_seconds`, `immich_clip_index_cached_pct`) → alerts `ImmichSmartSearchSlow`/`ImmichClipIndexColdCache`/`ImmichSearchProbeStale` + cluster-health check #46 (`check_immich_search`). immich PG role is a superuser so the CronJobs can run `pg_prewarm`/`pg_buffercache`. **Video transcoding is GPU-accelerated**: `immich-server` is pinned to GPU node1 (nodeSelector `nvidia.com/gpu.present` + NoSchedule toleration + `gpu-workload` priority) with a time-sliced `nvidia.com/gpu=1` slice — the stock immich-server image's ffmpeg already ships h264/hevc_nvenc + NVDEC. Activated via `ffmpeg.accel=nvenc` + `accelDecode=true` in the **DB** system-config (`system_metadata` table, key `system-config`, JSONB — NOT Terraform; app config is DB-managed here like oauth/smtp). Direct DB edits need a pod **recreate** to reload (config is cached at boot; only API-driven changes broadcast a reload). **Streaming bitrate is capped** to keep 4K playback smooth on the contended HDD and over remote uplinks: `ffmpeg.maxBitrate=20000k` + `preset=medium` + `transcode=bitrate` (set 2026-06-01 — was uncapped `maxBitrate=0` + `ultrafast` + `targetResolution=original`, which produced 77–264 Mbps 4K transcodes that stuttered for every client, local and remote, since even a single stream needs ~10–13.5 MB/s off the shared `sdc` spindle). 4K resolution is preserved (`targetResolution=original`); originals are NEVER modified — only the `encoded-video/` streaming copy. To re-apply transcode settings to EXISTING videos (config changes only affect new/missing ones): delete the offenders' `asset_file` rows `WHERE type='encoded_video'` (derived/regenerable — never touches originals) then run videoConversion `force=false` (admin Jobs API → "Missing"); it regenerates them to the deterministic `.mp4` path at concurrency 1 (gentle on sdc). See `docs/runbooks/immich-transcode-bitrate.md`. If Immich is ever reinstalled fresh (not restored), re-set these keys (accel, accelDecode, **maxBitrate=20000k, preset=medium, transcode=bitrate**). Thumbnails/previews live on SSD NFS (sdb) — do NOT move to block storage (HDD sdc = slower + the contended IO domain). **Background-job concurrency is capped to protect sdc** (DB-managed system-config, `system_metadata` key `system-config`, JSONB `job.*.concurrency`; re-set on fresh install): `thumbnailGeneration=2`, `metadataExtraction=2`, `library=2` — these jobs read ORIGINALS off the HDD library. Left uncapped (were 8/4/4) a library-wide job (e.g. Duplicate Detection on 2026-06-01) fans the ML/thumbnail backfill out into a read storm that saturates sdc and starves etcd → apiserver down. `sidecar`/`smartSearch`/`faceDetection` stay at Immich defaults (small `.xmp` / SSD previews). Apply via Job Settings UI or the `system-config` API; **direct DB edits need an `immich-server` pod recreate to reload** (config cached at boot). See `docs/post-mortems/2026-05-25-immich-anca-elements-io-storm.md`. | -| CrowdSec | Pin version, disable Metabase when not needed (CPU hog), LAPI scaled to 3, **DB on PostgreSQL** (migrated from MySQL), flush config: max_items=10000/max_age=7d/agents_autodelete=30d, DECISION_DURATION=168h in blocklist CronJob | +| CrowdSec | Pin version, disable Metabase when not needed (CPU hog), LAPI scaled to 3, **DB on PostgreSQL** (migrated from MySQL), flush config: max_items=10000/max_age=7d/agents_autodelete=30d, DECISION_DURATION=168h in blocklist CronJob. **Enforcement is out-of-band, NOT a Traefik plugin** (the Yaegi `crowdsec-bouncer-traefik-plugin` was dead on Traefik 3.7.5 and removed): `cs-firewall-bouncer` DaemonSet drops in-kernel via nftables on direct hosts (bouncer key `firewall`, v0.0.34 binary fetched at runtime, hostNetwork+NET_ADMIN, `stacks/crowdsec/modules/crowdsec/firewall_bouncer.tf`); `crowdsec-cf-sync` CronJob blocks at the CF edge for proxied hosts (bouncer key `kvsync`, `stacks/rybbit/crowdsec_edge.tf`). Both fail open. See `docs/architecture/security.md` | | Frigate | GPU stall detection in liveness probe (inference speed check), high CPU | | Authentik | 3 server replicas + 2-replica embedded outpost (PG-backed sessions), PgBouncer in front of PostgreSQL, strip auth headers before forwarding. **`authentik.*` Helm values are INERT** (existingSecret skips chart env rendering) — tune via `server.env`/`worker.env` in `modules/authentik/values.yaml`. Single-screen login (password embedded in identification stage); all first-party OIDC apps use implicit consent (2026-06-10). `/static` ingress carve-out serves assets with immutable Cache-Control. | | Kyverno | failurePolicy=Ignore to prevent blocking cluster, pin chart version | diff --git a/docs/architecture/networking.md b/docs/architecture/networking.md index e2c0ac2d..4659038a 100644 --- a/docs/architecture/networking.md +++ b/docs/architecture/networking.md @@ -4,7 +4,7 @@ Last updated: 2026-04-19 (WS E — Kea DHCP pushes dual DNS per subnet; Kea DDNS ## Overview -The homelab network is built on a dual-VLAN architecture with pfSense providing gateway services, Technitium for internal DNS, and Cloudflare for external DNS. Traefik serves as the Kubernetes ingress controller with a comprehensive middleware chain including CrowdSec bot protection, Authentik forward-auth, and rate limiting. All HTTP traffic flows through Cloudflared tunnels, avoiding the need for port forwarding or exposing public IPs. +The homelab network is built on a dual-VLAN architecture with pfSense providing gateway services, Technitium for internal DNS, and Cloudflare for external DNS. Traefik serves as the Kubernetes ingress controller with a middleware chain of anti-AI bot-blocking, Authentik forward-auth, rate limiting, and retry. CrowdSec IP-reputation enforcement is **out-of-band** (not a Traefik hop): banned IPs are dropped in-kernel via nftables on direct hosts and blocked at the Cloudflare edge on proxied hosts (see `docs/architecture/security.md`). All HTTP traffic flows through Cloudflared tunnels, avoiding the need for port forwarding or exposing public IPs. ## Architecture Diagram @@ -16,12 +16,14 @@ graph TB Traefik[Traefik Ingress
3 replicas + PDB] subgraph "Middleware Chain" - CS[CrowdSec Bouncer
fail-open] + AntiAI[Anti-AI bot-block
fail-open] Auth[Authentik Forward-Auth
3 replicas + PDB] RL[Rate Limiter
429 response] Retry[Retry
2 attempts, 100ms] end + CSdrop[CrowdSec drop
nftables / CF edge
out-of-band, pre-Traefik] + subgraph "Proxmox Host (eno1)" vmbr0[vmbr0 Bridge
192.168.1.127/24] vmbr1[vmbr1 Internal
VLAN-aware] @@ -53,8 +55,9 @@ graph TB Internet -->|DNS query| CF CF -->|CNAME to tunnel| CFD CFD --> Traefik - Traefik --> CS - CS --> Auth + CSdrop -.->|banned IPs dropped before Traefik| Traefik + Traefik --> AntiAI + AntiAI --> Auth Auth --> RL RL --> Retry Retry --> Service @@ -82,7 +85,7 @@ graph TB | Cloudflare DNS | SaaS | External | ~50 public domains under viktorbarzin.me | | Cloudflared | Container | K8s (3 replicas) | Tunnel ingress, replaces port forwarding | | Traefik | Helm chart | K8s (3 replicas + PDB) | Ingress controller, HTTP/3 enabled | -| CrowdSec | Helm chart | K8s (LAPI: 3 replicas) | Bot protection, fail-open bouncer | +| CrowdSec | Helm chart | K8s (LAPI: 3 replicas) | IP reputation. Out-of-band enforcement: `cs-firewall-bouncer` DaemonSet (in-kernel nftables drop, direct hosts) + Cloudflare edge WAF rule (proxied hosts). Fail-open | | Authentik | Helm chart | K8s (3 replicas + PDB) | SSO, forward-auth middleware | | MetalLB | v0.15.3 Helm chart | K8s | LoadBalancer IPs (10.0.20.200-10.0.20.220), all services on 10.0.20.200 | | Registry Cache | Container | 10.0.20.10 | Pull-through for docker.io:5000, ghcr.io:5010 | @@ -208,24 +211,31 @@ VMs tag traffic on vmbr1 to isolate workloads. pfSense bridges VLAN 20 to the up ### Ingress Flow +CrowdSec is **not** a step in this chain — banned IPs are dropped before the +request ever reaches Traefik (Cloudflare edge WAF rule on proxied hosts; host +nftables on direct hosts). The flow below is for a request that survives that +out-of-band gate. + ```mermaid sequenceDiagram participant Client - participant Cloudflare + participant CFedge as Cloudflare (edge WAF: crowdsec_ban block) participant Cloudflared participant Traefik - participant CrowdSec + participant AntiAI participant Authentik participant RateLimit participant Retry participant Service participant Pod - Client->>Cloudflare: HTTPS request to blog.viktorbarzin.me - Cloudflare->>Cloudflared: Forward via tunnel (QUIC) + Client->>CFedge: HTTPS request to blog.viktorbarzin.me + Note over CFedge: banned IP → blocked here (proxied hosts) + CFedge->>Cloudflared: Forward via tunnel (QUIC) Cloudflared->>Traefik: HTTP to LoadBalancer IP - Traefik->>CrowdSec: Apply bouncer middleware - CrowdSec->>Authentik: If allowed, check auth (protected=true) + Note over Traefik: on direct hosts, banned IPs already dropped in-kernel (nftables forward hook) + Traefik->>AntiAI: anti-AI bot-block (fail-open) + AntiAI->>Authentik: If allowed, check auth (protected=true) Authentik->>RateLimit: If authenticated, check rate limit RateLimit->>Retry: If within limit, continue Retry->>Service: Forward to Service @@ -234,24 +244,27 @@ sequenceDiagram Service-->>Retry: Response Retry-->>RateLimit: Response RateLimit-->>Authentik: Response (strip auth headers) - Authentik-->>CrowdSec: Response - CrowdSec-->>Traefik: Response + Authentik-->>AntiAI: Response + AntiAI-->>Traefik: Response Traefik-->>Cloudflared: Response - Cloudflared-->>Cloudflare: Response via tunnel - Cloudflare-->>Client: HTTPS response + Cloudflared-->>CFedge: Response via tunnel + CFedge-->>Client: HTTPS response ``` ### Middleware Chain -Every ingress created by the `ingress_factory` module follows this chain: +CrowdSec IP-reputation enforcement is **not** in this chain — it is out-of-band +(host nftables on direct hosts; the Cloudflare edge WAF `crowdsec_ban` rule on +proxied hosts), so banned IPs never reach the chain and there is no per-request +CrowdSec hop. Every ingress created by the `ingress_factory` module follows this +Traefik chain: -1. **CrowdSec Bouncer**: Checks IP against threat database. **Fail-open** mode — if LAPI is unreachable, traffic passes through to prevent outages. +1. **Anti-AI bot-block** (`ai-bot-block` ForwardAuth, on by default via `ingress_factory`): blocks/tarpits known AI crawlers. **Fail-open** (currently a no-op `return 200` — poison-fountain scaled to 0; see `docs/architecture/security.md`). 2. **Authentik Forward-Auth** (if `protected = true`): SSO authentication via OIDC. Non-authenticated users are redirected to login. Auth headers are stripped before forwarding to backend. 3. **Rate Limiting**: Per-IP throttling. Returns **429 Too Many Requests** (not 503) when limit exceeded. Default is `rate-limit` (average 10 req/s, burst 50). Services whose clients legitimately burst harder get a dedicated middleware via `skip_default_rate_limit = true` + `extra_middlewares`: Immich (`immich-rate-limit`, 1000/20000, photo uploads) and ActualBudget (`actualbudget-rate-limit`, 50/300 — the Actual web app boots with ~70 parallel asset/migration revalidations; the default burst 429'd the tail and stalled every page load). 4. **Retry**: 2 attempts with 100ms delay on transient failures (5xx errors, connection errors). Additional middleware: -- **Anti-AI**: On by default via `ingress_factory`. Blocks common AI crawler user-agents. - **HTTP/3 (QUIC)**: Enabled globally on Traefik. ### Entrypoint Transport Timeouts @@ -348,7 +361,7 @@ Containerd on all K8s nodes uses `hosts.toml` to redirect pulls to the local cac | pfSense | `stacks/pfsense/` | VM + cloud-init config | | Technitium | `stacks/technitium/` | Deployment, Service, PVC | | Traefik | `stacks/platform/` (sub-module) | Helm release, IngressRoute CRDs | -| CrowdSec | `stacks/platform/` (sub-module) | Helm release, LAPI + bouncer | +| CrowdSec | `stacks/crowdsec/` (+ edge in `stacks/rybbit/`) | Helm release, LAPI + agent; `cs-firewall-bouncer` DaemonSet (nftables, direct hosts) + Cloudflare edge sync (proxied hosts) | | Authentik | `stacks/authentik/` | Helm release, ingress, OIDC configs | | MetalLB | `stacks/platform/` (sub-module) | Helm release, IPAddressPool | | Cloudflared | `stacks/cloudflared/` | Deployment (3 replicas), tunnel config; runs `--no-autoupdate` (in-place self-updates exited the pods and severed all tunnel WebSockets, 2026-06-09/10) | @@ -436,13 +449,30 @@ Containerd on all K8s nodes uses `hosts.toml` to redirect pulls to the local cac **Decision**: Technitium handles internal `.lan` domains with near-zero latency. Cloudflare handles public domains with global DNS. K8s nodes use Technitium as primary, which forwards non-.lan queries to Cloudflare. -### Why Fail-Open on CrowdSec Bouncer? +### Why CrowdSec Enforcement Is Out-of-Band (and Fails Open) -**Alternatives considered**: -1. **Fail-closed**: Maximum security, but LAPI downtime blocks all traffic. -2. **Redundant LAPI**: Already scaled to 3 replicas, but resource pressure can still cause outages. +CrowdSec used to enforce inline as a Traefik middleware (the +`crowdsec-bouncer-traefik-plugin`). On Traefik 3.7.5 the Yaegi plugin handler was +never invoked, so it enforced nothing; the plugin was removed and enforcement +moved off the request path entirely (full history in +`docs/architecture/security.md`). It now runs on two surfaces: -**Decision**: Availability > strict bot blocking. CrowdSec LAPI is scaled to 3 replicas for resilience, but during cluster-wide resource exhaustion (e.g., memory pressure), bouncer falls back to allowing traffic. This prevents a complete service outage due to a security add-on. +- **Direct hosts** → `cs-firewall-bouncer` DaemonSet drops banned IPs in the host + nftables, in **both the `input` and `forward` hooks**. The `forward` hook is + the load-bearing one: with Traefik on a dedicated LB IP at + `externalTrafficPolicy=Local`, client packets are DNAT'd to the Traefik **pod** + and transit the node's `forward` chain (not `input`) — which is exactly why the + ingress must preserve the **real client IP** end-to-end (ETP=Local + PROXY-v2 + for IPv6; see the Traefik LB IP and IPv6 ingress notes above). Without the real + client IP the firewall-bouncer (and the CF edge rule) would have nothing to + match on. +- **Proxied hosts** → a Cloudflare edge WAF rule (`ip.src in $crowdsec_ban`) fed + by the `crowdsec-cf-sync` CronJob. + +Both **fail open**: if LAPI is unreachable, the firewall-bouncer simply stops +receiving new decisions (existing drops persist) and the CF sync skips a run — +neither ever blocks legitimate traffic. Availability > strict bot blocking, and +out-of-band enforcement adds **zero per-request latency** (no Traefik hop). ### Why HTTP/3 (QUIC)? @@ -473,9 +503,10 @@ Containerd on all K8s nodes uses `hosts.toml` to redirect pulls to the local cac **Symptoms**: All ingress routes return 503, Traefik dashboard shows no backends available. -**Diagnosis**: Middleware chain is blocking traffic. Check: -1. Authentik status: `kubectl get pod -n authentik` -2. CrowdSec LAPI status: `kubectl get pod -n crowdsec` +**Diagnosis**: Middleware chain is blocking traffic. (CrowdSec is **not** in the +chain — a CrowdSec/LAPI outage cannot cause 503s; it only stops new bans.) Check: +1. Authentik status: `kubectl get pod -n authentik` (ForwardAuth fails closed if the auth server is unreachable) +2. `bot-block-proxy` status: `kubectl get pod -n traefik -l app=bot-block-proxy` (anti-AI ForwardAuth target — also fails closed if down) 3. Traefik logs: `kubectl logs -n kube-system deploy/traefik` **Fix**: If Authentik is down and ingress uses forward-auth, pods won't pass health checks. Scale Authentik to 3 replicas or temporarily disable forward-auth middleware. diff --git a/docs/architecture/security.md b/docs/architecture/security.md index f8afd5e4..7d3043ea 100644 --- a/docs/architecture/security.md +++ b/docs/architecture/security.md @@ -2,40 +2,50 @@ ## Overview -The homelab implements defense-in-depth security at the application layer (L7) using CrowdSec for threat intelligence and IP reputation, Kyverno for policy enforcement and resource governance, and a 3-layer anti-AI scraping defense (reduced from 5 in April 2026 after removing the rewrite-body plugin). All security components operate in graceful degradation mode (fail-open) to prevent cascading failures. Security policies are deployed in audit mode first, then selectively enforced after validation. +The homelab implements defense-in-depth security using CrowdSec for threat intelligence and IP reputation, Kyverno for policy enforcement and resource governance, and a 3-layer anti-AI scraping defense (reduced from 5 in April 2026 after removing the rewrite-body plugin). CrowdSec enforcement is **out-of-band** (not a per-request Traefik hop — see the CrowdSec section): banned IPs are dropped in-kernel via nftables on direct hosts, and blocked at the Cloudflare edge on proxied hosts, so enforcement adds **zero per-request latency**. All security components fail open (a CrowdSec outage stops new bans but never blocks legitimate traffic). Security policies are deployed in audit mode first, then selectively enforced after validation. ## Architecture Diagram +CrowdSec enforcement is out-of-band (NOT an inline Traefik middleware hop). The +Traefik request chain is anti-AI → Authentik ForwardAuth → rate-limit → retry; +CrowdSec drops banned IPs *before* (direct hosts) or *off* (proxied hosts) that +chain entirely. + ```mermaid -graph LR +graph TB Internet[Internet] - CF[Cloudflare WAF] + + subgraph "Proxied hosts (orange-cloud)" + CFedge[Cloudflare edge
WAF rule: ip.src in $crowdsec_ban → block] + end + subgraph "Direct hosts (grey-cloud / internal)" + NFT[Host nftables
table crowdsec/crowdsec6
drop in input + forward] + end + Tunnel[Cloudflared Tunnel] - CrowdSec[CrowdSec Bouncer
Traefik Plugin] - AntiAI[Anti-AI Check
poison-fountain] - ForwardAuth[Authentik ForwardAuth] - RateLimit[Rate Limit Middleware] - Retry[Retry Middleware
2 attempts, 100ms] + Traefik[Traefik
anti-AI → Authentik → rate-limit → retry] Backend[Backend Service] LAPI[CrowdSec LAPI
3 replicas] - Agent[CrowdSec Agent] + Agent[CrowdSec Agent
parses Traefik logs] + FWB[cs-firewall-bouncer
DaemonSet, every node] + CFsync[crowdsec-cf-sync
CronJob, every 2 min] - Internet -->|1| CF - CF -->|2| Tunnel - Tunnel -->|3| CrowdSec - CrowdSec -.->|Query| LAPI - Agent -.->|Report| LAPI - CrowdSec -->|4. Pass/Block| AntiAI - AntiAI -->|5. Human/Bot| ForwardAuth - ForwardAuth -->|6. Authenticated| RateLimit - RateLimit -->|7. Under Limit| Retry - Retry -->|8. Success/Retry| Backend + Internet -->|proxied| CFedge + Internet -->|direct| NFT + CFedge -->|allowed| Tunnel + Tunnel --> Traefik + NFT -->|allowed| Traefik + Traefik --> Backend - style CrowdSec fill:#f9f,stroke:#333 - style AntiAI fill:#ff9,stroke:#333 - style ForwardAuth fill:#9f9,stroke:#333 - style RateLimit fill:#99f,stroke:#333 + Agent -.->|report| LAPI + LAPI -.->|all decisions incl. CAPI| FWB + FWB -.->|program drop rules| NFT + LAPI -.->|ban/captcha decisions, CAPI excluded| CFsync + CFsync -.->|push IP list| CFedge + + style CFedge fill:#f9f,stroke:#333 + style NFT fill:#f9f,stroke:#333 ``` ## Components @@ -44,7 +54,8 @@ graph LR |-----------|---------|----------|---------| | CrowdSec LAPI | Pinned | `stacks/crowdsec/` | Local API, threat intelligence aggregation (3 replicas) | | CrowdSec Agent | Pinned | `stacks/crowdsec/` | Log parser, scenario detection | -| CrowdSec Traefik Bouncer | Plugin | Traefik config | Plugin-based IP reputation check | +| cs-firewall-bouncer | v0.0.34 | `stacks/crowdsec/modules/crowdsec/firewall_bouncer.tf` | In-kernel nftables drop on every node (DIRECT hosts). Bouncer key `firewall` | +| crowdsec-cf-sync | — | `stacks/rybbit/crowdsec_edge.tf` | LAPI→Cloudflare-IP-List sync CronJob (PROXIED hosts). Bouncer key `kvsync` | | Kyverno | Pinned chart | `stacks/kyverno/` | Policy engine for K8s admission control | | poison-fountain | Latest | `stacks/poison-fountain/` | Anti-AI bot detection and tarpit service | | cert-manager/certbot | - | `stacks/cert-manager/` | TLS certificate management | @@ -54,11 +65,15 @@ graph LR ### Request Security Layers -Every incoming request passes through 6 security layers: +CrowdSec IP-reputation enforcement happens **before** a request reaches the +Traefik chain (banned IPs are dropped in-kernel on direct hosts, or blocked at +the Cloudflare edge on proxied hosts — see CrowdSec Threat Intelligence below). +A request that survives that out-of-band gate then passes through the Traefik +middleware chain: -1. **Cloudflare WAF** - DDoS protection, bot detection, firewall rules (external) -2. **Cloudflared Tunnel** - Zero Trust tunnel, hides origin IP -3. **CrowdSec Bouncer** - IP reputation check against LAPI (fail-open on error) +1. **Cloudflare WAF / edge** - DDoS protection, bot detection, firewall rules incl. the CrowdSec `crowdsec_ban` block rule (proxied hosts only) +2. **Cloudflared Tunnel** - Zero Trust tunnel, hides origin IP (proxied hosts) +3. **CrowdSec out-of-band drop** - nftables on direct hosts; *not* a Traefik hop (zero per-request latency) 4. **Anti-AI Scraping** - 3-layer bot defense (optional per service, updated 2026-04-17) 5. **Authentik ForwardAuth** - Authentication check (if `protected = true`) 6. **Rate Limiting** - Per-source IP rate limits (returns 429 on breach) @@ -80,58 +95,71 @@ CrowdSec operates in a hub-and-agent model: - Reports malicious IPs to LAPI - Shares threat intel with CrowdSec community (anonymized) -**Traefik Bouncer Plugin** (`crowdsec-bouncer-traefik-plugin`, `stacks/traefik/modules/traefik/middleware.tf`): -- Integrated as Traefik middleware (in the default ingress chain) -- Queries LAPI for IP reputation on each request -- **Registered with LAPI** via `BOUNCER_KEY_traefik` env on the LAPI container - (`stacks/crowdsec/modules/crowdsec/values.yaml`), seeded from the same Vault key - the middleware presents (`ingress_crowdsec_api_key`). **Before 2026-06-19 the - bouncer was never registered → LAPI returned 403 → the plugin failed open and - enforced nothing (no bans, no captcha).** The seed re-registers automatically on - every LAPI start, so a DB wipe (e.g. the MySQL→PostgreSQL migration that lost the - original registration) can't silently disable enforcement again. -- **Fail-open mode**: If LAPI unreachable, allows traffic (graceful degradation) -- **Only sees non-proxied (direct) apps' real client IPs** (ETP=Local). Proxied - apps arrive from cloudflared's pod IP (in `clientTrustedIPs`) and are bypassed — - extending enforcement to proxied apps needs `forwardedHeadersTrustedIPs` (future). -- Honours two LAPI remediation types (profiles in `stacks/crowdsec/modules/crowdsec/values.yaml`): - - **`ban`** → HTTP 403 (serious attacks: CVE exploits, scanners, brute force) - - **`captcha`** → **Cloudflare Turnstile challenge** so the flagged user can - self-unblock (lower-severity abuse: `http-429-abuse`, `http-403-abuse`, - `http-crawl-non_statics`, `http-sensitive-files`). The plugin is configured - with `captchaProvider=turnstile` + the widget keys; the `captcha.html` - template is mounted into the Traefik pod at `/captcha`. The widget is - Terraform-managed in `stacks/traefik/main.tf` - (`cloudflare_turnstile_widget.crowdsec_captcha`, scoped to `viktorbarzin.me` - so it covers every subdomain). **Before 2026-06-19 no captcha provider was - configured, so `captcha` decisions silently degraded to a 403 ban** — users - had no way to self-unblock; wiring Turnstile fixed that. +Enforcement is split across **two out-of-band surfaces**, neither of which adds +any per-request latency. (See "Why the Traefik bouncer plugin was removed" below +for the supersession history — there is no longer an inline Traefik bouncer.) -**Cloudflare Edge Enforcement for proxied hosts** (`stacks/rybbit/crowdsec_edge.tf` + `lapi_kv_sync.py`): -- Proxied (orange-cloud) hosts terminate at the Cloudflare edge, so the in-cluster - bouncer above never decides on them. Edge enforcement instead syncs LAPI - decisions into **one Cloudflare account IP List (`crowdsec_ban`)** + a single - **zone-scoped WAF custom rule** blocking `(ip.src in $crowdsec_ban)` across every - proxied host. CronJob `crowdsec-cf-sync` (rybbit ns, every 2 min) reconciles it. -- **BAN-ONLY (2026-06-20):** only `type=ban` decisions sync to the edge. `captcha` - decisions are deliberately NOT pushed — the CF account allows only ONE Rules List - with a single block action, so folding captcha in would hard-block a soft - challenge on every proxied host. (Before 2026-06-20 captcha was downgraded to a - hard block at the edge.) -- **Auth carve-out (2026-06-20):** the WAF rule excludes `authentik.viktorbarzin.me` - + `public-auth.viktorbarzin.me` (`… and not (http.host in {…})`), and the - Authentik UI ingress sets `exclude_crowdsec = true` for the in-cluster bouncer. A - CrowdSec hit must never wall a user out of the login / WebAuthn flow they - authenticate through; auth keeps `traefik-rate-limit` for brute-force protection. -- **⚠️ Currently NON-FUNCTIONAL (known issue, pre-existing since the 2026-06-20 - rollout):** `crowdsec-cf-sync` fails every run — `cf_list_items()` pagination - gets CF `HTTP 400 code 10027 "invalid or expired cursor"`, so the list never - populates (`num_items=0`) and the edge rule blocks nothing. LAPI also returns - ~31k ban IPs, likely exceeding CF IP-List capacity even once pagination is fixed. - **Edge enforcement for proxied hosts is therefore inert pending a fix** (the - in-cluster bouncer still protects direct apps; the auth carve-out is correct - regardless). Fix needs: (1) correct CF cursor pagination, (2) a capacity strategy - for the ban set. +**Surface 1 — DIRECT (non-Cloudflare-proxied) hosts → in-kernel nftables drop** +(`cs-firewall-bouncer` DaemonSet, `stacks/crowdsec/modules/crowdsec/firewall_bouncer.tf`): +- Runs on **every node** (no nodeSelector). Programs the HOST nftables — `table ip + crowdsec` / `table ip6 crowdsec6` — with drop rules in **both the `input` AND + the `forward` hooks**. The `forward` hook is required because Traefik is a + LoadBalancer with `externalTrafficPolicy=Local`: client traffic is DNAT'd to the + Traefik **pod** and transits the node's `forward` hook (not `input`) with the + real client IP preserved. Chains use `policy accept` (only set members drop — + it can never blackhole normal traffic). +- Pulls **all** decisions from LAPI, **including the CAPI community blocklist + (~31k IPs)**. Packets from banned IPs are dropped **in-kernel before reaching + Traefik** → zero per-request hops, no Traefik involvement at all. +- **Packaging**: cs-firewall-bouncer publishes no container image, so the + **v0.0.34** static binary is fetched at runtime by an initContainer onto a + `debian:bookworm-slim` runtime container. Needs `hostNetwork` + + `NET_ADMIN`/`NET_RAW` to talk netlink directly. Registered bouncer key: + **`firewall`**. +- **Fail-open**: if LAPI is unreachable it just stops receiving new decisions + (existing drop rules persist); it never blocks legitimate traffic. + +**Surface 2 — PROXIED (Cloudflare orange-cloud) hosts → Cloudflare edge block** +(`stacks/rybbit/crowdsec_edge.tf` + `lapi_kv_sync.py`): +- Proxied hosts terminate at the Cloudflare edge, so a host-level nftables drop + would never see them. Enforcement is instead a single Cloudflare Rules List + **`crowdsec_ban`** + a zone-scoped WAF custom rule `(ip.src in $crowdsec_ban)` + → **block** action, which covers every proxied host in the zone. +- Fed by the **`crowdsec-cf-sync` CronJob** (namespace `rybbit`, every 2 min, + pure-stdlib Python in a ConfigMap). It pulls local **ban/captcha ip-scoped** + decisions and pushes them into the CF list, but **EXCLUDES the ~31k CAPI + community blocklist** — that set is far too large for a CF Rules List (the CF + account hard-limits to **one** list), and CAPI is already covered in-kernel on + direct hosts and by Cloudflare's own managed protections on proxied hosts. + Registered bouncer key: **`kvsync`**. +- **Block-only**: the single-list limit precludes a separate + captcha/managed-challenge list, so both ban and captcha decisions are enforced + as a plain block at the edge. +- **Auth carve-out:** the WAF rule excludes `authentik.viktorbarzin.me` + + `public-auth.viktorbarzin.me` (`… and not (http.host in {…})`). A CrowdSec hit + must never wall a user out of the login / WebAuthn flow they authenticate + through; auth keeps `traefik-rate-limit` for brute-force protection. + +**Whitelist** (`stacks/crowdsec/whitelist.yaml`): a CrowdSec whitelist covers +RFC1918 + the tailnet + internal CIDRs (plus one specific external IP), so +internal users are never enforced. Internal access uses split-horizon DNS +straight to Traefik, and direct internal clients are RFC1918 — both whitelisted. + +#### Why the Traefik bouncer plugin was removed + +Enforcement used to run as an inline Traefik middleware — the +`crowdsec-bouncer-traefik-plugin` (Yaegi/Lua), which queried LAPI on every +request and could serve a Cloudflare Turnstile captcha for soft remediations. +On **Traefik 3.7.5 the Yaegi handler was never invoked**, so the bouncer was +registered but enforced **nothing** despite appearing healthy. Rather than chase +the Yaegi runtime, the whole plugin path was **removed** (2026-06): the plugin +static config + initContainer download, the `crowdsec` Middleware CRD, the +`captcha.html` template + its ConfigMap and volume mount, and the Cloudflare +Turnstile widget (`cloudflare_turnstile_widget.crowdsec_captcha`). It was +replaced by the two out-of-band surfaces above, which add zero per-request +latency and fail open. (The earlier `crowdsec-cf-sync` cursor-pagination / +IP-List-capacity issues are also moot now that CAPI is excluded from the edge +list and dropped in-kernel instead.) **Metabase** (disabled by default): - Dashboard for CrowdSec analytics @@ -377,10 +405,12 @@ Beads: `code-8ywc` W1.6 + W1.7. **Status: planned.** | Path | Purpose | |------|---------| -| `stacks/crowdsec/` | CrowdSec LAPI, agent, bouncer config | +| `stacks/crowdsec/` | CrowdSec LAPI, agent config + `whitelist.yaml` | +| `stacks/crowdsec/modules/crowdsec/firewall_bouncer.tf` | cs-firewall-bouncer DaemonSet (in-kernel nftables drop, direct hosts) | +| `stacks/rybbit/crowdsec_edge.tf` + `lapi_kv_sync.py` | Cloudflare IP-List + WAF block rule + LAPI→CF sync CronJob (proxied hosts) | | `stacks/kyverno/` | Kyverno deployment + policies | | `stacks/poison-fountain/` | Anti-AI service + CronJob | -| `stacks/platform/modules/traefik/middleware.tf` | Security middleware definitions | +| `stacks/traefik/modules/traefik/middleware.tf` | Security middleware definitions (no longer includes a CrowdSec bouncer) | | `stacks/platform/modules/ingress_factory/` | Per-service security toggles | ### Vault Paths @@ -490,7 +520,11 @@ spec: **Fix**: 1. Check LAPI decisions: `kubectl exec -it crowdsec-lapi-0 -- cscli decisions list` 2. Remove ban: `kubectl exec -it crowdsec-lapi-0 -- cscli decisions delete --ip ` -3. Whitelist if needed: Add to `stacks/crowdsec/whitelist.yaml` + — the in-kernel drop clears as soon as `cs-firewall-bouncer` reconciles (direct + hosts); for proxied hosts the `crowdsec-cf-sync` CronJob removes it from the + `crowdsec_ban` CF list within ~2 min. +3. Whitelist if needed: Add to `stacks/crowdsec/whitelist.yaml` (RFC1918 + tailnet + + internal CIDRs are already whitelisted, so internal clients are never banned). ### Kyverno Policy Blocking Deployment -- 2.49.1