viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

rybbit: proxied CrowdSec enforcement via Cloudflare IP Lists + WAF rule

Browse source 
Replaces the Worker+KV approach (which only covered the ~27 routed hosts) with a
zone-wide mechanism that covers ALL proxied hosts: two CF account IP Lists
(crowdsec_ban, crowdsec_captcha) + one zone WAF custom rule that blocks
`(ip.src in $crowdsec_ban)` and managed-challenges `(ip.src in $crowdsec_captcha)`.
No per-request Worker, no cookie machinery — the rybbit Worker stays
analytics-only. lapi_kv_sync.py now full-reconciles the two lists from LAPI
(fail-safe: a LAPI blip skips the run and freezes the last-known-good block set;
serializes CF bulk ops since CF allows one pending op per account). A
least-privilege CF API token (Account Filter Lists Edit) is minted in TF.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-06-20 09:18:33 +00:00
parent 7e646e1c7c
commit cc4bfb593b
2 changed files with 542 additions and 114 deletions
Download patch file Download diff file Expand all files Collapse all files

357
stacks/rybbit/lapi_kv_sync.py
View file

@ -1,48 +1,84 @@
#!/usr/bin/env python3
"""Sync CrowdSec LAPI decisions -> Cloudflare Workers KV.
"""Sync CrowdSec LAPI decisions -> two Cloudflare account IP Lists.
The rybbit edge Worker enforces CrowdSec bans/captchas for Cloudflare-PROXIED
hosts by reading an edge-local KV entry (`ip:<addr>` -> `ban`|`captcha`) on each
request. This job is the control plane that keeps that KV in sync with LAPI:
the request path NEVER calls LAPI (no per-request hop) — exactly the nginx
"tail logs -> inject rules" model, just projected onto Workers KV instead of
nftables.
Cloudflare-PROXIED hosts terminate at the CF edge, so the in-cluster CrowdSec
bouncer (which keys on the client IP Traefik sees) never decides on them. We
push the decisions into the edge instead: a zone-scoped WAF custom rule blocks
`(ip.src in $crowdsec_ban)` and managed-challenges `(ip.src in $crowdsec_captcha)`
across EVERY proxied host in the zone. This job is the control plane that keeps
those two IP Lists in sync with LAPI.
(Filename kept as lapi_kv_sync.py for path/ConfigMap continuity with the prior
Workers-KV design; it no longer touches KV — it reconciles CF Rules Lists.)
Design notes:
* Pure Python stdlib (no pip/apk at runtime) — runs on stock python:3.12-alpine
mounted from a ConfigMap, the alert_digest pattern (avoids the disk
anti-pattern of installing packages every run).
* FULL RECONCILE each run: read the complete current decision set from LAPI,
compute the desired KV state, then upsert present keys and delete stale ones.
This makes an un-ban (cscli decisions delete) clear from the edge within one
interval instead of lingering until the original TTL — important for getting
a false-positive un-blocked fast.
* FAIL-SAFE: if LAPI can't be read, we SKIP the run (leave KV untouched) rather
than wipe every ban. Existing KV entries then simply expire by their TTL, so
a LAPI outage degrades toward fail-OPEN, never toward a stale all-block.
* Only Ip-scope ban/captcha decisions are projected. Range-scope and other
remediations are ignored (the Worker keys on a single IP).
mounted from a ConfigMap, the alert_digest pattern.
* FULL RECONCILE each run: read the complete decision set from LAPI, partition
into ban / captcha desired sets, then for each list compute add (desired -
existing) and remove (existing - desired) and apply both. An IP listed for
BOTH ban and captcha is placed in BAN ONLY (ban wins; the WAF rule order
also blocks-before-challenges as belt-and-braces). A `cscli decisions
delete` therefore clears from the edge within one interval (<=2 min).
* FAIL-SAFE on LAPI: if LAPI can't be read we SKIP the run (lists untouched,
exit 0). A LAPI outage thus freezes the edge state rather than wiping every
ban — degrade toward the last-known-good block set, never toward all-block
or a thundering un-ban. (Decisions linger only until the next successful
sync, not their TTL — we reconcile to LAPI truth, we don't expire entries.)
* FAIL-LOUD on Cloudflare: any CF API error is logged and the job exits
non-zero so the failure is visible (CronJob backoff + missing success
metric + the next run retries).
Cloudflare Rules-Lists API (account-level IP list items), verified against the
official API reference (developers.cloudflare.com, 2026):
* GET /accounts/{acct}/rules/lists/{list}/items -> paginated; next page
cursor at result_info.cursors.after, passed back as ?cursor=. Each
item = {"id","ip","created_on",...}.
* POST /accounts/{acct}/rules/lists/{list}/items -> body JSON ARRAY
[{"ip":"1.2.3.4"},...]. APPENDS/upserts (does NOT replace the list).
ASYNCHRONOUS: returns {"result":{"operation_id":...}}.
* DELETE /accounts/{acct}/rules/lists/{list}/items -> body {"items":[{"id":
"<item_id>"},...]} (delete by item id, not ip). ASYNCHRONOUS.
* GET /accounts/{acct}/rules/lists/bulk_operations/{op_id} -> status in
{pending,running,completed,failed} (failed carries `error`).
ASYNC HANDLING: Cloudflare allows only ONE pending bulk operation per ACCOUNT.
So we must NOT fire add+delete (or both lists) concurrently — we serialize and
poll each operation_id to a terminal state (short bounded timeout) before the
next mutation. If a poll times out we stop mutating for this run and report
partial success (the next 2-min run reconciles the rest); we never abandon an
in-flight op and immediately issue another (that would 409/reject).
"""
import json
import os
import re
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
LAPI_URL = os.environ.get("LAPI_URL", "http://crowdsec-service.crowdsec.svc.cluster.local:8080").rstrip("/")
LAPI_URL = os.environ.get(
"LAPI_URL", "http://crowdsec-service.crowdsec.svc.cluster.local:8080"
).rstrip("/")
LAPI_KEY = os.environ["LAPI_KEY"] # kvsync bouncer key, registered in LAPI
CF_ACCOUNT_ID = os.environ["CF_ACCOUNT_ID"]
CF_NAMESPACE_ID = os.environ["CF_KV_NAMESPACE_ID"]
CF_API_TOKEN = os.environ["CF_API_TOKEN"] # scoped: Workers KV Storage:Edit
CF_API_TOKEN = os.environ["CF_API_TOKEN"] # scoped: Account Filter Lists Edit
CF_BAN_LIST_ID = os.environ["CF_BAN_LIST_ID"]
CF_CAPTCHA_LIST_ID = os.environ["CF_CAPTCHA_LIST_ID"]
PUSHGATEWAY = os.environ.get("PUSHGATEWAY_URL", "").rstrip("/") # optional
KEY_PREFIX = "ip:"
# CrowdSec remediation type -> KV value the Worker understands.
TYPE_MAP = {"ban": "ban", "captcha": "captcha"}
CF_API = "https://api.cloudflare.com/client/v4"
MIN_TTL = 60 # Cloudflare KV minimum expiration_ttl (seconds)
# Cloudflare item objects expose the ip differently between list and create
# responses; for an IP-kind list each GET item carries a top-level "ip".
# Batch sizes: no official per-request cap is documented, so keep batches
# generous but bounded (well under the global 1200 req / 5 min limit).
BATCH = 1000
# Async op polling: 1 pending bulk op per account, so poll to terminal state.
POLL_TIMEOUT = 25 # seconds to wait for one bulk op (the run has ~110s budget)
POLL_INTERVAL = 1.0
class CFError(Exception):
"""Cloudflare API failure -> job should exit non-zero (fail loud)."""
def _req(url, *, method="GET", headers=None, data=None, timeout=20):
@ -52,136 +88,229 @@ def _req(url, *, method="GET", headers=None, data=None, timeout=20):
return json.loads(body) if body else None
def parse_duration_seconds(dur):
"""Parse a Go duration string (e.g. '167h59m51.5s') into whole seconds.
LAPI returns the REMAINING duration of each decision here. We floor to the
second and clamp to Cloudflare's 60s minimum so the edge entry outlives the
next sync interval.
"""
if not dur:
return MIN_TTL
dur = dur.strip().lstrip("+")
if dur.startswith("-"): # already expired; give it the floor and move on
return MIN_TTL
total = 0.0
for value, unit in re.findall(r"(\d+(?:\.\d+)?)(h|m|s|ms|us|µs|ns)", dur):
v = float(value)
total += {"h": 3600, "m": 60, "s": 1, "ms": 1e-3, "us": 1e-6, "µs": 1e-6, "ns": 1e-9}[unit] * v
return max(MIN_TTL, int(total))
def _cf(url, *, method="GET", payload=None, timeout=20):
"""Call the CF API with the bearer token; raise CFError on any failure."""
headers = {"Authorization": f"Bearer {CF_API_TOKEN}"}
data = None
if payload is not None:
headers["Content-Type"] = "application/json"
data = json.dumps(payload).encode()
try:
res = _req(url, method=method, headers=headers, data=data, timeout=timeout)
except urllib.error.HTTPError as e:
detail = ""
try:
detail = e.read().decode(errors="replace")[:500]
except Exception:
pass
raise CFError(f"{method} {url} -> HTTP {e.code} {detail}") from e
except urllib.error.URLError as e:
raise CFError(f"{method} {url} -> {e}") from e
if res is not None and not res.get("success", True):
raise CFError(f"{method} {url} -> not success: {res.get('errors')}")
return res
# --------------------------------------------------------------------------- #
# LAPI
# --------------------------------------------------------------------------- #
def fetch_decisions():
"""Return desired KV state {('ip:<addr>'): (value, ttl_seconds)} from LAPI.
"""Return (ban_set, captcha_set) of IPs from LAPI.
Raises on transport/HTTP error so the caller can SKIP the run (fail-safe).
Only scope=="ip" decisions are projected (the WAF rule keys on ip.src). An
IP appearing in both ban and captcha is placed in BAN only. Raises on
transport/HTTP error so the caller can SKIP the run (fail-safe).
"""
out = {}
data = _req(f"{LAPI_URL}/v1/decisions", headers={"X-Api-Key": LAPI_KEY, "Accept": "application/json"})
data = _req(
f"{LAPI_URL}/v1/decisions",
headers={"X-Api-Key": LAPI_KEY, "Accept": "application/json"},
)
ban, captcha = set(), set()
for d in data or []:
if (d.get("scope") or "").lower() != "ip":
continue
value = TYPE_MAP.get((d.get("type") or "").lower())
if not value:
continue
ip = d.get("value")
if not ip:
continue
key = KEY_PREFIX + ip
ttl = parse_duration_seconds(d.get("duration"))
# If the same IP carries multiple decisions, ban beats captcha and the
# longest TTL wins.
if key in out:
prev_val, prev_ttl = out[key]
value = "ban" if "ban" in (value, prev_val) else value
ttl = max(ttl, prev_ttl)
out[key] = (value, ttl)
return out
dtype = (d.get("type") or "").lower()
if dtype == "ban":
ban.add(ip)
elif dtype == "captcha":
captcha.add(ip)
# other remediation types (e.g. throttle) are ignored
captcha -= ban # ban wins: never list the same IP in both
return ban, captcha
def cf_list_keys():
"""List existing `ip:` keys currently in the KV namespace (paginated)."""
keys = []
# --------------------------------------------------------------------------- #
# Cloudflare list items
# --------------------------------------------------------------------------- #
def cf_list_items(list_id):
"""Return {ip: item_id} for every item currently in the list (paginated)."""
out = {}
cursor = ""
while True:
url = f"{CF_API}/accounts/{CF_ACCOUNT_ID}/storage/kv/namespaces/{CF_NAMESPACE_ID}/keys?prefix={KEY_PREFIX}&limit=1000"
url = f"{CF_API}/accounts/{CF_ACCOUNT_ID}/rules/lists/{list_id}/items?per_page=1000"
if cursor:
url += f"&cursor={urllib.parse.quote(cursor)}"
res = _req(url, headers={"Authorization": f"Bearer {CF_API_TOKEN}"})
keys.extend(k["name"] for k in (res.get("result") or []))
cursor = (res.get("result_info") or {}).get("cursor") or ""
res = _cf(url)
for it in (res.get("result") or []):
ip = it.get("ip")
if ip:
out[ip] = it.get("id")
cursor = (((res.get("result_info") or {}).get("cursors") or {}).get("after")) or ""
if not cursor:
return keys
return out
def cf_bulk_put(items):
"""items: list of (key, value, ttl). Cloudflare bulk PUT (<=10000/call)."""
for i in range(0, len(items), 10000):
chunk = [{"key": k, "value": v, "expiration_ttl": t} for k, v, t in items[i:i + 10000]]
_req(
f"{CF_API}/accounts/{CF_ACCOUNT_ID}/storage/kv/namespaces/{CF_NAMESPACE_ID}/bulk",
method="PUT",
headers={"Authorization": f"Bearer {CF_API_TOKEN}", "Content-Type": "application/json"},
data=json.dumps(chunk).encode(),
)
def _wait_for_op(op_id):
"""Poll a bulk operation to a terminal state. Returns True if completed,
False if it timed out (still pending/running). Raises CFError if it failed.
We must reach a terminal state before issuing the next mutation: CF allows
only one pending bulk op per account, so firing another while this is
in-flight would be rejected.
"""
if not op_id:
return True
deadline = time.time() + POLL_TIMEOUT
url = f"{CF_API}/accounts/{CF_ACCOUNT_ID}/rules/lists/bulk_operations/{op_id}"
while time.time() < deadline:
res = _cf(url)
status = ((res.get("result") or {}).get("status") or "").lower()
if status == "completed":
return True
if status == "failed":
raise CFError(f"bulk op {op_id} failed: {(res.get('result') or {}).get('error')}")
time.sleep(POLL_INTERVAL)
print(f"[warn] bulk op {op_id} still {status or 'pending'} after {POLL_TIMEOUT}s; "
f"stopping further mutations this run (next run reconciles)", file=sys.stderr)
return False
def cf_bulk_delete(keys):
for i in range(0, len(keys), 10000):
_req(
f"{CF_API}/accounts/{CF_ACCOUNT_ID}/storage/kv/namespaces/{CF_NAMESPACE_ID}/bulk/delete",
def _op_id(res):
return ((res or {}).get("result") or {}).get("operation_id")
def cf_add_items(list_id, ips):
"""POST new IPs to the list (append). Returns the operation_id (async)."""
# If callers ever exceed one batch, each POST is a separate bulk op and the
# single-pending-op rule forces us to wait between them.
last_op = None
for i in range(0, len(ips), BATCH):
chunk = [{"ip": ip} for ip in ips[i : i + BATCH]]
res = _cf(
f"{CF_API}/accounts/{CF_ACCOUNT_ID}/rules/lists/{list_id}/items",
method="POST",
headers={"Authorization": f"Bearer {CF_API_TOKEN}", "Content-Type": "application/json"},
data=json.dumps(list(keys)).encode(),
payload=chunk,
)
last_op = _op_id(res)
if i + BATCH < len(ips): # more chunks coming -> serialize
if not _wait_for_op(last_op):
return last_op # timed out; bail (partial), next run continues
return last_op
def push_metrics(synced, deleted, ok):
def cf_delete_items(list_id, item_ids):
"""DELETE items by id. Returns the operation_id (async)."""
last_op = None
for i in range(0, len(item_ids), BATCH):
chunk = {"items": [{"id": iid} for iid in item_ids[i : i + BATCH]]}
res = _cf(
f"{CF_API}/accounts/{CF_ACCOUNT_ID}/rules/lists/{list_id}/items",
method="DELETE",
payload=chunk,
)
last_op = _op_id(res)
if i + BATCH < len(item_ids):
if not _wait_for_op(last_op):
return last_op
return last_op
def reconcile(label, list_id, desired):
"""Reconcile one list to `desired` (a set of IPs).
Returns (added, removed). Serializes add->wait->delete so we respect the
one-pending-bulk-op-per-account limit. Raises CFError on hard failure.
"""
existing = cf_list_items(list_id) # {ip: item_id}
existing_ips = set(existing)
to_add = sorted(desired - existing_ips)
to_remove_ids = [existing[ip] for ip in (existing_ips - desired)]
added = removed = 0
if to_add:
op = cf_add_items(list_id, to_add)
if _wait_for_op(op):
added = len(to_add)
else:
# add op didn't finish; skip the delete this run to avoid stacking a
# second pending op, and report what we attempted.
print(f"[warn] {label}: add op not confirmed; deferring deletes", file=sys.stderr)
return added, removed
if to_remove_ids:
op = cf_delete_items(list_id, to_remove_ids)
if _wait_for_op(op):
removed = len(to_remove_ids)
print(f"[ok] {label}: +{added} / -{removed} (desired={len(desired)}, was={len(existing_ips)})")
return added, removed
# --------------------------------------------------------------------------- #
# Metrics (best-effort)
# --------------------------------------------------------------------------- #
def push_metrics(ban_n, captcha_n, ok):
if not PUSHGATEWAY:
return
payload = (
"# TYPE crowdsec_kv_sync_decisions gauge\n"
f"crowdsec_kv_sync_decisions {synced}\n"
"# TYPE crowdsec_kv_sync_deleted gauge\n"
f"crowdsec_kv_sync_deleted {deleted}\n"
"# TYPE crowdsec_kv_sync_success gauge\n"
f"crowdsec_kv_sync_success {1 if ok else 0}\n"
"# TYPE crowdsec_kv_sync_last_run_seconds gauge\n"
f"crowdsec_kv_sync_last_run_seconds {int(time.time())}\n"
"# TYPE crowdsec_cf_list_ban_count gauge\n"
f"crowdsec_cf_list_ban_count {ban_n}\n"
"# TYPE crowdsec_cf_list_captcha_count gauge\n"
f"crowdsec_cf_list_captcha_count {captcha_n}\n"
"# TYPE crowdsec_cf_list_sync_success gauge\n"
f"crowdsec_cf_list_sync_success {1 if ok else 0}\n"
"# TYPE crowdsec_cf_list_sync_last_run_seconds gauge\n"
f"crowdsec_cf_list_sync_last_run_seconds {int(time.time())}\n"
)
try:
_req(f"{PUSHGATEWAY}/metrics/job/crowdsec-kv-sync", method="PUT",
headers={"Content-Type": "text/plain"}, data=payload.encode(), timeout=10)
_req(
f"{PUSHGATEWAY}/metrics/job/crowdsec-cf-list-sync",
method="PUT",
headers={"Content-Type": "text/plain"},
data=payload.encode(),
timeout=10,
)
except Exception as e: # metrics are best-effort, never fail the job
print(f"[warn] pushgateway: {e}", file=sys.stderr)
# --------------------------------------------------------------------------- #
def main():
# 1. Desired state from LAPI. Any failure here = SKIP (fail-safe, leave KV).
# 1. Desired state from LAPI. Any failure here = SKIP (fail-safe).
try:
desired = fetch_decisions()
ban, captcha = fetch_decisions()
except Exception as e:
print(f"[skip] LAPI unreadable ({e}); leaving KV untouched so existing "
f"bans expire by TTL (fail-open).", file=sys.stderr)
print(
f"[skip] LAPI unreadable ({e}); leaving CF lists untouched "
f"(fail-safe: freeze last-known edge state).",
file=sys.stderr,
)
push_metrics(0, 0, ok=False)
return 0
# 2. Current edge state.
existing = set(cf_list_keys())
desired_keys = set(desired)
print(f"[info] LAPI desired: {len(ban)} ban / {len(captcha)} captcha (ip-scope)")
upserts = [(k, v, t) for k, (v, t) in desired.items()]
stale = existing - desired_keys
# 2. Reconcile both lists. CF errors fail loud (non-zero exit).
try:
reconcile("ban", CF_BAN_LIST_ID, ban)
reconcile("captcha", CF_CAPTCHA_LIST_ID, captcha)
except CFError as e:
print(f"[error] Cloudflare API failure: {e}", file=sys.stderr)
push_metrics(len(ban), len(captcha), ok=False)
return 1
if upserts:
cf_bulk_put(upserts)
if stale:
cf_bulk_delete(stale)
print(f"[ok] synced {len(upserts)} decision(s) to KV, removed {len(stale)} stale; "
f"{sum(1 for _, v, _ in upserts if v == 'ban')} ban / "
f"{sum(1 for _, v, _ in upserts if v == 'captcha')} captcha")
push_metrics(len(upserts), len(stale), ok=True)
push_metrics(len(ban), len(captcha), ok=True)
return 0