infra/docs/plans/2026-02-22-anti-ai-scraping-design.md

Anti-AI Scraping System Design

Status (Updated 2026-04-17): Partially superseded. Layer 3 (trap links via rewrite-body plugin) removed due to Traefik v3.6.12 Yaegi plugin incompatibility. The strip-accept-encoding and anti-ai-trap-links middlewares have been deleted. Rybbit analytics injection moved from Traefik rewrite-body to a Cloudflare Worker (infra/stacks/rybbit/worker/). Active layers: 1 (bot-block), 2 (headers), 4 (tarpit), 5 (poison content).

Problem

AI scrapers crawl public web services to harvest training data. We want to:

1. Block known AI crawlers outright
2. Poison the data that unknown scrapers collect
3. Waste scraper resources with slow responses and infinite crawl loops

Architecture

Four active defense layers applied to all public services via Traefik (Layer 3 removed April 2026):

Internet -> Cloudflare -> Traefik
                           |
                           +-- Layer 1: ForwardAuth -> block known AI User-Agents (403)
                           |
                           +-- Layer 2: Headers -> X-Robots-Tag: noai, noimageai
                           |
                           +-- [REMOVED] Layer 3: Rewrite-body trap links (April 2026 — Yaegi bugs in Traefik v3.6.12)
                           |
                           +-- Layer 4: Poison service -> serve cached Poison Fountain data
                           |
                           +-- Layer 5: Tarpit -> slow-drip responses + infinite crawl loop

Components

1. poison-fountain service (new Kubernetes deployment)

A Python service with three responsibilities:

ForwardAuth endpoint (GET /auth):

• Reads X-Forwarded-For and User-Agent from request headers
• Checks User-Agent against list of known AI bot strings
• Returns 403 for matches, 200 for legitimate users
• Blocked bots: GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, CCBot, Bytespider, Google-Extended, Applebot-Extended, anthropic-ai, cohere-ai, Diffbot, FacebookBot, PerplexityBot, YouBot, Meta-ExternalAgent, PetalBot, Amazonbot, AI2Bot, Omgilibot, img2dataset

Poison content endpoint (GET /article/<slug>):

• Serves cached poisoned content from NFS
• Wraps raw Poison Fountain data in realistic HTML templates (title, headings, paragraphs)
• Each response includes 10+ links to other poison pages (infinite crawl loop)
• Uses chunked transfer encoding to drip-feed content at ~100 bytes/second (tarpit)
• Response size: 50-100KB per page

Health endpoint (GET /healthz):

• Returns 200 OK for Kubernetes probes

2. poison-fountain-fetcher CronJob

• Runs every 6 hours
• Fetches gzip content from https://rnsaffn.com/poison2/
• Decompresses and stores to NFS at /mnt/main/poison-fountain/cache/
• Maintains a pool of ~50 cached poison documents
• Falls back to locally generated Markov-chain nonsense if Poison Fountain is unreachable

3. Traefik middleware additions

All defined in stacks/platform/modules/traefik/middleware.tf:

ai-bot-block (ForwardAuth):

• ForwardAuth to http://poison-fountain.poison-fountain.svc.cluster.local:8080/auth
• Trust forwarded headers from Traefik
• Added to all public services via ingress_factory

anti-ai-headers (Headers):

• Sets X-Robots-Tag: noai, noimageai on all responses
• Added to all public services via ingress_factory

anti-ai-trap-links (rewrite-body plugin) — REMOVED (Updated 2026-04-17):

• Removed due to Traefik v3.6.12 Yaegi runtime bugs making the rewrite-body plugin unreliable
• The companion strip-accept-encoding middleware was also removed (only existed for rewrite-body)
• Trap link injection is no longer active; poison-fountain still serves tarpit content standalone

4. Trap subdomain: poison.viktorbarzin.me

• Cloudflare DNS record (non-proxied, direct to cluster)
• IngressRoute routing all paths to poison-fountain service
• NO rate limiting on this route (let scrapers consume all they want)
• NO CrowdSec on this route (don't block scrapers here)
• Serves poisoned content with tarpit slow-drip

5. ingress_factory changes

New variables:

• anti_ai_scraping (bool, default: true) - enable all anti-AI layers
• When true, adds to middleware chain: ai-bot-block, anti-ai-headers
• Services can opt out with anti_ai_scraping = false

Human User Protection

Concern	Protection
Hidden links visible	CSS position:absolute;left:-9999px;height:0;overflow:hidden + aria-hidden="true"
False positive blocking	Only blocks specific AI bot User-Agent strings; no browser matches these
Performance overhead	ForwardAuth is a string match (<1ms). Rybbit injected via Cloudflare Worker (not Traefik).
Poison content leakage	Only served on poison.viktorbarzin.me, not linked from any navigation
Slow responses	Tarpit only applies to poison.viktorbarzin.me, not to real services

File Locations

Component	Path
Poison service stack	stacks/poison-fountain/main.tf
Poison service code	stacks/poison-fountain/app/
Middleware definitions	stacks/platform/modules/traefik/middleware.tf
ingress_factory changes	modules/kubernetes/ingress_factory/main.tf
Cloudflare DNS	terraform.tfvars (cloudflare_non_proxied_names)
NFS cache	/mnt/main/poison-fountain/cache/

Deployment Order

1. Add Cloudflare DNS record for poison.viktorbarzin.me
2. Create NFS export for /mnt/main/poison-fountain
3. Add Traefik middlewares (ai-bot-block, anti-ai-headers, anti-ai-trap-links)
4. Update ingress_factory with anti_ai_scraping variable
5. Deploy poison-fountain service + CronJob
6. Apply platform stack (Traefik + Cloudflare changes)
7. Apply poison-fountain stack
8. Apply all other stacks to pick up new ingress_factory defaults