diff --git a/.claude/skills/traefik-rewrite-body-accept-header/SKILL.md b/.claude/skills/traefik-rewrite-body-accept-header/SKILL.md deleted file mode 100644 index a62a98f0..00000000 --- a/.claude/skills/traefik-rewrite-body-accept-header/SKILL.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -name: traefik-rewrite-body-accept-header -description: | - Fix for Traefik rewrite-body plugin (packruler/rewrite-body) silently skipping - content injection (rybbit analytics, trap links, or any HTML rewriting). Use when: - (1) rewrite-body middleware is enabled but injected content doesn't appear in responses, - (2) curl shows original HTML with no modifications but browser works fine, - (3) monitoring.types is set to ["text/html"] but responses aren't being rewritten, - (4) no errors in Traefik logs despite rewrite-body being in the middleware chain. - Root cause: the plugin checks the request Accept header (not response Content-Type) - against monitoring.types, and curl's default Accept: */* does not match text/html. -author: Claude Code -version: 1.0.0 -date: 2026-02-22 ---- - -# Traefik Rewrite-Body Accept Header Matching - -## Problem -The `packruler/rewrite-body` Traefik plugin silently skips HTML content injection -when the request's `Accept` header doesn't contain a value matching `monitoring.types`. -This means `curl` requests (which send `Accept: */*`) won't show injected content, -while browsers (which send `Accept: text/html,...`) will. - -## Context / Trigger Conditions -- rewrite-body middleware is in the ingress middleware chain and shows status "enabled" in Traefik API -- `curl https://example.com/` returns original HTML with no injected content -- Browser shows injected content (rybbit script, trap links, etc.) -- No errors in Traefik logs — the plugin silently skips processing -- `monitoring.types = ["text/html"]` is configured in the middleware spec -- Middleware chain order is correct (strip-accept-encoding before rewrite-body) - -## Misleading Symptoms -- Appears as if the middleware isn't working at all -- May look like a middleware ordering issue or configuration error -- `kubectl get middleware` shows the resource exists with correct spec -- Traefik API (`/api/http/middlewares/`) shows the middleware as "enabled" -- Checking the rewrite-body regex patterns seems pointless since nothing is being processed - -## Root Cause -In the plugin source code, `SupportsProcessing()` checks: -```go -func (r *Rewriter) SupportsProcessing(req *http.Request) bool { - accept := req.Header.Get("Accept") - for _, monitoringType := range r.monitoring.Types { - if strings.Contains(accept, monitoringType) { - return true - } - } - return false -} -``` - -It uses `strings.Contains(accept, "text/html")`. The curl default `Accept: */*` does -NOT contain the substring `text/html`, so the plugin returns false and skips all -processing. Browser requests include `Accept: text/html,application/xhtml+xml,...` -which does match. - -## Solution -This is **working as designed** — not a bug. The fix depends on context: - -### For testing with curl -Add the `Accept` header to simulate a browser: -```bash -curl -s -H "Accept: text/html,application/xhtml+xml" https://example.com/ -``` - -### For verifying injection is working -```bash -# Check for injected content (trap links, analytics, etc.) -curl -s -H "Accept: text/html,application/xhtml+xml" https://example.com/ \ - | grep -oE 'href="https://poison[^"]*"' - -# Check for rybbit analytics -curl -s -H "Accept: text/html,application/xhtml+xml" https://example.com/ \ - | grep -oE 'src="https://rybbit[^"]*"' -``` - -### For programmatic clients that need injection -If a non-browser client needs to receive injected content, ensure it sends -`Accept: text/html` in its request headers. - -## Verification -```bash -# Without Accept header — no injection (expected) -curl -s https://example.com/ | grep -c "rybbit" -# Output: 0 - -# With Accept header — injection works -curl -s -H "Accept: text/html" https://example.com/ | grep -c "rybbit" -# Output: 1 -``` - -## Notes -- This behavior is independent of the compression issue (see `traefik-rewrite-body-compression`) -- The check is on the **request** `Accept` header, not the **response** `Content-Type` -- `Accept: */*` does NOT match — `strings.Contains("*/*", "text/html")` is false -- Real AI scrapers typically send browser-like Accept headers, so trap links will be - injected for them correctly -- API calls (which typically send `Accept: application/json`) are correctly skipped - -## See Also -- `traefik-rewrite-body-compression` — Different issue: plugin fails to decompress - gzip responses, corrupting content. That skill covers the compression fix - (strip-accept-encoding middleware). -- `ingress-factory-migration` — Covers the ingress factory module middleware chain diff --git a/.claude/skills/traefik-rewrite-body-compression/SKILL.md b/.claude/skills/traefik-rewrite-body-compression/SKILL.md deleted file mode 100644 index 8531dace..00000000 --- a/.claude/skills/traefik-rewrite-body-compression/SKILL.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -name: traefik-rewrite-body-compression -description: | - Fix for Traefik rewrite-body plugin (packruler/rewrite-body) failing with - "flate: corrupt input before offset 5" errors when backends send gzip-compressed - responses. Use when: (1) rewrite-body plugin logs show "Error loading content: - flate: corrupt input before offset 5", (2) rybbit analytics script injection - breaks WebSocket connections or authentication flows, (3) HA Companion app - stuck in external_auth loop with repeated GET /?external_auth=1 requests, - (4) mobile apps fail to connect while browser works fine, (5) HTTP 499 errors - on webhook POST requests. Root cause: the plugin attempts to decompress all - responses before checking content-type, and fails on certain gzip encodings, - corrupting the response body. -author: Claude Code -version: 1.0.0 -date: 2026-02-11 ---- - -# Traefik Rewrite-Body Plugin Compression Fix - -## Problem -The `packruler/rewrite-body` Traefik plugin (used for injecting analytics scripts -like rybbit into HTML responses) fails to decompress gzip-compressed responses from -backends. Despite the `monitoring.types = ["text/html"]` filter, the plugin attempts -to decompress ALL responses before checking content type. When decompression fails, -it corrupts the response body, breaking: -- WebSocket upgrade handshakes -- Authentication flows (HA Companion app's `external_auth` callback) -- Mobile app connectivity (while browser appears to work due to auto-reconnect) - -## Context / Trigger Conditions -- Traefik logs show: `Rewrite-Body | ERROR ... Error loading content: flate: corrupt input before offset 5` -- Mobile apps (e.g., Home Assistant Companion) fail while browser works -- HA Companion app shows repeated `GET /?external_auth=1` requests (auth loop) -- WebSocket connections (`/api/websocket`) are very short-lived (seconds instead of minutes) -- HTTP 499 errors on API calls (client disconnects due to corrupted responses) -- Using `packruler/rewrite-body` plugin v1.2.0 with `monitoring.types = ["text/html"]` - -## Misleading Symptoms -- HTTP/3 (QUIC) may appear to be the cause because HTTP/3 requests show 499 errors. - This is a red herring - the rewrite-body plugin corruption affects all protocols. -- WebSocket issues may look like a timeout or proxy configuration problem. -- The `monitoring.types = ["text/html"]` config suggests the plugin should only touch - HTML, but it still processes all responses for decompression before filtering. - -## Solution - -### Step 1: Create a strip-accept-encoding middleware -Add a Traefik middleware that removes `Accept-Encoding` from requests, forcing -backends to send uncompressed responses that the plugin can safely process: - -```hcl -# In traefik/middleware.tf -resource "kubernetes_manifest" "middleware_strip_accept_encoding" { - manifest = { - apiVersion = "traefik.io/v1alpha1" - kind = "Middleware" - metadata = { - name = "strip-accept-encoding" - namespace = kubernetes_namespace.traefik.metadata[0].name - } - spec = { - headers = { - customRequestHeaders = { - "Accept-Encoding" = "" - } - } - } - } - depends_on = [helm_release.traefik] -} -``` - -### Step 2: Add middleware to routes with rewrite-body -In the ingress factory middleware chain, add `strip-accept-encoding` BEFORE the -rewrite-body middleware: - -```hcl -var.rybbit_site_id != null ? "traefik-strip-accept-encoding@kubernetescrd" : null, -var.rybbit_site_id != null ? "${var.namespace}-rybbit-analytics-${var.name}@kubernetescrd" : null, -``` - -The order matters: strip-accept-encoding must come first so the request reaches -the backend without Accept-Encoding, and the uncompressed response then passes -through the rewrite-body plugin. - -## Verification -1. Check Traefik logs for absence of `flate: corrupt input` errors: - ```bash - kubectl logs -n traefik -l app.kubernetes.io/name=traefik --tail=200 | grep -i "flate\|rewrite-body" - ``` -2. Verify the middleware chain includes strip-accept-encoding before rybbit: - ```bash - kubectl get ingress -n -o jsonpath='{.metadata.annotations.traefik\.ingress\.kubernetes\.io/router\.middlewares}' - ``` -3. Test mobile app connectivity (HA Companion, etc.) - -## Notes -- This affects ALL services using the rewrite-body plugin, not just HA -- The fix is applied conditionally: `strip-accept-encoding` is only added to the - middleware chain when `rybbit_site_id` is set, so services without analytics - are unaffected -- Both `ingress_factory` and `reverse_proxy/factory` modules need the fix -- Traefik may still compress responses to clients via its own compression middleware; - the strip only affects the backend request -- The plugin's `monitoring.types` filter works for deciding what to rewrite, but - decompression is attempted on all responses regardless - -## See Also -- `ingress-factory-migration` - Covers the ingress factory module that creates - rybbit analytics middlewares -- `traefik-http3-quic` - HTTP/3 configuration (not the cause, but often a red herring - when debugging this issue) diff --git a/.claude/skills/traefik-rewrite-body-troubleshooting/SKILL.md b/.claude/skills/traefik-rewrite-body-troubleshooting/SKILL.md new file mode 100644 index 00000000..5ff27fec --- /dev/null +++ b/.claude/skills/traefik-rewrite-body-troubleshooting/SKILL.md @@ -0,0 +1,200 @@ +--- +name: traefik-rewrite-body-troubleshooting +description: | + Troubleshooting guide for the Traefik rewrite-body plugin (packruler/rewrite-body). + Covers two failure modes: (1) Compression failure — plugin logs "flate: corrupt input + before offset 5" when backends send gzip-compressed responses, corrupting response + bodies and breaking WebSocket connections, authentication flows, and mobile app + connectivity. (2) Silent skip — plugin silently skips content injection (rybbit + analytics, trap links, or any HTML rewriting) when the request Accept header doesn't + contain "text/html" (e.g., curl's default Accept: */*), making it appear broken + despite correct configuration. +author: Claude Code +version: 1.0.0 +date: 2026-02-22 +--- + +# Traefik Rewrite-Body Plugin Troubleshooting + +Two distinct failure modes for the `packruler/rewrite-body` Traefik plugin used for +injecting analytics scripts (rybbit) and anti-AI trap links into HTML responses. + +--- + +## Problem 1: Compression Failure + +### Symptoms +- Traefik logs show: `Rewrite-Body | ERROR ... Error loading content: flate: corrupt input before offset 5` +- Mobile apps (e.g., Home Assistant Companion) fail while browser works +- HA Companion app shows repeated `GET /?external_auth=1` requests (auth loop) +- WebSocket connections (`/api/websocket`) are very short-lived (seconds instead of minutes) +- HTTP 499 errors on API calls (client disconnects due to corrupted responses) +- Using `packruler/rewrite-body` plugin v1.2.0 with `monitoring.types = ["text/html"]` + +### Root Cause +Despite the `monitoring.types = ["text/html"]` filter, the plugin attempts to decompress +ALL responses before checking content type. When decompression fails on certain gzip +encodings, it corrupts the response body, breaking: +- WebSocket upgrade handshakes +- Authentication flows (HA Companion app's `external_auth` callback) +- Mobile app connectivity (while browser appears to work due to auto-reconnect) + +### Misleading Symptoms +- HTTP/3 (QUIC) may appear to be the cause because HTTP/3 requests show 499 errors. + This is a red herring -- the rewrite-body plugin corruption affects all protocols. +- WebSocket issues may look like a timeout or proxy configuration problem. +- The `monitoring.types = ["text/html"]` config suggests the plugin should only touch + HTML, but it still processes all responses for decompression before filtering. + +### Solution + +#### Step 1: Create a strip-accept-encoding middleware +Add a Traefik middleware that removes `Accept-Encoding` from requests, forcing +backends to send uncompressed responses that the plugin can safely process: + +```hcl +# In traefik/middleware.tf +resource "kubernetes_manifest" "middleware_strip_accept_encoding" { + manifest = { + apiVersion = "traefik.io/v1alpha1" + kind = "Middleware" + metadata = { + name = "strip-accept-encoding" + namespace = kubernetes_namespace.traefik.metadata[0].name + } + spec = { + headers = { + customRequestHeaders = { + "Accept-Encoding" = "" + } + } + } + } + depends_on = [helm_release.traefik] +} +``` + +#### Step 2: Add middleware to routes with rewrite-body +In the ingress factory middleware chain, add `strip-accept-encoding` BEFORE the +rewrite-body middleware: + +```hcl +var.rybbit_site_id != null ? "traefik-strip-accept-encoding@kubernetescrd" : null, +var.rybbit_site_id != null ? "${var.namespace}-rybbit-analytics-${var.name}@kubernetescrd" : null, +``` + +The order matters: strip-accept-encoding must come first so the request reaches +the backend without Accept-Encoding, and the uncompressed response then passes +through the rewrite-body plugin. + +### Verification (Compression Fix) +1. Check Traefik logs for absence of `flate: corrupt input` errors: + ```bash + kubectl logs -n traefik -l app.kubernetes.io/name=traefik --tail=200 | grep -i "flate\|rewrite-body" + ``` +2. Verify the middleware chain includes strip-accept-encoding before rybbit: + ```bash + kubectl get ingress -n -o jsonpath='{.metadata.annotations.traefik\.ingress\.kubernetes\.io/router\.middlewares}' + ``` +3. Test mobile app connectivity (HA Companion, etc.) + +### Notes (Compression) +- This affects ALL services using the rewrite-body plugin, not just HA +- The fix is applied conditionally: `strip-accept-encoding` is only added to the + middleware chain when `rybbit_site_id` is set, so services without analytics + are unaffected +- Both `ingress_factory` and `reverse_proxy/factory` modules need the fix +- Traefik may still compress responses to clients via its own compression middleware; + the strip only affects the backend request +- The plugin's `monitoring.types` filter works for deciding what to rewrite, but + decompression is attempted on all responses regardless + +--- + +## Problem 2: Silent Skip (Accept Header Mismatch) + +### Symptoms +- rewrite-body middleware is in the ingress middleware chain and shows status "enabled" in Traefik API +- `curl https://example.com/` returns original HTML with no injected content +- Browser shows injected content (rybbit script, trap links, etc.) +- No errors in Traefik logs -- the plugin silently skips processing +- `monitoring.types = ["text/html"]` is configured in the middleware spec +- Middleware chain order is correct (strip-accept-encoding before rewrite-body) + +### Root Cause +In the plugin source code, `SupportsProcessing()` checks the **request** `Accept` +header (not the response `Content-Type`) against `monitoring.types`: + +```go +func (r *Rewriter) SupportsProcessing(req *http.Request) bool { + accept := req.Header.Get("Accept") + for _, monitoringType := range r.monitoring.Types { + if strings.Contains(accept, monitoringType) { + return true + } + } + return false +} +``` + +It uses `strings.Contains(accept, "text/html")`. The curl default `Accept: */*` does +NOT contain the substring `text/html`, so the plugin returns false and skips all +processing. Browser requests include `Accept: text/html,application/xhtml+xml,...` +which does match. + +### Misleading Symptoms +- Appears as if the middleware isn't working at all +- May look like a middleware ordering issue or configuration error +- `kubectl get middleware` shows the resource exists with correct spec +- Traefik API (`/api/http/middlewares/`) shows the middleware as "enabled" +- Checking the rewrite-body regex patterns seems pointless since nothing is being processed + +### Solution +This is **working as designed** -- not a bug. The fix depends on context: + +#### For testing with curl +Add the `Accept` header to simulate a browser: +```bash +curl -s -H "Accept: text/html,application/xhtml+xml" https://example.com/ +``` + +#### For verifying injection is working +```bash +# Check for injected content (trap links, analytics, etc.) +curl -s -H "Accept: text/html,application/xhtml+xml" https://example.com/ \ + | grep -oE 'href="https://poison[^"]*"' + +# Check for rybbit analytics +curl -s -H "Accept: text/html,application/xhtml+xml" https://example.com/ \ + | grep -oE 'src="https://rybbit[^"]*"' +``` + +#### For programmatic clients that need injection +If a non-browser client needs to receive injected content, ensure it sends +`Accept: text/html` in its request headers. + +### Verification (Accept Header) +```bash +# Without Accept header -- no injection (expected) +curl -s https://example.com/ | grep -c "rybbit" +# Output: 0 + +# With Accept header -- injection works +curl -s -H "Accept: text/html" https://example.com/ | grep -c "rybbit" +# Output: 1 +``` + +### Notes (Accept Header) +- This behavior is independent of the compression issue (Problem 1 above) +- The check is on the **request** `Accept` header, not the **response** `Content-Type` +- `Accept: */*` does NOT match -- `strings.Contains("*/*", "text/html")` is false +- Real AI scrapers typically send browser-like Accept headers, so trap links will be + injected for them correctly +- API calls (which typically send `Accept: application/json`) are correctly skipped + +--- + +## See Also +- `traefik-helm-configuration` -- Traefik Helm chart configuration and entrypoints +- `ingress-factory-migration` -- Covers the ingress factory module that creates + rybbit analytics middlewares