From 9488af2397c98de12d2e07d8a261e32c65676787 Mon Sep 17 00:00:00 2001 From: Viktor Barzin Date: Sun, 22 Feb 2026 21:41:07 +0000 Subject: [PATCH] [ci skip] Add rewrite-body Accept header skill, update NFS skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New skill: traefik-rewrite-body-accept-header — rewrite-body plugin silently skips injection when request Accept header doesn't contain text/html (curl default Accept: */* doesn't match). Updated: k8s-nfs-mount-troubleshooting v1.1.0 — added variant for non-root container UID permission denied on NFS writes. --- .../k8s-nfs-mount-troubleshooting/SKILL.md | 63 ++++++++++- .../SKILL.md | 106 ++++++++++++++++++ 2 files changed, 163 insertions(+), 6 deletions(-) create mode 100644 .claude/skills/traefik-rewrite-body-accept-header/SKILL.md diff --git a/.claude/skills/k8s-nfs-mount-troubleshooting/SKILL.md b/.claude/skills/k8s-nfs-mount-troubleshooting/SKILL.md index da4764dc..5f3fa3b8 100644 --- a/.claude/skills/k8s-nfs-mount-troubleshooting/SKILL.md +++ b/.claude/skills/k8s-nfs-mount-troubleshooting/SKILL.md @@ -1,14 +1,16 @@ --- name: k8s-nfs-mount-troubleshooting description: | - Debug Kubernetes NFS volume mount failures. Use when: (1) Pod stuck in ContainerCreating + Debug Kubernetes NFS volume mount failures. Use when: (1) Pod stuck in ContainerCreating for extended time, (2) kubectl describe shows "MountVolume.SetUp failed" with NFS errors, - (3) Error message shows "Protocol not supported" or "mount.nfs: access denied", - (4) NFS volume defined in pod spec but container won't start. Common root cause is - missing NFS export on the server, not a protocol issue. + (3) Error message shows "Protocol not supported" or "mount.nfs: access denied", + (4) NFS volume defined in pod spec but container won't start, (5) Container starts but + gets "Permission denied" writing to NFS volume (non-root container UID mismatch), + (6) CronJob or init container fails silently when writing to NFS. Common root causes + are missing NFS export on the server and UID mismatch for non-root containers. author: Claude Code -version: 1.0.0 -date: 2026-01-28 +version: 1.1.0 +date: 2026-02-22 --- # Kubernetes NFS Mount Troubleshooting @@ -91,6 +93,55 @@ ssh root@10.0.10.15 'mkdir -p /mnt/main/resume && chmod 777 /mnt/main/resume' - NFSv3 vs NFSv4 issues are rare in modern setups; missing paths are more common - Check that the NFS client packages are installed on Kubernetes nodes if this is a new cluster +## Variant: Non-Root Container UID Permission Denied + +### Problem +Container starts and mounts NFS successfully, but gets "Permission denied" when +writing files. The pod appears healthy but operations fail silently. + +### Trigger Conditions +- Container logs show "Permission denied" or "client returned ERROR on write" +- Pod is Running (not stuck in ContainerCreating) +- NFS directory exists and is mounted, but owned by root (uid 0) +- Container image runs as a non-root user (e.g., `curlimages/curl` runs as uid 101) +- CronJobs or init containers that write to NFS fail with no obvious error + +### Common Non-Root Container UIDs +| Image | UID | User | +|-------|-----|------| +| `curlimages/curl` | 101 | curl_user | +| `nginx` (unprivileged) | 101 | nginx | +| `node` | 1000 | node | +| `python` (slim) | 0 | root (safe) | +| `grafana/grafana` | 472 | grafana | + +### Solution +Fix permissions on the NFS server: +```bash +# Option 1: World-writable (simplest, suitable for non-sensitive data) +ssh root@10.0.10.15 "chmod -R 777 /mnt/main//" + +# Option 2: Match container UID (more secure) +ssh root@10.0.10.15 "chown -R : /mnt/main//" + +# Option 3: Use securityContext in pod spec to run as root +spec: + securityContext: + runAsUser: 0 +``` + +### Debugging +```bash +# Check what UID the container runs as +kubectl exec -n -- id + +# Test write access from inside container +kubectl exec -n -- sh -c 'echo test > /path/to/nfs/testfile' + +# Check NFS directory ownership on server +ssh root@10.0.10.15 "ls -la /mnt/main//" +``` + ## See Also - TrueNAS NFS configuration documentation - Kubernetes NFS volume documentation diff --git a/.claude/skills/traefik-rewrite-body-accept-header/SKILL.md b/.claude/skills/traefik-rewrite-body-accept-header/SKILL.md new file mode 100644 index 00000000..a62a98f0 --- /dev/null +++ b/.claude/skills/traefik-rewrite-body-accept-header/SKILL.md @@ -0,0 +1,106 @@ +--- +name: traefik-rewrite-body-accept-header +description: | + Fix for Traefik rewrite-body plugin (packruler/rewrite-body) silently skipping + content injection (rybbit analytics, trap links, or any HTML rewriting). Use when: + (1) rewrite-body middleware is enabled but injected content doesn't appear in responses, + (2) curl shows original HTML with no modifications but browser works fine, + (3) monitoring.types is set to ["text/html"] but responses aren't being rewritten, + (4) no errors in Traefik logs despite rewrite-body being in the middleware chain. + Root cause: the plugin checks the request Accept header (not response Content-Type) + against monitoring.types, and curl's default Accept: */* does not match text/html. +author: Claude Code +version: 1.0.0 +date: 2026-02-22 +--- + +# Traefik Rewrite-Body Accept Header Matching + +## Problem +The `packruler/rewrite-body` Traefik plugin silently skips HTML content injection +when the request's `Accept` header doesn't contain a value matching `monitoring.types`. +This means `curl` requests (which send `Accept: */*`) won't show injected content, +while browsers (which send `Accept: text/html,...`) will. + +## Context / Trigger Conditions +- rewrite-body middleware is in the ingress middleware chain and shows status "enabled" in Traefik API +- `curl https://example.com/` returns original HTML with no injected content +- Browser shows injected content (rybbit script, trap links, etc.) +- No errors in Traefik logs — the plugin silently skips processing +- `monitoring.types = ["text/html"]` is configured in the middleware spec +- Middleware chain order is correct (strip-accept-encoding before rewrite-body) + +## Misleading Symptoms +- Appears as if the middleware isn't working at all +- May look like a middleware ordering issue or configuration error +- `kubectl get middleware` shows the resource exists with correct spec +- Traefik API (`/api/http/middlewares/`) shows the middleware as "enabled" +- Checking the rewrite-body regex patterns seems pointless since nothing is being processed + +## Root Cause +In the plugin source code, `SupportsProcessing()` checks: +```go +func (r *Rewriter) SupportsProcessing(req *http.Request) bool { + accept := req.Header.Get("Accept") + for _, monitoringType := range r.monitoring.Types { + if strings.Contains(accept, monitoringType) { + return true + } + } + return false +} +``` + +It uses `strings.Contains(accept, "text/html")`. The curl default `Accept: */*` does +NOT contain the substring `text/html`, so the plugin returns false and skips all +processing. Browser requests include `Accept: text/html,application/xhtml+xml,...` +which does match. + +## Solution +This is **working as designed** — not a bug. The fix depends on context: + +### For testing with curl +Add the `Accept` header to simulate a browser: +```bash +curl -s -H "Accept: text/html,application/xhtml+xml" https://example.com/ +``` + +### For verifying injection is working +```bash +# Check for injected content (trap links, analytics, etc.) +curl -s -H "Accept: text/html,application/xhtml+xml" https://example.com/ \ + | grep -oE 'href="https://poison[^"]*"' + +# Check for rybbit analytics +curl -s -H "Accept: text/html,application/xhtml+xml" https://example.com/ \ + | grep -oE 'src="https://rybbit[^"]*"' +``` + +### For programmatic clients that need injection +If a non-browser client needs to receive injected content, ensure it sends +`Accept: text/html` in its request headers. + +## Verification +```bash +# Without Accept header — no injection (expected) +curl -s https://example.com/ | grep -c "rybbit" +# Output: 0 + +# With Accept header — injection works +curl -s -H "Accept: text/html" https://example.com/ | grep -c "rybbit" +# Output: 1 +``` + +## Notes +- This behavior is independent of the compression issue (see `traefik-rewrite-body-compression`) +- The check is on the **request** `Accept` header, not the **response** `Content-Type` +- `Accept: */*` does NOT match — `strings.Contains("*/*", "text/html")` is false +- Real AI scrapers typically send browser-like Accept headers, so trap links will be + injected for them correctly +- API calls (which typically send `Accept: application/json`) are correctly skipped + +## See Also +- `traefik-rewrite-body-compression` — Different issue: plugin fails to decompress + gzip responses, corrupting content. That skill covers the compression fix + (strip-accept-encoding middleware). +- `ingress-factory-migration` — Covers the ingress factory module middleware chain