diff --git a/cli/README.md b/cli/README.md index adc06920..186c1ee5 100644 --- a/cli/README.md +++ b/cli/README.md @@ -171,6 +171,37 @@ prints the bare token to stdout so it composes in `$(…)`; it's read-tier like not tied to whoever first wrote the workflow (the user's key must be enrolled on the HA host). +### v0.8 verbs — browser (headful anti-bot automation) + +Drive the cluster's **headful** Chrome (`chrome-service`, real Chrome under Xvfb) +from the devvm over CDP, for sites that detect and block headless automation. The +headless `@playwright/mcp` browser can *load* such a site and fill its forms, but +the gated action (submit/login) silently fails — the motivating case was the +Stirling Ackroyd Fixflo tenant portal, whose pre-submit check returned +`net::ERR_FILE_NOT_FOUND` and hung. This path connects via `connect_over_cdp`, +injects the same `stealth.js` the in-cluster callers use, and submits first try. + +The command owns only the *mechanics* (port-forward, stealth, lifecycle); the +agent supplies the Playwright script — judgment stays out of the CLI. + +| Command | Tier | What it does | +|---|---|---| +| `browser run [--url U] [--shared-context] [--keep-open] [--port N] [--timeout S]` | write | port-forward `svc/chrome-service:9222`, assert it's a real (non-headless) Chrome via `/json/version`, `connect_over_cdp`, `addInitScript(stealth.js)`, then run the script with `page`/`context`/`browser`/`log` in scope (top-level await ok; return a value to print it). Always tears the forward down. | +| `browser open [--shared-context] [--timeout S]` | write | open `` headful and print title + visible text + a screenshot path — a quick check. | +| `browser --help` | read | when-to-use signature + the error-code cheat-sheet (`ERR_FILE_NOT_FOUND` = automation-layer intercept, not egress; `ERR_CONNECTION_REFUSED`/`_TIMED_OUT`/`_NAME_NOT_RESOLVED` = real egress; one endpoint 500 while siblings 200 = bot rejection). | + +Default context is a **fresh incognito** one (closed on exit) — safe for the +shared browser and concurrent callers (e.g. tripit's fare scrape); `--shared-context` +reuses the warmed persistent profile when a pre-logged-in session is needed. +`port-forward` tunnels API-server→pod, so it bypasses the `:9222` NetworkPolicy +that gates in-cluster callers — no namespace label needed. The node CDP client is +pinned to **`playwright-core@1.48.2`** to match the chrome-service image minor +(Chromium 130; protocol changes between minors) and is installed once, lazily, +into `~/.cache/homelab/browser-client/` (no per-user setup). Because the client +runs on the devvm, `setInputFiles` streams local files to the remote browser over +CDP — no `chmod`/staging-dir workaround. See `docs/architecture/chrome-service.md` +and `docs/adr/0013`. + ## Build / install Built from source to `/usr/local/bin/homelab` during devvm provisioning @@ -190,4 +221,4 @@ original flag-based path unchanged, so the webhook handler is unaffected. ## Design -See `infra/docs/adr/0004`–`0012` for the architecture decisions. +See `infra/docs/adr/0004`–`0013` for the architecture decisions. diff --git a/cli/VERSION b/cli/VERSION index 63f2359f..b19b5211 100644 --- a/cli/VERSION +++ b/cli/VERSION @@ -1 +1 @@ -v0.7.1 +v0.8.0 diff --git a/cli/browser.go b/cli/browser.go new file mode 100644 index 00000000..39b6b0a0 --- /dev/null +++ b/cli/browser.go @@ -0,0 +1,388 @@ +package main + +import ( + _ "embed" + "encoding/json" + "fmt" + "io" + "net" + "net/http" + "os" + "os/exec" + "os/signal" + "path/filepath" + "strconv" + "strings" + "sync" + "syscall" + "time" +) + +// playwrightVersion pins the node CDP client to the chrome-service image minor +// (mcr.microsoft.com/playwright:v1.48.0-noble → Chromium 130). connect_over_cdp +// speaks the browser's CDP, so the client minor must track the server minor; +// see docs/architecture/chrome-service.md "Image pin". +const playwrightVersion = "1.48.2" + +// defaultBrowserTimeout is how long (seconds) to wait for the port-forwarded CDP +// endpoint to become ready before giving up. +const defaultBrowserTimeout = 60 + +const ( + chromeServiceNamespace = "chrome-service" + chromeServiceName = "chrome-service" + chromeServiceCDPPort = 9222 +) + +// stealthJS is vendored verbatim from stacks/chrome-service/files/stealth.js (the +// source of truth the in-cluster callers use). TestStealthJSEmbeddedMatchesCanonical +// guards against drift. +// +//go:embed browser_stealth.js +var stealthJS string + +// runnerJS is the node wrapper that connects to the port-forwarded CDP endpoint, +// installs the stealth init script, and runs the user's Playwright script. +// +//go:embed browser_runner.js +var runnerJS string + +// browserOpts is the parsed form of `homelab browser run|open` arguments. +type browserOpts struct { + mode string // "run" | "open" + script string // path to the user Playwright script (run mode) + url string // initial URL (run: optional; open: required positional) + sharedCtx bool // use the warmed persistent profile instead of a fresh context + keepOpen bool // leave the created context/pages open on exit + port int // explicit local port for the forward (0 = auto) + timeout int // CDP readiness timeout, seconds + help bool +} + +// parseBrowserArgs parses the args after `browser run` / `browser open`. +func parseBrowserArgs(mode string, args []string) (browserOpts, error) { + o := browserOpts{mode: mode, timeout: defaultBrowserTimeout} + var positionals []string + atoi := func(s, flag string) (int, error) { + n, err := strconv.Atoi(s) + if err != nil { + return 0, fmt.Errorf("%s expects an integer, got %q", flag, s) + } + return n, nil + } + for i := 0; i < len(args); i++ { + a := args[i] + switch { + case a == "-h" || a == "--help": + o.help = true + case a == "--shared-context": + o.sharedCtx = true + case a == "--keep-open": + o.keepOpen = true + case a == "--url": + if i+1 < len(args) { + o.url = args[i+1] + i++ + } + case strings.HasPrefix(a, "--url="): + o.url = strings.TrimPrefix(a, "--url=") + case a == "--port": + if i+1 < len(args) { + n, err := atoi(args[i+1], "--port") + if err != nil { + return o, err + } + o.port = n + i++ + } + case strings.HasPrefix(a, "--port="): + n, err := atoi(strings.TrimPrefix(a, "--port="), "--port") + if err != nil { + return o, err + } + o.port = n + case a == "--timeout": + if i+1 < len(args) { + n, err := atoi(args[i+1], "--timeout") + if err != nil { + return o, err + } + o.timeout = n + i++ + } + case strings.HasPrefix(a, "--timeout="): + n, err := atoi(strings.TrimPrefix(a, "--timeout="), "--timeout") + if err != nil { + return o, err + } + o.timeout = n + case strings.HasPrefix(a, "-"): + return o, fmt.Errorf("unknown flag %q (try: homelab browser --help)", a) + default: + positionals = append(positionals, a) + } + } + if o.help { + return o, nil + } + switch mode { + case "run": + if len(positionals) == 0 { + return o, fmt.Errorf("usage: homelab browser run [--url URL] [--shared-context] [--keep-open] [--port N] [--timeout S]") + } + o.script = positionals[0] + case "open": + if len(positionals) == 0 { + return o, fmt.Errorf("usage: homelab browser open [--shared-context] [--timeout S]") + } + o.url = positionals[0] + } + return o, nil +} + +// cdpHealthy parses a CDP /json/version body and reports whether the endpoint is +// a real (non-headless) Chrome — the entire reason chrome-service exists. +func cdpHealthy(jsonBody []byte) (browser string, healthy bool, err error) { + var v struct { + Browser string `json:"Browser"` + UserAgent string `json:"User-Agent"` + } + if e := json.Unmarshal(jsonBody, &v); e != nil { + return "", false, fmt.Errorf("parse /json/version: %w", e) + } + if v.Browser == "" { + return "", false, fmt.Errorf("/json/version had no Browser field") + } + healthy = strings.HasPrefix(v.Browser, "Chrome/") && + !strings.Contains(v.Browser, "Headless") && + !strings.Contains(v.UserAgent, "Headless") + return v.Browser, healthy, nil +} + +// buildPortForwardArgs is the kubectl invocation that exposes chrome-service's +// CDP locally. port-forward tunnels API-server→pod, so it bypasses the :9222 +// NetworkPolicy that gates in-cluster callers. +func buildPortForwardArgs(localPort int) []string { + return []string{"-n", chromeServiceNamespace, "port-forward", + "svc/" + chromeServiceName, fmt.Sprintf("%d:%d", localPort, chromeServiceCDPPort)} +} + +// browserClientPackageJSON is the auto-managed manifest for the pinned node CDP +// client kept under the user cache dir. +func browserClientPackageJSON() string { + return fmt.Sprintf(`{ + "name": "homelab-browser-client", + "private": true, + "description": "Pinned CDP client for 'homelab browser' — auto-managed, do not edit.", + "dependencies": { + "playwright-core": "%s" + } +} +`, playwrightVersion) +} + +// freePort asks the kernel for an unused ephemeral TCP port. +func freePort() (int, error) { + l, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + return 0, err + } + defer l.Close() + return l.Addr().(*net.TCPAddr).Port, nil +} + +// browserClientDir is where the pinned node client + managed runner files live. +func browserClientDir() (string, error) { + cache, err := os.UserCacheDir() + if err != nil || cache == "" { + home, herr := os.UserHomeDir() + if herr != nil { + return "", fmt.Errorf("locate cache dir: %v / %v", err, herr) + } + cache = filepath.Join(home, ".cache") + } + return filepath.Join(cache, "homelab", "browser-client"), nil +} + +// installedPlaywrightVersion reads the version of the playwright-core already +// installed in dir, or "" if absent/unreadable. +func installedPlaywrightVersion(dir string) string { + b, err := os.ReadFile(filepath.Join(dir, "node_modules", "playwright-core", "package.json")) + if err != nil { + return "" + } + var v struct { + Version string `json:"version"` + } + if json.Unmarshal(b, &v) != nil { + return "" + } + return v.Version +} + +// ensureBrowserClient writes the managed runner/stealth/package files into dir +// and lazily installs the pinned playwright-core (only when missing/mismatched), +// so no per-user setup is needed and the client tracks the binary version. +func ensureBrowserClient(dir string) error { + if err := os.MkdirAll(dir, 0o755); err != nil { + return err + } + files := map[string]string{ + "package.json": browserClientPackageJSON(), + "browser_runner.js": runnerJS, + "stealth.js": stealthJS, + } + for name, content := range files { + if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644); err != nil { + return err + } + } + if installedPlaywrightVersion(dir) == playwrightVersion { + return nil + } + fmt.Fprintf(os.Stderr, "homelab browser: installing pinned playwright-core@%s (one-time, ~a few seconds)…\n", playwrightVersion) + cmd := exec.Command("npm", "install", "--no-audit", "--no-fund", "--silent") + cmd.Dir = dir + cmd.Stdout = os.Stderr + cmd.Stderr = os.Stderr + if err := cmd.Run(); err != nil { + return fmt.Errorf("npm install playwright-core@%s in %s: %w (is node/npm installed?)", playwrightVersion, dir, err) + } + if got := installedPlaywrightVersion(dir); got != playwrightVersion { + return fmt.Errorf("playwright-core install mismatch in %s: want %s, got %q", dir, playwrightVersion, got) + } + return nil +} + +// waitForCDP polls the local CDP endpoint until it answers as a healthy +// (non-headless) Chrome, or the timeout elapses. +func waitForCDP(cdpURL string, timeout time.Duration) (string, error) { + deadline := time.Now().Add(timeout) + client := &http.Client{Timeout: 3 * time.Second} + var lastErr error + for time.Now().Before(deadline) { + resp, err := client.Get(cdpURL + "/json/version") + if err != nil { + lastErr = err + time.Sleep(300 * time.Millisecond) + continue + } + body, _ := io.ReadAll(resp.Body) + resp.Body.Close() + browser, healthy, herr := cdpHealthy(body) + if herr != nil { + lastErr = herr + time.Sleep(300 * time.Millisecond) + continue + } + if !healthy { + return browser, fmt.Errorf("CDP reports %q — expected a non-headless Chrome (wrong target?)", browser) + } + return browser, nil + } + if lastErr == nil { + lastErr = fmt.Errorf("timed out after %s", timeout) + } + return "", lastErr +} + +// runBrowser is the orchestration: pick a port, ensure the pinned client, start +// (and ALWAYS tear down) a CDP port-forward, wait for readiness, then run node. +func runBrowser(o browserOpts) error { + port := o.port + if port == 0 { + p, err := freePort() + if err != nil { + return fmt.Errorf("pick local port: %w", err) + } + port = p + } + + dir, err := browserClientDir() + if err != nil { + return err + } + if err := ensureBrowserClient(dir); err != nil { + return err + } + + // Start the forward in its own process group so the whole tree dies on cleanup. + pf := exec.Command("kubectl", buildPortForwardArgs(port)...) + pf.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + var pfLog strings.Builder + pf.Stdout = &pfLog + pf.Stderr = &pfLog + if err := pf.Start(); err != nil { + return fmt.Errorf("start kubectl port-forward (kubeconfig set?): %w", err) + } + + var once sync.Once + teardown := func() { + once.Do(func() { + if pf.Process != nil { + _ = syscall.Kill(-pf.Process.Pid, syscall.SIGKILL) + } + _ = pf.Wait() + }) + } + defer teardown() + + // Tear down on Ctrl-C / SIGTERM too, then exit non-zero. + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, os.Interrupt, syscall.SIGTERM) + defer signal.Stop(sigCh) + go func() { + if _, ok := <-sigCh; ok { + teardown() + os.Exit(130) + } + }() + + cdpURL := fmt.Sprintf("http://127.0.0.1:%d", port) + browser, err := waitForCDP(cdpURL, time.Duration(o.timeout)*time.Second) + if err != nil { + return fmt.Errorf("chrome-service CDP not ready on %s: %w\n--- port-forward log ---\n%s", cdpURL, err, pfLog.String()) + } + fmt.Fprintf(os.Stderr, "homelab browser: connected to %s via %s\n", browser, cdpURL) + + return runBrowserNode(dir, cdpURL, o) +} + +// runBrowserNode invokes the managed node runner with inputs passed via env. +func runBrowserNode(dir, cdpURL string, o browserOpts) error { + env := append(os.Environ(), + "HOMELAB_CDP_URL="+cdpURL, + "HOMELAB_BROWSER_MODE="+o.mode, + "HOMELAB_STEALTH_PATH="+filepath.Join(dir, "stealth.js"), + "NODE_PATH="+filepath.Join(dir, "node_modules"), + ) + if o.url != "" { + env = append(env, "HOMELAB_BROWSER_URL="+o.url) + } + if o.script != "" { + abs, err := filepath.Abs(o.script) + if err != nil { + return err + } + if _, err := os.Stat(abs); err != nil { + return fmt.Errorf("script %s: %w", o.script, err) + } + env = append(env, "HOMELAB_BROWSER_SCRIPT="+abs) + } + if o.sharedCtx { + env = append(env, "HOMELAB_BROWSER_SHARED=1") + } + if o.keepOpen { + env = append(env, "HOMELAB_BROWSER_KEEP_OPEN=1") + } + if o.mode == "open" { + shot := filepath.Join(os.TempDir(), fmt.Sprintf("homelab-browser-%d.png", os.Getpid())) + env = append(env, "HOMELAB_BROWSER_SCREENSHOT="+shot) + } + cmd := exec.Command("node", filepath.Join(dir, "browser_runner.js")) + cmd.Env = env + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + cmd.Stdin = os.Stdin + return cmd.Run() +} diff --git a/cli/browser_runner.js b/cli/browser_runner.js new file mode 100644 index 00000000..24a2db6b --- /dev/null +++ b/cli/browser_runner.js @@ -0,0 +1,106 @@ +// homelab browser — node CDP runner (auto-managed; regenerated each run from the +// homelab binary — DO NOT EDIT here). Connects to the port-forwarded +// chrome-service CDP endpoint, installs the stealth init script, then runs the +// user's Playwright script (run mode) or opens a URL (open mode). All inputs +// arrive via HOMELAB_* env vars set by the Go CLI. +'use strict'; +const fs = require('fs'); +const { chromium } = require('playwright-core'); + +async function main() { + const cdpURL = process.env.HOMELAB_CDP_URL; + if (!cdpURL) throw new Error('HOMELAB_CDP_URL not set'); + const mode = process.env.HOMELAB_BROWSER_MODE || 'run'; + const stealthPath = process.env.HOMELAB_STEALTH_PATH || ''; + const initURL = process.env.HOMELAB_BROWSER_URL || ''; + const scriptPath = process.env.HOMELAB_BROWSER_SCRIPT || ''; + const shared = process.env.HOMELAB_BROWSER_SHARED === '1'; + const keepOpen = process.env.HOMELAB_BROWSER_KEEP_OPEN === '1'; + const screenshotPath = process.env.HOMELAB_BROWSER_SCREENSHOT || ''; + + const browser = await chromium.connectOverCDP(cdpURL); + + // Fresh isolated context by default (safe for the shared browser + concurrent + // callers); --shared-context reuses the warmed persistent profile. + let context; + let createdContext = false; + if (shared) { + const existing = browser.contexts(); + if (existing.length) { + context = existing[0]; + } else { + context = await browser.newContext(); + createdContext = true; + } + } else { + context = await browser.newContext(); + createdContext = true; + } + + if (stealthPath) { + const stealth = fs.readFileSync(stealthPath, 'utf8'); + if (stealth.trim()) await context.addInitScript(stealth); + } + + const page = await context.newPage(); + const log = (...a) => console.error('[browser]', ...a); + + let exitCode = 0; + try { + if (initURL) { + await page.goto(initURL, { waitUntil: 'domcontentloaded' }); + } + if (mode === 'open') { + console.log('url: ' + page.url()); + console.log('title: ' + (await page.title())); + const text = (await page.evaluate(() => (document.body ? document.body.innerText : ''))).trim(); + console.log('--- visible text (truncated to 4000 chars) ---'); + console.log(text.slice(0, 4000)); + if (screenshotPath) { + await page.screenshot({ path: screenshotPath, fullPage: true }); + console.log('screenshot: ' + screenshotPath); + } + } else { + if (!scriptPath) throw new Error('run mode requires HOMELAB_BROWSER_SCRIPT'); + const src = fs.readFileSync(scriptPath, 'utf8'); + // Run the user's source with page/context/browser/log in lexical scope. + // AsyncFunction body permits top-level await. + const AsyncFunction = Object.getPrototypeOf(async () => {}).constructor; + const fn = new AsyncFunction('page', 'context', 'browser', 'log', src); + const result = await fn(page, context, browser, log); + if (result !== undefined) { + let out; + try { + out = typeof result === 'string' ? result : JSON.stringify(result, null, 2); + } catch (_) { + out = String(result); + } + console.log(out); + } + } + } catch (e) { + console.error('homelab browser: script error:', e && e.stack ? e.stack : e); + exitCode = 1; + } finally { + if (!keepOpen) { + try { + // Close only what we created; never tear down the shared persistent context. + if (createdContext) { + await context.close(); + } else { + await page.close(); + } + } catch (_) { /* ignore */ } + } + // Disconnect from the CDP endpoint; this does NOT kill the remote browser. + try { + await browser.close(); + } catch (_) { /* ignore */ } + } + process.exit(exitCode); +} + +main().catch((e) => { + console.error('homelab browser: fatal:', e && e.stack ? e.stack : e); + process.exit(1); +}); diff --git a/cli/browser_stealth.js b/cli/browser_stealth.js new file mode 100644 index 00000000..dfae98a8 --- /dev/null +++ b/cli/browser_stealth.js @@ -0,0 +1,54 @@ +// Minimal stealth init script for Playwright-driven Chromium. +// Vendored from puppeteer-extra-plugin-stealth/evasions/* (MIT) — covers: +// webdriver, chrome.runtime, navigator.plugins, navigator.languages, +// Permissions.query, WebGL getParameter (vendor + renderer spoof). +// Run via context.add_init_script() so it executes before any page script. +(() => { + // navigator.webdriver — most common detection, removed entirely. + Object.defineProperty(Navigator.prototype, 'webdriver', { get: () => undefined }); + + // window.chrome.runtime — many sites check that real Chrome exposes this. + if (!window.chrome) window.chrome = {}; + window.chrome.runtime = window.chrome.runtime || {}; + + // navigator.plugins — headless reports zero; spoof a plausible PDF viewer. + Object.defineProperty(navigator, 'plugins', { + get: () => [{ name: 'Chrome PDF Plugin' }, { name: 'Chrome PDF Viewer' }, { name: 'Native Client' }], + }); + + // navigator.languages — headless returns empty array. + Object.defineProperty(navigator, 'languages', { get: () => ['en-US', 'en'] }); + + // Permissions.query — headless returns 'denied' for notifications instead of 'default'. + const origQuery = window.navigator.permissions && window.navigator.permissions.query; + if (origQuery) { + window.navigator.permissions.query = (parameters) => + parameters && parameters.name === 'notifications' + ? Promise.resolve({ state: Notification.permission }) + : origQuery(parameters); + } + + // WebGL getParameter — spoof vendor + renderer strings to a real GPU. + const spoofGl = (proto) => { + if (!proto) return; + const orig = proto.getParameter; + proto.getParameter = function (parameter) { + if (parameter === 37445) return 'Intel Inc.'; // UNMASKED_VENDOR_WEBGL + if (parameter === 37446) return 'Intel Iris OpenGL Engine'; // UNMASKED_RENDERER_WEBGL + return orig.apply(this, arguments); + }; + }; + spoofGl(window.WebGLRenderingContext && window.WebGLRenderingContext.prototype); + spoofGl(window.WebGL2RenderingContext && window.WebGL2RenderingContext.prototype); + + // disable-devtool.js (theajack/disable-devtool) auto-inits via a script + // tag with `disable-devtool-auto`. Its Performance detector trips under + // Playwright (CDP adds console.log latency vs console.table) and the + // redirect URL is hard-coded — for hmembeds that's google.com. + // Hide the auto-init marker so the library's IIFE exits early. + const origQS = Document.prototype.querySelector; + Document.prototype.querySelector = function (sel) { + if (typeof sel === 'string' && sel.indexOf('disable-devtool-auto') !== -1) return null; + return origQS.apply(this, arguments); + }; +})(); diff --git a/cli/cmd_browser.go b/cli/cmd_browser.go new file mode 100644 index 00000000..181c83cf --- /dev/null +++ b/cli/cmd_browser.go @@ -0,0 +1,113 @@ +package main + +import "fmt" + +// browser verbs drive the cluster's HEADFUL Chrome (ns chrome-service) over CDP +// from outside the cluster, for sites that detect/block headless automation. +// The headless @playwright/mcp browser can load such sites but their gated +// actions (submit/login) silently fail; this path submits first try. Mechanics +// only — the agent supplies the Playwright script. See docs/adr/0013. + +func browserCommands() []Command { + return []Command{ + {Path: []string{"browser"}, Tier: TierRead, + Summary: "headful cluster-Chrome automation for anti-bot sites (run `browser --help`)", Run: browserTopHelp}, + {Path: []string{"browser", "run"}, Tier: TierWrite, + Summary: "run a Playwright script against headful cluster Chrome: browser run [--url U] [--shared-context]", Run: browserRun}, + {Path: []string{"browser", "open"}, Tier: TierWrite, + Summary: "open a URL in headful cluster Chrome; print title + text + screenshot: browser open ", Run: browserOpen}, + } +} + +func browserTopHelp([]string) error { + fmt.Print(browserHelp()) + return nil +} + +func browserRun(args []string) error { + o, err := parseBrowserArgs("run", args) + if err != nil { + return err + } + if o.help { + fmt.Print(browserHelp()) + return nil + } + return runBrowser(o) +} + +func browserOpen(args []string) error { + o, err := parseBrowserArgs("open", args) + if err != nil { + return err + } + if o.help { + fmt.Print(browserHelp()) + return nil + } + return runBrowser(o) +} + +// browserHelp carries the discoverability payload: WHEN to reach for this, and +// the diagnostic cheat-sheet that lets the agent self-correct instead of +// retrying a deterministic form blind (the failure mode that motivated this). +func browserHelp() string { + return `homelab browser — drive the cluster's HEADFUL Chrome (anti-bot) over CDP + +The shared chrome-service (ns chrome-service) runs a REAL, headed Chrome under +Xvfb. This connects to it via a port-forward + Playwright connect_over_cdp, +injects the same stealth.js the in-cluster callers use, and runs your script. + +USAGE + homelab browser run [--url URL] [--shared-context] [--keep-open] [--port N] [--timeout S] + homelab browser open [--shared-context] [--timeout S] + +WHEN TO USE THIS (not the headless @playwright/mcp browser) + A site LOADS fine but a gated action FAILS or HANGS — a submit/login/checkout + spins forever, or ONE request 500s / aborts while its siblings 200. That is + the signature of headless / anti-bot detection (navigator.webdriver, UA + "HeadlessChrome", disable-devtool traps). Switch to this command — it presents + as a real Chrome and typically succeeds first try. + +ERROR-CODE CHEAT-SHEET (diagnose BEFORE retrying) + ERR_FILE_NOT_FOUND (-6) request intercepted/resolved locally by the + automation layer — NOT a network/egress problem. + (This is what silently broke the headless submit.) + ERR_CONNECTION_REFUSED / real egress failure (DNS/route/firewall). These also + ERR_TIMED_OUT / break the initial page load — if the page loaded, + ERR_NAME_NOT_RESOLVED egress is fine and the cause is elsewhere. + one endpoint 500s while server-side bot rejection of the automation, not + its siblings 200 your payload. + +HABITS + - Inspect the network panel BEFORE retrying a deterministic form; a blind + retry just repeats the same silent failure. + - Don't park a half-filled multi-step form across a user pause — the session + can expire; re-run the whole flow from this command in one shot. + - Uploads stream over CDP via setInputFiles from THIS host — no chmod/staging + of $HOME needed; just point setInputFiles at a local path. + +CONTEXT + Default: a FRESH incognito context, closed on exit — safe for the shared + browser and concurrent callers (e.g. tripit). Your script does its own login. + --shared-context: reuse the warmed PERSISTENT profile (cookies from a manual + noVNC login at chrome.viktorbarzin.me) when you need a pre-logged-in session. + +SCRIPT CONTRACT (run mode) + Your file's body runs with page, context, browser and log() already in scope + (top-level await allowed). Return a value to print it. Example flow.js: + + await page.goto('https://portal.example.com/login'); + await page.fill('#user', 'me'); await page.fill('#pass', process.env.PW); + await page.click('button[type=submit]'); + await page.waitForURL('**/dashboard'); + return 'logged in: ' + page.url(); + + Run it: homelab browser run flow.js + +NOTES + - The Playwright client is pinned to playwright-core@` + playwrightVersion + ` to match the + chrome-service image (Chrome 130); installed once into ~/.cache/homelab/. + - The port-forward is always torn down, on success and on error. +` +} diff --git a/cli/cmd_browser_test.go b/cli/cmd_browser_test.go new file mode 100644 index 00000000..ee6bc25d --- /dev/null +++ b/cli/cmd_browser_test.go @@ -0,0 +1,160 @@ +package main + +import ( + "os" + "reflect" + "strings" + "testing" +) + +func TestParseBrowserArgsRun(t *testing.T) { + got, err := parseBrowserArgs("run", []string{ + "flow.js", "--url", "https://example.com", "--shared-context", + "--port", "19999", "--timeout", "45", "--keep-open", + }) + if err != nil { + t.Fatalf("parseBrowserArgs run: unexpected err: %v", err) + } + want := browserOpts{ + mode: "run", script: "flow.js", url: "https://example.com", + sharedCtx: true, keepOpen: true, port: 19999, timeout: 45, + } + if !reflect.DeepEqual(got, want) { + t.Fatalf("parseBrowserArgs run =\n %+v\nwant\n %+v", got, want) + } +} + +func TestParseBrowserArgsRunDefaults(t *testing.T) { + got, err := parseBrowserArgs("run", []string{"flow.js"}) + if err != nil { + t.Fatalf("unexpected err: %v", err) + } + if got.script != "flow.js" || got.sharedCtx || got.keepOpen || got.port != 0 { + t.Fatalf("defaults wrong: %+v", got) + } + if got.timeout != defaultBrowserTimeout { + t.Fatalf("timeout default = %d, want %d", got.timeout, defaultBrowserTimeout) + } +} + +func TestParseBrowserArgsRunRequiresScript(t *testing.T) { + if _, err := parseBrowserArgs("run", []string{"--url", "https://x"}); err == nil { + t.Fatalf("run without a script path should error") + } +} + +func TestParseBrowserArgsOpenRequiresURL(t *testing.T) { + got, err := parseBrowserArgs("open", []string{"https://example.com"}) + if err != nil { + t.Fatalf("unexpected err: %v", err) + } + if got.url != "https://example.com" || got.mode != "open" { + t.Fatalf("open parse wrong: %+v", got) + } + if _, err := parseBrowserArgs("open", []string{}); err == nil { + t.Fatalf("open without a URL should error") + } +} + +func TestParseBrowserArgsHelp(t *testing.T) { + for _, a := range [][]string{{"--help"}, {"-h"}, {"flow.js", "--help"}} { + got, err := parseBrowserArgs("run", a) + if err != nil { + t.Fatalf("help parse %v: %v", a, err) + } + if !got.help { + t.Fatalf("args %v should set help", a) + } + } +} + +func TestParseBrowserArgsEqualsForm(t *testing.T) { + got, err := parseBrowserArgs("run", []string{"flow.js", "--url=https://x", "--port=8123", "--timeout=10"}) + if err != nil { + t.Fatalf("unexpected err: %v", err) + } + if got.url != "https://x" || got.port != 8123 || got.timeout != 10 { + t.Fatalf("--flag=value form not parsed: %+v", got) + } +} + +func TestCDPHealthy(t *testing.T) { + real := []byte(`{"Browser":"Chrome/130.0.6723.31","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) Chrome/130.0.0.0 Safari/537.36","webSocketDebuggerUrl":"ws://127.0.0.1/devtools/browser/x"}`) + browser, ok, err := cdpHealthy(real) + if err != nil || !ok { + t.Fatalf("real Chrome should be healthy: ok=%v err=%v", ok, err) + } + if !strings.HasPrefix(browser, "Chrome/") { + t.Fatalf("browser = %q, want Chrome/ prefix", browser) + } + + headless := []byte(`{"Browser":"HeadlessChrome/130.0.6723.31","User-Agent":"Mozilla/5.0 HeadlessChrome/130.0.0.0"}`) + if _, ok, _ := cdpHealthy(headless); ok { + t.Fatalf("HeadlessChrome must be reported unhealthy (the whole point of chrome-service)") + } + + if _, _, err := cdpHealthy([]byte("not json")); err == nil { + t.Fatalf("malformed /json/version body should error") + } +} + +func TestBuildPortForwardArgs(t *testing.T) { + got := buildPortForwardArgs(18080) + want := []string{"-n", "chrome-service", "port-forward", "svc/chrome-service", "18080:9222"} + if !reflect.DeepEqual(got, want) { + t.Fatalf("buildPortForwardArgs =\n %v\nwant\n %v", got, want) + } +} + +func TestBrowserClientPackageJSONPinsVersion(t *testing.T) { + pj := browserClientPackageJSON() + if !strings.Contains(pj, `"playwright-core": "`+playwrightVersion+`"`) { + t.Fatalf("package.json must pin playwright-core to %s; got:\n%s", playwrightVersion, pj) + } +} + +func TestPlaywrightVersionPinnedToServerMinor(t *testing.T) { + // chrome-service runs mcr.microsoft.com/playwright:v1.48.0-noble; the CDP + // client minor MUST match (protocol changes between minors). + if !strings.HasPrefix(playwrightVersion, "1.48.") { + t.Fatalf("playwrightVersion = %q, must be 1.48.x to match the chrome-service image", playwrightVersion) + } +} + +func TestBrowserHelpHasDiagnosticCheatSheet(t *testing.T) { + h := browserHelp() + for _, want := range []string{ + "homelab browser run", + "ERR_FILE_NOT_FOUND", + "ERR_CONNECTION_REFUSED", + "network panel", + "headless", + "--shared-context", + } { + if !strings.Contains(h, want) { + t.Errorf("browser --help is missing %q (the discoverability/self-correction payload)", want) + } + } +} + +func TestStealthJSEmbeddedMatchesCanonical(t *testing.T) { + // The embedded copy must never drift from the source of truth that the + // in-cluster callers use, else the CLI's stealth and the cluster's diverge. + canonical, err := os.ReadFile("../stacks/chrome-service/files/stealth.js") + if err != nil { + t.Fatalf("read canonical stealth.js: %v", err) + } + if stealthJS != string(canonical) { + t.Fatalf("cli/browser_stealth.js has drifted from stacks/chrome-service/files/stealth.js — re-copy it") + } +} + +func TestFreePortReturnsUsablePort(t *testing.T) { + p, err := freePort() + if err != nil { + t.Fatalf("freePort: %v", err) + } + if p <= 1024 || p > 65535 { + t.Fatalf("freePort returned %d, want an ephemeral port", p) + } +} diff --git a/cli/homelab.go b/cli/homelab.go index fb12a169..5f781791 100644 --- a/cli/homelab.go +++ b/cli/homelab.go @@ -22,6 +22,7 @@ func buildRegistry() []Command { reg = append(reg, obsCommands()...) reg = append(reg, usageCommands()...) reg = append(reg, haCommands()...) + reg = append(reg, browserCommands()...) return reg } diff --git a/docs/adr/0013-homelab-browser-verbs.md b/docs/adr/0013-homelab-browser-verbs.md new file mode 100644 index 00000000..bba4e8e7 --- /dev/null +++ b/docs/adr/0013-homelab-browser-verbs.md @@ -0,0 +1,75 @@ +# homelab browser verbs: headful (anti-bot) web automation via cluster Chrome + +v0.8 adds `browser run`, `browser open`, and `browser --help`. They package a +capability that already existed but was undiscoverable: driving the cluster's +**headful** Chrome (`chrome-service` — real Chrome under Xvfb, CDP on +`svc/chrome-service:9222`) from the devvm, for sites that detect and block +headless automation. + +## Motivating incident (2026-06-22) + +Logging a washing-machine repair on the Stirling Ackroyd **Fixflo** tenant +portal: the headless `@playwright/mcp` browser loaded the site and filled the +entire multi-step form, but the **final submit silently failed** — Fixflo's +pre-submit `POST /IssuePreCreationCheck` returned `net::ERR_FILE_NOT_FOUND`, the +spinner hung, no issue was created. Root cause = headless-Chrome detection. The +fix was to drive the headful `chrome-service` over `connect_over_cdp` — it +submitted first try (Fixflo ref IS22657587). That capability was documented +(`docs/architecture/chrome-service.md`) but **not packaged or discoverable**, so +it took ~40 min, three redundant full form re-runs, and a user hint. The agent +also misread `ERR_FILE_NOT_FOUND` as "network egress" and retried blind instead +of inspecting the network panel. + +## Decisions + +- **Mechanics in `homelab`, not a `~/.claude` skill.** A standalone skill was + rejected: the CLI is run every session (so the verb is *discoverable*), is + versioned, multi-user, and test-covered. A private, untested skill is none of + those. The command owns only the deterministic *mechanics* (port-forward, + stealth injection, lifecycle) — the agent supplies the Playwright script, so + *judgment* stays out of the CLI (the founding rule, ADR-0004/0005). +- **The failure was judgment, not setup friction**, so the CLI is paired with a + one-line pointer in always-in-context `~/code/CLAUDE.md` and a diagnostic + payload in `browser --help`: the *when-to-use* signature (a site loads but a + gated action fails/hangs, or one request 500s/aborts while siblings 200 → + suspect headless detection) and an error-code cheat-sheet (`ERR_FILE_NOT_FOUND` + = request resolved/intercepted by the automation layer, **not** egress; + egress failures are `ERR_CONNECTION_REFUSED`/`_TIMED_OUT`/`_NAME_NOT_RESOLVED` + and would break the page load too). A command the agent doesn't think to run is + useless; the cheat-sheet is the actual fix for the misdiagnosis. +- **Reach the pod via `kubectl port-forward`, then `connect_over_cdp` to + localhost.** port-forward tunnels API-server→pod, so it **bypasses the `:9222` + NetworkPolicy** that gates in-cluster callers — the devvm needs no namespace + label. Readiness is asserted against `/json/version`: the endpoint must report + a real `Chrome/…`, never `HeadlessChrome` (the whole point). The forward is + **always** torn down (process-group kill + signal handler), on success and on + error — an acceptance requirement. +- **Default to a fresh incognito context; `--shared-context` opts into the warmed + profile.** chrome-service is a single shared browser with a persistent profile. + A fresh, always-closed context is safe for concurrent callers (tripit's fare + scrape connects per-quote) and is what production already does. The warmed + persistent profile (cookies from a manual noVNC login) is opt-in for flows that + need a pre-logged-in session. +- **Pin the node CDP client to `playwright-core@1.48.2`** to match the + chrome-service image minor (`mcr.microsoft.com/playwright:v1.48.0-noble`, + Chromium 130). `connect_over_cdp` speaks the browser's CDP, and protocol + changes between Playwright minors — the devvm's ambient Python Playwright was + 1.58, a 10-minor skew. The pin makes behaviour deterministic across the fleet + regardless of local drift. `playwright-core` (not `playwright`) because no + browser binary is needed — we connect to the remote one. +- **Self-provision the client lazily, no per-user setup.** The pinned client is + installed once into `~/.cache/homelab/browser-client/` (idempotent, version- + guarded) on first use, alongside the embedded runner + stealth files. node is + already fleet-wide; this avoids coupling the feature to a provisioner change + and keeps it self-contained and self-healing. The client runs on the devvm, so + `setInputFiles` streams local files to the remote browser over CDP — no + `chmod`/staging-dir workaround on the CDP path. +- **Vendor `stealth.js`, guard against drift.** The CLI embeds a byte-for-byte + copy of `stacks/chrome-service/files/stealth.js` (the source of truth the + in-cluster callers use) via `go:embed`; a unit test fails if the copy drifts. + `go:embed` can't reach outside the package dir, hence the vendored copy rather + than a path reference. +- **Scope held at two action verbs + help.** `run` (arbitrary script — the + workhorse) and `open` (navigate + title/text/screenshot — a quick check) cover + the surface. Both are write-tier; the bare `browser`/`--help` is read. Re-measure + via `usage top` (ADR-0011) before adding more. diff --git a/docs/architecture/chrome-service.md b/docs/architecture/chrome-service.md index 7b95d4a0..e1b5208d 100644 --- a/docs/architecture/chrome-service.md +++ b/docs/architecture/chrome-service.md @@ -180,6 +180,42 @@ minor, with Python-side bindings pre-installed. See `stacks/chrome-service/README.md` for the recipe (label namespace, inject `CHROME_CDP_URL`, vendor `stealth.js`). +## Driving from OUTSIDE the cluster (`homelab browser`) + +Agents on the devvm reach this browser through the **`homelab browser`** CLI +(`cli/`, ADR-0013) — the packaged, discoverable form of the ad-hoc +`connect_over_cdp` recipe. Use it when a site loads but a gated action +(submit/login) silently fails or hangs — the signature of headless / anti-bot +detection. + +```text +devvm: homelab browser run flow.js + │ kubectl port-forward svc/chrome-service :9222 (random local port) + ▼ + http://127.0.0.1: ──► chrome-service pod :9222 (CDP) + │ assert /json/version Browser is "Chrome/…", not "HeadlessChrome" + │ node + playwright-core@1.48.2 → connectOverCDP + │ context.addInitScript(stealth.js) ← same vendored file as in-cluster + │ run the user's Playwright script with page/context/browser in scope + └─ port-forward always torn down (success or error) +``` + +Key facts: + +- **port-forward bypasses the `:9222` NetworkPolicy.** It tunnels + API-server→pod, so the devvm needs no `chrome-service.viktorbarzin.me/client` + label — unlike in-cluster callers. +- **Client pinned to the image minor.** The node client is + `playwright-core@1.48.2` (matches `v1.48.0-noble` / Chromium 130), installed + lazily into `~/.cache/homelab/browser-client/`. Bump it in lockstep when the + server image bumps (same rule as the in-cluster Python clients — see "Image + pin" above). +- **Default context is a fresh incognito one** (closed on exit), safe for the + shared browser; `--shared-context` reuses the warmed persistent profile. +- **`stealth.js` is vendored** into the CLI (`cli/browser_stealth.js`) as a + byte-identical copy of `files/stealth.js`, guarded by a drift test — so the + CLI's stealth never diverges from the in-cluster callers'. + ## Limits + risks - **Anti-bot vs stealth arms race** — when an upstream beats us (DRM