homelab v0.8.0: browser verbs for headful anti-bot web automation
Add `homelab browser run|open` so agents can drive the cluster's headful Chrome (chrome-service) over CDP from the devvm. The headless playwright/mcp browser can load anti-bot sites and fill their forms, but the gated submit silently fails — e.g. the Stirling Ackroyd Fixflo tenant portal returned net::ERR_FILE_NOT_FOUND on its pre-submit check and hung, creating nothing. Driving the real headful Chrome submits first try. That capability already existed but was undiscoverable, so it cost ~40 min + redundant form re-runs to find; now it is one command, versioned, test-covered, and `browser --help` carries the when-to-use signature + an error-code cheat-sheet so the right tool is reached at the right moment (the failure was judgment, not setup). - port-forward svc/chrome-service:9222 (tunnels API-server->pod, so it bypasses the :9222 NetworkPolicy), assert non-headless via /json/version, connect_over_cdp, inject the same vendored stealth.js the in-cluster callers use; the port-forward is always torn down, on success and on error. - node CDP client pinned to playwright-core@1.48.2 to match the v1.48.0-noble image (Chromium 130); self-provisioned lazily into ~/.cache/homelab, no per-user setup. - default is a fresh incognito context (safe for the shared browser + concurrent callers); --shared-context reuses the warmed persistent profile. - TDD: cmd_browser_test.go covers arg parsing, headless detection, the version pin, the help cheat-sheet, and a stealth.js drift guard. Verified end-to-end against bot.sannysoft.com (real Chrome UA, webdriver hidden, plugins/WebGL spoofed) and `browser open`. - docs: README v0.8 section, ADR-0013, and a chrome-service.md "driving from outside the cluster" section. Closes: code-nepg Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
de163aa6af
commit
a6b52a5839
10 changed files with 966 additions and 2 deletions
|
|
@ -171,6 +171,37 @@ prints the bare token to stdout so it composes in `$(…)`; it's read-tier like
|
|||
not tied to whoever first wrote the workflow (the user's key must be enrolled on
|
||||
the HA host).
|
||||
|
||||
### v0.8 verbs — browser (headful anti-bot automation)
|
||||
|
||||
Drive the cluster's **headful** Chrome (`chrome-service`, real Chrome under Xvfb)
|
||||
from the devvm over CDP, for sites that detect and block headless automation. The
|
||||
headless `@playwright/mcp` browser can *load* such a site and fill its forms, but
|
||||
the gated action (submit/login) silently fails — the motivating case was the
|
||||
Stirling Ackroyd Fixflo tenant portal, whose pre-submit check returned
|
||||
`net::ERR_FILE_NOT_FOUND` and hung. This path connects via `connect_over_cdp`,
|
||||
injects the same `stealth.js` the in-cluster callers use, and submits first try.
|
||||
|
||||
The command owns only the *mechanics* (port-forward, stealth, lifecycle); the
|
||||
agent supplies the Playwright script — judgment stays out of the CLI.
|
||||
|
||||
| Command | Tier | What it does |
|
||||
|---|---|---|
|
||||
| `browser run <script.js> [--url U] [--shared-context] [--keep-open] [--port N] [--timeout S]` | write | port-forward `svc/chrome-service:9222`, assert it's a real (non-headless) Chrome via `/json/version`, `connect_over_cdp`, `addInitScript(stealth.js)`, then run the script with `page`/`context`/`browser`/`log` in scope (top-level await ok; return a value to print it). Always tears the forward down. |
|
||||
| `browser open <url> [--shared-context] [--timeout S]` | write | open `<url>` headful and print title + visible text + a screenshot path — a quick check. |
|
||||
| `browser --help` | read | when-to-use signature + the error-code cheat-sheet (`ERR_FILE_NOT_FOUND` = automation-layer intercept, not egress; `ERR_CONNECTION_REFUSED`/`_TIMED_OUT`/`_NAME_NOT_RESOLVED` = real egress; one endpoint 500 while siblings 200 = bot rejection). |
|
||||
|
||||
Default context is a **fresh incognito** one (closed on exit) — safe for the
|
||||
shared browser and concurrent callers (e.g. tripit's fare scrape); `--shared-context`
|
||||
reuses the warmed persistent profile when a pre-logged-in session is needed.
|
||||
`port-forward` tunnels API-server→pod, so it bypasses the `:9222` NetworkPolicy
|
||||
that gates in-cluster callers — no namespace label needed. The node CDP client is
|
||||
pinned to **`playwright-core@1.48.2`** to match the chrome-service image minor
|
||||
(Chromium 130; protocol changes between minors) and is installed once, lazily,
|
||||
into `~/.cache/homelab/browser-client/` (no per-user setup). Because the client
|
||||
runs on the devvm, `setInputFiles` streams local files to the remote browser over
|
||||
CDP — no `chmod`/staging-dir workaround. See `docs/architecture/chrome-service.md`
|
||||
and `docs/adr/0013`.
|
||||
|
||||
## Build / install
|
||||
|
||||
Built from source to `/usr/local/bin/homelab` during devvm provisioning
|
||||
|
|
@ -190,4 +221,4 @@ original flag-based path unchanged, so the webhook handler is unaffected.
|
|||
|
||||
## Design
|
||||
|
||||
See `infra/docs/adr/0004`–`0012` for the architecture decisions.
|
||||
See `infra/docs/adr/0004`–`0013` for the architecture decisions.
|
||||
|
|
|
|||
|
|
@ -1 +1 @@
|
|||
v0.7.1
|
||||
v0.8.0
|
||||
|
|
|
|||
388
cli/browser.go
Normal file
388
cli/browser.go
Normal file
|
|
@ -0,0 +1,388 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
_ "embed"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// playwrightVersion pins the node CDP client to the chrome-service image minor
|
||||
// (mcr.microsoft.com/playwright:v1.48.0-noble → Chromium 130). connect_over_cdp
|
||||
// speaks the browser's CDP, so the client minor must track the server minor;
|
||||
// see docs/architecture/chrome-service.md "Image pin".
|
||||
const playwrightVersion = "1.48.2"
|
||||
|
||||
// defaultBrowserTimeout is how long (seconds) to wait for the port-forwarded CDP
|
||||
// endpoint to become ready before giving up.
|
||||
const defaultBrowserTimeout = 60
|
||||
|
||||
const (
|
||||
chromeServiceNamespace = "chrome-service"
|
||||
chromeServiceName = "chrome-service"
|
||||
chromeServiceCDPPort = 9222
|
||||
)
|
||||
|
||||
// stealthJS is vendored verbatim from stacks/chrome-service/files/stealth.js (the
|
||||
// source of truth the in-cluster callers use). TestStealthJSEmbeddedMatchesCanonical
|
||||
// guards against drift.
|
||||
//
|
||||
//go:embed browser_stealth.js
|
||||
var stealthJS string
|
||||
|
||||
// runnerJS is the node wrapper that connects to the port-forwarded CDP endpoint,
|
||||
// installs the stealth init script, and runs the user's Playwright script.
|
||||
//
|
||||
//go:embed browser_runner.js
|
||||
var runnerJS string
|
||||
|
||||
// browserOpts is the parsed form of `homelab browser run|open` arguments.
|
||||
type browserOpts struct {
|
||||
mode string // "run" | "open"
|
||||
script string // path to the user Playwright script (run mode)
|
||||
url string // initial URL (run: optional; open: required positional)
|
||||
sharedCtx bool // use the warmed persistent profile instead of a fresh context
|
||||
keepOpen bool // leave the created context/pages open on exit
|
||||
port int // explicit local port for the forward (0 = auto)
|
||||
timeout int // CDP readiness timeout, seconds
|
||||
help bool
|
||||
}
|
||||
|
||||
// parseBrowserArgs parses the args after `browser run` / `browser open`.
|
||||
func parseBrowserArgs(mode string, args []string) (browserOpts, error) {
|
||||
o := browserOpts{mode: mode, timeout: defaultBrowserTimeout}
|
||||
var positionals []string
|
||||
atoi := func(s, flag string) (int, error) {
|
||||
n, err := strconv.Atoi(s)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("%s expects an integer, got %q", flag, s)
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
for i := 0; i < len(args); i++ {
|
||||
a := args[i]
|
||||
switch {
|
||||
case a == "-h" || a == "--help":
|
||||
o.help = true
|
||||
case a == "--shared-context":
|
||||
o.sharedCtx = true
|
||||
case a == "--keep-open":
|
||||
o.keepOpen = true
|
||||
case a == "--url":
|
||||
if i+1 < len(args) {
|
||||
o.url = args[i+1]
|
||||
i++
|
||||
}
|
||||
case strings.HasPrefix(a, "--url="):
|
||||
o.url = strings.TrimPrefix(a, "--url=")
|
||||
case a == "--port":
|
||||
if i+1 < len(args) {
|
||||
n, err := atoi(args[i+1], "--port")
|
||||
if err != nil {
|
||||
return o, err
|
||||
}
|
||||
o.port = n
|
||||
i++
|
||||
}
|
||||
case strings.HasPrefix(a, "--port="):
|
||||
n, err := atoi(strings.TrimPrefix(a, "--port="), "--port")
|
||||
if err != nil {
|
||||
return o, err
|
||||
}
|
||||
o.port = n
|
||||
case a == "--timeout":
|
||||
if i+1 < len(args) {
|
||||
n, err := atoi(args[i+1], "--timeout")
|
||||
if err != nil {
|
||||
return o, err
|
||||
}
|
||||
o.timeout = n
|
||||
i++
|
||||
}
|
||||
case strings.HasPrefix(a, "--timeout="):
|
||||
n, err := atoi(strings.TrimPrefix(a, "--timeout="), "--timeout")
|
||||
if err != nil {
|
||||
return o, err
|
||||
}
|
||||
o.timeout = n
|
||||
case strings.HasPrefix(a, "-"):
|
||||
return o, fmt.Errorf("unknown flag %q (try: homelab browser --help)", a)
|
||||
default:
|
||||
positionals = append(positionals, a)
|
||||
}
|
||||
}
|
||||
if o.help {
|
||||
return o, nil
|
||||
}
|
||||
switch mode {
|
||||
case "run":
|
||||
if len(positionals) == 0 {
|
||||
return o, fmt.Errorf("usage: homelab browser run <script.js> [--url URL] [--shared-context] [--keep-open] [--port N] [--timeout S]")
|
||||
}
|
||||
o.script = positionals[0]
|
||||
case "open":
|
||||
if len(positionals) == 0 {
|
||||
return o, fmt.Errorf("usage: homelab browser open <url> [--shared-context] [--timeout S]")
|
||||
}
|
||||
o.url = positionals[0]
|
||||
}
|
||||
return o, nil
|
||||
}
|
||||
|
||||
// cdpHealthy parses a CDP /json/version body and reports whether the endpoint is
|
||||
// a real (non-headless) Chrome — the entire reason chrome-service exists.
|
||||
func cdpHealthy(jsonBody []byte) (browser string, healthy bool, err error) {
|
||||
var v struct {
|
||||
Browser string `json:"Browser"`
|
||||
UserAgent string `json:"User-Agent"`
|
||||
}
|
||||
if e := json.Unmarshal(jsonBody, &v); e != nil {
|
||||
return "", false, fmt.Errorf("parse /json/version: %w", e)
|
||||
}
|
||||
if v.Browser == "" {
|
||||
return "", false, fmt.Errorf("/json/version had no Browser field")
|
||||
}
|
||||
healthy = strings.HasPrefix(v.Browser, "Chrome/") &&
|
||||
!strings.Contains(v.Browser, "Headless") &&
|
||||
!strings.Contains(v.UserAgent, "Headless")
|
||||
return v.Browser, healthy, nil
|
||||
}
|
||||
|
||||
// buildPortForwardArgs is the kubectl invocation that exposes chrome-service's
|
||||
// CDP locally. port-forward tunnels API-server→pod, so it bypasses the :9222
|
||||
// NetworkPolicy that gates in-cluster callers.
|
||||
func buildPortForwardArgs(localPort int) []string {
|
||||
return []string{"-n", chromeServiceNamespace, "port-forward",
|
||||
"svc/" + chromeServiceName, fmt.Sprintf("%d:%d", localPort, chromeServiceCDPPort)}
|
||||
}
|
||||
|
||||
// browserClientPackageJSON is the auto-managed manifest for the pinned node CDP
|
||||
// client kept under the user cache dir.
|
||||
func browserClientPackageJSON() string {
|
||||
return fmt.Sprintf(`{
|
||||
"name": "homelab-browser-client",
|
||||
"private": true,
|
||||
"description": "Pinned CDP client for 'homelab browser' — auto-managed, do not edit.",
|
||||
"dependencies": {
|
||||
"playwright-core": "%s"
|
||||
}
|
||||
}
|
||||
`, playwrightVersion)
|
||||
}
|
||||
|
||||
// freePort asks the kernel for an unused ephemeral TCP port.
|
||||
func freePort() (int, error) {
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer l.Close()
|
||||
return l.Addr().(*net.TCPAddr).Port, nil
|
||||
}
|
||||
|
||||
// browserClientDir is where the pinned node client + managed runner files live.
|
||||
func browserClientDir() (string, error) {
|
||||
cache, err := os.UserCacheDir()
|
||||
if err != nil || cache == "" {
|
||||
home, herr := os.UserHomeDir()
|
||||
if herr != nil {
|
||||
return "", fmt.Errorf("locate cache dir: %v / %v", err, herr)
|
||||
}
|
||||
cache = filepath.Join(home, ".cache")
|
||||
}
|
||||
return filepath.Join(cache, "homelab", "browser-client"), nil
|
||||
}
|
||||
|
||||
// installedPlaywrightVersion reads the version of the playwright-core already
|
||||
// installed in dir, or "" if absent/unreadable.
|
||||
func installedPlaywrightVersion(dir string) string {
|
||||
b, err := os.ReadFile(filepath.Join(dir, "node_modules", "playwright-core", "package.json"))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
var v struct {
|
||||
Version string `json:"version"`
|
||||
}
|
||||
if json.Unmarshal(b, &v) != nil {
|
||||
return ""
|
||||
}
|
||||
return v.Version
|
||||
}
|
||||
|
||||
// ensureBrowserClient writes the managed runner/stealth/package files into dir
|
||||
// and lazily installs the pinned playwright-core (only when missing/mismatched),
|
||||
// so no per-user setup is needed and the client tracks the binary version.
|
||||
func ensureBrowserClient(dir string) error {
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
files := map[string]string{
|
||||
"package.json": browserClientPackageJSON(),
|
||||
"browser_runner.js": runnerJS,
|
||||
"stealth.js": stealthJS,
|
||||
}
|
||||
for name, content := range files {
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if installedPlaywrightVersion(dir) == playwrightVersion {
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "homelab browser: installing pinned playwright-core@%s (one-time, ~a few seconds)…\n", playwrightVersion)
|
||||
cmd := exec.Command("npm", "install", "--no-audit", "--no-fund", "--silent")
|
||||
cmd.Dir = dir
|
||||
cmd.Stdout = os.Stderr
|
||||
cmd.Stderr = os.Stderr
|
||||
if err := cmd.Run(); err != nil {
|
||||
return fmt.Errorf("npm install playwright-core@%s in %s: %w (is node/npm installed?)", playwrightVersion, dir, err)
|
||||
}
|
||||
if got := installedPlaywrightVersion(dir); got != playwrightVersion {
|
||||
return fmt.Errorf("playwright-core install mismatch in %s: want %s, got %q", dir, playwrightVersion, got)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// waitForCDP polls the local CDP endpoint until it answers as a healthy
|
||||
// (non-headless) Chrome, or the timeout elapses.
|
||||
func waitForCDP(cdpURL string, timeout time.Duration) (string, error) {
|
||||
deadline := time.Now().Add(timeout)
|
||||
client := &http.Client{Timeout: 3 * time.Second}
|
||||
var lastErr error
|
||||
for time.Now().Before(deadline) {
|
||||
resp, err := client.Get(cdpURL + "/json/version")
|
||||
if err != nil {
|
||||
lastErr = err
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
continue
|
||||
}
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
browser, healthy, herr := cdpHealthy(body)
|
||||
if herr != nil {
|
||||
lastErr = herr
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
continue
|
||||
}
|
||||
if !healthy {
|
||||
return browser, fmt.Errorf("CDP reports %q — expected a non-headless Chrome (wrong target?)", browser)
|
||||
}
|
||||
return browser, nil
|
||||
}
|
||||
if lastErr == nil {
|
||||
lastErr = fmt.Errorf("timed out after %s", timeout)
|
||||
}
|
||||
return "", lastErr
|
||||
}
|
||||
|
||||
// runBrowser is the orchestration: pick a port, ensure the pinned client, start
|
||||
// (and ALWAYS tear down) a CDP port-forward, wait for readiness, then run node.
|
||||
func runBrowser(o browserOpts) error {
|
||||
port := o.port
|
||||
if port == 0 {
|
||||
p, err := freePort()
|
||||
if err != nil {
|
||||
return fmt.Errorf("pick local port: %w", err)
|
||||
}
|
||||
port = p
|
||||
}
|
||||
|
||||
dir, err := browserClientDir()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := ensureBrowserClient(dir); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Start the forward in its own process group so the whole tree dies on cleanup.
|
||||
pf := exec.Command("kubectl", buildPortForwardArgs(port)...)
|
||||
pf.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
var pfLog strings.Builder
|
||||
pf.Stdout = &pfLog
|
||||
pf.Stderr = &pfLog
|
||||
if err := pf.Start(); err != nil {
|
||||
return fmt.Errorf("start kubectl port-forward (kubeconfig set?): %w", err)
|
||||
}
|
||||
|
||||
var once sync.Once
|
||||
teardown := func() {
|
||||
once.Do(func() {
|
||||
if pf.Process != nil {
|
||||
_ = syscall.Kill(-pf.Process.Pid, syscall.SIGKILL)
|
||||
}
|
||||
_ = pf.Wait()
|
||||
})
|
||||
}
|
||||
defer teardown()
|
||||
|
||||
// Tear down on Ctrl-C / SIGTERM too, then exit non-zero.
|
||||
sigCh := make(chan os.Signal, 1)
|
||||
signal.Notify(sigCh, os.Interrupt, syscall.SIGTERM)
|
||||
defer signal.Stop(sigCh)
|
||||
go func() {
|
||||
if _, ok := <-sigCh; ok {
|
||||
teardown()
|
||||
os.Exit(130)
|
||||
}
|
||||
}()
|
||||
|
||||
cdpURL := fmt.Sprintf("http://127.0.0.1:%d", port)
|
||||
browser, err := waitForCDP(cdpURL, time.Duration(o.timeout)*time.Second)
|
||||
if err != nil {
|
||||
return fmt.Errorf("chrome-service CDP not ready on %s: %w\n--- port-forward log ---\n%s", cdpURL, err, pfLog.String())
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "homelab browser: connected to %s via %s\n", browser, cdpURL)
|
||||
|
||||
return runBrowserNode(dir, cdpURL, o)
|
||||
}
|
||||
|
||||
// runBrowserNode invokes the managed node runner with inputs passed via env.
|
||||
func runBrowserNode(dir, cdpURL string, o browserOpts) error {
|
||||
env := append(os.Environ(),
|
||||
"HOMELAB_CDP_URL="+cdpURL,
|
||||
"HOMELAB_BROWSER_MODE="+o.mode,
|
||||
"HOMELAB_STEALTH_PATH="+filepath.Join(dir, "stealth.js"),
|
||||
"NODE_PATH="+filepath.Join(dir, "node_modules"),
|
||||
)
|
||||
if o.url != "" {
|
||||
env = append(env, "HOMELAB_BROWSER_URL="+o.url)
|
||||
}
|
||||
if o.script != "" {
|
||||
abs, err := filepath.Abs(o.script)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := os.Stat(abs); err != nil {
|
||||
return fmt.Errorf("script %s: %w", o.script, err)
|
||||
}
|
||||
env = append(env, "HOMELAB_BROWSER_SCRIPT="+abs)
|
||||
}
|
||||
if o.sharedCtx {
|
||||
env = append(env, "HOMELAB_BROWSER_SHARED=1")
|
||||
}
|
||||
if o.keepOpen {
|
||||
env = append(env, "HOMELAB_BROWSER_KEEP_OPEN=1")
|
||||
}
|
||||
if o.mode == "open" {
|
||||
shot := filepath.Join(os.TempDir(), fmt.Sprintf("homelab-browser-%d.png", os.Getpid()))
|
||||
env = append(env, "HOMELAB_BROWSER_SCREENSHOT="+shot)
|
||||
}
|
||||
cmd := exec.Command("node", filepath.Join(dir, "browser_runner.js"))
|
||||
cmd.Env = env
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
cmd.Stdin = os.Stdin
|
||||
return cmd.Run()
|
||||
}
|
||||
106
cli/browser_runner.js
Normal file
106
cli/browser_runner.js
Normal file
|
|
@ -0,0 +1,106 @@
|
|||
// homelab browser — node CDP runner (auto-managed; regenerated each run from the
|
||||
// homelab binary — DO NOT EDIT here). Connects to the port-forwarded
|
||||
// chrome-service CDP endpoint, installs the stealth init script, then runs the
|
||||
// user's Playwright script (run mode) or opens a URL (open mode). All inputs
|
||||
// arrive via HOMELAB_* env vars set by the Go CLI.
|
||||
'use strict';
|
||||
const fs = require('fs');
|
||||
const { chromium } = require('playwright-core');
|
||||
|
||||
async function main() {
|
||||
const cdpURL = process.env.HOMELAB_CDP_URL;
|
||||
if (!cdpURL) throw new Error('HOMELAB_CDP_URL not set');
|
||||
const mode = process.env.HOMELAB_BROWSER_MODE || 'run';
|
||||
const stealthPath = process.env.HOMELAB_STEALTH_PATH || '';
|
||||
const initURL = process.env.HOMELAB_BROWSER_URL || '';
|
||||
const scriptPath = process.env.HOMELAB_BROWSER_SCRIPT || '';
|
||||
const shared = process.env.HOMELAB_BROWSER_SHARED === '1';
|
||||
const keepOpen = process.env.HOMELAB_BROWSER_KEEP_OPEN === '1';
|
||||
const screenshotPath = process.env.HOMELAB_BROWSER_SCREENSHOT || '';
|
||||
|
||||
const browser = await chromium.connectOverCDP(cdpURL);
|
||||
|
||||
// Fresh isolated context by default (safe for the shared browser + concurrent
|
||||
// callers); --shared-context reuses the warmed persistent profile.
|
||||
let context;
|
||||
let createdContext = false;
|
||||
if (shared) {
|
||||
const existing = browser.contexts();
|
||||
if (existing.length) {
|
||||
context = existing[0];
|
||||
} else {
|
||||
context = await browser.newContext();
|
||||
createdContext = true;
|
||||
}
|
||||
} else {
|
||||
context = await browser.newContext();
|
||||
createdContext = true;
|
||||
}
|
||||
|
||||
if (stealthPath) {
|
||||
const stealth = fs.readFileSync(stealthPath, 'utf8');
|
||||
if (stealth.trim()) await context.addInitScript(stealth);
|
||||
}
|
||||
|
||||
const page = await context.newPage();
|
||||
const log = (...a) => console.error('[browser]', ...a);
|
||||
|
||||
let exitCode = 0;
|
||||
try {
|
||||
if (initURL) {
|
||||
await page.goto(initURL, { waitUntil: 'domcontentloaded' });
|
||||
}
|
||||
if (mode === 'open') {
|
||||
console.log('url: ' + page.url());
|
||||
console.log('title: ' + (await page.title()));
|
||||
const text = (await page.evaluate(() => (document.body ? document.body.innerText : ''))).trim();
|
||||
console.log('--- visible text (truncated to 4000 chars) ---');
|
||||
console.log(text.slice(0, 4000));
|
||||
if (screenshotPath) {
|
||||
await page.screenshot({ path: screenshotPath, fullPage: true });
|
||||
console.log('screenshot: ' + screenshotPath);
|
||||
}
|
||||
} else {
|
||||
if (!scriptPath) throw new Error('run mode requires HOMELAB_BROWSER_SCRIPT');
|
||||
const src = fs.readFileSync(scriptPath, 'utf8');
|
||||
// Run the user's source with page/context/browser/log in lexical scope.
|
||||
// AsyncFunction body permits top-level await.
|
||||
const AsyncFunction = Object.getPrototypeOf(async () => {}).constructor;
|
||||
const fn = new AsyncFunction('page', 'context', 'browser', 'log', src);
|
||||
const result = await fn(page, context, browser, log);
|
||||
if (result !== undefined) {
|
||||
let out;
|
||||
try {
|
||||
out = typeof result === 'string' ? result : JSON.stringify(result, null, 2);
|
||||
} catch (_) {
|
||||
out = String(result);
|
||||
}
|
||||
console.log(out);
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
console.error('homelab browser: script error:', e && e.stack ? e.stack : e);
|
||||
exitCode = 1;
|
||||
} finally {
|
||||
if (!keepOpen) {
|
||||
try {
|
||||
// Close only what we created; never tear down the shared persistent context.
|
||||
if (createdContext) {
|
||||
await context.close();
|
||||
} else {
|
||||
await page.close();
|
||||
}
|
||||
} catch (_) { /* ignore */ }
|
||||
}
|
||||
// Disconnect from the CDP endpoint; this does NOT kill the remote browser.
|
||||
try {
|
||||
await browser.close();
|
||||
} catch (_) { /* ignore */ }
|
||||
}
|
||||
process.exit(exitCode);
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
console.error('homelab browser: fatal:', e && e.stack ? e.stack : e);
|
||||
process.exit(1);
|
||||
});
|
||||
54
cli/browser_stealth.js
Normal file
54
cli/browser_stealth.js
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
// Minimal stealth init script for Playwright-driven Chromium.
|
||||
// Vendored from puppeteer-extra-plugin-stealth/evasions/* (MIT) — covers:
|
||||
// webdriver, chrome.runtime, navigator.plugins, navigator.languages,
|
||||
// Permissions.query, WebGL getParameter (vendor + renderer spoof).
|
||||
// Run via context.add_init_script() so it executes before any page script.
|
||||
(() => {
|
||||
// navigator.webdriver — most common detection, removed entirely.
|
||||
Object.defineProperty(Navigator.prototype, 'webdriver', { get: () => undefined });
|
||||
|
||||
// window.chrome.runtime — many sites check that real Chrome exposes this.
|
||||
if (!window.chrome) window.chrome = {};
|
||||
window.chrome.runtime = window.chrome.runtime || {};
|
||||
|
||||
// navigator.plugins — headless reports zero; spoof a plausible PDF viewer.
|
||||
Object.defineProperty(navigator, 'plugins', {
|
||||
get: () => [{ name: 'Chrome PDF Plugin' }, { name: 'Chrome PDF Viewer' }, { name: 'Native Client' }],
|
||||
});
|
||||
|
||||
// navigator.languages — headless returns empty array.
|
||||
Object.defineProperty(navigator, 'languages', { get: () => ['en-US', 'en'] });
|
||||
|
||||
// Permissions.query — headless returns 'denied' for notifications instead of 'default'.
|
||||
const origQuery = window.navigator.permissions && window.navigator.permissions.query;
|
||||
if (origQuery) {
|
||||
window.navigator.permissions.query = (parameters) =>
|
||||
parameters && parameters.name === 'notifications'
|
||||
? Promise.resolve({ state: Notification.permission })
|
||||
: origQuery(parameters);
|
||||
}
|
||||
|
||||
// WebGL getParameter — spoof vendor + renderer strings to a real GPU.
|
||||
const spoofGl = (proto) => {
|
||||
if (!proto) return;
|
||||
const orig = proto.getParameter;
|
||||
proto.getParameter = function (parameter) {
|
||||
if (parameter === 37445) return 'Intel Inc.'; // UNMASKED_VENDOR_WEBGL
|
||||
if (parameter === 37446) return 'Intel Iris OpenGL Engine'; // UNMASKED_RENDERER_WEBGL
|
||||
return orig.apply(this, arguments);
|
||||
};
|
||||
};
|
||||
spoofGl(window.WebGLRenderingContext && window.WebGLRenderingContext.prototype);
|
||||
spoofGl(window.WebGL2RenderingContext && window.WebGL2RenderingContext.prototype);
|
||||
|
||||
// disable-devtool.js (theajack/disable-devtool) auto-inits via a script
|
||||
// tag with `disable-devtool-auto`. Its Performance detector trips under
|
||||
// Playwright (CDP adds console.log latency vs console.table) and the
|
||||
// redirect URL is hard-coded — for hmembeds that's google.com.
|
||||
// Hide the auto-init marker so the library's IIFE exits early.
|
||||
const origQS = Document.prototype.querySelector;
|
||||
Document.prototype.querySelector = function (sel) {
|
||||
if (typeof sel === 'string' && sel.indexOf('disable-devtool-auto') !== -1) return null;
|
||||
return origQS.apply(this, arguments);
|
||||
};
|
||||
})();
|
||||
113
cli/cmd_browser.go
Normal file
113
cli/cmd_browser.go
Normal file
|
|
@ -0,0 +1,113 @@
|
|||
package main
|
||||
|
||||
import "fmt"
|
||||
|
||||
// browser verbs drive the cluster's HEADFUL Chrome (ns chrome-service) over CDP
|
||||
// from outside the cluster, for sites that detect/block headless automation.
|
||||
// The headless @playwright/mcp browser can load such sites but their gated
|
||||
// actions (submit/login) silently fail; this path submits first try. Mechanics
|
||||
// only — the agent supplies the Playwright script. See docs/adr/0013.
|
||||
|
||||
func browserCommands() []Command {
|
||||
return []Command{
|
||||
{Path: []string{"browser"}, Tier: TierRead,
|
||||
Summary: "headful cluster-Chrome automation for anti-bot sites (run `browser --help`)", Run: browserTopHelp},
|
||||
{Path: []string{"browser", "run"}, Tier: TierWrite,
|
||||
Summary: "run a Playwright script against headful cluster Chrome: browser run <script.js> [--url U] [--shared-context]", Run: browserRun},
|
||||
{Path: []string{"browser", "open"}, Tier: TierWrite,
|
||||
Summary: "open a URL in headful cluster Chrome; print title + text + screenshot: browser open <url>", Run: browserOpen},
|
||||
}
|
||||
}
|
||||
|
||||
func browserTopHelp([]string) error {
|
||||
fmt.Print(browserHelp())
|
||||
return nil
|
||||
}
|
||||
|
||||
func browserRun(args []string) error {
|
||||
o, err := parseBrowserArgs("run", args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if o.help {
|
||||
fmt.Print(browserHelp())
|
||||
return nil
|
||||
}
|
||||
return runBrowser(o)
|
||||
}
|
||||
|
||||
func browserOpen(args []string) error {
|
||||
o, err := parseBrowserArgs("open", args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if o.help {
|
||||
fmt.Print(browserHelp())
|
||||
return nil
|
||||
}
|
||||
return runBrowser(o)
|
||||
}
|
||||
|
||||
// browserHelp carries the discoverability payload: WHEN to reach for this, and
|
||||
// the diagnostic cheat-sheet that lets the agent self-correct instead of
|
||||
// retrying a deterministic form blind (the failure mode that motivated this).
|
||||
func browserHelp() string {
|
||||
return `homelab browser — drive the cluster's HEADFUL Chrome (anti-bot) over CDP
|
||||
|
||||
The shared chrome-service (ns chrome-service) runs a REAL, headed Chrome under
|
||||
Xvfb. This connects to it via a port-forward + Playwright connect_over_cdp,
|
||||
injects the same stealth.js the in-cluster callers use, and runs your script.
|
||||
|
||||
USAGE
|
||||
homelab browser run <script.js> [--url URL] [--shared-context] [--keep-open] [--port N] [--timeout S]
|
||||
homelab browser open <url> [--shared-context] [--timeout S]
|
||||
|
||||
WHEN TO USE THIS (not the headless @playwright/mcp browser)
|
||||
A site LOADS fine but a gated action FAILS or HANGS — a submit/login/checkout
|
||||
spins forever, or ONE request 500s / aborts while its siblings 200. That is
|
||||
the signature of headless / anti-bot detection (navigator.webdriver, UA
|
||||
"HeadlessChrome", disable-devtool traps). Switch to this command — it presents
|
||||
as a real Chrome and typically succeeds first try.
|
||||
|
||||
ERROR-CODE CHEAT-SHEET (diagnose BEFORE retrying)
|
||||
ERR_FILE_NOT_FOUND (-6) request intercepted/resolved locally by the
|
||||
automation layer — NOT a network/egress problem.
|
||||
(This is what silently broke the headless submit.)
|
||||
ERR_CONNECTION_REFUSED / real egress failure (DNS/route/firewall). These also
|
||||
ERR_TIMED_OUT / break the initial page load — if the page loaded,
|
||||
ERR_NAME_NOT_RESOLVED egress is fine and the cause is elsewhere.
|
||||
one endpoint 500s while server-side bot rejection of the automation, not
|
||||
its siblings 200 your payload.
|
||||
|
||||
HABITS
|
||||
- Inspect the network panel BEFORE retrying a deterministic form; a blind
|
||||
retry just repeats the same silent failure.
|
||||
- Don't park a half-filled multi-step form across a user pause — the session
|
||||
can expire; re-run the whole flow from this command in one shot.
|
||||
- Uploads stream over CDP via setInputFiles from THIS host — no chmod/staging
|
||||
of $HOME needed; just point setInputFiles at a local path.
|
||||
|
||||
CONTEXT
|
||||
Default: a FRESH incognito context, closed on exit — safe for the shared
|
||||
browser and concurrent callers (e.g. tripit). Your script does its own login.
|
||||
--shared-context: reuse the warmed PERSISTENT profile (cookies from a manual
|
||||
noVNC login at chrome.viktorbarzin.me) when you need a pre-logged-in session.
|
||||
|
||||
SCRIPT CONTRACT (run mode)
|
||||
Your file's body runs with page, context, browser and log() already in scope
|
||||
(top-level await allowed). Return a value to print it. Example flow.js:
|
||||
|
||||
await page.goto('https://portal.example.com/login');
|
||||
await page.fill('#user', 'me'); await page.fill('#pass', process.env.PW);
|
||||
await page.click('button[type=submit]');
|
||||
await page.waitForURL('**/dashboard');
|
||||
return 'logged in: ' + page.url();
|
||||
|
||||
Run it: homelab browser run flow.js
|
||||
|
||||
NOTES
|
||||
- The Playwright client is pinned to playwright-core@` + playwrightVersion + ` to match the
|
||||
chrome-service image (Chrome 130); installed once into ~/.cache/homelab/.
|
||||
- The port-forward is always torn down, on success and on error.
|
||||
`
|
||||
}
|
||||
160
cli/cmd_browser_test.go
Normal file
160
cli/cmd_browser_test.go
Normal file
|
|
@ -0,0 +1,160 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParseBrowserArgsRun(t *testing.T) {
|
||||
got, err := parseBrowserArgs("run", []string{
|
||||
"flow.js", "--url", "https://example.com", "--shared-context",
|
||||
"--port", "19999", "--timeout", "45", "--keep-open",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("parseBrowserArgs run: unexpected err: %v", err)
|
||||
}
|
||||
want := browserOpts{
|
||||
mode: "run", script: "flow.js", url: "https://example.com",
|
||||
sharedCtx: true, keepOpen: true, port: 19999, timeout: 45,
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("parseBrowserArgs run =\n %+v\nwant\n %+v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseBrowserArgsRunDefaults(t *testing.T) {
|
||||
got, err := parseBrowserArgs("run", []string{"flow.js"})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected err: %v", err)
|
||||
}
|
||||
if got.script != "flow.js" || got.sharedCtx || got.keepOpen || got.port != 0 {
|
||||
t.Fatalf("defaults wrong: %+v", got)
|
||||
}
|
||||
if got.timeout != defaultBrowserTimeout {
|
||||
t.Fatalf("timeout default = %d, want %d", got.timeout, defaultBrowserTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseBrowserArgsRunRequiresScript(t *testing.T) {
|
||||
if _, err := parseBrowserArgs("run", []string{"--url", "https://x"}); err == nil {
|
||||
t.Fatalf("run without a script path should error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseBrowserArgsOpenRequiresURL(t *testing.T) {
|
||||
got, err := parseBrowserArgs("open", []string{"https://example.com"})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected err: %v", err)
|
||||
}
|
||||
if got.url != "https://example.com" || got.mode != "open" {
|
||||
t.Fatalf("open parse wrong: %+v", got)
|
||||
}
|
||||
if _, err := parseBrowserArgs("open", []string{}); err == nil {
|
||||
t.Fatalf("open without a URL should error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseBrowserArgsHelp(t *testing.T) {
|
||||
for _, a := range [][]string{{"--help"}, {"-h"}, {"flow.js", "--help"}} {
|
||||
got, err := parseBrowserArgs("run", a)
|
||||
if err != nil {
|
||||
t.Fatalf("help parse %v: %v", a, err)
|
||||
}
|
||||
if !got.help {
|
||||
t.Fatalf("args %v should set help", a)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseBrowserArgsEqualsForm(t *testing.T) {
|
||||
got, err := parseBrowserArgs("run", []string{"flow.js", "--url=https://x", "--port=8123", "--timeout=10"})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected err: %v", err)
|
||||
}
|
||||
if got.url != "https://x" || got.port != 8123 || got.timeout != 10 {
|
||||
t.Fatalf("--flag=value form not parsed: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCDPHealthy(t *testing.T) {
|
||||
real := []byte(`{"Browser":"Chrome/130.0.6723.31","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) Chrome/130.0.0.0 Safari/537.36","webSocketDebuggerUrl":"ws://127.0.0.1/devtools/browser/x"}`)
|
||||
browser, ok, err := cdpHealthy(real)
|
||||
if err != nil || !ok {
|
||||
t.Fatalf("real Chrome should be healthy: ok=%v err=%v", ok, err)
|
||||
}
|
||||
if !strings.HasPrefix(browser, "Chrome/") {
|
||||
t.Fatalf("browser = %q, want Chrome/ prefix", browser)
|
||||
}
|
||||
|
||||
headless := []byte(`{"Browser":"HeadlessChrome/130.0.6723.31","User-Agent":"Mozilla/5.0 HeadlessChrome/130.0.0.0"}`)
|
||||
if _, ok, _ := cdpHealthy(headless); ok {
|
||||
t.Fatalf("HeadlessChrome must be reported unhealthy (the whole point of chrome-service)")
|
||||
}
|
||||
|
||||
if _, _, err := cdpHealthy([]byte("not json")); err == nil {
|
||||
t.Fatalf("malformed /json/version body should error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPortForwardArgs(t *testing.T) {
|
||||
got := buildPortForwardArgs(18080)
|
||||
want := []string{"-n", "chrome-service", "port-forward", "svc/chrome-service", "18080:9222"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("buildPortForwardArgs =\n %v\nwant\n %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBrowserClientPackageJSONPinsVersion(t *testing.T) {
|
||||
pj := browserClientPackageJSON()
|
||||
if !strings.Contains(pj, `"playwright-core": "`+playwrightVersion+`"`) {
|
||||
t.Fatalf("package.json must pin playwright-core to %s; got:\n%s", playwrightVersion, pj)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlaywrightVersionPinnedToServerMinor(t *testing.T) {
|
||||
// chrome-service runs mcr.microsoft.com/playwright:v1.48.0-noble; the CDP
|
||||
// client minor MUST match (protocol changes between minors).
|
||||
if !strings.HasPrefix(playwrightVersion, "1.48.") {
|
||||
t.Fatalf("playwrightVersion = %q, must be 1.48.x to match the chrome-service image", playwrightVersion)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBrowserHelpHasDiagnosticCheatSheet(t *testing.T) {
|
||||
h := browserHelp()
|
||||
for _, want := range []string{
|
||||
"homelab browser run",
|
||||
"ERR_FILE_NOT_FOUND",
|
||||
"ERR_CONNECTION_REFUSED",
|
||||
"network panel",
|
||||
"headless",
|
||||
"--shared-context",
|
||||
} {
|
||||
if !strings.Contains(h, want) {
|
||||
t.Errorf("browser --help is missing %q (the discoverability/self-correction payload)", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStealthJSEmbeddedMatchesCanonical(t *testing.T) {
|
||||
// The embedded copy must never drift from the source of truth that the
|
||||
// in-cluster callers use, else the CLI's stealth and the cluster's diverge.
|
||||
canonical, err := os.ReadFile("../stacks/chrome-service/files/stealth.js")
|
||||
if err != nil {
|
||||
t.Fatalf("read canonical stealth.js: %v", err)
|
||||
}
|
||||
if stealthJS != string(canonical) {
|
||||
t.Fatalf("cli/browser_stealth.js has drifted from stacks/chrome-service/files/stealth.js — re-copy it")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFreePortReturnsUsablePort(t *testing.T) {
|
||||
p, err := freePort()
|
||||
if err != nil {
|
||||
t.Fatalf("freePort: %v", err)
|
||||
}
|
||||
if p <= 1024 || p > 65535 {
|
||||
t.Fatalf("freePort returned %d, want an ephemeral port", p)
|
||||
}
|
||||
}
|
||||
|
|
@ -22,6 +22,7 @@ func buildRegistry() []Command {
|
|||
reg = append(reg, obsCommands()...)
|
||||
reg = append(reg, usageCommands()...)
|
||||
reg = append(reg, haCommands()...)
|
||||
reg = append(reg, browserCommands()...)
|
||||
return reg
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue