homelab v0.8.0: browser verbs for headful anti-bot web automation
Add `homelab browser run|open` so agents can drive the cluster's headful Chrome (chrome-service) over CDP from the devvm. The headless playwright/mcp browser can load anti-bot sites and fill their forms, but the gated submit silently fails — e.g. the Stirling Ackroyd Fixflo tenant portal returned net::ERR_FILE_NOT_FOUND on its pre-submit check and hung, creating nothing. Driving the real headful Chrome submits first try. That capability already existed but was undiscoverable, so it cost ~40 min + redundant form re-runs to find; now it is one command, versioned, test-covered, and `browser --help` carries the when-to-use signature + an error-code cheat-sheet so the right tool is reached at the right moment (the failure was judgment, not setup). - port-forward svc/chrome-service:9222 (tunnels API-server->pod, so it bypasses the :9222 NetworkPolicy), assert non-headless via /json/version, connect_over_cdp, inject the same vendored stealth.js the in-cluster callers use; the port-forward is always torn down, on success and on error. - node CDP client pinned to playwright-core@1.48.2 to match the v1.48.0-noble image (Chromium 130); self-provisioned lazily into ~/.cache/homelab, no per-user setup. - default is a fresh incognito context (safe for the shared browser + concurrent callers); --shared-context reuses the warmed persistent profile. - TDD: cmd_browser_test.go covers arg parsing, headless detection, the version pin, the help cheat-sheet, and a stealth.js drift guard. Verified end-to-end against bot.sannysoft.com (real Chrome UA, webdriver hidden, plugins/WebGL spoofed) and `browser open`. - docs: README v0.8 section, ADR-0013, and a chrome-service.md "driving from outside the cluster" section. Closes: code-nepg Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
de163aa6af
commit
a6b52a5839
10 changed files with 966 additions and 2 deletions
113
cli/cmd_browser.go
Normal file
113
cli/cmd_browser.go
Normal file
|
|
@ -0,0 +1,113 @@
|
|||
package main
|
||||
|
||||
import "fmt"
|
||||
|
||||
// browser verbs drive the cluster's HEADFUL Chrome (ns chrome-service) over CDP
|
||||
// from outside the cluster, for sites that detect/block headless automation.
|
||||
// The headless @playwright/mcp browser can load such sites but their gated
|
||||
// actions (submit/login) silently fail; this path submits first try. Mechanics
|
||||
// only — the agent supplies the Playwright script. See docs/adr/0013.
|
||||
|
||||
func browserCommands() []Command {
|
||||
return []Command{
|
||||
{Path: []string{"browser"}, Tier: TierRead,
|
||||
Summary: "headful cluster-Chrome automation for anti-bot sites (run `browser --help`)", Run: browserTopHelp},
|
||||
{Path: []string{"browser", "run"}, Tier: TierWrite,
|
||||
Summary: "run a Playwright script against headful cluster Chrome: browser run <script.js> [--url U] [--shared-context]", Run: browserRun},
|
||||
{Path: []string{"browser", "open"}, Tier: TierWrite,
|
||||
Summary: "open a URL in headful cluster Chrome; print title + text + screenshot: browser open <url>", Run: browserOpen},
|
||||
}
|
||||
}
|
||||
|
||||
func browserTopHelp([]string) error {
|
||||
fmt.Print(browserHelp())
|
||||
return nil
|
||||
}
|
||||
|
||||
func browserRun(args []string) error {
|
||||
o, err := parseBrowserArgs("run", args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if o.help {
|
||||
fmt.Print(browserHelp())
|
||||
return nil
|
||||
}
|
||||
return runBrowser(o)
|
||||
}
|
||||
|
||||
func browserOpen(args []string) error {
|
||||
o, err := parseBrowserArgs("open", args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if o.help {
|
||||
fmt.Print(browserHelp())
|
||||
return nil
|
||||
}
|
||||
return runBrowser(o)
|
||||
}
|
||||
|
||||
// browserHelp carries the discoverability payload: WHEN to reach for this, and
|
||||
// the diagnostic cheat-sheet that lets the agent self-correct instead of
|
||||
// retrying a deterministic form blind (the failure mode that motivated this).
|
||||
func browserHelp() string {
|
||||
return `homelab browser — drive the cluster's HEADFUL Chrome (anti-bot) over CDP
|
||||
|
||||
The shared chrome-service (ns chrome-service) runs a REAL, headed Chrome under
|
||||
Xvfb. This connects to it via a port-forward + Playwright connect_over_cdp,
|
||||
injects the same stealth.js the in-cluster callers use, and runs your script.
|
||||
|
||||
USAGE
|
||||
homelab browser run <script.js> [--url URL] [--shared-context] [--keep-open] [--port N] [--timeout S]
|
||||
homelab browser open <url> [--shared-context] [--timeout S]
|
||||
|
||||
WHEN TO USE THIS (not the headless @playwright/mcp browser)
|
||||
A site LOADS fine but a gated action FAILS or HANGS — a submit/login/checkout
|
||||
spins forever, or ONE request 500s / aborts while its siblings 200. That is
|
||||
the signature of headless / anti-bot detection (navigator.webdriver, UA
|
||||
"HeadlessChrome", disable-devtool traps). Switch to this command — it presents
|
||||
as a real Chrome and typically succeeds first try.
|
||||
|
||||
ERROR-CODE CHEAT-SHEET (diagnose BEFORE retrying)
|
||||
ERR_FILE_NOT_FOUND (-6) request intercepted/resolved locally by the
|
||||
automation layer — NOT a network/egress problem.
|
||||
(This is what silently broke the headless submit.)
|
||||
ERR_CONNECTION_REFUSED / real egress failure (DNS/route/firewall). These also
|
||||
ERR_TIMED_OUT / break the initial page load — if the page loaded,
|
||||
ERR_NAME_NOT_RESOLVED egress is fine and the cause is elsewhere.
|
||||
one endpoint 500s while server-side bot rejection of the automation, not
|
||||
its siblings 200 your payload.
|
||||
|
||||
HABITS
|
||||
- Inspect the network panel BEFORE retrying a deterministic form; a blind
|
||||
retry just repeats the same silent failure.
|
||||
- Don't park a half-filled multi-step form across a user pause — the session
|
||||
can expire; re-run the whole flow from this command in one shot.
|
||||
- Uploads stream over CDP via setInputFiles from THIS host — no chmod/staging
|
||||
of $HOME needed; just point setInputFiles at a local path.
|
||||
|
||||
CONTEXT
|
||||
Default: a FRESH incognito context, closed on exit — safe for the shared
|
||||
browser and concurrent callers (e.g. tripit). Your script does its own login.
|
||||
--shared-context: reuse the warmed PERSISTENT profile (cookies from a manual
|
||||
noVNC login at chrome.viktorbarzin.me) when you need a pre-logged-in session.
|
||||
|
||||
SCRIPT CONTRACT (run mode)
|
||||
Your file's body runs with page, context, browser and log() already in scope
|
||||
(top-level await allowed). Return a value to print it. Example flow.js:
|
||||
|
||||
await page.goto('https://portal.example.com/login');
|
||||
await page.fill('#user', 'me'); await page.fill('#pass', process.env.PW);
|
||||
await page.click('button[type=submit]');
|
||||
await page.waitForURL('**/dashboard');
|
||||
return 'logged in: ' + page.url();
|
||||
|
||||
Run it: homelab browser run flow.js
|
||||
|
||||
NOTES
|
||||
- The Playwright client is pinned to playwright-core@` + playwrightVersion + ` to match the
|
||||
chrome-service image (Chrome 130); installed once into ~/.cache/homelab/.
|
||||
- The port-forward is always torn down, on success and on error.
|
||||
`
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue