pi-chrome-connector

Agent-first Chrome connector for the pi coding agent. Pi drives your real, signed-in Chrome through a small companion extension: tabs, keyboard and mouse, accessibility snapshots, screenshots, console, network (with bodies and HAR export), request mocking, storage, device emulation, iframes, dialogs, PDF.

Built for one workflow: the agent tests and verifies web apps in the browser you already use.

How it works

 pi (one bridge per session)                     Chrome (your profile)
 +--------------------------------+              +------------------------------+
 | chrome_* tools, /chrome        |   WebSocket  | pi-chrome-connector          |
 | WebSocket server 127.0.0.1     |<============>| (unpacked MV3 extension)     |
 |   :17417-17426                 |              | thin proxy for chrome.tabs,  |
 | CDP logic: snapshots, input,   |              | chrome.windows and           |
 |   console/network buffers,     |              | chrome.debugger (CDP)        |
 |   routes, emulation            |              +---------------+--------------+
 +--------------------------------+                              | DevTools Protocol
                                                          tabs and frames

• All automation logic runs inside pi in TypeScript. The extension is a ~300-line proxy that rarely changes, so package updates do not need an extension reload.
• The bridge only listens on loopback and only accepts sockets whose Origin is a chrome-extension:// URL, so web pages cannot reach it. It does not verify which extension is on the other end; read Security before installing this on a machine you share or alongside extensions you do not trust.
• The extension finds live sessions by fetching /status on each port in the range and opens a WebSocket only where a bridge answers. Ports with nothing behind them are probed with backoff (2, 4, 8, then every 15 s) and never show up as errors on chrome://extensions.
• A 20 s heartbeat over the socket keeps the extension's service worker alive (Chrome 116+), and a 30 s alarm wakes it if it ever sleeps.
• No authorization gate and no expiry: once the extension is connected, the tools work.
• The agent gets its own window; your tabs are left alone unless the agent is told to use one. When the connector launches or wakes Chrome, it marks its startup tab with a session-specific local URL. It reuses that window only if it contains just that tab and has the configured size. Otherwise it opens the agent window and removes only its marked startup tab, without leaving an extra empty window or closing your tabs.
• Agent-created tabs stay open when the session ends so you can inspect the result. /chrome cleanup closes them.

Install

pi install npm:pi-chrome-connector      # or: pi install /path/to/this/repo

Then, once, load the companion extension into Chrome:

/chrome onboard

It opens chrome://extensions, reveals the extension folder and copies its path. Turn on Developer mode, click Load unpacked, pick the folder (Cmd+Shift+G and paste on macOS). The extension shows up as "pi-chrome-connector". If you used pi-chrome before, remove its "Pi Chrome Connector" entry and pi remove npm:pi-chrome, since both register chrome_* tools. Then:

/chrome doctor

Everything should be a ✓. From now on, asking pi to "open localhost:3000 and check the dashboard" just works, including launching Chrome when it is closed.

Requirements: Chrome 125+ (Chrome for Testing and Chromium work too), Node 24+, pi 0.99+.

Tools

By default only one small tool, chrome, is declared to the model: calling it connects to Chrome and declares the chrome_* tools for the rest of the session. The 27 declarations below cost about 6k tokens per request, which sessions that never open a browser should not pay. Set "lazyTools": false to declare everything up front.

Tool	Purpose
chrome	Connect, report the current tab, declare the tools below (lazy mode only)
chrome_status	Connection state; launches Chrome if needed; lists secret names
chrome_tabs	list / new / use (by id, url, title) / close / focus
chrome_navigate	goto / back / forward / reload with load, domcontentloaded or networkidle
chrome_snapshot	Accessibility tree with refs (e12), [offscreen] and [clickable] markers
chrome_find	Elements by text, role, or CSS selector, with refs
chrome_click	Real mouse click by ref / selector / text / coordinates; occlusion detection
chrome_hover	Pointer move
chrome_fill	Set any form control's value with trusted input events; secret: NAME for credentials
chrome_type	Key-by-key typing for typeahead fields
chrome_press	Keys and chords: Enter, Control+a, Meta+Shift+p, sequences
chrome_scroll	Wheel scrolling of page or container; to top / bottom / element
chrome_select	<select> options by value or label
chrome_upload	Attach files to a file input without the OS picker
chrome_mouse	Raw move / down / up / wheel / drag
chrome_evaluate	JavaScript with awaited promises, refs bound as el, iframe support, CSP-proof
chrome_wait	selector state, text, function, URL, load state, networkidle, or plain time
chrome_screenshot	Viewport / full page / element, returned inline (at most 1024px wide unless scale is given) and saved to disk
chrome_console	console.*, uncaught exceptions, browser log entries; level / since / includes filters
chrome_network	Every request with headers, payloads, bodies, timing; filters; HAR export
chrome_route	Mock, abort, delay or capture requests by URL glob
chrome_dom	outerHTML, innerText, attributes, computed styles, box geometry, event listeners
chrome_storage	Cookies, localStorage, sessionStorage, clear site data
chrome_emulate	Device presets, viewport, UA, dark mode, reduced motion, locale, timezone, geolocation, offline, throttling
chrome_dialog	Accept / dismiss JavaScript dialogs, auto-handling policy
chrome_frames	List iframes (same-process and out-of-process)
chrome_performance	Navigation timing, Web Vitals, long tasks, resource summary, heap
chrome_pdf	Print to PDF

Finds and actions wait for their element. chrome_find, chrome_click, chrome_fill and the other element tools keep looking for up to 5 s (timeoutMs; 0 looks once), and clicks also wait for the element to be visible and uncovered, the way Playwright locators do. A page that has fired load is usually still rendering, so without this the call after a navigation or a click ran against the app shell and missed. A miss reports how long it waited and which page it was looking at.

Every action reports what changed since it started: navigation, new console errors, failed requests, open dialogs. Requests an ad blocker stopped, requests the page canceled itself, and requests a chrome_route rule aborted are counted apart from real failures, so telemetry noise never reads as an app error. Credential headers (authorization, cookie, API keys) are redacted in tool output, and so are credential-looking cookie and storage values (reveal: true shows them); the HAR export on disk stays complete. chrome_route mocks work for APIs on another origin than the page: CORS preflights are answered and access-control-allow-origin is added automatically. The bundled skill (skills/chrome-connector) teaches the agent the navigate → snapshot → act → wait → verify loop.

Commands

/chrome              status
/chrome onboard      load the companion extension (one time)
/chrome doctor       diagnostics
/chrome launch       launch Chrome and wait for the extension
/chrome tabs         list tabs
/chrome focus on|off bring Chrome to the front on actions (default on)
/chrome cleanup      close tabs this session opened

Configuration

Optional ~/.pi/agent/chrome-connector.json (or $PI_CHROME_CONNECTOR_CONFIG):

{
  "chromeApp": "Google Chrome",
  "launchArgs": ["--silent-debugger-extension-api"],
  "focus": true,
  "window": { "width": 1280, "height": 900 },
  "artifactDir": "~/.pi/agent/chrome-connector",
  "portRange": [17417, 17426],
  "reuseWindow": false,
  "lazyTools": true,
  "secrets": {
    "APP_EMAIL": "you@example.com",
    "APP_PASSWORD": "..."
  }
}

secrets lets the agent log in with chrome_fill secret: "APP_PASSWORD" without the value ever appearing in the conversation or the session transcript. Environment variables are used as a fallback for names not in the file. The agent can still read a value back out of the page after typing it (it is a browser, not a vault); treat this as convenience for test accounts, not as protection against a malicious model, and see Security for who else could receive a secret while pi is running. Logins persist because the agent runs in your real profile: cookies and "remember me" state survive Chrome restarts as usual. To test logged out, ask for chrome_storage store=site action=clear.

launchArgs only apply when the connector launches Chrome itself (open -a on macOS opens your default profile; the extension must be loaded in that profile). The default flag hides the "pi-chrome-connector started debugging this browser" bar. A Chrome you started yourself shows that bar; it is harmless, but its Cancel button detaches the agent from the tab (the next tool call reattaches). Quit Chrome and let the connector relaunch it if you want the bar gone.

Change portRange only if you also edit PORT_RANGE in chrome-extension/service_worker.js.

Connecting: each pi session runs its own bridge on the first free port of the range. The companion probes every port in the range every 2 s while its service worker is awake (any connected pi session keeps it awake); when no session is connected, Chrome stops the worker and an alarm wakes it about every 30 s. The extension only runs while its profile has a window: on macOS Chrome keeps running after its last window closes, and then nothing wakes the worker. So when a running Chrome stays silent for 3 s, the connector asks it to open a window (open -a, which uses the last-used profile) and keeps waiting, giving up after 60 s. The first browser call of a session therefore takes up to 2 s with another session open, about 3 to 5 s after all Chrome windows were closed, and up to about 30 s when a window is open but the worker is idle.

Security

Read this before installing. The trade-offs are deliberate, but you should accept them knowingly.

The bridge does not authenticate the extension on the other end. It listens on 127.0.0.1:17417-17426 and accepts any WebSocket whose Origin header starts with chrome-extension://. That check keeps web pages out, because browsers set Origin themselves and a page can never forge that value. It does not identify which extension connected, and any process running as your user can set that header by hand. So two things can take the companion's place:

• another extension installed in your Chrome profile, and
• any local program running as you.

An impostor that wins the handshake becomes "the browser" as far as pi is concerned. It receives whatever the agent sends, which includes values resolved from secrets, and it can answer with invented page content that the agent will then act on and report to you as fact. It cannot drive your real Chrome by itself; the damage is what it is told and what it tells the agent back.

What this does and does not change about your threat model:

• A local program running as you can already read your files, your Chrome profile and your cookies. If one is running, this package is not what is protecting you.
• Other browser extensions are the case worth thinking about. They are sandboxed from your filesystem but can open a WebSocket to localhost, so an extension that could not otherwise read your agent's traffic can read it while pi is running.
• Nothing off your machine can reach the bridge. It binds to loopback only.

How to keep the exposure small:

• Install browser extensions you trust, and prefer a Chrome profile without unknown ones.
• Put test-account credentials in secrets, not production ones.
• The bridge only exists while a pi session is running; it closes on exit.
• On a shared or multi-user machine, treat this as unsuitable until extension pinning lands.

Planned: pin the companion's extension id on first connect and require an explicit confirmation before a different id may take over. Until that ships, the paragraph above is the honest state.

Two more things worth knowing:

• The companion has debugger and <all_urls> permissions in the profile you load it into, and page content the agent reads is sent to your configured model provider.
• Credential-looking values are redacted in tool output (authorization headers, session cookies, JWT-shaped storage values), but a determined agent can still read a secret out of the page after typing it. Redaction reduces accidental exposure in transcripts; it is not a vault.

Limits

• Native Chrome UI is outside the DOM: file pickers (use chrome_upload), permission bubbles, password-manager popups, print dialogs, CAPTCHAs.
• Site permissions (geolocation, notifications) cannot be granted from the connector; Chrome prompts once and the user clicks Allow.
• Opening DevTools on a tab the agent controls detaches the agent from that tab.
• Google Chrome branded builds no longer accept --load-extension; the integration tests use Chrome for Testing, and real use loads the extension unpacked through chrome://extensions.

Development

npm install
npm run typecheck
npm test                  # unit tests (WebSocket framing, keyboard layout)
npm run test:integration  # launches Chrome for Testing with the extension; ~2 min

Integration tests need a Chrome for Testing or Chromium binary. They look in Playwright's cache (npx playwright install chromium) or use PI_CHROME_TEST_BINARY. Set PI_CHROME_TEST_HEADLESS=1 to run headless.

Layout:

chrome-extension/         the companion (load this folder unpacked)
extensions/chrome/
  index.ts                pi entry point
  bridge/                 WebSocket server, wire protocol, Chrome launch
  cdp/                    per-tab CDP session, console/network logs, snapshot, keyboard, routes
  tools/                  chrome_* tools
  commands.ts             /chrome
skills/chrome-connector/  agent skill
test/unit, test/integration

License

MIT