omp packages

@lennix1337/pi-agent-desktop0.1.5

Computer use for the Pi coding agent: see and control native Windows, macOS and Linux apps (click, type, read, screenshot) through the OS accessibility tree, powered by vercel-labs/agent-desktop.

GitHub Actions Published 13 hours ago MIT ~737 downloads/mo

Untested on omp

This is a pi package. No one has reported whether it works on omp yet. If you try it, add your result below.

Did it work for you?

Result

One report per person. A new report replaces your old one.

Reports 0

No reports yet.

Readme

pi-agent-desktop — computer use for Pi

npm check license

Give the Pi coding agent eyes and hands on your desktop. It reads any native app on Windows, macOS and Linux through the operating system's accessibility tree, then clicks, types, fills fields, reads values and takes screenshots. Powered by vercel-labs/agent-desktop.

pi install npm:@lennix1337/pi-agent-desktop

No global install and no Python: the native binary for your platform ships inside the package.

What you can ask Pi

  • "Open Notepad, write the release notes and save them to my Desktop."
  • "Read the error text in that installer window and tell me what it wants."
  • "Fill this desktop form with the data from customers.csv."
  • "Take a screenshot of the Settings window and check the toggle is on."

How it works

Pi gets one desktop tool and a pi-agent-desktop skill that teaches the loop:

  1. list-apps / list-windows find the target.
  2. snapshot returns the accessibility tree as compact JSON with @ref ids (no pixels, cheap in tokens).
  3. Act on a ref: click, set-value, type, press, launch, close-app...
  4. Verify with get or a fresh snapshot; screenshot attaches a PNG when the tree is not enough.

Several steps go in one call as a cell (command: "exec", code): async JavaScript with win, snap, click, type, keys, press, at, wheel, shot and ad helpers, values kept between cells, the same allow-list and headed rule per step, a stop at the first failed step, and a screenshot of the window after any action. It is the desktop counterpart of browser_harness's exec.

Native apps snapshot in about 0.2 s. Electron/Chromium apps (Slack, VS Code) expose deep trees: snapshot them with --max-depth 3; the tool suggests it on timeout.

Safety

  • Semantic, headless delivery by default: no cursor movement, no focus stealing.
  • Physical input (mouse-*, drag, hover, double-click, key-down/up) is refused unless the call sets headed: true; --headed cannot be smuggled through args.
  • Commands run without a shell; batch, session, cursor-overlay and --debug are not exposed.
  • macOS: grant Accessibility (and Screen Recording for screenshots) to the terminal that runs Pi.

vs. other options

pi-agent-desktop Screenshot + vision models Browser tools
Native desktop apps yes yes no
Reads exact text/values yes (accessibility tree) OCR guess web only
Token cost per step small JSON large images small

Not to be confused with abcwyc/pi-agent-desktop, a desktop chat app for Pi. This package gives Pi a tool; it is not an app.

Develop

npm install
npm run verify # types + all offline tests (same entry point as .pi/checks.json)
npm run check   # types
npm test        # unit + Pi loader test (runs the real binary's `version`)
npm run e2e     # real desktop: opens Notepad, writes, reads back, screenshots, closes (Windows)

Agents can discover verify, module (CLI tests only), and mutations in .pi/checks.json. These checks do not drive desktop apps; the real Notepad journey is separate.

Override the binary with PI_AGENT_DESKTOP_BIN.

Release

npm run release bumps the patch version, tags it and pushes. The Release workflow then runs the checks and publishes to npm with provenance through npm trusted publishing (no token, no OTP), and creates the GitHub release.

MIT. agent-desktop is Apache-2.0, © Vercel.