omp-cua

Native macOS desktop control for OMP agents, backed by the cua-driver CuaDriver.app daemon through its SDK (@trycua/cua-driver 0.36.0). Input goes to the target window in the background, so the agent does not take over your mouse or focus.

This package was renamed from omp-desktop and is published on npm as @edlsh/omp-cua. The tools, the /desktop command, the desktop skill, and the exported createDesktopExtension are unchanged. A plugin linked under the old name must be linked again.

Tools

• desktop_observe (read): windows, apps, state (element tree with [N] indices), screenshot, desktop, permissions.
• desktop_act (exec): up to 20 batched steps (click, type, set_value, key, hotkey, scroll, drag, menu, launch, focus, resize, clipboard, wait), optional verify predicates, and a fresh state afterwards.
• desktop_driver (exec, discoverable): direct access to other cua-driver tools such as zoom, page, and browser_*. The session label is set by omp-cua; args.session is refused.
• /desktop: status only, never a toggle: driver version, permissions, this conversation's session, and JEV. Anything other than the subcommands below shows usage; it is never sent to the agent as a prompt.
• /desktop jev on|off: JEV targeting for this conversation, off by default. While on, desktop_act click/double_click/right_click/type/set_value/scroll steps accept target: "Save button in the toolbar" instead of index. The typesafe-ai/jev model picks the control from a fresh walk and acts only at confidence ≥ 0.5; otherwise the step fails with the top candidates. Controls whose label looks destructive (delete, send, buy, …) are never clicked by target; the agent must pass the index. Needs a Vercel AI Gateway key: AI_GATEWAY_API_KEY, or a jevApiKey resolver passed to createDesktopExtension by a local loader (see Setup), called on first use. If the gateway refuses Jev for billing or key reasons (HTTP 402, 401, or 403), the step fails with jev_unavailable and JEV turns itself off with a notice. Privacy: control labels and visible row text of the target window are sent to Vercel AI Gateway; the menu bar, secure fields, disabled controls, and typed text-field values are not.
• /desktop reset: ends this conversation's driver sessions and clears element indices.

While the extension is enabled, a one-line system-prompt nudge steers native-app work to these tools; there is no separate switch for it. Turn the whole extension on or off per profile in OMP's /extensions panel.

Efficiency defaults: trees are capped at 150 nodes with a 3 s walk budget, menu bars are folded, screenshots are opt-in, and only the two newest desktop screenshots stay in context. A walk cut at its node budget reports elements=N (partial) and how many nodes it did not reach. The extension maps [N] indices to opaque driver-issued element tokens. A stale [N] index, including one the driver measures against a newer screenshot-only snapshot, is retried once, and only when a fresh walk shows the same element at that index. Screenshot-only observations keep the last element tree, so its indices stay usable. A pixel click refused with screenshot_context_missing is retried once with the window's newest screenshot capture, and desktop_act returns a screenshot with its state after pixel steps. Sliders and other range controls show their value in the state (AXSlider (Volume) = "8.33"), so a drag can be checked. drag always uses foreground delivery, because macOS has no background drag. A hotkey step accepts index (or x,y) to focus that control first, so editing chords such as cmd+a land in the intended field. Desktop calls within one conversation run one at a time; a cancelled call stops before its next step, and /desktop reset or session shutdown waits for the call in progress.

State text keeps one row per element. AppKit custom actions, which the driver reports as unresolved selector dumps, show as their names ([actions=[read,delete]]). Multi-line values stay on their row as \n. Values longer than 300 characters and rows longer than 500 are cut with a count of the hidden characters; a query for that control returns them in full. A query searches only the nodes the walk reached, so when the default 150-node walk is cut before any match, the extension searches once more up to 1,500 nodes and says so; an explicit max_elements is never raised.

Screenshots larger than 1,000,000 base64 bytes are re-encoded as lossy JPEG with macOS's built-in sips (quality 85, then 70 or 55 only if needed) without changing pixel dimensions, so click coordinates stay valid. Smaller images are unchanged. If encoding fails or the image still exceeds the budget, the tool reports an error asking for a smaller max_image_dimension rather than forwarding oversized image data.

cua-driver ends sessions after 5 idle minutes (this conversation's label and the transport's implicit session behind windows/apps), and /desktop reset ends the label immediately; the next desktop call restarts the ended session and retries, so a paused conversation keeps working without restarting OMP. A session switch ends this conversation's sessions and session shutdown ends them and detaches the binding; the next desktop call revives them. The OMP process keeps one daemon connection throughout, shared by the main agent and subagents, because the daemon only lets the connection that started a session label restart it.

Setup

Requires CuaDriver.app 0.36 in /Applications (installer: /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"). The first desktop call starts the daemon (open -n -g -a CuaDriver --args serve) when it is not running, and fails with driver_unavailable rather than falling back to an in-process runtime. Accessibility and Screen Recording are granted to CuaDriver.app, not to the app that launches OMP: run cua-driver permissions grant once and approve both. Until then calls fail with permissions_pending. Check with desktop_observe action=permissions or /desktop.

The SDK's Node addon connects to the daemon's socket. Load the extension one of two ways, never both (duplicate tools):

• As a plugin: omp plugin install @edlsh/omp-cua from npm, or omp plugin link . from a checkout after bun install (add --profile <name> per profile). A plugin is turned on or off with omp plugin enable|disable @edlsh/omp-cua and reads the JEV key only from AI_GATEWAY_API_KEY.
• From each profile's extensions directory, with a small local loader that imports createDesktopExtension from a checkout and passes a jevApiKey resolver backed by your configured secrets manager. The extension then appears in /extensions, where it can be turned on or off per profile.

Driver sessions are labeled omp-<profile>-<session>.

The package bundles the desktop agent skill in skills/desktop. OMP loads it with the plugin; with a local loader, link that folder into the profile's skills directory.

Development

bun run check   # typecheck, including the OMP host contract
bun run test    # unit tests