@jellyfish-p/codex-computer-use-mcp0.6.2

Codex Computer Use MCP

Expose OpenAI's official signed Computer Use tools for macOS and Windows directly to Pi and MCP clients. The calling agent chooses each method and its arguments. This package runs no nested model and generates no action plan.

OpenAI does not produce or endorse this independent project. It relies on an experimental app-server API and installed ChatGPT components that may change.

This fork uses the npm name @jellyfish-p/codex-computer-use-mcp. The unscoped codex-computer-use-mcp package belongs to the upstream project. The original author and MIT license are retained; the CLI command remains codex-computer-use-mcp.

Requirements

• Node.js 22 or newer
• Pi 0.99.0 or newer when using the Pi integration

On macOS:

• an unlocked user session
• the official ChatGPT macOS app, by default at /Applications/ChatGPT.app
• the Computer Use component installed by ChatGPT, by default under ~/.codex/computer-use/

macOS Screen Recording, Accessibility and TCC controls still apply.

On Windows:

• an interactive desktop session
• the official ChatGPT (Codex) Windows app with Computer Use enabled, which installs the @oai/sky runtime under %LOCALAPPDATA%\OpenAI\Codex\runtimes\cua_node\

Windows Computer Use asks for approval the first time an app is used. The adapter forwards that official request to Pi or the MCP client; it adds no prompts of its own.

Pi

Install the stable release of the scoped package:

pi install npm:@jellyfish-p/codex-computer-use-mcp@latest

To opt into Windows previews instead, use pi install npm:@jellyfish-p/codex-computer-use-mcp@windows-preview. Source checkouts and downloaded archives also work without an npm publication. When switching from the old unscoped or local package, remove that old Pi package registration first to avoid registering the same tools twice.

The extension registers the official Computer Use methods of the host platform as Pi tools named computer_use_<method>. Codemode is optional: Pi can declare these tools directly to the model when you activate them. The extension preserves Pi's codemode exposure for optional composition, but does not require a script executor or change the active tool set itself.

Included skill

The package includes an original computer-use skill, registered through pi.skills and shipped with Windows and macOS references. Pi advertises it for desktop interaction; use /skill:computer-use to load it explicitly. It covers target selection, input focus, observe–act–verify, stale references and recovery using this adapter's tools, not Codex's node_repl.

Active Computer Use tools also contribute short prompt guidelines, so direct-tool sessions receive the basic workflow without reading codemode namespace instructions. The skill and guidelines do not activate tools, add permission gates or replace official approvals. Installing or reloading the package discovers the skill unless you have disabled it in Pi's resource settings.

Direct calls without codemode

On Windows, add these entries to defaultTools in ~/.pi/agent/settings.json (merge with any existing entries):

{
  "defaultTools": [
    "+computer_use_list_apps",
    "+computer_use_list_windows",
    "+computer_use_get_window",
    "+computer_use_launch_app",
    "+computer_use_get_window_state",
    "+computer_use_click",
    "+computer_use_press_key",
    "+computer_use_type_text",
    "+computer_use_scroll",
    "+computer_use_set_value",
    "+computer_use_drag",
    "+computer_use_perform_secondary_action",
    "+computer_use_activate_window"
  ]
}

The + entries keep your existing default tools. Restart Pi or use /reload, then ask it to call computer_use_list_apps or computer_use_list_windows directly. Screenshots and text are returned in the tool result; no script or image() call is needed. On macOS, activate its ten computer_use_<method> tools instead; /computer-use-status lists the platform methods.

Optional script composition

If you prefer scripts, enable Pi's built-in codemode with pi --tools +codemode. The following examples are optional, not required for direct calls.

On macOS these are the ten app-server methods, such as computer_use_get_app_state and computer_use_click:

const state = await tools.computer_use_get_app_state({ app: "TextEdit" });
text(state.text);
if (state.screenshot) image(state.screenshot);
// Inspect this result before choosing an element or coordinate for the next call.

When neither a Computer Use tool nor codemode is active, the extension shows one notice explaining direct activation. It never changes other tools or your saved configuration.

computer_use_get_app_state resolves to { app, text, screenshot }, where screenshot is an image block for image() or null. It may return an accessibility-tree diff after the first inspection; pass disableDiff: true to request a fresh full tree. computer_use_list_apps resolves to the official text inventory produced by the app-server transport; unlike native @oai/sky, that transport does not provide the structured app array. Other methods resolve to their official text output. A failed call rejects with the official error text. Calls run one at a time, and calls after get_app_state reuse the signed session until the agent settles. The namespace instructions, available from describeNamespace("computer_use"), summarise this workflow.

On Windows the official surface is the @oai/sky window API, so the tools are list_apps, list_windows, get_window, launch_app, get_window_state, click, press_key, type_text, scroll, set_value, drag, perform_secondary_action and activate_window. Actions target a window object taken from list_apps or list_windows:

const { apps } = await tools.computer_use_list_apps({});
text(apps);
// Inspect the inventory and choose the task-specific window; do not assume windows[0].

Pass the chosen returned window unchanged to tools.computer_use_get_window_state({ window, include_text: true }), emit its tree with text() and its images with image(), and inspect the output before choosing a state-dependent action. Verify input focus before typing and refresh after acting. In codemode, use store() / load() for small JSON references across scripts; these are not a persistent Node REPL and saved references can become stale.

computer_use_list_apps resolves to { apps }, computer_use_list_windows to { windows }, computer_use_get_window to { window } and computer_use_get_window_state to { window, accessibility, screenshots }. These adapt the native arrays/window and screenshot URLs into tool-friendly values; each screenshot carries an image block for image(). Other methods resolve to ok. The first use of an app raises the official approval request, which Pi shows as a selection between a session-only and a permanent approval. Any call opens the retained helper session, which is closed when the agent settles.

Use /computer-use-status to inspect the installed component and transport status.

To run a source checkout:

npm ci
npm run build
pi -ne -e /absolute/path/to/codex-computer-use-mcp

Loading the package directory includes its skill. Loading only integrations/pi/index.ts supplies the extension and short tool guidelines, but does not discover the package skill automatically.

MCP

Running the package binary starts a stdio MCP server with the same platform methods and computer_use_status:

npx @jellyfish-p/codex-computer-use-mcp@latest

Example Pi MCP configuration:

{
  "mcpServers": {
    "computer-use": {
      "command": "npx",
      "args": ["@jellyfish-p/codex-computer-use-mcp@latest"],
      "lifecycle": "lazy",
      "requestTimeoutMs": 180000,
      "directTools": false
    }
  }
}

Behaviour

The adapter has one mode. Pi and MCP both expose the official methods of the host platform as typed tools; Pi gives them codemode exposure. Both are available without wrapper permission prompts. It adds no app allowlist, action gate, intent classifier, selector rewrite or focus policy. App-server uses Codex Full access. The adapter forwards any elicitation that the official host still emits.

On macOS, production calls require verified OpenAI-signed app-server and Computer Use binaries with Team ID 2DC432GLL2. The adapter uses an isolated, credential-free app-server context. It rejects any model-turn activity. Calls after get_app_state reuse the signed session, preserving element identifiers and official app state.

On Windows tools run through the @oai/sky JavaScript client rather than the macOS app-server MCP transport. That client runs the signed codex-computer-use.exe helper over stdio; the helper uses the verified Codex binary for its own authentication, without starting a model turn. The adapter loads that official client from the installed runtime after verifying the helper's Authenticode signature (status Valid, signer OpenAI OpCo, LLC) and its canonical location. The official client receives app-approval requests from the helper and expects a node_repl host to answer them; the adapter provides only that host surface and forwards each request as an elicitation. The host owns approval memory, as in Codex: a session approval is remembered for the lifetime of the Pi or MCP server process, and an "always" approval is written to windows-approvals.json in the state directory (mode 0600). Both map the call's target — the window's app field or launch_app's app — to the helper's own app id, which is what later requests are marked with; delete the file to forget permanent approvals. Any call opens the retained helper session; closing it ends the helper process.

While a helper turn is open, Computer Use shows its "ChatGPT is using your computer" overlay and holds the mouse and keyboard; element indexes from get_window_state are valid only within that turn and can become stale sooner when the UI changes. Pi owns the task lifecycle: it keeps the turn open across model thinking and pauses between calls, then closes the session when the agent settles (including cancellation) or the Pi session shuts down. Request timeouts and failures still clean up the helper. Long-running Pi tasks can therefore hold the desktop longer; pressing Esc cancels the current turn at any time.

MCP clients do not provide the same task-ended signal, so they retain an idle-turn fallback: CODEX_COMPUTER_USE_TURN_IDLE_MS (default 15000) ends a turn after that many milliseconds without calls. This environment setting does not shorten Pi-owned turns. Ending an idle turn releases the desktop without closing the helper; a later call starts a new turn and must inspect the window again before acting on an element. The MCP server separately closes retained sessions after 120 seconds of inactivity.

On macOS, closing a retained session sends a matching native turn-ended notification through the signed app-server before terminating its private processes. This lets the shared Computer Use service remove that session's software cursor without terminating the service. Cancellation and tool timeouts briefly wait for an outstanding native reply before notifying, so a late reply cannot recreate the cursor after cleanup. A stalled request, failed notification, or lost transport can prevent native cleanup. The call still returns its result, because the action already ran, and private process teardown still runs. brokerCleanupVerified describes private process cleanup, not visual confirmation that a cursor disappeared.

Audit records contain bounded metadata. They exclude arguments, app content, screenshots, prompts and credentials. MCP and CLI state defaults to ~/.direct-computer-use. Pi state defaults to direct-computer-use under the Pi agent directory. Set CODEX_COMPUTER_USE_HOME to override either default.

Paths

By default the adapter uses the Codex app-server at /Applications/ChatGPT.app/Contents/Resources/codex-cli/CodexCLI.app/Contents/MacOS/codex, falling back to /Applications/ChatGPT.app/Contents/Resources/codex on older ChatGPT builds. It uses the Computer Use app at ~/.codex/computer-use/Codex Computer Use.app, falling back to the copy bundled in older ChatGPT builds.

When ChatGPT is installed elsewhere or moves these files, set either path in config.json in the state directory:

{
  "codexPath": "/Applications/ChatGPT.app/Contents/Resources/codex-cli/CodexCLI.app/Contents/MacOS/codex",
  "computerUseAppPath": "/Users/me/.codex/computer-use/Codex Computer Use.app"
}

On Windows the adapter uses the newest @oai/sky package under %LOCALAPPDATA%\OpenAI\Codex\runtimes\cua_node\<runtime>\bin\node_modules\@oai\sky and the newest signed codex.exe under %LOCALAPPDATA%\OpenAI\Codex\bin\<runtime>. Set computerUseAppPath to another verified @oai/sky package or codexPath to another verified Codex binary.

The adapter reads this file when it starts each Computer Use session, so changes apply without restarting Pi or the MCP server. Configured binaries must pass the same OpenAI signature checks. When a component is missing, the error lists the paths checked and the config file to edit. /computer-use-status and codex-computer-use-mcp --status show the platform, the config path and the current error.

Pi limits returned text to 50KB or 2,000 lines. When text exceeds that limit, the complete text is written to a mode-0600 file in a private directory under /tmp. Images are returned directly and are never spilled to disk.

Development

npm ci
npm run check
npm run check:pi
npm test
npm run build

npm test includes one live call through the signed component of the host platform (list_apps); the other platform's transport tests are skipped. npm run test:ci skips those live calls, so it can run without ChatGPT/Codex installed. SDK tests also verify that both platforms' tools can be activated and called with no codemode extension registered.

The development-only Pi dependency pins an older Undici. The scoped overrides entry upgrades it to a patched version; it does not modify the user's installed Pi.

CI and releases

The CI and release workflow (.github/workflows/package.yml) checks, tests and packs on Windows with Node.js 24 for pushes to main, pull requests and manual runs. Those runs only produce an artifact; they do not publish.

Only version-tag pushes publish. The tag must match package.json, and only these version formats are accepted:

Tag	npm dist-tag	GitHub release
vX.Y.Z-windows.N	windows-preview	Prerelease
vX.Y.Z	latest	Stable release

The npm job downloads the tested archive, verifies its package name, version, commit, SHA-256 and SHA-512 integrity, then publishes with OIDC and provenance. It does not rebuild or require a desktop runtime. The GitHub Release receives that same archive, checksum and build manifest after the npm job succeeds. A rerun skips an existing npm version only when its integrity matches; it neither overwrites a published version nor moves dist-tags backwards.

Trusted Publishing must authorize jellyfish-p/codex-computer-use-mcp, workflow package.yml, for npm publish. No NPM_TOKEN secret is needed. The first package publication must be bootstrapped by a maintainer before npm allows this trust relationship. The initial manual publication has no GitHub OIDC provenance; subsequent new versions published by CI do.

To release, update the package and lockfile versions, commit to main, and push a matching version tag after CI succeeds. Preview versions stay off latest. Use the stable vX.Y.Z format only when intentionally releasing a stable version.

To install a downloaded preview in Pi without building it locally:

npm install --prefix ./computer-use-preview --omit=dev --legacy-peer-deps ./jellyfish-p-codex-computer-use-mcp-0.6.2-windows.4.tgz
pi install ./computer-use-preview/node_modules/@jellyfish-p/codex-computer-use-mcp

Keep that directory: Pi loads local packages in place. Enable direct calls with the defaultTools entries above, restart Pi (or use /reload), then run /computer-use-status. You can start Pi normally; --tools +codemode is not required.

npm publication preflight

The package targets the public npm registry under @jellyfish-p. Stable releases use latest; Windows previews use windows-preview. Check a stable release's publish payload without uploading:

npm publish --dry-run --access public --tag latest

For a preview version, use --tag windows-preview instead.

This runs prepublishOnly, including the full local test suite and its read-only official helper check. A successful dry run validates the package, not account ownership, 2FA or Trusted Publishing permissions. The default tag is deliberately windows-preview; stable CI releases explicitly select latest.

For the initial bootstrap, a maintainer can publish the already-checked CI archive with npm publish ./<archive>.tgz --access public --tag windows-preview, completing npm's authentication/2FA if requested. Then configure the exact trusted publisher with npm 11.15 or newer:

npm trust github @jellyfish-p/codex-computer-use-mcp --repo jellyfish-p/codex-computer-use-mcp --file package.yml --allow-publish
npm trust list @jellyfish-p/codex-computer-use-mcp

Account-level 2FA is required to configure trust. Linking a GitHub account to npm is not the same as configuring this OIDC publisher. Do not replace an existing publisher without reviewing it first.

License

MIT