OpenProgram · Web Use

Web Use: control only built-in browser Pages

An agent can call web_use, or let GUI Agent Harness execute continuously through the same interface. The runtime binds an exact Page and pins one of three browser backends for execution.

TARGET NAME: WEB_USE · PUBLIC NAME: WEB_USE · BROWSER-ONLY RUNTIME IMPLEMENTED

Names and boundaries

CapabilityResponsibilitiesCurrent status
Web UseBuilt-in browser BrowserContext, Page, tab, popup, split views, cross-window discovery, and DOM/ARIA/viewport operations.Current capability; its public name is web_use
Computer UseFuture desktop applications, full-screen or application-window screenshots, and OS-level mouse/keyboard input; Web Use can be called for webpage interaction.Reserved name; not currently implemented, with no pre-created configuration or background processes

System structure

Layered relationship between the agent, Web Use, and three optional browser backendsAn agent can call web_use directly or call the same tool through GUI Agent Harness. The runtime pins Playwright MCP, Chrome DevTools MCP, or the Open Claude in Chrome-compatible backend for the current WebSession. The selected backend operates the bound exact Page. AGENT / CONTROL AI Any AgentChoose direct or harness CALLERSDirect Tool · GUI Harness DIRECTHARNESSMCP web_use OpenProgram MCP RUNTIMEBind once. Execute exactly. Page binding•permission•verification Pin the backend when WebSession is created EXECUTION / CHOOSE ONE ONE BACKEND PER WEBSESSION DOM Playwright MCPDOM · ARIA · element refsDefault: structured page operations CDP Chrome DevTools MCPCDP · snapshot · diagnosticsConnect to the runtime through an MCP adapter VIS Open Claude in Chrometool contract · visual actionsOpenProgram Electron / CDP adapter BOUND PAGE exact PageBOUND Same Page · Split in any direction · No window activation required

The three backends are alternative execution paths. The runtime pins one when creating the WebSession and confines every action to the same Page.

Representing four webpages in one window

Three top-level tab entries contain four controllable PagesIn the same BrowserContext and OpenProgram window, the first two top-level tab entries each contain one Page. The third is a split group containing left and right Pages. list_pages returns both the tab_entries layout and pages control targets; the agent ultimately creates a WebSession using only one Page token. BROWSERCONTEXT ctx_1 · WINDOW win_1 Google · p1 GitHub · p2 Bilibili · p3YouTube · p4 tab_entry_1 · single · backgroundtab_entry_2 · single · backgroundtab_entry_3 · split · active PANE LEFTPANE RIGHT · FOCUSED p3p4 BilibiliYouTube visible = truevisible = true · focused = true list_pages GROUP-AWARE INVENTORY · IMPLEMENTED tab_entries3 UI entries singlep1 singlep2 splitp3 + p4 pages4 exact targets p1background p2background p3split / left p4split / right p4 token → exact WebSession The top-level tab structure describes the window; Page tokens support observation and operation. Split views do not introduce a separate set of agent commands.

Currently, list_pages returns both tab_entries + pages: the former describes single/split layout and pane ordering; the latter provides independently controllable exact Page tokens. Cross-window targets retain the same structure, adding an outer windows[], while each Page retains its original window_id. For the full fields, see Multi-Page data model.

Claude Code visual coordinate-click path

Claude Code browser extension visual coordinate-click flowThe visual path in the locally installed Claude Code 2.1.227 and Chrome extension 1.0.85 first captures a screenshot, then sends model-selected coordinates to the target Tab through MCP, Native Messaging, and the Chrome extension. The extension draws a visual cursor and performs the actual click using CDP mouse events. Claude Codeclaude-in-chromeMCP toolsNative MessaginghostChrome extensionservice workerVisual cursorcontent scriptTarget TabCDP ① computer(screenshot, tabId)② tool request③ JSON / stdio④ Page.captureScreenshot⑤ Return PNG + viewport metadata through the same path⑥ computer(left_click, [x,y], tabId)⑦ click request⑧ Validate tab, permissions, and coordinate mapping⑨ UPDATE_PHANTOM_CURSOR⑩ Input.dispatchMouseEvent: moved → pressed → released Pause here and request user authorization when needed The on-page cursor is only a visual cue. CDP sends the actual click to the specified Tab without moving the system mouse.

This diagram is based on inspection of the locally installed Claude Code 2.1.227 and Chrome extension 1.0.85 packages. It describes only the visual-coordinate path, not an officially stable internal interface.computer also supports clicking through page refs; that path does not require a screenshot first.

Chat rendering stability

When a live normal assistant reply or its corresponding runtime card exists, Web Use tree updates must not trigger transcript reload. A standalone run without a persisted runtime card may perform one initial hydration. This preserves existing historical message element identity and prevents streaming replies, scroll positions, and message-action state from being reset.

Extent of reuse

ConclusionComponentOpenProgram handling
Use directlyMicrosoft Playwright MCP, Playwright, MCP SDKOpen-source dependency; integrate a pinned version.
Call the interface directlyCDP, Electron webContents.debugger, Chrome debugger / Native MessagingNo Claude source copy is needed. The built-in browser uses Electron/CDP directly, without installing a separate Chrome extension.
Optional backendChrome DevTools MCPConnect through an MCP adapter to the shared runtime and bind an exact Page.
Optional backendOpen Claude in Chrome-compatibleUse its tool contract and visual interaction model; the OpenProgram Electron/CDP adapter binds an exact Page.
OpenProgram layerExact Page adapter, shared permissions, verification, and backend selectorShared by all three backends; only one backend may execute actions within a WebSession.

Task results and observation evidence

Successful browser task results retain the bounded, password-redacted observation associated with the verified frame, alongside the assertion and exact Page identity. The observation is read-only context for the caller, not authority to reuse a closed session or proof that every page field was verified. A caller can report observed fields without invoking another browser task merely to recover data discarded by the result adapter. Failed, stale, cancelled or privacy-rejected completion emits no successful observation.

The single-Page task executor receives an already validated binding. Its planner prompt distinguishes an unavailable inventory interface from an empty inventory and does not request discovery or switching when those tools are not exposed. Runtime owns Page identity and permissions; the model chooses actions on the supplied observation.

Acceptance uses the installed App and an existing task: select the authorized file, observe and verify the website receipt, then use GUI Agent delegation to return the requested page facts and verification evidence. Selecting a file alone is insufficient for website acceptance. Zero mutation steps is valid for a successful read-only verification. Network disconnection or an App restart is reported separately from an invalid target or unsupported action.

When no Page is bound, browser tasks use the same registered Page inventory and command driver as web_use. They do not fall back to a broadcast asking whichever window is visible. An existing explicit Page binding keeps its native controller; an explicitly disabled surface remains denied. Target selection and switching use owner-scoped inventory capabilities.

Transient binding failures and external file-input references

A desktop acknowledgement timeout fails the current command without closing its Page binding or WebSession. It clears cached observation authority and requires a successful observe before another act or verify. Actual disconnect, expiry, ownership or target revision changes still revoke access. No mutation is replayed automatically.

MCP references and controller references are separate namespaces even when both use e-prefixed numbers. Each successful observation exposes native file inputs in elements with opaque file_ references, preserving labels, disabled state and selected filenames. These references are session-local and observation-local, and only support upload and file_selected verification. The adapter maps them to the exact already-observed native handle; it never resolves inputs by label, index or an MCP reference. Do not click Choose File before upload. Both operations use the same guarded controller implementation on its owner thread. File permissions, size limits, freshness, cancellation and uncertain-selection protection remain authoritative. Selecting a file does not verify server acceptance.

Implementation status: component tests cover both MCP adapters for namespace collisions, reobservation revocation, cross-session references, stale references and permission rejection. Real Chromium integration tests cover file selection and independent verification without personal documents or live registration websites.