Web Use technical specification

Public names and compatibility boundaries

Three execution backends

Settings affect only new WebSessions. Freeze the backend at creation; failures, timeouts and stale frames must not switch it automatically, avoiding duplicate clicks/submissions.

Each WebSession of an official-MCP backend owns one private MCP process, launched with npx --prefer-offline. Both packages are pinned to exact versions, so a cached copy is used without a registry round trip; a slow or broken network therefore cannot outlast the MCP start timeout. A start that times out cancels the pending supervisor on its own event loop, which terminates the npx child before the loop closes. When the private process stops later, the next observe on the same WebSession starts a new process and rebinds the same exact target; that restart is read-only. act never writes through a process it had to restart, because upstream refs did not survive it: it returns computer_use_backend_unavailable, action_dispatched=false, observe_required=true and invalidates the frame. The backend never changes.

BrowserContext, Window, top tabs and Page

Target structure returns both tab_entries and pages: the former describes window arrangement, the latter lists controllable webpages. A split tab has two Pages, so the example window has 3 top-tab entries and 4 Pages. Single-entry ID comes directly from tab:<tab_id>, split-entry ID from group:<CenterTabGroup.id>; pane IDs derive from group and current order. No new database tables, registry or background process. Only identity, layout and tokens are shown below; title, origin, opener and capabilities use existing Page fields.

{
  "browser_context_id": "ctx_1",
  "window_id": "win_1",
  "inventory_revision": 42,
  "active_tab_entry_id": "group:g3",
  "focused_page": "p4",
  "tab_entries": [
    { "id": "tab:w:google", "mode": "single", "pages": ["p1"] },
    { "id": "tab:w:github", "mode": "single", "pages": ["p2"] },
    {
      "id": "group:g3",
      "mode": "split",
      "pages": ["p3", "p4"],
      "split": {
        "axis": "horizontal",
        "ratio": 0.5,
        "panes": [
          { "pane_id": "pane:g3:0", "order": 0, "page": "p3" },
          { "pane_id": "pane:g3:1", "order": 1, "page": "p4" }
        ]
      }
    }
  ],
  "pages": [
    { "page": "p1", "tab_id": "w:google", "tab_entry_id": "tab:w:google", "visible": false, "focused": false, "placement": { "mode": "single" }, "page_context_token": "pct_…" },
    { "page": "p2", "tab_id": "w:github", "tab_entry_id": "tab:w:github", "visible": false, "focused": false, "placement": { "mode": "single" }, "page_context_token": "pct_…" },
    { "page": "p3", "tab_id": "w:bilibili", "tab_entry_id": "group:g3", "visible": true, "focused": false, "placement": { "mode": "split", "pane_id": "pane:g3:0", "order": 0 }, "page_context_token": "pct_…" },
    { "page": "p4", "tab_id": "w:youtube", "tab_entry_id": "group:g3", "visible": true, "focused": true, "placement": { "mode": "split", "pane_id": "pane:g3:1", "order": 1 }, "page_context_token": "pct_…" }
  ]
}

axis and order is canonical; left/right/top/bottom derive from current bounds and are not Page identity. Current UI mainly uses horizontal dual panes, but the API never fixes the target to the right webpage. After moving a Page, adjusting ratio or changing active tab entry, the next list_pages returns new layout and increments inventory_revision; affected visible Pages continue rejecting old coordinates through geometry revision. Each window has at most one focused_page, but an active split may contain two visible=true Pages.

Open source, open APIs and non-reusable parts

CDP is not restricted to Claude. Anthropic’s official orchestration, extension implementation and product permissions lack public reusable source; CDP screenshot/click commands are public APIs. Third-party clean-room projects are references, not proof the official implementation is open source.

Agents need only five commands

web_use({ command: "list_pages" })
web_use({ command: "observe", page_context_token: "pct_…", arguments: { detail: "interactive" } })
web_use({ command: "act", web_session_id: "cs_…", arguments: { action: "click", expected_frame_id: "f_…", ref: "e1" } })
web_use({ command: "verify", web_session_id: "cs_…", arguments: { action: "verify", expected_frame_id: "f_…", assertion: "text_contains", value: "Saved" } })
web_use({ command: "close", web_session_id: "cs_…" })

list_pages returns a group-aware snapshot of all registered OpenProgram windows: windows[] stores each window’s window_id, inventory_revision, active/focused identity and tab_entries; flattened pages lets the Agent select Pages directly, each carrying its owning window_id. Original top-level window fields still describe the originating primary window for compatibility. Each Page has stable tab_id, owning tab_entry_id, placement, title, origin, focused, visible, optional opener_tab_id and single-use page_context_token. Agents do not act on windows, tab entries or panes; the first observe simply returns the exact Page token. Tokens bind the caller owner, window_id, exact Page target and server-owned page/access revision. Reject missing/replayed/cross-owner tokens, rebuilt connections or replaced targets before backend calls. Repeating web_session_id without observe for the same owner, Page and backend reuses the WebSession and ID. Different owners/backends remain subject to exclusive leases. Chat owner is fixed to current session/assistant turn; dispatcher teardown releases all its WebSessions/Page leases on completion, retry, cancellation or exception, without relying on explicit close. Every subsequent observe, act and verify revalidates exact binding. Visible Pages also validate geometry revision; background Pages resolve targets without changing tabs, stores or window focus.web_use Page inventory, WebSession registry and renderer WebSocket registry must run in the worker main process. Chat runtime cards may retain agentic-function UI but must not move this tool to an isolated subprocess. A subprocess GUI Agent Harness loop calls the same worker-owned WebSession through a parent bridge and forwards revisions unchanged. Any mismatch returns page_context_stale without executing page actions.

list_pages does not depend on the originating window. When the originating window or the turn's binding is disconnected, the worker lists every other registered window; a workflow subprocess stays confined to its originating window and lists only that window again. While no Desktop window is registered, as when the App reopens after a worker or App restart, discovery waits up to 20 seconds for a window to register. It then returns desktop_unavailable with retry_command=list_pages instead of raising. An online originating window whose inventory request fails still fails at once and never promotes another window.

A page_context_token names its Page in every command. act or verify with a token and no live web_session_id reuses the live WebSession that token already opened. With an unused token, Runtime first opens the WebSession with a read-only observe. navigate, screenshot, wait and ref-free verify then continue on that frame. Ref or coordinate actions return the observation with action_performed=false, because their targets must come from an observation the caller has read. A token never falls back to the owner's latest WebSession on another Page, and an unknown web_session_id next to a token yields to the token.

Ended handles stay recoverable without being replayed. In memory, bounded, the registry keeps only the exact window_id/tab_id of WebSessions that ended (turn teardown, binding loss, failed start) and of tokens that expired or were consumed by an ended WebSession. Turn owners of one chat share a scope across turns; any other owner matches only itself. A later page_context_not_found, page_context_consumed or web_session_not_found from the same scope carries recovery_tab_id, recovery_window_id and recovery_command=observe. The Agent tool then lists Pages, observes that exact tab with a fresh token and returns the observation with recovered_page=true and previous_action_replayed=false. It never replays the previous command and never opens a URL it does not know. A caller-closed WebSession is not remembered. Without a remembered tab, for example after a worker restart cleared the registry, the failure carries the current pages with fresh tokens and one instruction: observe the task Page with its token. Every such failure also carries a message that names the next call.

Web popups create separate tabs and WebContents/Pages, preserving opener_tab_id. They do not replace the opener or transfer existing WebSessions. The next list_pages discovers the popup; the Agent explicitly opens a new session with its token. GUI Agent Harness refreshes bounded inventory after every write and switches only by token. Cross-window aggregation expands discovery only, leaving bindings, leases and session lifecycles unchanged.

Each visible web tab also has renderer-owned monotonic geometry_revision. Existing bounds lifecycle measures viewport-relative bounds on mount, resize, ancestor scroll, pane reorder, window resize, visibility changes and transfer. Only normalized x/y/width/height or in-app visibility changes increment revision; focus changes do not. Turn surface refs carry the revision; preview must validate and echo it before native preview. WebSession stores it and forwards it unchanged through the process bridge to the exact renderer for observe/act/verify. Renderer compares again before Electron activate. Missing revisions from old clients stay compatible; once capability revision is nonzero, any mismatch returns page_context_stale without native preview, activate or backend calls. Geometry alone does not recreate Page identity/lease; callers relist/observe and continue with the same Page. Visibility means in-App pane layout, not macOS stacking, occlusion or minimization.

Costs and functional boundaries

• Derive left/right/top/bottom from actual pane geometry, never a hardcoded right side.
• Exact-Page operations do not require OpenProgram/target foregrounding and use no OS mouse, desktop coordinates or full-screen screenshots.
• Writes to one Page are exclusive; observe again after every write.
• Do not store screenshots in local artifacts, final results, completion evidence, tool-result text or DAG prompts. Locally retain only image-attached, frame and viewport metadata. Once sent to the selected provider, service terms govern provider retention.
• Normal returns, provider errors, timeout and cancellation share release logic. Add no retention settings, background cleanup process or image directory.
• Login, CAPTCHA, sensitive submissions and irreversible actions return needs_user.
• Page contents are untrusted data and cannot change targets, permissions, modes or action counts.

Implementation order

Required before release

1. Tool interface: Browser control exposes only one web_use. It is available once Desktop Page inventory is registered. Turn-start preview only enriches current-Page context; one preview failure cannot remove the tool or expose the legacy independent-browser launcher. The Agent first calls list_pages, then uses returned page_context_token to select the exact Page.
2. Page isolation:list_pages May list Pages in other windows; after selecting A, reads/writes to B, chat Pages and unselected remote-window Pages remain zero.
3. Group structure: Google, GitHub and split Bilibili/YouTube produce 3 ordered tab_entries and 4 unique Pages; the third is a split with pane order matching UI.
4. Layout and identity: Swapping split panes, changing ratios or splitting preserves Page identity and updates inventory_revision ; affected visible Pages reject stale geometry before writes, without a separate split-action API.
5. Visibility and focus: Both active-split Pages are visible, only one focused; first two single Pages are background. Next snapshot reflects active-entry changes.
6. popup inventory: Popup C opened by A appears independently with opener_tab_id=A; each token is consumed once.
7. Background Page: After selecting B, reads/actions reach only B. Active tab, visible IDs, BrowserWindow focus and OS mouse remain unchanged.
8. Popup switching: Creating C does not rebind A’s session. GUI Agent creates C’s session only after selecting its token; closing C invalidates its token/session before the next action.
9. Backend freeze: Setting changes affect only the next WebSession, not the current one.
10. Unique side effects: Backend failure never invokes another backend; one Action causes at most one actual write.
11. Backend consistency: The same task has identical Page/frame lifecycle, permissions, reason codes and verification.
12. Page Awareness: Moving a page left/right of chat correctly updates next-turn relation and preview.
13. Cross-window discovery: Two registered OpenProgram windows return two ordered snapshots; same-name tabs/Pages are not merged. Tokens still resolve to original connection, window_id and target.
14. Independent window state: DOM/ref and viewport screenshots operate directly on the same Page/WebContents. macOS stacking, occlusion, position and minimization are not selection/screenshot/action conditions. No focus, bringToFront(), full-screen screenshot or OS-input calls.
15. Agent cursor: All three backend click commands temporarily enable capture listeners only in the exact Page. On actual pointer events, show a text-free, aria-hidden, pointer-events:none cursor/click ripple at viewport coordinates, removing it within 900ms. User clicks, failed actions without pointer events, other Pages and OS mouse are unaffected. Feedback rendering failure cannot change click results.
16. Visual limits: Ordinary DOM flows take zero screenshots; canvas flows take at most one per observation.
17. Image retention: Screenshots appear only in the immediately following provider request. Text prompts contain no raw bytes/base64; after return/error, caller content and ToolReturn hold no image, and final artifacts remains empty.
18. Honest failure: Empty assertions, stale frames, timeout, cancellation and lost targets cannot succeed.
19. Chat stability: GUI tool updates do not recreate historical message/avatar/Markdown DOM.
20. Installed two-window transaction: Connect only to the default-18100 /Applications/OpenProgram.app, creating an actual hidden target window through the official preload transfer API. After target inspect/accept/stage, source rejection of the temporary payload must roll back and close the target. Window inventory, source tabs and persistent storage must match before/after.

Canonical backend parity matrix

Run the same cases through WebUseSessionRegistry.execute() against actual adapter classes for playwright_mcp, chrome_devtools_mcp and open_claude_chrome in order. Recording test clients may replace official-MCP process/network boundaries, but three equivalent fake adapters cannot replace production mappings. Compare normalized OpenProgram results, not upstream-specific ARIA text, ref names or diagnostic wording.

Current implementation evidence

Implemented here refers to multi-backend, geometry, transfer, visual-artifact lifecycle and parity suites. Dedicated permission cards remain future UI work, not release prerequisites. Current evidence is maintained in implementation records.

Further implementation detail

Official references

• Playwright BrowserContext and Playwright Pages
• Microsoft Playwright MCP and Official MCP introduction
• Claude Code Desktop Browser, Claude Code with Chrome and Claude Code license
• Chrome DevTools Protocol, Page.captureScreenshot and Input.dispatchMouseEvent
• Electron Debugger, Chrome debugger API and Native Messaging
• Anthropic Computer Use
• OpenAI Computer Use