Web Use implementation and verification records

This page records only implementation tasks, verification commands, and review conclusions for the Web Use target design, without changing its conceptual design. Task 1 delivers a low-cost baseline Browser Agent for the built-in browser.

TASK 1 · COMPLETE TASK 2 · COMPLETE TASK 3 · COMPLETE / APP E2E PASSED TASK 4–5 · COMPLETE / APP E2E PASSED TASK 6 · COMPLETE / PUBLIC API WEB_USE

Task 1: DOM-first Browser Agent baseline

Approved design: docs/reference/design/integrations/web-use.html.

Baseline: 717d4e17. Implementation starts in an isolated worktree and branch codex/browser-agent-20260814.

Production files: openprogram/programs/workflow/browser/__init__.py, openprogram/programs/_registry.py, openprogram/programs/__init__.py, openprogram/agent/authority.py.

Test files: tests/component/programs/browser/test_browser_agent.py.

Behavior contract

Public-entry RED

python -m pytest -q tests/component/programs/browser/test_browser_agent.py must first demonstrate these gaps:

Security, cancellation, error, and concurrency boundaries

Explicit exclusions

Desktop Computer Use, GUI Harness visual pipelines, new permission-card UI, permanent origin grants, tab leases, multi-window routing redesign, Windows, OS keychain/keyring, and browser-profile migration are outside Task 1.

Full gate manifest

Installed-build E2E verification

Test target: /Applications/OpenProgram.app 0.6.1. The test did not launch Electron from source. Desktop and the 18101 worker came from the application bundle; worker Python was Contents/Resources/runtime/python/cpython-3.12.10-macos-aarch64-none/bin/python3.12.

Isolation: use profile browser-agent-installed-20260814 and port 18101 to avoid affecting other concurrently running OpenProgram workers. Model configuration came from a local copy in that profile.

ScenarioExecution evidenceConclusion
DOM-first formSession local_38e090f085; enter browser-agent-installed-final, click pushState, and url_contains ?p=1 passes; 2 page-write actions, 0 screenshots, and 0 tool errors.PASS
Single-image visual inputSession local_7a8021f999; tool trace: observe → screenshot → verify; exactly 1 screenshot returning ImageContent and mime_type='image/png', without a validation error.PASS
Native model vision assessmentThe model judged from the screenshot that the current viewport contained a light purple gradient region. This is a model semantic judgment, not a DOM assertion.Visual result obtained
Deterministic completion evaluationIn the same visual session, title_contains Desktop Transfer Acceptance returns passed=true, so the runtime outputs status=succeeded, reason_code=verified. This assertion verifies only the title and does not independently prove the gradient judgment.PASS, with limited evidence scope

Installed-build defect resolution

DefectFixVerification
ToolReturn uses media_type to construct the provider image, causing a Pydantic validation error for missing mime_type during screenshot calls.3c850a91 changes to mime_type as required by the provider type. A screenshot without a frame ID reuses the latest observation while still allowing only one screenshot per frame.74 browser/tool-runtime tests passed; the installed visual trace confirmed the PNG image block reached the model.
After automatically detecting minimax-cn-coding-plan from provider.json, create_runtime() still indexed the six-entry built-in table directly, raising KeyError.2acf630b routes dynamic providers to the generic API runtime and normalizes provider-prefixed model names.6 provider-routing tests passed; the installed session successfully created minimax-cn-coding-plan:MiniMax-M3 Runtime.
The embedded desktop runtime installed only the base wheel, without the Playwright Python package, preventing CDP session creation.334d0abc makes the desktop runtime install the wheel’s [browser] extra and checks the import of playwright.sync_api at build time; it does not download Playwright Chromium.Release regressions, Bash syntax, and Ruff passed. Embedded Python imports passed, as did DOM and visual E2E tests.

The local macOS build is unsigned. Dependency installation also reported 8 high-severity Web findings and 2 moderate plus 5 high-severity Desktop findings. This task did not run dependency-tree-changing npm audit fix.

Task ledger

ItemEvidenceStatus
Base commit717d4e17Recorded
REDpython -m pytest -q tests/component/programs/browser/test_browser_agent.py: 9 failed, due to the missing public module and all constraints.Recorded
GREEN / affectedFocused browser/browser-control group: 51 passed; registry/tool-selection group: 25 passed.Passed
Specification reviewIndependent read-only review checked implementation commit 37b2c37c against all eight behavior contracts and security boundaries; zero remaining findings.PASS
Quality reviewInitial CHANGES_REQUIRED cancellation, URL-scheme, and cleanup findings were fixed by af33c608. Fresh review of candidate d0996f7b checked every changed boundary and confirmed the earlier KeyboardInterrupt finding was a false positive; zero remaining findings.PASS
Full gateReviewed candidate: browser/browser-control 58 passed; registry/tool-selection 25 passed; Ruff passed; docs 0 broken links; diff check passed.Passed

Task 2: Turn-scoped Surface Awareness and bound Web Use

Implementation scope: Chat turns automatically carry the current split web surface. The model’s first context includes a bounded DOM/ARIA preview. Expose web_use only when a valid surface exists; bind reads and actions to the originating WebSocket’s window_id + tab_id + target_id.

Foreground requirement: none. Electron directly reads the bound WebContents and returns its CDP target. activateView does not call BrowserWindow.focus() or another OS focus API. The target pane need not have keyboard focus; ordinary DOM/ARIA actions use neither desktop coordinates nor screenshots.

Runtime target: default profile ~/.openprogram, port 18100, and /Applications/OpenProgram.app. No second desktop process or visible worker is launched from Electron source.

Task 2 implementation layers

StepImplementationBoundary
1. Turn surface referenceThe composer constructs version/window/tab/region/access/focused/title/url references from the current compound group and submits them with the chat command. Regions use left, right, or center according to actual pane order; the surface chip shows the bound target and position.When access is disabled, retain only a disabled descriptor, with no preview or execution tool.
2. Exact previewThe server requests a preview only from the chat-originating WebSocket and validates exact window/tab/target. Electron derives the result from the current viewport’s DOM, ARIA landmarks, title, and URL.Text is limited to 2,000 characters, landmarks to 12 entries, and names to 160 characters. URLs retain only origin, without query values; no screenshot is sent.
3. Model context and tool schemaThe dispatcher injects Surface Context before the first model call.web_use is a resident tool but enters the provider tool array only with a valid surface binding; it is explicitly removed when no surface exists.The model cannot submit window/tab/CDP targets; it can use only the current turn’s surface alias.
4. Bound executionThe binding registry stores the originating WebSocket, window, tab, target, and expiry. Chat’s direct web_use calls retain runtime cards but execute in the main worker process holding the renderer WebSocket and WebSession registry. Only the GUI Agent Harness long loop uses the subprocess bridge. Browser Agent continues using DOM/ARIA refs and post-action observation.Do not use OS input, desktop coordinates, application-window focus, or a default active tab. Missing, expired, or identity-mismatched bindings fail.
5. Approval and verificationAfter the user sends a turn with an exact surface, the outer web_use does not request a second generic approval. Hard constraints, authority, and explicit deny/ask rules still execute first. Plan mode does not expose web_use. Parent-turn permission rules are passed to the agentic subprocess as a plain-data snapshot. Internal browser_page uses a scoped bypass only for the bound invocation and restores it afterward. Runtime assertions determine terminal state.High-risk action rules may still ask/deny. Write actions still require fresh DOM frames. Dynamic-page verification reads the current DOM of the same bound target and does not remain permanently stale because of unrelated mutations.
6. Cleanup ownershipFrom surface capture onward, the chat query enters try/finally. The dispatcher creates a stable Web Use owner for the current session and assistant turn. Normal return, retry, cancellation, and exception teardown release every WebSession, Page lease, and binding owned by it, without relying on the model explicitly calling close.prepare_turn, TurnBindings.bind or provider initialization fails, or the model never calls close, the Page must not remain page_in_use.

Task 2 installed-build acceptance

ScenarioEvidenceConclusion
First-turn page awarenessThe installed chat command contains an exact surface. The server sends preview to the same originating WebSocket; before its first reply, the provider receives the Bilibili title, origin, DOM excerpt, ARIA landmarks, and web_use schema.PASS, no screenshot
Bound-page clickSession local_340bc50ccd2c4eafb1fc12378e0489ab; the model calls web_use and internally clicks the homepage “Anime” navigation using a DOM ref. The command record includes both preview and activate.PASS
Deterministic resultThe target becomes https://www.bilibili.com/anime/?spm_id_from=333.1007.0.0; title_contains=番剧 in frame_6_a628e0d2 returns passed=true. The outer tool result is status=succeeded, reason_code=verified.PASS
Window state is not part of executiontest_web_use_background_app.py creates a temporary Page and immediately returns to the original tab, leaving the target Page as background WebContents of visible=false. It then executes list_pages → observe → viewport screenshot → DOM ref click → observe → verify → close; the original tab must remain visible after every agent action. Screenshots come directly from the exact Page’s CDP session. Web Use neither reads the system screen nor queries or changes system window ordering, position, occlusion, minimization, or foreground state. It does not call browser.close(), bringToFront(), window focus, full-screen capture, or OS input.Independent background-Page automated E2E

Task 2 verification commands

Task 2 current scope and remaining work

The current implementation supports turn-scoped exact Page binding, server-owned monotonic connection/Page/access revisions, renderer-owned monotonic geometry revisions, exclusive WebSession leases keyed by the canonical Page revision, and explicit screenshot for one current-viewport native-vision input. Within a session, action, verify, observe, and close execute serially. The initial observe and every subsequent observe, act, and verify revalidate the exact binding before calling the backend. Page capabilities store page/access/geometry revisions; agentic subprocesses return expected values through the parent-child bridge. The parent worker compares Page/access revisions, and the exact renderer checks in-app pane state and geometry revision both before native preview/activate and after asynchronous completion. Any mismatch returns page_context_stale without entering the backend. Stale asynchronous results contain neither preview nor target. Pane-bound or in-app visibility changes invalidate old WebSessions without rebuilding Page identity. Background Pages obtain exact targets directly through resolve. macOS window order, occlusion, position, and minimization do not affect binding or screenshot decisions. Leases are not released before close, initial-observe failure, owner disconnection, or process cleanup completes. Target replacement revokes old Page identity. Owner WebSocket disconnection immediately revokes all its bindings and wakes pending exact-socket requests. Component concurrency verification with two connections and 48 bindings confirmed commands go only to the originating connection and disconnecting one does not affect the other. Installed two-window transfer/rollback, temporary screenshot-artifact retention, and the unified three-backend parity suite are complete. A dedicated permission card is a later UI item outside this functionality-only scope.

Task 2 review record

ReviewFindings and handlingConclusion
Specification reviewThe first review found outdated target-design statements saying Surface Awareness was unimplemented and provider injection must wait for the complete registry. These now separately describe the current turn-scoped baseline and later full architecture; re-review found no new contradiction.PASS, zero findings
Quality reviewRe-review confirmed complete cross-process revision propagation and found a compatibility regression in custom single-argument binding_validator. The runtime now passes revision keywords only to the default production validator; custom validators retain the original one-argument contract. Both legacy and revision-aware paths have regression tests.PASS, zero findings
Geometry specification reviewReview of 47fad4a5, 33dd8f75, d77a17c3, 9c826f0c, 4ac96a21 and 2b3dda2e covered initial observe, second attach, preview/activate Promises, revision 0, and the actual hide lifecycle; no specification deviations remained.PASS
Geometry quality reviewIndependent review reproduced geometry changes during activation, missing initial-observe validation, stale preview-cache revisions, revision-0 compatibility differences, and incorrect in-app pane visibility. Fresh review passed after all corresponding regressions were added.PASS
Visual-artifact reviewScreenshots enter only the immediately following provider request. Runtime/DAG text retains only frame, viewport, and an image-attached marker. Normal return, provider errors, timeouts, and cancellation clear raw image blocks, ToolReturn.images, and coordinate permissions.Specification and Quality both PASS

Task 3: Unified command API and three backends

DeliverableDirect evidenceStatus
Unified entryweb_use(list_pages / observe / act / verify / close); a first-party OpenProgram MCP tool with the same name.Complete
Session selectorobserve selects playwright_mcp, chrome_devtools_mcp or open_claude_chrome; switching and automatic fallback are prohibited within the current session.Complete
Exact PagePlaywright MCP obtains a BrowserContext view containing only the exact target through the official createConnection(contextGetter); Chrome DevTools MCP resolves pageId using a one-time marker. Before initial and subsequent observe, act, and verify calls, the runtime revalidates the original bound target’s visibility, identity, and Page/access/geometry revisions. Controller app attach carries the same revision. Revision, identity, or actual visibility changes clean up the session before entering the backend. None of the three backends selects or foregrounds a tab.Components passed
Direct MCPWith one desktop control connection, inventory can obtain the current in-app Page. Multiple connections are rejected rather than selecting a random window.Components passed
Backend parityThe actual OfficialMCPPageBackend and ControllerBackend adapter runs the same matrix through public WebUseSessionRegistry.execute(), covering observe, seven allowlisted actions, stale/unsupported states, backend errors, built-in and Playwright timeouts, verify, PNG screenshots, and close. Every write action reaches the selected backend exactly once; failures never trigger fallback.Specification and Quality both PASS
Verification312 tests passed across Surface, permissions, cross-process bridge, backend parity, Runtime, threading, Browser Agent, MCP contracts, and routes. Full Web checks, TypeScript, production build, WebTab navigation/zoom, and installed transfer-client checks passed.Passed
Installed buildba665053 refreshed to /Applications/OpenProgram.app, using only default ~/.openprogram and the 18100 worker. Each of the three backends bound the same currently visible built-in Page and completed list_pages → observe → scroll → observe → text_contains verify → close; each path executed one write action and returned passed=true. Each backend also returned exactly one valid PNG with viewport metadata; acceptance did not save images. Actual two-window transfer/rollback also confirmed the target renderer was staged, source state unchanged, target window destroyed, and storage free of residual state.PASS

Task 4: Multiple browser Pages and popup handoff

Base: 54c466e9.Scope: Web Page inventory, background exact targets, and popup-opener ownership in the current OpenProgram window, plus explicit Page switching by browser_agent/gui_agent.

Production files: apps/desktop/main.js, apps/desktop/preload.js, apps/web/lib/desktop/desktop-bridge.ts, apps/web/lib/tabs/center-tabs-store.ts, apps/server/openprogram_server/_webui/ws_actions/webtab.py, openprogram/agent/surface_context.py, openprogram/programs/gui_browser_agent.py.Exclusions: Desktop application control, cross-window inventory, OCR/YOLO/memory, and automatic backend fallback.

RED: Inventory of A/B/popup C in one window; C retains opener=A; tokens bind exact targets; background B observe/act does not change visible/active/focus state; GUI Agent discovers C after a write and switches by token; no mounted native Page yields an empty inventory; close/target replacement/disconnection revoke capabilities.Gate: Affected Python tests, desktop navigation checks, Web split/compound checks, TypeScript, production Web build, docs build/links, installed 18100 E2E, and diff/status.

StatusEvidence
IMPLEMENTED · INSTALLED E2E PASSfe35dddf implements same-window Page inventory, background exact binding, popup openers, and explicit GUI Agent switching; 67a65e2a reads live URL/title from native WebContents for inventory, eliminating stale tab metadata after background navigation. No mounted native Page returns ok: true, pages: [] rather than HTTP 500.
Automated verification123 tests passed across Browser Agent, Web Use Runtime, backend parity/threading, Page bindings, and HTTP routes. WebTab navigation, Web split/compound, TypeScript, and production Web build passed. Behavior tests confirm background Page resolve/inspect never calls show, focus, or bringToFront().
Installed-build verificationUse only default ~/.openprogram, 18100, and /Applications/OpenProgram.app. Inventory returns foreground Grammarly and background Example together. open_claude_chrome performs observe → click Learn more → observe → close on background Example, navigating the actual Page to IANA while Grammarly remains foreground. The next inventory returns live background Example Domains / https://www.iana.org. The old application bundle was removed.
GROUP-AWARE INVENTORY · INSTALLED E2E PASSbrowserPageInventory() and list_pages now returns tab_entries + pages, active/focused identity, pane placement, and monotonic inventory_revision. Pane order follows actual visibleIds. Page capabilities are released independently; closing one session does not affect other Pages in the same inventory. Automated tests cover three tab entries, four Pages, member reordering, concurrent revisions, and sibling-session lifecycles. Installed acceptance completed list_pages → observe → click → observe → restore → close on the left Page of a real split entry, clicked Example’s Learn more to navigate to IANA, then restored the original page.
Retained scopeCross-window inventory is implemented in Task 5. Desktop application control, OCR/YOLO/memory, and automatic backend fallback are outside Web Use. Popup openers, token handoff, and stale/close revocation have component coverage; no installed-build fixture for public-site popups has been added yet.

Task 5: Cross-window Page inventory

Scope: Register each Desktop renderer’s window_id, allowing list_pages to aggregate all registered windows while retaining original connection/window/tab/target bindings. Screenshots still directly capture the exact Page’s current viewport without foregrounding a window or reading desktop imagery.

Compatibility: Retain existing top-level primary-window fields; add windows[] and Page-level window_id. Add no configuration ID, permission UI, OCR, desktop control, or automatic backend fallback.

StatusEvidence
IMPLEMENTED · INSTALLED E2E PASSd2e9baa2 defines the cross-window data model; 348e07f3 implements Desktop window_id registration and aggregation; 67e6ed09 uses window_id + tab_id to mark the GUI Agent’s current Page; 833c2314, 1699fb03 preserves independent window lifecycles and primary fail-closed behavior; 26c662e9, d4c4337e rejects delayed bindings after disconnection using connection-revision CAS.
Automated verification and re-review123 tests passed across Browser Agent, Web Use Runtime, Page surfaces, and process bridges. Ruff, Web split/compound/WS actions, TypeScript, and production Web build passed. Independent Specification and Quality re-reviews both returned PASS, covering unregistered clients, identical cross-window tab_id, origin disconnection/timeouts, materialization races, and late direct preview/active results.
Installed-build verificationUse only default ~/.openprogram, 18100, and /Applications/OpenProgram.app regenerated from current main. Actual list_pages returns main and detached-window snapshots were returned. The detached window’s Microsoft Edge Add-ons Page was selected for observe → screenshot → close, producing one exact-Page image/png current viewport. Execution used no window focus, bringToFront(), full-screen screenshots, or OS input.

Task 6: Public naming and boundary migration

Public capability: Agent registry, GUI Agent Harness, OpenProgram MCP, and canonical HTTP API all use web_use and web_session_id; model tool lists no longer expose computer_use.

Compatibility boundary: /api/computer-use and the old request field computer_session_id remain only as hidden HTTP compatibility entries mapped to the same Web Use Runtime; they do not appear in OpenAPI, Agent, or MCP tool lists.

Future boundary: Computer Use is reserved for future desktop applications, application-window or full-screen screenshots, and OS input. This change adds no desktop adapter, configuration, or background process.

Agent cursor task

BrowserPageController implements a click-only internal overlay on the exact Page; official MCP adapters reuse the same arming contract during click calls. Component acceptance covers DOM-ref clicks, viewport-coordinate clicks, no display for user clicks, feedback failures not changing action outcomes, and all three backends. 106 related regressions passed. Installed background-Page E2E completed a ref click with target visible=false and the original tab unchanged; the page click handler recorded cursor=true. Scope excludes system mouse, window focus, hover/type animations, configuration, dependencies, and persistent processes.

DeliverableDirect evidenceStatus
Runtime and sessionsweb_use_contract.py provides Chat Agent and MCP with the same command-conditioned schema. Empty legacy optional fields are treated as unspecified; act, verify and close each enforce required session/frame/arguments.web_use_runtime.py, WebUseSessionRegistry, the three browser backends, and exact Page lifecycles remain unchanged. Missing action fields return a complete missing_arguments without entering the backend or clearing a still-valid observation.Complete
Agent / GUI HarnessChat Agent can directly call web_use; gui_agent uses the same commands and session contract through the same Web Use adapter.Complete
MCP / HTTPMCP tools/list and tools/call exposes only web_use; the canonical route is /api/web-use.Complete
Cancellation and connection cleanup3a2e6baf closes WebSessions returned after MCP-call cancellation; cbc24778 precisely revokes unconsumed Page capabilities issued by the cancelled list_pages, without cleaning up other sessions or capabilities belonging to the same owner.Complete
Automated verification and re-reviewtest_web_use_runtime.py, test_web_use_backend_parity.py, test_web_use_backend_threading.py, test_web_use_surface.py, test_web_use_route.py and MCP/Agent groups total 453 passed. Independent Specification and Quality re-reviews both returned PASS.Passed
Current installed buildVerification uses default ~/.openprogram, 18100, and /Applications/OpenProgram.app: the Agent registry returns only web_use; OpenAPI exposes /api/web-use and hides legacy /api/computer-use plus the internal capability-release route. Actual list_pages successfully returned one window snapshot. No Page was registered during acceptance, so this round performed no observe or webpage actions and did not switch foreground windows for acceptance.API PASS · PAGE E2E NOT RUN