GUI Agent: complete computer-use design

Selected model for nested tools

A foreground turn binds the same provider and model for GUI and Workflow internal model calls as for its chat loop. Selection precedence is the request override, session selection, then the frozen or current agent profile. Session lookup is read-only for a frozen profile request. The resolved selection is frozen on the request and passed with the current work directory to tool subprocesses. A selected runtime setup error remains attached to the turn and is raised by an internal model call; it does not select another provider or inherit an outer runtime. Non-model tools can still execute. Releasing the turn restores the prior runtime context. An independently detected global provider is not the selected chat model.

Acceptance checks exercise TurnBindings.bind through override, session, frozen-profile and profile-default requests and verify restoration. Installed GUI and report execution must separately establish usable provider authentication and real tool actions; these component tests alone do not establish live execution.

Purpose and completion contract

The product goal is sustained work in browser and native applications: combine UI interaction with permitted code/file tools, verify deliverables, and remain controllable. Coverage includes dense controls, custom canvases, multiple windows, dialogs, downloads, saving, recovery, and concurrent human use where the OS supports it.

This is the canonical GUI Agent engineering design, rewritten in place. The body specifies the target; the current-source audit and final status appendix separate existing capabilities from implementation and live acceptance still required. Product instructions remain in GUI Agent usage.

• One public gui_agent(task), standard Agent configuration and shared history.
• Persistent code execution, complete UI primitives, browser/local desktop/VM adapters.
• Target ownership, inherited permissions, bounded execution, cancellation, restart recovery and final verification.
• Matched-model application tests. Foreground or VM operation does not satisfy a background-desktop criterion.

Foundation-model training, reproducing undisclosed OpenAI internals, bypassing app security and universal software compatibility are outside scope. Difficult background input remains a required workstream; it does not justify redefining the goal as Accessibility-only automation.

Reference systems and evidence limits

Docker/VNC supplies execution and display infrastructure, not model reasoning or automatic Windows-host control. A Linux container, a local macOS app and a Windows VM are distinct environments.

Current paths and verified gaps

The bridge registers one root and unwraps the Harness function. plan_next_capability serializes capability history into a prompt; call_capability selects the internal execution path.

gui_agent(task)
  -> plan_next_capability(task, history, availability)  # LLM
  -> computer_use(subtask)
       -> macOS window_session
       -> gui_step: observe
            -> verify_step(previous feedback)        # LLM, when feedback exists
            -> plan_next_action                      # LLM
            -> one registered action
  -> append inputs/outputs; choose again

browser_use(subtask)
  -> custom loop calling agent(tools=[browser_page], max_iterations=1)
  -> Page/frame checks, action, optional screenshot, terminal assertion

A desktop iteration with feedback normally has three model requests before optional localization/retry. This is a structural count, not measured latency. The first iteration omits prior-step verification. AX window actions bypass OCR/YOLO; other desktop/VM paths can use detection and localization.

The macOS window registry exposes only window_press, window_select, window_set_text, window_scroll and terminal markers; a screenshot-only window exposes only the terminal markers. Global-input functions elsewhere do not expand this entry. Browser actions include click/type/press/hover/scroll/select, but no exposed drag or persistent script entry.

Runtime.exec already uses DAG-rendered history and gates inner tools under caller authority. A second context engine is unnecessary. execute_code starts a fresh Python subprocess and cannot preserve UI handles or variables across calls.

Stable retained identity is the tab/resource association: tab id, conversation session, and branch provenance, plus last confirmed URL and title. The live page key, CDP/BrowserView handles, frames, and action leases are ephemeral to one process. Desktop views use the native persist:webtabs partition; the web UI persists center-tab descriptors in local storage. Restart restore recreates a new native Page/binding for each retained tab descriptor, loads the last confirmed URL, and keeps session/branch provenance. Old live handles may be marked unavailable. Orphan resource rows with no retained tab descriptor are not invented; they report failure in the original group.

Capability comparison

Partial is not full acceptance. Implementation gates for all targets appear in the final appendix.

Target architecture and responsibility

One gui_agent FunctionInvocation owns one standard Agent execution lifetime. It resolves configuration, acquires resources, exposes tools, enforces limits and returns verified results. It is not another planner. The model chooses observation, UI, ordinary tools and terminal verification.

gui_exec(code) is a normal function tool. Its persistent Python child accesses scoped proxies; a trusted broker owns Page/window handles, permissions, leases, coordinate transforms and action receipts. Model variable assignments cannot change these. The desktop helper provides input/capture; the worker retains Page ownership and task control.

computer_use, browser_use and vm_use become compatibility boundaries/internal adapters without additional task planners in this path. Keep the simple public form; resolve advanced settings through normal Agent configuration: explicit, then ambient, then default. Preserve tools=None, tools=[] and selected-list semantics. Add GUI tools only within outer authority, and remove recursive gui_agent. No GUI call restores denied shell/file/network tools.

Persistent runner and code contract

Use Python first because Runtime, Harness and adapters already use it. One isolated child preserves a namespace per GUI invocation, including top-level await. The broker may use Playwright internally; model code receives scoped proxies, not raw browser/CDP/AppKit/OS-input objects. A JavaScript language adapter is optional, not required for code-driven operation.

Reuse backend process lifecycle and ToolReturn images, with a dedicated mandatory GUI isolation policy. Do not change execute_code's fresh-process contract. Python namespace restrictions and Node vm are not security boundaries.

Prototype limits: 30 seconds and 100 broker operations per script, 256 KiB text, four images per return plus provider pixel limits. These are starting values, not measured optima. Long application rendering returns an asynchronous handle; bounded polling yields control. Timeout revokes write capabilities before killing the child.

# Proposed SDK; not currently implemented.
page = await ui.browser.open("https://example.org", background=True)
obs = await page.observe()
field = page.get_by_role("textbox", name="Search")
await field.fill("bearing dimensions")
await field.press("Enter")
await page.wait_for_text("Results", timeout_ms=5000)
display(await page.screenshot())
# page and field persist into the next gui_exec call.

Return ordered text/image blocks and typed details: environment, generation, operation ID, status, effects, observation, error and timing. Record each primitive as a child execution event. Errors identify the last completed primitive and uncertain effects. An executor receipt proves an operation was applied, not that the task succeeded.

SDK and surface semantics

Persisted Python proxies do not make stale targets valid. The broker revalidates every operation. Serialize writes to one target; independent reads may run concurrently.

Broker protocol and observation schema

Use an invocation-local IPC connection with length-delimited JSON messages and separate bounded image payloads. The host assigns the owner and generation at connection setup; generated code cannot choose them. Requests include an operation ID, resource handle, method, arguments and observation token. The server rejects unknown methods and oversized messages before executing.

# Proposed broker message shapes
request = {
  "operation_id": "op-17",
  "resource": "window-handle",
  "method": "drag",
  "observation": "frame-token",
  "arguments": {"path": [[120, 90], [180, 120]], "modifiers": ["Shift"]}
}
observation = {
  "target": {"kind": "window", "handle": "window-handle"},
  "generation": 3, "frame_id": "f-21", "base_frame_id": None,
  "image": {"width": 1600, "height": 1000, "detail": "original"},
  "transform": {"origin_x": 0, "origin_y": 0, "scale_x": 2, "scale_y": 2},
  "controls": [], "capabilities": ["observe", "click", "drag", "keypress"]
}
receipt = {
  "operation_id": "op-17", "state": "applied",
  "last_applied_index": 1, "effect_uncertain": False,
  "observation_token": "new-frame-token"
}

The broker persists admission and completion through existing execution effects before reporting a successful receipt. Admission is not proof of a side effect. For native input, event posting often has no transactional acknowledgement: after a lost response, observe rather than replay. Lease generation fences old connections after recovery.

Receipt admission follows normalization and current authority checks, and is serialized with revocation under a short lock. A receipt admitted before revocation may finish persistence afterward because it records an already-completed primitive; it does not authorize further input. No adapter, validator or database call holds that lock. Dynamic validators are followed by renewed execution/attempt and cancellation checks. Broker dispatch uses a separate copy of the owning host Context for each operation.

For an unrotated window capture, desktop points are x = origin_x + image_x / scale_x, similarly for y. Crops first map to parent-image coordinates. General display transforms use a matrix and must be invertible; the source observation defines the convention. Browser coordinates are viewport CSS pixels after mapping from screenshot device pixels. Reject points outside allowed content and stale transforms. Do not multiply Retina scale twice.

SDK calls may hide protocol bookkeeping, but cannot hide changing authority. A locator revalidation may update geometry inside one known target; opening another app or executing on a new origin must pass its own policy check.

macOS: complete input and background feasibility

An agentic Program subprocess inherits only the browser capability already authorized for its admitted parent turn. An existing parent surface context, including disabled access, is preserved without replacement. With no explicit context, the parent may capture a genuine registered OpenProgram Page inventory only when the ordinary registered web_use policy already permits it through SURFACE_GRANT. No task text, Program name, original-owner source reference or newly selected provider creates a grant. Capture failure retains denial; no synthetic Page binding or interactive authority is constructed. Capturing the same available inventory for a Program has a bounded renderer request cost and releases parent-owned temporary bindings through the existing finalizer even if the Program never calls browser tools.

The validated Page snapshot populates both the subprocess TurnRequest.surface_context and each derived inner request, matching the already bound execution context. At the subprocess operation boundary, the existing authenticated-owner permission snapshot resolves current mode and rules without changing the admitted request. An active parent plan-session flag further restricts that snapshot to plan mode. Page capture and subprocess mode/rules transport consume this same effective snapshot, so an existing Page context cannot discard either the current owner policy or the parent plan restriction. Inner requests inherit this snapshot; legacy callers without a mode snapshot retain approval mode. Runtime authority stays non-interactive; explicit deny/ask rules, uncertain-effect approval and target ownership continue to override the surface grant. The parent IPC broker keeps the existing exact window and binding allowlists, fresh revisions and cancellation cleanup. Disabled contexts, no registered App, failed capture and foreign bindings remain denied. A model-free owned registered Program exercises an actual spawned child and nested registered web_use permission gate; no user browser or provider is used for these checks. Unit delegation fixtures explicitly register the production web_use descriptor, provide isolated owner authority and controlled desktop availability, and assert the real policy decision. Missing registration, unavailable desktop or capture failure cannot create a fallback context. Unit capture callbacks prove transport and cleanup; actual spawned registered-tool checks separately prove Page readback.

The window session binds one ScreenCaptureKit window by its exact identity: window id, owning process, bundle identifier and process launch time. It then associates that capture with an Accessibility window by geometry: an AXWindows element whose AXPosition and AXSize agree with the captured frame within 2 points. With exactly one match, the session reads that window's controls, offers window_press, window_select, window_set_text and window_scroll, and window_match reports method: "unique_geometry". With several matches, the session refuses the window. With no match, the session is screenshot-only. This covers an empty or unreadable AXWindows list and one whose elements have no matching frame; WeChat for macOS can return a single AXApplication element without position or size, and its Accessibility state changes between runs. In a screenshot-only session, window_match reports {"verified": true, "method": "capture_only", "ax_association": "unavailable", "reason": ...}, where verified refers to the exact capture identity. The observation carries the exact window's screenshot with no controls (controls_available: false) and a rule that tells the planner controls are unavailable and the answer must be read from the screenshot. The planner is offered only done and fail. Dispatch refuses every action without touching the window and returns a recoverable result (action_status: "refused", reason_code: "window_controls_unavailable") with an explicit reason; there is no foreground, coordinate, global or process-targeted fallback. Each validation of a screenshot-only window checks the process launch identity and asks the window server whether the exact window id still belongs to that process and is on screen. Without an Accessibility window, a minimized window cannot be told apart from one on another Space, so an off-screen window is refused rather than captured. Every computer_use step binds afresh, so a window whose Accessibility state recovers regains its controls on the next step.

The macOS window feedback uses a non-activating, mouse-ignoring status pill at the top centre of the window and an agent pointer at the last action; Agent control overlay defines how they look and move. The helper starts after exact window validation and exclusive application control. Window visibility is the target rectangle minus covering windows, applied as a layer mask; an unrelated overlap cannot suppress the entire panel. Markers describe the successful action's pre-action control bounds, stored relative to the verified target window. The capability feedback retains this display metadata with its original window identity, and restores it before the next same-window step. It is never an input capability or success receipt. A normal step ends with bounded, explicitly ended feedback; failure, cancellation or lost identity closes it. Helper failures are diagnostic warnings and do not fabricate action success or perform foreground fallback. Source lifecycle changes do not establish installed native display acceptance; feature execution is unverified under the user's no-test instruction.

The macOS window adapter derives its required permissions from the host capability registry entries mapped to gui_agent:desktop. Availability and the adapter's system_access diagnostics contain only these required rows: Screen Recording and Accessibility. Unrelated Calendar, Reminders, microphone, camera, target-specific Automation and path-specific file access remain visible in the complete system report and retain their own operation checks; they cannot block capture or AX window control. A missing required row or a required state other than granted denies the window operation before native initialization. The same check is repeated before window actions, so revocation remains effective. Hosts without the registry API retain the existing nonprompting Quartz/AX checks. No read requests permissions or creates a foreground fallback.

Window inventory converts native macOS application names, bundle identifiers and window titles to plain Python strings before returning capability availability to a durable GUI step. Window identity also converts the native NSDate launch timestamp to an exact Python string before publishing it in the capability target or next-feedback window target. NSDate can return a bridged string subclass from its first string conversion; converting that string representation again preserves the date text while removing the bridge subtype. Missing values remain null, integer window/process identities remain integers, and text content is preserved. A missing launch date still rejects the target; restart and previous-target comparisons retain their existing meaning. The durable JSON validator remains strict; native bridge objects cannot enter stored step results. This conversion does not capture a window, request permission, change focus or replay an unresolved operation.

Shared-report compatibility fixtures supply both desktop-required ids, including a granted Screen Recording row when only control is denied. Their existing report, denial-reason and privacy assertions remain intact. Separate public adapter regressions deliberately omit each required row or supply unknown states and require rejection before native initialization.

This scoped adapter report also feeds the installed bridge's retained capability planner and desktop typed wait. Default browser tasks remain independent of desktop permissions; missing recoverable desktop grants still wait before the selected desktop effect. Delivery of the desktop access repair updates only the GUI commit in scripts/release/product-runtime.json to the independently reviewed Harness revision; the official refresh archives that exact commit and rejects source/pin drift. Other program pins, dependencies, permissions and runtime capabilities retain their existing values. A local reviewed commit and a successful local archive do not prove that the remote repository or CI contains it. The product runtime bundles the pinned Harness archive, while an owner catalog may resolve another source in a fresh loader namespace. Public bridge source metadata does not prove the cached worker's Harness module identity. Acceptance separately records the exact owned registered-entry source, installed native readiness and visible desktop execution.

Retain ScreenCaptureKit and AX. Add a stably signed native helper for full input, bound to application identity, window and broker lease. Use AX for supported controls; investigate process-directed events for canvases and keyboard workflows.

The local SDK declares CGEventPostToPid; AXUIElementPostKeyboardEvent is deprecated. A PID is not a window ID. Prototype correct window routing, coordinate interpretation, event acceptance and foreground/pointer preservation. No claim is made that OpenAI uses this API.

The opt-in Harness probe tests/probe_native_input.py uses an owned AppKit fixture. On macOS 15.7.3 arm64, raw process-directed mouse events reach the application without a window association. NSEvent supplies the window association, but subsequent mouse coordinates are incorrect. An experimental CGEventSetWindowLocation annotation with the full window frame, including its title bar, produces the expected click and drag coordinates. This setter is undocumented: the experiment establishes a local feasibility result, not a supported production contract. The default probe retains the public-API control; the undocumented path requires --private-window-location.

The event fixture receives ordinary and Unicode key-down characters without clipboard use. The separate --controls fixture checks native text replacement, an NSButton action, local WebKit textarea replacement and a canvas drag using actual control values and DOM event positions. Command+A requires the fixture's standard Edit menu; without it, the initial experiment appends text and fails. Control geometry is measured after WebKit loads. These tests establish a small AppKit/WebKit control sample, not installed-app compatibility, IME, general shortcuts, multiwindow isolation or screenshot-coordinate correctness.

Stronger application-identity and two-window checks invalidate a general background-input conclusion from that single-window sample. A temporary signed bundle launched through LaunchServices satisfies the existing MacWindow launch-identity guard and yields an exact-window screenshot. With an initially key sibling window, ordinary mouse flags do not focus WebKit; subsequent web text goes into the native field. Explicit Command-modified mouse events reach the canvas, but keyboard text goes into the sibling. This is an unresolved routing/focus failure, not a transparent fallback. The probe records both conditions as failed control acceptance and checks the sibling's actual text on every receipt, including an explicitly requested final state after input. A production adapter must establish the intended focus and preserve requested modifier semantics before sending keys.

Foreground identity and pointer position are sampled before and after; this does not exclude transient changes. Keep background_input unavailable until the full implementation gates pass. The exploratory background-click analysis identifies the undocumented annotation as a candidate; its claims about other software are not adopted as verified facts.

Advertise background_ax, background_input, foreground_input or unavailable. Background is default where verified; foreground requires a permitting task/session policy and visible control. Never silently activate an app. If public APIs cannot provide a required background behavior, evaluate a supported vendor integration or tested native approach and leave the criterion open.

Transform screenshot pixels using display identity, window origin, scale and crop metadata. Cover Retina, negative multi-display origins, moved windows and scale changes; reject stale geometry. Check capture and input permissions independently. Minimized windows, secure fields, lock state and permission dialogs have explicit outcomes.

Prefer AX replacement and verified key insertion. Default background and foreground text input never uses the global clipboard as an automatic transport. Use AX, Unicode/key events or an app-local insertion/import API. A clipboard change counter only protects restoration; it cannot prevent another process from replacing text before paste. Explicit system-clipboard tasks use host/pasteboard arbitration, preserve concurrent changes and do not carry a concurrent-user correctness guarantee. Browser-local paste is separately scoped. Serialize physical keyboard/pointer leases at host/session level, not merely per window. Bind modal sheets/menus as descendants or explicit new targets, never the frontmost window.

Browser: reuse ownership and extend operations

Editable-field observations keep four diagnostic ancestors and may include compact field_context.label_binding evidence from the nearest semantic container within twelve ancestors. A complete walk of at most 120 descendants must find exactly one visible editable control and one visible, non-editable leaf label. Ambiguous, incomplete or editable-value labels do not bind. The evidence records its source, exact normalized label, depth, counts and independent redacted/truncated flags; native and ARIA labels remain unchanged. Password controls do not expose a binding. Full secret masking precedes the existing string caps and 32768-byte aggregate context budget. Unrelated diagnostic ancestors may be dropped without changing complete binding evidence; clipped or masked binding text is unusable. Structural evidence does not grant write authority. Native ref actions retain the complete normalized native-label identity before public clipping and the full compact binding internally, then compare fresh semantic evidence before input. A changed, missing, newly added or ambiguous binding invalidates the observation; unrelated diagnostic layout text and classes are not action identities.

Reuse WebUseSessionRegistry, Page tokens, worker-owned WebContentsView/CDP, frame checks and cleanup. Attach to the exact submitted Page or open an owned background Page, never the currently active tab. Keep optional MCP adapters as execution adapters.

Page discovery does not establish that a selected MCP backend can start. A known MCP startup failure returns computer_use_backend_unavailable with availability=unavailable, the actual selected backend and the closed session identity; raw upstream diagnostics are not emitted. It never selects another backend implicitly. Failed observation releases the Page lease and captured bindings, removes temporary binding markers and closes the owned MCP event loop after its thread stops. Cancellation, permission failures and unexpected defects retain their existing error semantics.

Expose scoped role/name, label, text, test-ID and relative locators plus frame entry. Reuse Playwright actionability and bounded waiting. Resolve each operation afresh; ambiguity or semantic/owner change returns a stale-target error. Numeric refs are observation-local.

Add drag, multi-click, modifiers, horizontal scroll, file chooser/download, iframes and popups. Bind popups by opener and exact identity, recheck destination authority, and verify transfers through approved paths. Arm download/popup waits before the trigger so events are not missed.

Do not expose raw page.evaluate, arbitrary CDP send, cookies, profile directories or browser launch in UI scripts. Trusted code may evaluate DOM for bounded observation. Ordinary shell/file/code tools stay available through normal gates; explicit developer-debug capabilities use the same policy. A UI SDK is not an unrestricted browser control endpoint.

Windows, Linux and VM boundaries

Windows requires an explicit active-desktop mode: capture the intended display/window and execute input under a foreground lease. Detect user takeover, desktop switches, lock and integrity-level restrictions. UI Automation handles standard controls; coordinate input covers custom canvases. Background Windows control is not claimed by the reference product and is not silently promised here.

Linux X11 and Wayland are distinct adapters. X11 can use existing input/capture infrastructure; Wayland requires an explicitly supported compositor/portal or remote desktop backend. Return an unavailable capability rather than assuming X11 permissions. The first Linux acceptance target is a declared isolated desktop/VM image, not every distribution.

Keep OSWorld-compatible screenshot/execute transport, but move target identity out of process-global monkeypatching. One environment object owns its transport, screenshot path, input state and lease. Credentials stay in the executor. Validate images and timeouts, distinguish read retries from uncertain writes, and test two independent VM targets. Docker is optional Linux infrastructure; local macOS and Windows-host use do not require it.

On macOS, lock-screen automation requires a separate reviewed design for OS authorization. Do not add a privileged unlock helper as a side effect of UI completeness. Locked use is a disclosed reference-only capability until that gate passes; the task pauses with a precise reason and resumes from a fresh observation after manual unlock.

How context remains continuous

Use ordinary user/assistant/tool messages from the DAG as the sole model-visible history. UI code, every operation result and selected screenshots are recorded with the same invocation lineage. Remove duplicated JSON history injection after migration; preserve the legacy exported result history as a derived view of events, not a second source of truth.

Separate three lifetimes: Agent conversation, execution namespace, and external application state. A continued model response restores neither a Python object nor an application window. Record environment generation and resource bindings at safe points. Compaction retains the task, permissions, unresolved criteria, artifact paths, last verified state and uncertain effects. Never serialize live AX/CDP handles as portable checkpoints.

Return small AX/DOM diffs by default, with stable broker IDs and an explicit full-snapshot request. Missing or out-of-order base observations force a full snapshot. Images carry frame/target metadata and provider-compatible detail. Preserve the most recent relevant visual state, final evidence and failed-action observations under a declared retention policy; use the shared Context Engine for token budgets and summaries.

Crop on demand at original resolution. Each crop stores an affine transform back to its parent image; coordinate SDK calls require the crop's token or explicit parent frame. Reuse existing OCR/detector/zoom grounding as an optional tool for uncertain targets, not mandatory preprocessing or extra model calls on every native AX/DOM action. ScreenSpot-style localization measures one component, not long-task completion.

Latency, cost and verification strategy

Optimize measured end-to-end work. The current structural path repeats capability choice and per-step verify/plan. The target uses one Agent decision for a short deterministic group, returns state, then decides again. Do not remove outcome verification; move deterministic checks into the executor and reserve model verification for ambiguous visual states and milestones.

Report total time as model time + capture/encoding + broker/input + application waits + recovery. Count model requests, UI primitives, observations, image tokens, retries and manual interventions. An API call completed by the model is not an action applied by the executor.

Do not promise a speedup factor before measurement. Initial performance acceptance is no completion-rate regression and lower median model requests for the matched deterministic subset; publish latency distributions and failures for the whole suite. Dense visual tasks can reasonably spend more on images if completion improves.

Execution authority and isolation

Generated Python is arbitrary code, so OS isolation is mandatory. Launch the persistent child using a GUI-specific sandbox profile that intersects outer restrictions with an unconditionally enforced scratch/runtime/IPC-only policy. The ordinary Backend.spawn_spec path is not sufficient. Use a strict launch context that owns its scratch directory, fixes the interpreter and mandatory policy, requires the platform sandbox, and refuses fallback. Runtime bootstrap may read literal ancestor directory entries, but this does not permit descendant file contents. Neither danger-full-access, a broad inherited override, nor sandbox-unavailable may disable this per-tool isolation; in that case gui_exec is unavailable and ordinary separately gated tools remain unchanged. Deny direct network, desktop automation, arbitrary host-file access and raw CDP/native handles in this child. A filtered globals dictionary is not an isolation mechanism.

The trusted broker holds OS permissions and external handles. Authenticate the child connection using process identity and an invocation-bound channel; do not pass a reusable global bearer credential into generated code. Validate owner, generation, exact target, operation, current user grant, deadline and cancellation before each effect. A batch admission is not blanket authorization for future targets.

Ordinary Agent tools remain separately gated. If a script needs file or shell work, request the normal tool through an explicit brokered tool proxy, or return to the Agent to invoke it. Reuse the existing tool dispatcher and immutable outer request. Do not copy the legacy browser path's special bypass into a general code runner.

UI observations are untrusted data. Page text and screenshots cannot authorize new app access, alter system instructions or invoke restricted tools. Tool descriptions and app capabilities come from the trusted host. Preserve user approvals for the same authorized action; new apps or expanded effects follow current policy. Do not add a confirmation to every harmless click.

Revocation first fences new effects, releases held keys/buttons and stops the script; cleanup follows. Retain bounded private traces locally with explicit artifact retention. Redact secure fields and credentials, and never publish screenshots or user data as an automatic consequence of testing.

Failure, cancellation and restart recovery

Desktop access at the selected capability boundary

The task-only gui_agent entry admits planning and authorized browser work without requiring local desktop permissions. Explicit desktop execution retains its pre-effect access check. Default capability discovery separates recoverable missing OS grants from unavailable native dependencies: only a selected local desktop operation requests the required access. Browser and VM operations retain their own target and authorization boundaries.

A trusted host orchestration adapter retains each planner decision and capability result as JSON through the existing durable function records. Its resumable arguments contain task/configuration data; the live Runtime is bound by a host ContextVar. A fixed, versioned GUI-operation API connects retained orchestration to the pinned Harness planner, executor and conclusion functions. Snapshot admission recognizes only those operations; it does not permit arbitrary dynamic imports, opaque host objects or generated code. No public resume_state, history or cursor parameter can replace the persisted execution state.

1. Commit the next planner decision before selecting an external operation.
2. For computer_use, recheck desktop access before registering or dispatching its effect. A recoverable missing grant produces typed function suspension with the retained call identity and selected capability; an unavailable backend remains terminal.
3. The child reports suspension through IPC. The parent probes the execution host again and opens the existing system_access wait from its current owned attempt. Child or model reports never establish a grant.
4. Forced calls retain the function cursor in the wait checkpoint. Agent tool calls retain both the function cursor and the existing provider/tool continuation before opening the wait. A crash between those commits resumes the suspended function and recreates the required wait.
5. After actual authorization, resume the same task, recover committed planner and browser-step results, recheck access and dispatch the first pending desktop operation. Cancellation, revoked access, an old generation or an unresolved effect prevents dispatch.
6. If the parent already confirms access at admission, save a silent resolved wait and recover only that exact attempt/checkpoint after releasing its runtime and handle. Release-time recovery does not probe, resolve or resume other open waits; explicit permission waits retain the worker's normal reconciliation.

The access check is a separate pre-step boundary: step() marks an effect dispatched before calling its operation. Waiting inside a desktop step would create an uncertain effect and prevent safe continuation. The child also cannot open the parent-owned wait directly, because opening a wait ends the active attempt. Default forced calls capture authorized Page context for the browser capability as well as the selected native capability.

Reuse the existing execution-control safe points, cancellation and owner generations rather than creating a second scheduler. Persist serializable resource descriptors and last confirmed operation receipts. A crash destroys live namespace state: recover at a script boundary, rebind and reobserve; do not promise to resume a Python stack or replay a sequence of UI writes.

Public result semantics stay succeeded, infeasible or failed; cancellation remains the execution system's cancelled state. Pause/approval/takeover are resumable control states, not successful results. Include reason, unresolved criterion, last verified state and artifact evidence. Failed cleanup cannot rewrite an actual task failure as success.

Browser restart recovery

A worker or Desktop process death is not a user Close webpage. The retained tab/resource association (tab id, session, branch, last confirmed URL and title) survives. The previous live page key, frames, CDP/BrowserView handles, and action leases do not. After the App and worker are healthy, restore each still-retained tab automatically in the background by creating a new verified native Page/binding with a fresh capability identity, loading the last confirmed URL, without raising the window, adding extra top tabs, or showing a previously hidden PiP. Compact PiP chrome, the vertical Files / Activity / Resources sidebar, and branch-grouped resource rows stay as specified in Resources and operation preview. Locale follows App language settings.

Last confirmed URL and title must agree on that retained association. A later navigation that was persisted must not reopen an earlier Google URL under an arXiv title. Distinct retained tabs stay distinct even when URLs match. Explicitly closed Pages remain closed and appear only under Closed pages, not as live rows. Hidden PiP is a preview preference, not a failure.

Process incarnation may mark old live handles unavailable for safety and must drop them. It must not treat the retained tab association as user-closed, must not duplicate that tab, and must not move a recoverable restore or disconnect into a generic Unavailable group. The row stays in its original branch while restore runs or fails. The renderer registers its window again after it recreates retained native Pages; a register that arrives while that socket's restore is running queues exactly one more pass after it, so Pages created during the first pass are rebound instead of staying failed. The restore asks the renderer for its inventory in restore scope, which lists every Page the window holds, Pages private to a conversation included; Agent inventory requests keep their conversation scope.

Restoring page presence does not require or imply Continue Agent. Idle pages stay Ready to use. New task actions still need ordinary fresh observation and authorization after the new binding exists. Old binding, frame, and action handles remain invalid. Do not bypass authority or app security; do not auto-resume old Agent actions, write fences, or input leases. Restore reloads the last confirmed URL with existing site persistent storage in persist:webtabs; it does not restore unsaved DOM or form fields. Older orphan resource records with no retained tab descriptor cannot be recreated automatically; they report Could not restore page in the original group. This bounded restart contract is implemented. Native App acceptance on the default profile and port, including App-only restarts, verified hidden delayed navigation with query and fragment, surviving fixed and hidden preview preferences, site token and counter persistence, first human click after restore, and an explicitly closed page remaining closed. Combined three-run tests, Web, contracts, and Desktop/docs gates are green. Other GUI Agent recovery, desktop, and VM work remains pending. This row does not mark the GUI Agent project complete.

Resources and a unified operation preview

Resources groups retained targets by resource type. Resources has no search field. A row click shows a interactive chat preview and keeps that existing select behavior. An explicit Preview in conversation icon button does the same for the exact existing Page, without adding a top tab, restores a hidden PiP, and returns to the owning session when a webpage currently occupies center so the chat preview is visible. Hide keeps the Page. There is no standalone Show preview control. Explicit Open page reveals that same Page as a regular top-level tab and hides that preview together with any reserved space; returning to the conversation restores the preview unless the user hid it. TARGET (not current runtime): a webpage tab and its small chat preview may display at the same time and stay in sync on the same Page; hiding the preview does not close the Page or the main view. Runtime is unchanged. Desktop PiP presents the existing native Page with direct input and live rendering, including while dragged. Its bounds and layout zoom follow the preview size. A never-shown Page starts with a 1920×1080 CSS-pixel viewport. Plain-browser previews use a separate sandboxed iframe with browser embedding restrictions. Browser interaction is the first implementation scope. Native application and VM adapters, trace review and learning require separate implementation discussions. Verification and remaining work are recorded in the implementation-status appendix.

Resources in the current conversation's right sidebar lists all associated targets by resource type. Agent-opened pages remain in the background without adding top tabs. One chat preview follows the target most recently operated on by the viewed branch. Browser Pages, native application windows, the host desktop and VM displays share selection rules while retaining distinct input capabilities.

The preview header identifies Auto preview or Fixed preview beside the current control status. The More menu checkbox Automatically show the page the Agent is using selects Auto preview; disabling it keeps the current Page as Fixed preview. A different preview title does not retarget the Agent, resume or start the Agent, or change Page lifetime or top-tab placement. This display option exists only in the preview menu; Resources keeps its existing action buttons. Default-App acceptance is a separate release check.

Interactive appearance

The current interactive resource grouping and preview controls are maintained in Resource panel and preview interaction. It uses type groups, unchanged sidebar actions, and Open page, More and Close preview icons.

Resource and preview responsibilities

• Retained browser descriptors and branch associations provide the resource inventory. Temporary usage records describe execution use and do not determine Page lifetime. The same Page can have references in more than one branch.
• A resource row click selects a interactive chat preview and keeps the current tab. Explicit Preview in conversation uses the exact existing Page, enters Fixed preview, and does not pin or duplicate a top tab. If a webpage occupies center, that action activates the owning current session so the requested preview is visible, without changing the viewed branch. Explicit Open page uses the exact existing Page as an ordinary top view and hides that preview together with any reserved space. Changing automatic display in More does not steal tab focus. Returning to the conversation restores the preview unless the user hid it.
• Resource lifetime, preview preferences and execution input leases have separate state. A window-level projection keeps following and control status current when the Resources panel is closed. Hiding the preview releases its visible bounds and retains its target.

List and preview appearance

Keep the existing vertical Files, Activity and Resources navigation at the top of the right sidebar; do not replace it with horizontal tabs. The Resources redesign changes only its content. Use the existing sidebar's dark surfaces, system font and 16px type icons, with no bordered card per resource. Each group heading reuses the left sidebar Pinned and Projects SectionHeader: 14px normal secondary text, a 16px chevron immediately after the resource type title, the same padding and hover, and the resource count as trailing actions. Enter or Space expands or collapses that group only. Compact rows are approximately 44-48px tall: title first, then type, state and child-agent name when needed. Long titles truncate; details expose full titles, sanitized origins and target identity. Hover and selection use subtle surface changes. An operating indicator and preview selection are distinct, with text rather than color alone explaining state. Webpage rows expose three compact Lucide icon buttons beside the title: Preview in conversation, Open in tab, and Close webpage. They share the existing compact toolbar pattern (24px hit target, 14px glyph, same hover, disabled, and keyboard-focus treatment as PiP chrome). Do not use a literal arrow glyph or the global 36px Button for these controls.

The chat preview identifies its title. Chrome is one compact row that must not wrap, including at the 240px minimum width, the 300×198.75 default (300×168.75 image plus a single header), and the 720px expanded width: an ellipsized title, a compact control-status indicator, Open page, More, and Close preview (X). Title truncates before buttons. Show actions and Operation history belong in the existing native-safe More menu. Primary actions stay on the row; do not move them into a second row or a popover toolbar. The screenshot fills the remaining box with native layout zoom and keeps the captured aspect ratio; do not crop or zoom page content to enlarge the thumbnail, and do not accept clicks on the image. Idle and closed show Ready to use and have no Pause or Continue control. Failure text stays discoverable through the status indicator, title tooltip and aria-live without adding a second chrome row. Reuse the sidebar's adjustable width. On narrow product layouts, a Resources drawer selects a target while retaining chat and preview access. This document stacks its mock panes on narrow screens for review; it does not propose an additional product page.

Browser, application window, desktop and VM boundaries

Ownership, branches and concurrency

Resources persists serializable descriptors; the host retains actual Page/window/VM handles and leases. Reuse existing conversation and branch projections. Group keys require stable branch identity or verifiable fork lineage, not the changing message HEAD or a new group per execution. Renaming, continued turns and retries update presentation or associations; forking does not duplicate external page state.

• A descriptor includes resource ID, type, sanitized target description, actual owner, lifecycle, connection generation and preview capability. Branch associations separately track originating invocation, branch, child agent and sharing mode. Do not merge distinct Pages with identical URLs; merge UI and usage records for the same actual Page.
• Child-agent resources appear under the originating parent conversation branch with their actual owner identified. Later child work or branch renaming must not rewrite provenance. Existing session/execution authorization remains per resource; a shared display group grants no access.
• Ancestor resources show Referenced from. Browser branch writes default to an independent Page, without claiming external website or login-state isolation. Native windows and VMs cannot be cloned automatically; require explicit lease transfer or an independent target.
• Preview selection is not control acquisition. Shared references may appear in several branch rows with their current controller shown. Write leases exclude conflicts at the actual input scope: Page, application process with shared focus, VM or host desktop. Window grouping does not establish concurrent input support.
• Preview events carry session, branch, resource, connection generation and monotonic sequence. Discard stale events after branch changes, reconnection or revocation. Metadata is restorable; process-local handles are not. Legacy records without provable branch ownership belong in Unassigned, without guessed migration.

Inspection, takeover and disconnection

The image mirror is read-only; the actual tab or application remains normally usable. Its manual entry and yielding rules are specified below. Explicit stopping acknowledgement and ordinary human access are distinct; do not turn an Agent permission into a page-blocking mask.

Idle background targets retain their connection without high-frequency capture. Only the visible preview subscribes to images; do not periodically scan all host applications. Show frame freshness and operation progress. A disconnected resource labels its image Last frame, exposes metadata and disables input. App restart, Page crash and VM reboot require identity validation; durable metadata does not imply a live target. Under memory pressure, offer explicit resource release rather than silently discarding forms or stopping VMs.

Deterministic state transitions

A parent branch follows its own executions and child executions created in that branch, in host dispatch order. A child conversation window follows only its own branch. A shared target may have reference rows in several branches; each group counts unique targets and lists unavailable counts separately. Remove reference only detaches an idle association. Close webpage and Stop VM affect the actual target and enumerate all affected references. Completion retains grouping. App restart restores grouping and preview preferences, never remote human-input connections or Agent write leases. Titles and secret URL query values do not establish ownership; secret query parameters are not persisted.

Failed native focus or result checks show Input failed, revoke writes and retain evidence, without continued background retries. Legacy adapters lacking dispatch/completion events show Executing, operation state unverified, without an operating dot or unconfirmed interactive takeover. Host desktops bind display identity; layout changes invalidate old coordinates. Disconnecting either a user-opened or agent-opened application never quits it. Quit application is a separate explicit user action. Disconnect VM releases transport only; the guest and virtual display remain running.

The viewed branch is the branch checked out in the conversation pane; resource selection never changes it. Auto preview follows that branch and its attributed children. Continue Agent is a separate explicit action with fresh observation and authorization.

When the selected target closes or its VM stops, clear Operating and show that target's unavailable/last-frame state. Fixed preview keeps this state until selection changes. Auto preview also keeps it until a newer admitted operation in the viewed branch selects a target; it never silently falls back to an older resource. Restoring a hidden preview with a row click or Preview in conversation, and branch restoration, obey this rule.

Existing tabs, operation cues and human takeover

PiP is exclusively a chat preview of an existing Page. Opening any regular webpage tab hides that preview and any reserved space so the native Page fills the available content area. TARGET (not current runtime): a webpage tab and its small chat preview may display at the same time and stay in sync on the same Page; hiding the preview does not close the Page or the main view. Runtime is unchanged. Do not duplicate a Page screenshot beside or over the actual Page, dim the Page, add a blocking ownership layer, or hide an iframe under CSS while native geometry stays reserved. Opening the Page keeps the same Page identity, resource and follow preference. Returning to the conversation restores the preview for that branch unless the user explicitly hid it. A row click and Changing automatic display never steal tab focus. Explicit Preview in conversation does not reveal a top tab; if invoked while a webpage occupies center, it activates the owning session so the chat preview is visible. Only explicit Open page reveals the Page as a top tab. The live webpage stays undimmed and usable in its own tab. The full Web toolbar, control menu and actions remain above the Electron Page.

One native WebContentsView cannot occupy two rectangles simultaneously. Keep one live native interactive view when the Page tab is current, and deliver the same target's image stream to chat PiP only while that conversation is current. This restricts simultaneous native-view attachment, not the number of UI references to one Page. Additional presentations use mirrors; locating the existing tab selects that live primary view and hides the preview. Chat PiP remains a floating overlay on the conversation. The PiP image is strictly read-only: never forward scaled clicks, keys or scrolling from it, and do not crop or zoom the capture merely to enlarge the thumbnail. A never-shown Page is created with a 1920×1080 CSS-pixel viewport at the user's normal page zoom so capture has a stable 1080p layout; PiP size is independent of that native viewport. Returning a visible Page to the background keeps its last native bounds rather than shrinking them to the thumbnail or reapplying the creation default. Do not apply the legacy PiP layout-zoom path to thumbnail size. Open page locates the exact existing tab. If no view is visible, explicitly open a view of that Page, without another Page, login, reload or URL-based identity guess. Native windows and VMs also distinguish their actual control entry from an image mirror. Hiding the mirror leaves the live page available. Viewing or focusing the live tab does not request a pause.

While PiP is visible, subscribe to continuing captures of that exact Page, including changes caused by both Agent and human input; do not move the live view to capture it. Each frame carries resource ID, connection generation, sequence and capture time. Reject frames from an old binding. Set a bounded refresh rate and stale-age threshold during adapter acceptance, expose the last-frame age, and label delayed or unavailable capture as Last frame rather than Live. A static screenshot is not a live mirror. Hiding PiP unsubscribes its capture demand without closing the Page; do not capture every retained resource continuously. Dragging the floating chat PiP caches container bounds at pointer down and previews movement with a transform on animation frames. Pointer-move does not remeasure layout, write React or store state, or persist the rect. Pointer up, cancel and lost capture write the latest position after a single clamp against the current container bounds. Unmount, live-loss and a resource switch discard an in-progress drag, clear the transform and release pointer capture without writing that rect. Capture and image replacement pause for the drag and resume on pointer up, cancel or lost capture; they do not resume a stopped loop. A capture that started before pause is discarded even if resume happens before it settles; resume then starts a fresh capture when the old request finishes, keeping a single in-flight capture. Pausing must not permanently stop the loop by clearing isCurrent. The compact preview resizes from all four edges and four corners; there is no triangular corner-only handle. Aspect lock and native-acceptance of those handles remain a separate geometry check.

Operation cues without obstructing content

This example illustrates exclusive surfaces: the chat preview or the live page, never both. It does not change the vertical Files, Activity and Resources navigation. Simulate an Agent click, open the exact Page, toggle Show actions, edit the sample page, acknowledge stopping and resume. Everything is local document state, not control of a real webpage.

Which targets humans can use directly

Pausing while the human can still use the page

Agent active describes host-authorized automation writes, not a lock denying human page access. Actual human input closes subsequent Agent dispatch in the conflicting input scope, invalidates queued old-frame actions and requests a pause from the owning execution. Keep the page's brightness and usability, show Pausing… in its toolbar, and never copy or replay the human click later.

Dispatched operations may be impossible to retract immediately, especially native input and long browser calls. Late results can occur during yielding: do not claim zero overlap or confirmed stopping. Record unresolved effects and observe again, without automatically undoing human changes or resuming the agent. Show Agent paused only after stopping acknowledgement, held-input release and target-state reconciliation. On failure or timeout, show Could not pause. Try again in the toolbar with Retry pause while connected; normal human access remains available and new Agent dispatch stays closed. Advertise zero-overlap automatic takeover only after adapter acceptance establishes it.

Pause Agent to use page opens the exact human entry after stopping confirmation; sending a pause request is not success. While waiting, the button shows Pausing…; an already open live tab remains clickable, editable and navigable. Only subsequent Agent dispatch is gated, not human page input. Defer locating an unopened entry until acknowledgement; the separate Open page action still permits direct access and hides the chat preview. Idle and closed targets show Ready to use and have no Pause or Continue control. Could not confirm status, or Connection lost when the connection is known to be down, must not claim a pause failure and keep a disabled Pause Agent to use page control. Could not pause. Try again keeps Retry pause, enabled only while connected, and must not look paused. Keep one stopping request per input scope: an explicit pause reuses any request already started by direct human input and its acknowledgement, rather than starting a second handshake. Closing the pause presentation does not cancel stopping. Continue Agent is available only while Paused. Disable Continue Agent during Pausing…, unknown status, or pause failure. Allow status reconciliation retries and, when needed, reconnection to the same verified instance; enter Paused only after confirming old operations stopped and reconciling state. A late acknowledgement enters Paused without automatically resuming. Direct human input triggers pausing without a confirmation dialog. Continue Agent is a separate explicit action, requiring fresh observation and current authorization. Follow changes only preview selection, never input admission or tab focus.

Previewing, expanding, hiding and branch inspection are distinct from live input. Hiding a mirror neither revokes manual interaction in a visible live tab nor resumes the agent. Closing the actual controller view, losing its connection or invalidating target identity closes that connection's automation/remote-input authority and leaves the agent paused; OpenProgram does not claim to block normal local OS use. Inspecting another branch changes presentation only; pending requests remain bound to the original resource ID.

Pointer down in the real page (including starting text selection), page-operating key/input events, scrolling and user-initiated navigation request yielding. Viewing, focus alone (including Tab/Shift+Tab focus navigation and modifier-only keys), pointer movement/hover and read-only PiP inspection controls do not. Hover-induced page changes still invalidate old observations. The browser adapter must distinguish human events from Agent dispatch without relying only on page-content scripts. Adapters without reliable detection offer explicit pausing and do not advertise automatic yielding.

The actual input channel determines the pause scope: a browser uses Page identity, shared by its existing tab, additional views and mirror. A native window uses window scope only after independent window input is verified; otherwise use the actual shared process input domain or host keyboard/mouse domain. Global desktop input uses the host input domain. A VM uses the guest input domain, shared by all consoles connected to that guest. Close all Agent writes in that scope, not only one requesting view. The real tab remains interactive during Pausing… and Could not pause. Try again; only the Paused claim and completion of the explicit acknowledged pause path remain unavailable.

Edge cues, toolbar and action locations

Edges, toolbars and action markers below belong only to OpenProgram-owned presentation surfaces: the hosted webpage frame/toolbar, an OpenProgram guest console and the PiP preview layer. Original applications, the host desktop and external VNC/RDP clients are outside those surfaces. Do not alter their menus or draw their window borders; keep controller state, Show actions and history in OpenProgram PiP or its corresponding control panel, with locations over the captured preview. Locating the original window does not grant authority to change foreign application UI.

• Use a subtle approximately 1px outer glow while automation is active. Do not dim content or change layout, scrolling, focus or hit areas; avoid continuous flashing. Toolbar text identifies agent, branch and Active / Pausing… / Paused / Could not confirm status / Connection lost / Could not pause. Try again. Color is supplementary. The currently visible surface — the live Page tab or the chat PiP, never both — shows the status edge: blue for Active, subdued amber for Pausing…, and an ordinary edge plus explicit text for Paused, unknown, or pause failure, without an active glow.
• Add Show actions and Operation history to the existing page toolbar, not to external page DOM. Show actions defaults on and displays only the most recent action; history lists click, type, scroll, drag, navigation and verification in order without changing Files/Activity/Resources navigation.
• Brief rings indicate clicks, paths indicate drags and arrows indicate scrolling. Typing identifies a field/action without exposing passwords or full text. Shortcuts use text descriptions. Cues belong to the target, not the system pointer: never move the user's cursor to illustrate a trace.
• Only trusted host dispatch/receipt events produce cues. Distinguish attempted, dispatched, acknowledged, failed and unknown. DOM-ref operations without a precise point use element bounds or text rather than invented pixel accuracy. A primitive receipt does not establish overall task success.
• Bind resource ID, connection generation, observation frame and viewport mapping to both live-view and scaled-mirror coordinates. Account for content bounds, DPR, zoom, scrolling and iframe offsets. Navigation, layout/window changes or stale frames invalidate live markers; history retains old-frame labels instead of painting them on current content.
• Use a host-owned pointer-events:none presentation layer while original content receives input. Rendering above Electron native views requires actual acceptance; an ordinary DOM overlay is not assumed to work. Do not inject scripts into external pages to draw Agent traces. The live-view cues belong to the OpenProgram native host presentation layer; PiP cues belong to the preview layer. Both share the same resource/frame mapping and toggle state. Show actions hides both sets of action markers without affecting status edges, execution or operation history. Starting yielding clears both sets of markers. During Pausing…, Paused or Could not pause. Try again, late receipts enter labeled history only. Markers return only for newly authorized Agent actions with valid frames. Reduced motion uses brief static marks.

Later review and learning: retain verifiable records first

Reuse existing operation receipts with task/branch/resource, before/after observation, target description, action parameters, order, outcome and human corrections. Visible cues consume that same event source, not a separate invented trace. Records initially support debugging and inspection. With user approval, a later task may read selected sanitized excerpts to analyze targeting errors or propose reusable operation rules.

Record my next demonstration is a separate opt-in with explicit start, stop, scope, preview and deletion. Do not record all desktop activity by default or retain secret fields, passwords, secure input or credential-bearing URLs. Persist screen images only after permission and redaction checks. When reliable redaction is unavailable, keep content-free action metadata and omit the image/demonstration segment. Distinguish human demonstrations, automation receipts and failed traces; the user selects material before use.

Task-local review, saved reusable steps and model training are separate stages. This design does not train or upload automatically, or claim one interaction teaches the model. Validate proposed rules on independent test targets rather than replaying them directly on real user pages, applications or VMs. Training and data export require later discussion and authorization.

Implementation sequence

1. Resource-type grouping, click-to-preview, explicit Preview in conversation, Auto/Fixed preview, the automatic-display checkbox in More (Resources retains its existing buttons and has no search field), and closing semantics are the current Resources interaction. A hidden PiP is restored by the resource-row Preview in conversation icon; a row click keeps existing select behavior. There is no standalone Show preview control. Hide keeps the Page.
2. Resource records and branch projection retain descriptors, actual ownership, child-agent references and legacy migration.
3. Unified preview state and Resources rows preserve Page identities and separate view hiding, geometry and resource closing.
4. The live webpage tab and the chat preview are exclusive; human-input yielding, stopping acknowledgement and explicit resume follow the current takeover rules.
5. Native windows, the host desktop and background VM adapters remain separate. Advertise each capability only after real-target acceptance.
6. Connect edge cues, Show actions and receipt history; verify native overlays preserve input, coordinate freshness and image privacy. Demonstration recording and later review remain separate.

Primary locations: session-resources-panel.tsx, session-resources.ts, use-session-resources.ts, web-tab-pip-store.ts, desktop-bridge.ts, the existing ResourceUseStore, execution ownership and browser registry. Reuse execution control without another scheduler.

Acceptance cases

• One branch opens five pages: no extra top tabs, all five remain accessible in the vertical Resources list, and actual operations select the chat preview. Explicit Preview in conversation on a never-opened Page keeps that Page identity, adds no top tab, and enters Fixed preview.
• Two branches and their child agents run concurrently: resource type grouping with branch provenance retained, no task changes on checkout and no incorrect target rebinding.
• The user inspects A while the agent operates B: A stays the chat preview, B shows Operating, and enabling automatic display in More returns to B. Neither selection nor changing automatic display steals tab focus.
• Close preview, resizing and group collapse neither reload pages, stop VMs, grant control nor destroy targets.
• Closing an active target, revocation, late frames, restart and disconnection have explicit states. Unconfirmed stopping cannot be reported as confirmed pause; direct human access remains available.
• Opening a webpage tab hides the chat preview and any reserved space so the native Page fills the content area. Preview in conversation from that webpage returns to the owning session and shows the chat preview of the requested Page. Returning to chat restores the preview unless it was hidden. Undimmed content stays interactive, human input closes later Agent dispatch, late actions do not imply confirmed stopping, and focus alone does not pause.
• The visible surface of a Page keeps updating; failed capture is labeled Last frame. Markers never intercept input or display coordinates from stale frames or changed geometry.
• Same-process windows, concurrent user input and two VMs verify their conflict scopes independently. Successful capture does not establish background input.

Keep the single GUI Agent form, exact submitting window/Page identity, verified deliverables and existing native operation indicator. Product acceptance still uses the default App; a working document prototype is not implementation evidence.

[S13] Current Resources/PiP sources · [S12] App entry

Application suite and matched comparison

Two evaluation tracks prevent confounding: (A) same model through the current and new OpenProgram runners, isolating execution changes; (B) Codex product versus OpenProgram with matched model, task, OS/app versions, input files and practical budgets, measuring product outcomes. Tool capabilities and hidden prompts differ in B and must be reported. API-only results do not prove Codex parity.

Start with two tasks per family, three independent reset runs per task: 48 runs per system/model condition. This is an initial diagnostic suite, not a statistically sufficient universal benchmark. Freeze inputs, output checks, budgets and failure categories before running. Record every attempt, including unsupported tasks and infrastructure failures separately; do not remove hard cases from the denominator.

Report completion and false-success rates, p50/p95 duration, model/image tokens, API cost when known, tool calls, interventions and recovery success. Use paired task outcomes and uncertainty intervals; expand the sample if a parity conclusion is statistically unresolved. Keep visual judgments separate from deterministic artifact checks and use a reviewer unaware of the runner when possible.

Mandatory release conditions: all deterministic ownership/cancel/stale-frame/isolation tests pass; every declared critical capability passes its real-app fixture; no false success in the acceptance suite; no completion-rate regression against baseline; documented product comparison and unresolved limitations. Numeric safety targets are proposed as broker admission cancellation within 500 ms and held-input release within 1 s on the same host; measure and revise only with explicit evidence, never retroactively to turn a failing run green.

Adopted, modified and rejected choices

Implementation sequence and bounded work packages

The dependency order is P0 → P1 → P2/P3/P4 → P5 → P6. Browser and native adapter packages can be developed independently after the broker contract is fixed, but integration acceptance uses one combined candidate. Each package requires a public-entry failing test, minimum implementation, affected checks, independent specification review, then quality review and exact-commit evidence.

Each brief must list exact new test filenames and commands before code starts; names above identify ownership rather than pretend tests already exist. Keep API adapters backwards compatible while switching the single GUI root only after behavior gates pass. Legacy direct browser_agent, the browser_control compatibility name, web_use and CLI arguments remain explicit adapters until their consumers migrate. Never execute both old and new paths for one task.

At release, reconcile OpenProgram and the separately versioned GUI-Agent-Harness repository: exact source states, dirty work, installed source selection and app bundle contents. Preserve unrelated Harness changes. Build from the integrated checkout with scripts/refresh-local-app.sh once after the completed feature; do not restart from an old worktree or create another visible App. Publish only after explicit remote-write authorization.

Evidence index

Source locations

GitHub main links locate files; findings come from the inspected local sources. Immutable revisions, file digests, dirty state and command logs belong to the review package. A remote link does not prove it matches local code.

Implementation status and release gates

This document is a target design, not a completed Computer Use implementation. The browser runner, broker and explicit browser entry are merged. Source integration, installed-build checks and real-task acceptance are separate evidence categories; none establishes complete desktop support or product parity.

Engineering records keep current design in place without dated decision appendices or duplicate versions. Before updating any target capability to implemented, bind it to the public entry, tests, live evidence, reviewed source state and combined release gate. Keep baseline failures and unsupported cells visible.