# GUI Agent

Give it one natural-language task. Without an explicit browser selection, its root controller repeatedly chooses one bounded capability: `computer_use` for the local desktop, `browser_use` for an OpenProgram background Page, or `vm_use` for a configured remote virtual machine. Every capability's planner-selected arguments and full result are appended to the next model decision's context. The model ends the task by proposing a terminal result; action and time limits remain safety boundaries.

Local and VM perception combines YOLO component detection (GPA-GUI-Detector), OCR (Apple Vision on macOS, EasyOCR on Linux / Windows), and template matching. The action layer covers mouse, keyboard, and clipboard. Browser operations use the Page's DOM/CDP target instead of desktop coordinates.

## Availability

Every supported release registers this Program and ships Playwright Chromium plus the GPA detector weight. It does not ship PyTorch, OpenCV, or EasyOCR, so desktop perception that needs those libraries is unavailable in the packaged product. Source-development checkouts can still install the harness's own dependencies. Developers can use an editable GUI harness checkout or replace the OCR/browser backend for debugging and backend work.

## Usage

The public entry function is **`gui_agent`**, registered as a tool (`as_tool=True`, toolset `harness`). Its public input is only `task`. The planner and the three `*_use` functions remain traceable child function nodes, but they are not registered as separate public tools.

Run it directly from the command line:

```bash
openprogram programs run gui_agent -a task="Open Firefox and go to google.com"
```

The Programs card asks only for `task`. There is no required surface selector. On every iteration the controller reads the original task, the prior planner-selected capability arguments and full outputs, and current capability availability before choosing the next function.

For a task that is naturally satisfied by the current built-in browser Page, use the same entry:

```bash
openprogram programs run gui_agent -a task="Inspect and complete the current built-in browser form without foregrounding the window"
```

Trusted callers can also supply hidden controller settings: `max_steps` is the action safety limit (default 150); `max_seconds` is the wall-clock time budget; `app_name` selects component memory; `backend` pins an existing Page backend; and `vm_url` enables `vm_use`. `surface="browser"`, or a `backend` without another surface, selects the standard browser Agent path described below. Other surface settings remain compatibility preferences. These settings are absent from the public function schema.

Task-only execution does not require desktop permissions before planning or authorized browser work. If the planner selects `computer_use`, the execution checks native access before dispatching that capability. Missing recoverable OS grants pause the same task; authorization resumes it using saved planner decisions and capability results. Cancelling prevents continuation, and an unconfirmed operation is never automatically repeated. Missing native dependencies remain an unavailable capability rather than a permission wait. Explicit desktop execution retains its initial access check. Real permission recovery in the installed default App still requires acceptance.

## Explicit browser execution

Inside an active macOS Runtime execution, trusted callers can select `surface="browser"`. This path uses one standard `agent()` loop with the persistent, isolated `gui_exec` Python tool. It lists context-authorized Pages, acquires single-use Page capabilities, and exposes opaque handles. It does not open a new Page automatically or run the legacy capability planner. The default backend is `open_claude_chrome`; explicit MCP backends without operation guards return `infeasible` rather than silently selecting another backend.

`max_steps` limits Agent iterations on this path. Each Python call is separately bounded to 30 seconds and 100 broker operations. `allow_general=false` exposes only the GUI tool; `true` also makes standard tools available subject to inherited permissions and deny rules, excluding recursive `gui_agent`. Python state persists for this invocation; screenshot bytes are returned as normal Agent image content. Missing execution identity or unsupported process isolation fails closed.

Browser `observe` returns each element's `ref`, `name`, `role`, and `label`. The label comes from `aria-labelledby`, `aria-label`, or associated HTML labels, without guessing from ancestor text. Editable text inputs and textareas return the exact DOM `value`; contenteditable fields return rendered `innerText` with `value_source="inner_text"`. Values preserve whitespace. A fresh observation is required after a write. Exact field verification requires a string `value` with both `value_truncated=false` and `value_redacted=false`; ARIA whitespace normalization does not prove exact equality. Each value is capped at 8192 Unicode code points, with a 32768 UTF-8 byte budget across values in one frame. Any affected field has `value_truncated=true`. Passwords and ordinary fields containing a known password omit `value` and have `value_redacted=true`. Observation text and ARIA are redacted before display limits are applied. Truncated prefixes of known passwords (at least 32 characters) are also redacted and cannot serve as verification assertions.

`field_context` is a read-only DOM diagnostic, not a field label or permission to fill a form. For an editable field it describes the element and at most four ancestor elements, including IDs, up to six classes, visible editable counts, direct text, and up to six visible text leaves per ancestor (at most 120 descendants are scanned). Field contents are excluded from text leaves and ancestors' direct text, including read-only nodes inside contenteditable values. Text is redacted before its 400-code-point display cap. Contexts share a 32768 UTF-8 JSON byte budget; `field_context_truncated=true` marks incomplete diagnostics, and the context can be omitted when the budget is exhausted. Inspect the actual structure before adopting a field association; order alone does not establish one.

Native `type` keeps ordinary input and textarea filling. Contenteditable editing inserts plain text and sends an input event after the native editable check, preserving blank lines without changing the editor CSS. If the editor rewrites the text or its CSS collapses significant whitespace, the action returns `ok=false`, `reason_code="editable_value_mismatch"`, and `observe_required=true`. Observe the current value before deciding what to do; this result does not authorize an automatic repeat write.

Browser observations redact nonempty password-field values from DOM names, visible text, ARIA trees, and upstream MCP snapshots, including child frames and open shadow roots. Field labels and ordinary input values remain available; text matching a password is redacted wherever it occurs in an observation. Action metadata and the generic browser Accessibility action, including error and no-match replies, use the same protection. Freshness checks retain raw Page identity internally; display redaction does not invalidate a stable Page. Generated frame IDs and refs avoid known-password collisions. Verification checks actual Page content, not the redacted observation. Final results, including completion evidence, use the same redaction. Assertions containing a known password or its JSON-escaped form are rejected with `sensitive_assertion`, `passed=false`, and no echoed assertion value.

Browser completion uses the latest successful verification. A later rejected or failed verification replaces earlier success evidence. Before reporting success, the Page controller rechecks document identity and the recorded assertion against current content; navigation, document replacement, or a disappeared confirmation requires fresh verification.

The model proposes a final browser assertion. The host checks it against the owned Page and current frame after the Agent returns. Success requires that assertion to pass and no unresolved primitive effects from this invocation. The result includes the verification effect and assertion evidence; a model-selected assertion does not prove every aspect of an arbitrary task. Script completion alone cannot report success. Handles are revoked before Page leases are released, and cleanup errors prevent a successful result. Cleanup may still wait for an already-running browser operation; bounded in-flight cancellation and default-App acceptance remain unverified.

## Browser resources and human control

In the desktop App, Agent-opened Pages remain in the background. Open **Resources** in the right sidebar to see the current conversation's Pages grouped by branch, including Pages used by its child agents. Continuing a branch keeps its resource group. Distinct Pages stay distinct even when they have the same URL; pages without established branch ownership appear under **Unassigned**.

Select a resource to inspect its image preview in chat. **Preview in conversation** is the explicit control for the same existing Page: it does not add a top tab, enters Fixed preview, and if a webpage currently occupies center, returns to the owning session so the chat preview is visible. The chat preview chrome has a persistent **Pin preview** / **Unpin preview** control: pin holds the current page as Fixed preview; unpin returns to Auto preview of the Page most recently operated on by that branch. Selecting a resource keeps that selection while the Agent works elsewhere and does not steal tab focus. These pin controls exist only on the chat preview; Session resources has no pin control. **Hide** stops preview capture and keeps the Page available. Restore a hidden preview with the resource row or **Preview in conversation**; there is no separate Show preview control. **Expand** enlarges the image preview without changing Auto or Fixed preview. **Open page** opens the existing Page as an ordinary top tab and hides the chat preview. Returning to the conversation restores the preview unless it was hidden. These controls keep their selection per conversation and branch.

A webpage tab and its preview never display together. The actual webpage remains usable in its own tab. There is no dimming mask. Its toolbar identifies the current controller and provides **Pause Agent to use page**, **Continue Agent**, **Show actions**, and **Operation history**. The chat preview uses a single compact chrome row; Idle and closed pages show **Ready to use** and have no Pause or Continue control. The default overlay is 300×198.75. Drag the header to move it, or resize from any edge or corner; the image aspect stays linked and the page itself is not resized. Action markers are brief and do not intercept input; history contains operation types and results rather than the text entered into a webpage.

Using the real webpage requests a pause: clicking, typing, scrolling, and navigating close later Agent input admission and request execution pause. Merely focusing the page or inspecting its preview does not pause execution. An action already dispatched can still finish. **Pausing…** means stopping is pending; **Paused** requires execution acknowledgement and reconciliation. **Could not pause. Try again** means the App has not confirmed stopping; **Retry pause** repeats the pause request while connected, and the webpage remains usable. **Continue Agent** is explicit and requires a fresh observation and current permission.

Closing a Page affects every branch reference to that same Page. An active Page first requests stopping; it stays available until stopping is confirmed. Closing is separate from hiding the preview. Disconnection or an unavailable target shows the last image as stale. Saved resource metadata does not restore an Agent input lease or recreate a closed Page.

If the App or worker restarts, pages you had not closed come back in the background without extra top tabs. The retained tab, session, and branch stay; the last confirmed title and URL stay. The App creates a new live page for that retained tab. Previous live handles are invalid. Explicitly closed pages stay under **Closed pages**. A hidden preview stays hidden until you select the resource or use **Preview in conversation**. Rows that are restoring, failed to restore, or need a reconnect stay in the original branch and show **Restoring page…**, **Could not restore page**, or **Reconnect**. They do not move to a generic Unavailable group. Restoring a page does not require **Continue Agent**. Idle pages stay **Ready to use**. New task actions still use a fresh observation and current permission on the new page. Restore reloads the last confirmed URL with existing site persistent storage; unsaved DOM and form values from the old process are not restored. An older resource record with no retained tab descriptor cannot be recreated automatically and reports **Could not restore page** in its original group. This bounded restart recovery is implemented.

These browser controls operate on exact OpenProgram Pages. Native application windows, the shared host desktop, and VM displays retain their own capability and input-scope restrictions described below.

## Automatic capability execution

The following sequence applies when no explicit browser path is selected:

1. `plan_next_capability` receives the task, current availability, and complete ordered capability history.
2. It selects `computer_use`, `browser_use`, `vm_use`, or proposes a terminal result.
3. `call_capability` binds controller-owned runtime settings and invokes exactly the selected function.
4. The planner-selected arguments and full function output are appended to history and therefore visible to the next decision. Controller-bound feedback is recovered from the previous output's `next_feedback`; it is not duplicated inside the next history input.
5. A proposed terminal result is validated. Unsupported success is recorded and planning continues.

`computer_use` and `vm_use` each execute one existing Harness step: observe the current target, verify prior feedback when present, plan one action, execute it, and return the step plus next feedback. `browser_use` executes one bounded background Page sub-task and then returns control to the root loop. There is no special pre-route for screen-reading tasks.

The implementation uses OpenProgram's high-level agentic programming calls. `plan_next_capability`, desktop planning, verification, and conclusion call `llm()` with the active Runtime context. The Browser Page action loop calls `agent()` with its action tool and a single bounded iteration. GUI workflow code does not call `Runtime.exec` directly. The root controller does not wrap itself in a nested `goal()` call because it already owns the capability history, terminal proposal, evidence validation, timeout, cancellation, and no-progress decisions; a second goal controller would duplicate those decisions.

`vm_use` requires an OSWorld-compatible HTTP endpoint. Screenshots are read from `GET /screenshot`, and input commands are sent to `POST /execute`. VM target selection is serialized within the Harness process. Whether the call succeeds or raises, the prior input target and screenshot backend are restored before another capability runs. Endpoint credentials and query values are not included in planner availability context.

Desktop observations include the frontmost application and screenshot coordinate bounds. If the target application's windows are minimized or located in another macOS Space and remain unavailable after one bounded Window-menu recovery, the run stops as infeasible and asks the user to move or unminimize the window. It does not create additional windows indefinitely.

Desktop coordinate input always applies to the current foreground GUI. When the controller has an exact macOS process and window target, `computer_use` may instead use window-only capture and supported Accessibility press, text-value, or scroll actions without activating the target. During control, a compact dark pill at the top centre of the target window reads “OpenProgram is using WeChat” (in Chinese when that is the system's first language), with a pulsing dot. An agent pointer, an arrow in the agent accent colour, rests at the window centre until the first action. After a successful action it glides to the centre of the control it acted on, plays a short ripple for a press or selection, and shows a small tag naming the action: Click, Select, Type or Scroll. The tag never shows the typed text. Later steps on the same verified window restore the last pointer in place, without replaying the ripple, until the next successful action moves it. With Reduce motion turned on, the pointer moves without gliding and there is no ripple or pulse. The pill and pointer follow the window. Partial occlusion does not hide them. Minimizing or closing the window hides them. A normal step exit briefly shows “OpenProgram finished with WeChat”; failure or cancellation removes feedback immediately. The overlay takes no clicks; stop the agent from its conversation in OpenProgram, as with any other run. The indicator neither activates the application, receives mouse input nor moves the system pointer. Browser actions use the selected Page in the background and do not activate its tab, raise the OpenProgram window, or move the system pointer. The controller may switch between these capabilities when the recorded results require it.

All runs share the same terminal fields: `status` (`succeeded`, `infeasible`, or `failed`), `success`, `reason_code`, `summary`, and `handoff_instruction`. The runner, not the conclusion model, determines success. `success` is true only for `succeeded`. Infeasible and failed results always return `success=false`; infeasible results retain the blocker, marker, and user handoff instruction. The automatic capability path also contains its ordered capability history and timing.

`max_seconds` is enforced before each model or capability call and again after it returns. A terminal proposal that arrives after the deadline is rejected and normalized as a timeout failure. When the deadline and a step or decision limit are reached together, the result reports `timeout`. Provider cancellation is cooperative, so an in-flight provider request can return slightly after the configured wall-clock boundary; it still cannot turn that run into success.

A call started from a chat or the Run form without `max_seconds` gets a 300-second budget; set `OPENPROGRAM_AGENTIC_TIMEOUT_S` to change the default (`0` removes it). The budget is passed to the agent as `max_seconds`, so a run that runs out of time stops by itself and returns `status=failed` and `reason_code=timeout` with the capability history it completed, without the conclusion model call. The run executes in its own process, which OpenProgram ends only if it is still running after a further 20% of the budget (at least 30 and at most 120 seconds). That error, `agentic subprocess timed out after N seconds`, lists the last steps the run reported and how to allow more time.

The Function card displays that task result directly: `Succeeded` for a verified result, `Failed` when the task ended without satisfying the request, and `Needs takeover` when the handoff instruction requires user action. `Error` identifies a runtime exception or an invalid GUI result contract. An internal completed worker state never changes a failed GUI result into `Completed`.

## Dependency notes

- Product runtimes do not install PyTorch or EasyOCR.
- The release capability probe rejects an artifact if the detector model is missing.
- Program registration is included on macOS, Linux, and Windows x86_64
  runtimes; individual desktop backends still follow the harness's platform
  and dependency support.
- The runtime needs a working directory configured before running. Workflow records are stored under the OpenProgram state directory (`gui_harness/workflows/`), not in the source tree.

Source and README: `openprogram/programs/packages/gui_harness/`, upstream repository [Fzkuji/GUI-Agent-Harness](https://github.com/Fzkuji/GUI-Agent-Harness).

Page discovery through `web_use` does not verify that its selected MCP backend can start. The official MCP server is launched from the npm cache when its pinned version is already there, so a slow or offline network does not delay the start. If the server still cannot start or connect, observation returns `reason_code=computer_use_backend_unavailable`, `availability=unavailable`, the actual backend, and a closed session. OpenProgram releases the failed session and its Page lease; it does not silently select another backend. Observe the Page again to retry; each attempt starts the backend fresh. If the server stops during a session, the next observation restarts it. An action is never sent through the restarted server: the action returns `computer_use_backend_unavailable` with `observe_required=true`, and you observe again before retrying it.

Browser Workflow forms expose only the task and optional target URL. Action limits, timeout and backend remain internal settings with defaults; no Advanced section is shown. Explicit programmatic calls retain their supported overrides.

## System access diagnostics

System settings shows live optional desktop access on the execution computer.
Opening this page or running `openprogram doctor` does not request permission.
The TUI `/doctor` command and `GET /api/system/access` expose the same checks.
Missing optional desktop access does not prevent ordinary chat or upgrades.

On macOS, managed installations run the worker and its Python children through
**OpenProgram**, an embedded application with the OpenProgram icon and
a stable bundle identifier. This is the name to look for in System Settings;
the installer does not grant its permissions. Screen recording and Accessibility
are checked separately in a fresh process of the executable used for desktop tasks. In local System settings, **Set up access** explicitly requests
only a missing permission; an existing grant is left untouched. Complete the
system confirmation, then return to the page for automatic verification.
Remote clients must arrange authorization on the execution computer. A grant
for another application is not evidence that the worker is authorized.

Linux headless sessions and unsupported Wayland desktop capture are reported
separately from missing dependencies. An X11 display alone does not prove access.
Windows desktop access depends on the active session and target; ordinary
applications cannot assume access to secure desktops or elevated targets.
Neither platform is instructed to disable security or run the entire application
as administrator. Native Linux and Windows desktop acceptance remains unverified.

The current macOS development build is not production-signed. Permission
persistence across signed release upgrades, first-run capability onboarding,
and automatic task recovery after authorization remain unverified. System
access setup never retries a declined operation.


### System authorization

Local desktop tasks pause before desktop actions when system access is missing. OpenProgram requests the missing permission through the native authorization flow and opens the corresponding System Settings page when necessary. The conversation shows a brief paused waiting status and an Open System Settings text action, without a permission form, Continue button, null result, Retry, or Edit action on the paused runtime card. Genuine failed and cancelled runs keep their own terminal statuses.

The background worker saves the waiting task and checks permission through a fresh OpenProgram process, using the same executable as the desktop task. This avoids reusing a permission result retained before the system setting changed. These checks do not request permission. Once access is available, it continues the same task once. Closing the page does not stop this check. If an application restart is needed, the saved task is checked again after startup. Cancelling the task prevents later continuation. A failed check keeps the task waiting; it is never treated as permission granted. Viewing an old conversation does not itself start an operation or reopen system prompts. System access does not replace operation approval.

On local ad-hoc macOS builds, an update can invalidate a previously granted recording permission while the system switch remains enabled. OpenProgram remembers successful recording checks for its installed application. If a later application identity changes and recording is denied, the next local authorization action renews only OpenProgram’s obsolete recording registration before opening the native permission flow. This recovery runs at most once per changed identity; status checks and reconnects never reset permissions. Stable Developer ID releases use the normal system authorization flow.

Repeated delivery of the same waiting task does not reopen native authorization when you switch conversations. You can still open System Settings explicitly.

### Automation errors on macOS

Some desktop commands send Apple Events to System Events or a target application. macOS controls Automation separately for each target; screen recording and Accessibility access do not imply Automation access. If an operation reports an Automation denial, open **System Settings → Privacy & Security → Automation** on the execution Mac and inspect the requesting application's access to the named target. The failed operation remains failed; changing the system setting does not approve or repeat it. Other native command failures retain their actual error instead of being reported as success.

## Model selection

Programs whose bodies do not call a model can execute without a configured model. An unavailable model or credential fails when the body calls the Runtime; it does not prevent model-free steps from running and does not select another provider automatically.

Direct tool runs also need an explicit provider and model for internal model calls. An omitted or incomplete selection does not use global provider detection or a provider's default model.

When run from a chat, internal GUI and Workflow model calls inherit the turn's provider and model selection. A per-turn override takes priority over the session selection, followed by the agent profile. The selected provider and model also apply to tool subprocesses. Provider authentication must still be valid for that selected model; setup failures retain the selected model's error instead of silently using another provider.

Browser workflows retain an explicit initial URL when a backend is selected. Official MCP backends support guarded waits and propagate snapshot failures. Page recovery stays within the current window scope and does not replay the failed action. A verified result retains its actual evidence and observed target.