Agent control overlay

When an agent operates a native macOS window, OpenProgram draws two things above that window: a status pill that says the agent is in control, and an agent pointer at the last action. A webpage the agent operates shows the same pointer. This page defines how both look and move, which colour they use, and the rules that keep them from getting in the user's way.

Preview

The same moment in a chat app: the agent has just clicked the message box. The left panel is the earlier rendering; the right panel is the current one. Use the buttons to play a click, a typing step and the finished state.

Before

After

Accent #5b5bd6, a placeholder (see Accent colour). Everything else is neutral.

Surfaces

Target	What is drawn	Where the code is
Native macOS window (`computer_use` window actions)	Status pill and agent pointer, in a borderless, non-activating, click-through `NSPanel` run by a helper process per controlled window.	gui_harness/adapters/mac_indicator.py, started by `_WindowControlIndicator` in `mac_window.py`. OpenProgram ships it through program-patches/gui_harness.patch on top of the pinned Harness commit.
Webpage tab in the desktop App	Agent pointer in a transparent, unfocusable, click-through window above the page.	apps/desktop/main/action-cues.js places the window; menu-overlay/action-cue renders it.
Chat preview (PiP) and the web build	Agent pointer in a `pointer-events: none` layer over the preview image.	`ActionCueTravel` in browser-control-bar.tsx.

All three web surfaces draw the same component, AgentCursor. The webpage status (Active, Pausing…, Paused) stays in the page toolbar and the floating control described in GUI Agent. A webpage gets no second status pill.

Status pill

Position. Top centre of the controlled window, 6pt below its top edge, where most apps keep only a title or an empty title bar. It moves with the window and is clipped to it.
Shape. 26pt tall, fully rounded. Its width fits the text and never exceeds the window width minus 24pt; longer text is truncated with an ellipsis.
Surface. Neutral dark glass: rgba(28,28,33,.86) with a 0.5pt white edge at 10% and a soft 8pt shadow. It stays readable on light and dark apps.
Text. “OpenProgram is using WeChat”, or OpenProgram 正在操作 WeChat when the system's first preferred language is Chinese. The text names the app; it shows no window id or process id. The completed state reads “OpenProgram finished with WeChat” for at most 4 seconds.
Live dot. A 7pt dot in the accent colour, pulsing between full and 35% opacity every 0.9 s while the agent is in control. In the completed state the dot turns grey and stops pulsing.

Agent pointer

Glyph. A classic arrow about 22pt tall, a little larger than the system pointer so the two are not confused. It is filled with the accent colour, has a 1.5pt white keyline and a 2.5pt drop shadow. There is no ring around it.
Hot spot. The arrow's tip is the centre of its 42pt (native) or 40px (web) box, and the box is centred on the action point. For a native action the point is the centre of the control's bounds before the action.
Tag. A small accent label just below and right of the arrow names the action: Click, Select, Type or Scroll, or the Chinese equivalents. It never contains the typed text. The webpage pointer has no tag; the page toolbar's operation history names the action.
Parked. Before the first action, the native pointer rests at the window centre at 60% opacity, so the window visibly belongs to the agent before anything happens.
Travel. A new action point is reached by gliding with an ease-in-out curve over 0.18–0.42 s, depending on distance. The webpage pointer keeps its existing 80–280 ms ease-out travel. If the window moves mid-flight, the glide retargets.
Click feedback. After arriving at a press or selection, a ring 32pt across expands from 35% to 135% and fades out in 0.45 s. Typing and scrolling have no ripple.
Restore. A later step on the same window restores the last pointer in place. Restoring does not replay the travel or the ripple.
Reduced motion. When the system asks for reduced motion, the pointer jumps instead of gliding, and the ripple and the dot pulse are off.

Accent colour

The product is moving off the earlier orange and terracotta accent toward its own palette around deep indigo, violet and teal, which is being designed separately. The overlay therefore uses a cool indigo-violet, #5b5bd6, as a placeholder. That value lives in exactly two places, which must stay equal:

--agent-accent on :root in apps/web/app/styles/base.css. Every theme shares it.
ACCENT in mac_indicator.py, as sRGB fractions.

Only the pointer fill, its ripple, its tag and the live dot use the accent. The pill, its text and the keylines are neutral, so swapping in the final brand accent changes nothing else.

Safety rules

No input. The native panel sets ignoresMouseEvents and is non-activating. The webpage cue window ignores the mouse and cannot take focus. The preview layer is pointer-events: none. None of them moves the system pointer.
Always visible while in control. The helper runs for the whole window session and draws the pill even when other windows cover the target. Minimizing or closing the target hides it. Failure, cancellation or lost window identity closes it at once. Only a normal step exit shows the completed state.
One owner per window. Each controlled window has its own helper and panel. Two sessions on two windows each show their own pill and pointer; overlays never mask each other.
Display only. Drawing failures are logged as warnings. They never change an action's result and never fall back to foreground input.

Relation to the GUI Agent design

The GUI Agent design sets the rules this page follows: the indicator is non-activating and ignores the mouse, it follows the window and stays visible under occlusion, clicks show a brief ring, cues belong to the target and never move the user's pointer, and reduced motion uses static marks. The earlier rendering kept those rules but not its look:

Aspect	Earlier rendering	Now
Position	Window's upper-left corner, over the window's own controls.	Top centre of the window, in the title-bar area.
Wording	`控制中 · WeChat · 34327` (“In control · WeChat · 34327”): Chinese only, with the window id, which reads as a process id.	“OpenProgram is using WeChat” in the system language; no ids.
Size	Fixed 224 × 28pt badge; long names were truncated.	26pt tall, as wide as its text.
Colour	Saturated purple fill for the badge and the marker.	Dark neutral glass; the accent only on the pointer and the dot.
Pointer	42pt purple ring with a white SF Symbol inside, changing per action.	Accent arrow with its tip on the point, plus a small action tag.
Motion	The marker jumped between points.	Eased glide; parked pointer before the first action.
Click feedback	None; the ring was permanent.	One brief ripple after a press or selection.
Stopping	Nothing on the overlay.	Nothing on the overlay, by design (below).

Where this page departs from the request it answers

Top of the window, not the top of the screen. Each controlled window owns its overlay, and two sessions may control two windows at once. A pill pinned to the top of the screen would stack with other sessions' pills, could land on a different display from the window, and would not show which window is controlled. The top centre of the window keeps the pill at the top and tied to its target.
No glow border around the native window. The GUI Agent design keeps status edges and borders on OpenProgram's own surfaces (the webpage tab and the chat preview, which already have a status edge) and does not draw borders on other apps' windows. The pill and the pointer are enough to show control.
No Stop button on the native pill. The pill must ignore the mouse. A clickable area there would sit over the controlled app and catch clicks meant for it, and the helper has no checked channel back to the run. Stopping stays in OpenProgram: the conversation's stop control and the Running panel. The webpage floating control keeps its own Pause control, because that control is drawn over OpenProgram's own page view.

Implementation status

Implemented as described for the native helper (through the Harness source patch) and for all three web surfaces. The native panel has been rendered with real AppKit over a desktop window and captured from the screen: the pill, the arrow, the tag and the completed state draw as specified. It has not yet been accepted in the installed App while driving a real app such as WeChat. The ripple and the glide have been checked only through unit tests of the motion rules (tests/test_indicator_presentation.py in the Harness), not by watching them on screen.