GUI agent — call structure & context flow#

This document describes how context flows through gui_agent under the expose / render_range default semantics, and why each @agentic_function's decorator arguments are set the way they are.

References:

• Decorator semantics: agentic-programming/function-metadata.md
• render_context implementation: openprogram/context/nodes.py
• Code: openprogram/functions/agentics/GUI-Agent-Harness/gui_harness/

1. Call structure#

gui_agent(task)                   ← top-level @agentic_function (no render_range)
  └─ loop N times:
     gui_step(task, feedback)     ← @agentic_function orchestration (no runtime.exec)
       ├─ observe()               Python: screenshot + detect components + read state
       ├─ verify_step(...)        @agentic_function LLM leaf
       ├─ plan_next_action(...)   @agentic_function LLM leaf
       └─ dispatch_action(...)    Python: run the action plan picked
     ↓ return dict(goal, action, target, success, error, ...)
     ↓ passed in as the next gui_step's feedback
  └─ conclusion(task, ...)        @agentic_function LLM leaf

gui_step doesn't call runtime.exec directly; its job is to chain the four stages together. All LLM calls happen inside the three leaf functions verify_step / plan_next_action / conclusion.

2. render_range config per function#

The helper leaves (the ones in component_memory, learn, observe, general_action) all use {"callers": 0} — they are independent judges whose input is fully supplied via content=[...] and need no conversation context.

3. Key design points#

What plan_next_action sees#

From the perspective of plan step #5, the inputs to render_context:

• frame_entry_seq = the seq of the plan_next_action #5 code node
• head_seq = the current max seq in the DAG
• render_range = None (use defaults callers=None, subcalls=-1)

Walking render_context:

• pre-frame = all nodes with seq ≤ frame_entry_seq = the top-level chat user message, gui_agent's code node, the code nodes of gui_step #1..#4 (with their io), this round's gui_step #5 verify_step code node (with io), and the node for the observe Python result (if it goes through the DAG)
• in-frame = the nodes inside plan_next_action #5's own frame = empty (the leaf hasn't issued exec yet)
• pre-frame is not truncated; in-frame is not truncated (nothing to truncate)
• expose filtering: each gui_step is expose="io", so the gui_step io nodes are kept while the LLM calls of verify_step / plan_next_action inside gui_step are hidden

So plan_next_action #5's prompt ends up with:

user: <original task>
... prior chat history ...
[gui_step #1] input={...} output={"goal": "open Firefox", "action": "click", "target": "Firefox icon", "success": true}
[gui_step #2] input={...} output={"goal": "open url bar", "action": "click", ...}
[gui_step #3] input={...} output={...}
[gui_step #4] input={...} output={...}
[verify_step #5] input={...} output={"step_succeeded": true, "observation": ...}
[plan_next_action #5] input={...}   ← itself

The planner can therefore see "what the first four steps did, and how the most recent verify judged it" and make a non-repeating next-step decision.

Why verify_step is the opposite#

verify_step is a passive judgment: "did the thing the previous step did succeed, as seen from the current screenshot?". Its evidence is entirely local:

• the previous step's feedback dict (goal/action/target/success/error) — already pushed in via content=[feedback_text]
• the current screenshot — already pushed in via content=[{"type": "image", ...}]
• the components detected this step — already pushed in via content=[component_info]

Letting it see the full chat + gui_step chain would instead introduce noise that biases this simple "did the last step succeed" judgment with the macro narrative. Hence the explicit callers=0 wall.

conclusion, same reasoning#

The summary should state "what is visibly on the final screen", not narrate "step 3 did this, step 7 did that". callers=0 plus a hard constraint in the prompt to "use the concrete text visible on screen".

Considered and rejected: walling every function off#

The alternative is a full wall everywhere — {"callers": 0, "subcalls": 0} on every function — with a feedback dict built explicitly in Python and threaded down level by level. It is rejected because the planner then sees only the previous step's feedback rather than the full trace, and repeatedly performs the same action.

Letting the planner read history through the DAG's default behaviour removes the explicit accumulation entirely, and only the leaves that genuinely need isolation (verify / conclusion / tool-style judges) carry an explicit callers=0. It also means there is no "top level is a special case" branch: the gui_agent top level and the gui_step inside it go through exactly the same render_context code path.

A separate "collapsed" exposure mode is likewise unnecessary. expose="io" means the frame's own input/output is exposed and the LLM inside the frame is hidden, and a nested sub-function decides whether its own io is exposed through its own expose — so there is no in-between case of "expose io while hiding nested grandchildren" left to cover.

4. What to watch#

• For long tasks, plan_next_action's prompt may grow large because the gui_step chain gets long. Under token pressure, give plan_next_action or gui_agent a render_range={"callers": N} to explicitly truncate to the most recent N gui_steps
• A screenshot is an image node; N large images will blow up the context. We may need a "compress old screenshots into text descriptions" path, but that's a prompt-layer optimization, not a render_range-layer concern