Nested Agentic LLM streaming

0. Current state and gaps

1. End-to-end product behavior

1.1 ExecutionStrip cards (TreeStep · LLM rows)

• Open node: an LLM row appears in the tree, running; the secondary note may still show a prompt summary (see §8 / §12).
• tokens: the main content is a continuously growing reply preview (text channel). Thinking defaults to a collapsed “Thinking…” label or last-line summary, consistent with top-level ThinkingStep.
• Close node: status becomes completed/error/cancelled; the note settles to → short(final) plus duration. Clear the stream buffer and return authority to tree-node output/raw_reply.
• tree_update: still drives structure (child additions/removals and status changes); streamed text does not depend on the 1.2s poll and is driven by llm_delta.

1.2 Right-side details panel

• Selecting a running LLM node: Prompt still reads params._content; Reply prioritizes the full FE buffer with scrolling, falling back to tree fields if no buffer exists.
• After completion: Reply = persisted raw_reply/output; Thinking, if present, comes from metadata.blocks.

1.3 Multiple parallel LLMs

Multiple LLM children of one RuntimeBlock may run concurrently. Every node_id has an independent buffer and seq space; UI rows grow independently without overwriting each other.

1.4 Cancellation

User Stop / execution.cancel → node status=cancelled; retain generated partial text as visible, persisted output; clear the buffer; show a “Cancelled” badge and reject further deltas for that node.

1.5 Reconnection / refresh

• After refresh, read the running LLM node’s throttled partial output from SessionDB (§5); the card immediately shows existing text and running state.
• If the run remains active, WS recovery resumes new llm_delta / tree_update without requiring historical-delta replay.
• If the worker died, streaming-resume’s abort sweep marks the node aborted; UI shows partial output and aborted.

1.6 In-process and subprocess

User experience is identical. Only backend bridging differs (§7). Mode detection is diagnostic/logging only and does not branch frontend logic.

2. Complete event protocol

All events remain within the existing top-level envelope:

{
  "type": "chat_response",
  "data": { /* see below */ }
}

2.1 Add data.type = "llm_delta"

At the same level as stream_event / tree_update (nested LLM output is not top-level assistant-bubble text, so do not use inner event.type:"text" or accidentally write to reply.content).

// text channel
{
  "type": "chat_response",
  "data": {
    "type": "llm_delta",
    "session_id": "sess_7c2a",
    "msg_id": "runtime_a1b2",          // RuntimeBlock / live_progress reference (also called runtime_id)
    "node_id": "call_9f3e",            // DAG LLM Call.id (TreeStep aligns by path/id)
    "channel": "text",                 // "text" | "thinking"
    "delta": "Hello",
    "seq": 12                          // Optional; monotonically increasing within node_id+channel
  }
}

// thinking channel
{
  "type": "chat_response",
  "data": {
    "type": "llm_delta",
    "session_id": "sess_7c2a",
    "msg_id": "runtime_a1b2",
    "node_id": "call_9f3e",
    "channel": "thinking",
    "delta": "First, decompose the requirements…",
    "seq": 3
  }
}

2.2 Related existing events (field names unchanged)

// Shortly after node opening / structural change (poll or forced flush)
{
  "type": "chat_response",
  "data": {
    "type": "tree_update",
    "session_id": "sess_7c2a",
    "msg_id": "runtime_a1b2",
    "function": "gui_agent",
    "tree": {
      "path": "…/gui_agent",
      "name": "gui_agent",
      "status": "running",
      "children": [{
        "path": "call_9f3e",
        "name": "gpt-5",
        "node_type": "exec",
        "status": "running",
        "params": { "_content": "Summarize the page…" },
        "output": "",
        "raw_reply": ""
      }]
    }
  }
}

// Final tree_update after closure (root/child status=completed|error|cancelled)
// child.output / raw_reply = full text; FE treats it as authoritative and clears buffers

2.3 Lifecycle events (optional, but adopted here)

2.4 Ordering, coalescing, and backpressure

• Ordering: for the same (node_id, channel), if seq is present, append by seq; otherwise use arrival order. Drop out-of-order lower seq values rather than reordering history.
• Server coalescing: combine deltas within a 16–32ms window for the same node before broadcast (still one llm_delta with concatenated delta). Never coalesce across nodes or channels.
• FE coalesce: reuse the top-level rAF strategy—Map<nodeId, PartialStream> writes occur immediately, while DOM/store stamps update at most once per frame.
• Backpressure: bound the WS send queue (suggested: 512 frames per session). On overflow, discard intermediate text deltas first while retaining the latest snapshot frame (periodic full-snapshot llm_delta, using delta since the prior snapshot or a separate snapshot field). V1 skips intermediates on overflow but guarantees a complete final close tree. Thinking follows the same policy with more aggressive dropping.
• Relationship to tree_update: deltas never write into the tree subtree. §5 throttles partial text into nodes, which the next tree_update then includes.

3. Sequence diagram

sequenceDiagram
  participant AF as Agent method
  participant RT as Runtime.exec(llm)
  participant DAG as SessionStore/DAG
  participant BR as Broadcast(WS)
  participant FE as Web FE

  AF->>RT: llm(prompt)
  RT->>DAG: _open_model_call_node (running)
  Note over BR,FE: tree_update (child appears, output="")
  BR-->>FE: tree_update
  FE->>FE: TreeStep running; create PartialStream[nodeId]

  loop tokens
    RT->>BR: llm_delta(node_id, channel, delta, seq?)
    BR-->>FE: llm_delta
    FE->>FE: buffer.append; rAF updates note/details
  end

  RT->>DAG: _close_model_call_node(output, status)
  Note over BR,FE: terminal tree_update (output=full text)
  BR-->>FE: tree_update
  FE->>FE: replace with tree text; clear PartialStream[nodeId]

4. FE state architecture

4.1 Recommended store shape

// Alongside the existing Zustand session store, or as a new slice in it
type PartialStream = {
  text: string;
  thinking: string;
  lastSeqText?: number;
  lastSeqThinking?: number;
  updatedAt: number;
};

llmStreamsBySession: {
  [sessionId: string]: {
    // key = DAG node id (aligned with tree child.path / Call.id)
    [nodeId: string]: PartialStream;
  }
}

• Do not write high-frequency deltas into message-row contextTree (avoid deep-cloning the entire tree).
• TreeStep / DetailPanel reads buffers through (sessionId, node.path|id). With a buffer and running node, display streaming output; otherwise read note/raw_reply from tree fields.
• Isolate this from top-level pendingDeltas (assistant reply): nested LLMs never write to reply.content.

4.2 TreeStep selection rules

4.3 Clearing rules

• Node tree status ∈ {completed, error, cancelled, interrupted, aborted}
• or clearSessionByMsgId / session_loaded changes session
• or RuntimeBlock unmounts with no running children
• Before clearing, if the tree has no final output yet, backfill once from the buffer to avoid losing the last tokens in a race.

5. Persistence policy (decision within this design)

• Live authority: FE PartialStream (lowest latency).
• Recoverable authority: throttled node output/raw_reply snapshots plus terminal full text.
• Do not create child nodes or an append-only token table for every delta (non-goal).
• At the same level as operations/streaming-resume alignment: placeholder + last_update_at; worker-restart abort sweeps also cover nested LLM nodes.

6. Failure / cancellation / error UX

7. In-process vs Subprocess

Mode detection (diagnostics only)

• Parent: process_runner checks whether this execution_id has a live Process.
• Child: whether QueueTransport / a subprocess event_queue is installed.
• Log field agentic_isolation: "inprocess"|"subprocess"; no frontend branching.

8. UI wording and behavior (default proposal)

• Primary card content: while streaming, show growing reply (text); after completion, an → summary.
• Prompt _content: remains secondary by default. While running without tokens, show a short prompt; after tokens arrive, reply takes precedence. Full prompt remains available in details.
• Thinking: collapsed by default, titled “Thinking”; show the latest line while running; click to expand full text in details or an inline fold.
• Do not copy nested LLM text into the outer assistant bubble.

9. Acceptance criteria (complete feature)

9.1 Manual

1. A single llm() inside an agentic function: node opens → card grows token by token → closure matches raw_reply and removes the buffer.
2. Dual thinking/text channels: thinking collapses while main text grows.
3. Two parallel LLMs or consecutive loop calls: independent growing rows, no cross-contamination.
4. Stop: deltas stop immediately; partial remains; status is cancelled.
5. Mid-run refresh: partial is visible; new deltas resume if still running.
6. Subprocess tools such as gui_agent provide the same UX as in-process execution.
7. Details-panel Reply matches the card.

9.2 Suggested automation

• Unit: envelope shape (node_id/channel/seq), lower out-of-order seq rejection, and no cross-node coalescing.
• Unit: subprocess queue forwards llm_delta using mock Process/Queue.
• Component: TreeStep running+PartialStream snapshot; terminal buffer clearing.
• Contract: llm_delta in the chat-stream reducer does not contaminate reply.content.
• Optional E2E: fake provider emits fixed deltas; assert DOM text.

10. Non-goals (v1)

• Token-level audit logs or replayable delta recordings.
• Changing top-level chat stream_event.text semantics or complete TUI parity (TUI may later read and ignore llm_delta).
• Per-field streaming highlights for structured output.
• Multimodal image deltas.
• Cross-session LLM stream subscriptions.
• Replacing tree_update with llm_delta.

11. Implementation phases (ordered delivery of one complete design, not a minimal release followed by reconsideration)

1. Protocol and backend emission: in the _active_llm_node_id context, Runtime converts on_stream to llm_delta; add broadcast helpers and envelope unit tests.
2. Subprocess bridge: confirm queue forwarding, add isolation-mode diagnostics, and test both paths.
3. Throttled persistence: partial output/thinking on running nodes, including close-race tests.
4. FE store + chat-stream: llmStreams slice, reducer, clearing rules, and no reply contamination.
5. TreeStep + Detail: selection rules, thinking fold, and styling.
6. Cancellation/reconnection/parallelism: complete §6 and §1 behavior and sign off the manual checklist.
7. Documentation and check scripts: mark this page adopted; use check-*.mjs to preserve UI contracts where necessary.

12. Open questions (user decisions required · at most four)

Choices are recorded in localStorage under key op.design.llmStream.decisions for review comparison, not as a formal vote.

Interactive demonstration · Card lifecycle