Async Task Lifecycle#

This document describes the design of asynchronous sub-agent invocation: an explicit task entity, a background worker pool, and a queryable, cancellable lifecycle. The surface aligns with Claude Code's TaskCreate / TaskList / TaskGet / TaskUpdate / TaskStop.

The chassis it builds on already exists in the WebUI: one worker thread per session (_execute_in_context), a _cancel_events dict (_pause_stop.py), and a _running_tasks dict for the UI spinner. run_agent_turn (openprogram/agent/sub_agent_run.py) is the synchronous path that the /task tool, the /spawn WS action, and _merge.process_merge_turn all reuse; the task abstraction layers on top of it.

Part 1. The dimensions of the task lifecycle#

A full spawn / list / get / cancel lifecycle has to settle the 15 points below. Part 2 walks each scenario through the same checklist.

D1. What the Task entity stores#

A Task carries at least:

• job_id: a stable id returned immediately on spawn (independent of user_msg_id / assistant_msg_id)
• subject: a one-line summary (for list / panel display)
• description: the full prompt (reused as prompt when invoking the agent)
• agent_id: which profile to run
• status: the status enum from D2
• created_at / queued_at / started_at / completed_at: timestamps
• parent_session_id: which session the task runs on (a task is always bound to one session)
• parent_job_id: the task that spawned this task (None for a top-level user-spawn)
• parent_msg_id: the user / assistant msg id that triggered the spawn (used to hang the attach card back onto it)
• context_mode: inherit / clean
• head_id: the assistant msg id where the sub-agent lands after the task completes (None while running)
• result_text: the sub-agent's final reply (None while running)
• error: the error string on failure
• cancel_requested_at: the time the cancel signal was written
• attempt: the retry count (fixed at 0 in the first version)

Runtime objects like cancel_event / future stay out of the entity: the entity only describes "what," while "how to cancel / wait" is resolved by an internal map in the runner (see D5).

D2. State machine#

pending → queued → running → completed
                          ↘ cancelled
                          ↘ errored

• pending: the spawn API received the request, the entity was written to the persistence layer, but it hasn't been scheduled onto the worker pool yet.
• queued: handed to the ThreadPoolExecutor but not yet picked up (all workers busy).
• running: a worker picked it up and started process_user_turn.
• completed: the sub-agent returned normally, and head_id and result_text are written.
• cancelled: the cancel event was consumed and the worker exited (partial output may have landed).
• errored: the worker threw an exception, or a task was unfinished when recovering from a process crash.

Every state transition updates the corresponding timestamp and is irreversible. Jumping straight from pending → cancelled, skipping queued / running, is also allowed (the user stops before the worker picks it up).

D3. Worker pool model#

The pool is a concurrent.futures.ThreadPoolExecutor, isomorphic to the existing daemon-thread model of _execute_in_context, and it avoids async-coloring the whole codebase. Rationale:

• process_user_turn already opens its own asyncio.new_event_loop() internally to run the agent loop, so wrapping it in an outer thread causes no double-event-loop conflict.
• BashTool and file IO are blocking calls; the thread model costs nothing for them.
• The concurrency cap is configured via the OPENPROGRAM_JOB_WORKERS environment variable, default 4. Tasks over the cap stay queued, FIFO-fair.
• Backpressure: no per-session limit (tasks are siblings, and the serial spinners shown in the UI don't affect this); a global pool cap is enough. Priority can come later if needed.

D4. Persistence#

Persistence reuses the existing session-git meta: alongside meta.json in the session repo sits a jobs.json with structure {job_id: TaskRow}. Each task state transition calls commit_turn("task: ...") once, landing it in git history.

Rationale:

• A task is always bound to a single session (D6), so storing it under the session directory partitions it naturally.
• Sessions already follow git-as-truth; putting task state into git history actually helps debug "why is the task stuck."
• No new SQLite table or new DB file is needed.

Non-recoverable policy: after a process crash, all tasks with status=running / queued / pending are marked errored at startup (error="worker died before completion"). The reasoning: the LLM call was already issued and can't be retrieved, so making the user re-spawn is the cleanest semantics.

D5. Cancel signal propagation#

Each running task has a threading.Event inside the runner (not stored in the entity). When the Cancel API fires:

1. Write cancel_requested_at to the entity and transition status to cancelled (if still queued / pending), or keep it running and wait for the worker to exit naturally.
2. cancel_event.set() — via the contract already defined by _pause_stop.register_cancel_event, this propagates to process_user_turn(cancel_event=...).
3. process_user_turn already bridges cancel_event into an asyncio.Event (the agent_loop call), and the LLM provider stream checks for the cancel on the next chunk and then breaks.
4. BashTool / other subprocesses: reuse _pause_stop.kill_active_runtime (already exists). Cancellation at the tool layer is cooperative: each @agentic_function's pre-invocation hook checks is_cancelled (already exists), and the next tool-call entry raises CancelledError.
5. Fallback timeout: if the worker still hasn't exited 30 seconds after the cancel, the runner marks the entity cancelled (error="cancel timed out, worker may be stuck") and detaches the worker thread (no hard kill; let GC handle it).

A tool's own atomic operation (e.g. a Write halfway through) is not interrupted; it waits for the current atomic operation to finish before exiting.

D6. The relationship between Task and session#

A task always runs on one parent session. Cross-session is conceptually equivalent to merge / attach (already exists) and is out of scope for tasks.

task.parent_session_id is exactly the one the sub-agent uses in process_user_turn(session_id=...) — consistent with how run_agent_turn behaves. The sub-agent's output lands as a branch of that session (or a new root, depending on context_mode), so the session repo holds both the task entity and the task's product, which is self-consistent.

D7. The relationship between Task and sub-agent#

All spawns go through the task entity, so /task does not block synchronously. The semantics:

• The agent-facing tool agent(prompt, ...) is async — it returns job_id immediately without blocking the main conversation. Once the LLM has the id, it can choose to do other things, or immediately call job_output(job_id) for synchronous semantics (D15).
• Compatibility flag: /task --sync (or the tool's foreground default, run_in_background=False) wraps a spawn → await at the tool layer, returning the result synchronously, transparently to the LLM.
• _task_impl keeps its run_agent_turn call, but routes through the runner entry point.

/spawn (the user typing /spawn label: prompt in chat) goes through the same spawn API; the only difference is the caller is a WS handler rather than the LLM.

D8. The relationship between Task and ContextCommit#

The attach pointer (the function="attach" node) is written by _run_spawn / _task_impl:

• On spawn, a placeholder attach card is written immediately (function="attach", extra.attach.job_id = <job_id>, extra.attach.status = "running"), content="(running)", with source_commit_id left empty.
• When the task completes, the runner updates the same attach card node: fill in head_id / source_commit_id, replace content with final_text, and change status to completed / cancelled / errored.
• When the generator sees an attach node with status="running", it skips expansion (does not enter commit items) and only shows the card placeholder in the UI. When it sees status="completed", it follows the existing attach-expansion path (see scenario B in context.md).

This way, when the LLM triggers a new turn before the task finishes, it won't see half-baked attach content, but the user can still see the spinner.

D9. WS API#

Four ws actions (following the existing naming in ws_actions/):

• spawn_job — {action, session_id, prompt, description, agent_id?, context?, wait?} → immediately returns {job_id, status, parent_msg_id}.
• list_jobs — {action, session_id?, status_filter?, limit?} → {tasks: [...]}. No session_id means list globally.
• get_job — {action, job_id} → single entity + current head_id / partial result (if running).
• cancel_job — {action, job_id} → {job_id, status} (returns the current status synchronously; does not wait for the worker to exit).

Broadcast events:

• task_created, job_status (queued / running / completed / cancelled / errored), task_progress (optional; token progress can be added in the future).
• The existing running_task broadcast is reused with an added job_id field (UI compatibility).

D10. Agent-facing tools#

The agent uses a trio:

• agent(prompt, description, agent_id?, context?, run_in_background=False) → returns the final result (foreground default) or job_id (run_in_background=True). Wraps runner.submit(...).
• job_output(job_id, block=True, timeout=30000) → waits (timeout in ms, capped at 600000; Claude Code's TaskOutput shape) until completed / cancelled / timeout, returning the reply text plus terminal status; block=false peeks at the current status immediately. list_jobs() lists the session's background tasks (id, status, subject) so the LLM can keep track of parallel work.
• job_stop(job_id, reason?) → {ok, status}.

The variant await_tasks([id1, id2, ...], mode="all"|"any", timeout) is used for plan mode's wait_all (D14). The trio plus wait_all makes four tools, but plan-mode does not expose wait_all to ordinary agents — an ordinary agent can only await a single id, which keeps it from being misused.

D11. UI representation#

• A Tasks tab in the right-side panel (right next to the existing Branches / Context Commits panel). It lists all task entities for the current session: spinner + subject + status + creation time.
• Open a task: jump to its corresponding attach card (the placeholder already exists in chat) → check out its head_id branch (if completed).
• Each attach card carries its own status badge: running / done / cancelled / error. Once complete, behavior matches an ordinary attach card.
• The global sidebar shows a total count badge (how many running): reuse the existing running_task mechanism.

D12. Error recovery#

• Worker throws an exception: the runner catches it → status errored → fill the error field with f"{type}: {msg}" → the attach card status updates in sync → broadcast a job_status event.
• Main process crash: see D4. The startup hook scans jobs.json and marks all non-terminal states as errored.
• Pool shutdown: wait 5 seconds when the process closes; tasks that time out are marked errored ("worker pool shutdown").
• Re-spawning the same job_id is not allowed (the spawn API is idempotent on job_id, but by default mints a new id).

D13. Test boundaries#

Unit tests do not run a real LLM. The seams needed:

• runner.submit(req, *, sync_fn=...) — replace process_user_turn with a fake synchronous function (which receives cancel_event and returns a fake TurnResult).
• The entity store uses an in-memory MockTaskStore (a dict rather than git, but the same interface).
• State machine test matrix: one case per legal transition, one reject case per illegal transition.

Integration tests cover: the spawn → await → completed path, spawn → cancel mid-flight, spawn → worker raise → errored, multi-task FIFO queuing, and crash recovery marking running as errored.

D14. Plan mode integration#

The plan agent produces a plan (a set of sub-task specs); when exit_plan_mode completes:

• The plan tool / executor agent receives the spec list (each containing description + prompt + an optional agent_id).
• Call spawn_job in sequence to get N task_ids.
• Call await_tasks(ids, mode="all") to wait for all to complete.
• Once the results arrive, the executor synthesizes them (writing a user-visible summary, or automatically triggering merge).

The concurrency cap is the pool size from D3 — a plan listing 10 tasks with a pool of 4 will leave 6 stuck in queued, shown queued in the UI.

D15. Backward compatibility#

/task remains the user-input entry point in chat with unchanged behavior (the user sees an attach card + the full result), but it routes through the task entity underneath.

The LLM-facing agent(prompt, ...) tool provides two semantics:

• run_in_background=False (default): internally spawn + await + return result_text as the return value to the LLM. From the LLM's perspective, identical to the synchronous tool.
• run_in_background=True: returns job_id, and the LLM decides when to await on its own. Used by new code / plan mode.

The toggle lives in the tool signature, so prompts written against the synchronous tool run unchanged. The tool-capability description broadcast to the LLM explains both modes + recommended usage.

Part 2. Walking each scenario through the dimensions#

Scenario A: a single sync /task#

The most common case: the LLM calls agent(prompt="probe X"), expecting to block and get the result. Behavior matches the synchronous tool, and it routes through the task entity underneath.

Scenario B: a single async task (the agent actively chooses async)#

The LLM decides to do a long job, first spawns to get an id, then awaits or cancels later.

Scenario C: N concurrent async tasks (plan mode)#

The plan agent lists 5 research tasks, spawns 5 tasks, and calls await_tasks(ids, mode="all") to wait for them all.

Scenario D: a long-running task + cancel#

The agent spawns a 30-minute deep research task; 10 minutes in, the user clicks Stop in the UI or the agent decides to abort.

Part 3. Key invariants#

1. Terminal state is unique: every task entity must end in exactly one of completed / cancelled / errored, with no permanent running (pool shutdown / crash recovery must transition it to errored).
2. State is monotonic: each edge of the state machine is traversed only once. Once completed, it never goes → cancelled, and running → pending is not allowed.
3. Cancel is reachable: after the cancel API returns, the entity must reach a cancelled / errored terminal state within 30 seconds (the forced-timeout fallback).
4. Placeholder consistency: the attach card written on spawn stays in sync with the entity's state; every entity state change must update the card in sync.
5. Persistence is idempotent: after a crash reload, the state of unfinished tasks is determined to be errored; the same job_id is never revived.
6. Session binding is immutable: after a task is created, parent_session_id doesn't change; the same task doesn't run on two sessions.
7. Concurrency safety: the runner's _tasks + _cancel_events maps are held under a lock throughout; state reads and writes don't race.
8. Test controllability: the runner must accept an injectable sync_fn / store, so the full state machine can be run without depending on a real LLM.

Part 4. Out of scope for this design#

• Cross-process tasks: all workers are in the main process. Distributed / multi-host is left for later and requires a message broker.
• Task priority / SLA: FIFO is enough, with no high-priority preemption. A priority queue can be added later as needed.
• Resume / continuation: cancelled / errored tasks can't "pick up where they left off." A user retry equals spawning a new task.
• Task retry policy: the runner does not auto-retry; the upper-layer agent / plan decides for itself.
• Cross-session tasks: a task binds to only one session. Use attach / merge for cross-session.
• DAG-shaped task dependencies: await_tasks(mode="all"|"any") is already enough for plan mode; an explicit DAG / pipeline is left for later.
• Streaming subscription of task output: the first version only retrieves final_text on task completion. Subscribing to the stream mid-flight (so the parent agent sees the sub-agent thinking out loud) is left for later.
• Resource quotas: per-user concurrent task count / token caps are out of scope for this design and require a multi-tenant model first.

Appendix: Implementation Status#

None of the above has landed yet. The runtime today has no task entity — only the in-memory _running_tasks dict driving the spinner, which is lost on refresh — no explicit state machine (a function returning is what "done" means), no query interface, and no plan-mode concurrency, since /task is synchronous and serial. Cancel exists at whole-session granularity (_cancel_events) but not per task; the worker model is one daemon thread per session (_execute_in_context) rather than a shared pool; the attach card is written only on completion; the agent has only the synchronous task tool; and there is no Tasks panel in the UI.

The landing order, in dependency order: