Restart recovery and Goal design for the complete session runtime

Design question: after an App or worker restart, how do previously running sessions and Goals automatically continue, restoring subtasks, approvals, context, and usage without repeating unknown writes or resuming work the user explicitly stopped?

For framework development and product acceptance. This page defines the target design; implementation status is collected at the end. Scope includes Web, TUI, ACP, background Jobs, and authorized subtasks. Goals use the same chat execution layer. Retain the Python Workflow API without duplicating the execution engine, introducing a general scheduler, or promising exactly-once execution for arbitrary external operations.

Verified current behavior · Automatic recovery contract · Complete recovery-state matrix · Failure acceptance matrix

1. 1. Save the objectiveOriginal objective, completion conditions, permission scope, and explicit budget.
2. 2. Work through ordinary chatThe agent maintains Todo and uses existing tools and approvals.
3. 3. Check at turn completionSettle usage and decide whether to continue, wait, pause, or verify acceptance.
4. 4. Accept completionSubmit achieved once evidence supports every condition; otherwise continue working.

There is no default round cap. Round count is diagnostic information, not a substitute for progress or stopping conditions. Retain the short state tools create_goal/get_goal/update_goal; they only read and write objectives, rather than executing the entire task within one function call.

Five harnesses: what to adopt and what not to duplicate

The following are OpenProgram design choices based on these implementations, not claims that the five projects provide identical safety, acceptance, or cost guarantees. Reference source is pinned to inspected versions; external project tests were not run.

Verifiable Codex boundaries:Official App Server documentationdistinguishes thread/resume, turn/start, and persistent Goals;public extension sourceloads active Goals when resuming threads and attempts continuation while idle. This does not establish that arbitrary desktop exits, system restarts, or tool-process deaths can restore original memory or continue original tools. OpenProgram defines automatic continuation while prior run intent remains valid as its own explicit product contract, without relying on that unverified stronger assumption.

Current state: persistence exists, but consistent recovery decisions are missing

Inspection covers the shared entry points above, not individual failure tests for every adapter. A paused UI does not prove history was deleted, and existing records do not prove automatic continuation is possible.

Automatic recovery contract: run intent survives the process

Complete durable run_intent on top of existing admission / command / restart intent rather than adding a parallel scheduler. It records work_id, original input and unconsumed messages, desired_state (run / paused / cancelled / done), latest valid control version, permission scope, objective revision, execution host identity, parent/child relationships, last committed progress, and recovery policy. Acknowledge acceptance to the client only after reliably saving the original user request.

By default, automatically recover previously authorized work whose state remains run, consistently for ordinary sessions and Goals and without requiring the user to reopen the session. Preserve pauses, stops, completion, outstanding approval, and exhausted budgets. Ordinary requests without a Goal continue only until that request finishes, not as indefinite Goals. Background work retains its prior authorization and concurrency quota; restart does not authorize additional subtasks.

Closing a window or reloading the frontend only disconnects the client; “quit and stop all tasks” must persist user stop intent. Ordinary App exit, upgrades, worker crashes, and machine restarts do not implicitly mean user Pause. Graceful shutdown first stops new admission and saves accepted controls and checkpoints, then exits the worker; forced shutdown recovers from the last committed records. Client-only drafts without server acknowledgment must not be represented as submitted requests.

Remove the hidden two-hour recovery expiry. Long downtime preserves run intent, but continuation rechecks directory, host, model/tool availability, credentials, current permissions, and budget. Missing resources or revoked permissions show precise waiting reasons, without switching project, account, or model or expanding permissions automatically. Explicit automatic-recovery disabling remains effective; old paused records with unknown reasons stay paused with an explanation rather than being guessed to have been running.

1. Load facts:Read run intent, user controls, messages, checkpoints, effects, waits, resources, usage, and projection cursors. Read/migration failures show recovery failure without clearing old state.
2. Confirm ownership:Reconnect live executors; exclude dead executors through generation fencing. Only one host recovers each work item. Single-host recovery must not silently become cross-host takeover.
3. Repair context:Merge real results, supply missing tool outputs, and record interruption. Resume from a complete safe checkpoint when available; otherwise start a new turn with the same work_id without re-appending user input.
4. Decide:User stops take priority, followed by approval, permissions, budget, resources, and unknown effects. Runnable work becomes queued; dependency waits become waiting; faults expose retryable reasons. Persist all decisions in the same control service.
5. Admit and continue:Use existing fair scheduling and resource quotas; derive a stable continuation key from work_id + control version + predecessor commit point. Admission persistence and recovery-intent consumption must leave no duplication window; converge cross-database steps through a retryable outbox.
6. Restore the interface:App/Web/TUI reconstruct from canonical snapshots and monotonic event sequences, showing recovery, queueing, working, waiting, or failure; display is not a second control implementation.

Recovery coordination belongs to existing execution control. The server and JobRunner may call the same idempotent entry point but cannot make independent recovery decisions. UI orphan cleanup repairs only derived projections; Goal.after_terminal submits continuation requests only for the current version. Temporary faults use durable retry times and bounded backoff instead of idle model calls. Persistent faults retain pending-recovery records and show the error plus Retry / Pause / End without deleting tasks.

Complete recovery-state matrix

Recovery promises confirmed saved work state, run intent, and reconstructable execution, not arbitrary process-memory snapshots. Unpersisted stream fragments, interactive memory in a dead shell, and receipts lost by external services must be reported accurately, not hidden by clearing tasks or fabricated success.

One chat execution layer, five durable data categories

Continue using existing session, execution, effect, todo, and usage stores without building a second event platform. Cross-database updates use existing retryable notifications and unique operation keys; do not assume cross-database atomic transactions. Derived UI and statistics can be reconstructed and must not overwrite original records.

Goal states remain active / paused / blocked / achieved / cancelled / budget_exhausted. Execution, waiting, and acceptance belong in phase or existing wait records instead of a redundant combined state machine. Retain compatibility reads for paused_recoverable. Revision or permission changes invalidate old acceptance and automatic-continuation reservations; old usage retains its original objective attribution.

Interruption recovery: confirm execution ownership before restoring context

1. Confirm the old executor:Use the existing owner lease / generation; do not write repairs when the lease is uncertain or storage unreadable. A disconnected connection alone does not prove process death. Reject subsequent writes from the old generation.
2. Verify child processes:Reconnect trackable live processes and wait or explicitly cancel; do not treat a running shell as stopped. Preserve unknown-effect records for unreachable remote operations.
3. Choose exactly one recovery mode:Only reconnect a live owner. If death is confirmed and a verifiable safe checkpoint exists, create a new attempt under the same execution. Only when no reliable checkpoint exists, acquire exclusive recovery ownership, fill missing outputs using a stable repair key, end the old turn as interrupted, and admit a new turn. These choices are mutually exclusive and bind to the same durable recovery decision, control version, and continuation key; never resume a checkpoint and start a new turn together. Checkpoints must cover committed effects and satisfy current permission, wait, and safe-point contracts, without replaying unknown writes. Repeating repairs has no side effects; real results take priority and no usage is fabricated. Append late verifiable receipts as reconciliation events that replace synthetic placeholders in subsequent context.
4. Restore run intent:Previously running ordinary requests and active Goals automatically enter unified admission; user-paused work does not resume automatically. Only specific permission, budget, resource, or unknown-operation issues need human intervention; restart itself is not a reason to require manual Resume.
5. Decide tool behavior:New turns know which calls never started and which outcomes are unknown. Old unknown results do not globally block investigation or independent work; related writes retain original approval and recovery restrictions.

Safety boundary: read_only / idempotent / state_probe capabilities come from trusted tool registration and implementation identity saved at execution. Do not accept model claims of read-only behavior or infer them from a name containing read. General shell, third-party MCP, and custom tools default to unknown. Recovery turns related to unknown writes initially allow only trusted reads and state queries. Without reliable relevance determination, do not automatically allow writes; reconcile first or obtain explicit user authorization.

Request-history repair first matches the nearest call by original call ID, then converts provider IDs. Existing real outputs take priority; duplicate outputs use the last saved real record. Missing results get deterministic unknown placeholders. Calls and results are adjacent in the request projection while user-message order remains intact; original history is unchanged. Results without corresponding calls are not sent as valid tool-protocol items. Interrupted responses retain partial text and an interruption marker without reusing incomplete reasoning signatures. Ambiguous duplicate call IDs within one response produce an explicit error rather than guessed execution facts. This conversion grants no execution permission and cannot replace recovery decisions.

Recovery permission checks use the same conversation provenance to scope old executions and invoked Jobs, without expanding to all executions in the invoked session. Only actual built-in read, grep, glob, and list execution functions qualify for read-only investigation; names, acceptEdits flags, and plugin declarations cannot replace implementation identity. For uncertain effects within the applicable execution scope described below, all other operations require one-time approval; bypass, automatic classification, and existing allow rules cannot waive it. Durable approval binds to the new tool call and unknown-effect set; changes to that set require fresh approval and do not erase old effects. Reject execution if recovery state cannot be read. This check is a prerequisite for restricted new-turn recovery; it does not independently change old execution state or enable Resume.

This is neither an exception for one session nor a repair restricted to allowlisted files. Ordinary chat and Goals share the recovery entry point. Old effect audits may remain unresolved while their chat turns end as interrupted; adjust the old constraint that any unresolved effect prevents turn termination accordingly.

Only an explicitly activated Goal produces Goal UI. Starting, running, pausing and verifying do not add transcript cards. At termination, one compact record remains at the ending transcript position. It uses the existing attachment-card styling and opens the existing Goal details dialog in read-only historical mode. Each record carries its own Goal snapshot and transcript anchor; a later Goal cannot replace its contents. The composer continues to show the current Goal only. Canonically identified verifier messages are internal records, not assistant chat prose; ordinary agent replies remain unchanged. No assessment text, evidence identifiers or raw report is appended to chat. Terminal history is saved with the Goal state under optimistic concurrency. The implementation passes independent specification and quality review. Coverage includes terminal snapshot persistence, stable historical anchors, read-only reuse of the Goal dialog, keyboard focus restoration and preservation of ordinary JSON replies. Full verification and installed-App acceptance remain separate release gates.

Todo manages the plan; acceptance manages the objective

At creation, preserve the original objective and extract verifiable requirements; ask the user only when ambiguity would change scope. The working agent maintains Todo for multi-step tasks; simple tasks may omit it. Every necessary requirement has an evidence checkpoint; adding or removing Todo must not implicitly remove original requirements.

Ordinary work turns do not run a second judge. Enter verifying when the worker submits a complete candidate or all required Todo items finish without verification. First check current revision, required Todo, unfinished subtasks, and relevant unknown effects, then run one independent read-only acceptance review. The worker's completion text is not an acceptance fact.

Acceptance returns met / unmet / unknown and evidence references for each requirement. Deterministic requirements inspect actual test receipts, file versions, and runtime state; semantic requirements use independent model review. Tests retain ordinary tool approval and budget rules: calling shell verification does not make it read-only. Missing evidence returns to another work turn rather than fabricated acceptance.

Only the current goal_id + revision + verified result version may submit achieved. Recheck evidence versions and compare the Goal control version before committing. Controlled artifacts use version numbers or content digests; external facts retain verification time and query receipts without claiming atomic verification across all external systems. Completion means conditions held at that verification; it does not automatically monitor files or reopen the Goal afterward. Later edits, new requests, or explicit monitoring require separate acceptance. Human waivers must be explicit objective revisions with reasons; the agent cannot reduce scope itself. Independent acceptance can also err and does not guarantee absolute correctness.

Judge lack of progress by changes in results and evidence, not tool invocation count. Enter blocked only when the same real blocker recurs consecutively and no independent work is available. Waiting with a live handle enters waiting without idle model turns. A fixed total round count is not evidence of blocking or completion.

Waiting uses existing durable waits and execution-terminal notifications, saving work_id / Goal revision, handle, observed version, and unique consumption key. Completion, failure, or timeout notifications admit exactly once only if intent remains run, versions match, and no other current execution exists. Existing reconciliation polls for missing notifications with backoff. Save results arriving during restart, then automatically continue under the same rules. Pause, cancel, or revision invalidates old wakeups without discarding results; query timeout does not imply the external operation stopped.

Cost: attribute per request, settle per receipt

Fix goal_id / revision / turn_id / request_id before dispatch. Ordinary chat after pausing has no old Goal attribution. Subagent and acceptance requests explicitly inherit attribution; count each receipt once rather than adding both parent totals and child details. Retries are new provider attempts and count separately when they incur actual consumption.

Streaming cumulative usage updates a watermark for the same attempt; do not sum cumulative snapshots. Update Goal projections only after durable receipt writes succeed. Temporary failures retry with the same idempotency key, and restart can reconstruct statistics. Late receipts remain attributed to the original Goal even if paused, ended, or edited.

Store known tokens and known prices separately. Pricing-table changes do not rewrite historical quotes. Display estimates, known values, and partial unknowns separately, such as “Known $0.42; usage unknown for 1 additional request.” Preserve historical cursor data without inventing attribution. Accumulate active time from durable start/stop records, excluding pauses and downtime and without resetting on recovery. When an executor disappears, preserve its last heartbeat; crash-discovery time is not stop time. Show the known duration lower bound and unknown interruption interval. Late usage does not restart paused timers. If an explicit time limit cannot be verified, stop new admission without adding a default limit.

Budgets are unlimited by default. With a user-set hard cost budget, unknown prices or unverifiable consumption cannot be treated as available balance; pause new paid admission and explain why. Reserve a maximum request amount when calculable; otherwise do not promise absolute prevention of overspend. Recheck explicit budgets after ledger reload and before execution recovery or acceptance.

Background Jobs save Goal identity at the real active caller's admission boundary, restore it at startup, and clear unrelated temporary attribution. Cross-session calls preserve original caller provenance without creating a cross-session canonical parent. Goal and Job share start/settlement transactions in usage.db; receipts save Job reservation identity so metering can be reconstructed without an in-memory governor after restart. Unknown tokens and unknown costs retain separate reservations; partial cumulative data records only lower bounds. Expired unstarted reservations cannot dispatch; started requests are not automatically released. Hard budgets accept only audited input/output limits, explicit prices, and service tiers and add no default restriction to Goals without budgets.

Controls and interface: show state, reason, and action together

An empty Todo shows no progress placeholder or separator. Backend can_resume / can_verify / can_end and reason_code determine button permissions; the frontend must not infer them from status strings. UI read failures show “State temporarily unavailable” with refresh, not permanently disabled unexplained controls.

User input takes priority over automatic continuation using existing steer/queue semantics. Continuation keys bind goal_id, revision, and the previous committed turn; state checking and admission share version comparisons. Durable state and generation checks block stale actions when pause, edit, explicit cancellation, and admission race.

Implementation sequence and required acceptance

Each stage begins with a public-entry failure reproduction, followed by minimal implementation and regression. Recovery tests cover ordinary chat, not just one specific session. Acceptance distinguishes source, automated tests, local App, and remote release; designs without test or deployment evidence are not marked implemented.

Restart failure acceptance: more than testing Resume once

Create work through public chat, Goal, Job, approval, and control entry points; perform real termination using isolated test storage and non-user-visible process tests. Do not terminate the user's current work to demonstrate recovery. Final visible App acceptance uses only the default instance and first confirms that updating will not overwrite unrelated work.

• Cover frontend reload, normal App exit, graceful worker restart, forced worker termination, downtime beyond the old two-hour window, and another crash during recovery. Test host-restart semantics through process-identity changes; list actual OS reboot separately as unverified and do not reboot the user's machine yourself.
• Interruption points include before/after input acceptance, provider requests, before/after tool dispatch, before/after result commit, parent/child delivery, all Todo complete before acceptance, after completion commit before projection delivery, and before/after usage persistence.
• Recover multiple sessions together to prove fair queueing, shared quotas, one owner per work item, and no duplicate execution from repeated notifications. User Pause / End, permission changes, and new input take priority during recovery; old versions cannot continue.
• Live child processes only reconnect; unknown non-idempotent writes only undergo investigation. Consume accepted approvals once and never approve unaccepted requests automatically. Late results and usage retain their original identity.
• Unavailable databases, broken migrations, missing credentials/directories, changed tool versions, and lost projection events all need explicit states and recoverable actions, without clearing history or leaving unexplained disabled controls.