OpenProgram Docs
Web Runtime Reliability and Performance Design
This document consolidates verified issues in Web refresh, session loading, WebSocket distribution, background-error display, EventBus notifications, asynchronous Job state, static assets, and local state directories, defining compatible target contracts, task boundaries, and quantitative acceptance criteria.
Measured baseline
The following measurements use the current default state directory, largest current session, direct session-load handler execution, and cache-disabled local page loads. They are fixed before/after baselines, not universal machine timings.
handle_load_session; excludes network, browser parsing, and React state writes.chrome-profile approximately 6.1 GB; sessions approximately 8.5 MB.| Measurement | Method | Limitation | Subsequent use |
|---|---|---|---|
| Session payload | Execute the public handler using real SessionDB data and a WebSocket substitute recording sent content. | Disable cold context stats to exclude independent background refresh from the first frame. | Before/after on the same session; record bytes for total frames, messages, graph, and context tree separately. |
| Browser loading | Default-port page, cache disabled, Navigation/Resource Timing and response headers. | The diagnostic browser lacks default-App owner credentials, so authenticated chat was not exercised. | Accept static sizes, CSP, compression, and script counts here; use default App for real chat. |
| EventBus logs | Count types and byte sizes in global and session events.jsonl. | Log content changes with actual usage. | Compare event rows, duplicate IDs, large fields, and write latency afterward. |
| Worker resources | Short process sampling of default worker and state-directory size inventory. | Not a long-running workload profile. | Exclude sustained CPU saturation; add event-loop latency and queue metrics after implementation. |
Current data flow and failure points
The system currently has browser WebSockets, raw WS frames, EventBus, AgentEvent, SessionDB, and a MessageStore not integrated into production. The diagram shows frontend-to-persistence operations and background-event return paths.
Red marks verified gaps; yellow marks design/implementation mismatch; green marks existing useful optimizations. MessageStore v2 is a reuse candidate; do not introduce a third incremental protocol.
Issues and user impact
Prioritize data loss, invisible errors, event-loop blocking, unbounded resource growth, and ordinary-user trigger probability. Every item cites current source rather than presenting proposals as implemented.
| ID | Priority | Current behavior | User-visible impact | Root cause and evidence | Target state |
|---|---|---|---|---|---|
| R1 | P0 | Unhandled storage exceptions from public dispatcher and agent/channel/settings management actions now persist to profile-scoped UserErrorStore before sending final low-sensitivity operation_error; runtime/session top-level type="error", project empty/embedded errors, channel validation/QR-specific errors, and chat reply error turns remain. Frontend still lacks a durable global error store. | Migrated management failures are visible and remain recorded server-side after transport failure; other legacy forms still lack consistent display, acknowledgement, and refresh recovery. | server.py handles uncaught action exceptions uniformly; user_errors.py saves bounded low-sensitivity records; ws_actions/ still has forms awaiting migration. | All user-relevant failures enter one error contract and remain visibly scoped. |
| R2 | P0 | Public dispatcher unknown-action and handler exceptions now produce final low-sensitivity operation_error, with unique error/correlation IDs, explicit scope, and UTC time. Server supports principal-scoped list, exact-ID acknowledgement, and multi-window recovery; frontend remains compatible with legacy action_error. | Handler failures are no longer misreported as missing backend handlers; pre-send records support recovery. Unknown/future codes use generic failure text excluding wire message/error content. | server.py uses a shared safe builder, records correlation metadata, and closes with state_recovery_required on persistence failure; user_error.py provides strict pagination, acknowledgement binding, and same-principal recovery broadcasts; action-error.ts normalizes old/new envelopes. | Validation codes, remaining action migration, and frontend hydration/display remain later TASK 1 slices. |
| R3 | P0 | Agent/channel list, add/delete, binding, and alias storage exceptions no longer become empty arrays or silent success. Public receive loop persists them and returns one low-sensitivity operation_error, without sending normal lists or success notices. | Protocol distinguishes healthy empty states from these management failures. Pages receive no normal response replacing old data, but degraded markers and refresh recovery remain absent. | agent.py, channel.py storage exceptions propagate to public dispatcher; records are isolated by authenticated principal. | Durable records and log correlation IDs are implemented; degraded markers, acknowledgement, and recovery remain later TASK 1 slices. |
| R4 | P0 | Job broadcaster already sends error, but Branches state and AttachCard do not retain/display details. | Background tasks show only errored; after refresh, reason and retryability are unknown. | runner.py and branches/index.tsx fields are asymmetric. | Persist error summaries with Jobs; cards show safe reasons, retryability, and log correlation IDs. |
| R5 | P0 | Reconnect and session switch send complete session_loaded. | Large-session refresh transfers/parses over 11 MB, with response time growing with history. | session.py sends messages, graph, and context tree together; client reconnect directly calls load_session. | Initial manifest + current chain; Graph, context tree, and large fields on demand; reconnect incrementally by revision/seq. |
| R6 | P1 | Session loading synchronously aggregates, sorts, builds graphs, and JSON-serializes inside the WS event loop. | Refreshing one large session delays commands on other sockets in the same worker. | _fn_run_siblings repeatedly calls all_msgs.index, without separating computation and sending. | Compute immutable snapshots in a thread; async layer handles ordered sending and state checks only. |
| R7 | P0 | TASK 3 routes production connections through one sender with a 256-frame / 4 MiB queued + in-flight cap. | Concurrent slow-client send_text fell from RED’s 400 to 1; ordinary updates coalesce and terminal events use reserved capacity. | ws_delivery.py and public-handler slow-client regression. | complete; specification, quality, and full Python gates PASS. |
| R8 | P1 | Reconnect retries every 2 seconds and reloads the entire session on success. | Service restarts/network failures synchronize retries across pages and repeat full-state transfers. | use-ws.ts. | Exponential backoff with jitter; send known revision/seq after reconnection, without duplicate UI errors. |
| R9 | P0 | Ordinary EventBus notify handlers execute synchronously in emitter threads; async handlers are skipped when no event loop exists. | Slow plugins block Agents; some background events reach no consumer. | events/bus.py. | notify always enters a dedicated queue; gate stays synchronous with an explicit budget. |
| R10 | P1 | Event logs still write synchronously; duplicate tool.before rows are fixed with both rereviews passed; registry tests still miss multiline emits. | Frequent tool output still adds synchronous I/O; tests may miss unregistered events. | event_log.py, test_event_gate.py. | Implement asynchronous batched logs, AST registration checks, and large-field summaries. |
| R11 | P1 | TASK 6 removes runtime CDN resources and bundles marked/KaTeX. Starlette gzip responder negotiates gzip for compressible responses and excludes already-compressed MIME types. | Source, legacy templates, build output, and public HTTP entries have automated regressions. Browser home no longer auto-fetches Google favicons. Specification, quality, full Python, default App, and local browser traces pass. | markdown-render.ts and frontend.py. | complete; preserve self-only CSP. |
| R12 | P2 | Chrome profile lacks capacity display/retention policy; currently approximately 6.1 GB. | Long-term disk growth has no product-visible attribution or safe cleanup scope. | Largest directories are browser on-device model, Default profile, and model store. | Show categorized sizes; provide explicit recoverable cleanup only; never automatically delete login state. |
Unified error contract and frontend display
“Backend errors must appear in frontend” means every failure affecting user requests, durable state, background tasks, or recoverable operations produces an observable error record. It does not mean every internal retry/degradation becomes a Toast, nor permit traceback, file content, or credentials in browsers. This schema is the sole final wire contract. TASK 1 server generates complete fields for public dispatcher/initial management actions, records before sending, and supports principal-scoped list, exact-ID acknowledgement, and multi-window recovery. Frontend hydration/display and other sources remain later slices.
// WebSocket envelope; use null for nullable fields; do not omit semantically required fields { "type": "operation_error", "data": { "error_id": "err_opaque_unique_id", "request_id": "client_request_id_or_null", "scope": "session | job | settings | channel | agent | transport | system", "code": "stable_machine_code", "message": "safe user-facing summary", "action": "safe_action_or_null", "session_id": "session_id_or_null", "operation_id": "action_job_execution_id_or_null", "retryable": false, "severity": "info | warning | error | fatal", "correlation_id": "opaque log lookup id", "occurred_at": "ISO-8601 UTC" } }
| Error-contract field/state | Single meaning |
|---|---|
request_id | Client-generated for request/response actions; success/failure must echo it and finish pending.operation_id identifies a server entity, not a request ID. |
error_id | Unique per failure. Toast deduplication key is scope + code + session_id + operation_id; display once per identical key within 5 seconds, but preserve individual records. |
UserErrorStore | Use a separate SQLite file in the current profile, isolated by authenticated principal. Retain at most 7 days from creation and 1,000 records per principal. Open records use occurred_at, error_id strict newest-first pagination; acknowledgement closes transactionally and idempotently by principal + exact error_id. Send only after commit. Persistence failure must not send a falsely durable envelope; close the connection and require state recovery. |
| severity | info enters diagnostics only; warning shows local degraded state; error shows a card and one Toast; fatal shows a persistent blocking state and stops automatic retries. |
| Lifecycle | Request success only ends the matching request_id pending state.acknowledge_user_error sends on the requesting connection, using the same request_id, first user_error_acknowledged and then {type:"operation_recovered", data:{error_ids:[...], scope, operation_id, occurred_at}}; after durable closure, other same-principal windows independently receive the same recovery. No cross-connection order is guaranteed, and a failed requesting connection must not block other windows. Server explicitly lists affected error_id; frontend must not bulk-close by vague codes. Repeated acknowledgement may resend exact-ID recovery for convergence while DB closure stays idempotent. User acknowledgement/entity deletion closes explicit IDs only; unrelated later success cannot clear errors implicitly. |
| Job failure | job_status(status=errored) is the sole Job-state wire event; frontend derives error-store records from it. Do not send a second operation_error.operation_error for the same failure. It is used only when a Job action request itself fails. |
| Event category | Frontend behavior | Persistence | Must exclude |
|---|---|---|---|
| Current-session request failure | Show reason near the relevant card/control; one Toast on first receipt. | Store in session error records, visible after refresh. | Traceback, raw environment, unfiltered provider body. |
| Background Job failure | Session list shows an error marker; AttachCard/Jobs show summary and retryability. | Persist with the Job. | Complete subprocess stdout or file bodies. |
| Settings/management failure | Retain last known state and show operation failure; never interpret failure as an empty list. | Error center retains until acknowledgement or later success. | Credential values, tokens, absolute secret paths. |
| Recoverable internal degradation | No Toast by default; relevant surface shows degraded state, queryable in diagnostics. | Structured logs and metrics. | Repeated user-facing notices. |
| Transport disconnection | Show connection state without repeated identical errors during recovery. | Enter error center only after recovery budget expires or with unacknowledged operations. | One Toast for every reconnect attempt. |
One safe payload builder
The helper generates filtered error data and validates fields, log correlation, and session routing. WS action/channels/system wrap it as operation_error; JobRunner embeds the same data in the sole job_status, without a second error envelope.
One state entry
use-ws parses and forwards to the error store only; cards, session markers, and Toasts derive from one record rather than independently interpreting codes.
Explicit request completion
Request/response actions use request IDs. Success and failure end pending; no button may load indefinitely.
Incremental legacy-frame migration
During migration, frontend adapts existing error, action_error and Job errors into unified records; remove compatibility branches after each backend migration.
Session loading and refresh
Initial cost should depend on visible content rather than entire history. SessionDB/DAG remains canonical; Web wire sends only currently needed projections.
| Phase | Returned content | Trigger | Cache key |
|---|---|---|---|
| Session manifest | Title, status, head, revision, current-chain IDs, counts, feature state, error summary. | Session switch or missing valid manifest. | session_id + revision |
| Viewport messages | Visible chain tail and required compaction/attach references. | Initial view or scrolling into unloaded ranges. | session_id + head_id + cursor |
| Graph summary | Node IDs, edges, roles, states, minimal labels; no message bodies/tool results. | User opens DAG/branch panel. | session_id + graph_revision |
| Node/detail | Single-node context, complete tool output, runtime tree, attachment details. | User expands/navigates to a node. | session_id + node_id + content_revision |
| Reconnect delta | Durable commits after known revision, and state/terminal events after known seq for each running entity. | Reconnect while contiguous deltas remain available. | Client submits known_session_revision with the same known_stream_seqs: {entity_id: seq} |
| Full recovery | New manifest and bounded viewport, not full history. | Incompatible revision, evicted delta, or invalid client state. | Reset local session projection |
MessageStore contributes only block, seq, and delta computation, never another durable store. Remove unusable independent persistence/singleton interfaces in TASK 4. All full recovery comes from SessionDB projections.| Protocol quantity | Definition | Retention and fallback |
|---|---|---|
session_revision | Monotonically increasing integer per session, incremented once per atomic SessionDB/DAG commit. Manifest, message headers, Graph summary, and durable errors carry their read revision. | Persist permanently with the session. Client revision above server or incompatible schema rejects deltas and returns a new manifest. |
stream_seq | Independently monotonic per running message/job; describes uncommitted text/tree/progress changes only. | Current worker retains 256 deltas per entity or 15 minutes, whichever first. Restart persists unfinished entities as error/aborted, then clients perform manifest recovery. |
| delta journal | Durable-commit-generated from_revision → to_revision projection changes, without complete body copies. | In memory: last 512 commits or 15 minutes. Larger gaps return manifest + bounded viewport. |
| viewport cursor | Stable pair (created_at, message_id), paginated backward along current head chain. Initial view is at most 100 messages and 512 KiB uncompressed JSON, whichever first. | Paginate toward history; head changes yield a new cursor. If one message exceeds remaining budget, return only header, summary, and detail marker. |
| detail cursor | Single-node/tool-result reads cap at 1 MiB uncompressed; larger content is chunked at UTF-8-safe boundaries with next cursor. | Reauthorize session/node/caller on every read; cursors cannot cross nodes or sessions. |
Separating computation from sending
A synchronous thread reads an immutable SessionDB snapshot, builds ID indexes, current chain, and Graph summary, returning plain objects. Async handler rechecks revision before sending; concurrent writes cause manifest-plus-delta or a fresh snapshot. Threads never manipulate WebSockets.
Duplicate computation to remove
Prebuild id → insertion_index instead of sorting with all_msgs.index; group siblings once by predecessor; read the session row once per request; serialize Graph summary separately from message bodies.
Large fields and compatibility
Preserve current 32 KiB tool-result markers and get_full_tool_output authorization checks. Extend on-demand reads to context trees, runtime blocks, Graph-node content, and attachment metadata. Legacy clients may request old actions, but server payload caps must return upgrade guidance rather than unbounded frames.
WebSocket queues, ordering, and recovery
Standard browser WebSocket APIs provide no automatic backpressure, so servers must bound send queues. One sender task per connection sends sequentially; producer threads only enqueue thread-safely.
| Frame category | Example | Queue rule | Overflow behavior |
|---|---|---|---|
| Non-droppable terminal | error, question, chat final, job terminal, ack, permission result | Preserve order; no overwriting or cross-session merging. | If queue budget is unavailable, close and require state recovery; never drop silently. |
| Coalescible delta | text/thinking delta, tree update, progress, context stats | By session + entity + type, keep newest state or merge adjacent text. | Remove superseded deltas and increment coalesced counts. |
| Snapshot | sessions list, branches, settings, graph summary | Keep only the newest unsent snapshot per key. | Replace old snapshots without affecting terminal frames. |
| Control | pong, auth/status, reload required | Small separate budget prevents large-message starvation. | Authentication/protocol errors close immediately; stale pong values may be discarded. |
Queue limits and critical reserve
Per connection: 256 frames or 4 MiB, whichever first; reserve 64 frames and 1 MiB for non-droppable terminal events. Before critical enqueue, remove coalescible deltas; do not consume old ordinary snapshot versions.
Send ownership
Only sender task calls ws.send_text. On disconnect, cancel sender, free queue, and unregister focused session; other threads retain no WebSocket Futures.
Reconnect
Exponential backoff with jitter; page visibility or explicit retry may trigger earlier. Restore protocol state before session projection.
Metrics
Record queue_frames, queue_bytes, oldest_age, coalesced, send_failures, disconnect_reason, and recovery_mode without frame bodies.
UserErrorStore, queried by error_id, retaining at most 1,000 records or 7 days. Session manifests include only relevant open IDs; reconnect reads details through list_user_errors(cursor). Clients derive transport-close state locally from close/timeout, independent of server error frames. Chat ack follows durable user-message storage; other request acks must be idempotently queryable/replayable by request_id. If critical reserve is still full, close the connection and mark state_recovery_required; reconnect restores from manifest, UserErrorStore, pending-question registry, Job store, and request-result queries, not unsent memory frames.Standard WebSocket has no automatic backpressure. Retain its broad compatibility while bounding application queues.
Clients can use bufferedAmount to monitor unsent bytes, but it cannot replace server per-connection queues and recovery.
Use a FIFO queue with positive maxsize; cross-thread callers use event-loop thread-safe scheduling, never direct cross-thread Queue manipulation.
EventBus execution and logging
Keep two clear semantics: gate is synchronous decision-making; notify is asynchronous observation. They cannot share direct handler invocation in the emitter thread.
| Property | Gate | Notify |
|---|---|---|
| Purpose | Allow or reject before an action. | Record, update UI, trigger subsequent nonblocking work. |
| Execution location | Emitter thread; default total budget events.gate_timeout_ms=50. | EventBus-owned event loop; synchronous consumers use a thread pool, async consumers run on that loop. |
| Failure | Retain current fail-open with structured warnings and gate results. | Isolate consumers; retries belong to consumer policy; never skip silently. |
| Ordering | Registration order. | Same session + entity stays ordered; different keys may run concurrently. |
| Backpressure | No queue; budget expiry skips remaining gates and records it. | Bounded queues; ordinary telemetry cannot fill critical-event capacity. |
| Logging | Combine event and gate verdict into one record. | Asynchronous batch writes; large payloads use size, digest, and controlled previews. |
| Notify queue contract | Rule |
|---|---|
| Ordering key | Prefer session_id + entity_id; use session if only session exists; otherwise type + origin. Strict FIFO per key, concurrency across keys. |
| Capacity | Ordinary queue: 4,096 events maximum; critical state queue: 512. Critical types are operation error/recovered, question, Job terminal, turn end, and critical-marked ws.frame. |
| Full queue | Coalesce/drop keyed telemetry such as progress/context stats with explicit metrics. If critical queue fills, source durable state remains recovery authority; write an emergency loss marker and notify Web connections to recover state. Printing stderr alone must not imply delivery. |
| Shutdown and restart | Normal shutdown drains at most 2 seconds; remaining notify events get shutdown loss markers. Critical business state persists before emit and recovers from store scans after restart; EventBus is not a durable queue. |
| Gate security boundary | EventBus gates carry additional plugin/workflow decisions and stay fail-open. Permission, authentication, sandbox, and credential checks do not use this gate; their mandatory boundaries continue rejecting. |
| Log uniqueness | Choose session or global log according to session presence, never both; one row per event ID per target file. Gate events log once after verdict, not once during notify and again afterward. |
AST registration checks
Tests parse Python AST emit_safe, make_event calls and constant event names, including multiline calls. Dynamic names require explicit exceptions and consumers.
One event record
tool.before observation and gate share one Event; final logs include gate fields, without a preceding ordinary record.
Summarize high-volume fields
tool.after.result_text logs byte count, error status, hash, and bounded preview by default. Complete results remain in SessionDB/tool-result storage, not duplicated in event logs.
Errors are not raw forwarding of all events
The EventBus-Web adapter forwards registered user-relevant envelopes only. Internal logs correlate with user errors but never expose arbitrary payloads.
Frontend rendering, refresh, and static assets
Current stream reducer coalesces per animation frame; MessageRow subscribes by ID and recycles long-list rows. Retain these. Focus on less data entering stores, removing ineffective external dependencies, and bounding initial resources.
| Area | Keep | Change | Verification |
|---|---|---|---|
| Streaming messages | Coalesce text/thinking per animation frame; per-message selector. | Flush matching pending deltas before terminal state; atomically update error record and reply state. | At most one relevant store update per frame under high-frequency deltas; complete terminal content. |
| Long lists | MessageRow memo, viewport row recycling, stable ID arrays. | Hydrate viewport data only; Graph panels no longer force complete Graph bodies into chat stores. | Initial DOM and React commits for 1,000+ nodes do not grow linearly with full history. |
| Error UI | Existing Toast component and session unread state. | Add error store, durable cards, background-session markers, and Toast deduplication. | Automated assertions for current session, background session, and post-refresh. |
| marked / KaTeX | Markdown/math behavior. | Import locally from frontend bundle; remove CDN loader and WS initialization wait. | Zero CSP violations under strict script-src 'self'; works offline. |
| Static responses | Same-origin static export and cache headers. | Add gzip for text, JS, CSS, and SVG; optional build-time Brotli later. | Correct Accept-Encoding negotiation; no recompression of compressed formats; no SSE compression. |
| Code splitting | Next.js static export. | Lazy-load noninitial settings, DAG, browser, and programs surfaces. | Initial transfer/largest chunk meet budgets; interaction-driven loading shows pending. |
Use official ASGI middleware minimum-size/compression-level settings; exclude SSE and compressed formats.
'self' conflicts with CDN runtime scripts. Choose local dependencies without relaxing CSP or adding unsafe-inline/unsafe-eval.
Browser support is insufficient, and it does not solve server queues, persistence, errors, or incremental recovery. Keep standard WebSocket with application controls.
Runtime metrics, capacity, and state directories
Performance changes must remain measurable on the default worker. Record counts, durations, bytes, and correlation IDs only, not prompts, model responses, tool output, or credentials.
WebSocket
- Active connections and queue bytes per connection.
- Send latency, failures, coalesced count.
- Reconnect counts, recovery mode, full-recovery reasons.
Session hydration
- Separate manifest/messages/graph bytes.
- DB read, projection, serialization, and send durations.
- Viewport hits, detail fetches, delta fallbacks.
EventBus
- Notify enqueue and handler duration.
- Queue depth, failures, retries, drop reasons.
- Total gate time and budget overruns.
Storage
- Categorized sessions, logs, Chrome cache, models, and IndexedDB sizes.
- Show target and impact before user-triggered cleanup.
- Do not delete cookies, login state, extensions, or sessions by default.
Implementation tasks and dependencies
Each task has its own commit, specification review, and quality review. TASK 1 command isolation, first management-action migration, server persistence, and recovery are implemented in slices but remain partial overall. TASK 3 and TASK 6 pass both reviews, full gates, and respective runtime acceptance and are complete. TASK 5 gate single-record logging passed both rereviews, but TASK 5 overall remains incomplete. Later tasks must not depend on unreviewed earlier worktree state.
Unified error envelope and frontend error state partial
Public dispatcher exception isolation, agent/channel/settings storage-error migration, server UserErrorStore, and list/ack/recovery are implemented. Final envelopes persist by authenticated principal before enqueue, capped at 1,000 records/7 days; persistence failure requires state recovery. Acknowledgement uses exact error IDs, request-ID association, and same-principal recovery broadcasts only. Web supports operation_error/action_error, and TUI consumes both frame types using low-sensitivity text. Current-session cards, background markers, Toast deduplication, reconnect hydration, and other legacy sources remain unimplemented. Preserve chat reply error bubbles; TASK 2 handles Job-entity errors.
Complete background Job error display
Preserve Job errors from entity through broadcast/list loading to AttachCard/Branches, including after refresh. Use safe summaries; full tracebacks stay in logs.
Bounded per-connection send queue complete
Add server-layer _webui/ws_delivery.py, sender lifecycle, frame classification, coalescing, terminal guarantees, and disconnect cleanup. All broadcast paths only enqueue. Slow-client, send-failure, cross-thread concurrency, overflow, terminal-reserve, and in-flight capacity regressions pass; no changes to events/bus.py.
Session projection and incremental recovery
Fix payload measurement/protocol contracts first, then separate synchronous computation from async sending. Implement manifest, viewport messages, Graph summary, detail fetch, and reconnect deltas. Assess reuse/removal of unintegrated MessageStore.
EventBus notify and logging execution single-record slice complete
Gate observer notifications now defer disk logging until verdict, producing one record containing gate fields; independent specification/quality rereviews pass. Remaining scope: dedicated notify execution, synchronous gates, async batched logs, large-payload summaries, and AST registration checks. Preserve public plugin-hook signatures.
Local frontend dependencies, compression, resource budgets complete
Removed CDN runtime loaders/legacy tags; bundle marked/KaTeX locally. Replace CDN icons and automatic browser-home favicon requests with local text markers. Starlette gzip negotiates by weight and excludes compressed MIME types. Noninitial center-tab surfaces dynamically import with pending states. Build budgets, HTTP regressions, full Python, default-App visuals, and five cache-disabled browser traces pass.
Capacity diagnostics and explicit cleanup
Add read-only category sizes/diagnostics; design explicit cache/model cleanup. Identity/browser user data remain excluded by default. This does not block error/refresh fixes.
| Task | depends_on | Parallelism | Responsibility exclusions |
|---|---|---|---|
| TASK 1 | None | May run alongside TASK 3, 6, 7 | Define generic action errors/error store only, not Job-entity states. |
| TASK 2 | TASK 1 | After TASK 1, may run alongside TASK 4 | job_status solely carries Job terminal state; reuse error store without a second Job error frame. |
| TASK 3 | None | May run alongside TASK 1, 6, 7 | Change server queues and ws.frame enqueue adapters only, not events/bus.py notify scheduling. |
| TASK 4 | TASK 3 | May run alongside TASK 2, 5 | SessionDB/DAG is sole durable source; remove unusable MessageStore persistence interfaces. |
| TASK 5 | TASK 1, TASK 3 | No file overlap with TASK 4 | Sole owner of events/bus.py, registry, and event log; Web adapter uses TASK 1 envelopes and TASK 3 enqueue. |
| TASK 6 | None | May run independently | Preserve CSP trust boundary and WebSocket protocol. |
| TASK 7 | None | May run independently | Read-only diagnostics by default; deletion needs separate interaction design and explicit confirmation. |
Quantitative acceptance matrix
Targets are implementation contracts, not current achievements. Each needs automated tests or repeatable measurement commands, not observation alone.
| Public entry | Scenario | Acceptance criterion | Evidence type |
|---|---|---|---|
| Any registered WS action | Handler raises | Same socket receives safe operation_error; connection survives; UI displays within 250 ms; correlation ID locates logs. | Server receive-loop component test + Web reducer/UI test. |
| list/delete/settings/channel actions | Underlying storage failure | No normal empty result or silent success; retain old data with degraded state or show explicit failure. | Per-action contract tests. |
| JobRunner | Refresh after background-task failure | Job, AttachCard, and session marker show the same safe reason; retryable agrees. | Persistence + WS + React integration tests. |
| Initial load of largest known session | No client cache | First uncompressed wire projection ≤ 1 MiB; server event-loop occupancy ≤ 50 ms; remaining computation in threads. | Fixed-session fixture payload/time-budget test. |
| Reconnect same session | Known revision, at most 64 missing deltas | No full history; recovery frames ≤ 256 KiB; terminal states, errors, and questions preserved. | Disconnect/recovery protocol tests. |
| Slow WebSocket client | Continuous stream deltas and terminal events | Queue ≤ 256 frames or 4 MiB; deltas may coalesce; terminal states not lost; unrecoverable overflow explicitly closes and requires resync. | Controlled slow-sender component test. |
| EventBus notify | Slow sync plugin + async subscriber from worker thread | Emitter enqueue p99 ≤ 1 ms; async subscribers not skipped for missing current loop; failures do not block peers. | Concurrency benchmark + behavior tests. |
| Event log | One gated tool.before | Choose exactly one session/global log target; one row per event ID with gate verdict; no full large-result bodies. | Real event-file assertions. |
| Initial Web load | Cache disabled, accepts gzip | Zero CSP violations/CDN runtime requests; first transfer ≤ 2 MiB; largest compressed JS chunk ≤ 350 KiB. | Default-App browser trace + HTTP header tests. |
| State-directory diagnostics | Profile exceeds configured warning size | Correct categories; no deletion without explicit confirmation; cleanup report lists categories and recoverability. | Temporary-profile tree tests. |
Measurement definitions and threshold rationale
| Metric | Start/end and environment | Sample | Threshold rationale |
|---|---|---|---|
| Session wire bytes | UTF-8 uncompressed bytes of backend json.dumps output; fixed large anonymized default-profile fixture, cold context stats disabled. | Five runs per candidate; median/max, with message/graph/detail breakdown. | 1 MiB is the first-stage hard cap from 11.66 MB, bounding local JSON parse/remote transfer; each detail has its own 1 MiB chunk cap. |
| Event-loop occupancy | Async-handler entry to first await send, measuring only event-loop-thread wall time; thread projection time reported separately. | 20 fixture runs; report p50/p95/max. | 50 ms budgets interactive occupancy of the main loop, not total page completion. |
| Error-UI latency | Complete WS-frame receipt to error card appearing in DOM, excluding network. | 20 runs, report p95. | 250 ms allows state update/render while staying well below the existing 2-second reconnect interval. |
| Notify enqueue | emit entry to successful enqueue in the bus-owned queue, excluding consumers. | 10,000 calls each single-threaded and with 8 workers; p50/p95/p99. | p99 ≤ 1 ms prevents ordinary notify from materially extending tool/turn execution. |
| Static transfer | Default App, cache disabled, no extension injection; sum encodedBodySize for document-origin resources. | Five cold loads; median/max. | 2 MiB is the first-stage budget from 3.42 MB; 350 KiB caps individual compressed JS chunks to keep lazy surfaces out of the initial bundle. |
| CSP | Count violations caused by OpenProgram document origin only; report extension/devtool injection separately. | One cold load, math-message opening, Settings entry, and DAG entry. | Zero product-origin violations is a security/functionality contract. |
| Capacity categories | Actual allocated/logical filesystem bytes with classification rules; symlinks do not escape state root. | Fixed temporary tree and real read-only diagnosis. | Category sum differs from root by at most 1% or 1 MiB, whichever larger. |
Compatibility risks, adoption, and exclusions
| Choice | Disposition | Reason and boundary |
|---|---|---|
| Standard WebSocket + application-bounded queues | Selected design | Stable browser compatibility; server still needs explicit queue, coalescing, and recovery semantics. |
| Existing MessageStore seq/delta | Partial reuse | Reuse block/seq/delta computation only. SessionDB/DAG is the sole durable authority; remove independent store interfaces. |
| Unified operation_error | Selected design | Frontend needs stable machine codes and safe text; temporary adapters support old error frames. |
| Dedicated EventBus notify execution | Selected design | Satisfies nonblocking observation; gates stay synchronous. |
| Locally bundled marked/KaTeX | Selected design | Consistent with strict CSP, offline operation, and reproducible builds. |
| Relax CSP for CDNs | Reject | Adds remote-script trust/availability dependencies without solving initial-connection waiting. |
| Full WebSocketStream / WebTransport migration | Reject | Browser support/complexity exceeds scope and cannot replace durable recovery/error contracts. |
| Global React store/component rewrite | Reject | Existing rAF coalescing, per-message subscriptions, and row recycling are not proven bottlenecks. |
| Automatic Chrome-profile deletion | Reject | May remove login state, IndexedDB, extensions, or other user data; only explicit categorized cleanup is allowed. |
to_thread.Implementation-status appendix
Separates existing, partial, and unimplemented capabilities. Current status does not weaken target contracts.
| Capability | Status | Current evidence | Remaining work |
|---|---|---|---|
| Unified operation_error and error store | Server recovery protocol | Public dispatcher and migrated agent/channel/settings storage failures share final low-sensitivity operation_error; server persists by authenticated principal before send to separate SQLite UserErrorStore, capped at 1,000 records/7 days with concurrent-write protection and correlation IDs; list_user_errors paginates strictly, acknowledge_user_error closes exact IDs idempotently with request association, and same-principal windows receive exact-ID recovery. Web/TUI compatibility consumers exist. | Implement frontend hydration, scope display/deduplication, and remaining legacy sources. |
| Chat reply error bubble | Existing | Main chat dispatcher error turns terminate placeholders and use existing rendering. | Associate with error center without duplicate Toasts. |
| WebSocket command-level exception isolation | Four slices | Unknown actions, uncaught handler exceptions, and covered management failures persist first, then return low-sensitivity operation_error; valid request_id/session_id/action/operation_id associates requests, explicit scope/log correlation applies, and list/ack/recovery isolate authenticated principals. Web supports legacy action_error; TUI hides wire message/error and isolates other-session errors. | Add validation codes; migrate runtime/session/project/channel QR and other paths; implement frontend recovery/display. |
| On-demand tool output | Existing | Wire truncates tool results over 32 KiB; expansion authorizes session/node/caller reads. | Include Graph, context tree, and other duplicated large fields in projections. |
| Stream coalescing and long-list recycling | Existing | text/thinking deltas coalesce by animation frame; MessageRow uses stable selectors; long lists compute live rows. | Add budgets ensuring new error/delta stores preserve these properties. |
| R6 request-local sibling-navigation indexes | Partially complete | fn-run enumerates once for stable order; ordinary chat sibling_navigation_index builds request-local ID, target sibling-group, and deepest-leaf-child indexes, sorts only complete groups relevant to the visible chain, and materializes navigation only for that chain. Public handle_load_session regressions enforce linear sibling grouping/child traversal and cover equal timestamps/deep adjacent branches. Pure DAG-helper tests cover duplicate/empty IDs, runtime/tool/code, spawn, dangling predecessors, and cycles. A synthetic 2,000-message/100-visible-node median fell from 134.076 ms to 1.355 ms; 50,000 offscreen siblings remain near 15 ms and 2.75 MiB peak memory. Both independent rereviews pass. | Remaining R6 compute/send separation, single session-row snapshot, projection/delta recovery, and full event-loop budget are unimplemented; this does not mark R6 complete. |
| Background Job error fields | Partial | Runner broadcast and TypeScript types include error. | State storage, card rendering, refresh recovery, and Toast/error-center association. |
| Incremental session recovery | Unintegrated | MessageStore defines seq/delta/full, unused by production server/Web. | Implement projection recovery from sole SessionDB/DAG authority and remove independent store interfaces. |
| Per-connection WebSocket backpressure | complete | Each production connection has one sender; queued + in-flight ≤ 256 frames / 4 MiB. Slow-client peak concurrent sends fell from 400 to 1. Specification, quality, and full Python gates PASS. | None. |
| EventBus gate-log uniqueness | Partially complete | Registered gates defer disk logging while emit notifies observers. The same Event, after emit_gate completes its verdict, writes one JSONL record containing gate fields. Public regressions ensure observers still receive once; both rereviews PASS. For 1,000 paired tool.before, synchronous logging median fell from 105.15 ms to 58.48 ms and rows from 2,000 to 1,000. This is not full EventBus/end-to-end latency. | TASK 5 async notify, batched logs, large payloads, and AST registry checks remain unimplemented. |
| Guaranteed async EventBus notify | Unimplemented | Ordinary handlers execute synchronously; async handlers without a loop are skipped. | TASK 5. |
| Local marked/KaTeX and static compression | complete | Zero product CDN resources in source/legacy templates; local marked/KaTeX; passing JS/CSS/SVG gzip, weighted negotiation, identity, small-response, MIME exclusion, existing-encoding, and SSE regressions. Browser home uses local text icons. Build-budget size: 2,089,594 B; largest built gzip JS chunk: 186,752 B. Five cache-disabled local loads had median/max transfer 1,221,805 B and largest script 187,384 B. Every run had 65 local-origin requests, zero failures, and zero CSP/console errors. | Remote-deployment browser traces are outside local TASK 6 completion and require deployment-time verification. |
| Profile capacity diagnostics and cleanup | Unimplemented | Capacity is inspectable only through the filesystem. | TASK 7. |
TASK 1 long-term contract and slices
| Boundary | Long-term contract | Current implementation |
|---|---|---|
| Wire envelope | Section 04 schema is the sole final contract; nullable fields explicitly use null, never copying exceptions or command payloads to wire. | Public dispatcher unknown actions, uncaught exceptions, and first management group generate all final fields; chat reply’s own error turn is not duplicated. |
| Correlation identity | request_id is a client-generated per-command correlation value, not authorization/idempotency credentials. Echo only nonempty strings without controls and at most 128 characters.session_id with the same action likewise echoes only safe strings. | Missing/invalid correlation returns null, never original objects, oversized text, or internal exceptions. |
| Error classification | unknown_action means nonretryable protocol/version mismatch; handler_error means command handling failed, conservatively nonretryable. Later actions use registered codes; unknown frontend codes use generic low-sensitivity text. | Current public paths classify session, agent, settings, channel, or system; exceptions, tracebacks, paths, credentials, and command payloads stay out of wire/store. Full traceback stays in local structured logs. |
| Compatibility | New frontend consumes operation_error and legacy action_error; new server sends no duplicate frames. Web assets and worker are one release unit; new-server/old-frontend pairing is not promised. | Retain actionErrorNotice and consumeActionError export as a source-compatible alias, internally normalized into one notice. |
| Durable recovery | UserErrorStore indexes by error_id, session, and request, isolating authenticated principals. List, acknowledgement, and reconnect recovery use explicit IDs. | SQLite, 1,000-record/7-day bounds, persist-before-send, strict pagination, exact-ID acknowledgement, request-associated results, and same-principal recovery are implemented. Frontend hydration is not, so refresh recovery is not complete. |
| Display | Current-session errors, background markers, and Toasts share deduplication keys; existing chat error bubbles retain semantics. | Dispatcher command failure still displays one Toast; other-session errors cannot enter current-session-only views. |
_websocket_handler returns operation_error for unknown action/handler exceptions, echoes valid request_id, rejects invalid metadata, and continues handling ping. Frontend normalizes old/new frames to identical safe notices; unknown codes hide wire message/error._websocket_handler to produce exactly one safe operation_error, no normal empty/success response, and continue ping. TUI consumes both command-error frame types without rendering wire message/error; valid other-session errors stay out of current transcript, and only current-session chat failures end the corresponding turn.state_recovery_required. Store tests cover reopen, principal isolation, thread/process first-write races, cross-principal expiry, capacity, 7-day bounds, and low-sensitivity wire/store.list_user_errors returns only the authenticated principal’s open records with strict newest-first pagination; acknowledge_user_error sends acknowledgement then exact-ID recovery on the requesting connection and independently notifies same-principal peers only, even if requester sending fails. Duplicate/unknown exact IDs are idempotent without errors. Invalid fields, malformed JSON, and nonobject JSON produce safe invalid_request, and the connection continues. 35 store/component/protocol regressions pass.TASK 3 brief and implementation evidence
| Evidence item | Verified content |
|---|---|
| Approved source | This page’s WebSocket queue, TASK 3, and quantitative acceptance clauses; task began from the then-current main baseline. |
| Production files | server.py, _webui/ws_delivery.py, ws_actions/webtab.py, routes/misc.py. |
| Test files | test_websocket_delivery.py, test_websocket_command_lifecycle.py, test_webtab_control.py, test_web_owner_auth.py. |
| Required behavior | One sender; cross-thread enqueue only; bounded queued + in-flight; text/state coalescing; critical terminal reserve; overflow/send failure require state recovery; disconnect completes all waiters. |
| Compatibility and exclusions | Retain _ws_connections, await ws.send_text and existing exact-socket control semantics; no EventBus, Session projection, frontend reconnect, or error-contract changes. |
| RED | Inject 400 updates through real _websocket_handler with the same _broadcast; old implementation peak concurrent send_text was 400, expected 1. |
| GREEN and affected verification | 37 focused tests passed; expanded WebUI/web-use: 448 passed, 1 skipped, plus 1 reproduced baseline failure. |
| Reviews | Specification PASS; quality PASS. Confirmed waiter, failure-semantic, terminal-classification, and in-flight-capacity findings have focused regressions. |
| Full gate | Ruff PASS. After deterministic baseline fixes and independent review, full Python: 6,243 passed, 19 skipped, 7 deselected, 1 xfailed, zero failures/errors. |
| Delivery status | complete. Specification, quality, and full Python PASS; integrated into an isolated branch, not yet merged into main or pushed. |
TASK 6 brief and implementation evidence
| Evidence item | Verified content |
|---|---|
| Approved source | This page’s frontend/resource contract, TASK 6, first-load acceptance, and static-transfer definitions; isolated worktree created from TASK 3 baseline commit. |
| Production files | apps/server/openprogram_server/_webui/frontend.py; apps/web/app/layout.tsx; components/app-shell.tsx, components/chat/messages/markdown.ts, components/settings/provider-icon.tsx; lib/format-utils/markdown.tsx, lib/global.d.ts, lib/net/use-ws.ts, lib/runtime-bridge/conversations.ts, lib/runtime-bridge/markdown-render.ts; public/html/index.html; apps/web/package.json and root package-lock.json. Delete lib/external-libs.ts with the same lib/cdn-globals.d.ts. |
| Verification files | tests/component/webui/runtime/test_webui_frontend.py; apps/web/scripts/runtime/check-local-runtime-assets.mjs, check-static-resource-budget.mjs; synchronize check-chat-ui.mjs, check-markdown-sanitize.mjs with the same check-socket-access.mjs. |
| Required behavior | All product runtime assets local; marked/KaTeX never wait on external loaders; browser home never auto-fetches external favicons; CSP stays self-only. JS/CSS/SVG use Starlette gzip with case-insensitive, q-weight-aware Accept-Encoding; small responses, already-compressed MIME, SSE, and existing Content-Encoding are not compressed/recompressed. Noninitial center-tab surfaces load on demand with pending states. Build artifacts meet fixed 2 MiB / 350 KiB budgets. |
| Compatibility and exclusions | No changes to WebSocket, Session projection, EventBus, or error contracts; useMarkdownReady keeps compatibility exports; OwnerAuth CSP is not relaxed. |
| RED | Initial public-HTTP fixture lacked Content-Encoding on large static JS: 1 failed / 13 passed. Local-runtime script failed on CDN loader; budget script failed without production output. Specification-repair RED reproduced PNG/WOFF2 double gzip: 2 failed / 16 passed. Quality-repair RED added extensionless API MIME/q-weight paths: 6 failed / 19 passed; browser-component check failed on Google favicon helper. |
| GREEN and affected verification | After repair, all 25 frontend tests pass, covering gzip, identity, small responses, static/extensionless MIME exclusions, case/q weights, SSE, and existing encoding; combined OwnerAuth/CSP: 67 passed. Web 115 unit tests, browser/bookmarks/local-runtime source checks, TypeScript, and production build pass. |
| Resource budgets | chat.html conservative artifact check counts 110 initial/CSS-referenced resources totaling 2,089,594 B at gzip level 6, largest compressed JS 186,752 B, both under budget. |
| Reviews and repairs | Specification review confirmed extension-based exclusions missed extensionless responses; repair retained Starlette responder. Quality review found three issues: exclusions must use response MIME, Accept-Encoding must handle case/q weights, and browser home must not auto-fetch Google favicons. Repairs added MIME-aware responder, negotiation preferring explicit gzip over wildcard, and local DOM text icons. Final scoped specification and quality rereviews PASS. |
| Full gate | Deterministic baseline issues were separately fixed and independently reviewed. Full Python then reported 6,243 passed, 19 skipped, 7 deselected, 1 xfailed, zero failures/errors. Prior intermittent models.dev failures/background-thread errors did not recur. |
| Delivery status | complete. Specification, quality, full Python, default-App refresh/visual acceptance PASS. Five local cache-disabled traces meet 2 MiB / 350 KiB with no external requests, failures, or CSP/console errors. Remote-server traces and push were not performed. |
Current audit verification entries
PYTHONPATH=apps/server:. .venv/bin/python -m pytest \ tests/component/agent/turns/test_event_bus.py \ tests/component/agent/turns/test_event_gate.py \ tests/unit/agent/behavior/test_event_bridges.py \ tests/unit/agent/dispatcher/test_error_turn_finalized.py \ tests/component/agent/jobs/test_async_job_recovery.py \ tests/component/webui/chat/test_session_tool_output_transport.py \ tests/unit/webui/test_webui_chat_dispatcher.py -q cd apps/web && npm test cd apps/web && npx tsc --noEmit python3 -m scripts.docs_site.build python3 -m scripts.docs_site.checklinks
These entries show existing supported paths survive documentation changes, not that target behavior is implemented. Each TASK must first add corresponding error-UI, slow-client, projection, queue, EventBus-concurrency, or resource-budget RED.
Related authoritative documents
Update implementation evidence only after the task passes specification review, quality review, and full verification. This page keeps no dates or commit history; Git preserves evolution.
Bounded session history
Session loading separates the latest transcript page from older history. Pages retain their captured branch head and a stable oldest-message cursor. A page contains complete display rows and their caller descendants, with at most 50 top-level rows and a 512 KiB target serialized budget. One indivisible larger row is returned alone. Paging never edits stored history or model context. Historical reads do not change focus, live session head, execution state, or question replay.
Codex separates thread metadata, turns and items through paginated history APIs. DeepSeek Harness exposes message-aligned pages and sequence cursors. This implementation adopts stable branch snapshots, backward paging and incremental live updates; it retains OpenProgram's existing branch graph and execution recovery contracts. It does not introduce another session store or change model compaction.
Duplicate promoted fields are omitted from the wire-only extra object. Existing tool-output previews retain their authorized full-output expansion. Logical history and graph responses larger than the transport frame budget are sent as ordered, awaited fragments; the client reconstructs a complete response before dispatch, preserving first-fragment order with subsequent live notifications. Deferred responses share a per-connection budget of 64 million UTF-16 code units, eight incomplete transfers and 1,024 logical responses. This handles a single large row without silently dropping data or increasing the 4 MiB sender queue. Ordinary live updates continue independently. Fragment buffers belong to one connection and are discarded on disconnect. Reconnect loads a fresh recent page and uses existing execution recovery.
The client prepends older rows by stable identity without replacing newer live content or moving the selected conversation. Page requests capture the current load generation and are ignored after a reload, branch switch or socket replacement. An unavailable cursor produces an explicit retryable error instead of a reconnect loop. Scroll position remains anchored when older messages are inserted. The graph remains complete for existing branch and compaction controls; server-side history projection still scans the captured session. Storage indexing and lazy graph rendering are separate optimizations, not claimed here.
Acceptance: the copied failing session opens without queue overflow; every logical page retains exact content and caller ownership; page frames stay bounded, including Unicode and a single oversized row; cursor exhaustion loses or duplicates no messages; stale responses cannot overwrite live/newer state; disconnection releases partial buffers; branch, compaction, tool expansion and execution recovery remain functional. Implementation status: implemented and independently reviewed. The combined verification passes 8,591 Python tests, 635 Web tests, four browser tests and 190 CLI tests, together with Desktop, static, type, build and documentation checks. The installed App retrieves all 26 original display rows in five pages without duplicates, reordering or disconnects; the largest observed transport frame is 51,564 bytes. Reload and heartbeat remain operational.
References: Codex history protocol; DeepSeek Harness history.
Long-session layout and loading
The bounded history protocol separates transport size from transcript length. Long sessions also require stack-safe graph construction: the depth pass previously followed every ancestor for every node and recursively visited the conversation chain. A 1,500-node linear conversation exceeds Python's recursion limit before its first page can be delivered.
Graph depth now uses a memoized parent-chain classification followed by iterative preorder traversal. Nodes whose parent path enters a cycle remain independent roots, preserving the existing defensive behavior. Forks, retries, spawned branches and call children retain their current row order and offsets. Parent classification visits each node once; traversal does not depend on the Python call stack. Follow-up normalization indexes replies once. Lane column placement maintains per-lane widths and the largest occupied column rather than rescanning previously placed nodes.
Session message aggregation and byte-budget page preparation execute outside the event loop, while awaited WebSocket sends remain on the owning loop. Branch graph construction also executes outside that loop. Snapshot and incremental event ordering remain unchanged. The existing in-memory session index remains authoritative for reads; this change introduces no persistent cache, schema migration or history deletion.
Acceptance uses the public graph builder and session-loading handler with long chains, wide fork sets and invalid cycles. Small-graph output must match existing layout contracts, and a 10,000-node transcript must load a bounded recent page without recursion failures. Full graphs and initial store indexing remain proportional to session size; this work does not claim constant-memory storage or indexed transcript projection. The Codex and DeepSeek references in the bounded-history section continue to define the paging comparison.
Implementation status: implemented and independently reviewed. Public layout and session-loading tests cover a 10,000-node chain, wide forks and invalid cycles; complete graph output and bounded transcript pages are preserved. The combined verification and installed-App long-history replay pass. Initial graph and store memory still grow with session size.