OpenProgram · Design document

Error and timeout mechanisms (current-state inventory)

Scope: the complete path of one runtime.exec() from initiation to failure/timeout: retries, backoff, connection/stream timeouts, error classification and outer watchdog. Purpose: document current behavior before optimizing. Optimization items are listed separately in §8 with implementation state and corresponding commits. Companion documents: docs/design/llm-fault-tolerance.md(cross-framework comparison), error-retry.md.

1 Three timeout responsibilities

The system has three semantically different timeouts, owned by different layers and code. Confusing them causes most misunderstandings.

2 Complete call stack

From benchmark to network I/O. Each layer’s lower-right corner lists its own timeout/retry controls. There are three independent retry loops(locator, exec, stream_retry), without a shared wall-clock budget.

3 Existing timeout defenses (active)

These are all already active; environment variables can override every default. Defined centrally in providers/utils/timeouts.py and http_client.py.

4 Exec retry state machine

Retry logic within one exec() . Key point: the deadline is checked only between two _call between attempts(dashed box), so it cannot interrupt the currently running _call.

# runtime.py:746–818 (abridged)
_deadline = start + timeout_s   # timeout_s=None ⇒ _deadline=None ⇒ entire block inactive
for attempt in range(max_retries):          # default 6 (_default_max_retries, :58)

    ┌── Soft deadline check (between attempts only) ─────────────┐
    if _deadline and now() >= _deadline:           # :754
        raise LLMError(reason=TIMEOUT, retryable=False)
    └──────────────────────────────────────────────────┘

    try:
        reply = self._call(...)      # :771 synchronous blocking, no timeout argument; a stall here prevents the check above
        break
    except (TypeError, NotImplementedError):
        raise                       # Programming error: do not retry
    except Exception as e:           # :780 catches everything, including an external TimeoutError
        permanent = _is_permanent_error(e)   # :782 matches image/auth markers only
        if permanent or attempt == max_retries-1:
            raise _build_llm_error(...)   # → LLMError(reason, retryable, cause, …)
        sleep_s = _retry_sleep_seconds(attempt)  # 1.5·2^n, ±25% jitter; attempt5 ≈ 48s (:81)
        if _deadline and now()+sleep_s >= _deadline:  # :801 report TIMEOUT early if backoff would exceed the deadline
            raise LLMError(reason=TIMEOUT, retryable=False)
        time.sleep(sleep_s)          # :818

_is_permanent_error Recognizes only two classes runtime.py:104–120

The marker list contains only: not a valid image / invalid image / image data is not / login expired / login failed / re-auth / unauthorized / invalid api key. Any timeout-class exception misses these → permanent=False → it is retried. This directly explains why the outer SampleTimeoutError is treated as transient.

5 Core defect: three retry loops multiply with no wall-clock cap

With persistent ConnectError(dead route, VPN packet loss or unreachable proxy; logs show openai-codex stream retry … ConnectError), the three loops multiply, and no layer caps wall-clock time:

6 Timeline of a swallowed watchdog

ScreenSpot uses signal.alarm to stop at the sample deadline, but its exception reaches the two layers’ except Exception.

7 Error classification and retryability

When exec gives up, it raises structured LLMError (errors.py:62), already containing enough fields for precise caller-side scoring; callers should not label everything wrong_format.

classify_error Classification order errors.py:191–258

Note: override_reason=TIMEOUT deadline hits always use retryable=False (runtime.py:170–176); other exhausted budgets whose error type could be retried retain retryable=True, meaning “retry with a new budget”, not “retry immediately”.

8 Optimization items (implementation status)

Two parallel areas with different responsibilities: framework #1–#4 preventanycaller (channel worker, web turn, benchmark) from excessive nested retries; harness #5 handles sample-level scoring and a reliable watchdog. Subprocesses do not replace framework fixes, and framework fixes do not remove runner watchdog obligations.#1–#4 delivered in framework 4e64ac95 , #5 in harness 74bfd06 (subprocess isolation deferred); #6 deferred.