LLM-call fault tolerance & timeout management#

How OpenProgram calls LLMs robustly — retry, backoff, timeouts, connection handling, failover — and how reference agent frameworks solve the same problems.

Sources studied (all under references/, read-only):

1. The comparison matrix#

2. Notable per-project patterns#

openclaw (best transport layer)#

• Stream timeout = 30 min, set on the undici global dispatcher as bodyTimeout = headersTimeout = DEFAULT_UNDICI_STREAM_TIMEOUT_MS (src/infra/net/undici-global-dispatcher.ts:16), reset on any byte. This is the key insight: don't put a tight read timeout on a reasoning stream — give it 30 minutes, reset on any traffic.
• Forces HTTP/1.1 (allowH2:false) and Happy Eyeballs (autoSelectFamily) — avoids h2 stream resets and broken-IPv6 hangs (the classic VPN failure).
• Two-tier retry: outer retry.ts (3, 300ms→30s) + inner operation-retry.ts (2, 250ms→1s) for transient provider ops.
• API-key rotation (api-key-rotation.ts): outer loop over keys, inner transient retry per key.
• Failover categories (failover-matches.ts): rate_limit / overloaded / server / timeout / network — each a regex group.
• Positive-only jitter when honoring Retry-After (never sleep less than the server asked); SDK-retry bypass via x-should-retry.

hermes-agent (richest; Python, closest to us)#

• TCP keepalive socket injection (run_agent.py): SO_KEEPALIVE=1, TCP_KEEPIDLE=30s, TCP_KEEPINTVL=10s, TCP_KEEPCNT=3 → dead peer detected in ~60 s instead of hanging. Plus force-close TCP before SDK close to avoid CLOSE_WAIT pileup.
• Decorrelated jittered backoff seeded from time_ns ^ counter so concurrent sessions don't retry in lockstep (base 5s, ×2, cap 120s).
• Context-scaled stream-stale timeout: 180s base, →240s >50k tokens, →300s >100k tokens; disabled entirely for local providers.
• Separate TTFB vs inter-event timeouts (codex TTFB 120s, disabled above 25k context to avoid false positives during long prefill).
• Credential pool with rotation strategies (round-robin / least-used) and exhaustion cooldowns (401→5 min, 429/402→1 h, dead→prune 24 h).
• OAuth per-request token provider via httpx event hook (refresh skew 60 s) — tokens refresh mid-session without rebuilding the client.
• Partial-response recovery: on a break after the first token it returns the partial text + finish_reason=length and lets the next turn continue — no lost work, no blind retry.

opencode#

• No body/idle timeout on HTTP — streams unbounded (like pi-ai).
• WebSocket path: connect 15s, idle 5min (reset per frame), 55-min connection-age recycle, WS→HTTP fallback after 5 stream failures.
• Rate-limit header parsing for OpenAI + Anthropic into a structured object (enables proactive client-side throttling).
• Tagged-union error model; honors Retry-After (cap 10s).

pi-ai (our codex's reference)#

• MAX_RETRIES=3, BASE_DELAY_MS=1000, retries 429/5xx + body patterns — no explicit body-read timeout; relies on fetch + retry. A 120 s httpx read cap has no counterpart here, which is why adding one killed healthy long streams (see §3).

3. Timeout policy#

The governing principle, taken from openclaw: a reasoning stream must not carry a tight read timeout. A single httpx timeout=120 float caps the body read at 120 s and fires before the idle budget over a buffering proxy or VPN, which is how a healthy long stream gets killed.

Codex therefore uses Timeout(connect=30, read=1860, write=30, pool=30), and the SSE governor is two budgets plus a backstop:

• SSE_IDLE_TIMEOUT_S = 1800 (30 min) — "no bytes at all", reset on any line, pings included. This is openclaw's bodyTimeout equivalent.
• SSE_DATA_STALL_TIMEOUT_S = 900 (15 min) — "no real data", reset only on parsed events. This catches ping-flood stalls that a byte-level timeout cannot see.
• SSE_TOTAL_TIMEOUT_S = 7200 (2 h) — a runaway backstop.

All are env-overridable (OPENPROGRAM_SSE_*, OPENPROGRAM_HTTPX_*).

Backoff caps its exponential component at 30 s (OPENPROGRAM_PROVIDER_STREAM_BACKOFF_MAX_S, in utils/stream_retry.py); a larger server Retry-After is still honored. utils/errors.py parses all three Retry-After forms: retry-after-ms, integer seconds, and HTTP-date.

4. Transport and recovery#

The modules under providers/utils/ are generic and available to every HTTP provider; codex is wired to all of them.

• Central timeout policy (timeouts.py) — one source of truth at OpenClaw's 30-min level, with context-scaling helpers.
• Client builder (http_client.py):
  • TCP keepalive — SO_KEEPALIVE plus idle/interval/count, giving ~60 s dead-peer detection for the VPN-drop case. Applied defensively per OS; OPENPROGRAM_TCP_KEEPALIVE=0 disables it.
  • Force-IPv4 escape hatch (OPENPROGRAM_FORCE_IPV4=1) for broken-IPv6 VPNs, binding an IPv4 source address, since httpx has no Happy Eyeballs.
  • Connection reuse — get_shared_async_client is loop-keyed, so codex reuses its TLS connection across turns instead of re-handshaking.
  • Proxy via httpx 0.28 proxy=.
• Rate-limit header parsing (rate_limit.py) — x-ratelimit-* and anthropic-ratelimit-*; codex warns when a bucket is low or exhausted.
• Partial-response recovery (openai_codex.py) — a transient mid-stream break after content arrives finalizes the partial turn with stop_reason="length" rather than erroring, so no work is lost and no blind retry follows. Permanent failures (auth, invalid, context, policy) still hard fail. Toggle with OPENPROGRAM_PARTIAL_RECOVERY=0.
• Provider/model failover (failover.py + agent_loop.py) — a classifier (rate_limit / overloaded / server / timeout / network) plus a stream_with_failover wrapper that tries the primary and then each configured fallback on a pre-content failover-worthy failure. It forwards events, suppresses the duplicate start, and never switches after a token has streamed. On by default, conservatively: with nothing configured the chain is the user's other enabled models of the same provider (at most 2, in config-row order), so a failover reuses the credential the call was already going to use and never contacts a provider the user has not configured. Set OPENPROGRAM_FALLBACK_MODELS="provider/model,provider2/model2" to override with an explicit list, which may cross providers; set it to off (or none) to disable failover entirely.
• gemini_cli shares the same client, so it carries the same timeout semantics rather than its own single-float timeout.

Deliberately off or not built:

• API-key rotation — the machinery lives in the auth layer (auth/pool.py: pick rotation, mark_failure / record_call_failure cooldowns with strategies and TTLs). Rotation on acquire is automatic once a pool has more than one credential. Per-call failure-cooldown reporting is not wired into the live single-account path, because cooling down the only credential would lock the user out for no benefit. On a single account rotation is a clean no-op and activates by itself once multiple credentials exist.
• OAuth per-request token provider — codex already resolves and refreshes the bearer per call through the auth manager, so a full httpx event-hook provider would add machinery without fixing anything.

5. Tunables#