Request building: Context → per-provider parameters#

The providers layer has a single responsibility: take a ready-made Context, translate it into the wire request for the current provider, and apply prompt caching according to that provider's mechanism.

providers does not care how the Context was built — how identity/tools/memory are assembled in the system prompt, whether to segment it, that is the concern of the upstream ../context/. This layer only handles "what to receive, what to translate, what to send".

1. Core: one unified format + one translation per provider#

Upstream produces a provider-agnostic Context; this layer dispatches by model.api to the corresponding provider's translation, converting it into the request that provider expects. All provider differences are contained in the translation layer, and upstream only ever deals with Context.

Context (already built upstream)
    │  dispatch by model.api
    ▼
that provider's translation: Context → its wire request

The industry-consensus approach (opencode / hermes / openclaw all do this).

2. Unified format: Context#

Context {
  system_prompt    system prompt (this layer treats it as ready-made content, not asking how it came to be)
  messages         conversation (user / assistant / tool_result)
  tools            tool list
}

The three are stored separately, with system standalone. This framework's Context uses standalone system as its baseline — translating to APIs with standalone system (Anthropic / Bedrock / Gemini) maps directly; when translating to the OpenAI family, that provider's translation folds system into messages.

3. Translation layer: where system lands + field mapping#

One translation per provider (_build_system / _build_messages / _build_tools), dispatched by model.api. The core differences are where system lands and the field names:

The translation also smooths over each provider's idiosyncratic blocks: thinking-block signatures, whether tool_use arguments are objects or strings, and tool schema dialects (strict / additionalProperties).

4. Caching: three modes + a declaration layer#

Each provider's caching mechanism is fundamentally different, so don't force one abstraction over them all — split into three modes, with each provider declaring which one it belongs to.

The cache_spec declaration layer#

Copying the declarative paradigm of thinking under models/: one cache.json per provider declaring mode + the cache-key parameter name + TTL mapping + breakpoint limit. The shared module cache_spec.py loads it (get_cache_spec / cache_mode / ttl_for_retention / cache_key_param), and provider code reads the declaration to decide behavior, rather than hardcoding the rules in stream_simple. A provider with no declaration falls back to none (same as thinking's OpenAI-compatible fallback).

{
  "mode": "explicit",
  "breakpoint_format": "cache_control",        // or "cachePoint" (bedrock)
  "retention_ttl_map": {"short": null, "long": "1h"},
  "max_breakpoints": 4
}

{ "mode": "auto", "cache_key_param": "prompt_cache_key" }

{ "mode": "none" }

Where breakpoints go (explicit mode)#

The caller can explicitly mark cache_control on a content block, and it passes through verbatim to apply after that block (see ../../plans/cache-control-passthrough.md); when unmarked, the provider automatically marks the last block. The breakpoint limit (4 for Anthropic) is constrained by cache_spec's max_breakpoints; on overflow, the lower-priority ones are dropped by the tools > system > messages priority.

If breakpoint positions need to be decided uniformly by "the stable segments marked upstream", this relies on the existing per-block marker field TextContent.cache_control — upstream marks the block on a stable segment, and this layer passes it through. No new Context-level structure is introduced, nor is any change to the system_prompt type required.

5. The interface with upstream#

../context/                     providers (this layer)
build Context, decide content,  ──→   translate into each provider's wire, apply caching by mode
mark cache_control on blocks   Context     read cache_spec, pass through cache markers

The contract = Context (a content block may carry cache_control). How upstream builds the context is fully decoupled from this layer: adding a provider only touches this layer + one cache.json, and changing context building only touches ../context/.

6. The caching policy layer#

The caching policy layer follows opencode's cache-policy.ts + protocols/utils/cache.ts (references/opencode/packages/llm/src/), reproduced in Python:

Integration point: anthropic stream_simple calls apply_cache_policy before building messages/tools; _get_cache_control reads the ttl mapping and long_ttl_endpoints from cache.json rather than hardcoding them.

bedrock is also declared as explicit, but it uses cachePoint (a standalone block) rather than cache_control attached to a block, and it carries its own "place a breakpoint on the last message" logic, so it stays outside the unified apply_cache_policy; folding its tool breakpoints in is a possible later increment.

One difference from opencode: opencode's system is a segmented array and can mark a breakpoint individually on the "last system segment"; this layer's Context.system_prompt is a single string, and the system breakpoint is placed by each provider's _build_system on that whole block. The policy layer only covers tools + messages.