Skip to main content

​
Overview

GoModel ships with two response-cache layers for streaming and non-streaming requests on:
  • /v1/chat/completions
  • /v1/responses
  • /v1/messages
  • /v1/embeddings
Streaming and non-streaming variants of the same request are cached independently: a streaming miss stores the raw SSE bytes and a streaming hit replays them verbatim as text/event-stream. Exact-match cache returns byte-identical responses with:
Semantic cache uses embeddings plus vector search so meaning-equivalent prompts can reuse a stored response:
/v1/embeddings participates in the exact cache only. An embedding has to represent the exact text it was requested for, so replaying the vector of a merely similar input would be a wrong answer rather than an equivalent one. Embeddings requests are also never streamed, so only the JSON response is cached.

​
Enable the exact cache

Point response caching at Redis:
You can also configure it with environment variables:
  • REDIS_URL
  • REDIS_KEY_RESPONSES
  • REDIS_TTL_RESPONSES

​
Enable semantic caching

Add a semantic block with an embedder provider and a vector store:
Supported vector stores:
  • qdrant
  • pgvector
  • pinecone
  • weaviate
Both cache layers run after workflow and guardrail patching, so they operate on the final request sent upstream. Use Cache-Control: no-cache or Cache-Control: no-store to bypass caching per request. A plugin can also keep one response out of both layers with NoStore on its decision, which a guardrail that restores request-specific data into the reply uses so the restored text is never replayed to another caller. A background: true request on /v1/responses bypasses both layers as well: its queued body carries an id that belongs to one caller’s response. For the full semantic-cache design and storage options, see ADR-0006.

​
What the exact cache keys on

The exact cache hashes:
  • the request path
  • the resolved workflow context used for execution specifically execution mode, provider type, and resolved model
  • the guardrail chain the request resolves to, across the prompt, response and stream phases
  • the final request body
This means guardrails and workflows affect cache keys when they change the resolved workflow or the final body sent through execution. A cache hit replays the stored response without re-running the response and stream guardrail chains, so an entry is only ever served to a request whose chain matches the one that produced it. Editing, adding or removing a guardrail step — or a second workflow with a different chain — yields a miss instead of a reply those guardrails never saw. Changing guardrail configuration therefore invalidates the entries stored under the old chain. JSON object member order and insignificant whitespace are canonicalized before hashing. Requests that differ only in formatting therefore share one exact entry. Concurrent identical misses are also coalesced in-process: one request reaches the provider and the waiters replay its response. Failed or otherwise non-cacheable responses are never fanned out.

​
Provider prompt-cache planning

After routing resolves the concrete provider and model, GoModel adds a cache plan when the stable prefix meets that provider’s minimum size and the client did not already supply a cache directive:
  • OpenAI receives a stable prompt_cache_key; models supporting explicit caching also receive a breakpoint and explicit cache mode.
  • Anthropic receives top-level automatic ephemeral caching.
  • Amazon Bedrock Converse receives a cache point after the stable prefix.
  • Native Gemini AI Studio creates and reuses a five-minute cached-content object, with concurrent creation coalesced by prefix and stable credential.
The plan is applied post-routing so provider-only fields cannot leak through a fallback to another backend. Cache keys include the concrete provider instance and model, and caller-supplied cache controls always win. Provider prompt-cache planning is enabled by default. To disable GoModel’s automatic planning while continuing to pass through compatible client cache directives, set:
The flag accepts Go boolean values. Missing, empty, or invalid values keep the safe default (true) and an invalid value emits a startup warning.

​
user_path behavior

For the exact cache, user_path is not added to the cache key by itself. That is intentional. If two requests end up with the same path, resolved workflow, and final request body, they can share the same exact-cache entry even when they originate from different user_path values.
If you need tenant or path-specific cache behavior, use a scoped workflow or otherwise make the final request differ for that scope. user_path alone is not an exact-cache partition key.
Common patterns:
  • disable cache in a scoped workflow
  • use different scoped workflows for different user_path values
  • include scope-specific context so the final request body differs

​
Responses served through failover are not cached

When a request is answered by a failover target rather than the one routing picked first, the response is not stored. The exact-cache key is hashed from the pre-failover resolution, so storing the backup’s answer would serve it under the primary’s key once the primary recovers. The practical consequence: a virtual model whose first target is chronically down stops accumulating cache entries for as long as it stays down — every request that misses fails over and is served fresh, and nothing it produces is stored. Entries written before the outage keep serving until their TTL expires: a lookup runs ahead of dispatch, so a hit is returned without touching either target. If a target is going to be unavailable for a while, reorder the chain (or point the virtual model at a healthy target) so the responses that answer it are cacheable again. Retries within one target are unaffected: only a response produced after the sweep moved to another target is withheld from both cache layers.

​
Cache analytics

When response caching and usage tracking are enabled, the admin API exposes a cached-only overview at:
Cache hits are also recorded as usage entries, with the caller’s user_path and a cache_type of exact or semantic. The usage endpoints (/admin/usage/* and /v1/usage) leave them out by default so that totals and cost track provider spend; pass cache_mode=all to include them, or cache_mode=cached to see only the hits. /admin/usage/sessions is the exception: session request and token totals always include cache hits, while its cost fields stay provider-only.
See admin endpoints and the usage API.
Last modified on September 12, 2026