Cache identical responses

Answer a repeated request from Model Gateway’s cache instead of calling the provider again. The cache is off by default; an organization admin turns it on per model, and callers can opt out per request.

How the exact cache works

The gateway stores a model’s complete reply and replays it when the same caller sends the exact same request again before the reply expires. Exact means every part of the request matches: messages, tools, sampling settings, output constraints, and whether it streams.

A stored reply is reused only when…	Why
The same caller, with the same credential, in the same workspace, sends it	One person never receives another’s answer.
The request is identical, including `stream`	A streamed and an unstreamed request never share an entry.
It arrives on the same endpoint	Chat completions, Messages, and Responses each keep their own entries.
The model’s routing and settings haven’t changed	Any change to the model, its pool, or its provider starts a new entry.
The caller can still call the model	The gateway checks access before every lookup. A cached reply never grants access.

What is cached:

Complete chat replies, unary and streamed, and embedding replies.
A streamed reply only when the stream finished normally with its usage known, and stayed under 2 MiB.
Not decisions, images, audio, or any other modality.

A cache hit doesn’t call the provider and creates no new provider spend. The reply carries the usage the original call reported.

Before you begin

You’re an organization admin, to turn the cache on for a model.
The model’s modality is chat or embedding. Settings offers the cache only for those.

Open the model's settings

In Model Gateway studio, open Models, select the model, and open its Settings tab.

Turn on the exact-answer cache

Under Exact-answer cache, switch from Off (default) to On. In Keep an identical answer for, enter how long to keep a reply, from 1 to 3,600 seconds.

Save the model

Select Save changes.

Control the cache per request

Callers can narrow the cache for one request with headers. The model’s setting is always the upper bound: a caller can shorten the time a reply is kept, never lengthen it. These headers are never forwarded to the provider.

Header	Effect
`x-gateway-cache: bypass`	Ignore any stored reply and call the provider. A fresh successful reply may replace the stored one.
`x-gateway-cache: no-store`	Don’t read or write the cache for this request.
`x-gateway-cache-ttl: <seconds>`	Keep this reply for at most that long. `0` turns caching off for the request.
`Cache-Control: no-cache`	Same as `x-gateway-cache: bypass`.
`Cache-Control: no-store`	Same as `x-gateway-cache: no-store`.

curl https://api.atlan.com/openai/v1/chat/completions \
  -H "Authorization: Bearer $ATLAN_API_KEY" \
  -H "x-gateway-cache-ttl: 60" \
  -H "Content-Type: application/json" \
  -d '{"model": "<provider>/<model>", "messages": [{"role": "user", "content": "Hello"}]}'

Any value of x-gateway-cache other than bypass or no-store returns 400 with x-gateway-cache must be bypass or no-store. A x-gateway-cache-ttl that isn’t a whole number of seconds returns 400 with invalid x-gateway-cache-ttl.

Verify

On the model’s Overview tab, Routing and health reads Exact-answer cache on, 60 s. for a 60-second setting.
Send the same chat request twice, with the same key.
A fresh reply has an x-gateway-model-wire header naming the provider API that answered. A reply served from the cache has no x-gateway-model-wire header, because no provider API was called.

Exact cache versus provider prompt caching

Providers such as Anthropic cache the start of a long prompt and bill those tokens at a lower rate. That is a separate feature, and the gateway keeps it separate.

	Exact response cache	Provider prompt caching
Who runs it	Model Gateway	The provider
What it reuses	The whole reply to an identical request	The processed prefix of a prompt; the model still generates a new reply
Provider call	None on a hit	Every request
How you turn it on	Exact-answer cache on the model	The provider’s own controls, such as Anthropic’s `cache_control` blocks
Where you see it	No `x-gateway-model-wire` header on a hit	Cached input and Cache write on the call in Calls

The gateway forwards cache_control markers to providers whose API reads them: the Anthropic Messages API (direct, on Vertex AI, or on Azure AI Foundry), Amazon Bedrock, OpenRouter, and a LiteLLM proxy. On Bedrock only {"type": "ephemeral"} is carried. Other provider APIs have no such marker, so the gateway drops it there.

How Model Gateway works: what the gateway keeps and doesn’t.
Model API compatibility: every request and response header.
Review model usage: find cached input on a call.

Skill management

Agent management

AI Gateway

Metrics Studio

Ontology

How the exact cache works

Before you begin

Control the cache per request

Verify

Exact cache versus provider prompt caching

​How the exact cache works

​Before you begin

​Control the cache per request

​Verify

​Exact cache versus provider prompt caching

​Related

How the exact cache works

Before you begin

Control the cache per request

Verify

Exact cache versus provider prompt caching

Related