Response cache

The response cache answers a repeated request from stored results. A cached answer comes back at once and costs nothing. The cache works on POST /v1/chat/completions and POST /v1/messages.

The cache is off by default. Turn it on for one request, or ask RouterPlus support to turn it on for your organization.

Request

curl https://dev2-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{
    "model": "gpt-6-luna",
    "temperature": 0,
    "messages": [{ "role": "user", "content": "Summarize our refund policy." }],
    "cache": { "mode": "normalized", "ttl_seconds": 3600 }
  }'

Use the x-tm-cache header or the cache field in the body. The header wins over the field.

"cache": true turns the cache on with the defaults. "cache": false turns it off. The gateway removes the cache field before it sends the request to a model.

Modes

• exact: a hit needs the same request. The model, the messages, the system prompt, the tools, tool_choice, response_format and every sampling parameter (temperature, top_p, seed, max_tokens, stop and the others) must match.
• normalized: the same as exact, but the gateway trims the text of each message and turns each run of spaces, tabs and new lines into one space. Letter case still counts.

The match ignores stream, stream_options, user, metadata, session_id and prompt_cache_key. A streamed request can get an answer that a non-streamed request stored.

Semantic matching

Semantic matching returns a stored answer when the last user turn is close in meaning to a cached one, inside the same scope. It is off until RouterPlus sets it up for your organization. Ask for it on a request with x-tm-cache: semantic, "cache": { "semantic": true } or "mode": "semantic".

• The response carries x-tm-cache: hit and x-tm-cache-similarity (cosine similarity, 0.97 or more by default).
• The comparison ignores case, punctuation and spacing.
• Each lookup and store embeds the last user turn with text-embedding-3-small. This is an ordinary POST /v1/embeddings request on your key, billed at its price (about $0.02 per million tokens).

Scope

• A stored answer belongs to one organization. Another organization never gets it.
• Inside your organization, a stored answer belongs to one user by default. The user is the request's user field (on Messages, metadata.user_id) or its session. When the request names neither, the API key is the user.
• Your organization can turn on sharing between its users. Then any user of your organization can get an answer that another user stored.
• With sharing on, a request can keep its answer to its own user with "shared": false.
• With sharing off, a request cannot share its answer.

What a hit looks like

A hit is your stored answer with:

• a new id, and on chat completions a new created;
• usage with the stored token counts, "cost": 0 and "cached_response": true;
• the headers x-tm-cache: hit and x-tm-cache-age (seconds since the answer was stored).

A hit costs nothing. When you ask to stream, the gateway sends the stored answer as a stream in the live format: chat completion chunks on /v1/chat/completions, and Messages events on /v1/messages.

What is not cached

The gateway calls the model and stores nothing when a request:

• sets temperature above 0, unless allow_nonzero_temperature is true;
• asks for more than one choice (n above 1);
• uses a tool that runs on the provider's side, such as web search, code execution or an MCP server (tools that your code runs, functions, are cached);
• gives an image or a file by URL or by file id (images and files sent inline are cached).

The gateway stores only a complete answer. It does not store an error, an answer that ended with an error, or a streamed answer. A streamed request can get only an answer that a non-streamed request stored.