# Response cache

The response cache answers a repeated request from stored results. A cached answer comes back
at once and costs nothing. The cache works on `POST /v1/chat/completions` and
`POST /v1/messages`.

The cache is off by default. Turn it on for one request, or ask RouterPlus support to turn it
on for your organization.

## Request

```bash
curl https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{
    "model": "gpt-6-luna",
    "temperature": 0,
    "messages": [{ "role": "user", "content": "Summarize our refund policy." }],
    "cache": { "mode": "normalized", "ttl_seconds": 3600 }
  }'
```

Use the `x-tm-cache` header or the `cache` field in the body. The header wins over the field.

| `x-tm-cache` value | What happens |
|---|---|
| `on` | Use the cache for this request. |
| `off` | Do not use the cache for this request, even when your organization has it on. |
| `refresh` | Do not read the cache. Call the model and store the new answer. |
| `semantic` | Use [semantic matching](#semantic-matching). |

| `cache` field | Rule |
|---|---|
| `mode` | `"exact"` (default), `"normalized"` or `"semantic"`. |
| `ttl_seconds` | How long the answer stays, 1 to 86400. Default 300. |
| `shared` | `false` keeps the answer to one user. See [Scope](#scope). |
| `allow_nonzero_temperature` | `true` lets a request with `temperature` above 0 use the cache. |
| `semantic` | `true` turns on semantic matching. |

`"cache": true` turns the cache on with the defaults. `"cache": false` turns it off. The gateway
removes the `cache` field before it sends the request to a model.

## Modes

- `exact`: a hit needs the same request. The model, the messages, the system prompt, the tools,
  `tool_choice`, `response_format` and every sampling parameter (`temperature`, `top_p`, `seed`,
  `max_tokens`, `stop` and the others) must match.
- `normalized`: the same as `exact`, but the gateway trims the text of each message and turns
  each run of spaces, tabs and new lines into one space. Letter case still counts.

The match ignores `stream`, `stream_options`, `user`, `metadata`, `session_id` and
`prompt_cache_key`. A streamed request can get an answer that a non-streamed request stored.

## Semantic matching

Semantic matching returns a stored answer when the last user turn is close in meaning to a
cached one, inside the same [scope](#scope). It is off until RouterPlus sets it up for your
organization. Ask for it on a request with `x-tm-cache: semantic`,
`"cache": { "semantic": true }` or `"mode": "semantic"`.

- The response carries `x-tm-cache: hit` and `x-tm-cache-similarity` (cosine similarity, 0.97
  or more by default).
- The comparison ignores case, punctuation and spacing.
- Each lookup and store embeds the last user turn with `text-embedding-3-small`. This is an
  ordinary `POST /v1/embeddings` request on your key, billed at its price (about $0.02 per
  million tokens).

## Scope

- A stored answer belongs to one organization. Another organization never gets it.
- Inside your organization, a stored answer belongs to one user by default. The user is the
  request's `user` field (on Messages, `metadata.user_id`) or its session. When the request
  names neither, the API key is the user.
- Your organization can turn on sharing between its users. Then any user of your organization
  can get an answer that another user stored.
- With sharing on, a request can keep its answer to its own user with `"shared": false`.
- With sharing off, a request cannot share its answer.

## What a hit looks like

A hit is your stored answer with:

- a new `id`, and on chat completions a new `created`;
- `usage` with the stored token counts, `"cost": 0` and `"cached_response": true`;
- the headers `x-tm-cache: hit` and `x-tm-cache-age` (seconds since the answer was stored).

A hit costs nothing. When you ask to stream, the gateway sends the stored answer as a stream in
the live format: chat completion chunks on `/v1/chat/completions`, and Messages events on
`/v1/messages`.

## What is not cached

The gateway calls the model and stores nothing when a request:

- sets `temperature` above 0, unless `allow_nonzero_temperature` is `true`;
- asks for more than one choice (`n` above 1);
- uses a tool that runs on the provider's side, such as web search, code execution or an MCP
  server (tools that your code runs, functions, are cached);
- gives an image or a file by URL or by file id (images and files sent inline are cached).

The gateway stores only a complete answer. It does not store an error, an answer that ended
with an error, or a streamed answer. A streamed request can get only an answer that a
non-streamed request stored.