# Agent integration guide

This page is for an AI coding agent that builds RouterPlus into a product. It covers these
steps in order: the key, the SDK, model calls, provider pinning, sessions and caching, errors,
limits and cost. Each step links to the reference page for details.

- Every docs page is also raw markdown. Add `.md` to its URL, for example
  `https://demo.routerplus.com/docs/errors.md`.
- The index of all pages is `https://demo.routerplus.com/llms.txt`.
- All pages in one file: `https://demo.routerplus.com/llms-full.txt`.

> [!NOTE]
> **For agents.** Do not guess model ids, request fields or headers. Copy them from this page or from the live catalog. If an error message disagrees with your plan, trust the error message and read the page it names. Some steps need a person: see [Steps only a human can do](#steps-only-a-human-can-do). Stop and ask at those steps.

## The short version

1. Point the OpenAI SDK at `https://demo-api.routerplus.com/v1`, or the Anthropic SDK at `https://demo-api.routerplus.com`. Read the key from `TM_API_KEY`, on the server only.
2. Copy model ids from the catalog. The gateway never substitutes a model. An unknown id is a 404.
3. Send `max_tokens` on every request, from 1 to the `max_output_tokens` of the model. Without it, the model gets its own maximum (128,000 tokens on the Claude 5 and GPT-6 models).
4. For provider features such as server tools, call Claude models on `/v1/messages` and other models on `/v1/chat/completions`.
5. Keep the start of the prompt identical from turn to turn, and only append. This keeps prompt caches warm.
6. Pin a provider only when you must, with the `provider` object. Check the plan first with `POST /v1/route`.
7. Switch on `error_type`, never on message text. Retry `rate_limit` after `retry-after`. Stop on `insufficient_quota` and tell a human.
8. There is no idempotency key. A retry is a new request. It is billed again if it reaches a provider.
9. Keep at most 8 requests in flight per organization, unless you raised that limit.
10. Log `x-request-id` and `usage.cost` for every call.

## At a glance

| Item | Value |
|---|---|
| OpenAI-compatible base URL | `https://demo-api.routerplus.com/v1`, for `POST /v1/chat/completions`, and `POST /v1/responses` for OpenAI's models |
| Anthropic-compatible base URL | `https://demo-api.routerplus.com`, for `POST /v1/messages` (the SDK adds `/v1`) |
| API key | `tm_vk_` followed by 48 hex characters |
| Auth header | `Authorization: Bearer <key>` or `x-api-key: <key>`, on every route |
| Model list | `GET https://demo-api.routerplus.com/v1/models` with your key, or `https://demo.routerplus.com/api/models.json` (public, with prices) |
| Cost of a call | `usage.cost`, in USD, in every billed response |
| Audit of a call | the `x-request-id` response header, then `GET https://demo-api.routerplus.com/v1/generation?id=<id>` |
| Other routes | `POST /v1/images/generations`, `POST /v1/decisions`, `POST /v1/embeddings`, `/v1/audio`, `/v1/batches`. See [Overview](overview.md#gateway-endpoints) |

### What is not available

Do not build on these:

- Moderation. The path returns 404 `not_found`.
- Stored responses on the Responses API. `store` is always `false`, and `previous_response_id` is a 400. The Responses API serves the OpenAI models. Claude and the other models are a preview there. See [POST /v1/responses](api-responses.md).
- Server-side conversation memory and idempotency keys. Session ids exist: see [Sessions and caching](#sessions-and-caching).
- More than one answer per request. `n` above 1 is a 400.
- Model aliases. The one exception is the `rp-` names of the Claude models, for Cursor. In code, use the catalog ids. A `models` list is a fallback list. See [Model fallback](model-fallback.md).
- Other sort values. `provider.sort` takes only `"price"`, `"throughput"`, `"latency"` and `"quality"` (or the object form `{"by": ..., "partition": ...}`). Any other value is a 400.
- `provider.sort` and `provider.max_price` on a dedicated endpoint. Both are a 400 there, because the endpoint's own constraints apply.
- A price filter on the model list. `GET /v1/models?provider=...` with `max_price` is a 400 that names `provider.max_price`, because the list does not compare prices. Set `max_price` on each request. The list accepts `sort`, but `sort` changes nothing there.
- Provider keys or endpoints in a request. `api_key`, `base_url` and `connection_id` in the body are a 400.

## 1. Get a key and keep it safe

| Key | Where it comes from | Requests per minute | Use it for |
|---|---|---|---|
| First trial key | Shown after verified browser sign-in through Clerk | 20 | The first test calls |
| Console key | [https://demo.routerplus.com/console/keys](https://demo.routerplus.com/console/keys) | 300 | Production, staging and CI |
| Identity API key | `POST https://demo.routerplus.com/api/identity/keys` | 300 | One key for each end user or service (step 7) |

To get a key, the user:

1. Opens [Sign up](https://demo.routerplus.com/signup) and completes Clerk authentication and email verification.
2. Saves the first key shown after sign-in.
3. Adds paid credits in [Billing](https://demo.routerplus.com/console/billing). Signup does not grant free credit.

Existing customers sign in at [https://demo.routerplus.com/login](https://demo.routerplus.com/login) with the same verified email. They create a key at [API keys](https://demo.routerplus.com/console/keys), or copy an existing one from its row.

If purchase matching is offered for your account, Billing shows the cap and remaining matching credit.

Signup works only in the browser. No route signs up a user or sends a magic link. When `TM_API_KEY` is available, the agent can configure clients and make API calls.

Rules for the key:

1. Keep the key on the server. Read it from the `TM_API_KEY` environment variable or from your secret store. Never commit it, log it or put it in a URL.
2. Do not put the key in a browser or a mobile app. Anyone can copy it from there. The gateway accepts cross-origin requests, so a copied key works from any web page. If a browser-only internal tool must call the gateway, give the tool its own key with a low monthly spend cap.
3. Use one key for each service and each environment. RouterPlus keeps spend, caps and logs for each key. You can disable one key and the others continue to work.
4. Use a console key in production. A trial key allows only 20 requests per minute.
5. To rotate a key, create the new key, deploy it, then disable the old key in the console. A disabled key stops working within about 5 seconds. The console cannot enable it again. An owner or admin can enable it again with the identity API.

Details: [Authentication](authentication.md).

## 2. Connect the SDK

Change two values in the client that you already use: the base URL and the key. Set both in code. Do not depend on `OPENAI_BASE_URL` or `ANTHROPIC_BASE_URL` in the environment. If that variable is missing, the SDK sends the request and its key to the provider's own API.

Python, the OpenAI SDK:

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://demo-api.routerplus.com/v1",
    api_key=os.environ["TM_API_KEY"],
    # Optional: these name your app in usage analytics. They carry no content.
    default_headers={"HTTP-Referer": "https://your-app.example", "X-Title": "Your App"},
)
```

TypeScript, the OpenAI SDK:

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://demo-api.routerplus.com/v1",
  apiKey: process.env.TM_API_KEY,
  defaultHeaders: { "HTTP-Referer": "https://your-app.example", "X-Title": "Your App" },
});
```

With the Anthropic SDK, the base URL has no `/v1`:

```python
import os
import anthropic

client = anthropic.Anthropic(base_url="https://demo-api.routerplus.com", api_key=os.environ["TM_API_KEY"])
```

```typescript
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({ baseURL: "https://demo-api.routerplus.com", apiKey: process.env.TM_API_KEY });
```

### Which surface to call

Every chat model answers on both surfaces. When your surface matches the format of the route that serves the request, the gateway forwards the request as sent, with all provider features. The native surface of each model family's first route:

| Model ids | Native surface | Features that need it |
|---|---|---|
| `claude-*` | `/v1/messages` (Anthropic SDK) | `thinking`, server tools such as web search, PDFs given by URL |
| `gpt-*` | `/v1/chat/completions` (OpenAI SDK) | `response_format`, `reasoning_effort`, `seed`, `logprobs`, custom tools |

On the other surface, the gateway translates:

- A field that the translation cannot carry is dropped and named in the `x-tm-dropped-params` response header.
- If the drop would change the answer, the request is a 400 that names the field. Before it refuses, the gateway tries another provider of the same model that speaks your format.
- The `x-tm-provider` header names the provider that served you.
- The catalog gives each model's native format in `providers[0].dialect`.
- If your code sends only text, images, PDFs and function tools, either surface works for every model.
- Prompt caching marks (`cache_control`) reach Claude from both surfaces.

Details: [Wire compatibility](compat.md).

## 3. Call models

### Rules for every request

1. **Use exact model ids.** Copy them from `GET /v1/models` or `/api/models.json`. Claude and GPT ids have no prefix: `claude-sonnet-5`, `gpt-4o-mini`. Decision models keep their author prefix: `typesafe/jev-1.13`. Variants such as `:free` do not exist. A dated snapshot of a listed model, such as `claude-haiku-4-5-20251001`, also works. It routes and bills as its model.
2. **Keep model ids in one place in your configuration.** Check them against `GET /v1/models` when your service starts.
3. **Use the right route for the output.** Chat models answer on the two chat surfaces. Image models answer only on `POST /v1/images/generations`, and decision models only on `POST /v1/decisions`. The catalog field `output_modalities` (`text`, `image` or `decisions`) tells you which is which. A decision model also takes its own question format. The catalog field `decisions_wire` in `/api/models.json` names it (`systemone` for Jev, the Bespoke Nimble models, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider v1 27B, GPT-6 Luna Decisions, Sage and ScaleDown Classify-1, and `gliner`). [POST /v1/decisions](/docs/api-decisions) describes each one. Do not guess the format from the model's name.
4. **Send `max_tokens` every time**, from 1 to the model's declared `max_output_tokens` (16,384 on GPT-4o; 128,000 on the Claude 5 models). When a model declares no maximum, the maximum is 32,768.
   - If you leave it out, the gateway writes the model's own maximum into the request. When needed, it lowers that value so that the input and the output fit the model's context window.
   - A value above the model's maximum is a 400.
   - While the call runs, the gateway holds balance for the output limit that you send (at most 32,768 tokens when you send none). A realistic value helps on a small balance and lets more calls run at once.
5. **Ask for one answer.** `n` above 1 is a 400. Make separate calls for several samples.
6. **Send the full conversation on every turn.** The gateway keeps no conversation state between calls. See [Data policy](data-policy.md) for how request data is handled.
7. **Keep `temperature` at 1 or less** in code that can reach Claude models. On a translated request, a higher value is a 400.
8. **Refuse drops where they matter.** A field outside the forwarded set is dropped and named in `x-tm-dropped-params`. Send `"provider": {"require_parameters": true}` to get a 400 instead of a drop.
   - The 400 comes from the first route that would drop a field. The gateway does not go on to a later route that could carry it, so combine it with `only` or `order`.
   - Example: the `anthropic` route drops `seed`. A Claude request on `/v1/chat/completions` that sends `seed` and `"require_parameters": true` is a 400.

### Streaming

Stream answers that a person watches, and long answers.

- The OpenAI surface sends a role chunk, content chunks, a finish chunk, a usage chunk with an empty `choices` list and `usage.cost`, then `data: [DONE]`.
- The Anthropic surface sends the native event sequence, with usage and `cost` in the final `message_delta`.
- If the provider fails after the answer starts, the stream ends with one error event and nothing after it (no `[DONE]`). The SDKs raise an error. RouterPlus bills the partial answer. The gateway never continues an answer on another provider. To try again, send the full turn again.
- During a silence of 15 seconds or more, the gateway sends a keep-alive: an SSE comment on the OpenAI surface, a `ping` event on the Anthropic surface. The SDKs skip them. If you parse SSE yourself, skip them too.
- The first response headers must arrive within 20 seconds. If they do not, the gateway tries the next provider. A request can run for at most 450 seconds.
- To stop an answer, close the connection. The provider stops, and you pay for the usage that the provider reported.
  - If the provider reported no final usage, you pay for the input and for the output already streamed to you. The input count is the provider's count when it sent one, or else the gateway's estimate.
  - If nothing streamed yet, you pay for the input only.

Python, the OpenAI SDK, keeping the cost and handling a failure in the middle of an answer:

```python
from openai import APIError

parts, usage = [], None
try:
    stream = client.chat.completions.create(
        model="claude-sonnet-5",
        max_tokens=2048,
        stream=True,
        messages=messages,
    )
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            parts.append(chunk.choices[0].delta.content)
        if chunk.usage:  # the usage chunk; when usage is known, it also comes before an error
            usage = chunk.usage
except APIError:
    # The answer failed part way. What arrived is billed. Do not join a retry onto
    # `parts`: send the whole turn again, or show the error.
    raise
print("".join(parts), "cost:", usage.cost if usage else None)
```

Details: [Streaming](streaming.md).

### Calls that do not stream

- Set the client timeout above 450 seconds. The SDK default of 10 minutes works.
- The Anthropic SDKs refuse a call that does not stream when `max_tokens` is above about 21,000, unless you pass an explicit `timeout`. Stream those calls instead.
- If you close the connection, the gateway stops the provider. Nothing was streamed to you, so you pay for the input only.
- An image render does not stop when you close the connection. It finishes and is billed at the usage that the provider reports.
- A call that sends no `max_tokens` can reach the 450-second budget on a long answer. It then fails with a 504, billed $0. Send `max_tokens`, or stream.

### Tools, reasoning and structured output

- **Function tools** work on both surfaces for every chat model. The gateway translates tool calls and tool results.
- **Custom (free-form) tools**, `type: "custom"`, work on `/v1/chat/completions` for the GPT-6 and GPT-5.6 models. Their calls come back as `tool_calls` entries of `type: "custom"`.
- **Anthropic server tools** (web search, computer use, bash, text editor) work only for Claude models on `/v1/messages`, while the `anthropic` route serves them. Web search costs $10 per 1,000 searches (the Anthropic price). RouterPlus adds that cost to the tokens in `usage.cost`. The `anthropic` route refuses `mcp_servers` and `container`. A request with either one is a 400 that names the field.
- **`thinking` and `output_config.effort`** work on `/v1/messages` for every reasoning model.
  - For Claude they pass through as sent.
  - For a model that an OpenAI-format provider serves (GLM 5.3, Kimi K3, DeepSeek), effort becomes `reasoning_effort`. `adaptive` thinking leaves the model's reasoning on, and `disabled` turns it off ([details](/docs/compat#reasoning-and-structured-output-on-openai-dialect-routes)).
  - On `/v1/chat/completions`, Claude's thinking arrives as `reasoning_content`.
  - On `/v1/messages`, an answer from a non-Claude model has its reasoning as `thinking` blocks, streamed or not. A non-streamed answer starts with the reasoning. The block is readable. When you set `thinking.display` to `"omitted"` or `"updates"` (as Claude Code does), the block is empty and the text is in its `signature`.
  - You can send `thinking` blocks back in the next turn.
  - Claude Code works on these models with no change on your side. Set its model to the catalog id, for example `z-ai/glm-5.3`, with `z-ai/glm-5.3-flash` as its small model.
- **`response_format`** needs an OpenAI-format provider: GPT ids and the open models. On `/v1/messages`, `output_config.format` with a JSON schema reaches an open model's provider as a strict json_schema `response_format`.
- **JSON from a Claude model on `/v1/chat/completions`:** give the model one tool whose input schema is the JSON you want, and read the tool input. Claude Sonnet 4.5 and Haiku 4.5 take a forced `tool_choice`. The Claude 5 models refuse a forced tool choice (`required` or a named tool) with a 400, so use `auto` there.
- Do not end the message list with an assistant message when a Claude model can serve the request from the OpenAI surface. Claude refuses a prefilled answer, so the request is a 400.

### Images in prompts

Images and PDFs in user messages work on both surfaces. The gateway translates these parts between the OpenAI and Anthropic formats:

- JPEG, PNG, GIF and WebP images, as a data URL or an `http(s)` URL
- PDFs, as a data URL

Rules:

- When the gateway must translate, these parts are a 400 that names the part: an audio or video part, a file given by id, a file other than a PDF, and an image in an assistant message or a tool result.
- Send an image, document, audio or video part only to a model that takes it. `architecture.input_modalities` in `GET /v1/models` lists each model's inputs. A part that the model does not take is a 400 that names the part, and nothing is billed.
- While the call runs, each image or document part counts as 65,536 tokens against your limits and your balance hold.

### Other routes

- **Images:** `POST /v1/images/generations` takes the OpenAI Images API body. The image comes back as base64 in the response. `response_format: "url"` is a 400. `n` is 1 unless the model allows more: see `supported_parameters.n` in `/api/models.json` (GPT Image models allow up to 4). There is no streaming. A failed render bills nothing. See [POST /v1/images/generations](api-images.md).
- **Decisions:** `POST /v1/decisions` sends a state and typed questions to a decision model. It returns one typed answer for each question, for example a choice, a score or a yes/no probability. Use it for routing, classification and moderation points in your code. It replaces a chat model whose output you parse. The models:
  - `typesafe/jev-1.13` (Jev, from TypeSafe)
  - `bespokelabs/nimble-v4` and `bespokelabs/nimble-v4-max` (Bespoke Nimble v4 and v4 Max, from Bespoke Labs; `bespokelabs/nimble-latest` lets Bespoke pick the model)
  - `cloudflare/clef` and `cloudflare/clef-flash` (Clef and Clef-flash, from Cloudflare)
  - `routerplus/decider-2b` and `routerplus/kev-4b` (Decider 2B and Kev 4B, from RouterPlus)
  - `perplexity/pplx-decider-v1-27b` (Perplexity Decider v1 27B, from Perplexity)
  - `openai/gpt-6-luna-decisions` (GPT-6 Luna Decisions, from OpenAI; `gpt-6-luna` is OpenAI's chat model, not this one)
  - `levanto/sage-1.2` (Sage, from Levanto)
  - `scaledown/classify-1` (ScaleDown Classify-1, from ScaleDown)
  - `fastino/gliner-2.5-decide` (GLiNER-2.5-Decide, from Fastino)

  Rules for decision models:
  - The question format follows the model. System One questions (Jev, Nimble, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider, GPT-6 Luna Decisions, Sage and ScaleDown Classify-1) have a `type`. GLiNER takes a `schema` of classifications and extractions in place of `questions`. A question in another model's format is a 400.
  - GLiNER's numbers are confidences. They are not calibrated probabilities.
  - Perplexity Decider bills the state once per question, so ask it only the questions you need.
  - GPT-6 Luna Decisions adds about 150 input tokens for each question. It can decline one question, and that answer is `{"type": "refusal"}`, so check each answer's `type` before you read its value.
  - The gateway never fails over from one model to another. See [POST /v1/decisions](api-decisions.md).
- **Token counting:** `POST /v1/messages/count_tokens` is free, for models that have an Anthropic-format provider. It counts against your requests per minute.
- **Your deployed models:** an id such as `tm/<name>-v1` from [Optimize](model-search.md) works on `/v1/chat/completions` only, with keys of the organization that deployed it. With `stream: true`, it returns the finished answer as one chunk.

## 4. Pin providers (only when you must)

How routing works without a pin:

- By default, RouterPlus orders a model's providers by the estimated cost of one good answer. Providers close to the best share the traffic. Until providers have measurements, the cheapest provider goes first. See [Provider routing](provider-routing.md).
- Claude models go to `anthropic` first, and GPT models to `openai`.
- The gateway moves to the next provider only when the current one fails before it sends any output, or has no capacity left.
- Each provider charges its own price. You pay the price of the provider that serves the request: the one you name with `only`, or else the one that routing selects, fallbacks included. `/api/models.json` gives each provider's live price in `providers[].pricing_usd_per_million`.

Pin a provider in these cases only:

- A rule requires one provider.
- A feature exists at one provider only.
- You compare providers in a test.

`only` and `"allow_fallbacks": false` remove fallbacks. `order` keeps every fallback.

### The `provider` object

Send it in the request body on `/v1/chat/completions` or `/v1/messages`. The gateway reads it and never forwards it. An unknown key inside it is a 400.

| Field | Value | Effect |
|---|---|---|
| `order` | provider ids | Try these providers first, in this order. The others stay as fallbacks. |
| `only` | provider ids | Use only these providers. |
| `ignore` | provider ids | Never use these providers. |
| `allow_fallbacks` | boolean | `false` keeps only the first route. If that route is resting after failures, the request fails with 503 and `retry-after` and does not move on. |
| `sort` | `"price"`, `"throughput"`, `"latency"` or `"quality"` | Turns off the default order. `"price"`: marketplace routes cheapest first, after `order`, by the prompt and completion prices per million at each provider's live price. A route billed at the cost the provider reports goes after the priced routes. `"throughput"`: the highest measured output speed first. `"latency"`: the lowest measured time to first token first. `"quality"`: the highest share of well-formed tool calls first. Without `sort`, the default order applies (see [Provider routing](provider-routing.md)), unless your key's [routing policy](routing-policies.md) orders its routes. |
| `preferred_min_throughput`, `preferred_max_latency` | number (p50) or `{"p50", "p75", "p90", "p99"}` | Routes below the throughput (tokens per second) or above the latency (seconds to first token) go last. No route is left out. |
| `quantizations` | list of `int4`, `int8`, `fp4`, `mxfp4`, `nvfp4`, `fp6`, `fp8`, `mxfp8`, `fp16`, `bf16`, `fp32`, `unknown` | Only routes that serve the model at these quantizations. A route that declares none is `unknown`. |
| `max_price` | `{"prompt": USD, "completion": USD}` per million, at least one of them, each from 0 to 1,000,000 | Leaves out a marketplace route whose prompt or completion price is above the cap. See the rules below. |
| `require_parameters` | boolean | `true`: when the first route that would serve you would drop one of your request fields, the request is a 400. The gateway does not try a later route. |
| `connections`, `connection_order`, `funding` | ids, `"byok"` or `"house"` | Choose among your own provider connections and who pays. See [Routing policies](routing-policies.md). |
| `regions`, `zdr`, `data_collection` | region list, boolean, `"allow"` or `"deny"` | Hard filters on the provider's published terms. `"zdr": true`: the provider keeps no prompts. `"data_collection": "deny"`: the provider does not train on or keep them. A route on your own connection passes with operator evidence. A route whose terms are not known is left out. |

Rules for `max_price`:

- Cached input, cache write and reasoning prices are not compared, so they are not capped.
- The price compared is the price that you would pay, after the discount of a decision model.
- If no route is left, the request is a 404 that names `provider.max_price`. This also happens when a route that takes your images, files or `response_format` is above the cap and the routes within it do not take them.
- If the price cannot be read, the request is a 503 with `retry-after`.
- A route billed at the cost the provider reports (some image and video models) has no price to compare, so `max_price` leaves it out. When every route left out is such a route, the request is a 400: `provider.max_price is not available for model "<model>": it is billed at the cost the provider reports`.

`order`, `only` and `ignore` take route labels. Marketplace route labels are the `providers[].id` values in `/api/models.json`, for example `anthropic` and `openai`. Your own connections use their profile: `openai`, `anthropic`, `azure` or `bedrock`.

- A label that matches a route label selects or orders that route, even when that route does not serve the model. `only: ["anthropic"]` means the Anthropic route, so on a GPT model it leaves no route.
- Any other label matches no route: it selects, orders or skips nothing. An `only` list without a route label leaves no route.

| Goal | `provider` |
|---|---|
| Claude only from Anthropic's own API | `{"only": ["anthropic"]}` |
| Fail instead of falling back to another provider | add `"allow_fallbacks": false` |

Check a pin before you ship it. `POST /v1/route` returns the ordered candidates, and each excluded route with its reason. It calls no provider and costs nothing. If your controls exclude every route, a real request returns 404 `model_unavailable`. This is the same error as for an unknown model. When a pinned request gets a 404, run `/v1/route`.

```bash
curl -s https://demo-api.routerplus.com/v1/route \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-5","provider":{"only":["anthropic"]}}'
```

Python: the OpenAI SDK sends fields it does not know through `extra_body`.

```python
reply = client.chat.completions.create(
    model="claude-haiku-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={"provider": {"only": ["anthropic"]}},
)
```

TypeScript: pass a variable, so the SDK keeps the field its types do not know.

```typescript
const request = {
  model: "claude-haiku-4-5",
  max_tokens: 1024,
  messages: [{ role: "user" as const, content: "Hello" }],
  provider: { only: ["anthropic"] }, // read by the gateway, never forwarded
};
const reply = await client.chat.completions.create(request);
```

Check each answer with its response headers:

| Header | Value |
|---|---|
| `x-tm-provider` | The provider |
| `x-tm-served-by` | The provider that ran the request, when a route reports one |
| `x-tm-upstream-model` | The id the provider received, when it differs from yours |
| `x-tm-attempts` | The number of provider attempts |

`GET /v1/generation?id=<x-request-id>` shows the route of every attempt.

To pin for an organization, workspace or key, save a routing policy with the policy API. The policy API takes a console session, so a human sets it up. A request's `provider` object can narrow a saved policy. It can never add routes. See [Routing policies](routing-policies.md).

### Your own provider keys (BYOK)

You can route through your own OpenAI, Anthropic or Azure key, or an AWS Bedrock role. A human adds it one time in the console, in Access > Provider connections. Your code still calls the gateway with a RouterPlus key. The provider bills you directly, and `usage.cost` is 0. A key bound to a connection can call only that connection's model ids. It does not fall back to marketplace supply unless a routing policy allows it. See [Bring your own key](byok.md).

## 5. Keep sessions on one route and caches warm

A prompt cache saves money and time only when the next request reaches the same provider with the same prompt start.

- The gateway keeps no session state. Your app sends the full conversation on every turn.
- A conversation stays on the provider that served its first turn while that provider can serve. Requests that share a long prompt prefix (system prompt and tools) go to the same provider across sessions.
- A turn moves to the next route in three cases:
  - The first route fails before it sends output. One failure is enough.
  - The route's capacity pool is full.
  - Its provider's 429 pause is still running. The pause lasts for the provider's `retry-after`, 1 to 60 seconds.
- After two failures in a row, later turns skip the route for 30 seconds. A turn that moves can miss the cache one time.
- On a follow-up turn, the gateway first waits up to 5 seconds for a full pool. See [Waiting for the first route](#waiting-for-the-first-route).
- If you send a session id, the gateway gives it to each provider in that provider's own field. See [Sessions and caching](#sessions-and-caching). Claude Code's `x-claude-code-session-id` header counts, with no change on your side.

Rules for a high cache hit rate:

1. Keep the start of the prompt byte-identical: the same system prompt, the same tool definitions in the same order, and earlier messages unchanged. Only append.
2. Put content that changes (the time, ids, documents retrieved for this turn) after the stable part, at the end.
3. Keep the model and the `provider` object the same for every turn of the session.
4. For Claude, put `cache_control` on the block that ends the stable part (a system, tool or message block), or send the top-level `cache_control` field for automatic caching. Both surfaces keep the marks for Anthropic: see [Cache marks](#cache-marks). On marketplace routes the cache lasts 5 minutes, and `ttl: "1h"` is removed. If you send no marks, the gateway turns on automatic caching for a request with tools or earlier turns: see [Automatic caching on Claude](#automatic-caching-on-claude).
5. OpenAI caches long repeated prompt prefixes on its own, with no request setting.
6. Measure:
   - Cache reads are `usage.prompt_tokens_details.cached_tokens` on the OpenAI surface and `usage.cache_read_input_tokens` on the Anthropic surface.
   - Cache writes are `cache_write_tokens` and `cache_creation_input_tokens`.
   - RouterPlus bills cache reads at the lower `cached_prompt` price, and Claude cache writes at the higher `cache_write` price. The prices are in `/api/models.json`.

Python, the Anthropic SDK, with a cache mark on a long system prompt that does not change:

```python
reply = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": STABLE_INSTRUCTIONS,              # the same bytes on every turn
        "cache_control": {"type": "ephemeral"},   # cache everything up to here
    }],
    messages=history,                             # append only
)
print(reply.usage)  # cache_read_input_tokens, cache_creation_input_tokens and cost
```

The provider caches a prompt start only above a minimum length. A short system prompt shows no cache reads.

### Sessions and caching

#### Session ids

On `POST /v1/chat/completions` and `POST /v1/messages`, the session id is the first valid value among these inputs, in this order. Token counting and images do not use it.

| Order | Input | Kind |
|---|---|---|
| 1 | `session_id` | Top-level body field |
| 2 | `x-session-id` | Header |
| 3 | `x-session-affinity` | Header |
| 4 | `x-claude-code-session-id` | Header that Claude Code sends by itself |

- A valid value has 1 to 256 printable ASCII characters. `prompt_cache_key`, a top-level body field, follows the same rule.
- The gateway ignores an invalid value and uses the next input. An invalid value alone is never an error. `x-tm-dropped-params` names the ignored input as `session_id (invalid)`, `prompt_cache_key (invalid)`, `header:x-session-id (invalid)`, `header:x-session-affinity (invalid)` or `header:x-claude-code-session-id (invalid)`. With `"require_parameters": true`, that makes the request a 400, as any dropped field does.
- Valid session inputs are never named in `x-tm-dropped-params`, and they never trigger `require_parameters`.
- `prompt_cache_key` stays OpenAI's own field. It wins over the session id on OpenAI routes.
- When two valid inputs disagree, the order decides. There is no error.
- Codex's `session-id` header is not read. Codex calls `POST /v1/responses` and sends its conversation id as `prompt_cache_key` in the body, which reaches OpenAI unchanged.

What each route receives:

| Route | Field | Value |
|---|---|---|
| `openai` (marketplace), and `openai` and `azure` (your connections) | `prompt_cache_key` | Your `prompt_cache_key`, or else the session id. Nothing if you sent neither |
| `morphllm` | Header `x-session-id` | The session id, or else your `prompt_cache_key`. Nothing if you sent neither. Morph uses it to pick the region and the worker that hold your cache |
| `anthropic` and `bedrock` | Nothing | Anthropic has no session input. Its cache matches the prompt start |

- Values go to the provider unchanged. A value longer than the provider accepts is replaced by `tm-` followed by 43 base64url characters: a hash of the value.
- The gateway keeps no session table. The id goes to the provider, and the provider routes on it.
- `prompt_cache_retention` goes unchanged to `openai` and `azure` routes, both `"in_memory"` and `"24h"`. On other routes it is dropped and named.

#### End-user ids on marketplace routes

On marketplace routes, the provider gets a hash in place of your end user's id. The raw values do not go to the provider.

- The id is the first string among the body fields `safety_identifier`, `user` and `metadata.user_id`.
- The hash is `tm-` followed by 43 base64url characters, made from your organization id and that id. Without an id, it is made from your organization id alone.
- Each marketplace route gets one field:

| Marketplace route | Field that carries the hash | Fields removed |
|---|---|---|
| `openai`, `azure` | `safety_identifier` | `user` |
| `anthropic` | `metadata.user_id` | `user`, `safety_identifier` |
| every other route (OpenAI-compatible), dedicated capacity included | `user` | `safety_identifier`, `metadata.user_id` |
| `mistral-ai` | none: Mistral has no end-user field | `user`, `safety_identifier`, `metadata.user_id`, each named in `x-tm-dropped-params` when you sent it |

- When a provider blocks an id, it blocks one user of one customer. The RouterPlus account stays open.
- On your own connections, `user` and `metadata` pass unchanged. `safety_identifier` passes unchanged to `openai` and `azure` connections. It is dropped and named on `anthropic` and `bedrock` connections.
- Identity fields that a route uses are never named in `x-tm-dropped-params`. Mistral's route uses none, so it names each one you sent, and `require_parameters` refuses that route.

#### Cache marks

The gateway never adds a cache mark that you did not send. Automatic caching, below, adds only the top-level field, and only when you sent no marks. Block and part marks (`cache_control`):

| Request | Route | What happens to the marks |
|---|---|---|
| `/v1/chat/completions` | Anthropic direct or Bedrock | Each text part keeps its mark. A marked system part turns the system prompt into blocks. A tool message's last mark goes on its `tool_result` block. Other part keys are not forwarded |
| `/v1/chat/completions` | OpenAI direct or Azure | Part marks pass unchanged. OpenAI and Azure ignore them |
| `/v1/messages` | Anthropic direct or Bedrock | Block marks pass unchanged |
| `/v1/messages` | OpenAI direct or Azure | Marks are removed and named, for example `messages[i].content[j].cache_control` or `tools[i].cache_control` |

A top-level `cache_control` field (automatic caching), on either surface:

- Anthropic direct: sent unchanged.
- Bedrock: turned into a mark on the last block that can carry one, counted back from the end of the messages. If no block can carry it, it is dropped and named as `cache_control`.
- OpenAI direct and Azure: dropped and named as `cache_control`.
- Any route: a field that Anthropic would refuse is dropped and named as `cache_control`. Anthropic refuses:
  - a field other than `{"type": "ephemeral"}` with an optional `ttl` of `"5m"` or `"1h"`
  - a fifth mark (when four marks exist already)
  - a 1-hour field after a 5-minute mark
  - a field whose TTL differs from the mark on the last block

On marketplace routes, `ttl: "1h"` is removed from every mark, so the cache lasts 5 minutes. The drop is named as `<path>.cache_control.ttl`, for example `system[0].cache_control.ttl`, `messages[2].content[0].cache_control.ttl`, or `cache_control.ttl` for the top-level field. `ttl: "5m"` passes. Your own connections keep `ttl: "1h"`.

#### Automatic caching on Claude

When a request has no cache marks, the gateway adds the top-level `cache_control: {"type": "ephemeral"}` field (5 minutes) if all of these are true:

- Anthropic's own API serves the request: the `anthropic` route or your own Anthropic connection. Bedrock and dedicated endpoints do not get it.
- The request has no `cache_control` anywhere.
- It has tools, or an assistant turn before its last message.
- The prompt is at least the model's minimum cacheable size: 512 tokens on the Claude 5 models except Sonnet 5, 1,024 on Claude Sonnet 5 and Sonnet 4.5, and 4,096 on Claude Haiku 4.5.
- The request does not carry the header `x-tm-auto-cache: off`.

The response header `x-tm-auto-cache` is `applied`, or `skipped:` with the reason, for example `skipped:single-turn` or `skipped:client-markers`. The first turn pays a cache write (1.25 times the input price). Later turns read the prompt at the cache price. Details: [Wire compatibility](compat.md#automatic-caching-on-claude).

#### Waiting for the first route

On a follow-up turn (an assistant or tool message after a user message), the first route's capacity pool can be full. The gateway then waits for that pool for up to 5 seconds (an operator setting) before it moves the request to the next route. It never waits past the request's deadline. The `x-tm-affinity-wait-ms` response header gives the wait in milliseconds, only when the wait was more than 0.

## 6. Handle errors and retries

Every failure has one stable class, `error_type`. Read it from `error.metadata.error_type` on the OpenAI surface, or `error.error_type` on the Anthropic surface. The `x-tm-error-code` header has it too, but browsers cannot read that header. Never parse the message text. Log the `x-request-id` header with every failure.

| `error_type` | HTTP | Retry | What to do |
|---|---|---|---|
| `rate_limit` | 429 | Yes | Wait `retry-after` seconds, then retry. A capped wait can take a second try. If `x-tm-limit-kind` is `concurrency`, send fewer requests at once. |
| `email_not_verified` | 403 | No | Stop and tell a human to open the verify link in their email. The key starts working within seconds of the click. |
| `insufficient_quota` | 429 | No | Stop and tell a human to add credits or raise a spend cap. On a cap, `x-tm-cap-reset` gives the reset time. |
| `upstream_error` | 5xx | Yes, once, after about 2 seconds | The gateway already tried every other provider it could. If the retry fails too, report it with the `x-request-id`. |
| `upstream_error` | 4xx | No | The provider refused the request. The message has the provider's own words. |
| `gateway_error` | 503 | Yes | Wait `retry-after`. The request did not reach a provider. |
| `gateway_error` | 504 | Yes, once | The deadline passed while the gateway tried providers. An earlier attempt may be billed: check `GET /v1/generation?id=<x-request-id>` first. |
| `gateway_error` | 400 | No | Send one output limit, from 1 to the maximum the message names (32,768 only when no maximum is declared). |
| `gateway_error` | 500 | No | Report it with the `x-request-id`. |
| `model_unavailable` | 404 | No | Fix the model id, or fix your `provider` object (check it with `POST /v1/route`). |
| `model_unavailable` | 502 | Yes, once | The connection to every provider failed. |
| `invalid_request` | 400 | No | Fix the field that the message names. |
| `context_overflow` | 400 | No | Shorten the input, or use a model with a larger context. The body has the standard shape that agents compact on: `error.code` `context_length_exceeded` on the OpenAI surface, `prompt is too long: N tokens > M maximum` on the Anthropic surface. |
| `content_policy` | 400 | No | Change the prompt. The gateway never sends it to another provider. |
| `auth` | 401 | No | Check the header, the key, and whether the key was disabled. A 401, 402 or 403 relayed from every provider is a problem with the RouterPlus account: report it with the `x-request-id`. |
| `request_too_large` | 413 | No | Keep the body under 10 MB. |
| `not_found` | 404 | No | Use a route that exists. The list is in [Errors](errors.md). |

Rules:

1. Let the SDK retry. The official OpenAI and Anthropic SDKs retry 429 and 5xx responses twice by default, and they wait for `retry-after`. Keep that default. Do not add a second retry loop around it. The SDKs also retry an `insufficient_quota` response. That costs nothing, but your code must then stop.
2. There is no idempotency key. A retried request is a new request. If it reaches a provider, it is billed again. A refusal by the gateway's own limits (`x-tm-error-origin: gateway_admission`) costs nothing.
3. Never retry `insufficient_quota`, `invalid_request`, `context_overflow`, `content_policy` or `auth` in a loop. The request or the account must change first.
4. An error in the middle of a stream is final for that answer. Send the full turn again. Never join two partial answers.

Python, the OpenAI SDK, reading the class:

```python
from openai import APIStatusError

try:
    reply = client.chat.completions.create(model=MODEL, max_tokens=1024, messages=messages)
except APIStatusError as e:
    error = e.response.json().get("error") or {}
    kind = (error.get("metadata") or {}).get("error_type") or e.response.headers.get("x-tm-error-code")
    request_id = e.response.headers.get("x-request-id")
    if kind == "insufficient_quota":
        notify_operator(request_id, error.get("message"))  # money or a cap: a retry cannot help
    raise
```

Details: [Errors](errors.md).

## 7. Limits, spend caps and your users

Default limits:

| Scope | Requests per minute | Tokens per minute | In flight at once |
|---|---|---|---|
| Trial key | 20 | 1,000,000 | 8 |
| Console or identity API key | 300 | 1,000,000 | 8 |
| Organization, all keys together | 300 | 1,000,000 | 8 |

Rules:

1. Limit how many requests your code sends at once. By default the organization allows 8 in flight. A ninth gets 429 `rate_limit` with `x-tm-limit-kind: concurrency`. Use a semaphore or a fixed pool of workers.
   - To run more at once, a human first raises the organization and key limits in the console, in Billing > [Rate limits](https://demo.routerplus.com/console/limits). A key's requests per minute cannot go above its ceiling (300 for a console key).
   - Each marketplace provider pool also limits one organization to half of the pool's capacity. That refusal has `x-tm-limit-scope: pool`.
2. Read `x-tm-remaining-rpm` and `x-tm-remaining-tpm` on each response, and slow down before they reach zero. `GET https://demo-api.routerplus.com/v1/limits?model=<id>` returns the limits that apply to your key.
3. While a call runs, token limits count an estimate: one token for each four bytes of the request, rounded up, 65,536 for each image or document part, plus `max_tokens` (at most 32,768 of it). An image sent inline as base64 counts its 65,536 only, not its bytes. The real count replaces the estimate when the call ends. Small requests and a realistic `max_tokens` let more calls run at once.
4. Spend caps are monthly, for each key and for the organization, on the UTC calendar month. A human sets them in the console. A cap refuses with 429 `insufficient_quota` and the `x-tm-cap-reset` header. Give each feature or customer its own key and cap, so a bug or a loop cannot spend everything.

Your users:

- By default, one backend key serves all your users. The gateway does not know who your users are, so enforce per-user quotas in your app.
- `safety_identifier`, `user` and `metadata.user_id` identify your end user to the provider only (see [End-user ids on marketplace routes](#end-user-ids-on-marketplace-routes)). They do not identify, limit or bill anyone on the gateway. Do not put emails or names in them.
- For limits that the gateway enforces per user, use the identity API. Create an `end_user` principal for each user, issue a key for it, and set limits on the principal. Your backend keeps those keys. Never give a key to the user. The identity API takes a console session, not an API key, so a human with an owner or admin role sets it up. See [Workspaces and principals](identity.md).

Details: [Rate limits and spend caps](limits.md) and [Limits and capacity](admission.md).

## 8. Track cost

- Every billed response has `usage.cost` in USD. It is the full charge for the request, including provider attempts that failed before the answer. Store it with the `x-request-id`, the model and your own ids, such as the user and the feature.
- An Anthropic-surface stream that fails in the middle of the answer has no usage in it. Read its cost from `GET /v1/generation?id=<x-request-id>`.
- `GET /v1/usage` returns the balance and the last 20 attempts. For a key bound to its own workspace or principal, the balance fields are `null`. The console's Usage and Logs pages show all keys of the organization.
- Anthropic's web search adds its fee per search to `usage.cost` (see [Pricing and billing](pricing.md#web-search)).
- To check a charge, multiply the token counts by the prices in `/api/models.json`. The gateway rounds the final charge down.

Details: [Pricing and billing](pricing.md) and [GET /v1/usage and /v1/generation](api-usage.md).

## 9. Verify the integration

One cheap call shows the answer, the cost and the headers that the gateway adds:

```bash
curl -sS -i https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"gpt-4o-mini","max_tokens":16,"messages":[{"role":"user","content":"Reply with OK"}]}'
```

Then confirm that the ledger recorded it:

```bash
curl -s -H "Authorization: Bearer $TM_API_KEY" https://demo-api.routerplus.com/v1/usage
```

The integration is done when:

1. The rules in [The short version](#the-short-version) hold in the code.
2. The key is not in the repository, in a client bundle or in a log.
3. Every model id in the configuration appears in `GET /v1/models`.
4. One streamed and one non-streamed test call return `usage.cost`, and both appear in `GET /v1/usage`.
5. The `x-tm-dropped-params` header is absent, or it names only fields that you accept losing.
6. `x-tm-provider` shows the route that you expect.

## Leave instructions for the next agent

Add this block to the `AGENTS.md` or `CLAUDE.md` file of the repository that you integrated, so later coding sessions keep the integration correct:

```markdown
## LLM calls: RouterPlus gateway

- Every LLM call goes through the RouterPlus gateway. Full guide: https://demo.routerplus.com/docs/agent-integration.md
- OpenAI SDK base URL: https://demo-api.routerplus.com/v1. Anthropic SDK base URL: https://demo-api.routerplus.com. Key: the TM_API_KEY environment variable, on the server only.
- Model ids live in configuration. Each one must exist in https://demo.routerplus.com/api/models.json. Never invent or alias an id.
- Send max_tokens on every request (1 to the model's max_output_tokens in models.json; 32768 only when none is declared). Never send n above 1.
- Claude models: /v1/messages, with cache_control on the stable start of the prompt. GPT models: /v1/chat/completions.
- Keep the start of the prompt byte-identical across turns. Only append.
- Send one id per conversation as the x-session-id header (or session_id in the body). Providers then keep its cache warm.
- Provider pins go in the request body "provider" object (only, ignore, order, allow_fallbacks, require_parameters). Check a pin with POST https://demo-api.routerplus.com/v1/route.
- Errors: switch on error_type. rate_limit: wait retry-after, then retry; a capped wait can take a second try. insufficient_quota: stop and tell a human. Never retry invalid_request, context_overflow or content_policy.
- No idempotency key: a retry is a new, billed request. Add no retry loop on top of the SDK's own retries.
- At most 8 requests in flight per organization, unless the limit was raised.
- Log x-request-id and usage.cost for every call.
- Not available: moderation. POST /v1/responses serves OpenAI models; Claude and the other models are a preview there. On it, store is always false and previous_response_id is a 400.
```

## Steps only a human can do

- Complete browser signup through Clerk, including email verification, then buy credits in Billing. Signup and card verification alone do not add credit.
- Fix billing when `insufficient_quota` appears: add credits, or raise a spend cap in the console.
- Sign in to the console to create keys, set spend caps and change limits. See [Console](/docs/console).
- Add your own provider keys (BYOK), save routing policies, and set up workspaces and end-user principals.
- Approve large test runs. Every call is billed.

## Where next

- [Quickstart](/docs/quickstart) - the first streamed call, step by step
- [Errors](/docs/errors) - every error class, and what to do about it
- [Routing policies](/docs/routing-policies) - the provider object, saved policies and BYOK routes
- [Streaming](/docs/streaming) - frame order, keep-alives and failures in the middle of an answer
- [Rate limits & spend caps](/docs/limits) - the numbers behind every 429
- [Wire compatibility](/docs/compat) - which fields pass, which translate and which drop