# RouterPlus: all documentation

> One API key for many LLM providers. OpenAI-compatible /v1/chat/completions and
> Anthropic-compatible /v1/messages on one gateway. Pass-through pricing,
> per-request cost on the wire, honest failover that never switches mid-answer.

Gateway base URL: https://demo-api.routerplus.com
Create an account in your browser: https://demo.routerplus.com/signup. Complete secure verification, then create an API key at https://demo.routerplus.com/console/keys.

The index is https://demo.routerplus.com/llms.txt. Every page follows, in sidebar order, each under its source URL.

---

Source: https://demo.routerplus.com/docs/overview.md

# Overview

RouterPlus is an LLM gateway. One API key and one prepaid balance give you models from many providers, over an OpenAI-compatible API and an Anthropic-compatible API.

Every chat model works on both APIs. RouterPlus translates requests, streams and errors between them. You pay the listed per-token prices, and each billed response states its cost in `usage.cost`.

## Get a key

1. Open [Sign up](https://demo.routerplus.com/signup). Complete Clerk authentication and email verification.
2. Save the first API key shown after sign-in.
3. Buy credits in [Billing](https://demo.routerplus.com/console/billing). Signup does not add free credit.

If you have an account, [sign in](https://demo.routerplus.com/login) and create another key at [API keys](https://demo.routerplus.com/console/keys).

If purchase matching is offered for your account, Billing shows the cap and remaining matching credit.

## Make a call

```bash
export TM_API_KEY=tm_vk_...

curl -N https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-haiku-4-5","stream":true,"max_tokens":60,"messages":[{"role":"user","content":"hello"}]}'
```

The last usage chunk of the stream has the token counts and `cost`: the USD amount this request debited.

With an SDK, change only the base URL and the key:

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])
r = client.chat.completions.create(
    model="claude-haiku-4-5",   # a Claude model over the OpenAI format
    max_tokens=60,
    messages=[{"role": "user", "content": "hello"}],
)
print(r.model_dump()["usage"]["cost"])  # USD debit for this request
```

```python
import os
import anthropic

client = anthropic.Anthropic(base_url="https://demo-api.routerplus.com", api_key=os.environ["TM_API_KEY"])
msg = client.messages.create(
    model="gpt-4o-mini",        # a GPT model over the Anthropic format
    max_tokens=60,
    messages=[{"role": "user", "content": "hello"}],
)
```

## Two APIs, one key

| API | Base URL | Endpoint | Works with |
|---|---|---|---|
| OpenAI-compatible | `https://demo-api.routerplus.com/v1` | `POST /v1/chat/completions`, and `POST /v1/responses` for OpenAI's models | OpenAI SDKs, Codex, Cursor, Cline, Kilo Code, OpenCode, any OpenAI-format client |
| Anthropic-compatible | `https://demo-api.routerplus.com` | `POST /v1/messages` | Anthropic SDKs, Claude Code |

The key works on both APIs, as `Authorization: Bearer` or `x-api-key`. The API you use does not limit the model: OpenAI-format code can call Claude models, and Anthropic-format code can call GPT models. See [Authentication](/docs/authentication).

## How requests behave

- **Cost.** Every billed response has `usage.cost` in USD. It covers the full debit, including attempts that failed over before your answer started. See [Pricing & billing](/docs/pricing).
- **Failover.** Before any output reaches you, RouterPlus fails over to another provider. After output starts, the request stays with that provider. A failure then ends the stream with one error event. See [Routing & failover](/docs/routing).
- **Keep-alives.** During a silence, the stream sends a keep-alive every 15 seconds. See [Streaming](/docs/streaming).
- **No substitution.** An unknown model id returns a 404 that names the id you sent. RouterPlus does not pick another model.
- **Strict requests.** `n>1` returns a 400. Content or a parameter that cannot be translated returns a typed 400 that names the field. RouterPlus drops unknown top-level parameters and names each one in the `x-tm-dropped-params` header. See [Wire compatibility](/docs/compat).
- **Metadata only.** Metering and logs record token counts, timings, outcomes and cost. See [Data policy](/docs/data-policy).

## Gateway endpoints

All gateway endpoints at `https://demo-api.routerplus.com` take your API key.

| Endpoint | What it does |
|---|---|
| `POST /v1/chat/completions` | OpenAI-compatible completions, streamed or not |
| `POST /v1/messages` | Anthropic-compatible messages, streamed or not |
| `POST /v1/messages/count_tokens` | Anthropic token counting, not billed. Only models with an Anthropic-format deployment |
| `POST /v1/responses` | OpenAI Responses API. See [POST /v1/responses](/docs/api-responses) |
| `POST /v1/embeddings` | Text to vectors. See [POST /v1/embeddings](/docs/api-embeddings) |
| `POST /v1/audio/transcriptions`, `POST /v1/audio/speech` | Speech to text and text to speech. See [POST /v1/audio](/docs/api-audio) |
| `POST /v1/images/generations` | OpenAI Images API, base64 in the body. See [POST /v1/images/generations](/docs/api-images) |
| `POST /v1/videos` | Video from a text prompt. See [POST /v1/videos](/docs/api-videos) |
| `POST /v1/decisions` | Typed answers from a decision model. See [POST /v1/decisions](/docs/api-decisions) |
| `/v1/files`, `/v1/batches` | Many requests from one file. See [Batches](/docs/api-batches) |
| `GET /v1/models` | Models your key can call. OpenAI list shape by default, Anthropic shape when you send an `anthropic-version` header |
| `GET /v1/usage` | Balance, credited and spent totals, recent requests with cost |
| `GET /v1/generation?id=<request id>` | Audit of each attempt for one request: tokens, provenance, cost, outcome |
| `POST /v1/route` | Shows how a request would route, without sending it. See [Routing policies](/docs/routing-policies) |
| `GET /v1/limits?model=<id>` | The rate limits that apply to your key. See [Limits and capacity](/docs/admission) |

## Site

| Page | What it does |
|---|---|
| [`/api/models.json`](https://demo.routerplus.com/api/models.json) | Public catalog: ids, prices, context windows, who serves each model. No auth. See [Models & catalog](/docs/models) |
| `/models/<id>` | Each model's providers, prices, latency and uptime over the last 24 hours |
| [`/playground`](https://demo.routerplus.com/playground) | Chat with any catalog model, compare up to three side by side, judge with a decision model, and make images. See [Playground](/docs/playground) |
| [`/model-search`](https://demo.routerplus.com/model-search) | Optimize: find a cheaper model or mix that scores as well as yours on your Braintrust evals, then deploy it. See [Optimize](/docs/model-search) |
| [`/endpoints`](https://demo.routerplus.com/endpoints) | Your [dedicated endpoints](/docs/dedicated-endpoints) and the models you deployed from Optimize. Each has a `tm/...` model id that your keys can call |
| [`/console`](https://demo.routerplus.com/console) | Keys, usage, logs, credits, limits, team and governance. See [Console](/docs/console) |
| `/signup`, `/login` | Browser signup and sign-in through Clerk |
| [`/llms.txt`](https://demo.routerplus.com/llms.txt), [`/llms-full.txt`](https://demo.routerplus.com/llms-full.txt) | The docs index for agents, and every docs page as raw markdown in one file |

## Where next

- [Quickstart](/docs/quickstart) - browser signup to a streamed API call in five steps.
- [Agent integration guide](/docs/agent-integration) - the page an AI agent reads to build RouterPlus into your product.
- [Coding agents](/docs/install) - setup for Claude Code, Codex, Cursor, Cline, Kilo Code, OpenCode and more.
- [Migrate in one prompt](/docs/migration) - move the OpenAI SDK, Anthropic SDK, Claude Code or Codex with one paste.
- [Errors & remediation](/docs/errors) - every `error_type`, what it means and what to do.

> [!TIP]
> If a coding agent does the work, point it at
> [https://demo.routerplus.com/docs/agent-integration.md](https://demo.routerplus.com/docs/agent-integration.md) to build
> RouterPlus into a product, or at [Install](/docs/install) to set up a coding tool.
> [https://demo.routerplus.com/llms.txt](https://demo.routerplus.com/llms.txt) is the index it reads first.

---

Source: https://demo.routerplus.com/docs/quickstart.md

# Quickstart

Go from signup to a streamed, billed model call in five steps. You need a browser, `curl` and an email address.

One RouterPlus key works on two APIs:

| API | Endpoint | Works with |
|---|---|---|
| OpenAI-compatible | `POST https://demo-api.routerplus.com/v1/chat/completions` | OpenAI SDKs, any OpenAI-format client |
| Anthropic-compatible | `POST https://demo-api.routerplus.com/v1/messages` | Anthropic SDKs, Claude Code |

Every chat model works on both APIs. Image models use [POST /v1/images/generations](/docs/api-images).

> [!TIP]
> To set up with a coding agent, give it the runbook on [Coding agents](/docs/install).
> Complete browser signup first, then let the agent configure your client.

## 1. Sign up and save your key

1. Open [Sign up](https://demo.routerplus.com/signup) and complete Clerk authentication and email verification.
2. Save the `tm_vk_` API key shown after your first verified sign-in. You can copy it again later from its row at [API keys](https://demo.routerplus.com/console/keys).
3. Buy credits in [Billing](https://demo.routerplus.com/console/billing) before your first call. Signup does not add free credit.

If purchase matching is offered for your account, Billing shows the cap and remaining matching credit.

If you have an account, [sign in](https://demo.routerplus.com/login) with the same verified email and create a key at [API keys](https://demo.routerplus.com/console/keys). Your balance and keys stay.

You can create an account only in the browser. `POST /v1/signup` is removed. See [Authentication](/docs/authentication).

Export the key:

```bash
export TM_API_KEY=tm_vk_...
```

## 2. Pick a model

```bash
# Public catalog, no auth: ids, prices, context windows, provider retention
curl -s https://demo.routerplus.com/api/models.json

# Models your key can call, in your SDK's list shape
curl -s -H "Authorization: Bearer $TM_API_KEY" https://demo-api.routerplus.com/v1/models
```

`GET /v1/models` returns the OpenAI list shape. It returns the Anthropic shape when you send an `anthropic-version` header. See [Models & catalog](/docs/models).

> [!NOTE]
> An id that is not in the catalog returns a `404` with `error_type: model_unavailable`
> and the id you sent. RouterPlus does not substitute another model.

## 3. Make a streamed call on the OpenAI API

This calls a Claude model in the OpenAI format:

```bash
curl -N https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-haiku-4-5",
    "stream": true,
    "max_tokens": 60,
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'
```

The stream sends a role delta, content deltas, a finish chunk, a **usage chunk**, then `data: [DONE]`. The usage chunk holds the cost:

```json
{
  "prompt_tokens": 13,
  "completion_tokens": 9,
  "total_tokens": 22,
  "prompt_tokens_details": { "cached_tokens": 0, "cache_write_tokens": 0 },
  "completion_tokens_details": { "reasoning_tokens": 0 },
  "cost": 0.000058
}
```

`cost` is in USD. RouterPlus computes it with the same integer micro-USD math as the ledger. It covers the full request, including attempts that failed over before your answer started. You can compute it again from the token counts and the public prices. See [Pricing & billing](/docs/pricing).

With the OpenAI SDK, change only `base_url` and the key:

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

stream = client.chat.completions.create(
    model="claude-haiku-4-5",
    max_tokens=60,
    stream=True,
    messages=[{"role": "user", "content": "Say hello in five words."}],
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
    if chunk.usage:  # the final chunk before [DONE]
        print(f"\ncost: ${chunk.usage.cost}")
```

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://demo-api.routerplus.com/v1",
  apiKey: process.env.TM_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "claude-haiku-4-5",
  max_tokens: 60,
  stream: true,
  messages: [{ role: "user", content: "Say hello in five words." }],
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
```

> [!TIP]
> Before dispatch, RouterPlus reserves the cost of the call: about (estimated input +
> `max_tokens`) at the model's prices. Without `max_tokens` it reserves at most 32,768
> output tokens. After the call, it settles to the real usage. On a small balance, keep
> `max_tokens` low. If the reservation is larger than your balance, you get
> `insufficient_quota` before RouterPlus calls a provider. See [Rate limits & spend caps](/docs/limits).

## 4. Use the same key on the Anthropic API

This calls a GPT model in the Anthropic format:

```bash
curl -N https://demo-api.routerplus.com/v1/messages \
  -H "x-api-key: $TM_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "stream": true,
    "max_tokens": 60,
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'
```

The last `message_delta` event has the usage: `input_tokens`, `output_tokens`, `cache_read_input_tokens`, `cache_creation_input_tokens` and `cost` in USD.

```python
import os
import anthropic

client = anthropic.Anthropic(
    base_url="https://demo-api.routerplus.com",
    api_key=os.environ["TM_API_KEY"],
)

with client.messages.stream(
    model="gpt-4o-mini",
    max_tokens=60,
    messages=[{"role": "user", "content": "Say hello in five words."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="")
```

> [!NOTE]
> Both endpoints accept both auth headers: `Authorization: Bearer` and `x-api-key`.
> Use the one your SDK sends. See [Authentication](/docs/authentication).

## 5. Check the cost

```bash
curl -s -H "Authorization: Bearer $TM_API_KEY" https://demo-api.routerplus.com/v1/usage
```

```json
{
  "balance_usd": 4.999942,
  "credited_usd": 5.0,
  "spent_usd": 0.000058,
  "recent_attempts": [
    {
      "request_id": "5a2e…",
      "deployment": "anthropic",
      "model": "claude-haiku-4-5",
      "outcome": "completed",
      "usage_provenance": "observed",
      "input_tokens": 13,
      "output_tokens": 9,
      "cost_usd": 0.000058,
      "billing_source": "house",
      "at": "2026-09-04T10:14:03.201Z"
    }
  ]
}
```

The example is shortened. Each attempt also has its price snapshot and inference cost. `recent_attempts` lists your last 20 attempts. See [GET /v1/usage](/docs/api-usage).

For the full audit of one request, pass the `x-request-id` header of any response. The audit shows every attempt, including failovers, cache splits, and reserved and settled cost:

```bash
curl -s -H "Authorization: Bearer $TM_API_KEY" \
  "https://demo-api.routerplus.com/v1/generation?id=REQUEST_ID"
```

Both endpoints return metadata only: token counts, timings, outcomes and cost. See [Data policy](/docs/data-policy).

You can also see usage and each request in the console: open [Usage](https://demo.routerplus.com/console/usage) or [Logs](https://demo.routerplus.com/console/logs).

## Rules to know

- An unknown model id returns a `404`. RouterPlus does not substitute another model. See [Routing & failover](/docs/routing).
- `n > 1` returns a `400 invalid_request`.
- The first key shown after verified sign-in starts at 20 requests per minute. Until your organization buys credit, all its keys together stay at 20 requests per minute. After the first purchase, keys you create in the console get 300. Every organization also has token-per-minute and concurrency limits. See [Rate limits & spend caps](/docs/limits).
- Every error body has a stable `error_type`, on both APIs, in your SDK's error shape. See [Errors](/docs/errors).

## Next steps

- [Agent integration guide](/docs/agent-integration) - the page an AI agent follows to build RouterPlus into your product
- [Coding agents](/docs/install) - set up Claude Code, Codex, Cursor, Cline or OpenCode
- [Migrate](/docs/migration) - move an existing codebase with one prompt
- [Streaming](/docs/streaming) - event order, keep-alives and mid-stream failures
- [Pricing & billing](/docs/pricing) - the math behind `usage.cost`

---

Source: https://demo.routerplus.com/docs/agent-integration.md

# Agent integration guide

This page is for an AI coding agent that builds RouterPlus into a product. It covers these
steps in order: the key, the SDK, model calls, provider pinning, sessions and caching, errors,
limits and cost. Each step links to the reference page for details.

- Every docs page is also raw markdown. Add `.md` to its URL, for example
  `https://demo.routerplus.com/docs/errors.md`.
- The index of all pages is `https://demo.routerplus.com/llms.txt`.
- All pages in one file: `https://demo.routerplus.com/llms-full.txt`.

> [!NOTE]
> **For agents.** Do not guess model ids, request fields or headers. Copy them from this page or from the live catalog. If an error message disagrees with your plan, trust the error message and read the page it names. Some steps need a person: see [Steps only a human can do](#steps-only-a-human-can-do). Stop and ask at those steps.

## The short version

1. Point the OpenAI SDK at `https://demo-api.routerplus.com/v1`, or the Anthropic SDK at `https://demo-api.routerplus.com`. Read the key from `TM_API_KEY`, on the server only.
2. Copy model ids from the catalog. The gateway never substitutes a model. An unknown id is a 404.
3. Send `max_tokens` on every request, from 1 to the `max_output_tokens` of the model. Without it, the model gets its own maximum (128,000 tokens on the Claude 5 and GPT-6 models).
4. For provider features such as server tools, call Claude models on `/v1/messages` and other models on `/v1/chat/completions`.
5. Keep the start of the prompt identical from turn to turn, and only append. This keeps prompt caches warm.
6. Pin a provider only when you must, with the `provider` object. Check the plan first with `POST /v1/route`.
7. Switch on `error_type`, never on message text. Retry `rate_limit` after `retry-after`. Stop on `insufficient_quota` and tell a human.
8. There is no idempotency key. A retry is a new request. It is billed again if it reaches a provider.
9. Keep at most 8 requests in flight per organization, unless you raised that limit.
10. Log `x-request-id` and `usage.cost` for every call.

## At a glance

| Item | Value |
|---|---|
| OpenAI-compatible base URL | `https://demo-api.routerplus.com/v1`, for `POST /v1/chat/completions`, and `POST /v1/responses` for OpenAI's models |
| Anthropic-compatible base URL | `https://demo-api.routerplus.com`, for `POST /v1/messages` (the SDK adds `/v1`) |
| API key | `tm_vk_` followed by 48 hex characters |
| Auth header | `Authorization: Bearer <key>` or `x-api-key: <key>`, on every route |
| Model list | `GET https://demo-api.routerplus.com/v1/models` with your key, or `https://demo.routerplus.com/api/models.json` (public, with prices) |
| Cost of a call | `usage.cost`, in USD, in every billed response |
| Audit of a call | the `x-request-id` response header, then `GET https://demo-api.routerplus.com/v1/generation?id=<id>` |
| Other routes | `POST /v1/images/generations`, `POST /v1/decisions`, `POST /v1/embeddings`, `/v1/audio`, `/v1/batches`. See [Overview](overview.md#gateway-endpoints) |

### What is not available

Do not build on these:

- Moderation. The path returns 404 `not_found`.
- Stored responses on the Responses API. `store` is always `false`, and `previous_response_id` is a 400. The Responses API serves the OpenAI models. Claude and the other models are a preview there. See [POST /v1/responses](api-responses.md).
- Server-side conversation memory and idempotency keys. Session ids exist: see [Sessions and caching](#sessions-and-caching).
- More than one answer per request. `n` above 1 is a 400.
- Model aliases. The one exception is the `rp-` names of the Claude models, for Cursor. In code, use the catalog ids. A `models` list is a fallback list. See [Model fallback](model-fallback.md).
- Other sort values. `provider.sort` takes only `"price"`, `"throughput"`, `"latency"` and `"quality"` (or the object form `{"by": ..., "partition": ...}`). Any other value is a 400.
- `provider.sort` and `provider.max_price` on a dedicated endpoint. Both are a 400 there, because the endpoint's own constraints apply.
- A price filter on the model list. `GET /v1/models?provider=...` with `max_price` is a 400 that names `provider.max_price`, because the list does not compare prices. Set `max_price` on each request. The list accepts `sort`, but `sort` changes nothing there.
- Provider keys or endpoints in a request. `api_key`, `base_url` and `connection_id` in the body are a 400.

## 1. Get a key and keep it safe

| Key | Where it comes from | Requests per minute | Use it for |
|---|---|---|---|
| First trial key | Shown after verified browser sign-in through Clerk | 20 | The first test calls |
| Console key | [https://demo.routerplus.com/console/keys](https://demo.routerplus.com/console/keys) | 300 | Production, staging and CI |
| Identity API key | `POST https://demo.routerplus.com/api/identity/keys` | 300 | One key for each end user or service (step 7) |

To get a key, the user:

1. Opens [Sign up](https://demo.routerplus.com/signup) and completes Clerk authentication and email verification.
2. Saves the first key shown after sign-in.
3. Adds paid credits in [Billing](https://demo.routerplus.com/console/billing). Signup does not grant free credit.

Existing customers sign in at [https://demo.routerplus.com/login](https://demo.routerplus.com/login) with the same verified email. They create a key at [API keys](https://demo.routerplus.com/console/keys), or copy an existing one from its row.

If purchase matching is offered for your account, Billing shows the cap and remaining matching credit.

Signup works only in the browser. No route signs up a user or sends a magic link. When `TM_API_KEY` is available, the agent can configure clients and make API calls.

Rules for the key:

1. Keep the key on the server. Read it from the `TM_API_KEY` environment variable or from your secret store. Never commit it, log it or put it in a URL.
2. Do not put the key in a browser or a mobile app. Anyone can copy it from there. The gateway accepts cross-origin requests, so a copied key works from any web page. If a browser-only internal tool must call the gateway, give the tool its own key with a low monthly spend cap.
3. Use one key for each service and each environment. RouterPlus keeps spend, caps and logs for each key. You can disable one key and the others continue to work.
4. Use a console key in production. A trial key allows only 20 requests per minute.
5. To rotate a key, create the new key, deploy it, then disable the old key in the console. A disabled key stops working within about 5 seconds. The console cannot enable it again. An owner or admin can enable it again with the identity API.

Details: [Authentication](authentication.md).

## 2. Connect the SDK

Change two values in the client that you already use: the base URL and the key. Set both in code. Do not depend on `OPENAI_BASE_URL` or `ANTHROPIC_BASE_URL` in the environment. If that variable is missing, the SDK sends the request and its key to the provider's own API.

Python, the OpenAI SDK:

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://demo-api.routerplus.com/v1",
    api_key=os.environ["TM_API_KEY"],
    # Optional: these name your app in usage analytics. They carry no content.
    default_headers={"HTTP-Referer": "https://your-app.example", "X-Title": "Your App"},
)
```

TypeScript, the OpenAI SDK:

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://demo-api.routerplus.com/v1",
  apiKey: process.env.TM_API_KEY,
  defaultHeaders: { "HTTP-Referer": "https://your-app.example", "X-Title": "Your App" },
});
```

With the Anthropic SDK, the base URL has no `/v1`:

```python
import os
import anthropic

client = anthropic.Anthropic(base_url="https://demo-api.routerplus.com", api_key=os.environ["TM_API_KEY"])
```

```typescript
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({ baseURL: "https://demo-api.routerplus.com", apiKey: process.env.TM_API_KEY });
```

### Which surface to call

Every chat model answers on both surfaces. When your surface matches the format of the route that serves the request, the gateway forwards the request as sent, with all provider features. The native surface of each model family's first route:

| Model ids | Native surface | Features that need it |
|---|---|---|
| `claude-*` | `/v1/messages` (Anthropic SDK) | `thinking`, server tools such as web search, PDFs given by URL |
| `gpt-*` | `/v1/chat/completions` (OpenAI SDK) | `response_format`, `reasoning_effort`, `seed`, `logprobs`, custom tools |

On the other surface, the gateway translates:

- A field that the translation cannot carry is dropped and named in the `x-tm-dropped-params` response header.
- If the drop would change the answer, the request is a 400 that names the field. Before it refuses, the gateway tries another provider of the same model that speaks your format.
- The `x-tm-provider` header names the provider that served you.
- The catalog gives each model's native format in `providers[0].dialect`.
- If your code sends only text, images, PDFs and function tools, either surface works for every model.
- Prompt caching marks (`cache_control`) reach Claude from both surfaces.

Details: [Wire compatibility](compat.md).

## 3. Call models

### Rules for every request

1. **Use exact model ids.** Copy them from `GET /v1/models` or `/api/models.json`. Claude and GPT ids have no prefix: `claude-sonnet-5`, `gpt-4o-mini`. Decision models keep their author prefix: `typesafe/jev-1.13`. Variants such as `:free` do not exist. A dated snapshot of a listed model, such as `claude-haiku-4-5-20251001`, also works. It routes and bills as its model.
2. **Keep model ids in one place in your configuration.** Check them against `GET /v1/models` when your service starts.
3. **Use the right route for the output.** Chat models answer on the two chat surfaces. Image models answer only on `POST /v1/images/generations`, and decision models only on `POST /v1/decisions`. The catalog field `output_modalities` (`text`, `image` or `decisions`) tells you which is which. A decision model also takes its own question format. The catalog field `decisions_wire` in `/api/models.json` names it (`systemone` for Jev, the Bespoke Nimble models, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider v1 27B, GPT-6 Luna Decisions, Sage and ScaleDown Classify-1, and `gliner`). [POST /v1/decisions](/docs/api-decisions) describes each one. Do not guess the format from the model's name.
4. **Send `max_tokens` every time**, from 1 to the model's declared `max_output_tokens` (16,384 on GPT-4o; 128,000 on the Claude 5 models). When a model declares no maximum, the maximum is 32,768.
   - If you leave it out, the gateway writes the model's own maximum into the request. When needed, it lowers that value so that the input and the output fit the model's context window.
   - A value above the model's maximum is a 400.
   - While the call runs, the gateway holds balance for the output limit that you send (at most 32,768 tokens when you send none). A realistic value helps on a small balance and lets more calls run at once.
5. **Ask for one answer.** `n` above 1 is a 400. Make separate calls for several samples.
6. **Send the full conversation on every turn.** The gateway keeps no conversation state between calls. See [Data policy](data-policy.md) for how request data is handled.
7. **Keep `temperature` at 1 or less** in code that can reach Claude models. On a translated request, a higher value is a 400.
8. **Refuse drops where they matter.** A field outside the forwarded set is dropped and named in `x-tm-dropped-params`. Send `"provider": {"require_parameters": true}` to get a 400 instead of a drop.
   - The 400 comes from the first route that would drop a field. The gateway does not go on to a later route that could carry it, so combine it with `only` or `order`.
   - Example: the `anthropic` route drops `seed`. A Claude request on `/v1/chat/completions` that sends `seed` and `"require_parameters": true` is a 400.

### Streaming

Stream answers that a person watches, and long answers.

- The OpenAI surface sends a role chunk, content chunks, a finish chunk, a usage chunk with an empty `choices` list and `usage.cost`, then `data: [DONE]`.
- The Anthropic surface sends the native event sequence, with usage and `cost` in the final `message_delta`.
- If the provider fails after the answer starts, the stream ends with one error event and nothing after it (no `[DONE]`). The SDKs raise an error. RouterPlus bills the partial answer. The gateway never continues an answer on another provider. To try again, send the full turn again.
- During a silence of 15 seconds or more, the gateway sends a keep-alive: an SSE comment on the OpenAI surface, a `ping` event on the Anthropic surface. The SDKs skip them. If you parse SSE yourself, skip them too.
- The first response headers must arrive within 20 seconds. If they do not, the gateway tries the next provider. A request can run for at most 450 seconds.
- To stop an answer, close the connection. The provider stops, and you pay for the usage that the provider reported.
  - If the provider reported no final usage, you pay for the input and for the output already streamed to you. The input count is the provider's count when it sent one, or else the gateway's estimate.
  - If nothing streamed yet, you pay for the input only.

Python, the OpenAI SDK, keeping the cost and handling a failure in the middle of an answer:

```python
from openai import APIError

parts, usage = [], None
try:
    stream = client.chat.completions.create(
        model="claude-sonnet-5",
        max_tokens=2048,
        stream=True,
        messages=messages,
    )
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            parts.append(chunk.choices[0].delta.content)
        if chunk.usage:  # the usage chunk; when usage is known, it also comes before an error
            usage = chunk.usage
except APIError:
    # The answer failed part way. What arrived is billed. Do not join a retry onto
    # `parts`: send the whole turn again, or show the error.
    raise
print("".join(parts), "cost:", usage.cost if usage else None)
```

Details: [Streaming](streaming.md).

### Calls that do not stream

- Set the client timeout above 450 seconds. The SDK default of 10 minutes works.
- The Anthropic SDKs refuse a call that does not stream when `max_tokens` is above about 21,000, unless you pass an explicit `timeout`. Stream those calls instead.
- If you close the connection, the gateway stops the provider. Nothing was streamed to you, so you pay for the input only.
- An image render does not stop when you close the connection. It finishes and is billed at the usage that the provider reports.
- A call that sends no `max_tokens` can reach the 450-second budget on a long answer. It then fails with a 504, billed $0. Send `max_tokens`, or stream.

### Tools, reasoning and structured output

- **Function tools** work on both surfaces for every chat model. The gateway translates tool calls and tool results.
- **Custom (free-form) tools**, `type: "custom"`, work on `/v1/chat/completions` for the GPT-6 and GPT-5.6 models. Their calls come back as `tool_calls` entries of `type: "custom"`.
- **Anthropic server tools** (web search, computer use, bash, text editor) work only for Claude models on `/v1/messages`, while the `anthropic` route serves them. Web search costs $10 per 1,000 searches (the Anthropic price). RouterPlus adds that cost to the tokens in `usage.cost`. The `anthropic` route refuses `mcp_servers` and `container`. A request with either one is a 400 that names the field.
- **`thinking` and `output_config.effort`** work on `/v1/messages` for every reasoning model.
  - For Claude they pass through as sent.
  - For a model that an OpenAI-format provider serves (GLM 5.3, Kimi K3, DeepSeek), effort becomes `reasoning_effort`. `adaptive` thinking leaves the model's reasoning on, and `disabled` turns it off ([details](/docs/compat#reasoning-and-structured-output-on-openai-dialect-routes)).
  - On `/v1/chat/completions`, Claude's thinking arrives as `reasoning_content`.
  - On `/v1/messages`, an answer from a non-Claude model has its reasoning as `thinking` blocks, streamed or not. A non-streamed answer starts with the reasoning. The block is readable. When you set `thinking.display` to `"omitted"` or `"updates"` (as Claude Code does), the block is empty and the text is in its `signature`.
  - You can send `thinking` blocks back in the next turn.
  - Claude Code works on these models with no change on your side. Set its model to the catalog id, for example `z-ai/glm-5.3`, with `z-ai/glm-5.3-flash` as its small model.
- **`response_format`** needs an OpenAI-format provider: GPT ids and the open models. On `/v1/messages`, `output_config.format` with a JSON schema reaches an open model's provider as a strict json_schema `response_format`.
- **JSON from a Claude model on `/v1/chat/completions`:** give the model one tool whose input schema is the JSON you want, and read the tool input. Claude Sonnet 4.5 and Haiku 4.5 take a forced `tool_choice`. The Claude 5 models refuse a forced tool choice (`required` or a named tool) with a 400, so use `auto` there.
- Do not end the message list with an assistant message when a Claude model can serve the request from the OpenAI surface. Claude refuses a prefilled answer, so the request is a 400.

### Images in prompts

Images and PDFs in user messages work on both surfaces. The gateway translates these parts between the OpenAI and Anthropic formats:

- JPEG, PNG, GIF and WebP images, as a data URL or an `http(s)` URL
- PDFs, as a data URL

Rules:

- When the gateway must translate, these parts are a 400 that names the part: an audio or video part, a file given by id, a file other than a PDF, and an image in an assistant message or a tool result.
- Send an image, document, audio or video part only to a model that takes it. `architecture.input_modalities` in `GET /v1/models` lists each model's inputs. A part that the model does not take is a 400 that names the part, and nothing is billed.
- While the call runs, each image or document part counts as 65,536 tokens against your limits and your balance hold.

### Other routes

- **Images:** `POST /v1/images/generations` takes the OpenAI Images API body. The image comes back as base64 in the response. `response_format: "url"` is a 400. `n` is 1 unless the model allows more: see `supported_parameters.n` in `/api/models.json` (GPT Image models allow up to 4). There is no streaming. A failed render bills nothing. See [POST /v1/images/generations](api-images.md).
- **Decisions:** `POST /v1/decisions` sends a state and typed questions to a decision model. It returns one typed answer for each question, for example a choice, a score or a yes/no probability. Use it for routing, classification and moderation points in your code. It replaces a chat model whose output you parse. The models:
  - `typesafe/jev-1.13` (Jev, from TypeSafe)
  - `bespokelabs/nimble-v4` and `bespokelabs/nimble-v4-max` (Bespoke Nimble v4 and v4 Max, from Bespoke Labs; `bespokelabs/nimble-latest` lets Bespoke pick the model)
  - `cloudflare/clef` and `cloudflare/clef-flash` (Clef and Clef-flash, from Cloudflare)
  - `routerplus/decider-2b` and `routerplus/kev-4b` (Decider 2B and Kev 4B, from RouterPlus)
  - `perplexity/pplx-decider-v1-27b` (Perplexity Decider v1 27B, from Perplexity)
  - `openai/gpt-6-luna-decisions` (GPT-6 Luna Decisions, from OpenAI; `gpt-6-luna` is OpenAI's chat model, not this one)
  - `levanto/sage-1.2` (Sage, from Levanto)
  - `scaledown/classify-1` (ScaleDown Classify-1, from ScaleDown)
  - `fastino/gliner-2.5-decide` (GLiNER-2.5-Decide, from Fastino)

  Rules for decision models:
  - The question format follows the model. System One questions (Jev, Nimble, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider, GPT-6 Luna Decisions, Sage and ScaleDown Classify-1) have a `type`. GLiNER takes a `schema` of classifications and extractions in place of `questions`. A question in another model's format is a 400.
  - GLiNER's numbers are confidences. They are not calibrated probabilities.
  - Perplexity Decider bills the state once per question, so ask it only the questions you need.
  - GPT-6 Luna Decisions adds about 150 input tokens for each question. It can decline one question, and that answer is `{"type": "refusal"}`, so check each answer's `type` before you read its value.
  - The gateway never fails over from one model to another. See [POST /v1/decisions](api-decisions.md).
- **Token counting:** `POST /v1/messages/count_tokens` is free, for models that have an Anthropic-format provider. It counts against your requests per minute.
- **Your deployed models:** an id such as `tm/<name>-v1` from [Optimize](model-search.md) works on `/v1/chat/completions` only, with keys of the organization that deployed it. With `stream: true`, it returns the finished answer as one chunk.

## 4. Pin providers (only when you must)

How routing works without a pin:

- By default, RouterPlus orders a model's providers by the estimated cost of one good answer. Providers close to the best share the traffic. Until providers have measurements, the cheapest provider goes first. See [Provider routing](provider-routing.md).
- Claude models go to `anthropic` first, and GPT models to `openai`.
- The gateway moves to the next provider only when the current one fails before it sends any output, or has no capacity left.
- Each provider charges its own price. You pay the price of the provider that serves the request: the one you name with `only`, or else the one that routing selects, fallbacks included. `/api/models.json` gives each provider's live price in `providers[].pricing_usd_per_million`.

Pin a provider in these cases only:

- A rule requires one provider.
- A feature exists at one provider only.
- You compare providers in a test.

`only` and `"allow_fallbacks": false` remove fallbacks. `order` keeps every fallback.

### The `provider` object

Send it in the request body on `/v1/chat/completions` or `/v1/messages`. The gateway reads it and never forwards it. An unknown key inside it is a 400.

| Field | Value | Effect |
|---|---|---|
| `order` | provider ids | Try these providers first, in this order. The others stay as fallbacks. |
| `only` | provider ids | Use only these providers. |
| `ignore` | provider ids | Never use these providers. |
| `allow_fallbacks` | boolean | `false` keeps only the first route. If that route is resting after failures, the request fails with 503 and `retry-after` and does not move on. |
| `sort` | `"price"`, `"throughput"`, `"latency"` or `"quality"` | Turns off the default order. `"price"`: marketplace routes cheapest first, after `order`, by the prompt and completion prices per million at each provider's live price. A route billed at the cost the provider reports goes after the priced routes. `"throughput"`: the highest measured output speed first. `"latency"`: the lowest measured time to first token first. `"quality"`: the highest share of well-formed tool calls first. Without `sort`, the default order applies (see [Provider routing](provider-routing.md)), unless your key's [routing policy](routing-policies.md) orders its routes. |
| `preferred_min_throughput`, `preferred_max_latency` | number (p50) or `{"p50", "p75", "p90", "p99"}` | Routes below the throughput (tokens per second) or above the latency (seconds to first token) go last. No route is left out. |
| `quantizations` | list of `int4`, `int8`, `fp4`, `mxfp4`, `nvfp4`, `fp6`, `fp8`, `mxfp8`, `fp16`, `bf16`, `fp32`, `unknown` | Only routes that serve the model at these quantizations. A route that declares none is `unknown`. |
| `max_price` | `{"prompt": USD, "completion": USD}` per million, at least one of them, each from 0 to 1,000,000 | Leaves out a marketplace route whose prompt or completion price is above the cap. See the rules below. |
| `require_parameters` | boolean | `true`: when the first route that would serve you would drop one of your request fields, the request is a 400. The gateway does not try a later route. |
| `connections`, `connection_order`, `funding` | ids, `"byok"` or `"house"` | Choose among your own provider connections and who pays. See [Routing policies](routing-policies.md). |
| `regions`, `zdr`, `data_collection` | region list, boolean, `"allow"` or `"deny"` | Hard filters on the provider's published terms. `"zdr": true`: the provider keeps no prompts. `"data_collection": "deny"`: the provider does not train on or keep them. A route on your own connection passes with operator evidence. A route whose terms are not known is left out. |

Rules for `max_price`:

- Cached input, cache write and reasoning prices are not compared, so they are not capped.
- The price compared is the price that you would pay, after the discount of a decision model.
- If no route is left, the request is a 404 that names `provider.max_price`. This also happens when a route that takes your images, files or `response_format` is above the cap and the routes within it do not take them.
- If the price cannot be read, the request is a 503 with `retry-after`.
- A route billed at the cost the provider reports (some image and video models) has no price to compare, so `max_price` leaves it out. When every route left out is such a route, the request is a 400: `provider.max_price is not available for model "<model>": it is billed at the cost the provider reports`.

`order`, `only` and `ignore` take route labels. Marketplace route labels are the `providers[].id` values in `/api/models.json`, for example `anthropic` and `openai`. Your own connections use their profile: `openai`, `anthropic`, `azure` or `bedrock`.

- A label that matches a route label selects or orders that route, even when that route does not serve the model. `only: ["anthropic"]` means the Anthropic route, so on a GPT model it leaves no route.
- Any other label matches no route: it selects, orders or skips nothing. An `only` list without a route label leaves no route.

| Goal | `provider` |
|---|---|
| Claude only from Anthropic's own API | `{"only": ["anthropic"]}` |
| Fail instead of falling back to another provider | add `"allow_fallbacks": false` |

Check a pin before you ship it. `POST /v1/route` returns the ordered candidates, and each excluded route with its reason. It calls no provider and costs nothing. If your controls exclude every route, a real request returns 404 `model_unavailable`. This is the same error as for an unknown model. When a pinned request gets a 404, run `/v1/route`.

```bash
curl -s https://demo-api.routerplus.com/v1/route \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-5","provider":{"only":["anthropic"]}}'
```

Python: the OpenAI SDK sends fields it does not know through `extra_body`.

```python
reply = client.chat.completions.create(
    model="claude-haiku-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={"provider": {"only": ["anthropic"]}},
)
```

TypeScript: pass a variable, so the SDK keeps the field its types do not know.

```typescript
const request = {
  model: "claude-haiku-4-5",
  max_tokens: 1024,
  messages: [{ role: "user" as const, content: "Hello" }],
  provider: { only: ["anthropic"] }, // read by the gateway, never forwarded
};
const reply = await client.chat.completions.create(request);
```

Check each answer with its response headers:

| Header | Value |
|---|---|
| `x-tm-provider` | The provider |
| `x-tm-served-by` | The provider that ran the request, when a route reports one |
| `x-tm-upstream-model` | The id the provider received, when it differs from yours |
| `x-tm-attempts` | The number of provider attempts |

`GET /v1/generation?id=<x-request-id>` shows the route of every attempt.

To pin for an organization, workspace or key, save a routing policy with the policy API. The policy API takes a console session, so a human sets it up. A request's `provider` object can narrow a saved policy. It can never add routes. See [Routing policies](routing-policies.md).

### Your own provider keys (BYOK)

You can route through your own OpenAI, Anthropic or Azure key, or an AWS Bedrock role. A human adds it one time in the console, in Access > Provider connections. Your code still calls the gateway with a RouterPlus key. The provider bills you directly, and `usage.cost` is 0. A key bound to a connection can call only that connection's model ids. It does not fall back to marketplace supply unless a routing policy allows it. See [Bring your own key](byok.md).

## 5. Keep sessions on one route and caches warm

A prompt cache saves money and time only when the next request reaches the same provider with the same prompt start.

- The gateway keeps no session state. Your app sends the full conversation on every turn.
- A conversation stays on the provider that served its first turn while that provider can serve. Requests that share a long prompt prefix (system prompt and tools) go to the same provider across sessions.
- A turn moves to the next route in three cases:
  - The first route fails before it sends output. One failure is enough.
  - The route's capacity pool is full.
  - Its provider's 429 pause is still running. The pause lasts for the provider's `retry-after`, 1 to 60 seconds.
- After two failures in a row, later turns skip the route for 30 seconds. A turn that moves can miss the cache one time.
- On a follow-up turn, the gateway first waits up to 5 seconds for a full pool. See [Waiting for the first route](#waiting-for-the-first-route).
- If you send a session id, the gateway gives it to each provider in that provider's own field. See [Sessions and caching](#sessions-and-caching). Claude Code's `x-claude-code-session-id` header counts, with no change on your side.

Rules for a high cache hit rate:

1. Keep the start of the prompt byte-identical: the same system prompt, the same tool definitions in the same order, and earlier messages unchanged. Only append.
2. Put content that changes (the time, ids, documents retrieved for this turn) after the stable part, at the end.
3. Keep the model and the `provider` object the same for every turn of the session.
4. For Claude, put `cache_control` on the block that ends the stable part (a system, tool or message block), or send the top-level `cache_control` field for automatic caching. Both surfaces keep the marks for Anthropic: see [Cache marks](#cache-marks). On marketplace routes the cache lasts 5 minutes, and `ttl: "1h"` is removed. If you send no marks, the gateway turns on automatic caching for a request with tools or earlier turns: see [Automatic caching on Claude](#automatic-caching-on-claude).
5. OpenAI caches long repeated prompt prefixes on its own, with no request setting.
6. Measure:
   - Cache reads are `usage.prompt_tokens_details.cached_tokens` on the OpenAI surface and `usage.cache_read_input_tokens` on the Anthropic surface.
   - Cache writes are `cache_write_tokens` and `cache_creation_input_tokens`.
   - RouterPlus bills cache reads at the lower `cached_prompt` price, and Claude cache writes at the higher `cache_write` price. The prices are in `/api/models.json`.

Python, the Anthropic SDK, with a cache mark on a long system prompt that does not change:

```python
reply = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": STABLE_INSTRUCTIONS,              # the same bytes on every turn
        "cache_control": {"type": "ephemeral"},   # cache everything up to here
    }],
    messages=history,                             # append only
)
print(reply.usage)  # cache_read_input_tokens, cache_creation_input_tokens and cost
```

The provider caches a prompt start only above a minimum length. A short system prompt shows no cache reads.

### Sessions and caching

#### Session ids

On `POST /v1/chat/completions` and `POST /v1/messages`, the session id is the first valid value among these inputs, in this order. Token counting and images do not use it.

| Order | Input | Kind |
|---|---|---|
| 1 | `session_id` | Top-level body field |
| 2 | `x-session-id` | Header |
| 3 | `x-session-affinity` | Header |
| 4 | `x-claude-code-session-id` | Header that Claude Code sends by itself |

- A valid value has 1 to 256 printable ASCII characters. `prompt_cache_key`, a top-level body field, follows the same rule.
- The gateway ignores an invalid value and uses the next input. An invalid value alone is never an error. `x-tm-dropped-params` names the ignored input as `session_id (invalid)`, `prompt_cache_key (invalid)`, `header:x-session-id (invalid)`, `header:x-session-affinity (invalid)` or `header:x-claude-code-session-id (invalid)`. With `"require_parameters": true`, that makes the request a 400, as any dropped field does.
- Valid session inputs are never named in `x-tm-dropped-params`, and they never trigger `require_parameters`.
- `prompt_cache_key` stays OpenAI's own field. It wins over the session id on OpenAI routes.
- When two valid inputs disagree, the order decides. There is no error.
- Codex's `session-id` header is not read. Codex calls `POST /v1/responses` and sends its conversation id as `prompt_cache_key` in the body, which reaches OpenAI unchanged.

What each route receives:

| Route | Field | Value |
|---|---|---|
| `openai` (marketplace), and `openai` and `azure` (your connections) | `prompt_cache_key` | Your `prompt_cache_key`, or else the session id. Nothing if you sent neither |
| `morphllm` | Header `x-session-id` | The session id, or else your `prompt_cache_key`. Nothing if you sent neither. Morph uses it to pick the region and the worker that hold your cache |
| `anthropic` and `bedrock` | Nothing | Anthropic has no session input. Its cache matches the prompt start |

- Values go to the provider unchanged. A value longer than the provider accepts is replaced by `tm-` followed by 43 base64url characters: a hash of the value.
- The gateway keeps no session table. The id goes to the provider, and the provider routes on it.
- `prompt_cache_retention` goes unchanged to `openai` and `azure` routes, both `"in_memory"` and `"24h"`. On other routes it is dropped and named.

#### End-user ids on marketplace routes

On marketplace routes, the provider gets a hash in place of your end user's id. The raw values do not go to the provider.

- The id is the first string among the body fields `safety_identifier`, `user` and `metadata.user_id`.
- The hash is `tm-` followed by 43 base64url characters, made from your organization id and that id. Without an id, it is made from your organization id alone.
- Each marketplace route gets one field:

| Marketplace route | Field that carries the hash | Fields removed |
|---|---|---|
| `openai`, `azure` | `safety_identifier` | `user` |
| `anthropic` | `metadata.user_id` | `user`, `safety_identifier` |
| every other route (OpenAI-compatible), dedicated capacity included | `user` | `safety_identifier`, `metadata.user_id` |
| `mistral-ai` | none: Mistral has no end-user field | `user`, `safety_identifier`, `metadata.user_id`, each named in `x-tm-dropped-params` when you sent it |

- When a provider blocks an id, it blocks one user of one customer. The RouterPlus account stays open.
- On your own connections, `user` and `metadata` pass unchanged. `safety_identifier` passes unchanged to `openai` and `azure` connections. It is dropped and named on `anthropic` and `bedrock` connections.
- Identity fields that a route uses are never named in `x-tm-dropped-params`. Mistral's route uses none, so it names each one you sent, and `require_parameters` refuses that route.

#### Cache marks

The gateway never adds a cache mark that you did not send. Automatic caching, below, adds only the top-level field, and only when you sent no marks. Block and part marks (`cache_control`):

| Request | Route | What happens to the marks |
|---|---|---|
| `/v1/chat/completions` | Anthropic direct or Bedrock | Each text part keeps its mark. A marked system part turns the system prompt into blocks. A tool message's last mark goes on its `tool_result` block. Other part keys are not forwarded |
| `/v1/chat/completions` | OpenAI direct or Azure | Part marks pass unchanged. OpenAI and Azure ignore them |
| `/v1/messages` | Anthropic direct or Bedrock | Block marks pass unchanged |
| `/v1/messages` | OpenAI direct or Azure | Marks are removed and named, for example `messages[i].content[j].cache_control` or `tools[i].cache_control` |

A top-level `cache_control` field (automatic caching), on either surface:

- Anthropic direct: sent unchanged.
- Bedrock: turned into a mark on the last block that can carry one, counted back from the end of the messages. If no block can carry it, it is dropped and named as `cache_control`.
- OpenAI direct and Azure: dropped and named as `cache_control`.
- Any route: a field that Anthropic would refuse is dropped and named as `cache_control`. Anthropic refuses:
  - a field other than `{"type": "ephemeral"}` with an optional `ttl` of `"5m"` or `"1h"`
  - a fifth mark (when four marks exist already)
  - a 1-hour field after a 5-minute mark
  - a field whose TTL differs from the mark on the last block

On marketplace routes, `ttl: "1h"` is removed from every mark, so the cache lasts 5 minutes. The drop is named as `<path>.cache_control.ttl`, for example `system[0].cache_control.ttl`, `messages[2].content[0].cache_control.ttl`, or `cache_control.ttl` for the top-level field. `ttl: "5m"` passes. Your own connections keep `ttl: "1h"`.

#### Automatic caching on Claude

When a request has no cache marks, the gateway adds the top-level `cache_control: {"type": "ephemeral"}` field (5 minutes) if all of these are true:

- Anthropic's own API serves the request: the `anthropic` route or your own Anthropic connection. Bedrock and dedicated endpoints do not get it.
- The request has no `cache_control` anywhere.
- It has tools, or an assistant turn before its last message.
- The prompt is at least the model's minimum cacheable size: 512 tokens on the Claude 5 models except Sonnet 5, 1,024 on Claude Sonnet 5 and Sonnet 4.5, and 4,096 on Claude Haiku 4.5.
- The request does not carry the header `x-tm-auto-cache: off`.

The response header `x-tm-auto-cache` is `applied`, or `skipped:` with the reason, for example `skipped:single-turn` or `skipped:client-markers`. The first turn pays a cache write (1.25 times the input price). Later turns read the prompt at the cache price. Details: [Wire compatibility](compat.md#automatic-caching-on-claude).

#### Waiting for the first route

On a follow-up turn (an assistant or tool message after a user message), the first route's capacity pool can be full. The gateway then waits for that pool for up to 5 seconds (an operator setting) before it moves the request to the next route. It never waits past the request's deadline. The `x-tm-affinity-wait-ms` response header gives the wait in milliseconds, only when the wait was more than 0.

## 6. Handle errors and retries

Every failure has one stable class, `error_type`. Read it from `error.metadata.error_type` on the OpenAI surface, or `error.error_type` on the Anthropic surface. The `x-tm-error-code` header has it too, but browsers cannot read that header. Never parse the message text. Log the `x-request-id` header with every failure.

| `error_type` | HTTP | Retry | What to do |
|---|---|---|---|
| `rate_limit` | 429 | Yes | Wait `retry-after` seconds, then retry. A capped wait can take a second try. If `x-tm-limit-kind` is `concurrency`, send fewer requests at once. |
| `email_not_verified` | 403 | No | Stop and tell a human to open the verify link in their email. The key starts working within seconds of the click. |
| `insufficient_quota` | 429 | No | Stop and tell a human to add credits or raise a spend cap. On a cap, `x-tm-cap-reset` gives the reset time. |
| `upstream_error` | 5xx | Yes, once, after about 2 seconds | The gateway already tried every other provider it could. If the retry fails too, report it with the `x-request-id`. |
| `upstream_error` | 4xx | No | The provider refused the request. The message has the provider's own words. |
| `gateway_error` | 503 | Yes | Wait `retry-after`. The request did not reach a provider. |
| `gateway_error` | 504 | Yes, once | The deadline passed while the gateway tried providers. An earlier attempt may be billed: check `GET /v1/generation?id=<x-request-id>` first. |
| `gateway_error` | 400 | No | Send one output limit, from 1 to the maximum the message names (32,768 only when no maximum is declared). |
| `gateway_error` | 500 | No | Report it with the `x-request-id`. |
| `model_unavailable` | 404 | No | Fix the model id, or fix your `provider` object (check it with `POST /v1/route`). |
| `model_unavailable` | 502 | Yes, once | The connection to every provider failed. |
| `invalid_request` | 400 | No | Fix the field that the message names. |
| `context_overflow` | 400 | No | Shorten the input, or use a model with a larger context. The body has the standard shape that agents compact on: `error.code` `context_length_exceeded` on the OpenAI surface, `prompt is too long: N tokens > M maximum` on the Anthropic surface. |
| `content_policy` | 400 | No | Change the prompt. The gateway never sends it to another provider. |
| `auth` | 401 | No | Check the header, the key, and whether the key was disabled. A 401, 402 or 403 relayed from every provider is a problem with the RouterPlus account: report it with the `x-request-id`. |
| `request_too_large` | 413 | No | Keep the body under 10 MB. |
| `not_found` | 404 | No | Use a route that exists. The list is in [Errors](errors.md). |

Rules:

1. Let the SDK retry. The official OpenAI and Anthropic SDKs retry 429 and 5xx responses twice by default, and they wait for `retry-after`. Keep that default. Do not add a second retry loop around it. The SDKs also retry an `insufficient_quota` response. That costs nothing, but your code must then stop.
2. There is no idempotency key. A retried request is a new request. If it reaches a provider, it is billed again. A refusal by the gateway's own limits (`x-tm-error-origin: gateway_admission`) costs nothing.
3. Never retry `insufficient_quota`, `invalid_request`, `context_overflow`, `content_policy` or `auth` in a loop. The request or the account must change first.
4. An error in the middle of a stream is final for that answer. Send the full turn again. Never join two partial answers.

Python, the OpenAI SDK, reading the class:

```python
from openai import APIStatusError

try:
    reply = client.chat.completions.create(model=MODEL, max_tokens=1024, messages=messages)
except APIStatusError as e:
    error = e.response.json().get("error") or {}
    kind = (error.get("metadata") or {}).get("error_type") or e.response.headers.get("x-tm-error-code")
    request_id = e.response.headers.get("x-request-id")
    if kind == "insufficient_quota":
        notify_operator(request_id, error.get("message"))  # money or a cap: a retry cannot help
    raise
```

Details: [Errors](errors.md).

## 7. Limits, spend caps and your users

Default limits:

| Scope | Requests per minute | Tokens per minute | In flight at once |
|---|---|---|---|
| Trial key | 20 | 1,000,000 | 8 |
| Console or identity API key | 300 | 1,000,000 | 8 |
| Organization, all keys together | 300 | 1,000,000 | 8 |

Rules:

1. Limit how many requests your code sends at once. By default the organization allows 8 in flight. A ninth gets 429 `rate_limit` with `x-tm-limit-kind: concurrency`. Use a semaphore or a fixed pool of workers.
   - To run more at once, a human first raises the organization and key limits in the console, in Billing > [Rate limits](https://demo.routerplus.com/console/limits). A key's requests per minute cannot go above its ceiling (300 for a console key).
   - Each marketplace provider pool also limits one organization to half of the pool's capacity. That refusal has `x-tm-limit-scope: pool`.
2. Read `x-tm-remaining-rpm` and `x-tm-remaining-tpm` on each response, and slow down before they reach zero. `GET https://demo-api.routerplus.com/v1/limits?model=<id>` returns the limits that apply to your key.
3. While a call runs, token limits count an estimate: one token for each four bytes of the request, rounded up, 65,536 for each image or document part, plus `max_tokens` (at most 32,768 of it). An image sent inline as base64 counts its 65,536 only, not its bytes. The real count replaces the estimate when the call ends. Small requests and a realistic `max_tokens` let more calls run at once.
4. Spend caps are monthly, for each key and for the organization, on the UTC calendar month. A human sets them in the console. A cap refuses with 429 `insufficient_quota` and the `x-tm-cap-reset` header. Give each feature or customer its own key and cap, so a bug or a loop cannot spend everything.

Your users:

- By default, one backend key serves all your users. The gateway does not know who your users are, so enforce per-user quotas in your app.
- `safety_identifier`, `user` and `metadata.user_id` identify your end user to the provider only (see [End-user ids on marketplace routes](#end-user-ids-on-marketplace-routes)). They do not identify, limit or bill anyone on the gateway. Do not put emails or names in them.
- For limits that the gateway enforces per user, use the identity API. Create an `end_user` principal for each user, issue a key for it, and set limits on the principal. Your backend keeps those keys. Never give a key to the user. The identity API takes a console session, not an API key, so a human with an owner or admin role sets it up. See [Workspaces and principals](identity.md).

Details: [Rate limits and spend caps](limits.md) and [Limits and capacity](admission.md).

## 8. Track cost

- Every billed response has `usage.cost` in USD. It is the full charge for the request, including provider attempts that failed before the answer. Store it with the `x-request-id`, the model and your own ids, such as the user and the feature.
- An Anthropic-surface stream that fails in the middle of the answer has no usage in it. Read its cost from `GET /v1/generation?id=<x-request-id>`.
- `GET /v1/usage` returns the balance and the last 20 attempts. For a key bound to its own workspace or principal, the balance fields are `null`. The console's Usage and Logs pages show all keys of the organization.
- Anthropic's web search adds its fee per search to `usage.cost` (see [Pricing and billing](pricing.md#web-search)).
- To check a charge, multiply the token counts by the prices in `/api/models.json`. The gateway rounds the final charge down.

Details: [Pricing and billing](pricing.md) and [GET /v1/usage and /v1/generation](api-usage.md).

## 9. Verify the integration

One cheap call shows the answer, the cost and the headers that the gateway adds:

```bash
curl -sS -i https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"gpt-4o-mini","max_tokens":16,"messages":[{"role":"user","content":"Reply with OK"}]}'
```

Then confirm that the ledger recorded it:

```bash
curl -s -H "Authorization: Bearer $TM_API_KEY" https://demo-api.routerplus.com/v1/usage
```

The integration is done when:

1. The rules in [The short version](#the-short-version) hold in the code.
2. The key is not in the repository, in a client bundle or in a log.
3. Every model id in the configuration appears in `GET /v1/models`.
4. One streamed and one non-streamed test call return `usage.cost`, and both appear in `GET /v1/usage`.
5. The `x-tm-dropped-params` header is absent, or it names only fields that you accept losing.
6. `x-tm-provider` shows the route that you expect.

## Leave instructions for the next agent

Add this block to the `AGENTS.md` or `CLAUDE.md` file of the repository that you integrated, so later coding sessions keep the integration correct:

```markdown
## LLM calls: RouterPlus gateway

- Every LLM call goes through the RouterPlus gateway. Full guide: https://demo.routerplus.com/docs/agent-integration.md
- OpenAI SDK base URL: https://demo-api.routerplus.com/v1. Anthropic SDK base URL: https://demo-api.routerplus.com. Key: the TM_API_KEY environment variable, on the server only.
- Model ids live in configuration. Each one must exist in https://demo.routerplus.com/api/models.json. Never invent or alias an id.
- Send max_tokens on every request (1 to the model's max_output_tokens in models.json; 32768 only when none is declared). Never send n above 1.
- Claude models: /v1/messages, with cache_control on the stable start of the prompt. GPT models: /v1/chat/completions.
- Keep the start of the prompt byte-identical across turns. Only append.
- Send one id per conversation as the x-session-id header (or session_id in the body). Providers then keep its cache warm.
- Provider pins go in the request body "provider" object (only, ignore, order, allow_fallbacks, require_parameters). Check a pin with POST https://demo-api.routerplus.com/v1/route.
- Errors: switch on error_type. rate_limit: wait retry-after, then retry; a capped wait can take a second try. insufficient_quota: stop and tell a human. Never retry invalid_request, context_overflow or content_policy.
- No idempotency key: a retry is a new, billed request. Add no retry loop on top of the SDK's own retries.
- At most 8 requests in flight per organization, unless the limit was raised.
- Log x-request-id and usage.cost for every call.
- Not available: moderation. POST /v1/responses serves OpenAI models; Claude and the other models are a preview there. On it, store is always false and previous_response_id is a 400.
```

## Steps only a human can do

- Complete browser signup through Clerk, including email verification, then buy credits in Billing. Signup and card verification alone do not add credit.
- Fix billing when `insufficient_quota` appears: add credits, or raise a spend cap in the console.
- Sign in to the console to create keys, set spend caps and change limits. See [Console](/docs/console).
- Add your own provider keys (BYOK), save routing policies, and set up workspaces and end-user principals.
- Approve large test runs. Every call is billed.

## Where next

- [Quickstart](/docs/quickstart) - the first streamed call, step by step
- [Errors](/docs/errors) - every error class, and what to do about it
- [Routing policies](/docs/routing-policies) - the provider object, saved policies and BYOK routes
- [Streaming](/docs/streaming) - frame order, keep-alives and failures in the middle of an answer
- [Rate limits & spend caps](/docs/limits) - the numbers behind every 429
- [Wire compatibility](/docs/compat) - which fields pass, which translate and which drop

---

Source: https://demo.routerplus.com/docs/console.md

# Console

The console at [https://demo.routerplus.com/console](https://demo.routerplus.com/console) is where you manage your organization: keys, credits, limits, team, governance and the build tools. Sign in with the email you verified at signup. The sidebar shows the pages in the groups below.

If you are a member of two or more organizations, pick one in the **Organization** menu at the top of the sidebar. Each page shows and changes only that organization. The balance card at the bottom of the sidebar opens Credits.

## Overview, Usage, Analytics and Logs

| Page | What you do there | Docs |
|---|---|---|
| [Overview](https://demo.routerplus.com/console) | See spend per day, where it went and your recent requests. A new account sees its first steps: verify the email, create a key, make one call. | [Quickstart](/docs/quickstart) |
| [Usage](https://demo.routerplus.com/console/usage) | See requests, input and output tokens, success rate and spend for the last 24 hours, 7 days or this month, grouped by model, provider or key. | [GET /v1/usage](/docs/api-usage) |
| [Analytics](https://demo.routerplus.com/console/analytics) | Chart daily spend over 7, 30 or 90 days by model, provider, workspace, member, key or request label. Read the spend forecast and the savings table. | [Management API](/docs/api-management#activity) |
| [Logs](https://demo.routerplus.com/console/logs) | Search and filter the requests of the last 30 days by outcome. Open one request to see its attempts and its cost. Logs hold metadata only. | [Data policy](/docs/data-policy) |

## Access

| Page | What you do there | Docs |
|---|---|---|
| [API keys](https://demo.routerplus.com/console/keys) | Create, copy and disable keys. Set a monthly spend cap on a key. See each key's last use and spend this month. | [Authentication](/docs/authentication) |
| [Security](https://demo.routerplus.com/console/security) | Find risky keys: no spend limit, no expiry, idle for 30 days, expiring within 7 days or no IP allowlist. Disable a key from its row. | [Management API](/docs/api-management#security-report) |
| [Grants](https://demo.routerplus.com/console/grants) | Request, approve, reject and cancel time-boxed exceptions to a guardrail budget, a key limit or a guardrail's model rules. | [Guardrails](/docs/guardrails#grants) |
| [Provider connections](https://demo.routerplus.com/console/byok) | Connect your own provider account (BYOK) and use it through your RouterPlus key. Your provider bills you directly. | [Bring your own key](/docs/byok) |
| [Integrations](https://demo.routerplus.com/console/integrations) | Connect your Braintrust account so Optimize can read your evals. | [Optimize](/docs/model-search) |

## Billing

| Page | What you do there | Docs |
|---|---|---|
| [Credits](https://demo.routerplus.com/console/billing) | See your balance and buy credits. Read the statement, set up auto top-up and set the organization's monthly spend cap. | [Pricing & billing](/docs/pricing) |
| [Rate limits](https://demo.routerplus.com/console/limits) | See the gateway limits in effect. Set organization, key and model limits, and manage quota pools for your provider connections. | [Rate limits & spend caps](/docs/limits) |

## Team

| Page | What you do there | Docs |
|---|---|---|
| [Members](https://demo.routerplus.com/console/members) | See each member's organization role, workspace roles, status and last sign-in. The owner invites members, changes roles and removes members. | [Workspaces and principals](/docs/identity) |
| [Workspaces](https://demo.routerplus.com/console/workspaces) | See each workspace's spend this UTC month and its guardrail budgets. Create and manage workspaces. | [Workspaces and principals](/docs/identity) |
| [Single sign-on](https://demo.routerplus.com/console/sso) | Add OIDC and SAML connections, verify email domains with DNS records, and create or rotate the SCIM token. | [Single sign-on](/docs/sso) |
| [Audit log](https://demo.routerplus.com/console/audit) | Filter changes to governance, identity, connections and routing policies, and the sign-in log. Download the selection as CSV. | [Management API](/docs/api-management#audit-log) |

## Governance

| Page | What you do there | Docs |
|---|---|---|
| [Guardrails](https://demo.routerplus.com/console/guardrails) | Create guardrails, assign them, and see the effective policy of one key or member. | [Guardrails](/docs/guardrails) |
| [Budgets](https://demo.routerplus.com/console/budgets) | See every budget with this period's spend, percent used and reset time. Put a budget on a workspace or a member, and set the email, Slack or webhook channels for alerts. | [Guardrails](/docs/guardrails#budgets) |
| [Guardrail events](https://demo.routerplus.com/console/guardrail-events) | See the requests a guardrail blocked, redacted or flagged. Filter by action, key, guardrail and time. | [Guardrails](/docs/guardrails#activity) |
| [Presets](https://demo.routerplus.com/console/presets) | Create versioned request templates and copy their `@preset/<slug>` ids. | [Management API](/docs/api-management#presets) |

## Build

| Page | What you do there | Docs |
|---|---|---|
| [MCP servers](https://demo.routerplus.com/console/mcp) | Register MCP servers, set each server's tool policy, and see its recent tool calls. | [MCP gateway](/docs/mcp-gateway) |
| [Routers](https://demo.routerplus.com/console/routers) | Create custom routers, train them on labeled examples, and test which model a router picks for a prompt. | [Auto router](/docs/auto-router#custom-routers) |
| [Evals](https://demo.routerplus.com/console/evals) | Create an eval, follow its progress, read results per model and per case, and cancel it. | [Evals](/docs/evals) |
| [Exports and alerts](https://demo.routerplus.com/console/exports) | Send request data to webhook, OTLP, Langfuse, Datadog, ClickHouse, S3, Azure Blob, BigQuery or Snowflake. Set notification channels for budget alerts. | [Broadcast](/docs/broadcast) |

What you can change on a page depends on your role. Each docs page above gives the rules for its feature.

---

Source: https://demo.routerplus.com/docs/models.md

# Models & catalog

The catalog is one flat list of model ids. Every **chat** model works with one key on both APIs: OpenAI-compatible `/v1/chat/completions` and Anthropic-compatible `/v1/messages`. Image models answer only on `POST /v1/images/generations`, and decision models only on `POST /v1/decisions`. You pay the per-token prices in the catalog, and every billed response has `usage.cost`.

| Surface | URL | Auth | For |
|---|---|---|---|
| Model directory | `https://demo.routerplus.com/` | none | Browsing, filters, featured model order |
| Per-model pages | `https://demo.routerplus.com/models/<id>` | none | Providers with prices, latency, uptime and data policy; snippets to copy |
| Machine-readable JSON | `https://demo.routerplus.com/api/models.json` | none | Scripts, dashboards, price checks |
| SDK-shaped list | `https://demo-api.routerplus.com/v1/models` | API key | `client.models.list()`. See [GET /v1/models](/docs/api-models) |

## Model ids

Ids are exact strings. RouterPlus does no fuzzy matching and no substitution. An id that is not in the catalog is a 404 that names the id you sent.

The table shows the catalog at the time of writing. The live list is [`/api/models.json`](https://demo.routerplus.com/api/models.json).

| Model id | Providers | Context | Input /1M | Cached /1M | Cache write /1M | Output /1M |
|---|---|---|---|---|---|---|
| `claude-opus-5-5` | anthropic, deepinfra-labs, inference-net-chat | 1M | $4 | $0.2 | $5 | $20 |
| `claude-fable-5-1` | anthropic | 1M | $10 | $0.25 | $12.5 | $50 |
| `claude-sonnet-5-5` | anthropic, deepinfra-labs | 1M | $2 | $0.1 | $2.5 | $10 |
| `claude-haiku-5-5` | anthropic | 1M | $0.1 | $0.01 | $0.125 | $0.5 |
| `gpt-6-astra` | openai, inference-net-chat | 1.05M | $10 | $1 | $12.5 | $50 |
| `gpt-6.1-sol` | openai | 1.05M | $2 | $0.1 | $2.5 | $10 |
| `gpt-6-luna` | openai, inference-net-chat | 1.05M | $0.1 | $0.01 | $0.125 | $0.5 |
| `gpt-5.6-terra` | openai, inference-net-chat | 1.05M | $2 | $0.2 | $2.5 | $12 |
| `gpt-6-sol` | openai | 1.05M | $2 | $0.2 | $2.5 | $10 |
| `gpt-5.6-sol` | openai, inference-net-chat | 1.05M | $4 | $0.4 | $5 | $20 |
| `gpt-5.6-luna` | openai, inference-net-chat | 1.05M | $0.2 | $0.02 | $0.25 | $1.2 |
| `claude-sonnet-5` | anthropic | 1M | $2 | $0.2 | $2.5 | $10 |
| `claude-opus-5` | anthropic, deepinfra-labs, inference-net-chat | 1M | $5 | $0.5 | $6.25 | $25 |
| `claude-fable-5` | anthropic, deepinfra-labs, inference-net-chat | 1M | $10 | $1 | $12.5 | $50 |
| `claude-sonnet-4-5` | anthropic | 200K | $3 | $0.3 | $3.75 | $15 |
| `claude-haiku-4-5` | anthropic, deepinfra-labs, inference-net-chat | 200K | $1 | $0.1 | $1.25 | $5 |
| `gpt-4o-mini` | openai, inference-net-chat | 128K | $0.15 | $0.075 | – | $0.6 |
| `gpt-4.1-mini` | openai, inference-net-chat | 1.05M | $0.4 | $0.1 | – | $1.6 |
| `gpt-4o` | openai, inference-net-chat | 128K | $2.5 | $1.25 | – | $10 |
| `deepseek/deepseek-v4.1-flash` | runinfra-chat, doubleword-chat, wafer, inference-net-chat, morphllm, deepinfra-chat, nebius-chat, novita-chat | 1.05M | $0.3 | $0.006 | – | $1.2 |
| `tencent/hy4-preview` | deepinfra-chat, novita-chat | 1.05M | $0.834 | $0.042 | – | $2.501 |
| `z-ai/glm-5.3-flash` | wafer, sail-research-chat, inference-net-chat, runinfra-chat, morphllm, deepinfra-chat, nebius-chat, novita-chat | 1.05M | $0.15 | $0.015 | – | $0.5 |
| `z-ai/glm-5.3` | wafer, inference-net-chat, sail-research-chat, runinfra-chat, morphllm, deepinfra-chat, mistral-ai, nebius-chat | 1.05M | $1.4 | $0.06 | – | $4.4 |
| `deepseek/deepseek-v4-flash-0731` | sail-research-chat, doubleword-chat, deepinfra-chat, nebius-chat, streamlake, chutes-chat, inference-net-chat | 1.05M | $0.14 | $0.015 | – | $0.28 |
| `moonshotai/kimi-k3` | chutes-chat, sail-research-chat, wafer, morphllm, deepinfra-chat, nebius-chat, novita-chat, inference-net-chat | 1.05M | $3 | $0.285 | – | $15 |
| `z-ai/glm-5.2` | doubleword-chat, wafer, chutes-chat, inference-net-chat, morphllm, deepinfra-chat, novita-chat, streamlake | 1.05M | $1.4 | $0.26 | – | $4.4 |
| `minimax/minimax-m3` | deepinfra-chat, nebius-chat, streamlake | 524K | $0.3 | $0.056 | – | $1.2 |
| `minimax/minimax-m3.1-flash` | streamlake | 1M | $0.15 | $0.015 | – | $0.6 |
| `deepseek/deepseek-v4-pro` | deepinfra-chat, novita-chat, streamlake, inference-net-chat | 1.05M | $1.6 | $0.135 | – | $3.2 |
| `deepseek/deepseek-v4-pro-0813` | sail-research-chat, deepinfra-chat, nebius-chat, novita-chat, streamlake, doubleword-chat | 1.05M | $1.32 | $0.1 | – | $3.96 |
| `google/gemma-4-31b-it` | deepinfra-chat, novita-chat, sail-research-chat, doubleword-chat | 262K | $0.2 | – | – | $0.4 |
| `qwen/qwen3.6-27b` | chutes-chat, deepinfra-chat, streamlake | 262K | $0.32 | – | – | $3.2 |
| `qwen/qwen3.6-35b-a3b` | deepinfra-chat, doubleword-chat, streamlake | 262K | $0.1 | $0.1 | – | $0.95 |
| `xiaomi/mimo-v2.6-pro` | deepinfra-chat, novita-chat | 1.05M | $0.435 | $0.0036 | – | $0.87 |
| `xiaomi/mimo-v2.6-flash` | deepinfra-chat, novita-chat | 1.05M | $0.14 | $0.0028 | – | $0.28 |
| `tencent/hy3` | novita-chat | 262K | $0.14 | $0.035 | – | $0.58 |
| `claude-opus-4-8` | deepinfra-labs, inference-net-chat | 1M | $5 | – | – | $25 |
| `claude-opus-4-7` | deepinfra-labs, inference-net-chat | 1M | $5 | – | – | $25 |
| `claude-sonnet-4-6` | deepinfra-labs, inference-net-chat | 1M | $3 | – | – | $15 |
| `google/gemini-3.7-flash` | deepinfra-labs, inference-net-chat | 1M | $0.75 | – | – | $3.75 |
| `google/gemini-3.5-flash` | deepinfra-labs, inference-net-chat | 1M | $1.5 | – | – | $9 |
| `google/gemini-3.1-pro` | deepinfra-labs | 1M | $2 | – | – | $12 |
| `google/gemini-3.1-flash-lite` | deepinfra-labs, inference-net-chat | 1M | $0.25 | – | – | $1.5 |
| `google/gemini-2.5-pro` | deepinfra-labs, inference-net-chat | 1M | $1.25 | – | – | $10 |
| `google/gemini-2.5-flash` | deepinfra-labs, inference-net-chat | 1M | $0.3 | – | – | $2.5 |
| `openai/gpt-oss-120b` | deepinfra-chat | 131K | $0.037 | – | – | $0.17 |
| `openai/gpt-oss-20b` | deepinfra-chat | 131K | $0.03 | – | – | $0.14 |
| `morph/morph-v3-fast` | morphllm | 262K | $0.8 | – | – | $1.2 |
| `morph/morph-v3-large` | morphllm | 262K | $0.9 | – | – | $1.9 |
| `morph/kimi-k3-fast` | morphllm | 1.05M | $6 | $0.6 | – | $22.5 |
| `mistral-large-4` | mistral-ai | 1.05M | $1.36 | $0.14 | – | $4.18 |
| `mistral-medium-2604` | mistral-ai | 262K | $1.5 | $1.5 | – | $7.5 |
| `mistral-small-2603` | mistral-ai | 262K | $0.15 | $0.15 | – | $0.6 |
| `mistral-large-2512` | mistral-ai | 262K | $0.5 | $0.5 | – | $1.5 |
| `ministral-14b-2512` | mistral-ai | 262K | $0.2 | $0.2 | – | $0.2 |
| `ministral-8b-2512` | mistral-ai | 262K | $0.15 | $0.15 | – | $0.15 |
| `ministral-3b-2512` | mistral-ai | 131K | $0.1 | $0.1 | – | $0.1 |
| `codestral-2508` | mistral-ai | 256K | $0.3 | $0.3 | – | $0.9 |
| `qwen/qwen3.5-397b-a17b` | chutes-chat, nebius-chat, streamlake | 262K | $0.6 | $0.6 | – | $3.6 |
| `qwen/qwen3-235b-a22b-2507` | nebius-chat | 262K | $0.2 | $0.2 | – | $0.6 |
| `qwen/qwen3.8-27b` | runinfra-chat, chutes-chat, wafer, nebius-chat, doubleword-chat, streamlake | 262K | $0.45 | $0.45 | – | $3 |
| `nvidia/nemotron-3-ultra-550b-a55b` | nebius-chat, doubleword-chat | 1.05M | $1 | $1 | – | $3 |
| `nvidia/nemotron-3.5-lightning` | nebius-chat, runinfra-chat | 1.05M | $0.06 | $0.06 | – | $0.24 |
| `moonshotai/kimi-k2.6` | chutes-chat, nebius-chat, streamlake, inference-net-chat | 262K | $0.95 | $0.95 | – | $4 |
| `claude-opus-4-6` | inference-net-chat | 1M | $5 | $0.5 | – | $25 |
| `deepseek/deepseek-v4-flash` | streamlake, inference-net-chat | 1M | $0.23 | $0.0028 | – | $0.62 |
| `google/gemini-2.5-flash-lite` | inference-net-chat | 1.05M | $0.1 | $0.01 | – | $0.4 |
| `google/gemini-3-flash-preview` | inference-net-chat | 1.05M | $0.5 | $0.05 | – | $3 |
| `google/gemini-3.1-pro-preview` | inference-net-chat | 1.05M | $2 | $0.2 | – | $12 |
| `google/gemini-3.5-flash-lite` | inference-net-chat | 1.05M | $0.3 | $0.03 | – | $2.5 |
| `google/gemini-3.6-flash` | inference-net-chat | 1.05M | $1.5 | $0.15 | – | $7.5 |
| `google/gemma-3-4b-it` | inference-net-chat | 131K | $0.05 | – | – | $0.1 |
| `z-ai/glm-5` | inference-net-chat, streamlake | 205K | $0.95 | $0.2 | – | $2.55 |
| `z-ai/glm-5.2-fast` | inference-net-chat | 1M | $2.6 | $0.4 | – | $8.8 |
| `gpt-4.1` | inference-net-chat | 1.05M | $2 | $0.5 | – | $8 |
| `gpt-4.1-nano` | inference-net-chat | 1.05M | $0.1 | $0.025 | – | $0.4 |
| `gpt-5-mini` | inference-net-chat | 400K | $0.25 | $0.025 | – | $2 |
| `gpt-5-nano` | inference-net-chat | 400K | $0.05 | $0.005 | – | $0.4 |
| `gpt-5.4` | inference-net-chat | 1.05M | $2.5 | $0.25 | – | $15 |
| `gpt-5.5` | inference-net-chat | 1.05M | $5 | $0.5 | – | $30 |
| `x-ai/grok-4.5` | inference-net-chat | 500K | $2 | $0.3 | – | $6 |
| `x-ai/grok-4.6` | inference-net-chat | 500K | $2 | $0.5 | – | $6 |
| `moonshotai/kimi-k2.5` | inference-net-chat | 262K | $0.8 | $0.13 | – | $3.95 |
| `moonshotai/kimi-k3-fast` | inference-net-chat | 1.05M | $4.5 | $0.45 | – | $22.5 |
| `meta-llama/llama-3.2-1b-instruct` | inference-net-chat | 60K | $0.027 | – | – | $0.201 |
| `meta-llama/llama-3.2-3b-instruct` | inference-net-chat | 131K | $0.05 | – | – | $0.33 |
| `meta-llama/llama-3.3-70b-instruct` | inference-net-chat | 131K | $0.13 | – | – | $0.4 |
| `nvidia/nemotron-3-super-120b-a12b` | inference-net-chat | 262K | $0.102 | – | – | $0.48 |
| `qwen/qwen3-14b` | inference-net-chat | 131K | $0.2275 | – | – | $0.91 |
| `qwen/qwen3-30b-a3b` | inference-net-chat | 41K | $0.12 | – | – | $0.5 |
| `qwen/qwen3.7-plus` | inference-net-chat | 1M | $0.4 | $0.08 | – | $1.6 |
| `inference-net/schematron-v2-small` | inference-net-chat | 128K | $0.05 | $0.05 | – | $0.23 |
| `inference-net/schematron-v2-turbo` | inference-net-chat | 128K | $0.03 | $0.03 | – | $0.15 |
| `deepseek/deepseek-v4.1-flash-1017` | sail-research-chat, wafer, runinfra-chat, doubleword-chat, particle | 1.05M | $0.15 | $0.01 | – | $0.6 |
| `ornith-ai/ornith-1.5-122b` | runinfra-chat | 262K | $0.26 | $0.07 | – | $2.08 |
| `ornith-ai/ornith-1.5-35b-a3b` | runinfra-chat | 262K | $0.1 | $0.01 | – | $0.4 |
| `google/gemma-4-31b-it-nvfp4` | chutes-chat, sail-research-chat | 262K | $0.14 | $0.07 | – | $0.4 |
| `google/gemma-4-12b-it` | sail-research-chat | 16K | $0.3 | $0.15 | – | $2 |
| `deepseek/deepseek-v3.2` | chutes-chat | 131K | $1 | $0.1 | – | $1 |
| `z-ai/glm-5.1` | chutes-chat | 203K | $0.9 | $0.09 | – | $2.9 |
| `qwen/qwen3-vl-235b-a22b-instruct` | doubleword-chat | 262K | $0.21 | $0.1155 | $0.2625 | $1.9 |
| `qwen/qwen3.5-35b-a3b-dottxt` | doubleword-chat | 262K | $0.35 | $0.19999 | $0.4375 | $1.2 |
| `particle-ai/particle-1` | particle | 1.05M | $0.25 | $0.03 | – | $1 |

### Provider order

The Providers column lists, cheapest first, the routes that a request with no provider named can use. It has at most eight routes. [Provider routing](/docs/provider-routing) describes how RouterPlus picks among them.

- A provider past those eight can still be chosen with `provider.only` or `provider.order`. These providers are past the eight: `novita-chat`, `streamlake` and `doubleword-chat` on `z-ai/glm-5.3`; `streamlake`, `doubleword-chat` and `particle` on `z-ai/glm-5.3-flash`; `streamlake` on `deepseek/deepseek-v4.1-flash`; `doubleword-chat` on `moonshotai/kimi-k3`.
- The column order is not the order RouterPlus tries. By default RouterPlus orders a model's providers by Routing Cost, the lowest cost per good answer, and providers inside the competitive band share traffic. Before a provider has measurements, the order is cheapest first. `provider.order` or `provider.sort` in the request overrides the default. See [Provider routing](/docs/provider-routing#default-order-lowest-cost-per-good-answer).
- The Claude models have two providers at the same price: Anthropic and DeepInfra (`deepinfra-labs`).
- On the open models that Morph and DeepInfra both serve, Morph comes before DeepInfra when their prices are equal. On GLM 5.3, Mistral follows both at that same price.
- Chutes (`chutes-chat`) is first on Gemma 4 31B NVFP4, Kimi K2.6, Kimi K3, Qwen3.5 397B A17B and Qwen3.6 27B. It is second on GLM 5.2 (after `doubleword-chat`) and Qwen3.8 27B (after `runinfra-chat`). A request routed to Chutes is billed Chutes's own price, which on those seven is below the row's list price.
- On DeepSeek V4 Flash 0731, Chutes costs more than the row and is sixth, after `streamlake` at the same price.
- On DeepSeek V4 Flash, `streamlake` is first, at its own price, below the row's.

[`/api/models.json`](https://demo.routerplus.com/api/models.json) lists every provider of a model and its price.

### Model notes

- **Featured order.** The first eleven rows lead the Models page and the playground in this order.
- **MiniMax on StreamLake.** On StreamLake, `minimax/minimax-m3` and `minimax/minimax-m3.1-flash` accept text only. Those routes also refuse requests above 512,000 bytes of encoded JSON, because larger inputs can need a different rate. MiniMax M3 image requests can use its other providers. M3.1 Flash has no other provider, so an image request returns a 400 before RouterPlus calls the provider.
- **Qwen3.6 35B A3B on StreamLake.** StreamLake serves `qwen/qwen3.6-35b-a3b` with internal reasoning off, to keep output within the requested token limit. A requested reasoning effort other than `none` appears in `x-tm-dropped-params`. With `provider.require_parameters: true`, that request cannot use this route.
- **Gemma 4 31B NVFP4.** `google/gemma-4-31b-it-nvfp4` has the same weights as `google/gemma-4-31b-it`, quantized to NVFP4 (NVIDIA's 4-bit floating-point format), so it has its own row. Its 262K context is the largest of its routes. Chutes, tried first, holds 131,072 tokens. A longer prompt goes to `sail-research-chat`, which holds 262,144.
- **GPT-6 and GPT-5.6.** RouterPlus calls OpenAI's seven GPT-6 and GPT-5.6 models on OpenAI's Responses API, where they take function tools and custom (free-form) tools. You can call them on either chat API, or on [POST /v1/responses](/docs/api-responses). On these seven, the chat APIs drop `stop`, `n`, `seed`, `logprobs`, `logit_bias` and the penalties. An output limit under 16 tokens is sent as 16.
- **Default sampling only.** The Claude 5 models (Opus, Fable, Sonnet and Haiku) and OpenAI's GPT-6 and GPT-5.6 models take only their default sampling. Their providers refuse `temperature` other than 1, and any `top_p` or `top_k`, with a 400. RouterPlus leaves these out for those models on every route. `x-tm-dropped-params` names each drop, except `temperature: 1` and `top_p: 1`, which change nothing. Under `require_parameters`, a drop is a 400.
- **Long prompts.** OpenAI bills a prompt over 272K input tokens at twice the input rate and 1.5 times the output rate. Anthropic bills a Claude Haiku 5.5 prompt over 100K tokens at five times its rate. The catalog lists the base rate.
- **Your deployed models.** A model or mix you deploy from [Optimize](/docs/model-search) gets its own id, `tm/<name>-v<n>`. Call it on `POST /v1/chat/completions` with a key of the organization that owns it. It is not listed in the catalog.

### Fast Apply (code merge)

`morph/morph-v3-fast` and `morph/morph-v3-large` merge an edit into a file. They are chat models on `POST /v1/chat/completions` that take one user message with the instruction, the original file and the edit, each in its own tag. The edit can skip unchanged lines with `// ... existing code ...` (or `# ...` in Python). The reply's `content` is the complete merged file.

```bash
curl https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"morph/morph-v3-fast",
       "messages":[{"role":"user","content":"<instruction>Add an optional third number</instruction>\n<code>def total(a, b):\n    return a + b</code>\n<update>def total(a, b, c=0):\n    # ... existing code ...\n    return a + b + c</update>"}]}'
```

### Image models

Image models bill in tokens and answer only on [POST /v1/images/generations](/docs/api-images). The per-image ceiling is the most output tokens one image can bill. The balance hold is `n` times the ceiling.

| Model id | Sizes | Qualities | Per-image ceiling | Text in /1M | Image out /1M |
|---|---|---|---|---|---|
| `gpt-image-1` | `1024x1024 · 1536x1024 · 1024x1536` | `low · medium · high` | 6,240 | $5 | $40 |
| `gpt-image-1-mini` | `1024x1024 · 1536x1024 · 1024x1536` | `low · medium · high` | 6,500 | $2 | $8 |
| `gpt-image-1.5` | `1024x1024 · 1536x1024 · 1024x1536` | `low · medium · high` | 6,500 | $5 | $32 |
| `gpt-image-2` | `1024x1024 · 1536x1024 · 1024x1536` | `low · medium · high` | 8,232 | $5 | $30 |
| `gpt-image-2.5-flare` | `1024x1024 · 1536x1024 · 1024x1536` | `low · medium · high · xhigh · max` | 8,232 | $5 | $30 |
| `gpt-image-2.5-sunburst` | `1024x1024 · 1536x1024 · 1024x1536` | `low · medium · high · xhigh · max` | 8,232 | $5 | $30 |

`size` and `quality` also accept `auto`, which lets the provider choose. Each model's full descriptor map is in `/api/models.json` under `supported_parameters`.

### Decision models

Decision models answer only on [POST /v1/decisions](/docs/api-decisions). They write no text. They return one typed answer for each question about a state. Each model takes its own question format.

| Model id | Made by | Providers | Question kinds | Input /1M | Cached /1M |
|---|---|---|---|---|---|
| `typesafe/jev-1.13` | TypeSafe | `typesafe` | choice, score, yes/no (System One) | $0.042 | – |
| `bespokelabs/nimble-v4` | Bespoke Labs | `bespokelabs` | choice, score, yes/no (System One, the same as Jev) | $0.04 | – |
| `bespokelabs/nimble-v4-max` | Bespoke Labs | `bespokelabs` | choice, score, yes/no (System One, the same as Jev) | $0.09 | – |
| `cloudflare/clef` | Cloudflare | `workers-ai` | choice, score, yes/no (System One, the same as Jev) | $0.24 | – |
| `cloudflare/clef-flash` | Cloudflare | `workers-ai` | choice, score, yes/no (System One, the same as Jev) | $0.09 | – |
| `routerplus/decider-2b` | RouterPlus | `routerplus` | choice, score, yes/no (System One, the same as Jev) | $0 | $0 |
| `routerplus/kev-4b` | RouterPlus | `routerplus` | choice, score, yes/no (System One, the same as Jev) | $0 | $0 |
| `perplexity/pplx-decider-v1-27b` | Perplexity | `perplexity-decisions` | choice, score, yes/no (System One, the same as Jev) | $0.04 | – |
| `openai/gpt-6-luna-decisions` | OpenAI | `openai-decisions` | choice, score, yes/no (System One, the same as Jev) | $0.10 | – |
| `levanto/sage-1.2` | Levanto | `levanto` | choice, score, yes/no (System One, the same as Jev) | $0.05 | – |
| `scaledown/classify-1` | ScaleDown | `scaledown` | choice, score, yes/no (System One, the same as Jev) | $0.05 | – |
| `fastino/gliner-2.5-decide` | Fastino | `fastino` | classifications (single- and multi-label), entities, structures, relations | $0.03 | – |

Output is free on every decision model. A discount can apply to decision models: see [Pricing](/docs/pricing).

**Bespoke Nimble v4 and v4 Max**

- Nimble v4 is $0.04 per million input tokens, and Nimble v4 Max is $0.09. These are Bespoke's own prices with no markup.
- Nimble v4 is the faster, lower-cost model. v4 Max is Bespoke's most accurate model.
- Both have a 262,144-token context for each question's prompt, including the state, and take 1 to 64 questions per call.
- They share one pool, which takes 7 requests at once (5 per organization). Bespoke allows the RouterPlus account 8, and RouterPlus keeps one for its test environment.
- The gateway also accepts `bespokelabs/nimble-latest`, which the table does not list. Bespoke picks the model that answers it (Nimble v4 at the time of writing), at $0.04 per million input tokens.

**Clef and Clef-flash**

- Cloudflare's own decision models. Cloudflare Workers AI serves them at Cloudflare's own price with no markup: $0.24 and $0.09 per million input tokens.
- They take 1 to 64 questions per call, each question id 1 to 100 letters, digits, `_`, `.` or `-`, and a context of 65,536 tokens.
- They share one pool: 200 requests a minute (150 per organization) and at most 8 at a time (6 per organization).

**Decider 2B and Kev 4B**

- RouterPlus's own models, on RouterPlus GPUs in the United States, and free: $0 in and out.
- They share one pool. Their contexts are 25,600 and 8,192 tokens.
- RouterPlus answers an identical request from its cache for 600 s.

**Perplexity Decider v1 27B**

- Perplexity's own decision model, served by Perplexity's API at Perplexity's own price with no markup: $0.04 per million input tokens.
- Perplexity bills the state once per question, so the cost grows with the number of questions.
- It takes 1 to 128 questions per call, and 262,144 tokens for each question's prompt.
- Its pool takes 540 requests a minute (405 per organization) and 8 at once (6 per organization).

**GPT-6 Luna Decisions**

- OpenAI's own decision model, served by OpenAI's API at OpenAI's own price with no markup: $0.10 per million input tokens.
- OpenAI's API uses its own format. RouterPlus translates your System One body to it, and translates the answers back.
- OpenAI bills the state once per call, and each question adds about 150 input tokens.
- It takes 1 to 200 questions per call, 2 to 255 options in a choice, and a context of 272,000 tokens.
- Its pool takes 24,000 requests a minute (18,000 per organization) and 64 in flight at once (48 per organization), so your organization's own limits apply first.
- OpenAI's own name for it, `gpt-6-luna`, is the RouterPlus id of OpenAI's chat model GPT-6 Luna.

OpenAI can decline one question and answer the rest: that answer's `type` is
`refusal`.

**Sage** bills the state once per question, as Perplexity Decider does.

**ScaleDown Classify-1**

- ScaleDown's Jev-compatible decision model on its `/v1/scaledown` endpoint.
- It costs ScaleDown's own price with no markup: $0.05 per million input tokens, output free, the price of every ScaleDown model.
- It runs each question as its own call, so input grows with the number of questions.
- Its `state` can also carry `document` (a base64 image or PDF) with `document_mime_type`, which ScaleDown reads by OCR.
- Its context is 1,000,000 tokens.

### Cache pricing

Most models also have cache pricing: `cached_prompt` (the Cached column above), and `cache_write` where the provider bills it (the Cache write column). Each model's page and `/api/models.json` have the full prices per SKU. A model with no separate cache or reasoning price bills those tokens **at the base rate**. A missing SKU never means free.

### Task models (ScaleDown)

ScaleDown's two chat models each do one task. Call them on `POST /v1/chat/completions` (or `POST /v1/messages`). The answer's content is a JSON string with the task's result:

| Model id | Task | Input /1M | Cached /1M | Output /1M | Context |
|---|---|---|---|---|---|
| `scaledown/extract-1` | Fields and entities from the last user message, defined by a `response_format` of type `json_schema` (each property is a field; its `description` is the hint) | $0.05 | – | $0 | 1M |
| `scaledown/summarize-1` | An abstractive summary of the last user message; the system message holds instructions, and `max_tokens` is honored | $0.05 | – | $0 | 1M |

- Output is free. These models bill input tokens only, and report 0 output tokens.
- ScaleDown answers in one piece. With `stream: true` you still get a stream (role and content, the finish, usage with its cost, `[DONE]`). The stream starts when the complete answer is ready.
- To classify with ScaleDown, use `scaledown/classify-1` on [POST /v1/decisions](/docs/api-decisions).
- Both models also read an image or a PDF. On `/v1/chat/completions`, put one `image_url` part (a base64 `data:` URL) or one `file` part (base64 `file_data`) in the last user message, with a text part next to it. ScaleDown reads the attachment by OCR. A web URL, more than one attachment, or an attachment with no text is a 400 that says which.

### One model, several providers

More than one provider can serve a model id. The catalog shows one card. The model page lists every provider with its own uptime and TTFB.

**Routing**

- By default, RouterPlus orders providers by the estimated cost of one good answer. Until providers have measurements, the cheapest goes first. See [Provider routing](/docs/provider-routing).
- `provider.order` comes first. `provider.sort: "price"` orders providers by input price plus output price per million, at each live price, and also overrides a routing policy's saved order. `provider.max_price` leaves out a provider above your limit.
- RouterPlus fails over **before the first byte** only, never in the middle of an answer.
- A provider that fails twice in a row, or leaves two requests in a row without an answer, rests for 30 seconds (see [Routing](/docs/routing#health-circuits-and-cooldowns)).

**What you pay**

You pay the price of the provider that serves your request: the one you name with `provider.only`, else the one routing picks, failover included.

- Providers that do not set their own price share the model's list price. The catalog build enforces this.
- A provider reached through an aggregator is charged at the aggregator's listing price, which is the model's list price. Its row on the model page can also show the host's own published price, labelled "Host's list price". You do not pay that price.

**Prices on the pages**

- The pages use only the prices that you pay.
- The model page shows each provider's live price, the price in force now, without scheduled changes. Its bold cells mark the lowest input, cached input and output price, each in its own column.
- The model page's Cheapest sort ranks providers by input price plus output price, the same rule as `provider.sort: "price"`.
- `/api/models.json` also lists each provider's scheduled changes in `scheduled_pricing`, at most 20 per provider. A revision that leaves the price unchanged is not listed.
- In the directory, a decision model's price is the cheapest price you can get by naming a provider. It is labelled "from" when it is below the price of the provider that the page lists first. The model page's "from" price names the same provider.
- The directory's price sort, price filter and "from" price rank providers by input price, then by output price. On one model page, the "from" price can therefore name a provider that the Cheapest sort does not put first.

#### Self-serve price changes

A self-serve provider sets its own price and can change it without a release.

- Every price change starts at a 5-minute mark (10:00, 10:05, 10:10 UTC), at least 2 minutes after the change is written. A change written at 10:02:30 starts at 10:05. All gateways switch at the same time.
- A decrease starts at the next mark. An increase starts only after notice.
- A turn-off of a model, or a lower limit, applies at once. The pages hide the model at that provider and show the lower limits. Only a limit that the listing declares, on a chat model, can be lowered.
- A turn-off has no price, so it is never an increase and needs no notice.
- A turn-on is compared with the last price you could pay while the model was on. When it is higher, it is an increase, and the provider must schedule the turn-on at least the notice period ahead.
- A model that no provider serves now, but that one provider turns back on later, has a page that says it is not served now. It is not in the directory until it is back.

An operator can freeze a self-serve provider. While it is frozen:

- You pay its listing's own price from the manifest files, and the pages show that price.
- `/api/models.json` lists its scheduled turn-offs and turn-ons, but not its scheduled price changes.
- A model that it turned off stays hidden, and lower limits stay in force.
- When an operator unfreezes it and accepts its increases, they can apply at once.

Other exceptions:

- For a provider with a price feed, the site schedules a fed increase at the earliest for the notice period plus 5 minutes after it reads the feed. That can be later than the time the provider publishes.
- An operator can apply an increase without notice. Each such change is audited.

**How fresh the pages are**

| What | Delay |
|---|---|
| Pushed prices | The site reads them every 30 s |
| A page with a self-serve provider | Cached at the edge for up to 60 s (`s-maxage=60`), plus up to 60 s while it refreshes (`stale-while-revalidate=60`), never past the next 5-minute mark |
| A pushed change in `/api/models.json` | Listed as scheduled within about 2.5 minutes. The pages switch at its mark |
| A pushed increase in `/api/models.json` | Listed for its full notice period before it starts, also with a turn-on |
| Manifest files (a release) | Up to 60 s more, because the site reads them once a minute. A changed list price starts at the next 5-minute mark and shows as scheduled until then |
| Any other page | Changes only with a release. The edge can serve a copy up to about 11 minutes old |

After the site starts, no page is cached until its first read.

The gateway charges the price in force when your request arrives, whatever a cached page says. A price change applies at its 5-minute mark. A turn-off or a lower limit applies within about 30 s, when the gateway next reads the prices.

#### Who served the request

- `x-tm-provider` names the provider that served you.
- `x-tm-upstream-model` has the id sent to the provider, when it knows the model under its own id. The response's `model` field still names the model you asked for.
- `x-tm-served-by` names the provider that ran the request, when a route reports one.

**Inference.net.** `inference-net-chat` is an ordinary provider at its own price, also on models other providers serve. Routing tries it first where it is the cheapest (GLM 5.3 Flash, for example), else as a fallback. Name it in `provider.only` or `provider.order` (`"inference-net"` works too) to send a request there first. You pay Inference.net's price, which the model page shows on its row.

```json
{ "model": "moonshotai/kimi-k2.6", "messages": [...], "provider": { "only": ["inference-net-chat"] } }
```

### Date-pinned ids

The catalog lists **one id per model** (`claude-haiku-4-5`). Vendors also ship dated snapshots (`claude-haiku-4-5-20251001`, `gpt-4o-2024-08-06`) and `-latest` spellings, and tools such as Claude Code send them.

- A snapshot or `-latest` id of a catalog model works without being listed. RouterPlus resolves it to its family **for routing and pricing only**.
- RouterPlus forwards your original id to the provider unchanged. It never moves a snapshot pin to a newer version.
- The response's `model` field shows the snapshot that served you.
- If a vendor retires a pinned snapshot, you get the vendor's own error. RouterPlus does not upgrade the pin.
- Unknown ids, dated or not, return a 404 that names the id.
- A key bound to your own provider connection matches ids literally, so list the exact ids you send there.

### Claude names for Cursor

Cursor sends a model name that starts with `claude-` to Anthropic, never to a custom base URL. Each Claude model therefore also answers to a second name: `rp-` and its id without `claude-`.

| Name | Model |
|---|---|
| `rp-opus-5-5` | `claude-opus-5-5` |
| `rp-sonnet-5-5` | `claude-sonnet-5-5` |
| `rp-haiku-5-5` | `claude-haiku-5-5` |
| `rp-fable-5-1` | `claude-fable-5-1` |
| `rp-opus-5` | `claude-opus-5` |
| `rp-sonnet-5` | `claude-sonnet-5` |
| `rp-fable-5` | `claude-fable-5` |
| `rp-sonnet-4-5` | `claude-sonnet-4-5` |
| `rp-haiku-4-5` | `claude-haiku-4-5` |

- Each name is one model, billed at the price of the provider that serves it, as for the model's own id.
- A name takes no date pin. `rp-sonnet-4-5-20250929` is a 404.
- The provider receives the model's own id.
- The answer's `model` field has the name you sent. A Claude stream on `/v1/messages` is relayed as Anthropic sent it, so its first event names the model's own id.
- Billing, `/v1/usage` and `/v1/generation` record the model's own id.
- `GET /v1/models` lists each name right after its model, with `canonical_slug` set to the model's id, so model pickers that read it show the names.
- A key bound to your own provider connection or to a routing policy takes ids literally, so these names do not work on it.

In your own code, use the catalog ids.

### Id grammar

Ids match `[a-zA-Z0-9][a-zA-Z0-9._/-]{0,127}`: letters, digits, `.` `_` `/` `-`, at most 128 characters. `:` is not allowed, so an id has no suffix.

## Providers are manifests

A provider lists its models in a JSON manifest: its endpoint, wire format, settlement terms, data policy, and a model list with prices. [List your models](/docs/for-providers) is the full reference.

```json
{
  "provider": { "id": "anthropic", "name": "Anthropic (direct API)", "prompt_logging": "retained" },
  "endpoint": { "base_url": "https://api.anthropic.com/v1", "dialect": "anthropic" },
  "models": [{
    "id": "claude-haiku-4-5",
    "context_length": 200000,
    "max_output_tokens": 64000,
    "pricing": [
      { "type": "prompt", "unit": "token", "cost_usd_per_million": "1" },
      { "type": "completion", "unit": "token", "cost_usd_per_million": "5" }
    ]
  }]
}
```

What this means for you as a buyer:

- A model is listed only after it passes a live conformance suite against the provider's endpoint. The suite checks real usage, well-formed streams, non-empty answers, a 4xx with no fallback for an unknown model, and that the response names the model you asked for. A dated snapshot of the same id, or a target the manifest declares in `resolves_to`, passes. `gpt-4o` answered by `gpt-4o-mini` fails.
- `is_ready: false` stages a model: it is never routed or listed. After its `deprecation_date`, a model leaves the catalog and the public pages.
- Providers that are neither self-serve nor opt-in (`"routing": "opt_in"`) must agree on a shared model id's price and kind, or the catalog build fails. They share the model's list price. Self-serve and opt-in providers each charge their own price.
- When other providers also serve a model, an opt-in provider serves it only to a request that names that provider.
- Prices are stored as effective-dated, append-only rows. Old requests keep the price they were dispatched at. A changed list price starts at the first 5-minute mark (UTC) at least 2 minutes after the release writes it. A new model's first price applies at once.

## The directory

[`https://demo.routerplus.com/`](https://demo.routerplus.com/) lists every live model in three sections: chat, image and decision. Each kind is called on a different route.

- A row shows the model's context length and max output (for a chat model), or its sizes and qualities (for an image model).
- Directory rows and model pages show no request counts for any model, including newly added models.
- The default decision order starts with Jev 1.13, GPT-6 Luna Decisions, Nimble v4 Max, ScaleDown Classify-1 and Nimble v4.
- Search with `/?q=haiku`. It matches model, lab and provider names.
- Filter by kind, context length, supported parameters, zero data retention, input price, lab and provider.
- Sort by use, name, price or context.

Each model's page at `/models/<id>` shows the values that depend on the provider. It has one row per provider, with prices per SKU, latency, uptime and the provider's prompt-retention policy, and snippets to copy. See [Data policy](/docs/data-policy) for how RouterPlus handles request data.

### Uptime

Uptime and latency come from the last 24 hours of routed traffic. When a provider has fewer than 100 counted attempts in that window, the model page shows a dash (**-**) in place of a percentage. Failures you cause (your 400s, your cancelled streams) do not count against a provider.

## Machine-readable: /api/models.json

The catalog as JSON, with no auth:

```bash
curl -s https://demo.routerplus.com/api/models.json
```

```json
{
  "models": [
    {
      "id": "claude-haiku-4-5",
      "display_name": "Claude Haiku 4.5",
      "output_modalities": ["text"],
      "providers": [
        { "id": "anthropic", "name": "Anthropic (direct API)", "dialect": "anthropic", "prompt_logging": "retained",
          "pricing_usd_per_million": { "prompt": "1", "cached_prompt": "0.10", "cache_write": "1.25", "completion": "5" } }
      ],
      "served_by": [
        { "id": "anthropic", "name": "Anthropic", "zdr": false,
          "routes": [ { "via": "anthropic", "direct": true } ] }
      ],
      "context_length": 200000,
      "max_output_tokens": 64000,
      "pricing_usd_per_million": {
        "prompt": "1", "cached_prompt": "0.10",
        "cache_write": "1.25", "completion": "5"
      },
      "lab": { "id": "anthropic", "name": "Anthropic" },
      "weights": "proprietary"
    }
  ],
  "gateway": "https://demo-api.routerplus.com/v1",
  "docs": "https://demo.routerplus.com/docs"
}
```

| Field | Meaning |
|---|---|
| `id` | The exact string to put in your request's `model` field |
| `output_modalities` | `["text"]`, `["image"]` or `["decisions"]`: which route serves this model. Filter on it. Never guess from the id |
| `providers` | Every deployment serving this id, primary first, each with `id`, `name`, `dialect`, `prompt_logging`, its price, and `upstream_id` when the provider knows the model under its own name |
| `pricing_usd_per_million` (per provider) | What you pay when this provider serves the request, live: decimal-string USD per 1M tokens, keyed by SKU type. Absent for a `reported_cost` listing (see `price_card`) |
| `price_revision` / `price_effective_from` (per provider) | Only for a self-serve provider with a pushed price in force for this model: the revision number and when it took effect (ISO 8601, UTC). Absent when the revision in force does not have this model, and while the provider is frozen: the listing's own price from the manifest files applies then |
| `scheduled_pricing` (per provider) | Only when a change is scheduled: `[{effective_from, rises, pricing_usd_per_million}]` in time order. `rises` is `true` when any SKU goes up against the last price you could pay before it. An entry with `enabled: false` turns the model off: it has no `pricing_usd_per_million`, and its `rises` is always `false`. A later entry without `enabled` turns the model on again at its price. Its `rises` compares that price with the last one you could pay while the model was on, and the notice period applies when it rises. While the provider is frozen, only turn-offs and turn-ons are listed, at the price you pay then |
| `enabled` (per provider) | Only `false`: the provider has the model off now, and a turn-on is in its `scheduled_pricing`. Such an entry comes after the providers that serve the model, has no price, `price_revision` or `opt_in`, and is not in `served_by`. A provider that has the model off with no turn-on scheduled is not listed |
| `enabled` | Only `false`: no provider serves the model now, but one turns it back on later (see its `providers` entries). A request for it now that no BYOK route serves is answered as for an unknown model. Such an entry has no top-level `pricing_usd_per_million`, and `served_by` is empty |
| `dialect` (per provider) | The provider's **native** wire format, for information. Every chat model answers on both APIs |
| `prompt_logging` (per provider) | Provider's prompt retention: `none` or `retained` |
| `served_by` | The providers that run the model, one entry per provider. `routes[].via` is the `providers` entry the request goes to; `direct: true` means the request goes to the provider's own API. `zdr` is `true` when the provider offers Zero Data Retention for this model |
| `context_length` / `max_output_tokens` | Token limits; `max_output_tokens` may be `null`, and `context_length` is `null` for image models |
| `supported_parameters` | Image models only: the descriptor map the route enforces (`size`, `quality`, `n`, and the rest) |
| `pricing_usd_per_million` | The primary provider's live price (what a request that names no provider pays first): decimal-string USD per 1M tokens, keyed by SKU type. Other providers can charge another price: see each `providers[]` entry |
| `lab` | Who made the model, which can differ from who serves it. `null` when unknown |
| `weights` | `"open"` when the lab publishes the model's weights, `"proprietary"` when it never has, `null` when RouterPlus is not certain |

For an image model, `max_output_tokens` is the **per-image ceiling**. It does not cap a completion.

Prices are decimal **strings**, so float rounding loses nothing. Parse them with a decimal type. For the authenticated, SDK-shaped list (what `client.models.list()` calls), see [GET /v1/models](/docs/api-models).

---

Source: https://demo.routerplus.com/docs/migration.md

# Migrate

Most migrations change the base URL and the API key. If you come from another gateway, you also change the model ids. You keep your SDK, because RouterPlus serves both common API formats:

- OpenAI-compatible: `POST https://demo-api.routerplus.com/v1/chat/completions`, and
  `POST https://demo-api.routerplus.com/v1/responses` for OpenAI's Responses API
- Anthropic-compatible: `POST https://demo-api.routerplus.com/v1/messages`

Every chat model works on both APIs. RouterPlus translates requests, streams and errors in both directions. Image models use [POST /v1/images/generations](/docs/api-images). The translation rules are on [Wire compatibility](/docs/compat).

Every route accepts the key as `Authorization: Bearer $TM_API_KEY` or `x-api-key: $TM_API_KEY`.

> [!NOTE]
> To get a key, open [Sign up](https://demo.routerplus.com/signup), complete Clerk authentication
> and email verification, and save the first key shown after sign-in. Buy credits
> in [Billing](https://demo.routerplus.com/console/billing). Signup does not add free credit. If you have an account,
> [sign in](https://demo.routerplus.com/login) with the same verified email and create a key at
> [API keys](https://demo.routerplus.com/console/keys). There is no programmatic signup route.

## From the OpenAI SDK

Change two lines. `gpt-*` model ids are the provider's own ids (`gpt-4o-mini`, `gpt-4.1-mini`, `gpt-4o`), so your model strings usually stay the same.

```diff
 from openai import OpenAI

 client = OpenAI(
-    api_key=os.environ["OPENAI_API_KEY"],
+    base_url="https://demo-api.routerplus.com/v1",
+    api_key=os.environ["TM_API_KEY"],
 )
```

```diff
 import OpenAI from "openai";

 const client = new OpenAI({
-  apiKey: process.env.OPENAI_API_KEY,
+  baseURL: "https://demo-api.routerplus.com/v1",
+  apiKey: process.env.TM_API_KEY,
 });
```

The same client can call Claude models:

```python
client.chat.completions.create(
    model="claude-sonnet-4-5",  # Anthropic model, OpenAI format
    stream=True,
    messages=[{"role": "user", "content": "hello"}],
)
```

> [!NOTE]
> Output limits:
> - On the chat APIs, a request with no `max_tokens` or `max_completion_tokens` gets the
>   model's own maximum output. RouterPlus lowers it when needed so the input and output
>   fit the model's context window.
> - On the Responses API, RouterPlus adds no limit, and OpenAI applies the model's own maximum.
> - While such a request runs, the balance hold counts at most 32,768 output tokens. You pay
>   for the real usage.
> - A limit that you send (`max_tokens` on the chat APIs, `max_output_tokens` on the
>   Responses API) is held in full.

Code on OpenAI's Responses API (`client.responses.create`) works with the same base URL for OpenAI's models. Claude and the other chat models are a preview there. See [POST /v1/responses](/docs/api-responses).

**Codex** (the CLI, the desktop app and the IDE extension) calls the Responses API. Add a provider block to `~/.codex/config.toml`:

```toml
model = "gpt-6-luna"
model_provider = "routerplus"

[model_providers.routerplus]
name = "RouterPlus"
base_url = "https://demo-api.routerplus.com/v1"
env_key = "TM_API_KEY"
wire_api = "responses"
```

Then run `codex` with `TM_API_KEY` set. For the desktop app, which needs the key in the file, see [Coding agents](/docs/install#codex).

## From the Anthropic SDK

Change the base URL and the key. The Anthropic-compatible API is at the gateway root. Do not add `/v1` to `base_url`: the SDK appends `/v1/messages`.

```diff
 from anthropic import Anthropic

 client = Anthropic(
-    api_key=os.environ["ANTHROPIC_API_KEY"],
+    base_url="https://demo-api.routerplus.com",
+    api_key=os.environ["TM_API_KEY"],
 )
```

Claude model ids are Anthropic's own (`claude-sonnet-4-5`, `claude-haiku-4-5`, dated snapshots included). This SDK can also call GPT models on the same `/v1/messages` endpoint.

**Claude Code** needs environment variables only:

```bash
ANTHROPIC_BASE_URL=https://demo-api.routerplus.com ANTHROPIC_AUTH_TOKEN=$TM_API_KEY claude
```

- RouterPlus forwards your `anthropic-beta` header when an Anthropic-format provider serves the request.
- It keeps only allowlisted betas that do not change the price of a token: `prompt-caching`, `claude-code`, `interleaved-thinking`, and the values Claude Code needs on the Claude 5 models. The full list is on [Authentication](/docs/authentication).
- The `x-tm-dropped-params` header names each dropped value.
- Claude Code's `?beta=true` query string is accepted and routes normally. RouterPlus does not forward query strings to the provider.

For Claude Code's output limit, auto mode and advisor, see [Coding agents](/docs/install#output-limit-auto-mode-and-the-advisor).

## From another gateway

If your code calls another multi-provider gateway in the OpenAI format, change the client as in [From the OpenAI SDK](#from-the-openai-sdk). Then change the model ids. Usage comes in the last stream chunk with a `cost` field. The optional `HTTP-Referer` and `X-Title` attribution headers also work.

```diff
 completion = client.chat.completions.create(
-    model="anthropic/claude-sonnet-4.5",
+    model="claude-sonnet-4-5",
     stream=True,
     messages=[{"role": "user", "content": "hello"}],
 )
```

| Before | RouterPlus |
|---|---|
| `author/model` slugs (`anthropic/claude-sonnet-4.5`) | Provider ids (`claude-sonnet-4-5`). The full list is at [https://demo.routerplus.com/api/models.json](https://demo.routerplus.com/api/models.json) |
| Variant suffixes (`model:variant`) | None. An id with a suffix is not in the catalog and returns 404 |
| `usage.cost` in credits | `usage.cost` in USD, computed with the ledger's integer math. It covers the full request debit, including attempts that failed over |
| A usage lookup by generation id | `GET https://demo-api.routerplus.com/v1/generation?id=REQUEST_ID` returns every attempt with its tokens, outcome and settled cost |
| Out of credits: HTTP 402 | HTTP 429 `insufficient_quota`, the response OpenAI's SDKs already handle |
| A `models` fallback array with `route: "fallback"` | Supported: up to five models, tried in order. See [Model fallback](/docs/model-fallback) |
| `transforms` | Dropped. The `x-tm-dropped-params` header names it. Remove it |
| A `provider` object | Supported keys: `order`, `only`, `ignore`, `allow_fallbacks`, `require_parameters`, `sort`, `max_price`, `quantizations`, `preferred_min_throughput`, `preferred_max_latency`, `zdr`, `data_collection`, `enforce_distillable_text`, and the connection controls on [Routing policies](/docs/routing-policies). `sort` takes `"price"`, `"throughput"`, `"latency"` or `"quality"`. Other keys return a 400. See [Provider routing](/docs/provider-routing) |

> [!WARNING]
> RouterPlus never substitutes a model. An id that is not in the catalog returns a 404
> that names the id you sent. Check every migrated model id against
> `https://demo.routerplus.com/api/models.json` before you ship.

Failover never switches providers in the middle of an answer. If a provider fails after output starts, you get the tokens streamed so far and one terminal error event. You decide whether to retry. See [Wire compatibility](/docs/compat).

## Migrate with one prompt

Paste this into your coding agent:

```text
Migrate this project to RouterPlus.

1. If TM_API_KEY is not set, have me open https://demo.routerplus.com/signup and complete
   Clerk authentication and email verification in the browser. Existing customers
   sign in at https://demo.routerplus.com/login and create a key at https://demo.routerplus.com/console/keys.
   Save the key as TM_API_KEY (env var and .env; never commit it). Continue once
   the key is available; there is no programmatic signup endpoint.

2. Find every LLM client in this codebase and repoint it:
   - OpenAI SDK / any OpenAI-compatible client:
       base_url -> "https://demo-api.routerplus.com/v1", api_key -> TM_API_KEY
   - Anthropic SDK:
       base_url -> "https://demo-api.routerplus.com", api key -> TM_API_KEY
   - Claude Code:
       run with ANTHROPIC_BASE_URL=https://demo-api.routerplus.com ANTHROPIC_AUTH_TOKEN=$TM_API_KEY
   - Codex: add a [model_providers.routerplus] block to ~/.codex/config.toml with
       base_url "https://demo-api.routerplus.com/v1", env_key "TM_API_KEY", wire_api "responses",
       and set model_provider = "routerplus" and model = "gpt-6-luna".
   - Clients of another gateway: also strip "author/" prefixes and ":variant"
     suffixes from model ids, and remove the transforms request field.
     Keep a "models" fallback list (at most five) with route "fallback".
     Inside a "provider" object keep only: order, only, ignore, allow_fallbacks,
     require_parameters, sort ("price", "throughput", "latency" or "quality"),
     max_price (USD per million), quantizations, preferred_min_throughput,
     preferred_max_latency, zdr, data_collection; remove the rest.
   - Raw HTTP: change the host to https://demo-api.routerplus.com and the bearer/x-api-key to TM_API_KEY.

3. Check every model id against https://demo.routerplus.com/api/models.json and substitute the
   closest listed id where needed. Never invent an id: unlisted models return 404.

4. Run one real streamed test call per repointed client and show me each response's
   final usage object (it must contain "cost").

5. Keep the diff minimal. Do not refactor unrelated code.
```

## Verify the cutover

One streamed call tests auth, routing and billing:

```bash
curl -N https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-haiku-4-5","stream":true,"max_tokens":40,"messages":[{"role":"user","content":"Reply with exactly: MIGRATED"}]}'
```

The last usage chunk must have `cost` (USD). Then check that the ledger recorded the call:

```bash
curl -s -H "Authorization: Bearer $TM_API_KEY" https://demo-api.routerplus.com/v1/usage
```

Use the response headers to confirm that traffic moved:

- `x-tm-provider` names the deployment that served the request.
- `x-tm-attempts` counts the dispatches.

Every error body has a stable `error_type`. The fixes are on [Errors](/docs/errors).

---

Source: https://demo.routerplus.com/docs/playground.md

# Playground

The Playground at [`https://demo.routerplus.com/playground`](https://demo.routerplus.com/playground) runs chat, typed
decisions and images on the same catalog, routing, prices and ledger as the API. Each answer
shows the provider that served it, the token counts and the cost. You can look around without
an account. To send, you need one (see [Without an account](#without-an-account)).

![Playground: Cmd+K opens the model picker, a search finds DeepSeek V4.1 Flash, and one message gets a reply with its cost and provider](media/playground.gif)

## What it does

- **Any catalog model.** `⌘K` / `Ctrl+K` opens the model picker.
  - Models of your current mode come first (chat models in Chat, decision models in Decision).
  - Chat starts with Claude Opus 5.5, then Claude Sonnet 5.5, then the rest, newest first.
  - Search matches a model's name, id, lab, providers and kind:
    type `decision` for every decision model, `image` for the image models, `free` for the
    free ones.
  - Each row shows the lab's mark and the price per million tokens, in / cached / out (a dash
    where there is no cached price). An image model shows the most one image can cost. A chat
    model shows its context length.
  - To pin a dated snapshot of a catalog model, type its id. An id that is not listed is used
    as typed.
- **Decision order.** Decision mode lists Jev 1.13, GPT-6 Luna Decisions, Bespoke Nimble v4 Max,
  ScaleDown Classify-1, Bespoke Nimble v4, Clef, Clef-flash, Kev 4B, GLiNER-2.5-Decide,
  Decider 2B and Sage 1.2, in that order.
- **Compare up to three chat models side by side** (Decision mode compares up to four).
  **Compare** adds a column. One prompt goes to every column. Each column streams into its own
  card with its own provider, tokens, latency and cost. Each column keeps its own thread: a
  follow-up sees that model's replies only. Removing a column leaves the rest unchanged.
- **Sample prompts** under the composer send a real prompt in one click, until the
  conversation starts.
  - The first is always the **strawberry test** (how many times "r" appears in "strawberry").
    Each reply says **passed**, **failed** with the count the model gave, or that it gave no
    clear count.
  - The others are random: a reasoning trap, a code task, structured JSON, a table, a six-word
    story and more.
- **Streaming replies** with stop, regenerate, a collapsible "thinking" section for reasoning
  models, and safe rendering of code, lists, links and tables. A table is drawn only for
  a real markdown table: a header row, then a divider row such as `|---|---|` with the same number of
  cells. Other text with a pipe shows as written. A wide table scrolls sideways.
- **Code** and **Parameters** sit to the left of Send.
  - **Code** shows the next request as curl, Python or Node.
  - **Parameters** holds a system prompt, temperature, top_p and max output tokens, per
    conversation.
- **Cost on every reply:** the same `usage.cost` the API returns, the time to first byte, and
  an `audit` link to the request's ledger rows.
- **Organizations:** if you belong to several, the sidebar selector sets which one the
  playground bills.

## Tetris Royale

[Tetris Royale](https://demo.routerplus.com/playground/tetris-royale) compares nine System One decision
models on the same block sequence. GPT-6 Luna Decisions plays next to Jev, Bespoke Nimble v4
Max is Bespoke's player, and ScaleDown Classify-1 plays last. Open it from the Playground mode
bar. The arrow button at the top left goes back.

- It bills the selected organization through its own managed key, **Tetris Royale**, held on
  the server. Ordinary Playground keys keep their own limit. No API key reaches the browser.
- The server validates the board and match counters, then builds the decision prompt and the
  legal choices. The game relay accepts no custom prompts or questions.
- Every request follows the organization's normal credits, limits, billing and routing.

Match rules:

- A match is 200 pieces by default. Settings offers 20 up to a thousand.
- Each model plays on its own, so a slow or rate-limited model does not hold up the others.
- A transient 429 cools down only that model. It retries up to three times and respects
  Retry-After. Other errors offer a retry button.
- Matches in one organization share per-model pacing on the server. A busy queue for one model
  does not pause the others. Gateway quotas still apply.
- Pause finishes current moves and stops new calls. Switching tabs pauses the match. Reset
  starts over. Match state lives only in the open page.

Settings offers five **Block sequences**, each a repeatable order of pieces that all models
share. The boards keep a fixed order: Jev, GPT-6 Luna Decisions, Perplexity, Bespoke, Clef,
Clef-flash, Kev, Decider.

**Standings** rank the models by score at each model's current progress. A row moves when its
model passes another. Equal scores share a place. A line clear or a change of place shows for a moment. Final standings appear when every
board finishes or tops out.

Both move modes use a 10 × 20 board, seven-bag pieces, line-clear and level points, hard-drop
points, combos, back-to-back clears and all-clear bonuses. Settings → **Moves** picks how a
piece reaches its place:

- **Drop and slide** (the default): the piece turns at the top, then falls. While it falls it
  can move left or right, so it can tuck into a gap under an overhang. It never turns while it
  falls. The model picks any position the piece can reach this way. Each choice says whether it
  is a straight drop or a tuck, and the points it scores now. The board shows the piece turn,
  drift toward its column and tuck.
- **Drop only**: the model picks a rotation and a column, and the piece drops straight down.

Neither mode has hold, wall kicks, T-spins or a gravity deadline. **Rules & scoring** gives the
exact rules.

## Decision

A *decision model* returns typed answers in place of text. You give it a **state** (a message,
a ticket, a list of items) and typed **questions**. It returns a typed answer for each
question, with the probability or confidence behind it. Use it for routing, classification and
other decision points in an application. The catalog has twelve:

| | Jev 1.13 | Bespoke Nimble v4 and v4 Max | Clef and Clef-flash | Decider 2B and Kev 4B | Perplexity Decider v1 27B | GPT-6 Luna Decisions | Sage 1.2 | ScaleDown Classify-1 | GLiNER-2.5-Decide |
|---|---|---|---|---|---|---|---|---|---|
| Catalog id | `typesafe/jev-1.13` | `bespokelabs/nimble-v4`, `bespokelabs/nimble-v4-max` | `cloudflare/clef`, `cloudflare/clef-flash` | `routerplus/decider-2b`, `routerplus/kev-4b` | `perplexity/pplx-decider-v1-27b` | `openai/gpt-6-luna-decisions` | `levanto/sage-1.2` | `scaledown/classify-1` | `fastino/gliner-2.5-decide` |
| Made by | TypeSafe | Bespoke Labs | Cloudflare | RouterPlus, our own models | Perplexity | OpenAI | Levanto | ScaleDown | Fastino |
| Numbers it returns | probabilities | probabilities, at full precision | probabilities, to 4 decimals | probabilities | probabilities, at full precision | probabilities, to 2 decimals | probabilities | probabilities, at full precision | confidences, not calibrated probabilities |
| Price | $0.042 per million input tokens | $0.04 (v4) or $0.09 (v4 Max) per million input tokens | $0.24 (Clef) or $0.09 (Clef-flash) per million input tokens | free: our own models, at $0 | $0.04 per million input tokens, the state billed once per question | $0.10 per million input tokens, about 150 tokens for each question | $0.05 per million input tokens, the state billed once per question | $0.05 per million input tokens, each question its own call | $0.03 per million input tokens |

Jev, the two Nimble models, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider, GPT-6
Luna Decisions, Sage and ScaleDown Classify-1 are the eleven **System One** models. On this
page, "System One" means all eleven.

- **System One models** use Jev's question format (System One). They take choice, rating (2 to
  10 levels) and yes/no questions, and tags as one yes/no per tag.
  When unsure, they give their best answer, with probabilities. GPT-6 Luna Decisions can also
  decline a question (**Refused**).
- **GLiNER-2.5-Decide** uses GLiNER's schema. It takes yes/no, choice, rating (exactly 5
  levels) and tags, all as labels. When unsure, it gives its best label, with a confidence.

### Per-model limits

Decision mode checks these limits before a run. It counts the questions in order, and only the ones the column will send. On Nimble, Clef and Clef-flash, a question that would cross 64 shows **Not askable on Bespoke Nimble v4** (or on the other Nimble models, or on Clef, or on Clef-flash) and stays out of that column's request. It takes none of the 64, and a later question that still fits is sent. So the provider never refuses the whole column. Perplexity Decider and GPT-6 Luna Decisions are counted the same way, against their own limits.

| Model | Own limits |
|---|---|
| Nimble, Clef and Clef-flash | A choice needs at least 2 options. One request holds at most 64 questions, each tag counted as one. |
| Clef and Clef-flash | Also: every answer key is 1 to 100 letters, digits, `_`, `.` or `-`. A question id in Decision mode always fits. A tag can break the rule: Decision mode asks each tag under the key `<question id>__<tag id>`, and a long question id with a long tag id can pass 100 characters. |
| Perplexity Decider v1 27B | One request holds at most 128 questions, each tag counted as one. It takes a one-option choice, as Jev does. A question that would cross 128 shows **Not askable on Perplexity Decider v1 27B**. Perplexity bills the state once for each question and each tag. |
| GPT-6 Luna Decisions | A choice needs at least 2 options. One request holds at most 200 questions, each tag counted as one. A question that would cross 200 shows **Not askable on GPT-6 Luna Decisions**. Each question and each tag adds about 150 input tokens. OpenAI can decline one question and answer the rest. The column shows that answer as **Refused**, and the call is still billed. |
| Decider 2B and Kev 4B | None. For Decider 2B and Kev 4B, Decision mode checks no limit of their own: probed on 2026-10-02, each answers a one-option choice and a request of 200 questions. Their columns follow the family's rules, as a Jev column does. |
| ScaleDown Classify-1 | None in Decision mode. Probed on 2026-10-12, it answers a one-option choice and a request of 65 questions. It reads System One as text, so the gateway sends an option without a description as `""` and a JSON state as its JSON text. The answers mean the same. |

### The decision console

In **Decision** mode, the composer becomes a decision console: a box for what the models
judge, and up to 20 questions for every column.

- Each question is a line of text and an answer type: **Yes / no**, **Pick one** (one of your
  options), **Rating** (a scale, lowest level first; a new rating starts with five levels) or
  **Tags** (which of your tags apply).
- Options, tags and levels are rows: a short name, which the model answers with, and what it
  means.
- You never write an answer key: Decision mode names each question from its words, and
  **Code** shows the names it sends.
- A yes / no saved by the earlier editor with descriptions of yes and no keeps them, under the
  question. Only the System One models take them. **Remove descriptions** makes the question
  plain.

An example loads by itself. The **Example** menu picks another, or **Start blank**. The
examples cover checking an answer against its sources, reviewing an agent's run, catching a
prompt injection, triage, bug severity, lead qualification, code review, moderation, feedback
tags and a JSON state. **Run** runs it once. **Run 10 times** runs every column ten times (see
[Run 10 times](#run-10-times)).

### One form for every model

You write each question once. Each decision model has its own question format, so Decision
mode translates the form for each column:

| Question | System One (Jev, Nimble, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider, GPT-6 Luna Decisions, Sage, ScaleDown Classify-1) | GLiNER-2.5-Decide |
|---|---|---|
| Yes / no | a `noul` question | a classification: the question text is the task, the labels are `yes` and `no` |
| Choice, 2 to 20 options, descriptions optional | a `choice` question: each option, with its description or none | a classification: one label per option, `"name: description"` (or `"name"`), read back as the option's name |
| Rating, exactly 5 levels, low to high | a `score` question with 5 levels | a classification: the labels `"0: <level 0>"` to `"4: <level 4>"`, read back as the level |
| Tags, 1 to 20, descriptions optional | one `noul` question per tag, asked as `<question> Does this apply: <tag>?` (with the tag's description after the tag, when it has one) | a multi-label classification that returns every tag with its confidence; a tag applies at 0.5 or more |

- GLiNER reads each label as text, so Decision mode puts an option's or level's description
  into its label. The description changes GLiNER's answer.
- Decision mode asks GLiNER at threshold 0 (`cls_threshold: 0`), so GLiNER always names its top
  label. At GLiNER's default of 0.5, a task answers `null` when no label reaches it. Decision
  mode shows a `null` as Not sure.
- Each tag is one question in a System One column's request. The column shows the tags
  together.
- Decision mode builds a System One body from the catalog's record of the model's format,
  never from its name.

### Compare

Put up to four models side by side with **Compare**: up to four of the twelve decision
models, or three decision models and a chat model. A chat or image conversation
stops at three. One turn cannot hold all twelve decision models. A column's chip swaps one
model for another.

With two or more decision models, the question builder follows the tables above:

- Before the run, it marks a question that one of the chosen models cannot take, and names the
  models that can. Examples: a rating that does not have 5 levels (System One models only), a
  choice with one option
  (Jev, Decider 2B, Kev 4B and Perplexity Decider only: Nimble, Clef, Clef-flash and GPT-6
  Luna Decisions refuse a one-option choice), more than 20 options, a question past the 64
  keys of one Nimble, Clef or Clef-flash request, the 128 keys of one Perplexity Decider
  request or the 200 keys of one GPT-6 Luna Decisions request, and
  a tag whose key is longer than Clef's 100 characters.
- Models refused for the same reason share one line. A Nimble, Clef, Decider 2B, Perplexity
  Decider or GPT-6 Luna Decisions column is named as itself, never Jev.
- Each column still gets the questions its model can take. A question it cannot take shows
  **Not askable on** that model and stays out of that column's request.
- GLiNER uses the question text as its task name, at most 256 characters. Two questions with
  the same text, or a question longer than 256 characters, are not askable on GLiNER.

Every example uses forms that every decision model takes, the tags example included.

### Run 10 times

**Run 10 times** runs the whole comparison ten times with the same state and questions. It
shows one ledger in place of a card per column. Each column sends up to two requests at a
time, and the ledger fills in as runs come back. The ledger shows:

- For each model and question, the most common answer and in how many runs (**10/10** when it
  never changed). A pick-one question lists every option the runs chose, with its count.
- The averages: the mean probability of yes, the mean confidence of a choice, the mean level of
  a rating (a marker on the track, with a faint dot for each run), and for tags how many runs
  each tag applied in. Bars and markers move as each run lands.
- At the head of each column, the mean time per run and the cost of all runs. Each run's time
  is a dot on a strip, and the mean is a line.
- For each row, whether the decision models agree on their most common answers (for a rating,
  mean levels within 0.5 of each other).

Each column is billed as one request per run: ten runs of four columns are 40 requests. A
column that hits a rate limit stops and says so, and the other columns go on. **Stop** stops them all.

On Decider 2B and Kev 4B, the ten runs send the same request, so RouterPlus answers the
repeats from its cache, which keeps an answer for 600 s. Those runs come back
sooner than the first one, cost $0, and repeat the first run's answer exactly. On these
models, the ten runs do not show how much the model's answer varies. The first request after
a quiet period can take 15 to 20 s while RouterPlus starts the model.

### Compare view

Below the columns, the compare view draws each question as one row, with one cell per column:

| Question | Each cell shows |
|---|---|
| Yes / no | The verdict (Yes, No or Not sure) and a bar from 0 to 1: the probability of yes for System One, the confidence in the chosen label for GLiNER (labelled **confidence**), the stated answer for a chat model. |
| Choice | The chosen option (or Not sure) and its probability or confidence. Per-option bars where the model returns them: a System One model's add up to 1, and GLiNER returns the chosen label only. |
| Rating | One 0 to 4 track, with one marker per model (a System One score or GLiNER's chosen level), and the confidence beside it. |
| Tags | A grid of tag by model: applies ✓, does not apply ✗ or not sure ?, with the score. |

- Each row says whether the decision models agree: the same verdict, the same option, a level
  within 0.5, or the same set of tags with every tag answered. A tag that one model is not sure
  about cannot be compared, and the row says so.
- The view shows only numbers a model returned, each labelled with what it means.
- **GLiNER's numbers are confidences, not calibrated probabilities**: a GLiNER 0.9 and a Jev
  0.9 mean different things.
- A Clef confidence and a Jev confidence also mean different things, because Cloudflare
  computes Clef's in its own way. So compare their probabilities, not their confidences.
- GPT-6 Luna Decisions' confidence is OpenAI's own.
- A Decider 2B or Kev 4B column costs $0.00, with no list price or discount beside it: the
  model is free, so there is nothing to take off.

### Answer cards

- A System One model's cards (Jev, Nimble, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity
  Decider, GPT-6 Luna Decisions and Sage) show bars: the chosen option, the probability of each
  alternative, and the model's confidence.
- A question that OpenAI declined on a GPT-6 Luna Decisions column shows **Refused**, with no
  bars.
- GLiNER's cards show the chosen label and its confidence, and for tags each tag's confidence.

A chat model in a column gets the same questions in words, tags included where the prompt can
say them. When it answers in the JSON it was
asked for, its answers render as the same cards and in the compare view.

### Billing for decisions

A Decision run is a real API call to [`POST /v1/decisions`](/docs/api-decisions) under the
playground key.

- It is billed to your organization at the model's price.
- It counts against the playground key's limits and the organization's.
- It appears in `GET /v1/generation`. Each answer card has an `audit` link to its ledger rows.
- A discount can make a run free. The cost line shows what you paid and, with a discount, the
  list price and the percent off.

From your code, call [`POST /v1/decisions`](/docs/api-decisions) in the model's own question
format. Only Decision mode translates the form. The API does not.

## Images

Pick an image model and the composer describes an image. The picker marks image models
**image**, with the most one image can cost.

### Size and controls

- The toolbar keeps **Aspect ratio**, **Resolution / size** and **Add reference images** in
  the same place for every model. A control the model does not support stays visible and
  disabled, with an explanation. Local reference images show a preview and a remove button.
- A model with a native `size` menu, such as GPT Image, shows those choices in
  **Resolution / size**. The aspect ratio follows the size. The request keeps the model's
  `size` field, with no added resolution parameter.
- A model with width and height inputs offers eight canvases within its limits: 1024×1024 and
  2048×2048 squares, 1536×1024 landscape, 1024×1536 portrait, 1280×720 HD, 1920×1080 Full HD,
  and portrait HD and Full HD.
- On Krea 3, a fixed size sets the aspect ratio, and a native resolution clears custom width
  and height. Krea 3 rounds a requested size to a supported 1K or 2K canvas, so the output
  size can differ. Krea 3 Ultra uses native 2K and takes no size preset.
- Other models keep resolution and aspect ratio separate from pixel size. For example, Nano
  Banana can keep native 4K while a preset sets width and height.
- **Model default size** clears explicit dimensions and keeps the native choices.
- **Advanced settings** holds the rest: image count, custom dimensions, quality, format,
  background, seed, thinking mode, and reference URLs or weights. Closing it keeps your values.
  A hidden field that needs attention opens by itself.
- The **Code** dialog waits for reference files to load and asks you to fix invalid settings
  before it shows a request. It cannot copy an older valid value by mistake.

### Cost

The estimate beside the send button is the most the render can cost: the per-image ceiling
the gateway reserves. The bill is the actual cost, which is usually less. Receipts show token
counts only when the provider reports nonzero counts. Models billed per image, such as Krea,
show cost and render time in place of `0 in / 0 out`.

### Compare and render

**Compare** puts up to three image models side by side, each with the same description and
the options it accepts. A comparison holds image models only. A chat model as the lead starts
a new conversation, so text and pictures never share a history.

- A render usually takes 10 to 60 seconds. Some models take several minutes. Each tile shows
  the time elapsed.
- You cannot stop a render, and the provider bills it either way.
- A render continues while you use another chat. Its chat shows a dot in the list until the
  picture arrives.
- **A render survives a reload** or a sleeping background tab. It runs on the server, and the
  page picks it up again.
- A finished render waits 15 minutes, then is gone. It is still billed.
- Each image has a **Download** button.
- Saved conversations and exports leave out local reference uploads. After a reload or import,
  **make again** asks you to reattach them and sends a new request. It keeps the previous
  result and never turns an edit into a text-only generation. Provider-hosted reference URLs
  still work.
- The sample descriptions include hard cases: exact spelling, counting, hands and a
  transparent background. A sample fills the description and does not render, because a
  render costs more than a chat reply.

**Images stay in your browser**, in its IndexedDB storage, filed by your sign-in, conversation
and reply, so a reload or a visit tomorrow shows them again. They are never uploaded. A render is metered the same as
[`POST /v1/images/generations`](/docs/api-images), under the playground key.

## Without an account

Without signing in, you can pick and compare models, switch modes, set parameters and read a
request as code.

- Nothing runs. Send, Generate, Run, Run 10 times and a sample prompt open the sign-in form:
  email and password, an email link, or Google, and **Create an account**. After it, you are
  back in the playground.
- Your unsent prompt, models, parameters and Decision form come back in the same browser tab,
  for up to 30 minutes. Nothing is sent until you send it.
- A signed-out playground saves no conversations, and does not show those a signed-in person
  saved in that browser.

## Keyboard shortcuts

| Keys | What it does |
|:---|:---|
| `⌘⇧K` / `Ctrl+Shift+K` | Start a new chat |
| `/` | Put the cursor in the message box (in Decision, the box for what the models judge) |
| `⌘Enter` / `Ctrl+Enter` | Send the message, or run the Decision form |
| `Enter` | Send, from the message box. `Shift+Enter` adds a new line |
| `⌘K` / `Ctrl+K` | Open the model picker |
| `Esc` | Close the picker or a dialog |

`/` does nothing while you type in a field.

## Install as a desktop app

The app opens the playground in its own window, with its own dock or taskbar icon. The console
is in the same app.

- **Chrome and Edge.** Open the playground. Click **Install app** in the top bar, or the
  install icon at the right of the address bar, and confirm. To remove it, open the app menu
  and choose **Uninstall**.
- **Safari on macOS.** Open the playground. Choose **File** > **Add to Dock**, name it and
  click **Add**.

The app is the same site. It shares the sign-in of the browser profile you installed it from,
and updates by itself.

**Offline**, the app shows an offline page. It keeps only its static files: icons, styles,
scripts and that page. Saved conversations stay in the browser. Sending, the console, sign-in
and every API call need the network. The app never stores API answers, console data or
sign-in pages.

## How it is billed

The playground meters usage the same as an API call.

- The console creates one key named **Playground** per organization, tagged **managed** on
  [`/console/keys`](https://demo.routerplus.com/console/keys). That page hides managed keys until you turn
  on **Show playground keys**.
- Every reply is an attempt under that key. It is charged to the organization's credits,
  subject to its spend caps, and shows in the console and `GET /v1/generation`.
- BYOK connections bound to that key apply.
- Nobody sees the key's value. The console holds it in memory and calls the gateway for you.
- RouterPlus replaces the key every 24 hours and after a console restart, and retires the old
  one. If you disable it, the playground creates a new one on the next message.

## What is stored

- Your browser's local storage holds up to 100 conversations. **Export** and **Import** move
  them. **Clear all** deletes them.
- Your browser's IndexedDB holds images, up to 1 GB, oldest removed first. Deleting a chat, or
  **Clear all**, deletes its files.
- Export carries conversations, not files. Download the images you want to keep.
- RouterPlus handles requests per the [Data policy](/docs/data-policy). Clearing browser
  history does not delete server-side request logs.

## Limits

- No chat attachments. Image models that accept references or editing controls show them in
  **Advanced settings**.
- Up to three image renders in flight per organization (one per comparison column), and up to
  four images per render.
- A request holds at most 400 000 characters, and a conversation 200 messages. A system prompt
  holds at most 32 000 characters. Start a new chat when a model reports a context overflow.
- The managed key has its own rate limit (60 requests per minute) on top of the organization's
  limits. Each comparison column is one request, with its own credit hold, so on a low balance
  one column can be refused while another answers. Decision runs count the same way. **Run 10
  times** sends ten requests per column, two at a time.
- Each model shows its lab's mark in the lab's colours, or the model's own mark. An id the
  catalog does not list gets a neutral chip.

### Provider pools

Columns served by one provider share that provider's pool. A pool has room for a full
comparison (three chat models, or four Decision columns) and for other users at the same time.
A comparison that exhausts it gets `rate_limit` on the later columns, with **Try again**.

| Models | Pool |
|---|---|
| Nimble | The Nimble models share one pool, which takes 7 requests at once in all, 5 per organization: Bespoke allows our whole account 8, and one is kept for our test environment. |
| Clef and Clef-flash | Clef and Clef-flash share one pool: 200 requests a minute and at most 8 at a time in all, 150 and 6 per organization. |
| Decider 2B and Kev 4B | Decider 2B and Kev 4B share one RouterPlus pool, which is large (11,400 requests a minute and 59 in flight at once in all, 44 per organization), so your organization's own limits bind first. |
| Perplexity Decider | 540 requests a minute and 8 in flight at once in all, 405 and 6 per organization. Perplexity allows our organization 10 requests a second. |
| GPT-6 Luna Decisions | A large pool (24,000 requests a minute and 64 in flight at once in all, 18,000 and 48 per organization), so your organization's own limits bind first. |

## Errors

Errors show the class and remediation from the API's [error table](/docs/errors). A
`rate_limit` asks you to wait, `insufficient_quota` points you at credits, and
`content_policy` is never rerouted. Every failed reply carries its request id.

---

Source: https://demo.routerplus.com/docs/provider-routing.md

# Provider routing

Many models have more than one provider. RouterPlus picks the provider for each request. When a
provider cannot serve, RouterPlus fails over to the next one before the first byte reaches you.
Use the `provider` object to change the order or filter the providers.

## Request

```bash
curl https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{
    "model": "moonshotai/kimi-k3",
    "messages": [{ "role": "user", "content": "Hello" }],
    "provider": { "sort": "price", "max_price": { "prompt": 1, "completion": 4 } }
  }'
```

## Default order: lowest cost per good answer

For each provider of the model, RouterPlus estimates the cost of one good answer:

- the price of the model's typical request: uncached input, cached input at that provider's
  cache hit rate, and output;
- plus the cost of waiting: time to first token and output time at the provider's measured speed;
- plus the time lost to rate limits;
- divided by the share of requests that succeed, with a heavy penalty for provider errors and a
  penalty for malformed tool calls.

RouterPlus splits traffic as follows:

- Providers whose cost is close to the best share the traffic.
- Every provider keeps a small share, so RouterPlus keeps measuring new and recovering providers.
- A provider that fails or rate-limits gets less traffic until it recovers.
- Traffic above a provider's learned capacity goes to the next provider.
- Until providers have measurements, the cheapest goes first.

A conversation stays on the provider that served its first turn while that provider can serve.
This keeps the provider's prompt cache warm. Requests that share a long prompt prefix (system
prompt and tools) go to the same provider across sessions.

The response header `x-tm-routing` names the strategy: `routing_cost` (the default), `price`,
`throughput`, `latency` or `quality`. It adds `;pinned` or `;prefix` when a conversation pin or a
shared prefix chose the provider.

## The `provider` object

| Field | Effect |
|---|---|
| `order` | Try these providers first, in this order. Turns off the default order. |
| `only` | Only these providers. |
| `ignore` | Never these providers. |
| `allow_fallbacks` | `false`: only the first provider, with no failover. |
| `require_parameters` | Refuse the request instead of dropping a parameter a provider does not take. |
| `sort` | `"price"`, `"throughput"`, `"latency"` or `"quality"`, or `{ "by": ..., "partition": "model" \| "none" }`. Turns off the default order. See [Sort](#sort). |
| `preferred_min_throughput` | Tokens per second (a number means p50, or `{ "p50", "p75", "p90", "p99" }`). Slower providers go last. None is excluded. |
| `preferred_max_latency` | Seconds to first token, same forms. Slower providers go last. |
| `max_price` | `{ "prompt", "completion" }` in USD per million tokens, plus `request` and `image`. A provider above a cap is left out. |
| `quantizations` | Only providers that serve the model at these quantizations (`int4`, `int8`, `fp4`, `mxfp4`, `nvfp4`, `fp6`, `fp8`, `mxfp8`, `fp16`, `bf16`, `fp32`, `unknown`). |
| `zdr` | Only providers that keep no prompts or completions. |
| `data_collection` | `"deny"`: only providers that do not train on or keep request data. |
| `enforce_distillable_text` | Only `false` is accepted for now. |

[Guardrails](guardrails.md) on your key apply first. A request can narrow them. It cannot widen
them.

### Sort

| `sort` | Order |
|---|---|
| `price` | Cheapest first. |
| `throughput` | Highest measured tokens per second first. |
| `latency` | Lowest measured time to first token first. |
| `quality` | Highest share of well-formed tool calls first. Ties go by the default cost estimate. |

In the object form, `partition` is `"model"` (the default) or `"none"`. Both give the same
order, because RouterPlus routes one model per request.

After the sort, a conversation pin or a shared prompt prefix can still choose the provider.
`preferred_min_throughput` and `preferred_max_latency` then move providers that miss them to the
end.

### Errors

An unknown field in `provider`, or a `sort` or `partition` value outside the lists above, returns
`400`.

## Your default

An organization or workspace can make price its default order with
`PATCH /v1/org {"default_sort": "price"}` or `PATCH /v1/workspaces/<id>` (see the
[management API](api-management.md)). A `sort` in the request wins over this default.

## Rankings and benchmarks

| Endpoint | Returns |
|---|---|
| `GET /v1/rankings?period=day\|week\|month` | Models ranked by tokens served on RouterPlus. A model used by fewer than 3 organizations is not listed. |
| `GET /v1/rankings?task=code\|reasoning\|extraction\|chat\|creative\|vision\|long_context` | Models ranked for one kind of request. RouterPlus labels each request with its kind (the `task` label). |
| `GET /v1/rankings?view=sessions` | Models ranked by sessions, with the median cost and the median number of requests per session. |
| `GET /v1/benchmarks/<model>` | What the model takes (context, output limit, inputs, reasoning efforts, response formats) and how it serves now across its providers: the best latency, throughput, error rate and tool-call quality measured. |
| `GET /v1/benchmarks?task=coding\|math\|reasoning&limit=20` | Catalog models ranked by a quality index. See below. |
| `GET /v1/models/<model>/endpoints` | Each provider with its price, data policy, status and measurements. Its `quality` field holds the model's quality scores. |

`GET /v1/benchmarks?task=...` ranks `coding` by the coding index, `math` by the math index and
`reasoning` by the intelligence index. `limit` is 1 to 100 (default 20). When a model has no
coding or math index, the mean of its published coding or math evaluations (out of 100) ranks
it. A model with neither is not listed.

### Quality scores

`GET /v1/benchmarks/<model>` and `GET /v1/models/<model>/endpoints` carry a `quality` object:

```json
{ "quality": {
    "intelligence_index": 58.4, "coding_index": 55.0, "math_index": 90.1,
    "evaluations": { "gpqa": 0.9, "hle": 0.4, "livecodebench": 0.8 },
    "aa_model": { "slug": "claude-opus-5-5", "name": "Claude Opus 5.5 (Max)" },
    "source": "Artificial Analysis",
    "source_url": "https://artificialanalysis.ai/",
    "model_url": "https://artificialanalysis.ai/models/claude-opus-5-5",
    "fetched_at": "2026-10-31T09:00:00.000Z" },
  "scores_status": "ready" }
```

- The scores come from [Artificial Analysis](https://artificialanalysis.ai/). RouterPlus does not
  run these evaluations.
- `evaluations` holds every other evaluation Artificial Analysis publishes for the model, by its
  name.
- `quality` is `null` when Artificial Analysis has no entry for the model.
- `scores_status` is `ready`, `loading` or `unavailable`. `loading` means RouterPlus is fetching
  the scores. Ask again shortly.
- Scores refresh every 6 hours. `fetched_at` gives the time of the copy in use.
- When you show these scores, credit Artificial Analysis and link to
  https://artificialanalysis.ai/. Artificial Analysis asks for this attribution.

---

Source: https://demo.routerplus.com/docs/routing.md

# Routing & failover

You name a model. The gateway picks a deployment that serves it, and fails over to another
deployment when a provider fails. Three rules apply to every request:

1. **You get the model you name.** An unknown model is a 404 that names the id. The gateway
   never substitutes another model.
2. **Failover happens only before your answer starts.** Once the first token of real output
   reaches you, the request stays with that provider.
3. **The gateway never reroutes a content-policy refusal** to another provider.

To change the provider order or filter providers, see [Provider routing](/docs/provider-routing).

## How a request picks a provider

A **deployment** is a provider endpoint with a wire dialect, the model ids it serves and a
routing `priority`. Today each Claude and GPT model has one deployment: the lab's own API
(`anthropic` for Claude, `openai` for GPT). Routing runs in memory and makes no remote calls.
Billed requests still check the ledger and the shared admission store for authentication,
limits and the reserve check.

1. **Model filter.** Keep deployments that list the requested id. The match is exact, with two
   exceptions:
   - A dated snapshot of a listed id (`claude-haiku-4-5-20251001`) routes and bills as its
     family. The provider still gets your original id.
   - The `rp-` name of a Claude model (`rp-sonnet-5-5`, for Cursor) routes, bills and goes to
     the provider as that model. See [Models & catalog](/docs/models#claude-names-for-cursor).
2. **Health filter.** Drop deployments whose circuit is open.
3. **Order.** The base order is cheapest first: input plus output price per million tokens, at
   each provider's live price.
   - Equal prices keep ascending `priority`, then catalog order.
   - A route billed at the cost its provider reports (some image and video models) has no price
     to compare. It comes after the priced routes.
   - The providers that `provider.order` names go first, in its order.
   - The [routing strategy](/docs/provider-routing) then sets the final order.
   - A key under a [routing policy](/docs/routing-policies) keeps the policy's saved order.

The gateway tries the list in order. The first deployment to produce real output wins. A
failure that can fail over moves to the next deployment with no backoff. The gateway translates
requests, streams and errors between dialects, so you can call any listed model from either
surface.

A key with a BYOK connection or a routing policy runs the same steps over its own routes, with
the policy's order, funding rule and attempt limit on top. `POST /v1/route` shows the plan
without sending the request.

A [dedicated endpoint](/docs/dedicated-endpoints) (`<your-namespace>/<name>`) routes first to
capacity reserved for your organization, then to public deployments of the same model that meet
its constraints. It never routes to a different model. A
[closed](/docs/dedicated-endpoints#closed-endpoints) endpoint names RouterPlus as the provider,
whichever route served.

## The commit boundary

**A request commits to a deployment at its first semantic output:** the first text delta,
reasoning delta, tool call, refusal or usage report that reaches you.

**Before commit**, failover is possible only while the gateway has written nothing to you.

- A provider that fails before its first frame is skipped, and the next one is tried at once.
  This covers a refused connection, a 5xx, a failure before any bytes, and a miss of the
  20-second headers deadline.
- The gateway sends your response headers only when a live attempt starts writing.
- After the headers and (on the OpenAI surface) the role-priming delta go out, the gateway does
  not switch providers. A later failure becomes the terminal in-stream error in
  [Streaming](/docs/streaming), even with no semantic output.

**After commit**, the gateway never switches providers. If the provider fails mid-answer, you
get the tokens it produced plus one terminal error event inside the 200 stream. You decide
whether to retry. A retry starts fresh.

> [!NOTE]
> You pay for every settled attempt, including attempts that failover discarded before commit.
> A provider that fails after it reports prompt usage still bills RouterPlus for that prompt,
> and pass-through pricing passes the charge on. The inline `usage.cost` is the **full request
> debit** across all attempts, so it matches the ledger. `x-tm-attempts` gives the number of
> physical dispatches.

## No model substitution

A model that is not in the catalog returns a 404 that names it:

```json
{
  "error": {
    "code": "model_unavailable",
    "message": "model \"gpt-5-ultra\" is not in the catalog; GET /v1/models lists what this key can serve",
    "type": "invalid_request_error",
    "metadata": { "error_type": "model_unavailable" }
  },
  "request_id": "…"
}
```

There is no closest-match fallback and no default model. `n>1` returns a typed 400, because the
gateway sends every stream as a single choice.

## What never fails over

The gateway decides whether an error can fail over once, when it normalizes the error. Routing
never reads raw provider errors.

- **`content_policy`** never retries and **never** goes to another provider. This check runs
  before every status-code rule. Error metadata on this path never echoes your flagged input.
  To try a different provider, send a new request.
- **`context_overflow`** never fails over, because every deployment of a model has the same
  context window. Shorten the request or choose a model with a larger context.
- Other buyer-fault 4xx errors, such as a malformed request, return directly.

## Failover eligibility by error class

| Upstream signal | `error_type` | Retryable | Fails over |
|---|---|---|---|
| 429 (Retry-After honored, capped 60s) | `rate_limit` | yes | yes |
| 401 / 402 / 403 from the provider | `auth` | no | **yes** |
| Context / length errors | `context_overflow` | no | no |
| Content policy / refusal shapes | `content_policy` | no | **never** |
| 404 / model_not_found upstream | `model_unavailable` | no | yes |
| 5xx / 529 / overloaded | `upstream_error` | yes | yes |
| Other 4xx | `upstream_error` | no | no |

> [!NOTE]
> A 401, 402 or 403 **from a provider** is a problem with RouterPlus's account at that
> provider. The gateway routes around the deployment and counts the error against that
> provider's health. You see an `auth` error only if every candidate failed.

## Health circuits and cooldowns

Each deployment has its own circuit breaker:

- **Two failures in a row open the circuit** for a 30-second cooldown. Routing skips the
  deployment while the circuit is open.
- **After the cooldown the circuit goes half-open.** One live request goes through as a probe.
  Success closes the circuit. Failure opens it again with a new cooldown.
- **Only failures that can fail over count.** Buyer-fault errors (400/413-class, content
  policy, context overflow) never count.
- An upstream 429 counts against the circuit. Uptime scoring excludes it as buyer-caused. It
  also pauses that provider's admission pool for the provider's `retry-after`, 1 to 60 seconds.
- A client cancellation does not count.
- **A hanging provider counts as a failure.** This applies when the provider sends no response
  headers for 60 seconds (20 seconds on a stream) and the request then ends, by the gateway's
  deadline or your cancellation.
  - The attempt counts on the deployment's circuit and on a second circuit for that model at
    that deployment.
  - Two such attempts in a row pause the provider for that model for 30 seconds. Its other
    models keep answering. Then one request probes it.

Public uptime numbers use the same rules. A model page shows no uptime percentage for a
provider RouterPlus calls directly until it has 100 counted requests in the window.

## When everything is cooling down

If every deployment of your model has an open circuit, the gateway refuses the request at once,
with no queue. The response is a `503` with:

- `retry-after: 5`
- `x-tm-error-code: gateway_error`
- `x-tm-limit-kind: health`
- the message `all deployments cooling down`

A cooldown lasts 30 seconds, then one request goes through as the probe. Retry after a few
seconds.

## Reading what happened

Every response says how it was served:

| Header | Meaning |
|---|---|
| `x-request-id` | your handle for audit and support (always present) |
| `x-tm-provider` | the deployment that served the request; `routerplus` on a closed dedicated endpoint |
| `x-tm-served-by` | the provider that ran the request, when a route reports one; absent on a closed dedicated endpoint |
| `x-tm-upstream-model` | the id sent to the provider, when it differs from the one you sent; absent on a closed dedicated endpoint |
| `x-tm-attempts` | physical dispatches, failovers included |
| `x-tm-upstream-status` | the HTTP status the provider returned |
| `x-tm-dropped-params` | request fields the gateway removed before dispatch, by name |
| `x-tm-route-plan-id` | the plan id, for correlation with `/v1/route` |
| `x-tm-dedicated-endpoint`, `x-tm-route-role` | on a dedicated endpoint: its id, and whether your dedicated capacity (`primary`) or the shared pool (`fallback`) served |
| `x-tm-queue-ms` | on a dedicated endpoint with a [paced limit](/docs/dedicated-endpoints#the-paced-limit): how long the request waited for its turn, in milliseconds |
| `x-tm-remaining-rpm`, `x-tm-remaining-tpm` | what your rate limits had left at admission |
| `x-tm-error-code` | canonical error class, on failures |
| `x-tm-error-origin` | on failures: `upstream`, `gateway_admission`, `upstream_quota`, `gateway_infrastructure` or `authorization` |
| `retry-after` | on rate limits and cooldowns; whole seconds, at most 59 (`usage_limited` sends 60 or 3,600) |

On failures the body carries the canonical class: `error.metadata.error_type` on the OpenAI
surface, `error.error_type` on the Anthropic surface. Each dialect's native type string is there
too, so your SDK's retry rules keep working. The gateway never relays a provider's raw error
body. See [Errors & remediation](/docs/errors) for the full table.

### Per-attempt audit

The generation audit endpoint returns every physical attempt for a request id: which
deployments the gateway tried, in what order, and what each one cost.

```bash
curl -s -H "Authorization: Bearer $TM_API_KEY" \
  "https://demo-api.routerplus.com/v1/generation?id=<x-request-id>"
```

In this trimmed example, a Claude request under a `byok_with_house_fallback` policy failed on
your own Anthropic connection, then succeeded on the marketplace `anthropic` deployment. Each
attempt also carries its billing source, price snapshot, error origin, admission and route
context, and dropped parameters.

```json
{
  "request_id": "2f6d8e3b-…",
  "admission_events": [],
  "attempts": [
    {
      "attempt": 1,
      "deployment": "byok:5f3a9c1e-…:0b7d42e8-…",
      "model": "claude-sonnet-5",
      "outcome": "failed",
      "usage_provenance": "unknown",
      "tokens": { "input": 0, "cached": 0, "cache_write": 0, "output": 0, "reasoning": 0 },
      "reserved_max_usd": 0,
      "cost_usd": 0,
      "error_code": "upstream_error",
      "dispatched_at": "2026-09-04T09:12:44.120Z"
    },
    {
      "attempt": 2,
      "deployment": "anthropic",
      "model": "claude-sonnet-5",
      "outcome": "completed",
      "usage_provenance": "observed",
      "tokens": { "input": 412, "cached": 0, "cache_write": 0, "output": 638, "reasoning": 0 },
      "reserved_max_usd": 0.04,
      "cost_usd": 0.007204,
      "error_code": "",
      "dispatched_at": "2026-09-04T09:12:45.010Z"
    }
  ]
}
```

The response holds metadata only. See [Data policy](/docs/data-policy).

| Field | Values |
|---|---|
| `outcome` | `completed`, `failed`, `cancelled`, `incomplete` (failed after your answer started) or `unknown_after_crash` (settled at the reserved maximum until reconciled) |
| `admission_events` | Refusals before any attempt, such as a rate limit |

Use this endpoint to recompute an Anthropic-surface stream that failed mid-answer (see
[Streaming](/docs/streaming)).

---

Source: https://demo.routerplus.com/docs/model-fallback.md

# Model fallback

Model fallback tries up to five models in your order until one answers. It works on
`POST /v1/chat/completions` and `POST /v1/messages`, streamed or not.

## Request

Send `models` next to `model`:

```bash
curl https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-5-5",
    "models": ["gpt-6-luna", "z-ai/glm-5.3"],
    "messages": [{ "role": "user", "content": "Hello" }]
  }'
```

| Field | Rule |
|---|---|
| `models` | An array of 1 to 5 model ids. |
| `model` | Tried first. Without `model`, the first entry of `models` goes first. |
| `route` | Optional. The only value is `"fallback"`. |

- The gateway removes duplicates and keeps your order.
- Each model gets your request with its own id in `model`. All other fields go as you sent them.
- Each model first fails over between its own providers (see
  [Routing & failover](/docs/routing)). The gateway tries the next model only after every
  provider of the current model failed.

## When the next model is tried

The gateway moves to the next model only before any part of an answer reaches you.

| Response from a model | What happens |
|---|---|
| 2xx | You get this answer. The request stays with this model. |
| 429, 404, 5xx, other 4xx | The gateway tries the next model. |
| 400 because the model cannot take the request | The gateway tries the next model. This covers context length, a model that is not available, an unsupported parameter, an input type (for example images) and `response_format`. |
| Any other 400 | You get this error. The request itself is wrong, so another model would fail the same way. |
| 401, 403 | You get this error at once. This covers a bad key and a guardrail refusal. |
| 413 | You get this error at once. The request is too large for every model. |

When every model fails, you get the last model's error unchanged.

The gateway does not switch models after a stream starts. A failure in the stream ends it with
the error chunk in [Streaming](/docs/streaming).

## Response

The response is the answering model's own answer, and `model` names that model. The usual
headers (`x-request-id`, `x-tm-provider`, `x-tm-attempts` and the others) come from that
model's call. The gateway adds two headers:

| Header | Meaning |
|---|---|
| `x-tm-fallback-model` | The model that answered. |
| `x-tm-fallback-tried` | How many models were tried, the answering model included. Also set on an error. |

## Billing

RouterPlus bills each model call as a direct call to that model. A model that failed before it
produced output costs what its provider billed for the attempt, as in
[Routing & failover](/docs/routing#the-commit-boundary).

## Errors

| Status | Cause |
|---|---|
| 400 `invalid_request` | `models` is not an array of 1 to 5 non-empty strings, or `route` is not `"fallback"`. |

---

Source: https://demo.routerplus.com/docs/auto-router.md

# Auto router

Send the model `routerplus/auto`, and RouterPlus selects a model for each request. The router
reads what the request asks for and stays in the cost tier that you select. It keeps one
conversation on one model.

## Request

```json
{
  "model": "routerplus/auto",
  "messages": [{ "role": "user", "content": "Why does this throw?\n```js\nnull.x\n```" }],
  "auto": { "tier": "economy", "prefer": "cost", "models": ["deepseek/deepseek-v4.1-flash", "claude-haiku-5-5"] }
}
```

The `auto` object and all of its fields are optional.

| Field | Rule |
|---|---|
| `auto.tier` | `economy`, `balanced` (default) or `premium`. |
| `auto.prefer` | `quality` (default) or `cost`. With `cost`, the router can also select models in cheaper tiers, and price counts for more. |
| `auto.models` | The models that the router can select. Leave it out to use the full list below. |

## Tiers

| Tier | Models |
|---|---|
| `premium` | `claude-opus-5-5`, `gpt-6-astra` |
| `balanced` | `claude-sonnet-5-5`, `gpt-6-luna`, `moonshotai/kimi-k3`, `z-ai/glm-5.3` |
| `economy` | `claude-haiku-5-5`, `deepseek/deepseek-v4.1-flash`, `openai/gpt-oss-120b`, `z-ai/glm-5.3-flash` |

The router selects from your tier. When no model in your tier can serve the request, it selects
from the next tier up.

## How it selects

1. The router reads the conversation and names the task:
   - `code`: code fences, stack traces, file paths or SQL.
   - `reasoning`: proofs, math or requests for step-by-step work.
   - `extraction`: `response_format` with a JSON schema.
   - `vision`: images in the latest message.
   - `long_context`: 120,000 or more characters of input.
   - `chat` or `creative` otherwise.
2. It removes the models that cannot serve the request:
   - models without image input, when you send images
   - models without tools, when you send tools
   - models with a context window too small for your input
3. It removes the models that your key's guardrails or `auto.models` do not allow.
4. It ranks the remaining models by how well each one does the task, then by price.

A short follow-up such as "thanks, now make it faster" keeps the conversation's task.

## One model per conversation

The router keeps a conversation on the model that it selected first. The router identifies a
conversation by your system prompt plus your first user message. It selects again only in these
cases:

- The latest message asks for a different task, for example a code conversation that now asks
  for a poem.
- The model can no longer serve the request.

## Fallback

The router also ranks up to 4 more models. When the selected model fails before its answer
starts, RouterPlus tries the next model, as in [Model fallback](/docs/model-fallback).

## Response

The response is the selected model's answer, and `model` names that model. Two headers
tell what the router did:

| Header | Meaning |
|---|---|
| `x-tm-auto-model` | The model that the router selected. |
| `x-tm-auto-reason` | A short reason, for example `task code; tier economy; strong at code`. |

## Billing

RouterPlus bills the request as a direct call to the model that answered. The router adds no
charge.

## Custom routers

Your organization can store its own router and call it as the model `routerplus/auto@<slug>`.
A router has a list of models that it can select, rules that it tries in order, and a default
model. In the console, open Build > Routers.

```json
{
  "model": "routerplus/auto@support",
  "messages": [{ "role": "user", "content": "Where is my invoice for October?" }]
}
```

The router selects a model in this order:

1. The trained classifier, if the router has one and its prediction is above the threshold. See
   [Training](#training).
2. The rules, in order. The first rule that matches selects its model.
3. The built-in ranking above, over the router's candidates.
4. The default model.

- The router never selects a model that your key's guardrails or `auto.models` do not
  allow. It goes to the next step instead.
- `auto.tier` and `auto.prefer` in the request apply to step 3.
- In step 3, one conversation keeps its model, as above. The classifier and the rules decide
  again on each request.

The response has the `x-tm-auto-model` and `x-tm-auto-reason` headers. It also has
`x-tm-router` with the router's slug and version, for example `support@3`. The reason
names the step: `classifier: claude-haiku-5-5 at 94%`, `rule 2`, a ranking reason, or `default`.
An unknown slug returns 404 `model_unavailable`.

### Spec

```json
{
  "candidates": [
    { "model": "claude-haiku-5-5", "tier": "economy" },
    { "model": "claude-sonnet-5-5", "strengths": ["code", "creative"] }
  ],
  "rules": [
    { "when": { "contains": ["\\binvoice\\b", "\\brefund\\b"] }, "use": "claude-haiku-5-5" },
    { "when": { "task": ["code"], "min_input_chars": 20000 }, "use": "claude-sonnet-5-5" }
  ],
  "default": "claude-haiku-5-5",
  "prefer": "cost"
}
```

| Field | Rule |
|---|---|
| `candidates` | Up to 20 models for step 3. `tier` and `strengths` are optional. A model from the tiers table keeps its tier, strengths and context window. Any other model is `balanced`, and the router assumes that it takes images and tools. |
| `rules` | Up to 50 rules. Each rule has `when` and `use` (a model id). |
| `default` | The model for step 4. Required. |
| `prefer` | `cost` or `quality`, for step 3. `auto.prefer` in the request wins. |

Every condition in `when` must be true. A rule with an empty `when` always matches.

| Condition | Matches when |
|---|---|
| `task` | The task is one of these: `code`, `reasoning`, `extraction`, `chat`, `creative`, `vision`, `long_context`. |
| `contains` | One of these regular expressions matches the first 20,000 characters of the latest user message. Case is ignored. 1 to 10 patterns of up to 512 characters. |
| `min_input_chars` | The input, with the system prompt and the tool definitions, has at least this many characters. |
| `has_images` | The conversation has an image (`true`), or has none (`false`). |
| `has_tools` | The request sends tools (`true`), or sends none (`false`). |

`contains` patterns follow the same safety rules as guardrail regex rules. RouterPlus refuses
these patterns with 400 `invalid_router`:

- a lookahead or a lookbehind
- a backreference
- a repeated group that holds a quantifier or an alternation, such as `(a+)+`

### Manage routers

Use a management key, under `https://demo-api.routerplus.com/v1` or `https://demo-api.routerplus.com/api/v1`.

| Method and path | Body | Result |
|---|---|---|
| `GET /v1/routers` | | The latest version of each router. |
| `POST /v1/routers` | `{ "slug": "support", "spec": { ... } }` | `201`. A slug that exists returns `409`. |
| `GET /v1/routers/:slug` | | The latest version. `?version=2` reads an older version. |
| `PUT /v1/routers/:slug` | `{ "spec": { ... }, "expected_version": 3 }` | A new version. `expected_version` is optional. When it is not the latest version, the response is `409`. |
| `DELETE /v1/routers/:slug` | | The router stops working. |
| `POST /v1/routers/:slug/train` | `{ "examples": [ ... ], "threshold": 0.7 }` | A new version with a trained classifier. |

A slug is 1 to 80 characters from `a-z`, `0-9`, `_` and `-`, and starts with a letter or a
digit. Each change writes a new version. Requests use the latest version within 5 seconds.

```bash
curl -s https://demo-api.routerplus.com/v1/routers \
  -H "Authorization: Bearer $TM_MANAGEMENT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"slug":"support","spec":{"rules":[{"when":{"contains":["\\binvoice\\b"]},"use":"claude-haiku-5-5"}],"default":"claude-sonnet-5-5"}}'
```

## Training

A router can learn from your labeled requests. Send examples of text, each with the model that
must answer it:

```bash
curl -s https://demo-api.routerplus.com/v1/routers/support/train \
  -H "Authorization: Bearer $TM_MANAGEMENT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"threshold":0.8,"examples":[
        {"text":"I was charged twice this month","model":"claude-haiku-5-5"},
        {"text":"The export to CSV crashes with error 500","model":"claude-sonnet-5-5"}
      ]}'
```

- Send 10 to 5,000 examples. Each `text` has up to 10,000 characters.
- Label at least two models. Each label must be a candidate or the router's default.
- The gateway fits a naive Bayes text classifier on words and word pairs. It keeps the 20,000
  most frequent features.
- `threshold` is optional, above 0 and below 1. Default 0.7.

At request time, the classifier reads the latest user message. When its top model has a
probability above the threshold, that model answers, before the rules. Otherwise the rules
decide.

The response has the new version and a `training` object with these fields: `examples`,
`labels`, `features`, `threshold` and `training_accuracy`. `training_accuracy` is the share of
your examples that the classifier labels correctly. Router objects show a summary of the
classifier: its labels, feature count, example count, threshold and training time.

To replace the classifier, train again. A `PUT` without `model_classifier` keeps the current
classifier. To remove it, send `"model_classifier": null`.

## Errors

| Status | Cause |
|---|---|
| 400 `invalid_request` | The `auto` object is invalid. The message names the field. |
| 400 `invalid_request` | No allowed model can serve the request. The message names the need: image input, tools or context length. |
| 400 `invalid_router` | A custom router's spec or training request is invalid. The message names the field. |
| 400 `invalid_request` | No step of a custom router can select an allowed model. |
| 404 `model_unavailable` | The custom router slug is unknown. |

---

Source: https://demo.routerplus.com/docs/fusion.md

# Fusion

Fusion sends one prompt to several models at once. A judge model then picks the best answer or
writes one answer from all of them. Call `POST /v1/chat/completions` with the model
`routerplus/fusion`.

## Request

```bash
curl https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{
    "model": "routerplus/fusion",
    "messages": [{ "role": "user", "content": "Explain what a mutex is." }],
    "fusion": {
      "models": ["claude-sonnet-5-5", "gpt-6-luna", "moonshotai/kimi-k3"],
      "judge": "claude-opus-5-5",
      "mode": "synthesize",
      "max_tokens_each": 1024
    }
  }'
```

| Field | Rule |
|---|---|
| `fusion.models` | 2 to 5 different model ids. These are the candidates. |
| `fusion.judge` | Optional. The model that judges. The default is the first candidate. |
| `fusion.mode` | `"synthesize"` (default): the judge writes one answer from the candidates. `"judge"`: the judge picks the best candidate, and you get that answer word for word. |
| `fusion.max_tokens_each` | Optional. The output limit for each candidate. Without it, each candidate gets your `max_tokens`. |

- Each candidate gets your messages and parameters. Candidates do not stream.
- The judge sees the candidate answers as A, B, C and so on, never the model names.
- `stream: true` streams the answer. In `synthesize` mode the judge's answer streams as the
  judge writes it. In `judge` mode the picked answer streams after the judge picks.

## Response

The response is an OpenAI chat completion with `model: "routerplus/fusion"`. `usage` is the sum
over every candidate and the judge. `usage.cost` is present when the calls report a cost. A
streamed response carries `usage` and `fusion` on its last chunk before `[DONE]`.

```json
{
  "model": "routerplus/fusion",
  "choices": [{ "index": 0, "message": { "role": "assistant", "content": "..." }, "finish_reason": "stop" }],
  "usage": { "prompt_tokens": 1830, "completion_tokens": 892, "total_tokens": 2722, "cost": 0.0213 },
  "fusion": {
    "mode": "synthesize",
    "candidates": [
      { "model": "claude-sonnet-5-5", "finish_reason": "stop", "usage": { "prompt_tokens": 20, "completion_tokens": 300, "total_tokens": 320 } },
      { "model": "gpt-6-luna", "finish_reason": null, "usage": { "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0 }, "error": "the provider is unavailable" },
      { "model": "moonshotai/kimi-k3", "finish_reason": "stop", "usage": { "prompt_tokens": 20, "completion_tokens": 280, "total_tokens": 300 } }
    ],
    "judge": { "model": "claude-opus-5-5", "usage": { "prompt_tokens": 1790, "completion_tokens": 312, "total_tokens": 2102 } }
  }
}
```

In `judge` mode, `fusion.judge` also has `pick` (the letter) and `selected` (the model whose
answer you got).

## When a call fails

- RouterPlus leaves out a candidate that fails. Its entry in `fusion.candidates` has `error`.
- When only one candidate answers, you get its answer with no judge call.
  `fusion.judge.skipped` says so.
- When every candidate fails, you get the first candidate's error.
- When the judge fails, you get the longest candidate answer. `fusion.judge.error` says why
  and `fusion.judge.fallback` names the candidate.

## Billing

RouterPlus bills each candidate call and the judge call as a direct call to its model. A fusion
request costs the sum of those calls.

## Errors

| Status | Cause |
|---|---|
| 400 `invalid_request` | The `fusion` object is missing or invalid. The message names the field. |
| 400 | `n` is above 1. Fusion returns one choice. |

Fusion does not work with [server tools](server-tools.md).

---

Source: https://demo.routerplus.com/docs/routing-policies.md

# Routing policies and Azure connections

A routing policy is a saved rule set for a key. It says which of your connections and which
marketplace deployments the key may use, in what order, and who pays. You can bind a policy to
an org, a workspace or a key.

## Create and bind a policy

Policy management uses the site origin, with the same session cookie, JSON content type and
same-origin write protection as [BYOK connection management](/docs/byok). Inference keys cannot
manage policies. Viewers can read them.

`POST /api/routing-policies` creates a policy revision. A revision never changes.

```json
{
  "models": ["your-exact-model-id"],
  "connections": ["FIRST_CONNECTION_UUID", "SECOND_CONNECTION_UUID"],
  "funding": "byok_only",
  "allow_fallbacks": true,
  "max_attempts": 4,
  "timeout_ms": 450000
}
```

The response is `201` with `{"policy_id":"UUID"}`. The connections must belong to your org.

| Request | Effect |
|---|---|
| `POST /api/routing-policies/{id}/bind` with `{"key_id":"KEY_UUID"}` | Binds the policy to an existing key. |
| `POST /api/routing-policies/{id}/bind` with `{"scope":"org"}` | Binds the policy to the org. |
| `GET /api/routing-policies` | Lists the newest 100 revisions. |
| `GET /api/routing-policies/{id}` | Returns one revision. |

Binding rules:

- An org policy applies to every key in the org. A key policy can narrow it.
- A key with no policy and no single-connection binding of its own uses the org policy.
- An older single-connection binding stays as an extra restriction when an org policy exists.
- Binding a policy to a key replaces the key's single-connection binding. Binding a single
  connection replaces the key's policy.
- To change a policy, create a new revision and bind it. You cannot delete a policy or unbind a
  key back to marketplace-only routing.
- Cached marketplace-only decisions can take up to 5 seconds to see a new binding. Calls already
  dispatched can finish on their pinned revision.
- Workspace bindings use the [identity API](identity.md).

## Policy fields

| Field | Behavior |
|---|---|
| `models` | 1 to 100 exact IDs. Every policy that applies must allow the requested model. |
| `connections` | Up to 8 BYOK connection UUIDs that your org owns, in order. |
| `house_deployments` | Up to 8 marketplace deployment IDs. |
| `funding: "byok_only"` | The default. Needs connections. Forbids marketplace deployments. |
| `funding: "house_only"` | Needs marketplace deployments. Forbids connections. |
| `funding: "byok_with_house_fallback"` | Needs both. Marketplace deployments are always the last phase. |
| `allow_fallbacks` | Default `true`. `false` keeps only the first eligible route, before health filtering. |
| `max_attempts` | Integer 1 to 8. Default 4. |
| `timeout_ms` | Admission and generation deadline, 1,000 to 600,000 ms. Default 450,000. |
| `regions`, `zdr`, `data_collection` | Hard restrictions, checked against provider facts that RouterPlus operators record. A route with no recorded fact is refused. |

### Fallback

- Under a policy, automatic fallback stays within the first selected **provider label**.
- You can authorize and select both OpenAI and Azure. A failed OpenAI call does not switch to
  Azure on its own.
- RouterPlus does not decide that two models are the same. It rejects
  `attest_equivalence:true`.
- Azure deployment mappings are your declarations.
- When fallbacks are off, an unavailable first route does not authorize a fallback.
- No retry crosses the [streaming commit boundary](/docs/routing#the-commit-boundary) or a
  content-policy refusal.

### How restrictions combine

- The restrictions in force are the intersection of the org policy, workspace policy, key
  policy or binding, the server catalog and the request. An empty intersection refuses the
  request. Lower limits win.
- Order: request connection preference, then provider preference, then saved order, then
  deployment ID as a stable tie-breaker.
- With mixed funding, marketplace deployments come after every BYOK route.

### Deadline

`timeout_ms` covers admission and generation after planning, including active streams. Request
upload, policy and price reads, and the final billing settlement are separate phases. A timeout
never skips billing settlement. A reservation that arrives after the deadline is released, and
RouterPlus never dispatches it.

## Request controls

The inference `provider` object accepts:

```json
{
  "only": ["openai"],
  "ignore": [],
  "order": ["openai"],
  "connections": ["CONNECTION_UUID"],
  "connection_order": ["CONNECTION_UUID"],
  "funding": ["byok"],
  "allow_fallbacks": false,
  "require_parameters": true
}
```

`only`, `ignore` and `order` name route labels: marketplace `anthropic` and `openai`, and a BYOK
connection's profile (`openai`, `anthropic`, `azure`, `bedrock`). A route label keeps that
meaning even when its route does not serve the model. Any other label matches no route. It
selects, orders or skips nothing, and an `only` list with no route label leaves no route.

On `POST /v1/decisions` the labels are the decision providers:

- `typesafe` for Jev
- `bespokelabs` for the Bespoke Nimble models
- `workers-ai` for Clef and Clef-flash
- `routerplus` for Decider 2B and Kev 4B
- `perplexity-decisions` for Perplexity Decider v1 27B
- `openai-decisions` for GPT-6 Luna
Decisions
- `levanto` for Sage
- `fastino` for GLiNER-2.5-Decide

Clef's label is `workers-ai`, not `cloudflare`: on `POST /v1/decisions`, `cloudflare` names no
provider. Perplexity Decider's label is `perplexity-decisions`, not `perplexity`. GPT-6 Luna
Decisions' label is `openai-decisions`, not `openai`, the label of the RouterPlus OpenAI chat
route.

Other controls:

- `require_parameters: true` refuses a deployment that would drop one of your request fields.
  Without it, RouterPlus calls the deployment without that field.
- `regions`, `zdr` and `data_collection` work here too, as hard restrictions.
- `sort` (`price`, `throughput`, `latency` or `quality`) and `max_price` apply to marketplace
  routes only. See [Provider routing](/docs/provider-routing). The
  [Agent integration guide](agent-integration.md) covers how to keep a session's prompt cache
  warm.
- Raw request keys and arbitrary destinations return 400.

Request preferences never grant connections, models, providers or funds that the saved policy
does not allow. You cannot write the provider facts used for region and privacy restrictions. A
restriction is not a residency or ZDR guarantee.

### Explain a route

`POST /v1/route` takes `{"model":"ID","provider":{...}}` with the inference key. It returns,
with `advisory: true`:

- the plan, catalog and policy IDs;
- the candidates in order, with their funding and pinned marketplace price IDs;
- the excluded routes, each with a reason;
- the time and source of the health data.

It makes no provider request, decryption or reservation. Health and authorization can change
before the real dispatch.

Related endpoints:

- `GET /v1/models` applies the same restrictions. An optional URL-encoded JSON `provider` query
  parameter narrows the list. `output_modalities=text,image,decisions` filters by kind.
- Token counting uses the same plan but needs an Anthropic route. It sends one count request
  and does not fall back.
- `GET /v1/generation?id=...` shows each attempt's `route_context` and fallback cause.
- Policy requests return `x-tm-route-plan-id` for correlation.

### Funding

- BYOK needs no marketplace credit.
- Marketplace fallback reserves credit only when the request reaches it, and charges under the
  standard pricing rules.
- Earlier marketplace holds stay until the request finishes, so fallback admission is
  conservative while settlements are pending.
- Each attempt pins its own funding and credential.
- RouterPlus does not know what a provider charges you on BYOK. Marketplace prices come from the
  model price list and are fixed before dispatch.

## Azure/Foundry API-key connections

Create a connection with `POST /api/connections`. In the console, open Access > Provider
connections.

```json
{
  "profile": "azure",
  "models": ["your-public-model-id"],
  "api_key": "YOUR_RESOURCE_API_KEY",
  "endpoint_config": {
    "resource": "your-resource-name",
    "host": "foundry",
    "api_version": "v1",
    "deployments": {"your-public-model-id": "your-deployment-name"}
  }
}
```

- `host` is `openai` (`{resource}.openai.azure.com`) or `foundry`
  (`{resource}.services.ai.azure.com`). RouterPlus builds the public hostname. You cannot send a
  URL.
- Every model you declare needs a deployment mapping.
- `endpoint_config` cannot change. A change needs a new connection.
- Validate the connection, then bind it to a key or include it in a policy. Rotation and
  revocation follow the [connection lifecycle](/docs/byok).

RouterPlus calls `/openai/v1/chat/completions?api-version=v1`, with the deployment name in the
model field and the native `api-key` header. Validation calls the model-list endpoint and
generates nothing. These shapes follow Microsoft's [chat reference](https://learn.microsoft.com/en-us/rest/api/microsoft-foundry/azureopenai/chat),
[models reference](https://learn.microsoft.com/en-us/rest/api/microsoft-foundry/azureopenai/models)
and [v1 lifecycle guide](https://learn.microsoft.com/en-us/azure/foundry/openai/api-version-lifecycle).

| Supported | Not supported |
|---|---|
| API-key authentication, public v1 endpoints | Entra or workload identity, legacy dated APIs, preview versions, sovereign or private endpoints, Azure capacity or residency checks |

Validation confirms endpoint authentication only. It does not confirm deployment entitlement,
model equivalence, quota or capacity.

## Limits

- Shared admission needs declared [quota pools and gateway limits](/docs/admission).
- A full quota can skip a route that the policy allows. It never adds providers or marketplace
  funding.
- Model lists and route explanations do not reserve capacity.
- The attempt record stores at most 32 exclusion details, with `excludedTotal` for the full
  count. The catalog hash and the full route explanation keep the rest of the context.

---

Source: https://demo.routerplus.com/docs/dedicated-endpoints.md

# Dedicated endpoints

A dedicated endpoint is provider capacity reserved for your organization on one model, with its
own limits, prices and failover rules. RouterPlus sets one up with you under a contract. There
is no self-serve setup.

You call the endpoint with the model id RouterPlus gives you, in your organization's namespace:
`<your-namespace>/<name>`, for example `acme/gemma-31b`.

## Calling an endpoint

Use the endpoint's model id in place of a catalog model id. Both surfaces
(`/v1/chat/completions` and `/v1/messages`), streaming, and every request field the model
accepts work as usual.

```bash
curl https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "acme/gemma-31b", "messages": [{"role": "user", "content": "Hello"}]}'
```

Only your organization's keys can call the endpoint, and it can be limited to named
workspaces. Any other key gets the same 404 `model_unavailable` as for an unknown model.

On an open endpoint, the response's `model` is the model that ran, for example
`"model": "google/gemma-4-31b-it"`. On a closed endpoint it is the endpoint id (see
[Closed endpoints](#closed-endpoints)). Two headers say how the request was served:

| Header | Value |
|---|---|
| `x-tm-dedicated-endpoint` | The endpoint's id, for example `acme/gemma-31b`. |
| `x-tm-route-role` | `primary` (your dedicated capacity) or `fallback` (the shared pool). |

- In [`/v1/generation`](/docs/api-usage), each attempt's `route_context.dedicated` records the
  endpoint id, the revision of its configuration in force, and the role.
- `GET /v1/models` lists your organization's active endpoints with `owned_by` `routerplus`, on
  the base URL that serves the endpoint.
- The [Endpoints](https://demo.routerplus.com/endpoints) page lists your endpoints before the models you
  deployed from Optimize. For each endpoint it shows the id, model, base URL, status, provider
  and limits, and the price per million tokens: input, cached input and output, and the cache
  write where the rate card has one above $0.

## What serves your request

1. **Your dedicated capacity first.**
2. **Then the shared pool, for the same model only.** The request moves on when the dedicated
   deployment fails before your answer starts, when its circuit is open, or when its capacity is
   full. It goes to public deployments of the same model that meet your endpoint's constraints,
   never to a different model.
3. **The commit boundary holds.** Once the first token reaches you, the request stays with that
   provider. See [Routing & failover](/docs/routing).

Your endpoint's configuration sets the constraints. They apply to every fallback:

| Constraint | A fallback host qualifies only if |
|---|---|
| Precision | It serves one of the listed precisions, for example `bf16`. |
| Zero data retention | It keeps no prompts. |
| Data collection denied | It neither stores nor trains on requests. |
| Maximum price | Its price per million tokens is at or below the limit. This is the price of the provider that serves it (its live price when that provider sets its own). |
| Regions | It runs in one of the listed countries. |
| Hosts | It is one of the named hosts. |
| Require parameters | It accepts every parameter the request sends. |

- A host that has not stated a value for a constrained property does not qualify.
- A fallback that reaches several hosts through one route sends the constraints with the
  request. Only hosts that meet them serve it.
- A region constraint excludes such a route, because the route makes no promise about where a
  host runs.

An endpoint set to `primary_only` never falls back. When the dedicated deployment cannot serve,
you get the error.

### Request-level provider fields

For one request you can narrow the routes. You cannot widen them.

- `"provider": {"allow_fallbacks": false}` keeps the request on your dedicated capacity. It
  works on open and closed endpoints.
- On an open endpoint, `provider.only` and `provider.ignore` filter by provider name, as on any
  request.
- A closed endpoint selects its providers and hosts through the contract configuration.
  `provider.only`, `provider.ignore`, `provider.order`, `provider.upstream` and
  `provider.require_parameters` return `400 invalid_request`, empty selectors included. Omit
  those fields on closed endpoints.
- `provider.sort` and `provider.max_price` return `400 invalid_request` on every dedicated
  endpoint, open or closed. The message names the field. The endpoint's own constraints,
  including its maximum price, decide its routes.

## Closed endpoints

An endpoint is open unless your contract says otherwise. An open endpoint names the deployment
and host of the shared pool that served you, like any request. Your dedicated capacity shows as
`routerplus`.

A closed endpoint never names a deployment or a host. Every surface names RouterPlus as the
provider, whichever route served the request. Only the route role differs between routes:

| Surface | On a closed endpoint |
|---|---|
| `x-tm-provider` | `routerplus` |
| `x-tm-served-by`, `x-tm-upstream-model`, `x-tm-upstream-status`, `x-tm-dropped-params`, `x-tm-affinity-wait-ms` | Absent. |
| `x-tm-dedicated-endpoint`, `x-tm-route-role` | As on an open endpoint. |
| `x-tm-remaining-rpm`, `x-tm-remaining-tpm` | Counted against the endpoint's own limits only. |
| Chat completion body | A fixed set of fields, below. `model` is the endpoint id. |
| Stream chunks | The same fields in each chunk. A provider's SSE comment lines are dropped. A provider's error in the stream ends it with the gateway's own error event, and nothing follows that event. |
| Error messages | They name `routerplus`, never a deployment. A provider's own error body is never relayed. |
| Error statuses | A provider's 401, 402, 403 or 404 is a retryable 503 `upstream_error`. A `retry-after` is the gateway's own, never a provider's. |
| Limits | A refusal because the capacity behind one route is full has `x-tm-limit-scope: endpoint` and `x-tm-limit-id: endpoint:<endpoint id>`. |
| `provider.only`, `provider.ignore`, `provider.order`, `provider.upstream` | Refused with 400 `invalid_request`. |
| [`/v1/generation`](/docs/api-usage) | `deployment` is `routerplus` and `model` the endpoint id. `route_context` keeps only `dedicated` (the endpoint, its revision and the role). `admission_context` is null, `dropped_params` is empty, and admission events carry no pool scope id. |
| [`/v1/usage`](/docs/api-usage) | `deployment` is `routerplus` and `model` the endpoint id. |
| Console: Overview, Usage, Logs and a request's details | RouterPlus as the provider, no host, the endpoint id as the model, and "contract rate" as the price. |
| The Playground's receipt | Served by RouterPlus. |

Records stay masked once an endpoint has been closed. Requests it served before it closed, and
after it opens again, read as above.

The fields of a closed chat completion:

| Level | Fields |
|---|---|
| Top level | `id` (`chatcmpl-` and the request id without dashes), `object`, `created` (when the gateway received your request), `model`, `choices`, `usage` |
| A choice | `index`, `message` (not streamed) or `delta` (streamed), `finish_reason` (`stop`, `length`, `tool_calls`, `content_filter` or `function_call`; null until a stream ends), `logprobs` (only when you ask for them) |
| `message` or `delta` | `role`, `content`, `tool_calls` (`id`, `type`, `function.name`, `function.arguments`, and `index` in a stream), `function_call`, `refusal`, `reasoning_content` |
| `usage` | `prompt_tokens`, `completion_tokens`, `total_tokens`, `prompt_tokens_details.cached_tokens` and `completion_tokens_details.reasoning_tokens` (both always present, 0 when none), `cost` |

- Every other field is dropped, including any field a provider adds later.
- A field without a value is left out. A message keeps `content`: null beside tool calls.
- A model's thinking arrives as `reasoning_content` on every route.
- A tool call's id is the gateway's own: `call_` and 24 hex characters. It is the same on every
  chunk of the call, and it is the `tool_use` id on `/v1/messages`.
- In a stream, the first delta of a choice has the role. The first delta of a tool call has its
  id, type and name. The finish and the usage each come in a chunk of their own.

## Prices

| Attempt | Price |
|---|---|
| On your dedicated capacity | Your contract rate card, per million tokens. |
| On a fallback | The price of the provider that serves it, as for any other request (its live price when that provider sets its own). Your rate card instead, when your contract sets one price on every route. |

- A closed endpoint bills your rate card on every route. A second price would tell you which
  route served.
- RouterPlus tries a fallback only when its price is known and at or below the endpoint's
  maximum price, if one is set. This holds whichever price you then pay.
- In `/v1/generation`, an attempt billed at your rate card has `inference_price_source`
  `customer_rate` and a `price_snapshot_id` of the form `dedicated:<endpoint id>:r<revision>`,
  which names the rate card that produced the charge.
- `usage.cost` on the response is the full debit across all attempts, as on every request.

## Limits

For the endpoint's traffic, its own limits replace the default organization, workspace,
principal and key limits:

| Limit | Counted for |
|---|---|
| Requests per minute | The endpoint |
| Requests per second (optional): a count per second, or a paced rate | The endpoint |
| Tokens per minute | The endpoint |
| Concurrent requests | The endpoint |

A limit your organization set in Billing > Rate limits
([`/console/limits`](https://demo.routerplus.com/console/limits)) still applies on top. You can lower what one key or workspace may use. You cannot raise the
endpoint's limits.

A refusal is the usual 429 `rate_limit` (see [Rate limits & spend caps](/docs/limits)), with
`x-tm-limit-scope: endpoint`:

| Case | `x-tm-limit-kind` | `x-tm-limit-id` |
|---|---|---|
| Over the endpoint's RPM, RPS, TPM or concurrency | `rpm`, `rps`, `tpm` or `concurrency` | `endpoint:<endpoint id>` |
| Above a paced endpoint's hard rate | `rps` | `endpoint:<endpoint id>` |
| A paced endpoint's wait would be longer than its bound | `rps_queue` | `endpoint:<endpoint id>` |
| The dedicated capacity is full and the endpoint does not spill over to the shared pool | the measure that is full | `endpoint:<endpoint id>` |
| A closed endpoint: the capacity behind every route it could take is full | the measure that is full | `endpoint:<endpoint id>` |
| Fallback traffic is over the endpoint's fallback cap | `rpm`, `tpm` or `concurrency` | `endpoint-fallback:<endpoint id>` |

`GET /v1/limits?model=<your-namespace>/<name>` returns your key's effective limits on the
endpoint and a `dedicated` object: the endpoint, its revision, the model, the output default,
the fallback caps and whether it spills over.

When the shared counter store is unreachable:

- In outage mode `closed`, requests are refused.
- In the default `bounded_open` mode, requests are admitted under the endpoint's own limits,
  divided among the gateways that serve it, for up to 10 minutes. See
  [Limits and capacity](/docs/admission).

### The paced limit

When traffic arrives at random, a limit that counts each calendar second refuses requests in many
seconds, even when the average rate is below the limit. A paced limit queues a short burst
instead. It has three settings: the paced rate, a hard rate above it, and the longest wait (the
bound).

- Requests leave admission no faster than the paced rate. A request that must wait waits in the
  gateway. `x-tm-queue-ms` on the response gives the wait in milliseconds.
- A request whose wait would pass the bound gets 429 with `x-tm-limit-kind: rps_queue`. Its
  `retry-after` says when its wait would fit.
- The hard rate refuses a request at once, with `x-tm-limit-kind: rps`. It allows a burst as
  long as the bound (at least one second). It never refuses a request the queue would hold.
  With a bound of a second or more, a full queue refuses first.
- The wait comes before the request's own deadline starts, so it does not use the endpoint's
  total time.
- A waiting request holds no concurrency slot. Its place in the pace stays used if you
  disconnect.

For example, with a paced rate of 6 requests a second, a hard rate of 6.5 and a 2 s bound:

| Average rate, arriving at random | Refused (429) | Typical wait |
|---|---|---|
| 4 a second | Almost none | About 0.1 s |
| 5 a second | About 0.3 % | About 0.3 s |
| 6 a second | About 4 % | About 0.9 s |
| Steadily above 6 a second | The excess, once the queue is full | Up to 2 s |

A queue cannot hold a steady excess. Over time, the endpoint passes its paced rate.

## Timeouts and defaults

| Setting | On other requests | On an endpoint |
|---|---|---|
| Attempts per request | Up to 8 | 1 to 8. Default 3. |
| Total time per request | 450 s | 1 to 600 s |
| Wait for a streamed attempt's response headers before the next route | 20 s | 1 to 120 s |
| Time one attempt may take on a request that is not streamed, before the next route | 450 s | 1 to 600 s, below the total |
| `max_tokens` written into a request that sets none | The model's own maximum, fitted to its context window | Set per endpoint |

A shorter header wait (streamed) or attempt limit (not streamed) makes failover faster when the
dedicated deployment stalls. A stalled request still takes that limit plus the fallback's own
time. The attempt limit applies only while another route remains. The last route can run for
the rest of the total time.

## Other limits

- `POST /v1/route` does not explain dedicated endpoints yet.
- Every request to a deployment shares its health circuit. Two failures in a row on the
  dedicated deployment send all its traffic to the fallbacks for 30 seconds. Per-endpoint
  circuit settings are not available yet.
- RouterPlus changes an endpoint's configuration on request. Each change is a new revision. It
  reaches every gateway within seconds.

---

Source: https://demo.routerplus.com/docs/server-tools.md

# Server tools

Server tools are tools that RouterPlus runs for your model. Add them to `tools` on
`POST /v1/chat/completions`. The model calls them like any function tool. RouterPlus runs the
call, gives the result to the model and calls the model again. You get the final answer.

| Tool | What the model gets |
|---|---|
| `routerplus:advisor` | Advice from a second model that sees the conversation. |
| `routerplus:subagent` | The answer of a helper model to a task that the model writes. |
| `routerplus:web_fetch` | The text of a web page. |

## Request

```json
{
  "model": "moonshotai/kimi-k3",
  "messages": [{ "role": "user", "content": "Compare the two pricing pages and recommend one plan." }],
  "tools": [
    { "type": "routerplus:advisor", "advisor": { "model": "claude-opus-5-5", "max_calls": 2 } },
    { "type": "routerplus:subagent", "subagent": { "models": ["gpt-6-luna", "claude-sonnet-5-5"], "max_calls": 4 } },
    { "type": "routerplus:web_fetch", "web_fetch": { "allowed_domains": ["example.com"], "max_bytes": 300000 } },
    { "type": "function", "function": { "name": "save_report", "parameters": { "type": "object" } } }
  ]
}
```

| Field | Rule |
|---|---|
| `advisor.model` | The advisor model. |
| `advisor.max_calls` | 1 to 5. The default is 3. |
| `subagent.models` | 1 to 5 helper models. The model can pick one. The default is the first. |
| `subagent.max_calls` | 1 to 8. The default is 4. |
| `web_fetch.allowed_domains` | Optional. Fetch only these domains and their subdomains. |
| `web_fetch.max_bytes` | Up to 2000000. The default is 500000. RouterPlus cuts a longer page at this size. |
| `web_fetch.max_calls` | 1 to 10. The default is 5. You can send `web_fetch` without its object. |

List each tool type once. When a server tool is in the request, your own function tools cannot
use its name (`advisor`, `subagent` or `web_fetch`).

## What the model sees

RouterPlus gives the model one function tool for each server tool:

- `advisor(question)`: the advisor gets the conversation so far as text, plus the question. It
  answers with advice for the model.
- `subagent(task, model?)`: the helper gets the task as its only message. The model gets the
  helper's final answer.
- `web_fetch(url)`: RouterPlus fetches the URL and gives the model the page as plain text.

When a tool has used its `max_calls`, the model gets a message that says so and continues
without that tool.

## How the loop ends

- The model answers without a tool call. You get that answer.
- The model calls one of your own function tools. You get that answer with your tool calls.
  RouterPlus removes server tool calls from the same answer. Run your tools and send the next
  request as usual.
- After 12 model calls, you get the last answer without its tool calls, and
  `server_tools.stopped` is `"round_limit"`.

Your `tool_choice` applies to the first model call only.

## Response

The response is an OpenAI chat completion. `model` is your model. `usage` is the sum over every
model call: your model, the advisor and the helpers. `usage.cost` is present when the calls
report a cost.

```json
{
  "model": "moonshotai/kimi-k3",
  "choices": [{ "index": 0, "message": { "role": "assistant", "content": "..." }, "finish_reason": "stop" }],
  "usage": { "prompt_tokens": 5210, "completion_tokens": 940, "total_tokens": 6150, "cost": 0.0182 },
  "server_tools": {
    "rounds": 3,
    "stopped": "answer",
    "calls": [
      { "type": "web_fetch", "url": "https://example.com/pricing", "status": "ok", "bytes": 48211 },
      { "type": "subagent", "model": "gpt-6-luna", "status": "ok", "usage": { "prompt_tokens": 900, "completion_tokens": 300, "total_tokens": 1200 } },
      { "type": "advisor", "model": "claude-opus-5-5", "status": "ok", "usage": { "prompt_tokens": 2100, "completion_tokens": 120, "total_tokens": 2220 } }
    ]
  }
}
```

`server_tools.stopped` is `"answer"`, `"buyer_tools"` or `"round_limit"`. Each entry in
`server_tools.calls` has a `status`:

| Status | Meaning |
|---|---|
| `ok` | The tool ran. |
| `error` | The tool failed. `error` says why. The model got the error as the tool result. |
| `refused` | `web_fetch` did not fetch the URL. `error` says why. |
| `limit` | The tool had used its `max_calls`. |

A `web_fetch` entry has `truncated: true` when RouterPlus cut the page at `max_bytes`.

`stream: true` streams the final answer only. The last chunk before `[DONE]` carries `usage`
and `server_tools`.

## Web fetch rules

- RouterPlus fetches only `http` and `https` URLs. It refuses URLs with a user name or password.
- RouterPlus resolves the host name first. It refuses the URL when any address is private,
  loopback, link-local (cloud metadata addresses included), shared, multicast or reserved. This
  applies to IPv4, IPv6 and IPv4 addresses inside IPv6.
- The connection goes to the address RouterPlus checked.
- RouterPlus follows up to 3 redirects and checks each with the same rules, `allowed_domains`
  included.
- HTML becomes plain text. Other text types come back as they are. Images and other binary
  types are refused.
- A fetch stops after 20 seconds.

## Billing

RouterPlus bills each model call as a direct call to its model. Your key's limits and
[guardrails](guardrails.md) apply to each call. A request with server tools costs the sum of
those calls. `web_fetch` has no charge of its own.

## Errors

| Status | Cause |
|---|---|
| 400 `invalid_request` | A server tool is invalid. The message names the field. |
| 400 | `n` is above 1. Server tools answer with one choice. |
| 403 `guardrail_blocked` | A guardrail blocks server tools (checked before any model call), or does not allow the model, advisor model or helper model. |

## Limits

- Server tools do not work with `routerplus/fusion`, `routerplus/auto` or a `models` list.
- When a call of your model fails, you get that error unchanged.

---

Source: https://demo.routerplus.com/docs/mcp-gateway.md

# MCP gateway

The MCP gateway puts your MCP servers behind one RouterPlus endpoint. Your agents call the
servers with their RouterPlus inference key. The gateway adds the server's credentials,
applies your tool policy and logs each tool call.

The gateway uses the MCP Streamable HTTP transport (JSON-RPC 2.0). It accepts JSON and
`text/event-stream` responses from the server. In the console, open Build > MCP servers.

## Register a server

Send `POST /v1/mcp/servers` with a management key (`Authorization: Bearer tm_mk_...`):

```json
{
  "slug": "github",
  "url": "https://mcp.example.com/mcp",
  "auth_header": "Authorization",
  "auth_value": "Bearer ghp_...",
  "allowed_tools": ["search_*", "get_issue"],
  "blocked_tools": ["search_admin*"],
  "workspace_ids": []
}
```

| Field | Rule |
|---|---|
| `slug` | Required. 1 to 40 of `a-z`, `0-9` and `-`, starting with a letter or digit. Unique in your org. `servers` and `calls` are reserved. |
| `url` | Required. An `https` URL. `http` works only for `localhost`. No user name or password in the URL. |
| `auth_header` | Optional. The header name that the server expects, for example `Authorization` or `X-Api-Key`. |
| `auth_value` | Required with `auth_header`. Up to 3,000 bytes. RouterPlus stores it encrypted and never returns it. |
| `allowed_tools` | Optional. Tool names or globs. Empty (the default) allows every tool. |
| `blocked_tools` | Optional. Tool names or globs. A blocked tool stays blocked even when `allowed_tools` matches it. |
| `workspace_ids` | Optional. The workspaces whose keys can use the server. Empty (the default) means every workspace. |
| `enabled` | Optional. Default `true`. A disabled server returns 404. |

In a glob, `*` matches any run of characters. Every other character matches itself. Names are
case-sensitive.

The response is `201`, with the server. It shows `auth_header` and `has_auth_value`. It never
shows the value.

## Manage servers

| Route | What it does |
|---|---|
| `GET /v1/mcp/servers` | Lists your servers. |
| `GET /v1/mcp/servers/{id}` | Reads one server. |
| `PATCH /v1/mcp/servers/{id}` | Changes the fields you send. To rotate the secret, send `auth_value` alone. To remove the auth header and its value, send `"auth_header": null`. |
| `DELETE /v1/mcp/servers/{id}` | Deletes the server. Its call log stays. |
| `GET /v1/mcp/calls` | Reads the tool call log. |

Every route also works under `/api/v1`. Each change writes a row to the audit log
(`GET /v1/audit`).

## Call a server

Point your MCP client at `https://<gateway>/v1/mcp/{slug}` and send your inference key:

```
Authorization: Bearer tm_vk_...
```

| Method | Use |
|---|---|
| `POST` | Sends a JSON-RPC message or batch. The response is JSON or an SSE stream, in the form that the server sends. |
| `GET` | Opens the server's SSE stream. |
| `DELETE` | Ends the session. |

- The gateway passes `mcp-session-id`, `mcp-protocol-version` and `last-event-id` to the
  server. It returns the server's `mcp-session-id` to you.
- It sends the server's `auth_header` with the stored value. It never sends your
  RouterPlus key to the server.

## Tool policy

- `tools/list`: the gateway removes the tools that your policy does not allow, in JSON and in
  SSE.
- `tools/call` for a tool that your policy does not allow: the gateway returns this error and
  does not call the server:

```json
{ "jsonrpc": "2.0", "id": 3, "error": { "code": -32601, "message": "tool not allowed by RouterPlus policy" } }
```

- The gateway checks a batch one message at a time. Allowed messages go to the server. Refused
  calls get the error above, in the same response.
- Other methods (`initialize`, `resources/*`, `prompts/*`, notifications) pass through.

## Tool call log

The gateway records each `tools/call`: the tool name, the key, the server, the status and the
duration. It does not store arguments or results.

| Status | Meaning |
|---|---|
| `ok` | The server returned a result. |
| `error` | The server returned a JSON-RPC error or a result with `isError: true`. |
| `blocked` | Your policy refused the call. The gateway did not call the server. |
| `upstream_error` | The server was unreachable or returned an HTTP error. |
| `timeout` | The server did not answer in time. |
| `no_answer` | The server ended its response without an answer to the call. |

`GET /v1/mcp/calls` takes these query parameters: `server_id`, `key_id`, `tool`, `status`,
`from` and `to` (ISO 8601 times), and `limit` (1 to 500, default 100). The newest calls come
first.

## Limits and errors

| Limit | Value |
|---|---|
| Request body | 4 MiB. A larger body gets `413`. |
| Server timeout | 120 s until the server starts its response, and 120 s between parts of a stream. When the server does not start in time, the response is `504`. |
| JSON response from the server | 16 MiB. |

| Status | Cause |
|---|---|
| `401` | The inference key is missing or not valid. |
| `404` | No enabled server has this slug in your org, or the server is not open to your key's workspace. |
| `400` with JSON-RPC code `-32700` | The body is not valid JSON. |
| `400` `upstream_refused` | The server URL resolves to a private, loopback or link-local address. The gateway does not call it. |
| `502` `upstream_auth_failed` | The server refused the stored credentials (HTTP 401 or 403). |
| `502` `upstream_error` | The server was unreachable, or sent a redirect or invalid JSON. |

Other HTTP errors from the server (for example `404` for an expired session) reach you
unchanged.

---

Source: https://demo.routerplus.com/docs/streaming.md

# Streaming

Set `"stream": true` to receive the answer as Server-Sent Events (SSE). Both chat surfaces
stream: `POST https://demo-api.routerplus.com/v1/chat/completions` (OpenAI) and
`POST https://demo-api.routerplus.com/v1/messages` (Anthropic).

```bash
curl -N https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-5","stream":true,"max_tokens":100,
       "messages":[{"role":"user","content":"Say hello"}]}'
```

The images route (`POST https://demo-api.routerplus.com/v1/images/generations`) does not stream. There,
`stream: true` and `partial_images` are typed 400s.

## Relayed and translated streams

RouterPlus handles a stream in one of two ways. The provider's dialect decides which.

**Same dialect: relayed.** The provider speaks the dialect of your surface, for example a Claude
model on `/v1/messages` or a GPT model on `/v1/chat/completions`. RouterPlus relays the
provider's stream record for record, with the payloads unchanged.

- Provider features reach you as they do from the provider: server tool use, web search
  results, citations, annotations, `pause_turn` and the matched `stop_sequence`.
- RouterPlus reads the stream for billing and failover. It adds `usage.cost` to the provider's
  usage record and makes sure the stream ends with a well-formed terminator.
- On the OpenAI surface, each chunk keeps only OpenAI's fields and names the model you asked
  for. RouterPlus drops fields the provider adds (its cost, its accounting, its host's name).
  The delta passes as the provider sent it. See
  [what an answer keeps](/docs/compat#what-an-answer-from-an-openai-dialect-provider-keeps).
- An explicit `stream_options: {"include_usage": false}` turns the relay off. RouterPlus must
  then remove the usage chunk it asked the provider for.

**Other dialect: translated.** RouterPlus translates the provider's stream into the framing of
your surface. The frame shapes below apply.

## Deadlines

| Deadline | Value |
|---|---|
| First byte (headers) | 20 seconds. A provider that misses it is failed over, and you get no error. See [Routing & failover](/docs/routing). |
| After the first byte | None. RouterPlus does not cut a slow stream. |
| Whole request | 450 seconds from dispatch, or the `timeout_ms` of a [routing policy](/docs/routing-policies) (1 to 600 s). |

A stream that is still open at the request deadline ends with the terminal error event in
[When a provider dies mid-stream](#when-a-provider-dies-mid-stream).

## Response headers

Every stream response carries `x-request-id`, `x-tm-provider` (the deployment that served you),
`x-tm-attempts` and `x-tm-upstream-status`. It carries `x-tm-dropped-params`,
`x-tm-upstream-model` and `x-tm-served-by` when they apply. See
[Wire compatibility](/docs/compat#response-headers).

## Frame order: OpenAI surface

Each frame is a complete `data: {json}\n\n` line in the `chat.completion.chunk` shape. The
order is fixed: role-priming delta, content deltas, finish chunk, usage chunk, `data: [DONE]`.

```text
data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":" there."},"finish_reason":null}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{},"finish_reason":"stop","native_finish_reason":"end_turn"}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":9,"total_tokens":21,"prompt_tokens_details":{"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"reasoning_tokens":0},"cost":0.000114}}

data: [DONE]
```

- RouterPlus sends the role-priming delta (`{"role":"assistant","content":""}`) when the
  upstream responds. It shows that the stream is alive and carries no content.
- Tool calls stream as `delta.tool_calls` entries with stable, ascending `index` values.
  Concatenate the argument fragments. A tool called with no arguments gets one `"{}"` fragment.
- Reasoning models stream their thinking as `delta.reasoning_content`.
- Refusal text streams as `delta.refusal`.
- On a translated stream, the finish chunk also carries `native_finish_reason`, the provider's
  own stop reason (here `end_turn`).
- The usage chunk is always sent, zero-filled if the provider reported nothing. The exception
  is a request with `stream_options.include_usage: false`.
- `[DONE]` comes only after a successful finish. It never follows an error.

## Frame order: Anthropic surface

Frames are named SSE events (`event: <name>\ndata: {json}\n\n`) in the Anthropic Messages API
order: `message_start`, `content_block_start`, `content_block_delta`, `content_block_stop`,
`message_delta`, `message_stop`.

```text
event: message_start
data: {"type":"message_start","message":{"id":"msg_2f6d8e3b","type":"message","role":"assistant","model":"gpt-4o-mini","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":0,"output_tokens":0}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello there."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"input_tokens":12,"cache_read_input_tokens":0,"cache_creation_input_tokens":0,"output_tokens":9,"cost":0.000007}}

event: message_stop
data: {"type":"message_stop"}
```

- On a translated stream, the token counts in `message_start` are zeros. Read the final usage
  and `cost` in the last `message_delta`. On a relayed stream, `message_start` carries the
  provider's own counts.
- Tool calls stream as `tool_use` content blocks with `input_json_delta` fragments.
- Thinking streams as `thinking` blocks with `thinking_delta`, plus `signature_delta` where the
  provider signs it.
- `input_tokens` excludes cache reads and writes, per the [billing contract](/docs/compat).
  `cache_read_input_tokens` and `cache_creation_input_tokens` report cache traffic.

## Usage is always in the stream

You do not opt in to stream usage. RouterPlus adds `stream_options.include_usage` to every
OpenAI-dialect upstream request, because OpenAI streams carry no usage without it.

- RouterPlus honors an explicit `stream_options: {"include_usage": false}`. You are billed the
  same, and you get no usage chunk.
- Every other stream ends with the token counts the provider reported.
- Every billed stream carries `usage.cost` in USD. RouterPlus computes it with the integer
  micro-USD math the ledger settles with. It covers the full request debit, including attempts
  that failed over before your answer started.

> [!TIP]
> `cost` on the wire is the amount you are charged. You can recompute it from the token
> counts and the public prices at
> [https://demo.routerplus.com/api/models.json](https://demo.routerplus.com/api/models.json). Settlement rounds down.

## Keep-alive frames

Proxies, load balancers and some HTTP clients close idle TCP connections. Slow reasoning models
can be silent for a long time mid-answer. After your stream starts, RouterPlus sends a
keep-alive frame after any silence of 15 seconds or more.

| Surface | Frame | Who sees it |
|---|---|---|
| OpenAI | The SSE comment `: processing` | The SSE spec requires clients to ignore it. SDKs never see it. |
| Anthropic | `event: ping` / `data: {"type":"ping"}` | The same frame Anthropic's API sends. Anthropic SDKs skip it. |

RouterPlus measures the silence from the last real write, so a gap is at most about 15 s.
Nothing is sent after a terminal event, keep-alives included.

## Cancelling a stream

To cancel, close the connection (abort the HTTP request). RouterPlus then aborts the upstream
request, and the provider stops generating.

Billing:

- You pay for the input and for the output streamed to you before the abort. The attempt has
  the outcome `cancelled`, visible at `GET /v1/generation?id=<x-request-id>`.
- If the provider reported no final usage, you pay for the input and the output you received,
  never the full hold. The input count is the provider's if it sent one, else RouterPlus's
  estimate.
- A cancel before the first token costs the input only.

Uptime:

- A cancel does not count against the provider's uptime. The public uptime numbers and the
  health circuits exclude cancelled attempts.
- Exception: if the provider sent no response headers for 60 seconds before you closed the
  connection, the cancel counts as a provider failure. See
  [Routing & failover](/docs/routing#health-circuits-and-cooldowns).

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://demo-api.routerplus.com/v1",
  apiKey: process.env.TM_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "claude-sonnet-5",
  max_tokens: 4096,
  messages: [{ role: "user", content: "Explain SSE, thoroughly." }],
  stream: true,
});

let chars = 0;
for await (const chunk of stream) {
  chars += chunk.choices[0]?.delta?.content?.length ?? 0;
  if (chars > 2000) {
    stream.controller.abort(); // upstream generation stops; billed to here
    break;
  }
}
```

## When a provider dies mid-stream

If the serving provider fails after your answer started, RouterPlus does not restart the
request on another provider (see the commit boundary in [Routing & failover](/docs/routing)).
Instead, you get one terminal error event inside the open 200 stream.

**OpenAI surface.** RouterPlus first sends any known usage and its `cost`, so you can recompute
the billed partial attempt. Then it sends a final chunk with `finish_reason: "error"` and a
top-level `error`:

```text
data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}],"error":{"code":"upstream_error","message":"upstream failure","type":"gateway_error","metadata":{"error_type":"upstream_error"}}}
```

**Anthropic surface.** RouterPlus sends a native `error` event with the Anthropic type string
and the stable `error_type`:

```text
event: error
data: {"type":"error","error":{"type":"api_error","message":"upstream failure","error_type":"upstream_error"}}
```

Nothing follows a terminal event: no `[DONE]` and no other frames. The same event ends a
stream that reaches the request deadline. It also ends a stream whose provider closed the
connection without a proper terminator.

> [!NOTE]
> The Anthropic wire format has no usage slot outside `message_delta`. So an Anthropic-surface
> stream that fails mid-answer cannot carry its partial usage. Read the settled numbers from
> `GET /v1/generation?id=<x-request-id>`.

[Errors & remediation](/docs/errors) lists every `error_type` and what to do about it.

## SDK streaming examples

Both official SDKs work unmodified. Change the base URL and the key.

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

stream = client.chat.completions.create(
    model="claude-sonnet-5",
    max_tokens=200,
    messages=[{"role": "user", "content": "Write a haiku about failover."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:  # the final usage chunk has an empty choices list
        print(f"\ncost: ${chunk.usage.cost}")
```

```python
import os
import anthropic

client = anthropic.Anthropic(base_url="https://demo-api.routerplus.com", api_key=os.environ["TM_API_KEY"])

with client.messages.stream(
    model="gpt-4o-mini",  # a GPT model over the Anthropic wire
    max_tokens=200,
    messages=[{"role": "user", "content": "Write a haiku about failover."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()
    print(f"\nstop_reason: {final.stop_reason}")
```

---

Source: https://demo.routerplus.com/docs/response-cache.md

# Response cache

The response cache answers a repeated request from stored results. A cached answer comes back
at once and costs nothing. The cache works on `POST /v1/chat/completions` and
`POST /v1/messages`.

The cache is off by default. Turn it on for one request, or ask RouterPlus support to turn it
on for your organization.

## Request

```bash
curl https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{
    "model": "gpt-6-luna",
    "temperature": 0,
    "messages": [{ "role": "user", "content": "Summarize our refund policy." }],
    "cache": { "mode": "normalized", "ttl_seconds": 3600 }
  }'
```

Use the `x-tm-cache` header or the `cache` field in the body. The header wins over the field.

| `x-tm-cache` value | What happens |
|---|---|
| `on` | Use the cache for this request. |
| `off` | Do not use the cache for this request, even when your organization has it on. |
| `refresh` | Do not read the cache. Call the model and store the new answer. |
| `semantic` | Use [semantic matching](#semantic-matching). |

| `cache` field | Rule |
|---|---|
| `mode` | `"exact"` (default), `"normalized"` or `"semantic"`. |
| `ttl_seconds` | How long the answer stays, 1 to 86400. Default 300. |
| `shared` | `false` keeps the answer to one user. See [Scope](#scope). |
| `allow_nonzero_temperature` | `true` lets a request with `temperature` above 0 use the cache. |
| `semantic` | `true` turns on semantic matching. |

`"cache": true` turns the cache on with the defaults. `"cache": false` turns it off. The gateway
removes the `cache` field before it sends the request to a model.

## Modes

- `exact`: a hit needs the same request. The model, the messages, the system prompt, the tools,
  `tool_choice`, `response_format` and every sampling parameter (`temperature`, `top_p`, `seed`,
  `max_tokens`, `stop` and the others) must match.
- `normalized`: the same as `exact`, but the gateway trims the text of each message and turns
  each run of spaces, tabs and new lines into one space. Letter case still counts.

The match ignores `stream`, `stream_options`, `user`, `metadata`, `session_id` and
`prompt_cache_key`. A streamed request can get an answer that a non-streamed request stored.

## Semantic matching

Semantic matching returns a stored answer when the last user turn is close in meaning to a
cached one, inside the same [scope](#scope). It is off until RouterPlus sets it up for your
organization. Ask for it on a request with `x-tm-cache: semantic`,
`"cache": { "semantic": true }` or `"mode": "semantic"`.

- The response carries `x-tm-cache: hit` and `x-tm-cache-similarity` (cosine similarity, 0.97
  or more by default).
- The comparison ignores case, punctuation and spacing.
- Each lookup and store embeds the last user turn with `text-embedding-3-small`. This is an
  ordinary `POST /v1/embeddings` request on your key, billed at its price (about $0.02 per
  million tokens).

## Scope

- A stored answer belongs to one organization. Another organization never gets it.
- Inside your organization, a stored answer belongs to one user by default. The user is the
  request's `user` field (on Messages, `metadata.user_id`) or its session. When the request
  names neither, the API key is the user.
- Your organization can turn on sharing between its users. Then any user of your organization
  can get an answer that another user stored.
- With sharing on, a request can keep its answer to its own user with `"shared": false`.
- With sharing off, a request cannot share its answer.

## What a hit looks like

A hit is your stored answer with:

- a new `id`, and on chat completions a new `created`;
- `usage` with the stored token counts, `"cost": 0` and `"cached_response": true`;
- the headers `x-tm-cache: hit` and `x-tm-cache-age` (seconds since the answer was stored).

A hit costs nothing. When you ask to stream, the gateway sends the stored answer as a stream in
the live format: chat completion chunks on `/v1/chat/completions`, and Messages events on
`/v1/messages`.

## What is not cached

The gateway calls the model and stores nothing when a request:

- sets `temperature` above 0, unless `allow_nonzero_temperature` is `true`;
- asks for more than one choice (`n` above 1);
- uses a tool that runs on the provider's side, such as web search, code execution or an MCP
  server (tools that your code runs, functions, are cached);
- gives an image or a file by URL or by file id (images and files sent inline are cached).

The gateway stores only a complete answer. It does not store an error, an answer that ended
with an error, or a streamed answer. A streamed request can get only an answer that a
non-streamed request stored.

---

Source: https://demo.routerplus.com/docs/api-batches.md

# Batches

The Batches API runs many requests from one file and returns the answers in result files. It
uses the shapes of the OpenAI Files and Batches API. The OpenAI SDKs work when you change
`base_url` and the key.

A batch has four steps:

1. Upload a JSONL file of requests: `POST /v1/files`.
2. Create a batch from the file: `POST /v1/batches`.
3. Poll the batch until it ends: `GET /v1/batches/{batch_id}`.
4. Download the output and error files: `GET /v1/files/{file_id}/content`.

Every route also works under `/api/v1/`.

## Example

```python
import os, time
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

while batch.status not in ("completed", "failed", "cancelled", "expired"):
    time.sleep(30)
    batch = client.batches.retrieve(batch.id)

print(batch.request_counts)
if batch.output_file_id:
    print(client.files.content(batch.output_file_id).text)
if batch.error_file_id:
    print(client.files.content(batch.error_file_id).text)
```

To cancel:

```python
client.batches.cancel(batch.id)
```

## Authentication, billing and governance

Send your inference key as `Authorization: Bearer <key>`. Files and batches belong to the
key's org. A file or batch of another org returns 404.

Each line of a batch runs as an ordinary request to the gateway:

- RouterPlus bills it to your org at the normal price of the model and provider that serve it.
- Your org, workspace, member and key guardrails apply to it.
- It has the same rate limit as your key.
- Routing, provider failover and a `models` fallback list work as on a direct request.
- It shows in your usage and activity, with your key's labels and finance category.

When you create a batch, the gateway makes a key for that batch. The batch key copies these
settings from the key that created the batch: workspace, member, labels, finance category, rate
limit, routing and key guardrail. The batch key stops working when the batch ends.

> [!NOTE]
> Your key's spending limit and IP allowlist do not apply to the batch key. To limit batch
> spend, use a guardrail budget.

## The input file

Put one request on each line, as JSON:

```json
{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-6-luna", "messages": [{"role": "user", "content": "Hello"}]}}
```

| Field | Rule |
|---|---|
| `custom_id` | 1 to 64 characters. Unique in the file. The answer line has the same `custom_id`. |
| `method` | `"POST"`. |
| `url` | `/v1/chat/completions`, `/v1/messages` or `/v1/responses`. The same on every line. |
| `body` | The request body, as you send it to that endpoint. |

Limits:

- A file has at most 50 MiB.
- A file has at most 10,000 requests.
- The gateway always sends `stream` as `false`. Each line gets one complete answer.

The upload checks every line. A bad line returns 400. The message names the first bad line:

```json
{"error": {"message": "Line 2: method must be \"POST\".", "type": "invalid_request_error", "param": "file", "code": "invalid_file"}}
```

## Files

### Upload

Send `POST /v1/files` as `multipart/form-data` with the fields `purpose=batch` and `file`:

```bash
curl https://demo-api.routerplus.com/v1/files \
  -H "Authorization: Bearer $TM_API_KEY" \
  -F purpose=batch \
  -F file=@requests.jsonl
```

Or send the file as the body, with `content-type: application/jsonl`:

```bash
curl "https://demo-api.routerplus.com/v1/files?purpose=batch&filename=requests.jsonl" \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/jsonl" \
  --data-binary @requests.jsonl
```

The response is a file object:

```json
{"id": "file-5b2e0c7d9a1f4e3b8c6d0a2f", "object": "file", "bytes": 1832, "created_at": 1793577600,
 "expires_at": 1796169600, "filename": "requests.jsonl", "purpose": "batch", "status": "processed", "status_details": null}
```

RouterPlus keeps a file for 30 days.

### Other file routes

| Route | Response |
|---|---|
| `GET /v1/files/{file_id}` | The file object. |
| `GET /v1/files/{file_id}/content` | The file's JSONL content. |
| `DELETE /v1/files/{file_id}` | `{"id": "...", "object": "file", "deleted": true}`. You cannot delete the input file of a batch that has not ended (400). |
| `GET /v1/files?purpose=&limit=&after=&order=` | A list of file objects, newest first. `purpose` is `batch`, `batch_output` or `batch_error`. |

## Batches

### Create

`POST /v1/batches`:

| Field | Rule |
|---|---|
| `input_file_id` | A file with purpose `batch`. |
| `endpoint` | The `url` on the file's lines. |
| `completion_window` | `"24h"`. |
| `metadata` | Optional. Up to 16 pairs. Keys have up to 64 characters. Values are strings of up to 512 characters. |

```bash
curl https://demo-api.routerplus.com/v1/batches \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"input_file_id": "file-5b2e0c7d9a1f4e3b8c6d0a2f", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'
```

The response is a batch object:

```json
{"id": "batch_8f3a1c2e4b5d6f708192a3b4", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null,
 "input_file_id": "file-5b2e0c7d9a1f4e3b8c6d0a2f", "completion_window": "24h", "status": "validating",
 "output_file_id": null, "error_file_id": null, "created_at": 1793577600, "in_progress_at": null,
 "expires_at": 1793664000, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null,
 "cancelling_at": null, "cancelled_at": null, "request_counts": {"total": 3, "completed": 0, "failed": 0}, "metadata": null}
```

### Other batch routes

| Route | Response |
|---|---|
| `GET /v1/batches/{batch_id}` | The batch object. `request_counts` grows while the batch runs. |
| `POST /v1/batches/{batch_id}/cancel` | The batch object, with status `cancelling`. |
| `GET /v1/batches?limit=&after=` | A list of batch objects, newest first. `limit` is 1 to 100, default 20. `after` is a batch id. |

### Statuses

| Status | Meaning |
|---|---|
| `validating` | The batch is waiting to start. |
| `in_progress` | Lines are running. |
| `finalizing` | All lines ran. The gateway is writing the output and error files. |
| `completed` | The output and error files are ready. |
| `failed` | The batch could not run. `errors` gives the reason. No line ran after the failure. |
| `cancelling` | You cancelled. Lines that already started finish. No new lines start. |
| `cancelled` | The cancel is done. The files hold the lines that ran. |
| `expired` | The batch did not end within 24 hours of creation. The files hold the lines that ran. Every line that did not run is in the error file with code `batch_expired`. |

`request_counts.completed` counts the lines with a 2xx answer. `request_counts.failed` counts
all other lines.

### Retries

The gateway retries a line that gets 429 or 5xx, or that it cannot send. It makes up to 3
attempts in total. Before each retry it waits for the answer's `retry-after`, up to 60
seconds. With no `retry-after`, it waits 1 second, then 2 seconds. The batch records the last
answer.

## Output and error files

When a batch ends, `output_file_id` and `error_file_id` name its result files.

- A file id is `null` when the file has no lines.
- The lines follow the order of the input file. Match them to your requests by `custom_id`.

A line with a 2xx answer goes to the output file:

```json
{"id": "batch_req_1c9e7a0b3d5f2e4a6c8b0d1f", "custom_id": "req-1", "response": {"status_code": 200, "request_id": "8d1c0f6e-2b7a-4e39-9a51-3c0d2e4f6a7b", "body": {"id": "chatcmpl-1", "object": "chat.completion", "choices": []}}, "error": null}
```

A line with any other answer goes to the error file, with that answer's status and body:

```json
{"id": "batch_req_7e2d4c6a8b0f1e3d5c7a9b2e", "custom_id": "req-2", "response": {"status_code": 400, "request_id": "4f2a...", "body": {"error": {"message": "..."}}}, "error": null}
```

A line that got no answer has `response: null` and an `error` with a `code` and `message`.

`request_id` is the gateway's id for that request. Use it to find the request in your activity.

## Errors

Errors use the OpenAI shape:

```json
{"error": {"message": "No batch found with id 'batch_...'.", "type": "invalid_request_error", "param": "batch_id", "code": "batch_not_found"}}
```

| Status | When |
|---|---|
| 400 | A bad file, field or value. `param` names it. |
| 401 | No key, or a key that is not valid. |
| 404 | No such file or batch in your org. |
| 409 | The batch cannot be cancelled in its status. |
| 413 | The upload is larger than 50 MiB. |

---

Source: https://demo.routerplus.com/docs/evals.md

# Evals

An eval compares candidate models on your own cases. RouterPlus sends each case to each model.
Then a judge can score the answers, compare them with an expected answer, or select the best
answer for each case. When the eval ends, you get a summary for each model: quality, cost and
latency.

1. Create the eval with your cases and models: `POST /v1/evals`.
2. Poll the eval until it ends: `GET /v1/evals/{eval_id}`.
3. Read the answers for each case: `GET /v1/evals/{eval_id}/results`.

Every route also works under `/api/v1/`. In the console, open Build > Evals.

## Authentication, billing and governance

Send your inference key as `Authorization: Bearer <key>`. An eval belongs to the key's
org. An eval of another org returns 404.

Each candidate call and each judge call runs as an ordinary chat completion:

- RouterPlus bills it to your org at the normal price of the model and provider that serve it.
- Your org, workspace, member and key guardrails apply. Your key's rate limit applies.
- Routing and provider failover work as on a direct request.
- It shows in your usage and activity, with your key's labels and finance category.

When you create an eval, the gateway makes a key for that eval. The eval key copies these
settings from your key: workspace, member, labels, finance category, rate limit, routing and
key guardrail. The eval key stops working when the eval ends.

Your key's spending limit and IP allowlist do not apply to the eval key. To limit eval
spend, use a guardrail budget.

## Create an eval

`POST /v1/evals`:

| Field | Rule |
|---|---|
| `name` | 1 to 200 characters. |
| `cases` | An array of 1 to 2,000 cases. Send one of `cases` and `file_id`. |
| `file_id` | A file with purpose `evals`. See [Eval files](#eval-files). |
| `models` | 1 to 10 candidate model ids. |
| `judge` | Optional. See [Judge modes](#judge-modes). Without a judge, the summary has cost, latency, tokens and errors only. |
| `params` | Optional. `temperature` (0 to 2) and `max_tokens` (1 to 128,000). RouterPlus sends them on every candidate call. |

A case:

| Field | Rule |
|---|---|
| `id` | Optional. 1 to 64 characters, unique in the eval. Default `case-1`, `case-2` and so on. |
| `input` | A string of up to 200,000 characters, sent as one user message. Or an array of chat messages. |
| `expected` | Optional. The reference answer, up to 20,000 characters. Mode `match` needs it on every case. The other modes show it to the judge. |
| `criteria` | Optional. What a good answer to this case does, up to 4,000 characters. The judge reads it. |

```bash
curl https://demo-api.routerplus.com/v1/evals \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "name": "support answers",
    "cases": [
      {"input": "How do I reset my password?", "criteria": "Names the settings page."},
      {"input": [{"role": "system", "content": "Answer in one word."}, {"role": "user", "content": "Capital of France?"}], "expected": "Paris"}
    ],
    "models": ["gpt-6-luna", "deepseek/deepseek-v4.1-flash", "moonshotai/kimi-k3"],
    "judge": {"model": "gpt-6-luna", "mode": "score", "rubric": "Correct, short and polite."},
    "params": {"temperature": 0, "max_tokens": 400}
  }'
```

The response is the eval object, with status `queued`.

## Judge modes

| Mode | What happens | Needs `judge.model` |
|---|---|---|
| `score` | The judge reads each answer and gives a score from 1 to 10. One judge call per answer. | Yes |
| `match` | RouterPlus compares each answer to `expected`. No judge call. | No |
| `pairwise` | The judge reads all answers to a case and picks the best one. One judge call per case. Needs 2 or more models. | Yes |

- `rubric` is optional, up to 4,000 characters. The judge reads it on every call.
- The judge sees the answers with the labels A, B, C. It never sees the model ids. In
  `pairwise` mode the label order changes from case to case.
- In `pairwise` mode, when only one model answered a case, that model wins with no judge call.
  When no model answered a case, the case has no winner.
- In `match` mode, an answer is an `exact` match when it equals `expected` after trimming. It is
  a `normalized` match when both are equal after these changes: lower case, each run of
  whitespace as one space, and no quotes, brackets or end punctuation.
- A judge reply with no score from 1 to 10, or with no named answer, is a judge error. That
  answer or case has no score or winner.

## Eval files

For many cases, upload a JSONL file with one case on each line. Then send its id as `file_id`:

```json
{"id": "q1", "input": "Capital of France?", "expected": "Paris"}
{"id": "q2", "input": [{"role": "user", "content": "2 + 2?"}], "expected": "4", "criteria": "A number only."}
```

```bash
curl https://demo-api.routerplus.com/v1/files \
  -H "Authorization: Bearer $TM_API_KEY" \
  -F purpose=evals \
  -F file=@cases.jsonl
```

- The upload checks every line and names the first bad line.
- A file holds at most 2,000 cases.
- The other file routes are the same as for [batches](api-batches.md).
- The eval keeps its own copy of the cases. You can delete the file after you create the eval.

## Read an eval

`GET /v1/evals/{eval_id}`:

```json
{
  "id": "eval_3f9a0c1e2b4d6f8091a2b3c4", "object": "eval", "name": "support answers", "status": "completed",
  "models": ["gpt-6-luna", "deepseek/deepseek-v4.1-flash", "moonshotai/kimi-k3"],
  "judge": {"model": "gpt-6-luna", "rubric": "Correct, short and polite.", "mode": "score"},
  "params": {"temperature": 0, "max_tokens": 400}, "file_id": null,
  "progress": {"cases_total": 2, "cases_done": 2, "calls_done": 12, "call_errors": 0},
  "created_at": 1793577600, "started_at": 1793577601, "finished_at": 1793577640, "error": null,
  "summary": {
    "mode": "score",
    "models": [
      {"model": "deepseek/deepseek-v4.1-flash", "cases": 2, "answered": 2, "errors": 0, "mean_score": 8.5, "scored": 2,
       "match_rate": null, "exact_match_rate": null, "win_rate": null, "wins": 0, "cost_usd": 0.000412,
       "mean_latency_ms": 1840, "p50_latency_ms": 1620, "p95_latency_ms": 2060,
       "prompt_tokens": 64, "completion_tokens": 190, "total_tokens": 254}
    ],
    "judge": {"model": "gpt-6-luna", "calls": 6, "errors": 0, "cost_usd": 0.000951, "prompt_tokens": 2310, "completion_tokens": 140},
    "total_cost_usd": 0.002104
  }
}
```

`summary` is `null` until the eval ends. A cancelled eval has a summary of the calls that ran.

Summary fields for each model:

| Field | Meaning |
|---|---|
| `cases`, `answered`, `errors` | Cases sent to the model, answers with text, and calls that failed or came back empty. |
| `mean_score`, `scored` | Mode `score`: the mean judge score and the number of scored answers. |
| `match_rate`, `exact_match_rate` | Mode `match`: matched cases over all cases. A failed call counts as no match. |
| `win_rate`, `wins` | Mode `pairwise`: cases the model won over cases the judge decided. |
| `cost_usd` | The billed cost of the model's calls, in USD. Judge calls are in `judge.cost_usd`. |
| `mean_latency_ms`, `p50_latency_ms`, `p95_latency_ms` | Time to the full answer, over answered calls. |
| `prompt_tokens`, `completion_tokens`, `total_tokens` | Tokens of the model's calls. |

`total_cost_usd` is the cost of all candidate and judge calls. It matches your usage.

### Statuses

| Status | Meaning |
|---|---|
| `queued` | The eval is waiting to start. |
| `running` | Cases are running. `progress` grows. |
| `completed` | All cases ran. `summary` is ready. |
| `failed` | The eval could not run. `error` has a `code` and `message`. |
| `cancelled` | You cancelled. Cases that already started finish. No new cases start. |

An eval that has not ended 7 days after creation fails with code `eval_expired`.

## Results per case

`GET /v1/evals/{eval_id}/results?limit=&offset=` lists the cases in order. `limit` is 1 to 500,
default 100. `offset` is the first case's index, default 0.

```json
{
  "object": "list", "eval_id": "eval_3f9a0c1e2b4d6f8091a2b3c4", "status": "completed", "offset": 0, "limit": 100, "total": 2, "has_more": false,
  "data": [
    {"case_index": 1, "case_id": "case-2", "expected": "Paris", "results": [
      {"model": "deepseek/deepseek-v4.1-flash", "status": "ok", "answer": "Paris", "answer_truncated": false, "error": null,
       "score": 10, "matched": null, "won": null, "judge_reason": "Correct and short.",
       "cost_usd": 0.000021, "judge_cost_usd": 0.00016, "latency_ms": 980,
       "prompt_tokens": 22, "completion_tokens": 2, "request_id": "8d1c0f6e-2b7a-4e39-9a51-3c0d2e4f6a7b"}
    ]}
  ]
}
```

- `status` is `ok`, `error` or `pending` (not run yet).
- `answer` holds the first 4,000 characters. `answer_truncated` is `true` when the answer was
  longer.
- `score` is the judge score in mode `score`, and 1 or 0 in mode `match`.
- In mode `pairwise`, `won` tells if the model won the case. Each case has a `judge` object with
  `best_model`, `reason` and `cost_usd`.
- `request_id` is the gateway's id for the call. Use it to find the call in your activity.

## Other routes

| Route | Response |
|---|---|
| `POST /v1/evals/{eval_id}/cancel` | The eval object, with status `cancelled`. An eval that already ended returns 409. |
| `GET /v1/evals?limit=&after=` | A list of eval objects, newest first. `limit` is 1 to 100, default 20. `after` is an eval id. |

## Retries

RouterPlus retries a call that gets 429 or 5xx, or that it cannot send. It makes up to 3 attempts
in total. Before each retry it waits for the response's `retry-after`, up to 60 seconds.
With no `retry-after`, it waits 1 second, then 2 seconds. The eval records the last response.

## Errors

Errors use this shape:

```json
{"error": {"message": "No eval found with id 'eval_...'.", "type": "invalid_request_error", "param": "eval_id", "code": "eval_not_found"}}
```

| Status | When |
|---|---|
| 400 | A bad field or value, or a bad eval file. `param` names it. |
| 401 | No key, or a key that is not valid. |
| 404 | No such eval or file in your org. |
| 409 | The eval cannot be cancelled in its status. |
| 413 | The request body is larger than 32 MiB. Use an eval file. |

---

Source: https://demo.routerplus.com/docs/model-search.md

# Optimize

[Optimize](https://demo.routerplus.com/model-search) finds a cheaper model or mix that scores as well as
your model on your own Braintrust evals. It deploys the result as an endpoint: one model id that
you call like any other.

> [!NOTE]
> Optimize is paused. Searches cannot start, continue or deploy for now. Past results stay
> available. A deployed endpoint answers only while every model it calls is in the catalog.

## Before you start

1. In the console, open [Access > Integrations](https://demo.routerplus.com/console/integrations) and
   connect Braintrust with a Braintrust API key.
   - RouterPlus checks the key with Braintrust and stores it encrypted. The page shows only the
     last four characters.
   - Owners and admins can connect, replace or disconnect the key. Viewers can only look.
   - Langfuse is on the same page, under **Request access**. The RouterPlus team sets it up
     with you.
2. In Braintrust, make sure that you have one of these sources to search on:

| Source | What the search uses |
|---|---|
| An **experiment** that ran on a dataset | The dataset is the test cases. The experiment's `metadata.model` is the baseline. The prompt comes from `metadata.prompt_id` or `metadata.prompt_slug`, or is the project's only prompt. |
| A **dataset** | The rows are the test cases. You select the prompt and, optionally, a baseline model. Without a baseline, the strongest candidate becomes the baseline. |
| **Production logs** | The logged calls of the last 1, 7 or 30 days. Each call is a test case, with the production reply as the expected turn. A candidate scores on whether its reply is as good as the production reply. The baseline is the model that most logged calls used, unless you select one. |

The baseline must be a catalog model or one of your own deployed endpoints (`tm/...`). A search
takes up to 2,000 cases.

For an experiment or a dataset, these parts of the rows are used:

| Braintrust | Used as |
|---|---|
| A row's `input.messages` | One test of the next assistant turn. |
| The project's scorers | Quality. A case's score is the mean of its scorers. |
| A row's `tags` | The slices in a result's detail view. |
| A row's `metadata.conversation_id` and `metadata.goal` | Keep the turns of one conversation together. Simulated customers use the goal. |

Code scorers run in Braintrust. Prompt scorers (LLM graders) run through the gateway and bill
your credits.

## Start a search

1. Click **New search** at the bottom of the searches list.
2. Select the source and name the search.
3. Set the **search budget**: the most that the search spends on model calls, from $5 to $100.

Model calls bill your credits while the search runs. They use a managed key that RouterPlus
makes for the search. To see the key, turn on **Show playground keys** on `/console/keys`. The
key has a **managed** tag. Viewers cannot start or change a search.

The search runs in stages. The budget pays for them in this order:

1. **Shortlist.** Public benchmarks select the open models to try: the Pareto frontier of
   benchmark score against price. The search tries each model on up to three hosts: the
   cheapest host, the host with the highest precision, and the model maker's own host.
2. **Pilot.** Every candidate answers a sample of the conversations.
3. **Dev.** The leaders answer all the dev conversations.
4. **Mixes.** The search builds cascades, routers and votes from the saved answers. A search
   agent proposes mixes to try. Mixes cost nothing extra.
5. **Held-out.** The finalists, and a fusion of the best models, answer conversations that the
   search never saw. The winner comes from these.
6. **Customers.** The finalists talk to a simulated customer who has the goal of a held-out
   conversation.

| Action | Effect |
|---|---|
| Budget reached | The search stops and keeps its results. |
| **Add $25** | Continues the search from where it stopped. |
| **Stop** | Ends the search and keeps its results. |

If a Braintrust read fails while the search loads its cases, RouterPlus tries 3 more times,
after 2 s, 10 s and 30 s. Then it stops the search. A stopped search shows the reason. Fix the
cause (for example, connect the key again) and add budget to continue.

## Read the results

The chart plots each candidate's cost against its score. A line shows the best-value
frontier. The table below the chart has these columns:

| Column | Meaning |
|---|---|
| Model | The model, or `tm-mix-N` for a mix. Open a row to see what the mix does. |
| Provider | The host that served the model, with its precision when the host states it. |
| Score | The mean scorer score, in percent. |
| $ / 1k req | The measured cost of 1,000 requests like your cases. |
| p50 | The median time to a full answer. |
| TTFT | The median time to the first token. |
| Out tok | Output tokens per answer. |
| Status | The stage that scored the row: `pilot`, `dev` or held-out. |

The ★ row is the winner. It is the cheapest finalist that meets all three rules:

- It costs less than the baseline.
- It is not worse than the baseline on a paired test over the same held-out cases.
- It scores within 2 points of the baseline.

When two rows cost the same, the row with the lower median latency wins. Rows marked `dev` or
`pilot` scored on fewer conversations and cannot win. If no row meets the rules, no row has a
star.

Open a row to see the cases where it did worse than the baseline, side by side, and its score
for each tag.

## Deploy the result

**Deploy** makes a result callable as `tm/<name>-v1`. The name has lowercase letters, digits
and dashes, 40 characters at most:

```bash
curl https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model": "tm/billing-agent-v1", "messages": [{"role": "user", "content": "Hi"}]}'
```

- Only keys of your organization can call it.
- RouterPlus bills each model call that the endpoint makes as a direct call to that model.
  `usage.cost` in the response is the sum.
- Send it on `POST /v1/chat/completions`. With `"stream": true`, it returns the finished answer
  as one chunk, then usage.
- A second deploy makes a new version (`-v2`). Earlier versions continue to work.

[Endpoints](https://demo.routerplus.com/endpoints) lists everything that your organization deployed, with
the search that each result came from. **Try it** sends one held-out case of that search to the
endpoint, and bills your credits like any call.

## What is stored

A search stores its cases and the candidates' answers, for your organization only, until
you delete the search. A deployed model id continues to work after you delete its search.

---

Source: https://demo.routerplus.com/docs/broadcast.md

# Broadcast to storage and warehouses

Broadcast sends a trace of each request to the destinations you configure. This page covers
the storage and warehouse kinds: `s3`, `azure_blob`, `bigquery` and `snowflake`. In the console,
open Build > Exports and alerts.

The routes and the `sampling`, `privacy` and `filter` fields are the same for every kind. See
[Broadcast destinations](api-management.md#broadcast-destinations).

Each trace is one JSON object, after your privacy rules apply. A trace leaves out prompt and
completion content unless `privacy.include_content` is `true`.

## Delivery

- The gateway sends traces in batches of up to 50, at least every 2 seconds.
- Each request has a 10-second timeout.
- The gateway retries a failed batch up to 3 times, after 0.5, 1 and 2 seconds.
- It retries on a `5xx`, `408` or `429` status, or a network error. It does not retry other
  errors.
- A retry sends the same batch with the same batch id. Storage writes the same object again.
  BigQuery and Snowflake drop the duplicate rows. See each kind below.

`POST /v1/broadcast/destinations/:id/test` sends one sample trace. It returns `{ok, status}`.
On a failure it returns `{ok: false, status?, error}`, with the reason in `error`.

## Object layout (`s3` and `azure_blob`)

Each batch is one object. Each line of the object is one trace (NDJSON).

```
<prefix>/dt=YYYY-MM-DD/hour=HH/<batch-id>.ndjson
<prefix>/dt=YYYY-MM-DD/hour=HH/<batch-id>.ndjson.gz    (with "gzip": true)
```

- The date and hour are the UTC time at which the gateway sent the batch.
- `<batch-id>` is a UUID. The same traces always give the same id.
- With no `prefix`, the key starts at `dt=`.
- A plain object has `Content-Type: application/x-ndjson`. A gzip object has
  `Content-Type: application/gzip`.

The `dt=` and `hour=` folders are Hive-style partitions. Athena, BigQuery external tables,
Snowflake external stages and Spark read them as columns.

## Amazon S3 and S3-compatible storage (`s3`)

```json
{
  "kind": "s3",
  "config": {
    "bucket": "acme-llm-traces",
    "region": "us-east-1",
    "prefix": "routerplus",
    "access_key_id": "AKIA...",
    "secret_access_key": "...",
    "gzip": true
  }
}
```

| Field | Rule |
|---|---|
| `bucket` | Required. An S3 bucket name. |
| `region` | Required. The bucket's region. Use `auto` for Cloudflare R2. |
| `prefix` | Optional. Folders before `dt=`. The gateway removes a leading and a trailing `/`. |
| `endpoint` | Optional. The origin of an S3-compatible store, such as `https://<account>.r2.cloudflarestorage.com`. https only, except `http://localhost` for MinIO. No path. |
| `access_key_id` | Required. |
| `secret_access_key` | Required. Secret. |
| `session_token` | Optional. For temporary credentials. Secret. |
| `gzip` | Optional. `true` compresses each object. Default `false`. |

- Requests use AWS Signature Version 4 with a signed payload hash.
- With no `endpoint`, the URL is `https://<bucket>.s3.<region>.amazonaws.com/<key>`.
- With an `endpoint`, or a bucket name with a dot, the URL is path style:
  `<endpoint>/<bucket>/<key>`.
- The access key needs `s3:PutObject` on `arn:aws:s3:::<bucket>/<prefix>/*`. It needs no read
  access.

## Azure Blob Storage (`azure_blob`)

```json
{
  "kind": "azure_blob",
  "config": { "account": "acmetraces", "container": "routerplus", "prefix": "prod", "sas_token": "sv=2022-11-02&ss=b&srt=o&sp=cw&sig=..." }
}
```

| Field | Rule |
|---|---|
| `account` | Required. The storage account name. |
| `container` | Required. The container must exist. |
| `prefix` | Optional. Folders before `dt=`. |
| `sas_token` | A SAS token with create and write permission. The gateway removes a leading `?`. Secret. |
| `account_key` | The storage account key (base64). The gateway signs requests with Shared Key. Secret. |
| `gzip` | Optional. `true` compresses each object. Default `false`. |

Send one of `sas_token` and `account_key`. Each batch is a Put Blob request with
`x-ms-blob-type: BlockBlob`.

## Google BigQuery (`bigquery`)

```json
{
  "kind": "bigquery",
  "config": {
    "project_id": "acme-analytics",
    "dataset": "llm",
    "table": "routerplus_events",
    "service_account_json": "{\"client_email\": \"...\", \"private_key\": \"-----BEGIN PRIVATE KEY-----\\n...\", \"token_uri\": \"https://oauth2.googleapis.com/token\"}"
  }
}
```

| Field | Rule |
|---|---|
| `project_id` | Required. |
| `dataset` | Required. |
| `table` | Required. The table must exist. |
| `service_account_json` | Required. The service account key file, as a string or an object. The gateway keeps `client_email`, `private_key`, `token_uri` and `private_key_id`. Secret. |

- The service account needs `bigquery.tables.updateData` on the table. The role
  `roles/bigquery.dataEditor` on the dataset has it.
- The gateway gets an access token with the
  `https://www.googleapis.com/auth/bigquery.insertdata` scope. It uses the token again until one
  minute before it expires.
- The gateway inserts rows with `tabledata.insertAll`. Each row's `insertId` is the event
  id, so BigQuery drops a row that arrives twice.
- A row that BigQuery rejects fails its batch. The gateway logs the row index and the reason.

Create the table:

```sql
CREATE TABLE `acme-analytics.llm.routerplus_events` (
  event_id STRING NOT NULL,      -- the attempt id (generation_id), else the request id
  received_at TIMESTAMP NOT NULL, -- when the gateway sent the batch
  event JSON NOT NULL             -- the trace
)
PARTITION BY DATE(received_at);
```

Query it:

```sql
SELECT JSON_VALUE(event, '$.model') AS model, SUM(CAST(JSON_VALUE(event, '$.cost_usd') AS FLOAT64)) AS cost_usd
FROM `acme-analytics.llm.routerplus_events`
WHERE DATE(received_at) = CURRENT_DATE()
GROUP BY model;
```

## Snowflake (`snowflake`)

```json
{
  "kind": "snowflake",
  "config": {
    "account": "xy12345.us-east-1",
    "user": "ROUTERPLUS_LOADER",
    "private_key_pem": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----",
    "database": "ANALYTICS",
    "schema": "LLM",
    "table": "ROUTERPLUS_EVENTS",
    "warehouse": "LOAD_WH",
    "role": "ROUTERPLUS_LOADER"
  }
}
```

| Field | Rule |
|---|---|
| `account` | Required. The account identifier from your Snowflake URL, such as `xy12345.us-east-1` or `myorg-myaccount`. |
| `user` | Required. |
| `private_key_pem` | Required. An unencrypted RSA private key in PEM (PKCS#8 or PKCS#1). Secret. |
| `database`, `schema`, `table` | Required. Unquoted identifiers: letters, digits, `_` and `$`. |
| `warehouse`, `role` | Optional. Default: the user's defaults. |

The gateway signs in with key-pair authentication. It sends one SQL API request
(`POST /api/v2/statements`) per batch: a parameterized multi-row insert with the traces as
bindings.

```sql
INSERT INTO ANALYTICS.LLM.ROUTERPLUS_EVENTS (event) SELECT PARSE_JSON(column1) FROM VALUES (?), (?), ...
```

Each request has a `requestId` from the batch id. A retry sets `retry=true`. Snowflake then
does not run the same batch twice.

Set up the user, the key and the table:

```sql
-- Generate the key pair (shell):
--   openssl genrsa 2048 | openssl pkcs8 -topk8 -nocrypt -out routerplus_key.p8
--   openssl rsa -in routerplus_key.p8 -pubout -out routerplus_key.pub
CREATE ROLE ROUTERPLUS_LOADER;
CREATE USER ROUTERPLUS_LOADER TYPE = SERVICE DEFAULT_ROLE = ROUTERPLUS_LOADER DEFAULT_WAREHOUSE = LOAD_WH
  RSA_PUBLIC_KEY = 'MIIBIjANBgkqh...';   -- routerplus_key.pub without the BEGIN and END lines
GRANT ROLE ROUTERPLUS_LOADER TO USER ROUTERPLUS_LOADER;

CREATE TABLE ANALYTICS.LLM.ROUTERPLUS_EVENTS (
  event VARIANT NOT NULL,
  received_at TIMESTAMP_LTZ NOT NULL DEFAULT CURRENT_TIMESTAMP()
);

GRANT USAGE ON WAREHOUSE LOAD_WH TO ROLE ROUTERPLUS_LOADER;
GRANT USAGE ON DATABASE ANALYTICS TO ROLE ROUTERPLUS_LOADER;
GRANT USAGE ON SCHEMA ANALYTICS.LLM TO ROLE ROUTERPLUS_LOADER;
GRANT INSERT ON TABLE ANALYTICS.LLM.ROUTERPLUS_EVENTS TO ROLE ROUTERPLUS_LOADER;
```

Query it:

```sql
SELECT event:model::string AS model, SUM(event:cost_usd::float) AS cost_usd
FROM ANALYTICS.LLM.ROUTERPLUS_EVENTS
WHERE received_at >= DATEADD(day, -1, CURRENT_TIMESTAMP())
GROUP BY model;
```

## Secrets

A read returns each secret as `"__secret__"`. To keep a stored secret on an update, send
`"__secret__"` in its place.

| `kind` | Secret fields |
|---|---|
| `s3` | `secret_access_key`, `session_token` |
| `azure_blob` | `sas_token`, `account_key` |
| `bigquery` | `service_account_json` |
| `snowflake` | `private_key_pem` |

---

Source: https://demo.routerplus.com/docs/slack-agents.md

# Slack agents

A Slack agent connects your own Slack app to RouterPlus. People mention the app in a channel or
send it a direct message. RouterPlus answers in the same thread, with the model you select.

Each agent has its own RouterPlus inference key. Every answer is an ordinary request on that
key. Your guardrails, budgets, routing and billing apply.

## Set up an agent

1. Create a Slack app at api.slack.com/apps:
   - Select "From scratch" and install the app in your workspace.
   - Copy the bot token (`xoxb-...`) from "OAuth & Permissions".
   - Copy the signing secret from "Basic Information".
2. Create the agent with a management key (`Authorization: Bearer tm_mk_...`):

```
POST /v1/slack/agents
```

```json
{
  "slug": "helpdesk",
  "bot_token": "xoxb-...",
  "signing_secret": "8f74...",
  "model": "gpt-6-luna",
  "system_prompt": "You answer questions about our product.",
  "allowed_channel_ids": ["C0123ABCD"],
  "allowed_user_ids": [],
  "tools": { "server": [{ "type": "web_fetch" }], "mcp_servers": ["docs"] },
  "max_cost_usd_per_day": 10
}
```

3. Get the app manifest:

```
GET /v1/slack/agents/{id}/manifest
```

4. In your Slack app, open "App Manifest", paste the manifest and save. Slack checks the events
   URL. When Slack asks, install the app again to give it the new scopes.
5. Invite the app to a channel (`/invite @helpdesk`). Mention it, or send it a direct message.

`POST` returns `201` with the agent. Slack sends events to the agent's `events_url`:
`https://<gateway>/v1/slack/events/{id}`.

## Fields

| Field | Rule |
|---|---|
| `slug` | Required. 1 to 40 of `a-z`, `0-9` and `-`, starting with a letter or digit. Unique in your org. |
| `bot_token` | Required. The bot token, `xoxb-...`. Stored encrypted. Never returned. |
| `signing_secret` | Required. The app's signing secret. Stored encrypted. Never returned. |
| `model` | Required. A model id, a preset (`@preset/<slug>`) or `routerplus/auto`. |
| `system_prompt` | Optional. Up to 8,000 characters. |
| `allowed_channel_ids` | Optional. Channel ids where the agent answers. Empty (the default) means every channel that the app is in. This list does not limit direct messages. |
| `allowed_user_ids` | Optional. User ids that the agent answers, in channels and in direct messages. Empty (the default) means anyone in the workspace. |
| `tools.server` | Optional. Server tools: `{ "type": "web_fetch" }` with optional `allowed_domains`, `max_bytes` and `max_calls`, and `{ "type": "advisor", "model": "..." }`. Not with `routerplus/auto`. See [Server tools](server-tools.md). |
| `tools.mcp_servers` | Optional. Slugs of your MCP servers. The agent calls them through the [MCP gateway](mcp-gateway.md) with its own key, so their tool policies apply. |
| `max_cost_usd_per_day` | Optional. The most that the agent's key spends in one UTC day. `null` (the default) means no daily limit. |
| `workspace_id`, `principal_id` | Optional, on create only. The workspace and principal of the agent's key. Default: your default workspace. |
| `slack_team_id`, `slack_app_id` | Optional. When set, the agent ignores events from another Slack workspace or app. |
| `enabled` | Optional. Default `true`. A disabled agent acknowledges events but does not answer. |

## Manage agents

| Route | What it does |
|---|---|
| `GET /v1/slack/agents` | Lists your agents. |
| `POST /v1/slack/agents` | Creates an agent and its key. |
| `GET /v1/slack/agents/{id}` | Reads one agent. |
| `PATCH /v1/slack/agents/{id}` | Changes the fields you send. Send `bot_token` or `signing_secret` to rotate it. |
| `DELETE /v1/slack/agents/{id}` | Deletes the agent and its thread memory, and disables its key. |
| `GET /v1/slack/agents/{id}/manifest` | The Slack app manifest for the agent. |

- Every route also works under `/api/v1`.
- Each change writes a row to the audit log (`GET /v1/audit`).
- Responses show `has_bot_token` and `has_signing_secret`, never the values.
- The agent's key is a normal inference key with the name `slack:<slug>`. It shows in
  `GET /v1/keys` and in activity. To stop all spend immediately, disable the key or the agent.

## Scopes and events

The manifest asks for these bot scopes:

| Scope | Use |
|---|---|
| `app_mentions:read` | Receive mentions of the app. |
| `chat:write` | Post answers. |
| `im:history`, `im:read`, `im:write` | Receive and answer direct messages. |
| `channels:history` | Read messages in threads of public channels. |

It subscribes to two bot events: `app_mention` and `message.im`.

## How the agent answers

- A mention in a channel gets an answer in the thread of that message.
- A direct message gets an answer in the direct message. Direct messages outside a thread are
  one conversation.
- The agent does not answer bots (itself included), edits, deletions, or people who are not on
  the allow lists.
- RouterPlus removes the app mention from the text before the model sees it.
- The model sees the system prompt, the thread's earlier messages (up to 30), and the new
  message.
- When the agent's key has reached `max_cost_usd_per_day` today, the agent posts a short
  notice and does not call the model.
- When the model call fails, the agent posts a short message, for example "Sorry, I could not
  answer that. Try again later."

## Security

- RouterPlus checks Slack's signature on every request. The signature is an HMAC-SHA256 of
  the timestamp and the body, with your signing secret. A request without a valid signature gets
  `401`.
- A request with a timestamp more than 5 minutes from now gets `401`. Thus nobody can send a
  captured request again.
- Slack sends an event again when it gets no acknowledgement in time. RouterPlus answers each
  event id one time.
- RouterPlus acknowledges each event immediately and answers in the background.

## What is stored

| Data | How long |
|---|---|
| Bot token, signing secret, agent key | Encrypted, until you delete the agent. |
| Thread memory: the last 30 messages of each thread, from people and from the agent | 7 days after the thread's last message, then deleted. |
| Event ids, to answer each event once | 1 day. |

RouterPlus keeps the model requests like any other request on your key. See
[Data policy](data-policy.md).

---

Source: https://demo.routerplus.com/docs/guardrails.md

# Guardrails

A guardrail is a set of rules for requests. You assign it to the organization, a workspace, a
member or a key. Every guardrail that covers a request applies. When two guardrails disagree,
the stricter rule wins.

Guardrails apply to `POST /v1/chat/completions`, `/v1/messages`, `/v1/responses`,
`/v1/images/generations`, `/v1/videos` and `/v1/decisions`. You manage them with the
[management API](api-management.md).

In the console, open Governance > Guardrails for the rules. Open Governance > Budgets for the
spending limits.

## Create and assign a guardrail

1. Create the guardrail with a management key:

   ```bash
   curl https://demo-api.routerplus.com/v1/guardrails \
     -H "Authorization: Bearer $TM_MANAGEMENT_KEY" \
     -H "Content-Type: application/json" \
     -d '{"name": "support-bot",
          "blocked_models": ["openai/gpt-oss-20b"],
          "content_rules": [{"kind": "pii", "preset": "email", "action": "redact"}]}'
   ```

   The response is `201`. `data` holds the guardrail and its `guardrail_id`.
2. Assign the guardrail. It has no effect until you do. `target_kind` is the organization, a
   workspace, a member or a key:

   ```bash
   curl https://demo-api.routerplus.com/v1/guardrails/$GUARDRAIL_ID/assignments \
     -H "Authorization: Bearer $TM_MANAGEMENT_KEY" \
     -H "Content-Type: application/json" \
     -d '{"target_kind": "org", "target_id": "'$ORG_ID'"}'
   ```

A change applies within 5 seconds. With the guardrail above:

- A call to `openai/gpt-oss-20b` returns `403` with `error.code` `guardrail_blocked`.
- A call that contains an email address goes to the provider with `[REDACTED:email]` in its
  place. The response carries `x-tm-guardrail-actions: redacted`.

## Rules

Unknown fields are refused.

| Field | Type | Effect |
|---|---|---|
| `allowed_models` | string[] | Only these models. `*` matches any run of characters, `?` one character. Empty means every model. |
| `blocked_models` | string[] | Never these models. |
| `allowed_providers` | string[] | Only these providers. Empty means every provider. |
| `blocked_providers` | string[] | Never these providers. |
| `zdr` | boolean | Only providers that keep no prompts or completions. |
| `data_collection` | `"deny"` | Only providers that do not train on or keep request data. |
| `allowed_regions` | string[] | Only providers that process requests in these regions (`us`, `eu`, `in`, ...). `global` allows all. |
| `blocked_infrastructure` | string[] | Never providers that run on these clouds (`aws`, `gcp`, `azure`, ...). |
| `budgets` | object[] | Spending limits. See [Budgets](#budgets). |
| `limit_usd`, `reset_interval`, `include_byok_in_budgets` | | One budget in a short form: a USD limit, its period, and whether BYOK spend counts. |
| `rpm`, `tpm` | integer | Requests and tokens per minute for each key the guardrail covers. |
| `max_output_tokens` | integer | The most output a request may ask for. RouterPlus lowers a larger limit to this value. |
| `server_tools` | `"allow"` or `"block"` | `block` refuses requests that ask the provider to run tools (web search, web fetch, code execution). |
| `content_rules` | object[] | Rules on request text. See [Content rules](#content-rules). |
| `on_scan_limit` | `"allow"` or `"block"` | What to do with a request too long to scan in full. RouterPlus scans the first 2,000,000 characters. Default `allow`. |
| `overridable` | `false` | Makes the guardrail a hard limit. See [Grants](#grants). |

A filter on a provider fact leaves out each provider for which RouterPlus does not know that
fact. For example, `zdr: true` leaves out a provider that does not publish its retention.

### Regions and infrastructure

- RouterPlus records a provider's regions and infrastructure only from the provider's published
  terms: its privacy policy, data processing terms, subprocessor list or docs.
- `allowed_regions` keeps only providers whose terms state where they process requests.
- RouterPlus does not record a region that a provider offers only as an option RouterPlus does
  not use.
- `blocked_infrastructure` leaves out a provider that names a blocked cloud for serving or
  hosting.
- `GET /v1/models/<model>/endpoints` shows each provider's `regions` and `infrastructure` under
  `data_policy`.

### Budgets

```json
{ "budgets": [
  { "id": "monthly", "limit_usd": 500, "period": "month" },
  { "id": "opus", "limit_usd": 50, "period": "week", "model": "claude-opus-*" },
  { "id": "team-pool", "limit_usd": 2000, "period": "month", "shared": true, "include_byok": true }
] }
```

- `period` is `day`, `week` (from Monday), `month` or `lifetime`. Periods start at 00:00 UTC. A
  budget with no period is `lifetime`.
- `model` and `provider` count only the spend on that model or provider.
- `include_byok` also counts the provider cost of requests on your own provider keys.
- `shared: true` makes one budget for everything the guardrail is assigned to. Without it, each
  assignment has its own budget. For example, a guardrail on a workspace counts the spend of that
  workspace.
- A key also has its own limit: `limit` with `limit_reset` on the key.

RouterPlus refuses a request when its budget is used up. Requests that are already running
finish. Spend can go past the limit by the cost of those requests.

### Content rules

```json
{ "content_rules": [
  { "kind": "pii", "preset": "credit_card", "action": "block" },
  { "kind": "pii", "preset": "email", "action": "redact" },
  { "kind": "regex", "pattern": "PROJ-[0-9]{4}", "action": "redact", "label": "project_code" },
  { "kind": "injection", "action": "block" }
] }
```

- `pii` presets: `email`, `phone`, `ssn`, `credit_card`, `iban`, `ip_address`, `credential`,
  `aws_access_key`, `private_key`, `jwt`, `us_passport`.
- `regex`: up to 512 characters, with flags `i`, `m`, `s` and `u`. A pattern with lookarounds,
  backreferences or nested quantifiers returns `invalid_regex_pattern`.
- `injection` finds prompt-injection and jailbreak patterns in user text and tool results.
- `action`:
  - `redact` puts `[REDACTED:<label>]` in place of the match before RouterPlus sends the request.
  - `block` refuses the request.
  - `flag` records the match and sends the request unchanged.
- When two guardrails have the same pattern with different actions, `block` wins over `redact`,
  and `redact` wins over `flag`.

Content rules read message text, tool-call arguments and tool results. They do not read images
or tool definitions. RouterPlus does not store the matched text. The record holds the rule, the
count and the place in the request where it matched.

## Errors

| Status | `error_type` | When |
|---|---|---|
| 401 | `key_expired` | The key's `expires_at` has passed. |
| 403 | `ip_not_allowed` | The key has an IP allowlist and the request comes from another address. |
| 403 | `guardrail_blocked` | A model, server tool or content rule refused the request. The message names the guardrail and its id, or the content rule ids. |
| 403 | `budget_exceeded` | A budget is used up. `x-tm-cap-reset` gives the time it resets. |
| 429 | `rate_limit` | A guardrail's `rpm` or `tpm` is reached. |
| 404 | `model_unavailable` | No provider of the model meets the guardrails. |

The `401` and `403` refusals carry `x-should-retry: false`. Do not retry them.

Response headers:

- `x-tm-guardrails`: the ids of the guardrails that applied.
- `x-tm-guardrail-actions`: `redacted` or `flagged`, when a content rule acted.
- `x-tm-guardrails: unavailable`: RouterPlus could not read your guardrails, so the request ran
  without them.

## Grants

A grant is a time-boxed exception to one rule. An approved grant applies from `starts_at` until
`expires_at`. After `expires_at`, the base rule applies again.

| Grant type | What it changes |
|---|---|
| `budget` | Raises one budget of a guardrail to a higher `limit_usd`. |
| `key_limit` | Raises a key's own limit to a higher `limit_usd`. |
| `guardrail` | Lets one model (`allow_model`, an id or pattern) pass that guardrail's `allowed_models` and `blocked_models`. Every other guardrail still applies. |

- A grant lasts at most 90 days.
- A budget or key limit grant must raise the limit. When two grants cover the same limit, the higher one applies.
- A guardrail with `overridable: false` is a hard limit. It takes no grant. Its organization-level budgets also cap every budget grant for the same period.
- A management key can request, approve, reject, cancel and list grants. An API key can request a grant for itself, and list, read and cancel the grants it requested.
- RouterPlus sends a notice when a grant is requested, decided or cancelled, and one day before it expires.

In the console, open Access > Grants. For the endpoints, see [Grants](api-management.md#grants)
in the management API.

## Activity

`GET /v1/guardrails/events` lists the requests that a guardrail blocked, redacted or flagged. In
the console, open Governance > Guardrail events.

---

Source: https://demo.routerplus.com/docs/limits.md

# Rate limits & spend caps

RouterPlus checks these limits before a request reaches a provider:

- the key's requests-per-minute (RPM) limit
- the organization's token and concurrency limits
- the provider pool's capacity
- guardrail budgets and key spend limits
- the monthly spend caps
- the balance

Each limit refuses with a typed error. The error names its class in the body and in
`x-tm-error-code`. RouterPlus does not slow down or queue a request without telling you. A
refused request costs nothing.

- [Errors](/docs/errors) lists the error shapes.
- [Limits and capacity](/docs/admission) covers pools, fairness and outages.

In the console, open Billing > Rate limits.

## Per-key requests per minute

Every key has an RPM limit over a rolling 60-second window. All gateway instances count in one
shared store. The count for a request can stay in the window for up to 61 seconds.

| Key | RPM |
|---|---|
| First trial key shown after verified browser sign-in | 20 |
| Key created in the console or with the identity API | 300 |
| The managed Playground key of the console | 60 |

### Trial rate

- Until the organization buys credit, every customer key and the organization as a whole run at
  20 requests per minute. Credit that RouterPlus granted counts as bought credit. More keys do
  not add more requests.
- The first purchase removes the trial rate within seconds. No key changes.
- The trial rate does not apply to requests served only by your own provider keys (BYOK). It
  does not apply to the managed Playground key.
- Past card-verification bonuses are promotional credit. They do not count as purchases.

An organization that joined through a RouterPlus signup invitation (for example, at a
hackathon) counts as paid from the start. By default, the organization and each key get these
limits: 600 requests per minute, 10,000,000 tokens per minute and 32 requests in flight.

`GET /v1/limits` shows the values that apply to your key.

RouterPlus can also limit an organization on trial credit when it detects unusual usage. The
request gets `429 usage_limited` (with `retry-after`) or `403 usage_restricted`. The message asks
you to contact support. Support can review the account. A purchase usually removes the limit.

### The rate-limit refusal

Above the limit, RouterPlus refuses the request before any reservation or provider work:

```
HTTP/1.1 429 Too Many Requests
retry-after: 23
x-tm-error-code: rate_limit
x-tm-error-origin: gateway_admission
x-tm-limit-scope: key
x-tm-limit-kind: rpm
x-tm-limit-id: key:<key id>
```

```json
{
  "error": {
    "code": "rate_limit_exceeded",
    "message": "configured quota exhausted",
    "type": "requests",
    "metadata": {
      "error_type": "rate_limit",
      "origin": "gateway_admission",
      "limit_scope": "key",
      "limit_kind": "rpm",
      "limit_id": "key:<key id>",
      "retryable": true,
      "retry_at": "2026-09-24T10:14:04.000Z"
    }
  },
  "request_id": "..."
}
```

| Surface | `code` | `type` |
|---|---|---|
| `/v1/chat/completions`, `/v1/responses` | `rate_limit_exceeded`, the OpenAI code that OpenAI SDKs and coding agents read | `requests`, or `tokens` for a token limit |
| `/v1/decisions`, images, video | `rate_limit` | `invalid_request_error` |
| Anthropic surface | A native `rate_limit_error` with the same `error_type` and `metadata`, so the Anthropic SDK's backoff runs | |

`error_type` and `x-tm-error-code` are always `rate_limit`.

### `retry-after`

- A rate limit's `retry-after` is in whole seconds, 1 to 59. The `usage_limited` refusal
  sends 60 or 3,600.
- On an RPM or token limit, it is the time until the rolling window has room for the request.
- On a limit on requests at once, it is 10 seconds. A slot becomes free when one of your running
  requests ends. No window can predict that time.
- `metadata.retry_at` is the same instant.
- Other requests can take the room first. Then a retry can get another 429 with the rest of the
  wait. Wait for `retry-after`, then retry.

> [!NOTE]
> You cannot raise a key's RPM ceiling yourself. On [`/console/limits`](https://demo.routerplus.com/console/limits) you can only lower a key's RPM and set its other limits.

### Provider rate limits

A provider can rate-limit RouterPlus upstream. Then RouterPlus fails over to another deployment
and pauses that provider's pool. You see a provider `rate_limit`
(`x-tm-error-origin: upstream`) only when every candidate was rate-limited. Its `retry-after` is
the provider's value, up to 59 seconds. A provider 429 that names no wait gets 10 seconds.

## Organization tokens and concurrency

Every organization and every key also has a tokens-per-minute limit and a concurrency limit, in
the same shared window:

| Scope | RPM | Tokens per minute | Concurrent requests |
|---|---|---|---|
| Organization (all keys together) | 300 (20 until the first purchase) | 1,000,000 | 8 |
| Key | its ceiling above | 1,000,000 | 8 |

- These are the defaults. Set your own on [`/console/limits`](https://demo.routerplus.com/console/limits)
  or with `POST /api/admission-limits`.
- A key's RPM cannot go above its ceiling. You can add a limit for one model on top.
- Under a contract, RouterPlus can replace these defaults and the keys' RPM ceiling for
  your organization. A limit that you set still applies and can only lower them.
- At admission, tokens are an estimate: one token for each four UTF-8 bytes of the request,
  rounded up, plus a media allowance, plus the output limit (at most 32,768 tokens of it). When
  the request completes, observed usage replaces the estimate. The balance hold in
  [The worst-case reserve](#the-worst-case-reserve) counts one token for each byte.
- A refusal has the 429 shape above, with `limit_scope` `org` or `key` and `limit_kind` `tpm` or
  `concurrency`. On the OpenAI surface, a token-limit refusal has `"type": "tokens"`. A refusal
  for requests at once sends `retry-after: 10`.

Every admitted response has `x-tm-remaining-rpm` and `x-tm-remaining-tpm`. Each is the smallest
remaining amount across the scopes that the request claimed, at that moment.
`GET /v1/limits?model=<id>` on the gateway returns the effective limits for your key.

### Provider pools

Each house deployment has a declared pool. One organization can use at most half of a pool's usable
capacity, or three quarters on the decision-model pools. When the pool is full, you
get a 429 with `x-tm-error-origin: upstream_quota` and `x-tm-limit-scope: pool`.
[Limits and capacity](/docs/admission) covers workspace and principal scopes.

### Dedicated endpoints

A [dedicated endpoint](/docs/dedicated-endpoints) has its own limits. For its traffic, they
replace the defaults above.

- The limits can include a requests-per-second limit: a count per second, or a
  [paced limit](/docs/dedicated-endpoints#the-paced-limit) that holds a short burst in a queue.
- A paced request that waited has `x-tm-queue-ms`, the wait in milliseconds.
- Above the hard rate, the refusal is `x-tm-limit-kind: rps`. When the wait would pass its
  bound, it is `rps_queue`.
- Refusals have `x-tm-limit-scope: endpoint`.

## Budgets, key limits and grants

A guardrail budget limits the spend of its targets for a day, week, month or lifetime. A key's own
spend limit (`limit` and `limit_reset` on the key) limits one key. A used-up budget or
key limit returns `403 budget_exceeded`. To see every budget and key limit with the spend of
this period, open Governance > Budgets in the console.

A grant raises a budget or a key limit for a set time. Then the base value applies again. To
request, approve, reject or cancel a grant, open Access > Grants in the console. The
[management API](/docs/api-management#grants) has the same actions.

Granted limits show in these places:

- `x-tm-grants` on the response: the ids of the active grants that applied to the request. A
  refusal also has it.
- The `budget_exceeded` message. It adds "(raised by grant <id>)" when a grant raised the limit.
- `GET /v1/guardrails/:id/status`: `limit_usd` is the granted limit, `base_limit_usd` the
  guardrail's own limit, and `grant_id` the grant.
- Key reads (`GET /v1/keys`, `GET /v1/keys/:id`) and `GET /v1/key`: `limit` is the granted
  limit, `base_limit` the key's own limit, and `grant_id` the grant.
- Budget alerts. They count against the granted limit.

A guardrail with `"overridable": false` takes no grant. On the org, its budgets limit every
grant.

## Monthly spend caps

You can set two optional hard caps in the console:

- An **org-wide** cap on every key, in Billing > Credits
  ([https://demo.routerplus.com/console/billing](https://demo.routerplus.com/console/billing)).
- A **per-key** cap on one key, with **Set cap** in Access > API keys
  ([https://demo.routerplus.com/console/keys](https://demo.routerplus.com/console/keys)).

Rules:

- Both caps run on the **UTC calendar month** and reset at the first instant of the next month.
- They can overlap. The strictest one wins.
- In the console, a blank cap field changes nothing. A cap of 0 removes the cap.
- The console marks keys near their cap. It shows each key's month-to-date spend against
  its cap.

### How enforcement works

Before RouterPlus sends anything, it reserves a worst-case cost for the request. See
[The worst-case reserve](#the-worst-case-reserve). The cap check adds this month's settled
spend and the worst-case cost of every request in flight, this one included. If the sum is
above the cap, RouterPlus refuses the request before it costs anything. RouterPlus claims the
in-flight amount synchronously, so concurrent requests cannot pass the cap together.

The check is conservative. RouterPlus can refuse a request near the cap even when its settled
cost would have fit. A tight `max_tokens` lowers the worst-case reserve and gives you more room
near the cap.

### The cap refusal

A cap breach is the standard out-of-credits 429: `insufficient_quota`, a class that OpenAI SDKs
recognize. The message and a header give the reset time:

```
HTTP/1.1 429 Too Many Requests
x-tm-error-code: insufficient_quota
x-tm-cap-reset: 2026-10-01T00:00:00.000Z
x-should-retry: false
```

```json
{
  "error": {
    "code": "insufficient_quota",
    "type": "insufficient_quota",
    "message": "org monthly cap exceeded; resets 2026-10-01T00:00:00.000Z",
    "metadata": { "error_type": "insufficient_quota", "origin": "gateway_admission", "retryable": false }
  },
  "request_id": "..."
}
```

- The message names the scope of the cap that was reached: `org` or `key`.
- On the Anthropic surface, the body is a native `billing_error` with the same `error_type` and
  message.
- The refusal has no `retry-after`. `x-should-retry: false` tells the OpenAI and Anthropic SDKs
  and Claude Code not to retry.
- Do not retry before the reset time. Backoff does not change a hard cap.

## The worst-case reserve

The cap and balance checks use a pessimistic cost estimate. There is one exception: a request
without an output limit holds at most 32,768 output tokens. A longer answer is still charged in
full.

| Input | Value |
|---|---|
| Estimated input tokens | One per UTF-8 byte of the request body, plus 65,536 per image or document part. An image sent inline as base64 counts the 65,536 only, not its bytes. |
| Assumed output tokens | Your `max_tokens` (or `max_completion_tokens`), all of it. When omitted, at most 32,768, and the request is sent with the model's own maximum, fitted to its context window. More than the model's own `max_output_tokens` (32,768 when no maximum is declared) is a 400. |
| Images (`/v1/images/generations`) | Assumed output = `n` × the model's per-image ceiling. |
| Rounding | The reserve rounds **up**. Settlement rounds **down**. |

The reserve exists only while the request is in flight. At settlement, RouterPlus releases it and
records the observed cost: the usage that the provider reports, at the prices pinned when the
request was admitted. RouterPlus debits your balance and caps only for observed usage, with two
exceptions:

- An attempt can end before the provider reports usage, because you cancelled it or the
  provider failed during the answer. It is charged its input and the output already streamed to
  you.
- An attempt whose outcome the gateway could not observe (`unknown_after_crash`) settles at the
  reserved maximum, because the provider may have billed the call. See
  [Pricing & billing](/docs/pricing).

## Balance exhaustion

The org balance must cover the worst-case reserve of every request that the org has in flight,
this one included. When it cannot, the refusal is the same `insufficient_quota` 429, with a
different message:

```json
{
  "error": {
    "code": "insufficient_quota",
    "type": "insufficient_quota",
    "message": "insufficient marketplace credits for authorized house fallback",
    "metadata": { "error_type": "insufficient_quota", "origin": "gateway_admission", "retryable": false }
  },
  "request_id": "..."
}
```

- To tell the two `insufficient_quota` cases apart in code: only cap breaches have
  `x-tm-cap-reset`.
- A balance refusal has no `retry-after` and no `x-should-retry`. When your own requests in
  flight hold the balance, the balance becomes free as they end, and a retry can succeed.
  Otherwise, add credits.

> [!TIP]
> Signup alone does not add free credit. An invitation link from RouterPlus does. Buy credits in [Billing](https://demo.routerplus.com/console/billing). Eligible purchases may receive matching promotional credit. If calls return `insufficient_quota`, check your balance and spend caps there.

## Handling the two 429s

Both refusals are 429s, so the OpenAI SDK raises `RateLimitError` for both. Read `error_type`.
Retry a `rate_limit` after `retry-after`. Stop on `insufficient_quota`.

```python
import os, time
from openai import OpenAI, RateLimitError

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

def create(**kw):
    try:
        return client.chat.completions.create(**kw)
    except RateLimitError as e:
        kind = e.response.json()["error"]["metadata"]["error_type"]
        if kind == "rate_limit":
            time.sleep(int(e.response.headers.get("retry-after", "10")))
            return client.chat.completions.create(**kw)  # one retry
        raise  # insufficient_quota: retrying cannot help; add credits or raise the cap
```

```typescript
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://demo-api.routerplus.com/v1", apiKey: process.env.TM_API_KEY });

try {
  await client.chat.completions.create({
    model: "claude-haiku-5-5",
    messages: [{ role: "user", content: "hello" }],
  });
} catch (err) {
  if (err instanceof OpenAI.APIError && err.status === 429) {
    const kind = (err.error as { metadata?: { error_type?: string } })?.metadata?.error_type;
    if (kind === "rate_limit") {
      // wait Number(err.headers?.get("retry-after")) seconds, then retry
    } else {
      // insufficient_quota: stop; x-tm-cap-reset (cap breaches only) says when a cap reopens
      const capReset = err.headers?.get("x-tm-cap-reset");
    }
  }
}
```

## Other limits

- **Monthly spend caps only.** The monthly caps use the UTC calendar month and nothing shorter.
  For a daily or weekly limit, use a guardrail budget or a key limit with `limit_reset`.
- **No self-serve RPM raises.** Trial keys are 20 RPM. Console keys are 300, or 20 until the
  organization's first purchase. A limit that you set can only lower a key's RPM.
- **RouterPlus refuses sign-ups that look automated or abusive.** The page says that unusual
  activity was detected. If this happens to you, contact support. Existing accounts sign in
  without limit.
- **Request size:** a body over 10 MB gets a 413 `request_too_large`.
- **Image response size:** an upstream image response over 32 MiB is a 502 `gateway_error` and
  bills nothing. It is not a 413. Lower `n` or the quality.
- **An image request holds one admission slot for the whole render**, typically 10 to 60 s.
  The gateway sends no first byte before the image is ready.

---

Source: https://demo.routerplus.com/docs/admission.md

# Rate limits, capacity and outage behavior

This page is the full reference for admission: limit scopes, provider pools, fairness, and what
happens when a store is down. For per-key RPM, the org defaults and spend caps, see
[Rate limits & spend caps](/docs/limits). Pools attach to BYOK connections and routing policies.
See [BYOK setup](/docs/byok) and [routing policies](/docs/routing-policies).

All gateway instances count every limit on this page in one shared store. Adding instances
does not raise any limit.

## Manage limits

In the console, open Billing > Rate limits
([`/console/limits`](https://demo.routerplus.com/console/limits)). There you view or set gateway limits,
create provider pools and assign connections.

Gateway limits overlap. The strictest platform, org, workspace, principal, key, model and
provider-account limit that applies wins.

The limits API uses the site origin (`https://demo.routerplus.com`), a session cookie, a JSON content type
and same-origin protection on writes. Inference bearer keys cannot call it. Owners and admins
can write. Viewers can read.

| Endpoint | Purpose |
|---|---|
| `GET /api/admission-limits` | Lists your org's configured limits, every scope. |
| `POST /api/admission-limits` | Replaces one scope's limits. |
| `GET /api/quota-pools` | Lists the newest 100 org-owned provider pools. |
| `GET /api/quota-pools/{id}` | Reads one pool. |
| `POST /api/quota-pools` | Creates a pool. |
| `POST /api/quota-pools/{id}` | Replaces its limits and parent. |
| `POST /api/quota-pools/{id}/bind` | Assigns an owned connection to that pool. |

Example limit body:

```json
{"scope":"key","subject":"VIRTUAL_KEY_UUID","rpm":100,"tpm":200000,"concurrency":4,"outage_mode":"bounded_open"}
```

| Field | Rule |
|---|---|
| `scope` | `org`, `workspace`, `principal`, `key` or `model`. |
| `subject` | `org` for the org scope. The UUID for the workspace, principal and key scopes. The exact model ID for the model scope. |
| `rpm` | 1 to 1,000,000. |
| `tpm` | 1 to 1,000,000,000. |
| `concurrency` | 1 to 10,000. |
| `outage_mode` | `bounded_open` (the default) or `closed`. `closed` refuses new work while the counter store is down. A closed mode at any scope that applies wins. |

A numeric value that is null or left out inherits the default or the parent's limit. A key's
original RPM ceiling still applies: a larger key limit does not raise it.

### Defaults

| Scope | Default |
|---|---|
| Org, workspace, principal | 300 RPM, 1M TPM, 8 concurrent requests |
| Key | Its RPM ceiling, 1M TPM, 8 concurrent requests |
| Model | None. Set one to add a limit for one model. |

- Until an org adds credit, the org and each of its keys are held to 20 RPM.
- The operator sets platform limits separately.
- Under a contract, the operator can replace the defaults and the keys' RPM ceiling for one org.
  This override can add a per-second limit on the org. It also replaces the 20 RPM hold. Limits
  you set still apply and can only lower an override.
- Traffic to a [dedicated endpoint](/docs/dedicated-endpoints#limits) keeps the endpoint's own
  limits.

## Declare provider account capacity

```json
{
  "provider":"openai",
  "rpm":500,
  "tpm":1000000,
  "concurrency":20,
  "headroom_percent":90,
  "parent_pool_id":null
}
```

- `provider` is `openai`, `anthropic`, `azure` or `bedrock` for customer pools.
- Send finite positive values.
- `headroom_percent` is the usable percentage, 1 to 100. Default 90.
- Fractional limits round down, with a minimum of one unit.
- A parent pool must belong to the same org and use the same provider.
- Pools nest up to four levels, the leaf included. Cycles are not allowed.
- An update replaces the limits and parent. Send the complete configuration.

To bind a connection, send `{"connection_id":"CONNECTION_UUID"}` to
`/api/quota-pools/{id}/bind`. The connection's profile must match the pool's provider.

Keys for the same upstream account should share a pool, or use child pools under a shared
parent. Rotating credentials does not reset capacity. RouterPlus checks parent and child claims
together. RouterPlus cannot tell your real provider account from a key, so your pool
declarations must be accurate.

The operator owns the house pools and maps them to catalog deployment IDs. You cannot change
them. A route with no declared pool is unavailable (`503`, `x-tm-limit-kind:
capacity_unknown`). RouterPlus does not treat it as unlimited.

RouterPlus does not see provider usage outside the gateway. A provider 429 pauses the affected
leaf pool for the provider's `retry-after` (1 to 60 seconds), across all workers. An error of
unknown scope does not mark the whole provider as down.

## Counting and token estimates

- One admitted logical request uses gateway RPM once.
- Every physical attempt, an eligible fallback included, uses upstream RPM and its own token
  reservation.
- A request sent to a provider is not refunded. A request proven not dispatched can release its
  provider claim.
- Concurrency uses renewable 10-second leases, so a crashed worker does not hold slots.

### Input estimate

The input reservation is the UTF-8 byte size plus 65,536 per image or document part. The base64
of an inline image is left out of the byte size, so the image counts only its 65,536. This is an
estimate, not the vendor's tokenizer count.

### Output bound

- RouterPlus sends an output bound of up to the model's own `max_output_tokens` (16,384 on
  GPT-4o, 128,000 on the Claude 5 models, 32,768 when the model declares none).
- A request without an output limit gets the model's own maximum, written into the request. It
  is lowered when needed so that the input and output fit the model's context window.
- The token estimate counts at most 32,768 of the output limit, so a large default does not
  crowd out other requests.
- The balance hold counts all of an output limit that the request sends, and at most 32,768
  tokens of the default.
- A larger or conflicting output limit returns 400.

### Settlement

- Observed input and output usage of a completed attempt replaces the estimate. Cache and
  reasoning tokens are not counted twice.
- Partial prefill is not a final bill. Cancelled, incomplete, unknown or estimated usage keeps
  its reservation until the rate window expires. Work proven never dispatched is the exception.
- The rolling minute includes a partial second, so a charge can stay for up to 61 seconds.

### Anthropic token counting

Anthropic token counting uses the same authorization and shared limits. It uses gateway and
provider RPM, reserves estimated input capacity and writes a request audit before it calls the
provider. It creates no inference debit and no inference-attempt row. Its answer is
`{"input_tokens": N}` and nothing else.

## Inspect limits and errors

`GET /v1/limits?model=MODEL_ID` on the gateway returns the effective gateway limits for the key,
inherited limits included. `model` is optional. The answer also has:

| Field | Meaning |
|---|---|
| `max_output_tokens` | The output bound: the model's own maximum, or 32,768 without a model. |
| `default_max_output_tokens` | The limit a request without one gets, for a short prompt. A long prompt gets less. |
| `quota_max_output_tokens` | The most output the token estimate counts: 32,768. |
| outage mode | The outage mode that applies. |

On a dedicated endpoint, RouterPlus caps a configured `default_max_output_tokens` at the model's
maximum, and `/v1/limits` reports the capped value. RouterPlus still rejects an output limit
that a client sends above the model maximum.

A successful admission returns `x-tm-remaining-rpm` and `x-tm-remaining-tpm`: the lowest
remaining amount across the claimed scopes at that moment. They do not promise capacity for a
later request. On a dedicated endpoint with a
[paced limit](/docs/dedicated-endpoints#the-paced-limit), `x-tm-queue-ms` gives how long the
request waited before admission.

Failures keep SDK-compatible statuses and include metadata and CORS-readable headers:

| Origin | Typical status | Meaning |
|---|---:|---|
| `gateway_admission` | 429 or 503 | A customer quota or spend limit, fairness headroom, or worker capacity. |
| `upstream_quota` | 429, or 503 if unconfigured | A declared account pool is used up or cooling down, or its capacity is unknown. |
| `upstream` | The provider's status | A provider response, including its 429. |
| `gateway_infrastructure` | 503 | A counter, journal or recovery failure. |
| `authorization` | 401, 404 or 503 | Authority is invalid or unavailable, or a snapshot changed or expired. |

Headers:

- `x-tm-error-origin`
- `x-tm-limit-scope`
- `x-tm-limit-kind` (`rpm`, `rps`, `rps_queue`, `tpm`, `concurrency`, `cooldown`, `spend`,
  `health`, `capacity_unknown`, …)
- `x-tm-limit-id` (optional)
- `retry-after`

The metadata says whether to retry and, where known, the retry or cap-reset time. A
rolling-window retry time is advisory. It does not guarantee capacity.

RouterPlus records quota and admission refusals after authentication, also when no attempt was
made. The record is best effort: when it cannot be written, you still get the 429, not a 503.
Read them with `GET /v1/generation?id=REQUEST_ID`. `admission_events` come with the attempts,
and each attempt includes its receipt context and error origin. This audit cannot tie
unauthenticated requests or HTTP-parser protections to an org.

## Availability and fairness

- **Healthy shared store:** limits are shared across workers. Pilot fairness protects idle
  tenant headroom. It caps each org at a share of a shared house pool's usable capacity, with a
  one-unit minimum. The share is half by default, and 75% on the decision-model pools
  (`typesafe`, `bespokelabs`,
  `workers-ai`, `levanto`, `fastino`, `routerplus`, `perplexity-decisions`, `openai-decisions`).
  RouterPlus rejects excess work at once. There is no waiting queue, except on a dedicated
  endpoint with a [paced limit](/docs/dedicated-endpoints#the-paced-limit). There is no general
  no-starvation guarantee for any number of tenants.
- **Redis unavailable:** the default local allowance is at most 5 logical requests a minute, 2
  concurrent local requests and 25,000 estimated TPM per scope per worker, for at most 30
  seconds. A dedicated endpoint's traffic instead gets the endpoint's own limits divided among
  the gateways that serve it, for at most 10 minutes by default. A paced endpoint paces each
  gateway at its share of the rate. `x-tm-admission-mode: bounded_local` marks this path.
  Limits and balances can be approximate across workers during this time. Strict mode refuses
  new work. Every provider call still needs a durable journal intent.
- **Postgres unavailable:** verified snapshots can serve for a bounded time (5 minutes by
  default, up to 1 hour if the operator sets it for an outage) while the live Redis authority
  fence still matches. A single failed read falls back to a snapshot at most 5 minutes old.
  After reads have failed for 5 seconds with none succeeding, the gateway answers from the
  snapshot at once, up to 15 seconds after the last failure. It does not wait on Postgres again
  in that time. Cold, evicted, expired or changed authority fails closed.
  `x-tm-authorization-mode: bounded_snapshot` marks requests that used this fallback. If both
  stores are unavailable, RouterPlus does not use cached authority.
- **Management during Redis failure:** RouterPlus refuses changes, so revocation stays
  coordinated. A failed publication can leave a fence that needs operator recovery. Provider
  calls in progress can finish on their pinned credential version.
- **Redis replaced or state lost:** shared counters need explicit recovery. RouterPlus does not
  treat an empty store as a full unused allowance.

These are pilot controls. They give no provider capacity commitments, exact vendor TPM
compliance, private connectivity, regional processing or ZDR. Under a routing policy, automatic
fallback stays within one provider label. See [routing policies](/docs/routing-policies).

## Workspace and principal scopes

`workspace` and `principal` limits add up across the keys bound to that scope. RouterPlus claims
them in the same atomic step as the org, key, model, platform and pool limits. Bindings come
from server-managed keys, never from request headers. See [identity management](identity.md).

---

Source: https://demo.routerplus.com/docs/pricing.md

# Pricing & billing

You pay the provider's per-token price, published per model in USD per million tokens.
RouterPlus adds no per-token markup and no fee line.

Every billed response has `usage.cost`: the amount debited from your balance. You can recompute
it from the response.

One provider
is our own: RouterPlus, which runs its decision models on its own GPUs. These models cost $0
for input and output (see [Decision models](#decision-models)).

In the console, open Billing > Credits to buy credits and see your balance.

## Where prices live

The catalog is public and needs no key:

```bash
curl -s https://demo.routerplus.com/api/models.json \
  | jq '.models[] | select(.id == "claude-sonnet-4-5").pricing_usd_per_million'
```

```json
{
  "prompt": "3",
  "cached_prompt": "0.30",
  "cache_write": "3.75",
  "completion": "15"
}
```

Prices are **decimal strings** with up to six fractional digits, in USD per million tokens. They
convert without loss to the ledger's unit, so billing uses no floating-point rounding.

| SKU | What it bills | Example (`claude-sonnet-4-5`) |
|---|---|---|
| `prompt` | Non-cached input tokens | $3 / M |
| `cached_prompt` | Prompt-cache **reads** | $0.30 / M |
| `cache_write` | Prompt-cache **writes** | $3.75 / M |
| `completion` | Output tokens, including reasoning unless a distinct rate is declared | $15 / M |
| `internal_reasoning` | The reasoning part of output, when a provider declares a distinct rate | (none) |

> [!NOTE]
> A model that declares no cache or reasoning SKU bills those tokens at its base rate.
> `cached_prompt` and `cache_write` default to the `prompt` price. `internal_reasoning`
> defaults to the `completion` price. A missing price never means $0.

## Image models

Image models bill on the **same two SKUs**, in USD per million tokens: `prompt` for the text you
send, `completion` for the image that comes out. OpenAI reports image output in tokens, and the
size and quality you ask for set the count. The per-image ceiling is the most output tokens one
image can bill. It is `max_output_tokens` on the model page and in `/api/models.json`.

| Model id | Text in /1M | Image out /1M | Per-image ceiling |
|---|---|---|---|
| `gpt-image-1` | $5 | $40 | 6,240 tokens |
| `gpt-image-1-mini` | $2 | $8 | 6,500 tokens |
| `gpt-image-1.5` | $5 | $32 | 6,500 tokens |
| `gpt-image-2` | $5 | $30 | 8,232 tokens |
| `gpt-image-2.5-flare` | $5 | $30 | 8,232 tokens |
| `gpt-image-2.5-sunburst` | $5 | $30 | 8,232 tokens |

Every output token bills at the image rate. This includes the text output tokens that
`gpt-image-1.5` counts in its `output_tokens`. See [POST /v1/images/generations](/docs/api-images).

## Models billed at the provider's reported cost

Some image and video models are billed at the provider's listed price for the render or the job,
with no margin. RouterPlus reserves the model's ceiling (per image, or per second of video),
charges the cost the provider reports, and releases the rest. The prices and ceilings are on
[POST /v1/images/generations](/docs/api-images) and [Videos](/docs/api-videos).

## Decision models

Decision models answer typed questions on [POST /v1/decisions](/docs/api-decisions). They are
priced in USD per million tokens, like every other model.

| Model id | Input /1M | Cached /1M | Output /1M | What the token counts are |
|---|---|---|---|---|
| `typesafe/jev-1.13` | $0.042 | – | $0 | The provider's input count |
| `bespokelabs/nimble-v4` | $0.04 | – | $0 | The provider's input count |
| `bespokelabs/nimble-v4-max` | $0.09 | – | $0 | The provider's input count |
| `cloudflare/clef` | $0.24 | – | $0 | The provider's input count |
| `cloudflare/clef-flash` | $0.09 | – | $0 | The provider's input count |
| `routerplus/decider-2b` | $0 | $0 | $0 | The provider's input count; an answer from its cache bills $0 |
| `routerplus/kev-4b` | $0 | $0 | $0 | The provider's input count; an answer from its cache bills $0 |
| `perplexity/pplx-decider-v1-27b` | $0.04 | – | $0 | The provider's input count, which holds the state once per question |
| `openai/gpt-6-luna-decisions` | $0.10 | – | $0 | The provider's input count: the state once, and about 150 tokens for each question |
| `levanto/sage-1.2` | $0.05 | – | $0 | The provider's input count, which holds the state once per question |
| `scaledown/classify-1` | $0.05 | – | $0 | The provider's input count, summed over its questions (each one is a call) |
| `fastino/gliner-2.5-decide` | $0.03 | – | $0 | Fastino's `prompt_tokens` |

Output is free on every decision model. Each model is billed on its provider's input count at
the provider's price, with no markup, unless its section below says otherwise.

### Bespoke Nimble

- $0.04 per million input tokens for Nimble v4, $0.09 for Nimble v4 Max.
- Bespoke counts the state and the questions once per call.
- A three-question call of 178 input tokens costs $0.000007 on Nimble v4 and $0.000016 on
  Nimble v4 Max.
- `bespokelabs/nimble-latest` is in no list. It bills at $0.04 per million input tokens,
  Bespoke's price for `nimble-latest`, whichever model Bespoke picks for it.

### Clef and Clef-flash

- $0.24 and $0.09 per million input tokens. Cloudflare reports `output_tokens: 0`.
- Cloudflare bills Workers AI in neurons, at $0.011 per 1,000 neurons. Its price page lists Clef
  at 21,818 neurons ($0.24) and Clef-flash at 8,182 neurons ($0.09) per million input tokens.
- A call of 400 input tokens costs $0.000096 on Clef and $0.000036 on Clef-flash.

### Decider 2B and Kev 4B

Decider 2B and Kev 4B are RouterPlus's own models, on RouterPlus's own GPUs. RouterPlus
charges nothing for them.

- Input, cached input and output are free. `usage.cost` is `0` on every call.
- RouterPlus still reports its input count, which is at most the model's context. A long input
  that Decider 2B (25,600 tokens) cuts counts only the tokens the model read.
- A free call is still metered, counted against your limits and its pool, and recorded in
  `GET /v1/generation`. It still needs an organization with credit.
- RouterPlus answers an identical request from its cache for 600 s.

### Perplexity Decider v1 27B

- $0.04 per million input tokens.
- Perplexity runs one prompt for each question, and each prompt holds the whole state.
- Five questions about a state of 1,843 tokens bill 9,215 input tokens, $0.000368.
- One call with many questions still saves round trips.

### GPT-6 Luna Decisions

- $0.10 per million input tokens. OpenAI reports `output_tokens: 0`.
- It bills at its list price: no discount applies to it today.
- OpenAI bills the state once per call. It runs a prompt for each question, so each question
  adds about 150 input tokens. One yes/no about a two-character state billed 156 tokens, and
  five billed 760.
- A three-question call (a choice, a score and a yes/no about a short complaint)
  billed 396 input tokens, $0.000039.
- One call with many questions saves round trips and the state's tokens. The questions' tokens
  stay the same.
- A question that OpenAI declines (its answer is `{"type": "refusal"}`) is billed too.
- The listing's context is 272,000 tokens. Up to there, OpenAI's listed price holds. OpenAI
  bills input above it at twice the rate.

### GLiNER-2.5-Decide

- $0.03 per million input tokens.
- `usage.input_tokens` is Fastino's `prompt_tokens`. `usage.output_tokens` is its
  `completion_tokens`.
- A call of 1,000 input tokens costs $0.00003.

### Sage

- $0.05 per million input tokens.
- Sage reads and bills the state once per question, like Perplexity Decider.

### The decisions discount

A discount can apply to decision models: a percent off the charge of every decision model, or of
some of them, each at its own percent. It applies to every key, the Playground's included. It
does not change the list price, and it can change or end.

A discounted model's page and directory row show the percent beside the struck list price. Its
row in `/api/models.json` has `discount_percent`. Every discounted response says what applied:

| Where | What it shows |
|---|---|
| `usage.cost` | What you paid, after the discount |
| `usage.cost_before_discount` | The charge at the list price |
| `usage.discount_percent` and the `x-tm-discount-percent` header | The percent taken off |
| `cost_usd` in [GET /v1/generation](/docs/api-usage) | What you paid, after the discount |
| `inference_cost_usd` in GET /v1/generation | The cost at the list price |

- The discounted charge is the list charge × (100 − percent) / 100, rounded down to the
  micro-dollar.
- With no discount, the three discount fields are absent and `usage.cost` is the list charge.
- A discount needs an organization with credit. With a balance of $0, a discounted request gets
  `429 insufficient_quota`, as any request does, even at 100 %.

A discount can cover a model whose list price is already $0 (Decider 2B and Kev 4B today). Then
the discount has nothing to take off: `usage.cost` and `usage.cost_before_discount` are both
`0`. The gateway still reports `usage.discount_percent` and the header, and `/api/models.json`
still carries `discount_percent` on its row, so one reader works for all
twelve models. The model's page, the directory and the Playground show no "N % off" beside a $0
price, because the model is free, not discounted.

## The math: integer micro-USD

All money math is in integer micro-USD (millionths of a dollar). A published price of `"2.50"`
becomes `2500000` micro-USD per million tokens. Two rounding rules apply:

- **Reserve rounds up** (ceiling), so the hold is conservative.
- **Settlement rounds down** (floor).

The settlement formula:

```
non_cached_input = max(0, input_tokens − cached_tokens − cache_write_tokens)
reasoning        = clamp(reasoning_tokens, 0, output_tokens)
plain_output     = output_tokens − reasoning

cost_micro = floor((
    non_cached_input   × prompt_per_M
  + cached_tokens      × cached_prompt_per_M
  + cache_write_tokens × cache_write_per_M
  + plain_output       × completion_per_M
  + reasoning          × reasoning_per_M
) / 1,000,000)
```

Each `*_per_M` is the integer micro-USD price. `usage.cost` is this number divided by 10⁶.

## Reserve, then settle

Every billed request has four steps:

1. **Reserve.** Before RouterPlus contacts a provider, the gateway holds an estimate of the
   highest cost.
   - **Input (chat surfaces):** one token per byte of the request body, plus 65,536 tokens for
     each image or document part. An image sent inline counts the 65,536 only, not its base64
     bytes.
   - **Output:** all of your `max_tokens` (or `max_completion_tokens`, or `max_output_tokens`
     on [POST /v1/responses](/docs/api-responses)).
   - **No output limit:** the hold covers at most 32,768 output tokens, and the model gets its
     own maximum. On the chat surfaces, the gateway writes that maximum into the request,
     lowered when needed so that the input and output fit the model's context window (see
     [POST /v1/chat/completions](/docs/api-chat-completions)). On `/v1/responses` the gateway
     adds none, and OpenAI applies the same maximum.
   - **Rate:** each side is priced at the model's highest rate for that side (cache writes and
     reasoning, when they cost more than plain input or output). The sum rounds **up** to
     micro-USD.
   - **Web search:** a request with a web search tool also holds the search fee for the tool's
     `max_uses` searches (at most 20), or for 5 when it sets none (see [Web search](#web-search)).
   - **Images:** on `/v1/images/generations`, the output is reserved at `n` × the model's
     per-image ceiling (not `max_tokens`), and the input at the prompt bytes ÷ 4.
2. **Dispatch.** The request goes to a provider. Prices are pinned at this moment (see
   [Price changes](#price-changes-frozen-snapshots)).
3. **Settle.** When the attempt finishes, RouterPlus prices the provider-reported token counts
   with the formula above and rounds **down**. You pay that amount, also when it is more than the
   hold (a long answer to a request without a limit).
4. **Release.** RouterPlus releases the hold in full.

### When usage is not reported

An attempt can end before the provider reports usage: you cancelled it, or the provider failed
after your answer started. You then pay for the input and for the output already streamed to
you. RouterPlus does not charge the hold.

- **Input:** the provider's own count when it sent one. Claude sends it when its answer starts,
  with its cache reads and writes, each at its own price. Otherwise, the request's likely input
  (its size at four bytes a token, with 1,600 tokens for each image or document part) at the
  plain input price.
- **Output:** what was streamed to you before the end (text, tool-call arguments and any
  reasoning you received), at four characters a token. If you cancel before the first token, or
  the call does not stream, the output part is $0.

A model that hides its thinking (the Claude 5 models, by default) streams nothing while it
thinks. A cancel during that time costs the input only. Such a charge has the provenance
`estimated`.

You pay the reserved maximum in two cases:

- The gateway stopped mid-request and cannot observe the outcome (`unknown_after_crash` in
  [GET /v1/usage & /v1/generation](/docs/api-usage)), because the provider may have billed the
  call.
- An attempt completes and the provider reports no usage. This is rare.

### Refusals before dispatch

| Cause | Refusal |
|---|---|
| **Balance too low.** Your balance must cover every reservation your org has in flight, this one included. | `429` with `error_type: insufficient_quota` (the OpenAI shape SDKs handle; native type `billing_error` on the Anthropic surface) |
| **Monthly spend cap hit.** Caps are org-wide and per key, per UTC calendar month. The strictest wins. | The same `429 insufficient_quota`, with the reset time in the message and in the `x-tm-cap-reset` header |
| **A rate limit hit.** The key's RPM, a shared limit of the organization, or the provider's pool. | `429` with `error_type: rate_limit` and `retry-after` |

[Errors & remediation](/docs/errors) gives the fix for each.

### Failed images cost nothing

A failed attempt, a response over the 32 MiB cap, and a 200 with no decodable image all settle
at zero. RouterPlus never returns or bills a partial image.

If you hang up mid-render, the render still finishes upstream. That attempt settles the
**provider-reported** usage: never the reservation, and never $0.

## The cost on the wire: `usage.cost`

Every billed response has `usage.cost` (USD, a JSON number):

- **Non-streaming:** in the response body's `usage`, on both surfaces.
- **Streaming, OpenAI surface:** in the final usage chunk, before `[DONE]`.
- **Streaming, Anthropic surface:** in the `message_delta` usage at stream end.

```bash
curl -s https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-4-5","max_tokens":350,"messages":[{"role":"user","content":"hello"}]}' \
  | jq .usage
```

```json
{
  "prompt_tokens": 1200,
  "completion_tokens": 350,
  "total_tokens": 1550,
  "prompt_tokens_details": { "cached_tokens": 800, "cache_write_tokens": 0 },
  "cost": 0.00669
}
```

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])
r = client.chat.completions.create(
    model="claude-sonnet-4-5", max_tokens=350,
    messages=[{"role": "user", "content": "hello"}],
)
print(r.usage.model_dump()["cost"])  # the ledger debit, in USD
```

```python
import os
import anthropic

client = anthropic.Anthropic(base_url="https://demo-api.routerplus.com", api_key=os.environ["TM_API_KEY"])
msg = client.messages.create(
    model="claude-sonnet-4-5", max_tokens=350,
    messages=[{"role": "user", "content": "hello"}],
)
print(msg.usage.model_dump()["cost"])
```

> [!WARNING]
> `usage.cost` is the **full request debit**, not the serving attempt's share. If a
> provider failed mid-prompt and the gateway failed over before your answer started,
> the number includes the tokens the failed attempt used. The per-attempt breakdown is
> at [GET /v1/generation](/docs/api-usage).

Usage reporting is always on:

- Streams and non-streams both carry usage.
- The gateway requests usage from the provider whatever you send, because RouterPlus cannot
  recompute unreported usage.
- Only an explicit `stream_options.include_usage: false` changes anything. You are billed the
  same, but no usage chunk is written to you.

On the Anthropic surface, a **stream that fails after your answer started** has no place for
usage outside `message_delta`. Its terminal error event carries no cost. Read those costs from
[/v1/generation](/docs/api-usage). The OpenAI surface sends any known usage and cost before its
terminal error.

## Cache reads and writes

Cache tokens bill at their own rates on **both** surfaces. Both surfaces report the split:

| Concept | OpenAI surface | Anthropic surface | Billed at |
|---|---|---|---|
| Non-cached input | `prompt_tokens` minus the two details below | `input_tokens` | `prompt` |
| Cache reads | `prompt_tokens_details.cached_tokens` | `cache_read_input_tokens` | `cached_prompt` |
| Cache writes | `prompt_tokens_details.cache_write_tokens` | `cache_creation_input_tokens` | `cache_write` |
| Output | `completion_tokens` | `output_tokens` | `completion` |

On the OpenAI surface, `prompt_tokens` includes cache reads and writes. Billed responses always
have `prompt_tokens_details` with both fields, set to zero when the provider reports none. Stream
and non-stream usage have the same shape.

On Claude, a cache write costs 1.25 times the input price (`cache_write`). A cache read costs a
tenth of it or less (`cached_prompt`).

When an agent sends a conversation with tools or earlier turns and no cache marks of its own,
the gateway turns on Claude's automatic caching for it. The first turn pays a cache write on its
prompt. The next turns read that prompt at the cache price.

- The `x-tm-auto-cache` response header says whether automatic caching applied.
- The request header `x-tm-auto-cache: off` turns it off.
- [Wire compatibility](/docs/compat#automatic-caching-on-claude) has the rules.

## Web search

RouterPlus bills a web search that the provider runs for the model at the provider's price per
search, plus the tokens:

| Tool | Where | Price |
|---|---|---|
| Anthropic's web search server tool (`web_search_20250305` and later) | Claude models on `POST /v1/messages` | **$10 per 1,000 searches** ($0.01 a search) |
| Anthropic's web fetch server tool | Claude models on `POST /v1/messages` | No fee: only the tokens of the fetched page |
| OpenAI's hosted `web_search` tool | OpenAI models on [POST /v1/responses](/docs/api-responses) | $10 per 1,000 calls |
| OpenAI's older `web_search_preview` tool | GPT-4o, GPT-4o mini and GPT-4.1 mini on `POST /v1/responses` | $25 per 1,000 calls |

- Each search counts once, whatever the number of results. A failed search is not billed.
- The pages a search returns become input tokens at the model's input price, as on the
  provider's own API. On a stream, the input is the count at the end of the turn.
- The fee is part of `usage.cost`, and of `cost_usd` in
  [GET /v1/usage & /v1/generation](/docs/api-usage). There, each attempt also shows
  `server_tool_use` (`web_search_requests`, `web_fetch_requests`) and `server_tool_fee_usd`. On
  `/v1/messages`, the answer's `usage.server_tool_use.web_search_requests` also gives the number
  of searches.
- While the request runs, the hold includes the fee for each web search tool's `max_uses`
  searches (at most 20), or for 5 searches when the tool sets no `max_uses`. A tool of your own
  named `web_search` holds nothing.
- If the model searches more often than the hold allows, you pay for every search it made. The
  hold limits what is set aside, and the charge has no such limit.
- On your own provider connection (BYOK), RouterPlus charges no fee. The provider bills you.

Anthropic's server tools, web search included, work only for Claude models on
`POST /v1/messages`. Claude Code calls them there. See [POST /v1/messages](/docs/api-messages).

## Reasoning tokens

Reasoning tokens are **part of output**. They are never billed on top of it.

- The settlement math splits them out and prices them at the model's reasoning rate.
- Today every cataloged model prices reasoning equal to `completion`, so the split changes
  nothing until a provider declares a distinct rate.
- The OpenAI surface reports them in `completion_tokens_details.reasoning_tokens`. The Anthropic
  surface does not report them separately.

## Price changes: frozen snapshots

You pay the price in effect at **dispatch time**.

- Price rows are append-only. A change is a new effective-dated row, never an edit.
- The row in effect when your request is reserved is pinned into the reservation and written
  onto every attempt's ledger row.
- A price change published while your request is in flight does not affect it.
- Past attempts always settle, and audit, against the prices they were dispatched under.

## Paid credits and purchase matching

Create an account at [Sign up](https://demo.routerplus.com/signup), complete Clerk authentication and email
verification, and save your first API key. Signup and card verification alone do not grant
credit. Existing customers keep their balance and API keys.

Standard credit purchases start at **$10** in [Billing](https://demo.routerplus.com/console/billing). The
payment adds the amount you buy as paid credit. Your balance is the sum of credit ledger entries
minus settled spend.

If purchase matching is offered for your account, Billing shows the cap and remaining matching credit.

Only eligible paid purchases receive a match, when the offer is available in Billing.

- New matching offers have a lifetime maximum of **$100 per account** across eligible
  purchases. Billing shows the offer and the matching credit left for your account.
- Matching credit is promotional and shows as its own ledger entry.
- Refunds and chargebacks reverse the matching credit. A partial refund reverses its
  proportional matching credit.
- Adding a card without a purchase does not earn a bonus.
- Existing trial and card-verification grants stay in the ledger. Those signup offers are no
  longer available.

Before an organization buys credit, its customer keys share the trial rate of 20 requests a
minute. A purchase lifts that cap. Each key then has its own ceiling (300 for a console key).
RouterPlus can also limit past trial credit when it detects unusual usage. Contact support if
that happens. See [Rate limits & spend caps](/docs/limits).

## Worked example

A `claude-sonnet-4-5` request reports 1,200 prompt tokens (800 of them cache reads, no cache
writes) and 350 completion tokens:

```
non-cached input:   400 × 3,000,000  = 1,200,000,000
cache reads:        800 ×   300,000  =   240,000,000
output:             350 × 15,000,000 = 5,250,000,000
                                       ─────────────
sum / 1,000,000 (floor)              = 6,690 micro-USD
```

`usage.cost` in the response is `0.00669`. The attempt's `cost_usd` in
[GET /v1/usage & /v1/generation](/docs/api-usage) is the same number, from the same computation.

## Worked example: an image

A `gpt-image-1` request for two medium 1024×1024 images (`n: 2`) with a 12-token prompt
estimate. Text in is $5 / M, image out is $40 / M, and the per-image ceiling is 6,240 tokens.

The reservation, before RouterPlus contacts a provider:

```
prompt estimate:  12 ×  5,000,000 =         60,000,000
output ceiling:   2 × 6,240 × 40,000,000 = 499,200,000,000
                                           ───────────────
ceil(sum / 1,000,000)                    = 499,260 micro-USD
```

The provider reports 12 input tokens and 2,112 output tokens (1,056 per image). Settlement:

```
input:            12 ×  5,000,000 =         60,000,000
output:        2,112 × 40,000,000 =     84,480,000,000
                                        ──────────────
floor(sum / 1,000,000)            =     84,540 micro-USD
```

`usage.cost` in the response is `0.08454`. RouterPlus releases the hold in full, and you pay the
settled amount.

---

Source: https://demo.routerplus.com/docs/postpaid.md

# Postpaid billing

With postpaid billing, your organization uses RouterPlus on a credit line and pays after the
month ends. RouterPlus sends an invoice each month for that month's usage.

To get a credit line, contact RouterPlus. RouterPlus sets the limit, the payment terms and the
email address that gets the invoices.

## How a credit line works

- Without a credit line, RouterPlus refuses new requests when your balance is at zero.
- With a credit line, your balance can go below zero, down to minus the line's limit. With a
  $5,000 line, your organization can use $5,000 before it pays. Your balance then shows -$5,000.
- Credit you buy is added to your balance and gives you more room on the line.
- RouterPlus holds the maximum cost of each request while it runs, as for prepaid credit.
  Requests in flight count against the line until they finish.

## At the limit

When a request would take your balance past minus the limit, RouterPlus refuses it with the
usual out-of-credit error: `429 insufficient_quota`. The message says that the organization
reached the end of its credit line. Do not retry. The request fails until the balance goes up.

To continue, pay the open invoice or ask RouterPlus for a higher limit. A new limit applies
within a few seconds.

Your spend caps and guardrail budgets still apply. RouterPlus refuses a request that fits the
credit line but breaks a monthly cap or a budget.

## Invoices

- An invoice covers one UTC calendar month. It bills the month's usage at the prices you paid on
  each request.
- It has one line per model and provider, with the token counts and the amount.
- RouterPlus issues it after the month ends and emails it to your billing address. It shows the
  period, the amount due, the terms and the due date.
- When RouterPlus records your payment, it adds the invoice amount to your balance. When all
  your usage is invoiced and paid, your balance is back at zero.
- Quote the invoice number (`inv_...`) with your payment.

| Terms | Due |
|---|---|
| `net15` | 15 days after the invoice is issued |
| `net30` | 30 days after the invoice is issued (the default) |
| `net45` | 45 days after the invoice is issued |

## Read your credit line and invoices

These routes need a management key (`Authorization: Bearer tm_mk_...`). Each shows only your own
organization. They also work under `/api/v1/`.

| Route | Returns |
|---|---|
| `GET /v1/billing/credit-line` | Your credit line. `404` without one. |
| `GET /v1/billing/invoices` | Your invoices, newest month first, without `lines`. |
| `GET /v1/billing/invoices/{id}` | One invoice with its lines. |

`GET /v1/billing/credit-line`:

```json
{
  "data": {
    "limit_usd": 5000,
    "terms": "net30",
    "billing_email": "ap@example.com",
    "enabled": true,
    "balance_usd": -1250.5,
    "available_usd": 3749.5,
    "open_invoices_usd": 900,
    "created_at": "2026-10-01T09:00:00.000Z",
    "updated_at": "2026-10-01T09:00:00.000Z"
  }
}
```

| Field | Meaning |
|---|---|
| `balance_usd` | Your credit minus your settled usage. Below zero while you use the line. |
| `available_usd` | What you can still use: the balance plus the limit while the line is enabled. Requests in flight are not subtracted. |
| `open_invoices_usd` | The total of issued invoices not yet paid. |

`GET /v1/billing/invoices/{id}`:

```json
{
  "data": {
    "id": "inv_3f2a9c0d1e4b5a6978c0d1e2",
    "period_start": "2026-09-01",
    "period_end": "2026-09-30",
    "amount_usd": 900,
    "amount_microusd": 900000000,
    "status": "issued",
    "terms": "net30",
    "issued_at": "2026-10-02T10:00:00.000Z",
    "due_at": "2026-11-01T10:00:00.000Z",
    "paid_at": null,
    "lines": [
      { "model": "deepseek/deepseek-v3.2", "provider": "deepinfra", "attempts": 41200,
        "input_tokens": 512000000, "cached_input_tokens": 120000000, "cache_write_tokens": 0,
        "output_tokens": 61000000, "reasoning_tokens": 0,
        "amount_microusd": 900000000, "amount_usd": 900 }
    ]
  }
}
```

`status` is `issued` (not paid yet), `paid` or `void` (cancelled).

---

Source: https://demo.routerplus.com/docs/partners.md

# Partner accounts

A partner is an organization that creates RouterPlus accounts for its own customers. Each
account is an organization of its own. The partner funds each account from its balance and earns
a share of the account's spend.

To become a partner, contact RouterPlus. RouterPlus sets your revenue share and turns the partner
API on for your organization.

```bash
curl https://demo-api.routerplus.com/v1/partner/accounts \
  -H "Authorization: Bearer $PARTNER_MANAGEMENT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"external_id": "cust-1042", "name": "Acme Corp", "credit_usd": 25}'
```

## Authentication

The partner API uses a management key of the partner organization:
`Authorization: Bearer tm_mk_...`. Create one with `POST /v1/management-keys` (see the
[management API](api-management.md)). Every route is served under `/v1/...` and `/api/v1/...`.

- A partner sees and changes only its own accounts. An account of another partner returns `404`.
- A management key of an organization that is not a partner returns `403 not_a_partner`.
- An account's own keys cannot call the partner API.

## Routes

| Route | Does |
|---|---|
| `POST /v1/partner/accounts` | Creates an account. Returns `201`. |
| `GET /v1/partner/accounts` | Lists every account. |
| `GET /v1/partner/accounts/{external_id}` | Reads one account. |
| `PATCH /v1/partner/accounts/{external_id}` | `{"disabled": true}` or `{"disabled": false}`. |
| `POST /v1/partner/accounts/{external_id}/credits` | `{"usd": n}`. Moves credit to the account. |
| `POST /v1/partner/accounts/{external_id}/keys` | `{"name"?}`. Creates another inference key. Returns `201`. |
| `GET /v1/partner/earnings?period=YYYY-MM` | Spend and revenue share per account for one month. |

Each account shows `balance_usd` and `month_to_date_spend_usd`, its spend since the start of the
current UTC month.

## Create an account

| Field | Type | Notes |
|---|---|---|
| `external_id` | string | Your own id for the customer. 1 to 128 letters, digits or `. _ : @ -`. Unique among your accounts. |
| `name` | string | The account's organization name, 1 to 100 characters. |
| `credit_usd` | number | Optional. Credit to move from your balance to the account, 0 to 10,000. |

The answer is `201`:

```json
{
  "data": {
    "external_id": "cust-1042",
    "org_id": "6c1f0d1e-2a35-4a3e-9a51-0f4c2d7b9e10",
    "name": "Acme Corp",
    "disabled": false,
    "balance_usd": 25,
    "month_to_date_spend_usd": 0,
    "workspace_id": "0b8e3f52-7c41-4d55-8f0a-3e2d9a6c1b77",
    "created_at": "2026-10-31T12:00:00.000Z"
  },
  "key": "tm_vk_...",
  "key_id": "…",
  "management_key": "tm_mk_...",
  "management_key_id": "…"
}
```

The account gets a default workspace and two keys:

- `key` is an inference key. Give it to your customer, or call models with it for them.
- `management_key` is a management key for the account's own organization. With it the account
  can create keys, set guardrails and read its activity.

The answer shows each raw key once. RouterPlus stores only a hash. Keep both keys safe.

An `external_id` that you already use returns `409`.

## Move credit to an account

Send `{"usd": 10}` to `POST /v1/partner/accounts/{external_id}/credits`.

- The amount is above 0 and at most 10,000.
- RouterPlus takes the amount from your balance and adds it to the account's balance in one
  step. The answer gives the account, and your new balance in `transfer.partner_balance_usd`.
- When your balance is less than the amount, the answer is `402 insufficient_balance` and no
  credit moves. The same rule applies to `credit_usd` on create: the account is not created.
- Credit on an account cannot move back to the partner.

## Create another key

`POST /v1/partner/accounts/{external_id}/keys` returns `201` with a new inference key in `key`,
shown once. The key is in the account's default workspace.

## Disable an account

`{"disabled": true}` turns off every key of the account: its inference keys and its management
keys. Requests with those keys get `401`. You cannot create keys for a disabled account.

`{"disabled": false}` turns on the keys that the disable turned off. A key that was already off
before the disable stays off. A key can take up to 5 seconds to work again.

## Earnings

`period` is a UTC month as `YYYY-MM`. Without it, the current month.

```json
{
  "period": "2026-10",
  "revenue_share_percent": 20,
  "data": [
    { "external_id": "cust-1042", "org_id": "…", "spend_usd": 412.5, "revenue_share_usd": 82.5 }
  ],
  "totals": { "spend_usd": 412.5, "revenue_share_usd": 82.5 }
}
```

- `spend_usd` is the account's usage at RouterPlus prices in the month. Usage on the account's
  own provider keys is not in it.
- `revenue_share_usd` is `spend_usd` times your revenue share, rounded to a millionth of a
  dollar.
- `totals` adds up all accounts.

If RouterPlus set a markup for you, each account also shows `markup_usd`, and the answer shows
`markup_percent` and `markup_note`. The markup is a reporting field: the amount you can charge
your customers on top. RouterPlus bills your accounts at its own prices.

## Audit

Every change writes an audit row on your organization and on the account. Read your rows with
`GET /v1/audit`.

## Errors

| Status | Type | When |
|---|---|---|
| `400` | `invalid_request` | A field is missing, unknown or out of range. |
| `401` | `unauthorized` | No management key, or the key is not valid. |
| `402` | `insufficient_balance` | Your balance is less than the credit to move. |
| `403` | `not_a_partner` | The key's organization is not a partner. |
| `403` | `partner_disabled` | RouterPlus turned off the partner API for your organization. |
| `404` | `not_found` | No account with this `external_id` among your accounts. |
| `409` | `conflict` | The `external_id` is already in use. |
| `409` | `account_disabled` | You asked for a key on a disabled account. |

---

Source: https://demo.routerplus.com/docs/authentication.md

# Authentication

Every gateway request authenticates with a **virtual key**. One key works on both wire
surfaces, spends one org balance and never touches your provider credentials. Send it in one of
the [two headers](#header-forms). In the console, open Access > API keys.

## Virtual keys

| Property | Value |
|---|---|
| Format | `tm_vk_` followed by 48 hex characters (24 random bytes) |
| Shown | at creation, and again from its row at [API keys](https://demo.routerplus.com/console/keys) |
| Stored | the SHA-256 hash (the gateway checks only this), the first 12 characters so the console can label it, and an encrypted copy for copying it again |
| Spends | the org's shared prepaid balance |
| Limits | per-key requests per minute; optional per-key and org-wide monthly spend caps |

> [!NOTE]
> An owner or admin can copy a key again at any time with **copy** on its row at
> [https://demo.routerplus.com/console/keys](https://demo.routerplus.com/console/keys). The copy is encrypted at rest
> (AES-256-GCM, with its key held outside the database), and RouterPlus records each copy. A
> key created before 17 October 2026 is marked **shown once**. RouterPlus stored only its hash,
> so it cannot be recovered. If you lose such a key, or a key leaks, disable it and create a
> new one.
>
> If your organization needs keys that are never shown again, ask RouterPlus support to turn
> copying off. RouterPlus then deletes the copies of your keys and keeps none for new keys.
> Each key is then shown once, at creation.

## Header forms

The gateway reads your key from either header, on **both** surfaces:

| Header | Convention | Typically sent by |
|---|---|---|
| `Authorization: Bearer tm_vk_...` | OpenAI | OpenAI SDKs; Claude Code via `ANTHROPIC_AUTH_TOKEN` |
| `x-api-key: tm_vk_...` | Anthropic | Anthropic SDKs via `api_key` |

The header does not have to match the surface. `x-api-key` works on
`POST /v1/chat/completions`, and `Authorization: Bearer` works on `POST /v1/messages`. When both
headers are present, `Authorization: Bearer` wins.

```bash
# OpenAI surface, bearer auth
curl -s https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'

# Anthropic surface, x-api-key auth
curl -s https://demo-api.routerplus.com/v1/messages \
  -H "x-api-key: $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-haiku-4-5","max_tokens":50,"messages":[{"role":"user","content":"hi"}]}'
```

> [!NOTE]
> `anthropic-version` is **not required** on `/v1/messages`. The gateway pins its own
> version (`2023-06-01`) when it calls the provider.
>
> The gateway forwards your `anthropic-beta` header when the serving provider speaks the
> Anthropic dialect, filtered to an allowlist:
>
> - any Anthropic-dialect provider: the `prompt-caching`, `token-efficient-tools`,
>   `fine-grained-tool-streaming`, `interleaved-thinking`, `claude-code`, `oauth` and
>   `computer-use` betas;
> - Anthropic's own API also: `per-turn-control`, `mid-conversation-output-config`,
>   `context-management`, `mid-conversation-tool-changes` and `inline-tools`.
>
> The gateway drops any other value and names the drop in `x-tm-dropped-params`.
>
> `anthropic-version` changes one thing. On `GET /v1/models` it returns the Anthropic-shaped
> model list instead of the OpenAI shape. Anthropic SDKs send it automatically.

In the SDKs, put the key where the provider's own key goes:

```python
import os
from openai import OpenAI
import anthropic

openai_client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])
anthropic_client = anthropic.Anthropic(base_url="https://demo-api.routerplus.com", api_key=os.environ["TM_API_KEY"])
# anthropic.Anthropic(auth_token=...) also works. It sends Authorization: Bearer,
# which the gateway accepts too.
```

```typescript
import OpenAI from "openai";
import Anthropic from "@anthropic-ai/sdk";

const openai = new OpenAI({ baseURL: "https://demo-api.routerplus.com/v1", apiKey: process.env.TM_API_KEY });
const anthropic = new Anthropic({ baseURL: "https://demo-api.routerplus.com", apiKey: process.env.TM_API_KEY });
```

## Getting a key

### In the browser

1. Open [Sign up](https://demo.routerplus.com/signup) to create an account, or
   [Sign in](https://demo.routerplus.com/login) if you have one. Signup and sign-in use Clerk.
2. Complete the authentication and email-verification steps shown there.
3. For a new account, the first verified sign-in mints your first API key. Save it. You can
   copy it again from its row at [API keys](https://demo.routerplus.com/console/keys), where you can also
   create more named keys.
4. Add paid credits in [Billing](https://demo.routerplus.com/console/billing) before you make model calls.
   Signup and email verification grant no free credit.

Existing accounts:

- Sign in with the verified email of your existing marketplace account to keep its
  organization, keys and balance. A new sign-in method does not create another trial.
- If you have an older marketplace account with no Clerk sign-in, use **Sign up** with the same
  email. This connects it to your existing account.
- Browser sign-in does not change how your applications authenticate with `tm_vk_` keys.

There is no programmatic signup. `POST /v1/signup` and `POST /login` return HTTP `404`. The
`?magic=1` query parameter gives no other signup or sign-in form.

A marketplace session lasts up to 7 days and ends with the linked Clerk session. To end it
sooner, use **Sign out** in the account menu (the avatar in the app bar) or at the bottom of the
console rail. Sign out ends this browser's session on the server. Other devices stay signed in.

## Key rate limits

Each key has a request limit over a rolling 60-second window. The limit is per key, across all
gateway instances.

| Key origin | Requests per minute |
|---|---|
| First key shown after verified sign-in (trial) | 20 |
| Created in the console, or with the identity API | 300 |
| The console's managed Playground key | 60 |

Until your organization buys credit, its keys and the organization as a whole run at the trial
rate of 20 requests per minute, whatever a key's own limit says. The first purchase lifts it
within seconds. The trial rate does not apply to requests served only by your own provider keys
(BYOK) or to the managed Playground key.

Over the limit, the gateway returns `429` with `error_type: "rate_limit"` and a `retry-after`
header. Wait that long, then retry. Your organization also has token-per-minute and concurrency
limits. See [Rate limits & spend caps](/docs/limits).

## Disabling a key

At [https://demo.routerplus.com/console/keys](https://demo.routerplus.com/console/keys), each key row has a **Disable**
action. Disabling is permanent. There is no re-enable in v1, so create a new key instead. The
gateway caches key rows briefly, so a disabled key can keep working for up to about 5 seconds.
After that, every request returns 401.

To cap a key without disabling it, set a per-key **monthly spend cap** on the same page
(**Set cap**). Set the org-wide cap at
[https://demo.routerplus.com/console/billing](https://demo.routerplus.com/console/billing). A breached cap returns
`429 insufficient_quota`. The message and the `x-tm-cap-reset` header give the UTC reset time.

## What a 401 looks like

A missing, invalid or disabled key gets `401` with `error_type: "auth"`. The body has the shape
of the calling SDK family, so your SDK's own error class fires, and it carries `error_type`.
Headers always include `x-request-id`, `x-tm-error-code: auth` and
`x-tm-error-origin: authorization`.

OpenAI surface (`/v1/chat/completions` and every non-Anthropic path):

```json
{
  "error": {
    "code": "auth",
    "message": "missing or invalid API key",
    "type": "invalid_request_error",
    "metadata": { "error_type": "auth", "origin": "authorization", "retryable": false }
  },
  "request_id": "..."
}
```

Anthropic surface (`/v1/messages`):

```json
{
  "type": "error",
  "error": {
    "type": "authentication_error",
    "message": "missing or invalid API key",
    "error_type": "auth",
    "metadata": { "origin": "authorization", "retryable": false }
  },
  "request_id": "..."
}
```

Both SDK families raise their own `AuthenticationError`. The message is the same for every 401
cause. Check in order: the header form, the key value, and whether someone disabled the key in
the console.

## Key hygiene

- Keep keys in environment variables or a secret store. Never commit them.
- Never put keys in URLs. The gateway reads only the two headers above.
- Use one key per app or agent. The console then shows spend and caps per integration, and
  disabling one key stops one integration.
- The console mints its own keys: one **Playground** key per organization, and one per Optimize
  search. They show on the keys page with a **managed** tag. They are never displayed, and they
  bill your credits like any key.

Access > Security in the console lists each active key with its flags: no limit, no expiry,
idle 30 days, expires within 7 days and no IP allowlist:

![Console Security page with three keys: one with three flags, one with no flags, and one that expires within 7 days](media/console-security.gif)

Next: [Quickstart](/docs/quickstart) for browser signup and the first API calls, or
[Errors](/docs/errors) for the complete `error_type` table.

---

Source: https://demo.routerplus.com/docs/byok.md

# Bring your own provider key

Bring your own key (BYOK) routes inference through your organization's own OpenAI, Anthropic or
Azure API keys, or an AWS Bedrock workload role. You still call the gateway with a RouterPlus
key. RouterPlus stores the provider credential encrypted, and the provider bills you directly.
RouterPlus charges nothing for BYOK inference.

In the console, open [Access > Provider connections](https://demo.routerplus.com/console/byok), or use the
API below.

## Set up in the console

1. Sign in, open [`/console/byok`](https://demo.routerplus.com/console/byok) and select your organization.
2. Click **Add connection**. Name it, choose a provider, enter its credential and add exact
   model IDs.
   - Azure also needs a resource and a deployment mapping.
   - You can select Bedrock only when RouterPlus has approved a role for your organization.
   - You cannot change the provider or model list after you save. Create a new connection
     instead.
3. Click **Save and validate**. If validation fails, the connection stays pending. Rotate its
   credential or retry validation. Do not create a duplicate.
4. Choose the limits of an existing provider account, or enter RPM, TPM, concurrency and
   headroom for a new account. Credentials for the same account should share one pool or
   parent. A connection with no pool cannot serve requests.
5. Create a new RouterPlus key or select an existing one.
   - RouterPlus creates and binds a new BYOK key in one transaction before it returns it.
   - The page warns before it replaces an existing key's connection or key-level policy. An
     organization or workspace policy still applies.
6. Copy the new key once and use the generated request example. The optional
   **Send test request** sends "Reply OK" with at most 8 output tokens. It uses provider quota
   and can cost a small provider charge.

Each connection card can validate, edit shared-account limits, rotate, disable and delete.

- Viewers can read metadata. Owners and admins make changes.
- RouterPlus never returns provider secrets.
- A disabled connection becomes active again only after rotation and a successful validation.
- You cannot restore a deleted connection.

When you issue a key, you can select workspaces and principals from the
[identity API](/docs/identity). The console keeps their limits and restrictions.

## Set up with the API

1. Sign in to the console and create a virtual key. BYOK inference needs no marketplace credit.
2. Send `POST /api/connections` on the **site origin**, with JSON:

   ```json
   {"profile":"openai","models":["your-exact-provider-model-id"],"api_key":"YOUR_PROVIDER_KEY"}
   ```

   | Field | Rule |
   |---|---|
   | `profile` | `openai` or `anthropic`. `azure` needs a deployment mapping: see [Azure and policy setup](/docs/routing-policies). `bedrock` takes `workload_role` instead of `api_key`: see [Bedrock](/docs/bedrock). |
   | `models` | 1 to 100 exact model IDs. No wildcards. |
   | `name` | Optional label. |

   The response is `201` with `connection_id` and `status: "pending"`. It never returns the key.
3. Send `POST /api/connections/{connection_id}/validate` with `{}`. Success returns
   `valid: true` and `status: "active"`. For API-key profiles, validation makes one
   authenticated model-list GET, with a 10-second timeout and no generated tokens. It does not
   check access to every declared model or reserve capacity.
4. Send `POST /api/connections/{connection_id}/bind` with `{"key_id":"YOUR_VIRTUAL_KEY_UUID"}`.
   This binds an enabled virtual key in the same org to an active connection. A key that served
   marketplace traffic takes up to 5 seconds to switch. You can use a new virtual key instead.
5. [Declare a quota pool](/docs/admission) for the provider account and assign the connection
   to it. A route with no declared pool returns 503. Credentials for the same account must share
   one pool or parent.
6. Use the virtual key with `/v1/chat/completions` or `/v1/messages`. Both request formats and
   streaming work with each profile, within the wire compatibility limits.

Management rules:

- Management needs the `tm_s` session cookie.
- Writes also need `Content-Type: application/json` and an `Origin` header equal to
  `https://demo.routerplus.com`.
- A same-origin browser `fetch` sends the cookie and Origin for you. A CLI client must send
  both. Keep cookies and provider keys out of shell history.
- An inference key cannot manage connections.
- Viewers can read. Owners and admins can write.

## Management operations

| Method and site path | JSON body | Effect |
|---|---|---|
| `GET /api/connections` | none | Newest 100 connections of your org, deleted ones included |
| `GET /api/connections/{id}` | none | One connection's metadata and fully masked label |
| `POST /api/connections` | `profile`, `models`, `api_key` or `workload_role`, optional `name`, `endpoint_config` | Create a pending connection |
| `POST /api/connections/{id}/validate` | `{}` | Check the credential and activate on success. A failed validation leaves the connection pending. |
| `POST /api/connections/{id}/rotate` | `api_key` or `workload_role` | Revoke old versions and create a pending version. Validate it before use. |
| `POST /api/connections/{id}/disable` | `{}` | Disable the connection and revoke its credential versions |
| `POST /api/connections/{id}/bind` | `key_id` | Bind or rebind a virtual key of your org to an active connection |
| `DELETE /api/connections/{id}` | `{}` | Revoke and mark deleted. Attribution and bindings are kept. |

Lifecycle rules:

- To restore a disabled connection, rotate and validate a new credential.
- You cannot restore a deleted connection.
- Rotation stops new inference until validation succeeds.
- A profile or model-list change needs a new connection and an explicit rebind.
- You cannot unbind a key back to marketplace routing. Use a `house_only` policy from the
  [policy API](/docs/routing-policies) instead.
- Management responses never show a secret suffix, ciphertext, envelope or provider validation
  body.

## Inference behavior

### Routing

- A bound key can use only its connection's exact models. It never falls back to marketplace
  supply on its own.
- `/v1/models` lists the connection's models.
- Anthropic token counting uses the same connection, on an Anthropic profile only.
- Image generation works through an `openai` connection. List the exact image model ids you
  will send (matching is literal). OpenAI bills the render to your account, so `usage.cost` is
  `0`. See [POST /v1/images/generations](/docs/api-images).
- Supported destinations are OpenAI, Anthropic and configured Azure v1 public endpoints. Bedrock
  uses the [workload-role profile](bedrock.md). RouterPlus refuses redirects and arbitrary
  endpoints.
- BYOK gives no private-link, residency or retention guarantee.
- [Routing policies](/docs/routing-policies) add ordering, filtering and funding controls.

### Errors

| Status | Cause |
|---|---|
| `401 auth` | The virtual key is invalid. |
| `404 model_unavailable` | The binding is unavailable, disabled or deleted. |
| `503 gateway_error` | A database or decryption failure. |
| 400 | The request sends its own credentials or destination (`api_key`, `base_url` and the like). |

RouterPlus never relays provider HTTP error bodies, so an echoed credential cannot leak.

### Billing

- RouterPlus meters and audits BYOK inference.
- RouterPlus charges `0`. The attempt's `billing_source` is `byok` and its `cost_usd` is `0`.
- The inference cost is `null`, because RouterPlus does not price your provider account. The
  provider bills your account.
- Marketplace spend caps do not limit that provider bill. Virtual-key RPM limits and your
  organization's token and concurrency limits still apply, including to token counting.
- A BYOK key spends RouterPlus credit in one case: a policy with marketplace fallback. The
  marketplace attempt reserves and charges like any marketplace call.

### Changes and outages

- New calls see a disable or rotation within 5 seconds. Calls and streams already started can
  finish on their original credential version.
- When the configuration database is unreachable, authority that RouterPlus already verified
  can serve for a bounded time. See [limits and outage behavior](/docs/admission).
- Validation and token-count calls are not inference ledger attempts. Validation writes a
  management audit event with no content.
- Shared limits and pilot fairness are in the [admission guide](/docs/admission).

## Related

- [Workspaces and principals](identity.md): scoped keys and trusted end-user limits.
- [Bedrock workload-role profile](bedrock.md): native regional Claude inference. Its write-only
  `workload_role` replaces `api_key`.

The connection lifecycle, the quota-pool rule and the zero RouterPlus charge are the same for
every profile.

---

Source: https://demo.routerplus.com/docs/identity.md

# Workspaces, members and principals

Every inference key belongs to one organization, one workspace and one principal. A principal
is the identity a key acts for: a service, an end user or a member. Use workspaces and
principals to split limits and usage inside an organization. To sync members and groups from
an identity provider, use [SCIM provisioning](scim.md).

In the console, open Team > Members and Team > Workspaces.

## Rules for keys and principals

- A key's organization, workspace and principal cannot change. To change them, mint a new key.
- All keys of one principal share its RPM, TPM and concurrency limits.
- Request values never authenticate a principal: `user`, `metadata`, `x-user-id`,
  `x-tm-principal-id` and `x-tm-workspace-id`.
- Your backend creates an end-user principal and keeps that principal's key. Do not let an end
  user choose a different backend key.
- There is no per-request delegation or SSO.

These keys use the organization's **Default** workspace and its shared `_default` service
principal:

- existing keys;
- keys created through browser onboarding;
- keys created in the console, unless you choose another workspace there.

A shared older key keeps its access, but RouterPlus cannot tell its individual users apart.

## Management authentication

Management calls use a verified console session cookie. Inference keys have no management
rights.

- `x-tm-org-id: <UUID>` selects an organization. The session email must be an active member
  of it. The header never grants access.
- Without the header, RouterPlus picks the caller's own organization first, then their oldest
  active membership.
- `GET /api/identity/organizations` lists your memberships.
- The console shows your default organization. [`/console/byok`](https://demo.routerplus.com/console/byok)
  and the playground take `?org=<UUID>` to switch. Manage the other identities with the API
  below.

Request rules:

- POSTs need `Origin: https://demo.routerplus.com` (origin only) and `Content-Type: application/json`.
- Responses are not cacheable.
- Adding a membership sends no invitation or email.
- RouterPlus normalizes membership and member-principal emails like account emails: lowercase,
  `+tag` removed, and Gmail dots and domain normalized.

| Role | Read org configuration | Manage workspaces, principals, keys, connections, policies, limits | Manage members |
|---|---|---|---|
| owner | Yes | Yes | Yes |
| admin | Yes | Yes | No |
| viewer | Yes | No | No |

This API cannot disable, demote or replace the initial owner.

### Workspace roles

A person also has a role in each workspace: `admin`, `member` or `viewer`.

- An org owner or admin is a workspace admin in every workspace.
- An org viewer is a workspace viewer, unless the workspace has a row for them. Then they get
  that row's role in that workspace only.
- Set a row with `PUT /v1/workspaces/<id>/members/<email>` and `{role}`.

| Workspace role | View usage | Create keys | Disable keys | Manage members and settings |
|---|---|---|---|---|
| admin | Yes | Yes | Yes | Yes |
| member | Yes | Yes | Yes | No |
| viewer | Yes | No | No | No |

- The workspace role applies to key and principal writes, in the console and in the API below.
- A key's workspace is the workspace of its principal.
- A write that the role does not allow returns `403` with the reason.
- The API keys page in the console shows your role in each workspace.
- Capping or copying a key needs an org owner or admin.
- Removing a membership removes that person's access to this organization. Their memberships in
  other organizations stay valid.

## Identity endpoints

Each list GET returns at most 500 entries. Use the returned UUIDs for later writes. A create
returns `201`. An update or a member write returns `200`.

| Request | JSON body / result |
|---|---|
| `GET /api/identity/organizations` | `{organizations:[{org_id,name,role}]}` |
| `GET /api/identity/members` | `{members:[{email,role,disabled}]}` |
| `POST /api/identity/members` | `{email,role:"admin"\|"viewer",disabled?:boolean}`. Owner only. Creates or updates. |
| `GET /api/identity/workspaces` | Workspace metadata and policy pointers |
| `POST /api/identity/workspaces` | `{name}` → `{workspace_id}` |
| `POST /api/identity/workspaces/<id>` | Any of `{name,disabled,route_policy_id}`. `null` clears its policy. |
| `GET /api/identity/principals` | Principal metadata |
| `POST /api/identity/principals` | `{workspace_id,kind:"service"\|"end_user"\|"member",subject,member_email?}` → `{principal_id}` |
| `POST /api/identity/principals/<id>` | `{disabled:boolean}` |
| `GET /api/identity/keys` | Metadata only, including workspace and principal bindings |
| `POST /api/identity/keys` | `{principal_id,name,connection_id?}` → `{key_id,key,copyable}`. 300 RPM. `copyable: true` means an owner or admin can copy the key again at `/console/keys`. It is `false` when the organization has copying turned off. |
| `POST /api/identity/keys/<id>` | `{disabled:boolean}`. Disabling deletes the key's copy, so it cannot be copied again. |

Principal fields:

- `subject` is a stable identifier from your backend. It is unique within a workspace and at
  most 160 characters. A leading `_` is reserved.
- `kind:member` with an existing member's `member_email` ties the principal to that member.
  Revoking the membership also stops their inference keys.
- An `end_user` or `service` principal does not depend on a membership. Disable it to revoke
  it.

With `connection_id`, RouterPlus checks that the connection is active and belongs to the same
organization. It then creates the key with that binding in one transaction, so an interrupted
BYOK setup never leaves a key that RouterPlus funds. Without `connection_id`, key creation works
as for any non-BYOK key.

### Revocation

- Disabling a workspace denies all its keys.
- Disabling a principal denies all its keys.
- Disabling a member denies their control access and their `member` principals.
- New requests see a revocation within 5 seconds. A stream already running can finish.
- A successful write returns only after every gateway can see it. If a write returns 503, check
  the state before you retry. The change may have saved anyway.

## Routes, limits and usage

The restrictions in force are the intersection of **org → workspace → key → request**. Attach a
[routing policy](/docs/routing-policies) to a workspace with
`POST /api/identity/workspaces/<id>` and `{route_policy_id:"..."}`. A workspace policy cannot
grant a route that an org or key policy denies.

The `/api/admission-limits` API accepts `scope:"workspace"` and `scope:"principal"`, with the
matching UUID in `subject`.

- The defaults for each are 300 RPM, 1,000,000 TPM and 8 concurrent requests.
- All org, key, model, platform and provider-pool limits also apply.
- These limits create no separate workspace or principal balances or spend caps. Money stays
  with the organization and the key.
- Set `outage_mode:"closed"` when your contract needs RouterPlus to refuse work while the
  counters are down.

Usage:

- `GET /v1/usage` and `/v1/generation?id=...` show the attempts of the calling principal and
  workspace, including its other keys.
- Unknown generations, and generations of other principals, return 404.
- Org balance fields are `null` for scoped principals. The default service principal keeps
  them.
- The console shows org-wide accounting to authorized members.
- Each attempt's route record holds `workspaceId`, `principalId` and up to three policy
  revisions.

---

Source: https://demo.routerplus.com/docs/oauth.md

# OAuth PKCE: let users connect your app

Your app can get a RouterPlus inference key for a user. The user does not copy a key. The user
signs in, selects a workspace and approves. Then your app receives the key.

The flow is OAuth 2.0 with PKCE (RFC 7636). It uses the common request and response shapes for
PKCE key exchange. To use an existing PKCE client, change its two URLs.

## Steps

1. Make a code verifier: 43 to 128 random characters from `A-Z a-z 0-9 - . _ ~`. Keep it in
   your app.
2. Make the code challenge: `base64url(sha256(verifier))`, with no padding.
3. Send the user's browser to the consent page:

```
https://demo.routerplus.com/auth?callback_url=https://yourapp.example.com/callback&code_challenge=CHALLENGE&code_challenge_method=S256
```

4. The user approves. The browser goes to your `callback_url`, with `?code=CODE` added.
5. From your app, exchange the code for a key:

```bash
curl -s https://demo-api.routerplus.com/api/v1/auth/keys \
  -H "Content-Type: application/json" \
  -d '{"code":"CODE","code_verifier":"VERIFIER","code_challenge_method":"S256"}'
```

```json
{"key": "tm_vk_...", "user_id": "user@example.com"}
```

6. Send the key as a Bearer token on inference calls: `Authorization: Bearer tm_vk_...`.

## Consent page parameters

| Parameter | Required | Meaning |
|---|---|---|
| `callback_url` | yes | Where the browser returns with the code. https only, except plain http on `localhost` and `127.0.0.1`. No user name, password or fragment. Its own query parameters are kept. |
| `code_challenge` | recommended | The PKCE challenge. Without it, the code alone is enough to get the key. |
| `code_challenge_method` | no | `S256` (recommended) or `plain`. Default `plain`. |
| `key_label` | no | The key's name on the user's API keys page, up to 80 characters. Default: your callback host. |
| `limit` | no | A spending limit for the key in USD, for example `25`. The user can change it before approval. |

The user sees your callback host, the workspace list and the limit. Owners and admins can
approve. Viewers cannot approve. A user who is not signed in signs in first and then returns to
the consent page.

## Exchange endpoint

`POST https://demo-api.routerplus.com/api/v1/auth/keys` (also `POST /v1/auth/keys`)

Request body, JSON:

| Field | Required | Meaning |
|---|---|---|
| `code` | yes | The `code` from the callback. |
| `code_verifier` | when a challenge was sent | The verifier you made in step 1. |
| `code_challenge_method` | no | If sent, it must equal the method on the consent request. |

Response `200`: `{"key": "...", "user_id": "<email of the user who approved>"}`. RouterPlus
returns the key one time. Store it in your app.

## Rules

- A code expires 10 minutes after approval.
- A code works one time. A second exchange of the same code fails and makes no key.
- A wrong `code_verifier` fails and leaves the code usable until it expires.
- The key belongs to the workspace that the user selected. Its spend goes to that organization,
  up to the limit.
- The user can disable the key at any time in the console, under Access > API keys.
- If the user stops being an owner or admin before the exchange, the exchange fails. The code
  is then used up.

## Errors

Errors have the shape `{"error": {"code": <status>, "message": "..."}}`.

| Status | Cause |
|---|---|
| `400` | No `code`; the code is unknown, expired or used; or `code_challenge_method` is not `S256` or `plain`, or differs from the consent request. |
| `403` | The `code_verifier` does not match the challenge, or the user or workspace can no longer make keys. |
| `405` | The method is not POST. |
| `503` | A temporary failure. Retry with the same code. |

---

Source: https://demo.routerplus.com/docs/workload-identity.md

# Workload identity: keys without stored secrets

A workload sends an OIDC token from its platform and gets a short-lived RouterPlus inference
key back. You store no RouterPlus key in CI or in the cluster.

Any issuer that publishes OIDC discovery and a key set over https works. Examples are GitHub
Actions, Google Cloud, AWS and Kubernetes service account tokens.

## Trusts

Set up one trust for each workload with the
[management API](api-management.md#workload-identities). A trust has these fields:

| Field | Meaning |
|---|---|
| `issuer` | The token's `iss`, an https URL. For GitHub Actions: `https://token.actions.githubusercontent.com`. |
| `audience` | A value that the token's `aud` must hold. You choose the value, and you request tokens for it. |
| `subject_pattern` | A pattern that the token's `sub` must match. `*` matches any run of characters. `?` matches one character. The pattern matches the whole `sub`. |
| `required_claims` | Claims the token must carry, as `{"claim": "value"}`. RouterPlus compares values as text. Use a dotted path for a nested claim, for example `kubernetes.io.namespace`. |
| `key_ttl_seconds` | How long the key works: 300 to 86400 seconds. Default 3600. |
| workspace and principal | Where the key's requests and spend go. |

Each trust has an `identity_id`. A disabled trust accepts no tokens.

## Exchange endpoint

`POST https://demo-api.routerplus.com/api/v1/auth/token` (also `POST /v1/auth/token`)

Request body, JSON:

```json
{"token": "<OIDC JWT>", "identity_id": "<trust id, optional>"}
```

The endpoint also accepts the RFC 8693 form (`application/x-www-form-urlencoded`):
`grant_type=urn:ietf:params:oauth:grant-type:token-exchange`, `subject_token=<JWT>`,
`subject_token_type=urn:ietf:params:oauth:token-type:jwt` (or `...:id_token`).

Without `identity_id`, RouterPlus tries the enabled trusts for the token's issuer, oldest
first. The first trust that matches makes the key.

Response `200`:

```json
{
  "access_token": "tm_vk_...",
  "token_type": "Bearer",
  "expires_in": 3600,
  "issued_token_type": "urn:ietf:params:oauth:token-type:access_token"
}
```

Send `access_token` as a Bearer token on inference calls. It stops working after `expires_in`
seconds. To get a new key, exchange a new token.

## What RouterPlus checks

- The signature, with the issuer's published keys. Accepted algorithms: RS256, RS384, RS512, PS256, ES256, ES384. RouterPlus refuses `none`, HS256, HS384 and HS512.
- `exp` (required), `nbf` and `iat`, with 60 seconds of clock skew.
- `aud` holds the trust's audience. `aud` can be a string or a list.
- `sub` matches the trust's pattern.
- Each required claim equals the token's claim.
- The token was not exchanged before. A token (its `iss` and `jti`) works one time.

## Errors

| Status | Body |
|---|---|
| `400` | `{"error": "invalid_request", "error_description": "..."}`: no token, a bad `identity_id`, or a bad `grant_type` or `subject_token_type`. |
| `401` | `{"error": "invalid_token", "error_description": "Token rejected: <reason>."}` |
| `405` | `{"error": "invalid_request", ...}`: the method is not POST. |
| `503` | `{"error": "temporarily_unavailable", ...}`: retry. |

The `401` reasons:

| Reason | Meaning |
|---|---|
| `malformed` | The token is not a valid JWT, is longer than 16,384 characters, or has no `exp` or https `iss`. |
| `unsupported_alg` | The algorithm is not on the accepted list. |
| `unknown_identity` | No enabled trust exists for this issuer, or for this `identity_id`. |
| `issuer_unavailable` | RouterPlus could not read the issuer's discovery document or key set. |
| `signature` | The signature does not verify with the issuer's keys. |
| `expired`, `not_yet_valid`, `issued_in_future` | A time check failed. |
| `not_trusted` | No trust matched the audience, subject and required claims. |
| `replayed` | This token was already exchanged. |

## GitHub Actions example

1. Set up a trust with issuer `https://token.actions.githubusercontent.com` and audience
   `routerplus`.
2. Set a subject pattern, for example `repo:acme/app:ref:refs/heads/main`.
3. Add `repository_owner: acme` as a required claim.
4. Give the job the `id-token: write` permission.

```yaml
permissions:
  id-token: write
  contents: read

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - name: Get a RouterPlus key
        run: |
          OIDC=$(curl -sS -H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
            "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=routerplus" | jq -r .value)
          KEY=$(curl -sS https://demo-api.routerplus.com/api/v1/auth/token \
            -H "Content-Type: application/json" \
            -d "{\"token\":\"$OIDC\",\"identity_id\":\"YOUR_IDENTITY_ID\"}" | jq -r .access_token)
          echo "::add-mask::$KEY"
          echo "TM_API_KEY=$KEY" >> "$GITHUB_ENV"
      - name: Call a model
        run: |
          curl -sS https://demo-api.routerplus.com/v1/chat/completions \
            -H "Authorization: Bearer $TM_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{"model":"MODEL_ID","messages":[{"role":"user","content":"Hello"}]}'
```

GitHub Actions subjects have one of these forms:

- `repo:<owner>/<repo>:ref:refs/heads/<branch>`
- `repo:<owner>/<repo>:environment:<name>`
- `repo:<owner>/<repo>:pull_request`

---

Source: https://demo.routerplus.com/docs/bedrock.md

# Bedrock workload identity

The `bedrock` connection profile calls AWS Bedrock in your AWS account. RouterPlus signs each
call with AWS SigV4, using temporary credentials from AssumeRole on a role you own. You store no
AWS key with RouterPlus.

| Supported | Not supported |
|---|---|
| Regional Anthropic Claude Messages through `InvokeModel` and `InvokeModelWithResponseStream`, on both buyer surfaces | A general Converse adapter; `count_tokens` |
| Direct `anthropic.claude-...-vN:N` model ids | Cross-region or global inference profiles, profile ARNs, provisioned-throughput ARNs and other endpoints (rejected) |

The model mapping and the AWS region are part of the connection and cannot change. Check that
the model is available and that your account has access in the region you choose.

## Set up the AWS role

1. Create a role in your account. Its trust policy names the RouterPlus source role and
   requires `sts:ExternalId` equal to `tm:<org UUID>`.
2. Allow `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` only on the regional
   foundation-model ARNs you approve.
3. Ask RouterPlus to grant your organization that role ARN. The console offers the Bedrock
   profile only to an organization with a grant.

How RouterPlus operators grant a role:

- The gateway gets its own AWS credentials from its instance role through IMDSv2. It uses no
  static AWS key from the environment.
- The instance role gets `sts:AssumeRole` on your exact role ARN.
- The grant map is set on both gateway processes, and takes effect after a restart:

```sh
TM_BEDROCK_ROLE_GRANTS='{"ORG_UUID":["arn:aws:iam::123456789012:role/tm-customer"]}'
```

- An organization cannot register or use another organization's role without a grant.
- The grant map does not replace AWS IAM authorization. Keep trust grants least-privilege, with
  a distinct external ID per organization.

## Create and validate

Send `POST /api/connections` with the same session and Origin headers as other
[connection management](/docs/byok) calls. In the console, open Access > Provider connections.
Replace the example model with a regional model your account can invoke.

```json
{
  "profile": "bedrock",
  "models": ["your-public-model-id"],
  "endpoint_config": {
    "region": "us-east-1",
    "deployments": {
      "your-public-model-id": "anthropic.claude-3-haiku-20240307-v1:0"
    }
  },
  "workload_role": {
    "role_arn": "arn:aws:iam::123456789012:role/tm-customer"
  }
}
```

`workload_role` is write-only. RouterPlus encrypts it with envelope encryption. Do not send
`api_key`, AWS access keys, an external ID override, a bearer token or a request-time
credential.

Then:

1. Call `POST /api/connections/<id>/validate`. It checks that RouterPlus can assume the role,
   and makes no billable completion. An active result does not prove model access, capacity,
   private connectivity, processing residency or ZDR.
2. Declare a quota pool with `provider:"bedrock"` and bind it.
3. Bind the connection or a policy to a virtual key.
4. Run one small real request to confirm model permissions and streaming.

| Operation | Behavior |
|---|---|
| `/rotate` with `{workload_role:{role_arn:"..."}}` | Revokes the old connection version and returns the connection to pending. The account pool stays the same. |
| Disable or delete | Works as for other connections. |

## Runtime behavior

Credentials:

- STS uses the regional endpoint, 15-minute sessions and an external ID bound to your
  organization.
- Temporary credentials live only in process memory, with at most 256 sessions and 64
  concurrent refreshes.
- They refresh at least one minute before expiry.
- They are never written to a database, cache, log or telemetry.

Requests:

- The AWS clients make one attempt each. The gateway owns retries. SDK retries and endpoint
  overrides from the environment are off.
- The gateway records the request before it assumes the role and calls the model. Authorization
  and admission checks still run before the signed call.
- A credential failure before inference counts as never dispatched.
- When the outcome of the inference call is uncertain, the normal usage reservation stays.
- The gateway turns AWS binary events into the Anthropic stream. After output starts, an error
  ends the request. There is no silent retry.
- The gateway does not relay AWS exception text, because it can echo sensitive input.

Billing and privacy:

- [BYOK billing](/docs/byok#billing) applies: RouterPlus charges `0`.
- The AWS cost is unknown to RouterPlus.
- A regional endpoint is not a ZDR guarantee. Privacy and residency route filters still need
  provider facts that RouterPlus operators record.

References: [AWS Claude request format](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages-request-response.html),
[stream operation](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModelWithResponseStream.html),
[Bedrock PrivateLink](https://docs.aws.amazon.com/bedrock/latest/userguide/vpc-interface-endpoints.html).

---

Source: https://demo.routerplus.com/docs/sso.md

# Single sign-on

With single sign-on (SSO), your members sign in to the RouterPlus console through your identity
provider. RouterPlus supports OpenID Connect (OIDC) and SAML 2.0, with Okta, Microsoft Entra ID
and Google Workspace.

An organization can have several connections. Each connection signs in the emails of its
domains. For example, `acme.com` signs in through Okta and `acme.co.uk` through Entra ID.

- Members sign in at `https://demo.routerplus.com/sso` with their work email. RouterPlus sends them to the
  connection for that domain.
- The sign-in link for one connection is
  `https://demo.routerplus.com/sso/start?connection=<connection id>`.
- Email links and the normal sign-in page continue to work.

In the console, open Team > Single sign-on.

## Set up a connection

1. Create the application in your identity provider. See the sections below.
2. Create the connection with a management key: `POST https://demo-api.routerplus.com/v1/sso/connections`.
3. Add the DNS TXT record that the answer shows for each domain.
4. Verify the domains: `POST https://demo-api.routerplus.com/v1/sso/connections/<id>/verify-domains`.
5. Add your members to the organization, or turn on just-in-time provisioning.

A connection signs in no one until you verify its domain.

## The URLs your identity provider needs

| Use | URL |
|---|---|
| OIDC redirect URI (sign-in redirect URI) | `https://demo.routerplus.com/sso/oidc/callback` |
| SAML assertion consumer service (ACS, reply URL) | `https://demo.routerplus.com/sso/saml/acs` |
| SAML audience (entity id of RouterPlus) | the connection's `sp_entity_id` |
| SAML metadata of RouterPlus | `https://demo.routerplus.com/sso/saml/metadata?connection=<connection id>` |

## Create a connection

OIDC:

```bash
curl https://demo-api.routerplus.com/v1/sso/connections \
  -H "Authorization: Bearer $ROUTERPLUS_MANAGEMENT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "oidc",
    "name": "Okta",
    "email_domains": ["acme.com"],
    "issuer": "https://acme.okta.com",
    "client_id": "0oa1b2c3d4",
    "client_secret": "...",
    "default_role": "viewer",
    "jit_provisioning": false
  }'
```

SAML:

```bash
curl https://demo-api.routerplus.com/v1/sso/connections \
  -H "Authorization: Bearer $ROUTERPLUS_MANAGEMENT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "saml",
    "name": "Entra ID",
    "email_domains": ["acme.com"],
    "idp_entity_id": "https://sts.windows.net/<tenant id>/",
    "idp_sso_url": "https://login.microsoftonline.com/<tenant id>/saml2",
    "idp_certificate": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----"
  }'
```

The response is `201`, with the connection in `data`.

### Fields

| Field | Kind | Meaning |
|---|---|---|
| `kind` | both | `oidc` or `saml`. You cannot change it later. |
| `name` | both | A label, 1 to 100 characters. |
| `email_domains` | both | 1 to 50 domains. The connection signs in only emails at these domains. |
| `enabled` | both | `false` stops all sign-ins through the connection. Default `true`. |
| `default_role` | both | `viewer` or `admin`. The role a member gets from just-in-time provisioning. Default `viewer`. |
| `jit_provisioning` | both | `true` adds a new email to the organization at its first sign-in. Default `false`. |
| `issuer` | OIDC | The issuer URL of your identity provider, `https` only. RouterPlus reads `<issuer>/.well-known/openid-configuration`. |
| `client_id` | OIDC | The client id of the application. |
| `client_secret` | OIDC | The client secret. Leave it out for a public client with PKCE only. |
| `allowed_tenant_ids` | OIDC | Up to 50 Entra ID tenant ids that can sign in. See [Tenant filtering](#tenant-filtering). |
| `idp_entity_id` | SAML | The issuer (entity id) of your identity provider. |
| `idp_sso_url` | SAML | The single sign-on URL of your identity provider, `https` only. |
| `idp_certificate` | SAML | The signing certificate of your identity provider, in PEM or as bare base64. RSA keys only. |
| `sp_entity_id` | SAML | The audience that your identity provider puts in the assertion. Default `urn:routerplus:sso:<connection id>`. |

RouterPlus stores the client secret encrypted and never returns it. A connection with a secret
shows `"client_secret": "__secret__"`.

### Routes

All routes take a management key. They also work under `/api/v1`.

| Route | What it does |
|---|---|
| `GET /v1/sso/connections` | Lists your organization's connections. |
| `POST /v1/sso/connections` | Creates a connection. |
| `GET /v1/sso/connections/<id>` | Shows one connection. |
| `PATCH /v1/sso/connections/<id>` | Changes the fields you send. Send `"client_secret": null` to remove the secret. |
| `DELETE /v1/sso/connections/<id>` | Deletes the connection. |
| `POST /v1/sso/connections/<id>/verify-domains` | Looks up the DNS records and marks the domains verified. |

RouterPlus writes every change to the audit log (`GET /v1/audit`).

## Verify your domains

Each domain in the response has a `verification` object:

```json
{
  "domain": "acme.com",
  "verified": false,
  "verification": { "type": "TXT", "name": "_routerplus-sso.acme.com", "value": "routerplus-sso=4f1c..." }
}
```

1. At your DNS provider, add a TXT record with that name and value.
2. Call `verify-domains`. The response lists each domain with `verified`. A domain that fails
   also has a `reason`.
3. If a domain fails, call `verify-domains` again after your DNS change propagates.

- In your organization, a domain belongs to one connection.
- A verified domain belongs to one organization. No other organization can verify or add it.
- A domain stays verified when you change the connection's other fields. A domain that you
  remove loses its verification.

## Okta

OIDC:

1. In the Okta admin console, open **Applications**, then **Create App Integration**.
2. Select **OIDC - OpenID Connect** and **Web Application**.
3. Set the sign-in redirect URI to `https://demo.routerplus.com/sso/oidc/callback`.
4. Assign the application to your people or groups.
5. Copy the client id and client secret. The issuer is your Okta domain, for example
   `https://acme.okta.com`. With a custom authorization server, the issuer is that server's
   URL, for example `https://acme.okta.com/oauth2/default`.

SAML:

1. Create an app integration and select **SAML 2.0**.
2. Set **Single sign-on URL** to `https://demo.routerplus.com/sso/saml/acs`.
3. Set **Audience URI (SP Entity ID)** to the connection's `sp_entity_id`.
4. Set **Name ID format** to `EmailAddress` and **Application username** to `Email`.
5. Open **View SAML setup instructions**. Copy the identity provider issuer, the single sign-on
   URL and the X.509 certificate.

To use the default `sp_entity_id`, create the connection first. Then add the Okta fields with
`PATCH`.

## Microsoft Entra ID

OIDC:

1. In the Entra admin center, open **App registrations**, then **New registration**.
2. Add a **Web** redirect URI: `https://demo.routerplus.com/sso/oidc/callback`.
3. Under **Certificates & secrets**, create a client secret.
4. Under **Token configuration**, add the optional claim `email` to the ID token.
5. The issuer for a single-tenant app is `https://login.microsoftonline.com/<tenant id>/v2.0`.

Entra ID does not send `email_verified`. Make sure that each user's `email` attribute
holds their real work email.

SAML:

1. Open **Enterprise applications**, then **New application**, then
   **Create your own application**.
2. Under **Single sign-on**, select **SAML**.
3. Set **Identifier (Entity ID)** to the connection's `sp_entity_id`.
4. Set **Reply URL (Assertion Consumer Service URL)** to `https://demo.routerplus.com/sso/saml/acs`.
5. Download **Certificate (Base64)**. Copy **Login URL** (the `idp_sso_url`) and
   **Microsoft Entra Identifier** (the `idp_entity_id`).

RouterPlus reads the email from the NameID when it is an email. Otherwise it reads the
`emailaddress` claim.

## Google Workspace

OIDC:

1. In Google Cloud console, open **APIs & Services**, then **Credentials**.
2. Create an **OAuth client ID** of type **Web application**.
3. Add the authorized redirect URI `https://demo.routerplus.com/sso/oidc/callback`.
4. The issuer is `https://accounts.google.com`.

Set the OAuth consent screen to **Internal**. Then only your Workspace accounts can sign in.

SAML:

1. In the Google Admin console, open **Apps**, then **Web and mobile apps**, then
   **Add custom SAML app**.
2. Copy the SSO URL, the entity id and the certificate.
3. Set **ACS URL** to `https://demo.routerplus.com/sso/saml/acs` and **Entity ID** to the connection's
   `sp_entity_id`.
4. Set **Name ID format** to `EMAIL` and **Name ID** to **Primary email**.
5. Turn the app on for your organizational units.

## Tenant filtering

A connection signs in a person only when both conditions are true:

- The email's domain is one of the connection's verified `email_domains`.
- For Entra ID with `allowed_tenant_ids` set, the `tid` claim is in that list.

A multi-tenant Entra ID app uses the issuer
`https://login.microsoftonline.com/organizations/v2.0` or `/common/v2.0`. Any Entra tenant can
sign in to such an app, and a tenant admin can set any email on a user. Thus a connection with a
multi-tenant issuer needs `allowed_tenant_ids`. Without the list, the connection refuses every
sign-in.

## Just-in-time provisioning

- `jit_provisioning` off: only existing members sign in. Add members in the console or with
  [SCIM](/docs/scim).
- `jit_provisioning` on: a person with an email at a verified domain joins the organization at
  the first sign-in, with `default_role` (`viewer` or `admin`).

SSO never makes an owner. A member that an admin turned off stays off.

## What a sign-in checks

OIDC:

- Authorization code flow with PKCE (`S256`), `state` and `nonce`.
- The ID token signature, against the identity provider's published keys.
- `iss`, `aud` (your `client_id`), `exp`, `nbf`, `iat` and `nonce`.
- `email_verified` must be `true` when the claim is present.

SAML:

- RouterPlus sends an AuthnRequest with the HTTP-Redirect binding. It reads the response with the
  HTTP-POST binding.
- The response or the assertion must be signed with RSA-SHA256 and exclusive canonicalization.
  Every signature in the response must verify with `idp_certificate`.
- `Destination`, `Recipient`, `Audience`, `InResponseTo`, `NotBefore` and `NotOnOrAfter` must
  match.
- RouterPlus accepts an assertion one time.
- RouterPlus refuses encrypted assertions, and XML with a DOCTYPE or entity declarations.

The sign-in must finish in the browser that started it, within 10 minutes. Sign-ins start only
from RouterPlus. RouterPlus refuses an unsolicited (IdP-initiated) SAML response.

## Errors

The sign-in page shows the reason.

| Message | Fix |
|---|---|
| Single sign-on is not set up for this email domain | Add the domain to a connection, verify it and set `enabled` to `true`. |
| This connection does not sign in emails at `<domain>` | Add the domain to the connection and verify it. |
| You are not a member of this organization | Add the person as a member, or turn on `jit_provisioning`. |
| Your account in this organization is turned off | An admin turned the member off. Turn the member on again. |
| The ID token has no email claim | Add the `email` scope or the `email` optional claim. |
| Your identity provider has not verified this email address | Verify the email in the identity provider. |
| Your Entra ID tenant is not allowed | Add the tenant id to `allowed_tenant_ids`. |
| Set allowed tenant ids on the connection | A multi-tenant Entra ID issuer needs `allowed_tenant_ids`. |
| The identity provider did not complete the sign-in | Check `client_id`, `client_secret` and the redirect URI. |
| The SAML response was refused: ... audience ... | Set the audience in the identity provider to `sp_entity_id`. |
| The SAML response was refused: ... signature ... | Upload the identity provider's current signing certificate. |
| The SAML response was refused: ... encrypted assertions ... | Turn assertion encryption off. |
| This sign-in request has expired or was already used | Start the sign-in again. |
| This sign-in was started in another browser | Start the sign-in again in this browser. |

---

Source: https://demo.routerplus.com/docs/scim.md

# SCIM provisioning

With SCIM, your identity provider creates, updates and deactivates the members of your
organization, and pushes groups. Okta, Microsoft Entra ID and any SCIM 2.0 client work. A group
mapping turns group membership into a workspace role, an org role and a guardrail.

In the console, open Team > Single sign-on for the SCIM base URL and token.

## Connect your identity provider

| Setting | Value |
|---|---|
| SCIM base URL | `https://demo-api.routerplus.com/scim/v2` |
| Authentication | HTTP header, Bearer token |
| Token | A management key of the organization (`tm_mk_...`) |
| Unique identifier for users | `userName` (the email address) |
| Supported actions | Create users, update user attributes, deactivate users, push groups |

- Create a management key with `POST /v1/management-keys`. See the
  [management API](api-management.md). Use the key for this connection only. To stop
  provisioning, disable the key.
- The token selects the organization. A user or group id of another organization returns `404`.
- A missing, wrong, disabled or expired token returns `401`.
- In Okta, set the user name format to the email address.
- In Entra ID, map `userPrincipalName` or `mail` to `userName`.

## Users

A SCIM user is a member of the organization.

| Request | What it does |
|---|---|
| `POST /Users` | Adds the member with the org role `viewer`, or links the existing member with that email. Returns `201`. |
| `GET /Users?filter=userName eq "a@b.com"` | Finds a user. `startIndex` (from 1) and `count` (at most 200) page through the list. |
| `GET /Users/:id` | Reads one user. |
| `PUT /Users/:id` | Replaces `externalId`, `displayName`, `name` and `active`. If you leave out `active`, it keeps its value. |
| `PATCH /Users/:id` | `add`, `replace` or `remove` operations, with or without a `path`. |
| `DELETE /Users/:id` | Deactivates the member and removes it from every group. Returns `204`. |

Rules:

- `userName` is the member's email. RouterPlus lowercases it and removes a `+tag`. You
  cannot change `userName`. For a new email, create a new user.
- `active: false` disables the member. The member loses access to the organization. Keys bound
  to the member stop working.
- After `DELETE`, the user id returns `404`. RouterPlus keeps the member's history. A later
  `POST` with the same email enables the member again, with a new id.
- SCIM can link the org owner. It never deactivates or deletes the owner, and never changes the
  owner's role. Those requests return `400` with `scimType: "mutability"`.
- `active` accepts `true`, `false`, `"True"` and `"False"`.
- RouterPlus accepts and ignores attributes other than `userName`, `externalId`, `displayName`,
  `name`, `active` and `emails`. `emails` must match `userName`.

## Groups

| Request | What it does |
|---|---|
| `POST /Groups` | `{displayName, externalId?, members?: [{value: <user id>}]}`. Returns `201`. |
| `GET /Groups?filter=displayName eq "Engineering"` | Finds a group. `excludedAttributes=members` leaves out the members. |
| `GET /Groups/:id` | Reads one group. |
| `PUT /Groups/:id` | Replaces `displayName`, `externalId` and the member list. |
| `PATCH /Groups/:id` | Adds or removes members, or replaces `displayName`. |
| `DELETE /Groups/:id` | Deletes the group and its mapping. Returns `204`. |

These PATCH forms work:

```json
{"schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
 "Operations": [
   {"op": "add", "path": "members", "value": [{"value": "<user id>"}]},
   {"op": "remove", "path": "members[value eq \"<user id>\"]"},
   {"op": "remove", "path": "members", "value": [{"value": "<user id>"}]},
   {"op": "replace", "value": {"displayName": "Platform"}}
 ]}
```

Each member must be a user of the same organization. Otherwise the request returns `400` with
`scimType: "invalidValue"`.

## Filters

Filters use `eq`, joined by `and`.

| Resource | Attributes |
|---|---|
| Users | `userName`, `externalId`, `id`, `displayName`, `active`, `emails.value`, `emails[type eq "work"].value` |
| Groups | `displayName` (case is ignored), `externalId`, `id`, `members[value eq "<user id>"]` |

Other operators, `or` and `not` return `400` with `scimType: "invalidFilter"`.

## Group mappings

A mapping gives each member of a group these values:

- `workspace_id` and `workspace_role` (`admin`, `member` or `viewer`): the member's role in
  that workspace. Give both or neither.
- `org_role` (`admin` or `viewer`): the member's org role.
- `guardrail_id`: a guardrail assigned to the member.

A mapping needs at least one of the three.

| Route | What it does |
|---|---|
| `GET /v1/scim/groups` | Lists the SCIM groups with their member count and mapping. |
| `GET /v1/scim/groups/:id/mapping` | Reads a group's mapping. Returns `404` when the group has none. |
| `PUT /v1/scim/groups/:id/mapping` | `{workspace_id?, workspace_role?, org_role?, guardrail_id?}`. Sets the mapping and applies it to the members. |
| `DELETE /v1/scim/groups/:id/mapping` | Removes the mapping and applies the change to the members. |

These routes use [management API](api-management.md) authentication. They are served
under `/v1/...` and `/api/v1/...`.

How mappings apply:

- Every change applies immediately. Changes include a member that joins or leaves a group, a
  changed mapping, and a deleted group or user.
- For a member of several mapped groups, the strongest workspace role wins in each workspace.
  For the org role, `admin` wins over `viewer`.
- When several of a member's groups name a guardrail, the group with the lowest id wins. Ids
  compare as lowercase text.
- When no group grants a value any more, the member gets back the value from before SCIM set it.
  If someone changed the value by hand after that, SCIM does not change it.
- A member that SCIM added has the org role `viewer` when no group grants an org role.
- A mapping never changes the org owner.
- A deleted guardrail never applies.

## Sign-in audit

`GET /v1/audit/sign-ins` lists the sign-ins and sign-outs of the organization's members,
newest first.

| Query | What it does |
|---|---|
| `since`, `until` | ISO 8601 times. `since` is inclusive, `until` is exclusive. |
| `email` | Only this member. |
| `limit` | 1 to 500. Default 100. |

Each row is `{at, email, event, source}`.

| `event` | Meaning |
|---|---|
| `sign_in` | The member signed in to the console. |
| `sign_out` | The member signed out. |
| `session_revoked` | The session ended for another reason, for example the identity provider ended it. |

| `source` | Meaning |
|---|---|
| `clerk` | Single sign-on. All three events are recorded. |
| `email_link` | An email sign-in link. Only `sign_in` is recorded. |

Each event has one row for each organization that the person is a member of. Disabled
memberships are included.

## Errors

Errors use the SCIM error form, with `content-type: application/scim+json`:

```json
{"schemas": ["urn:ietf:params:scim:api:messages:2.0:Error"], "status": "409", "scimType": "uniqueness", "detail": "a user with this userName exists"}
```

## Discovery

`GET /ServiceProviderConfig`, `GET /ResourceTypes` and `GET /Schemas` need no token.
RouterPlus supports PATCH and filters. It does not support bulk, sort, ETags or password
changes.

---

Source: https://demo.routerplus.com/docs/data-policy.md

# Data policy

RouterPlus sits between your application and model providers. This page lists what RouterPlus
stores, for how long, and what each provider keeps.

- RouterPlus records request metadata for billing and operations.
- RouterPlus records request and response content from non-dedicated endpoints for debugging.
  It keeps these request logs in a private, encrypted storage bucket for up to 30 days.
- Request logging excludes dedicated endpoints and private installations, also when a dedicated
  endpoint falls back to a public model. It also excludes organizations that opt out.
- [Optimize](/docs/model-search) stores the eval cases and candidate answers of a search, for
  your organization only, until you delete the search.
- Each provider has its own retention policy. See
  [What providers do with your prompts](#what-providers-do-with-your-prompts).

## Request data

Request content goes only to the request log. Billing and analytics hold no content:

| Store | What it holds | Content? |
|---|---|---|
| Request log | Requests and responses from non-dedicated endpoints, with recognized credentials removed and size limits applied, for up to 30 days ([details](#request-logging)) | Yes |
| Process logs | Request id, surface, model, outcome, HTTP status. Every buyer-controlled string is sanitized first. | Never |
| Telemetry (analytics) | Ids, org and key ids, provider, deployment, model, region, attempt ordinal, outcome, error code, statuses, timestamps, latency, five token counts, request and response *byte sizes*, cost, provider request id (house routes only, see below). Also: the calling software's name, version and User-Agent header (keys and email addresses removed); the finish reason; the output limit sent and whether the gateway set it; the number and types of tools (never their names or schemas); whether the request was cancelled before the first token; and the provider's error code with its message scrubbed of content (quoted text, names you chose, values, keys and addresses removed). | Never |
| Billing ledger | One row per attempt: the prices in force, reserved maximum, settled cost, token counts, HTTP status, error code, provider request id (house routes only, see below), timestamps, route and admission context | Never |
| Rendered pages | Model directory, model pages, console, status. All render from metadata. | Never |

Details:

- **Process logs include model names**, as routing metadata. RouterPlus sanitizes the
  buyer-controlled strings in those logs: it strips control characters and caps the length.
  Request content goes to the request log as structured records.
- **Error bodies never echo your input.** Every error body is the gateway's own typed shape.
  RouterPlus never relays a provider's raw error body. When a refusal is translated across
  dialects or sent mid-stream, the gateway sends only the typed error class.
- **Telemetry records byte sizes, not bodies.** It keeps `request_bytes` and `response_bytes` to
  measure throughput.
- **Image requests follow the request-log policy.** On
  [`/v1/images/generations`](/docs/api-images), prompts, generated image content and any
  `revised_prompt` can appear in request logs, within the size limits below. Telemetry and the
  billing ledger do not hold them.
- **Video creation requests follow the request-log policy.** The prompt and creation response
  can appear in request logs. The video file streams from the provider when you download it.
  RouterPlus does not keep it.
- **The Playground keeps its chat history in your browser:** conversations in local storage and
  images in IndexedDB, filed by your sign-in, the conversation and the reply. Requests you send
  from the Playground follow the request-log policy. Deleting a chat, or **Clear all**, deletes
  the browser's copy. It does not delete a request log already recorded.
- **Playground decisions follow the request-log policy.** Their state, questions, options and
  answers can appear in request logs. Telemetry and the billing ledger keep metadata such as
  model, provider, token counts and cost.
- **Request logs exclude authentication headers.** The gateway removes recognized API-key formats
  and the request's provider credentials before it writes a record. It authenticates API keys by
  comparing a SHA-256 hash. See the redaction limits below.
- **The provider request id is kept on house routes only.** A provider gives each call this
  reference number, so you can trace a failed call with that provider. A house route runs on
  RouterPlus's own provider account. A call through a provider credential you connect (BYOK)
  never keeps the id. A provider can put the API key into that header, so the gateway drops an
  id that contains the key it sent, or any 16 characters of the key in a row. It also drops an
  id longer than 256 characters.

> [!NOTE]
> For local development, `TM_UNSAFE_LOG_BODIES=1` prints request bodies to stdout.
> The flag refuses to activate outside the `local` and `demo` regions. A production
> deployment that sets it gets a logged warning and no body logging. In every region,
> it never touches telemetry or the ledger, and it never prints an image prompt.

Tests check that content stays out of process logs, telemetry and the billing ledger. They also
check that request logging respects endpoint exclusions, opt-outs, credential redaction and size
limits.

## What we do store

To run an account, RouterPlus stores this data:

| Data | Why | Form |
|---|---|---|
| Email address | Sign-in and billing receipts | Canonicalized: lowercased, Gmail dots and `+tags` collapsed, `+tags` stripped elsewhere. Aliases share one account and one promotional-credit eligibility. |
| Members' emails | Organization membership and roles | Canonicalized the same way |
| API keys | Gateway authentication, and copying a key again from the console | SHA-256 hash plus the first 12 characters, so the console can show you *which* key. The gateway checks only the hash. Keys created since 17 October 2026 also keep an envelope-encrypted copy (AES-256-GCM, its key held outside the database), so an owner or admin of your organization can copy the key again. Each copy is recorded. Disabling a key deletes its copy. An organization can ask RouterPlus to turn copying off. RouterPlus then deletes its copies and keeps no new ones. |
| Legacy and operator-issued email-link tokens | Redeeming sign-in links issued earlier | SHA-256 hash only, single use, 30-minute expiry |
| Clerk identity and session references | Connect verified sign-ins to the same account, and sign-out | Clerk issuer, user and session ids, verified canonical email, organization id and session timestamps |
| Authentication email delivery ids | Prevent duplicate sends when the Clerk email relay is on | Message id, outcome and timestamp only. No email body, sign-in link or recipient. |
| Per-attempt metadata | Billing and the usage you read back | The ledger row described above |
| Provider credentials you connect (BYOK) | Routing through your own OpenAI, Anthropic, Azure or Bedrock account | Envelope-encrypted. No API or page returns them. A Bedrock connection stores a role ARN, never AWS keys. |
| Braintrust API key (Optimize) | Reading your evals | Encrypted. Only its last four characters are shown. |
| Optimize searches | Comparing models on your evals | The eval cases and the candidates' answers, scoped to your organization, deleted with the search |
| Video jobs | Following a render across reloads | Job id, model, length, shape, status and charge. Never the prompt or the file. |
| App attribution | Optional `HTTP-Referer` and `X-Title` headers you send | Stored as sent, in telemetry. Send them only if you want your app identified. |
| Request log | Find and fix errors on non-dedicated endpoints | Requests and responses, recognized credentials removed, size-limited, encrypted, deleted after 30 days ([details](#request-logging)) |

RouterPlus does not store passwords. Clerk handles passwords, email verification and Google
sign-in.

- Google sign-in requests basic identity information (OpenID, email and profile). It does not
  request access to Gmail, Drive or Calendar.
- RouterPlus uses the verified email to find your existing account, including accounts first
  created through an email link.
- Clerk processes authentication data under its
  [privacy policy](https://clerk.com/legal/privacy).

Resend can deliver authentication emails, including Clerk-generated email links. The relay
verifies the sender's signature. It does not log or keep the email body or sign-in link. Resend
processes delivery data under its [privacy policy](https://resend.com/legal/privacy-policy).

## Request logging

| Topic | Rule |
|---|---|
| **Which requests** | Model requests to non-dedicated endpoints: chat, messages, responses, compaction, image generation, decisions, video creation and token counting. Dedicated endpoints are excluded, also when a request is refused or falls back to a public model. Private installations and organizations that opt out are excluded too. |
| **What is stored** | The request as the gateway received it, the upstream request and response details it has, and the response returned to the caller (JSON or streamed content). Also the request metadata above, selected client headers such as `User-Agent`, and your key's first 12 characters, which the console shows too. |
| **Size** | A record is normally at most 2 MiB, and up to 16 MiB for a configured program. Large fields can be truncated. Logging is best effort, so not every request has a complete record. |
| **What is removed** | The `Authorization` and `x-api-key` headers, cookies, and headers outside the request log's allowlist. The gateway also replaces with `[REDACTED]`: keys in recognized formats (RouterPlus, OpenAI, Anthropic, AWS, Google, GitHub, Stripe, Slack and private keys) and the provider credential the request used, including one you connect (BYOK). Other secrets in your prompts can be stored as you sent them. |
| **Where, and how long** | An encrypted, non-public storage bucket in RouterPlus's AWS account. The gateway can write records but cannot read them. Records are deleted after 30 days. |
| **Why** | To investigate errors and replay requests in tests, so RouterPlus can fix their cause. RouterPlus does not train models on these records and does not share them. |
| **Opt out** | Contact support to stop request logging for your organization. This does not change the provider's own retention policy, and it does not delete records already kept. |

## Hackathons and similar programs

Requests from accounts in a hackathon or similar program are stored for up to 30
days to find and fix errors.

Program invitations show this notice. A program's logging mark has an end date. After that date,
requests to non-dedicated endpoints can still be logged under the general policy above.
Dedicated endpoints, private installations and organization opt-outs stay excluded.

## Account authentication

Public signup and sign-in use Clerk in the browser at [Sign up](https://demo.routerplus.com/signup) and
[Sign in](https://demo.routerplus.com/login). Complete the email-verification and authentication steps
there. There is no magic-link submission or programmatic signup route. Clerk connects a verified
email to the same account, so existing accounts and API keys stay as they are.

Sessions:

- After Clerk verifies a sign-in, RouterPlus issues its own seven-day signed session cookie
  (`HttpOnly`).
- **Sign out** revokes that cookie and the linked Clerk session. Other devices stay signed in.
- RouterPlus rechecks Clerk-backed sessions during authenticated requests, with cached proof for
  at most 30 seconds. After that, a revoked or expired Clerk session cannot authorize new
  requests.

The browser also stores two items beside the session cookie. Neither is a credential, and signing
out clears both:

- A sign-in hint (`tm_in`, the same expiry, readable by script) that says only "signed in".
- A display cache (`localStorage["tm-acct"]`): your email and your balance as the last page
  showed them.

They let every page draw your account menu and credits at once.

Email links issued earlier, and operator-issued links, use stored SHA-256 token hashes, so a
leaked database row is not a working link. These links are single-use and expire. They are not a
public account-creation route.

See [Authentication](/docs/authentication) for browser onboarding and API keys.

## What providers do with your prompts

The provider that serves your request has its own retention policy, apart from RouterPlus's
request logs. Every provider in the catalog declares its prompt retention in its manifest before
it can serve traffic:

```json
{
  "provider": {
    "id": "anthropic",
    "privacy_policy_url": "https://www.anthropic.com/legal/privacy",
    "prompt_logging": "retained"
  }
}
```

`prompt_logging` is `"none"` or `"retained"`. A manifest without it, or without a privacy policy
URL, fails conformance and never goes live. You can see the value in two places:

- On every **model page**. The Data policy card links to this policy and says how many of the
  model's providers offer zero data retention. The providers table has a **Retention** column
  per provider. It describes the provider, not RouterPlus's request log.
- In the catalog, per provider of each model:

```bash
curl -s https://demo.routerplus.com/api/models.json | python3 -c \
  'import json,sys; [print(m["id"], ":", [(p["id"], p["prompt_logging"]) for p in m["providers"]]) for m in json.load(sys.stdin)["models"]]'
```

Every provider in the catalog today declares `"retained"`, per their published
policies, except Fastino, Cloudflare Workers AI, Perplexity, DeepInfra (its open models) and RouterPlus.
RouterPlus reports each provider's flag as declared. The sections below give the basis for each
provider with a specific policy.

### Levanto

Levanto, which serves the decision model Sage, keeps no ordinary request. It keeps the full
request of a failed or slow one for 30 days to debug it. Levanto declares `"retained"`.

### Fastino

Fastino, which serves the decision model GLiNER-2.5-Decide, declares `"none"`.

- Fastino keeps an inference unless the request says `store: false`. The gateway sends
  `store: false` on every request.
- Fastino's terms let it train on inputs unless a team has Zero Data Retention on. Our Fastino
  team has it on.

### Cloudflare Workers AI

Cloudflare Workers AI, which serves the decision models Clef and Clef-flash,
declares `"none"`, on Cloudflare's own statements.

- Of Clef, Cloudflare says: "we don't read, store, or train on your requests or responses
  (unless you want to use our fine-tuning product)". RouterPlus does not use that product.
- Cloudflare's Workers AI data-usage page says that Cloudflare does not use a customer's content
  to train any AI model on Workers AI, or to improve any Cloudflare or third-party service,
  without the customer's explicit consent.
- The same page says that Cloudflare stores the content only if the customer uses a Cloudflare
  storage service (R2 or KV, for example) with Workers AI. RouterPlus uses none.
- The gateway calls the Workers AI REST API directly. It never sends Cloudflare's AI Gateway
  header (`cf-aig-gateway-id`), so requests do not go through AI Gateway,
  whose logs are on by default and can hold the full prompt and response. Cloudflare's replies carry none of AI
  Gateway's `cf-aig-*` headers.

### Perplexity

Perplexity, which serves the decision model Perplexity Decider v1 27B, declares `"none"`, on
Perplexity's own statements.

- Its API FAQ says: "We do not retain any query data sent through the API and do not train on
  any of your data."
- It also says that the API has zero day retention of user prompt data by default, never used
  for AI training, and that its compute is hosted on Amazon Web Services in North America.
- Perplexity keeps the usage metadata it bills from, such as the number of requests and tokens
  (its API FAQ).
- The gateway calls Perplexity's API directly, with our one Perplexity key.

### DeepInfra

DeepInfra, which serves fifteen open chat models (DeepSeek, GLM, Kimi, MiniMax, Hy4, Gemma, MiMo
and gpt-oss), declares `"none"`, on DeepInfra's own statements.

Its data-privacy page says that it does
not store the data sent to its inference APIs on disk. It holds the data in memory only while it
runs the request. It does not log request content. It records only metadata such as the request
id, the cost and the sampling parameters.

- DeepInfra does not train on API data. Its exceptions are Google's and Anthropic's models,
  which this listing does not carry.
- The gateway calls DeepInfra's OpenAI-compatible API directly, never its bulk inference API
  (which can keep data longer).

DeepInfra's second listing, `deepinfra-labs`, carries Google's and Anthropic's models: Claude
and Gemini, which DeepInfra relays to Anthropic and Google under their own data and training
policies. This listing declares `"retained"`, as the Anthropic listing does. It is Claude's
fallback after Anthropic direct.

### Morph

Morph, which serves its Fast Apply models, Kimi K3 Fast and five open models, declares
`"retained"`. Morph publishes zero data retention only for self-hosted deployments. Its hosted
API records requests for its traces dashboard. Its inference runs in the US, and DeepSeek V4.1
Flash in Japan.

### Mistral AI

Mistral AI, which serves its own models (Mistral Large, Medium, Small, Ministral and Codestral)
and is GLM 5.3's last route, declares `"retained"`. Mistral keeps API prompts and answers for 30
days to monitor abuse, unless the account has zero data retention on.

### Bespoke Labs

Bespoke Labs, which serves the decision models Bespoke Nimble v4 and v4 Max, declares
`"retained"`. Bespoke keeps API inputs, outputs and request logs for up to 30 days, to debug the
service and check for abuse, and then deletes them. It also keeps the metadata it bills from
(token counts, times, key id, status). Bespoke does not train on API content unless an
organization opts in, and ours does not.

### OpenAI

OpenAI, which serves the decision model GPT-6 Luna Decisions (deployment
`openai-decisions`), declares `"retained"`. OpenAI keeps abuse-monitoring logs of
`/v1/decisions` requests for up to 30 days. Zero Data Retention needs an eligible account and an
agreement with OpenAI, and our organization does not have it. OpenAI does not train
on API data by default. The gateway calls OpenAI's API directly, with our one OpenAI key.

### RouterPlus

RouterPlus serves its own decision models, Decider 2B and Kev 4B, and
this page is its data policy. Its GPUs run in the United States.

- The RouterPlus model service writes no request logs.
- It keeps a cache of its answers: each answer and its usage, for 600 s (10 minutes).
- The cache key is the RouterPlus API key, the model and the exact request body, byte for byte.
  Only an identical request gets a cached answer.
- A cache entry is never shared across RouterPlus keys. The gateway sends every request with
  our one RouterPlus key, so an identical request from any buyer within 600 s can get the cached
  answer, which repeats your option names.
- The cache is all it keeps, so the listing declares `"none"`.
- RouterPlus does not use your states, questions or answers for training.

On the gateway, requests to RouterPlus models follow the request-log policy above, as for every
provider.

## Reading your own metadata back

Read your billing and usage metadata with your API key:

```bash
curl -s -H "Authorization: Bearer $TM_API_KEY" https://demo-api.routerplus.com/v1/usage
curl -s -H "Authorization: Bearer $TM_API_KEY" "https://demo-api.routerplus.com/v1/generation?id=<request_id>"
```

Both endpoints read the billing ledger: balance, attempts, token counts and settled cost.
Neither returns message content. See [usage and generation](/docs/api-usage).

> [!TIP]
> To audit a request later, keep the `x-request-id` response header from the
> original call. It is the lookup key for `/v1/generation`, and the id RouterPlus
> uses if you write in about a billing dispute.

See [Errors](/docs/errors) for the error taxonomy.

---

Source: https://demo.routerplus.com/docs/install.md

# Coding agents

Point a coding agent at RouterPlus to run it on any catalog model, billed to one balance. Each tool below has the same parts: requirements, setup steps, a config snippet and its limits through the gateway.

Tools: [Claude Code](#claude-code), [Codex](#codex), [Cursor](#cursor), [Cline](#cline), [Kilo Code](#kilo-code), [OpenCode](#opencode), [Hermes Agent](#hermes-agent) and [other tools](#other-tools).

- Apps built with Replit, Lovable, Bolt or v0 have [their own page](/docs/app-builders).
- The [Playground](#playground) needs no setup.
- A coding agent can follow the [installation runbook](#the-installation-runbook) at the end of this page.
- To build the gateway into a product, give your agent the [Agent integration guide](agent-integration.md).

Every docs page is served twice: `/docs/<slug>` rendered for people, and `/docs/<slug>.md` as raw markdown for agents. The machine index is [https://demo.routerplus.com/llms.txt](https://demo.routerplus.com/llms.txt).

## Before you start

1. **Get a key.** Open [Sign up](https://demo.routerplus.com/signup), complete Clerk authentication and email verification, and save the first key shown after sign-in. Buy credits in [Billing](https://demo.routerplus.com/console/billing). Signup does not add credit.
   - An invitation link from RouterPlus (for example, at a hackathon) adds its credit and makes the account count as paid, with higher limits.
   - To create more keys, open Access > API keys in the console ([API keys](https://demo.routerplus.com/console/keys)). A key looks like `tm_vk_...`. Give each tool its own key.
2. **Pick the base URL.** Tools that speak OpenAI's format use `https://demo-api.routerplus.com/v1`. Tools that speak Anthropic's format (Claude Code) use `https://demo-api.routerplus.com`, without `/v1`.
3. **Pick a model.** These ids work in every tool. In Cursor, use the name in the second column. The live list, with every price, is [https://demo.routerplus.com/api/models.json](https://demo.routerplus.com/api/models.json).

   | Model id | Name in Cursor | Context | Max output | Input /1M | Cached /1M | Cache write /1M | Output /1M |
   |---|---|---|---|---|---|---|---|
   | `claude-opus-5-5` | `rp-opus-5-5` | 1,000,000 | 128,000 | $4 | $0.20 | $5 | $20 |
   | `claude-fable-5-1` | `rp-fable-5-1` | 1,000,000 | 128,000 | $10 | $0.25 | $12.50 | $50 |
   | `claude-sonnet-5-5` | `rp-sonnet-5-5` | 1,000,000 | 128,000 | $2 | $0.10 | $2.50 | $10 |
   | `claude-haiku-5-5` | `rp-haiku-5-5` | 1,000,000 | 128,000 | $0.10 | $0.01 | $0.125 | $0.50 |
   | `gpt-6-astra` | `gpt-6-astra` | 1,050,000 | 128,000 | $10 | $1 | $12.50 | $50 |
   | `gpt-6.1-sol` | `gpt-6.1-sol` | 1,050,000 | 128,000 | $2 | $0.10 | $2.50 | $10 |
   | `gpt-6-luna` | `gpt-6-luna` | 1,050,000 | 128,000 | $0.10 | $0.01 | $0.125 | $0.50 |
   | `gpt-5.6-terra` | `gpt-5.6-terra` | 1,050,000 | 128,000 | $2 | $0.20 | $2.50 | $12 |
   | `gpt-6-sol` | `gpt-6-sol` | 1,050,000 | 128,000 | $2 | $0.20 | $2.50 | $10 |
   | `gpt-5.6-sol` | `gpt-5.6-sol` | 1,050,000 | 128,000 | $4 | $0.40 | $5 | $20 |
   | `gpt-5.6-luna` | `gpt-5.6-luna` | 1,050,000 | 128,000 | $0.20 | $0.02 | $0.25 | $1.20 |

   The other chat models in the catalog work too: `claude-sonnet-5`, `claude-opus-5`, `claude-fable-5`, `claude-sonnet-4-5` and `claude-haiku-4-5` (in Cursor `rp-sonnet-5`, `rp-opus-5`, `rp-fable-5`, `rp-sonnet-4-5` and `rp-haiku-4-5`), and `gpt-4o`, `gpt-4o-mini` and `gpt-4.1-mini`.

### What the gateway does for coding agents

- **Full-length answers.** When an agent sends no output limit, the model gets its own maximum: 128,000 tokens on the models above, fitted to the context window on the chat surfaces. The [hold](#your-balance-and-the-hold) counts at most 32,768 of them.
- **Prompt caching on Claude.** When a conversation carries no cache marks, the gateway turns on Claude's [automatic caching](/docs/compat#automatic-caching-on-claude). The `x-tm-auto-cache` response header says whether it did.
- **Screenshots and PDFs.** Images and PDFs reach Claude from tools that speak OpenAI's format, and reach GPT models from tools that speak Anthropic's format.
- **Compaction.** A prompt longer than the model's window gets the standard error that agents compact on: `context_length_exceeded` on the OpenAI format, `prompt is too long` on the Anthropic format.
- **The provider's own words.** An error carries the provider's message, so an agent can see what the model refused. A 502 or 504 is the exception: it is Cloudflare's own short error page, without the message or `x-request-id` (see [Errors](/docs/errors)). Agents retry it by themselves.
- **Default sampling only.** The Claude 5, GPT-6 and GPT-5.6 models take only their default sampling. When a tool sends another `temperature`, or a `top_p` or `top_k`, the gateway drops it and names it in `x-tm-dropped-params`. The request does not fail.
- **Cancels.** When you stop an answer (Esc, Stop), you pay for the input and the output already streamed. You never pay the full hold.
- **Hidden thinking.** The Claude 5 models do not show their thinking. Nothing reaches the tool until the first word or tool call, so a hard turn can look stuck for a minute or more.

### Your balance and the hold

While a request runs, the gateway holds balance for the input plus the output limit. A limit that the tool sends is held in full. With no limit, the hold counts 32,768 output tokens. Agents often run several requests at once. The output part of the hold per request:

| Model | Hold for 32,768 output tokens |
|---|---|
| Claude Fable 5.1, GPT-6 Astra | $1.64 |
| Claude Opus 5.5, GPT-5.6 Sol | $0.66 |
| GPT-5.6 Terra | $0.39 |
| Claude Sonnet 5.5, GPT-6.1 Sol, GPT-6 Sol | $0.33 |
| GPT-5.6 Luna | $0.04 |
| Claude Haiku 5.5, GPT-6 Luna | $0.02 |

> [!WARNING]
> You pay for what a request used, so a long answer can cost more than its hold. A small
> balance can refuse a request (`insufficient_quota`) before it starts. Then pick a
> cheaper model, or keep the tool's output limit small (each tool's section shows how).

### When you hit a rate limit

A request over one of your limits gets a 429, or a 503 when the gateway as a whole is full. Its `retry-after` header gives the seconds to wait, from 1 to 59:

- For a one-minute limit: the time until its window has room again.
- For a limit on requests at once: 10 seconds. The limit frees when one of your running requests ends.

Most coding agents wait and try again by themselves:

| Tool | What it does on a 429 |
|---|---|
| Claude Code | waits at least `retry-after` and tries again, up to 10 times (`CLAUDE_CODE_MAX_RETRIES`, at most 15) |
| Codex | waits `retry-after` and tries again, up to `stream_max_retries` times (5 by default), showing `Reconnecting... 1/5`. To wait longer, add `stream_max_retries = 10` to `[model_providers.routerplus]` |
| Cursor | does not try again. It shows the error, so send your message again after the wait |
| Cline | waits `retry-after` and tries again, up to 5 times, with no message while it waits. A turn can pause for up to a minute |
| Kilo Code, OpenCode | wait `retry-after` and try again, up to 5 times, showing the error with a countdown |

When the tries run out, the task stops with the error. Tell the agent to continue.

Out of credit (`insufficient_quota`) is also a 429. If your own requests in flight hold the balance, it frees as they end, and retries can wait it out. If not, add credits in [Billing](https://demo.routerplus.com/console/billing). Codex stops on it at once. Claude Code stops at once on a monthly spend cap, because that 429 says not to retry.

## Claude Code

Claude Code speaks Anthropic's format, which the gateway's `/v1/messages` surface accepts as sent. It works with Claude Code's default model (Claude Opus 5.5, sent as `claude-opus-5-5`) and with every Claude id in the catalog.

### Claude Code CLI

**Requirements:** the `claude` CLI and a `tm_vk_...` key.

**Setup:**

1. Put the gateway in your user settings, `~/.claude/settings.json`:

   ```json
   {
     "env": {
       "ANTHROPIC_BASE_URL": "https://demo-api.routerplus.com",
       "ANTHROPIC_AUTH_TOKEN": "tm_vk_..."
     }
   }
   ```

2. Run `claude`. There is no login screen. The first run shows the theme, the security notes and the folder trust question, then a session on Opus 5.5 with API usage billing.
3. At the folder trust question, choose **Yes, I trust this folder**. The default, **No, exit**, closes Claude Code.
4. Run `/status` to check the base URL and `ANTHROPIC_AUTH_TOKEN`.

Do not put the key in a project's `.claude/settings.json`: that file is committed, and its `env` block applies only after the first-run screens, so the login screen still appears.

For one shell only, export the same two values:

```bash
export ANTHROPIC_BASE_URL=https://demo-api.routerplus.com
export ANTHROPIC_AUTH_TOKEN=$TM_API_KEY
claude
```

> [!WARNING]
> Use `ANTHROPIC_AUTH_TOKEN`, not `ANTHROPIC_API_KEY`. With `ANTHROPIC_API_KEY`, Claude
> Code asks once whether to use "a custom API key", and the default **No** opens the login
> screen. If that happened, choose the custom API key in `/config`. You do not need a
> claude.ai sign-in: `ANTHROPIC_AUTH_TOKEN` comes first. For headless or CI use, give
> Claude Code a fresh config directory:
> `CLAUDE_CONFIG_DIR=$(mktemp -d) ANTHROPIC_BASE_URL=https://demo-api.routerplus.com ANTHROPIC_AUTH_TOKEN=$TM_API_KEY claude -p "..."`

**How the gateway handles Claude Code:**

- `ANTHROPIC_AUTH_TOKEN` goes as `Authorization: Bearer …`, which the gateway accepts everywhere, as it does `x-api-key`.
- Query strings (`/v1/messages?beta=true`) pass through. The gateway routes on the path.
- **Dated** ids (`claude-haiku-4-5-20251001`) resolve to their model, and the original id goes to the provider (see [Models & catalog](/docs/models)). An unknown id gets a `404` that names it.
- The gateway forwards the `anthropic-beta` values Claude Code needs on the Claude 5 models (Opus 5.5, Fable 5.1, Sonnet 5.5, Haiku 5.5): effort inside a message, context editing (`context_management`), and tools added during a session. [Wire compatibility](/docs/compat) lists them all.
- Claude Code's own cache marks work as on Anthropic's API.
- Web search works, at Anthropic's price per search (see [Pricing & billing](/docs/pricing#web-search)).
- A conversation that outgrows the window gets Anthropic's `prompt is too long` error, which Claude Code compacts on.
- Every request appears in `https://demo-api.routerplus.com/v1/usage` with its token counts and `cost_usd`.
- Claude Code does not know the context windows of open models and assumes 200,000 tokens. Set `CLAUDE_CODE_MAX_CONTEXT_TOKENS` to the model's window, for example `CLAUDE_CODE_MAX_CONTEXT_TOKENS=1000000` for `moonshotai/kimi-k3`.

**Limits:**

- Remote Control does not work through the gateway. It needs a claude.ai subscription sign-in. Claude Code turns it off when `ANTHROPIC_BASE_URL` points at a host other than `api.anthropic.com`, or when `ANTHROPIC_AUTH_TOKEN` or `ANTHROPIC_API_KEY` is set. To use Remote Control, start that session without the gateway variables and sign in with claude.ai.
- Voice dictation is off while a gateway key is in use.

> [!NOTE]
> The gateway serves these buyer endpoints: `/v1/chat/completions`, `/v1/responses` and
> `/v1/responses/compact`, `/v1/messages`, `/v1/messages/count_tokens` (unbilled, models
> with an Anthropic-dialect deployment), `/v1/models`, `/v1/images/generations`,
> `/v1/decisions`, `/v1/usage`, `/v1/generation`, `/v1/route` and `/v1/limits`, plus an
> unauthenticated `/healthz`. Any other path that an authenticated tool probes gets a
> well-formed `404` with `error_type: not_found` in the caller's own dialect. It does not
> hang or return an HTML page.

### Claude Code in VS Code

**Requirements:** VS Code 1.94 or later and the Claude Code extension.

**Setup:** the setting must be in your **user** settings. Run **Preferences: Open User Settings (JSON)** and add:

```json
{
  "claudeCode.environmentVariables": [
    { "name": "ANTHROPIC_BASE_URL", "value": "https://demo-api.routerplus.com" },
    { "name": "ANTHROPIC_AUTH_TOKEN", "value": "tm_vk_..." }
  ],
  "claudeCode.disableLoginPrompt": true
}
```

`claudeCode.disableLoginPrompt` stops the sign-in prompt. The extension reads `~/.claude/settings.json` only for the process it starts, not for its own sign-in check, so put the key in `claudeCode.environmentVariables`.

**Limits:** through a gateway, the extension shows no plan usage bars, no voice dictation and no Web tab.

### Claude Code in JetBrains IDEs

**Requirements:** the Claude Code CLI. The plugin for JetBrains IDEs (IntelliJ IDEA, PyCharm, WebStorm and the others) does not include it.

**Setup:**

1. Set up the CLI as in [Claude Code CLI](#claude-code-cli). The plugin runs the `claude` command in the IDE's terminal and uses `~/.claude/settings.json`.
2. If the IDE cannot find the CLI, set its path in **Settings → Tools → Claude Code [Beta] → Claude command**.

### Claude desktop app

**Requirements:** the Claude desktop app on macOS or Windows. The app sends its requests to a gateway in third-party mode.

**Setup:**

1. **Help → Troubleshooting → Enable Developer Mode**. On Windows the Help menu is under ☰, at the top left of the sign-in screen. The app restarts with a **Developer** menu.
2. **Developer → Configure Third-Party Inference…**
3. Under **Connection**, set **Inference provider** to **Gateway**.
4. Under **Gateway credentials**, set:
   - **Gateway base URL:** `https://demo-api.routerplus.com` (without `/v1`)
   - **Gateway API key:** your `tm_vk_...` key
   - **Credential kind:** Static API key
   - **Gateway auth scheme:** Bearer
5. **Apply Changes**, then **Save & Restart**.
6. On the sign-in screen, start in third-party mode. Do not sign in to claude.ai or create an Anthropic account.

An administrator can set the same values for a whole company with managed settings: `inferenceProvider: "gateway"`, `inferenceGatewayBaseUrl`, `inferenceGatewayApiKey`, `inferenceGatewayAuthScheme: "bearer"`.

**Limits:** in third-party mode, Remote Control, Anthropic's cloud sessions, the mobile apps, voice mode, computer use and the connectors made by Anthropic do not work. Local MCP servers work. Web search works, and the gateway passes it to Anthropic.

### Output limit, auto mode and the advisor

- **Output limit and your balance.** Claude Code asks the Claude 5 models for up to 128,000 output tokens (64,000 on Fable 5.1), and the gateway accepts up to each model's `max_output_tokens`. The hold is the input plus that output: about $2.56 per request on Opus 5.5 and $3.20 on Fable 5.1.
  - You pay for the tokens the provider reports.
  - An interrupt (Esc) costs the input and the output already streamed. An interrupt while the model thinks costs the input only, because the Claude 5 models do not show their thinking.
  - On a small balance, set a lower limit:

  ```bash
  export CLAUDE_CODE_MAX_OUTPUT_TOKENS=32000
  ```

- **Auto mode.** Claude Code starts in auto mode. In auto mode and plan mode, it checks some tool calls before it runs them (a web search, for example; in tests, file edits and simple commands needed no check).
  - Each check is a separate request to `claude-sonnet-5` with the conversation, billed at the full input price. One check of a 47,000-token conversation cost about $0.12.
  - The first check shows a notice that auto mode keeps working. Press Enter. To stop the notice, set `CLAUDE_CODE_AUTO_MODE_SERVER=0`.
  - To avoid the checks, press Shift+Tab to change to another permission mode.
- **Advisor.** Do not turn on `/advisor`. It runs a second model at its own rates, so the gateway does not forward its beta, and Anthropic refuses the request with a 400 that names `Input tag 'advisor_…'`. Claude Code 2.1.280 or later then retries without the advisor and leaves it off until it exits.

### Pinning which models Claude Code uses

Claude Code picks its own model ids. Set them with the standard variables, in the shell or in the `env` block of `settings.json`, with any Claude id from the catalog:

```bash
export ANTHROPIC_MODEL=claude-sonnet-5-5               # main model
export ANTHROPIC_DEFAULT_HAIKU_MODEL=claude-haiku-5-5  # background and fast tasks
```

- Set `ANTHROPIC_DEFAULT_HAIKU_MODEL`. Otherwise, behind a gateway, background tasks run on the main model (Opus 5.5 by default).
- Aliases resolve as on Anthropic's API: `opus` is `claude-opus-5-5`, `sonnet` is `claude-sonnet-5-5`, `haiku` is `claude-haiku-5-5`, `fable` is `claude-fable-5-1`.
- `ANTHROPIC_DEFAULT_OPUS_MODEL`, `ANTHROPIC_DEFAULT_SONNET_MODEL`, `ANTHROPIC_DEFAULT_FABLE_MODEL` and `CLAUDE_CODE_SUBAGENT_MODEL` pin the others.
- `ANTHROPIC_MODEL` overrides your `/model` choice at every launch. To set only the starting model, use `ANTHROPIC_DEFAULT_MODEL` (Claude Code 2.1.236 or later).
- Inside a session, `/model` accepts any catalog id.

Spend from every session appears per request in the console under Usage ([Usage](https://demo.routerplus.com/console/usage)) and in `https://demo-api.routerplus.com/v1/usage`.

## Codex

Codex calls the Responses API, which the gateway serves at [POST /v1/responses](/docs/api-responses).

**Requirements:** Codex (the CLI, the IDE extension, or the desktop app, which is Codex in the ChatGPT desktop app). All of them read one file, `~/.codex/config.toml`. The CLI needs no sign-in for this provider.

**Setup:**

1. Add the gateway as a provider. Keys such as `model` must come before the first `[table]`:

   ```toml
   # ~/.codex/config.toml
   model = "gpt-6-luna"
   model_provider = "routerplus"

   [model_providers.routerplus]
   name = "RouterPlus"
   base_url = "https://demo-api.routerplus.com/v1"
   env_key = "TM_API_KEY"
   wire_api = "responses"
   ```

2. Start Codex with the key in the environment:

   ```bash
   export TM_API_KEY=tm_vk_...
   codex
   ```

- Use any GPT-6 or GPT-5.6 model as `model` (`gpt-6-astra`, `gpt-6.1-sol`, `gpt-6-sol`, `gpt-6-luna`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`), or start Codex with `codex -m gpt-6-astra`.
- `wire_api = "responses"` is Codex's default, so you can leave it out. With `wire_api = "chat"`, current Codex does not start ("`wire_api = "chat"` is no longer supported").
- Do not name the provider `OpenAI`. Codex then treats it as OpenAI's own service and changes how it compacts a long conversation.

**The desktop app** may not see your shell's variables when started from the Dock or Finder. Put the key in the provider block instead of `env_key`, never both: Codex fails when `env_key` names a missing variable.

```toml
[model_providers.routerplus]
name = "RouterPlus"
base_url = "https://demo-api.routerplus.com/v1"
wire_api = "responses"
experimental_bearer_token = "tm_vk_..."
```

- The file then holds your key. Do not commit it or share it.
- To keep the key out of the file on a Mac, keep `env_key`, run `launchctl setenv TM_API_KEY tm_vk_...`, and quit and reopen the app. Do it again after a restart.
- The desktop app's model picker lists only OpenAI's own models, so set `model` in the file.

**Claude models (preview).** Codex can run Claude through the gateway's preview translation:

```toml
model = "claude-sonnet-5-5"
model_provider = "routerplus"
model_context_window = 272000
web_search = "disabled"
```

- Codex warns "Model metadata for `claude-sonnet-5-5` not found. Defaulting to fallback metadata". Only a full model catalog of your own (`model_catalog_json`) removes the warning.
- Codex caps `model_context_window` at 272,000 tokens for an unknown id, so set that.
- Codex ignores `model_max_output_tokens`, with a warning.
- `web_search = "disabled"` stops Codex from sending its hosted web search tool, which does not run on Claude (the gateway would drop it and name it in `x-tm-dropped-params`).
- `model_reasoning_effort` becomes Claude's `output_config.effort`. `minimal` becomes `low`. `none` turns thinking off on Claude Haiku 5.5, Sonnet 5 and Opus 5, and is `low` on the other Claude models. On Claude Sonnet 4.5 and Haiku 4.5 it is dropped and named in `x-tm-dropped-params`.

> [!WARNING]
> Do not sign in to ChatGPT in Codex while you use the gateway. A known bug in the
> desktop app ([openai/codex#48369](https://github.com/openai/codex/issues/48369))
> disables the send button when the ChatGPT plan's usage runs out, even for a custom
> provider such as this one.

**Limits:**

- Stored responses are not supported: `store` is always false, and `previous_response_id` is a 400. Codex is not affected: it sends `store: false` and the full conversation on every turn, and compacts with an ordinary `/v1/responses` request.
- Claude through Codex is a preview. Claude's reasoning does not carry from turn to turn.
- Codex sends no output limit (see [the hold](#your-balance-and-the-hold)).
- When your balance runs out, Codex stops with "Quota exceeded. Check your plan and billing details." This is the gateway's `insufficient_quota`. Add credits in [Billing](https://demo.routerplus.com/console/billing).
- Codex retries a rate limit only when the 429 carries `retry-after`. The gateway's 429s do.

## Cursor

Cursor's Chat and Agent can call the gateway with an OpenAI key and a base URL.

**Requirements:** a paid Cursor plan (Pro, Pro+, Ultra or Teams). On the free plan, outside keys do not work even when they are set up.

**Setup:**

1. Open **Cursor Settings → Models**, and expand **API Keys** at the bottom.
2. Under **OpenAI API Key**, paste your `tm_vk_...` key and turn on **Use OpenAI API Key**.
3. Turn on **Override OpenAI Base URL** and enter `https://demo-api.routerplus.com/v1`.
4. Add the models by name in the model field on the same page:
   - GPT models such as `gpt-6-astra`, `gpt-6.1-sol`, `gpt-6-luna` and `gpt-5.6-terra`.
   - Claude models by their `rp-` names: `rp-sonnet-5-5`, `rp-opus-5-5`, `rp-haiku-5-5` and `rp-fable-5-1`.
5. Restart Cursor, open a new chat, and pick one of these models in Chat or Agent.

Use `rp-sonnet-5-5`, not `claude-sonnet-5-5`. Cursor sends a name that starts with `claude-` with its Anthropic key, never to your base URL, and without that key it fails with "Model name is not valid". Each `rp-` name is one Claude model at its price (see [Claude names for Cursor](/docs/models#claude-names-for-cursor)).

**Limits:**

- Only Chat and Agent use your key. A subagent runs on its chat's model. Tab completion, Auto, the Cursor CLI and cloud agents always use Cursor's own models.
- While the override is on, Cursor sends every model that is not Claude or Gemini to the gateway. Its own GPT models arrive under Cursor's names and may get a 404. Its Composer and Grok models fail. Turn **Use OpenAI API Key** off to use them; your key and URL stay saved.
- If Cursor says a name you add "is already available", it is one of Cursor's own models, and with the override on it comes to the gateway under Cursor's name.
- Cursor calls the gateway from Cursor's servers, not from your computer.
- Cursor sends no output limit (see [the hold](#your-balance-and-the-hold)).
- You cannot set a context window or a reasoning effort for a custom model.
- On a Teams or Enterprise plan, Cursor charges its own fee per token on requests with your key, and an admin can hide the API Keys section.
- Cursor edits files on GPT models with a custom (free-form) tool, which the gateway supports on the GPT-6 and GPT-5.6 models. `gpt-4o`, `gpt-4o-mini` and `gpt-4.1-mini` take none, so use a GPT-6 or GPT-5.6 model for Agent.

## Cline

**Requirements:** Cline in VS Code, in a JetBrains IDE, as Cline Desktop, or as the Cline CLI (Node 20 or later). All of them share their settings in `~/.cline`.

**Setup:**

1. Pick the **OpenAI Compatible** provider and set these fields:

   | Field | Value |
   |---|---|
   | API Provider | OpenAI Compatible |
   | Base URL | `https://demo-api.routerplus.com/v1` |
   | OpenAI Compatible API Key | `tm_vk_...` |
   | Model ID | `claude-sonnet-5-5`, or any id from the catalog (**Use custom model ID…** if the list does not show it) |

2. Open **Model Configuration**. Cline assumes a 128,000-token context window for unknown models. Set:
   - **Context Window Size:** the model's window, 1,000,000 for `claude-sonnet-5-5`.
   - **Supports Images:** on, for the Claude 5, GPT-6 and GPT-5.6 models.
   - **Max Output Tokens:** leave it empty, so the model gets its maximum and the [hold](#your-balance-and-the-hold) stays at 32,768 output tokens. A number that you set is sent and held in full.
   - **Input Price** and **Output Price** (per 1M tokens): the model's prices, if you want Cline's cost display to match your bill.

Where to set it in each Cline:

- **JetBrains IDEs:** install **Cline** from **Settings → Plugins → Marketplace** and open it from **View → Tool Windows → Cline**.
- **Cline Desktop:** **Add Provider → OpenAI-Compatible Provider** takes the same base URL, key and model ids.
- **Cline CLI** takes the same values:

```bash
npm install -g cline
cline auth --provider openai-compatible --apikey tm_vk_... --modelid claude-sonnet-5-5 --baseurl https://demo-api.routerplus.com/v1
cline "Add a README for this project"
```

**Limits:**

- Cline sends no cache marks. The gateway's [automatic caching](/docs/compat#automatic-caching-on-claude) covers it on Claude.
- On a rate limit, Cline waits and tries again by itself, with no message while it waits ([When you hit a rate limit](#when-you-hit-a-rate-limit)).
- The Cline CLI's file editor takes at most 6,000 characters per edit, so a long file takes several edits and requests.
- Cline's own Anthropic and OpenAI providers go to those providers' servers. Use OpenAI Compatible.

## Kilo Code

**Requirements:** Kilo Code for VS Code, the Kilo Code plugin for JetBrains IDEs, or the Kilo CLI (`npm install -g @kilocode/cli`, then `kilo`).

**Setup in VS Code:** open **Settings** (the gear icon) → **Providers**. Find **Custom provider**, click **Connect**, fill in these fields and click **Submit**:

| Field | Value |
|---|---|
| Provider ID | `routerplus` |
| Display name | RouterPlus |
| Provider API | OpenAI Compatible |
| Base URL | `https://demo-api.routerplus.com/v1` |
| API key | `tm_vk_...` |
| Models | `claude-sonnet-5-5`, `gpt-6-luna` and any other catalog id. Turn on **Image** for the Claude 5, GPT-6 and GPT-5.6 models |

**Setup in JetBrains IDEs:** open **Settings → Kilo Code → Providers → Add custom provider**.

1. Enter the Provider ID, Display name, Base URL and API key (or the name of an environment variable that holds the key).
2. Type the model ids, separated by commas, in **Model IDs**, and click **Add**.
3. This dialog has no Image switch. Turn images on in `kilo.jsonc` (below).

**Config:** set the context window and output limit in `~/.config/kilo/kilo.jsonc`, which the VS Code extension, the JetBrains plugin and the Kilo CLI all read. The settings screen has neither, and without a context window Kilo Code compacts only after the gateway refuses a prompt as too long:

```json
{
  "provider": {
    "routerplus": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "RouterPlus",
      "options": {
        "baseURL": "https://demo-api.routerplus.com/v1",
        "apiKey": "tm_vk_..."
      },
      "models": {
        "claude-sonnet-5-5": { "name": "Claude Sonnet 5.5", "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 1000000, "output": 128000 } },
        "gpt-6-luna": { "name": "GPT-6 Luna", "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 1050000, "output": 128000 } }
      }
    }
  },
  "model": "routerplus/claude-sonnet-5-5"
}
```

**Limits:**

- Keep `modalities` on each model. Without it, Kilo Code sends no images.
- Kilo Code sends `limit.output` as `max_tokens`, up to 32,000 unless you set `KILO_EXPERIMENTAL_OUTPUT_TOKEN_MAX` (128,000 is the models' maximum). A lower number keeps the [hold](#your-balance-and-the-hold) small.

> [!NOTE]
> Roo Code was shut down in May 2026, and its repository is archived. Use Kilo Code or
> Cline instead.

## OpenCode

**Requirements:** OpenCode, the CLI or the desktop app. Both read providers from `opencode.json`, globally at `~/.config/opencode/opencode.json` or in a project's root.

**Setup:**

1. Add the gateway with the OpenAI-compatible package, and list the models you want with their limits:

   ```json
   {
     "$schema": "https://opencode.ai/config.json",
     "provider": {
       "routerplus": {
         "npm": "@ai-sdk/openai-compatible",
         "name": "RouterPlus",
         "options": {
           "baseURL": "https://demo-api.routerplus.com/v1",
           "apiKey": "{env:TM_API_KEY}"
         },
         "models": {
           "claude-sonnet-5-5": { "name": "Claude Sonnet 5.5", "attachment": true, "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 1000000, "output": 128000 } },
           "claude-opus-5-5": { "name": "Claude Opus 5.5", "attachment": true, "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 1000000, "output": 128000 } },
           "gpt-6-luna": { "name": "GPT-6 Luna", "attachment": true, "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 1050000, "output": 128000 } }
         }
       }
     },
     "model": "routerplus/claude-sonnet-5-5"
   }
   ```

2. Give OpenCode the key in one of two ways:
   - Export `TM_API_KEY` before you start OpenCode. The desktop app reads your login shell's variables.
   - Store the key: run `/connect` in OpenCode (or `opencode auth login` in a shell), choose **Other**, enter the provider id `routerplus`, paste the key, and leave `apiKey` out of the file.
3. Pick a model with `/models`, or start with `opencode -m routerplus/claude-sonnet-5-5`.

**Limits:**

- Screenshots need both `"attachment": true` and `"modalities": { "input": ["text", "image"], "output": ["text"] }` on the model. Without `modalities`, OpenCode sends no image.
- OpenCode asks for at most 32,000 output tokens, even when `limit.output` is higher, unless you set `OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX`. That also keeps the hold small.
- `@ai-sdk/openai-compatible` calls `/v1/chat/completions`, which serves every model.

## Hermes Agent

**Requirements:** Hermes Agent. It reads its endpoint from `config.yaml` in its home folder (`~/.hermes`, or the folder in `HERMES_HOME`).

**Setup:**

1. Add the gateway as a custom endpoint in `config.yaml`:

   ```yaml
   model:
     provider: custom
     base_url: https://demo-api.routerplus.com/v1
     api_key: ${TM_API_KEY}
     default: claude-sonnet-5-5
   ```

2. Run Hermes with the key in the environment:

   ```bash
   export TM_API_KEY=tm_vk_...
   hermes
   ```

- Hermes calls `/v1/chat/completions` with the key as a bearer token. Every catalog model works, for example `moonshotai/kimi-k3` or `deepseek/deepseek-v4.1-flash`.
- Hermes reads each model's context window from `GET /v1/models`. To set it yourself, add `context_length:` with the model's window.
- To call Claude on `/v1/messages`, set `base_url: https://demo-api.routerplus.com` and add `api_mode: anthropic_messages`.
- For one task without a session, run `hermes chat -q "..."`. Add `--yolo` only if you accept that shell commands run without a prompt.

**Limits:** Hermes sends no output limit, so the [hold](#your-balance-and-the-hold) counts at most 32,768 output tokens.

## Other tools

Most tools that speak "OpenAI-compatible" take the same two values: **base URL** `https://demo-api.routerplus.com/v1` and **API key** `tm_vk_...`. Where a tool asks for the model's limits, use the table in [Before you start](#before-you-start).

### Goose

**Requirements:** the Goose desktop app or the Goose CLI.

**Setup in the desktop app:** open **Settings → Models → Configure providers → Add Custom Provider**.

1. Choose the provider type **OpenAI Compatible**.
2. Enter a display name such as RouterPlus, the API URL `https://demo-api.routerplus.com/v1/chat/completions`, your key, and the model ids you want (for example `claude-sonnet-5-5, gpt-6-luna`), with streaming on.
3. Click **Create Provider**.

**Setup in the CLI:** `goose configure` → **Custom Providers** → **Add A Custom Provider** asks for the same values.

**Config:** Goose stores each custom provider as a file in `~/.config/goose/custom_providers/`. This one reads the key from `TM_API_KEY`:

```json
{
  "name": "custom_routerplus",
  "engine": "openai",
  "display_name": "RouterPlus",
  "description": "RouterPlus gateway",
  "api_key_env": "TM_API_KEY",
  "base_url": "https://demo-api.routerplus.com/v1/chat/completions",
  "models": [
    { "name": "claude-sonnet-5-5", "context_limit": 1000000 },
    { "name": "gpt-6-luna", "context_limit": 1050000 }
  ],
  "supports_streaming": true,
  "requires_auth": true
}
```

Start a session with `goose session --provider custom_routerplus --model claude-sonnet-5-5`.

**Limits:**

- Goose calls GPT-6 and GPT-5 models on `/v1/responses` by itself, which the gateway serves.
- Set `GOOSE_MAX_TOKENS` to give Goose an output limit.

### Zed

**Requirements:** Zed's agent.

**Setup:**

1. Add an OpenAI-compatible provider to Zed's `settings.json` (below). `max_tokens` is the model's context window.
2. Enter the key in **Settings → AI → LLM Providers** (Zed keeps it in the system keychain), or set `ROUTERPLUS_API_KEY` in the environment. Do not put the key in `settings.json`.

```json
{
  "language_models": {
    "openai_compatible": {
      "routerplus": {
        "api_url": "https://demo-api.routerplus.com/v1",
        "available_models": [
          {
            "name": "claude-sonnet-5-5",
            "display_name": "Claude Sonnet 5.5",
            "max_tokens": 1000000,
            "max_output_tokens": 128000,
            "capabilities": { "tools": true, "images": true }
          },
          {
            "name": "gpt-6-luna",
            "display_name": "GPT-6 Luna",
            "max_tokens": 1050000,
            "max_output_tokens": 128000,
            "capabilities": { "tools": true, "images": true }
          }
        ]
      }
    }
  }
}
```

For Claude in Anthropic's format, Zed also has `anthropic_compatible` providers. They have the same shape, with `api_url` set to `https://demo-api.routerplus.com`, without `/v1`.

### VS Code chat with your own key

**Requirements:** VS Code 1.122 or later. VS Code's built-in chat takes your own key, with no GitHub account and no Copilot plan.

**Setup:**

1. In the model picker, choose **Manage Language Models** → **Add Models** → **Custom Endpoint**.
2. Give it a name and your key, and pick the API type.
3. VS Code opens `chatLanguageModels.json`. Add the models there:

```json
[
  {
    "name": "RouterPlus",
    "vendor": "customendpoint",
    "apiKey": "${input:routerplusApiKey}",
    "models": [
      {
        "id": "claude-sonnet-5-5",
        "name": "Claude Sonnet 5.5",
        "url": "https://demo-api.routerplus.com/v1/messages",
        "apiType": "messages",
        "toolCalling": true,
        "vision": true,
        "maxInputTokens": 872000,
        "maxOutputTokens": 128000
      },
      {
        "id": "gpt-6-luna",
        "name": "GPT-6 Luna",
        "url": "https://demo-api.routerplus.com/v1/responses",
        "apiType": "responses",
        "toolCalling": true,
        "vision": true,
        "maxInputTokens": 922000,
        "maxOutputTokens": 128000
      }
    ]
  }
]
```

- `maxInputTokens` plus `maxOutputTokens` must fit the model's context window.
- `toolCalling: true` puts a model in agent mode.

**Limits:** without a GitHub account, inline suggestions and semantic search do not work. Set `chat.utilityModel` to one of these models for titles and commit messages.

### Continue

**Requirements:** Continue. Agent mode needs `tool_use`.

**Config:** in `~/.continue/config.yaml`:

```yaml
name: RouterPlus
version: 1.0.0
schema: v1
models:
  - name: Claude Sonnet 5.5
    provider: openai
    model: claude-sonnet-5-5
    apiBase: https://demo-api.routerplus.com/v1
    apiKey: tm_vk_...
    roles: [chat, edit, apply]
    capabilities: [tool_use]
    defaultCompletionOptions:
      contextLength: 1000000
      maxTokens: 32000
```

To keep the key out of the file, put `TM_API_KEY=tm_vk_...` in `~/.continue/.env` and refer to it from `apiKey` with Continue's secrets syntax.

### aider

**Setup:**

```bash
export OPENAI_API_BASE=https://demo-api.routerplus.com/v1
export OPENAI_API_KEY=tm_vk_...
aider --model openai/claude-sonnet-5-5
```

**Config:** aider warns that it does not know the model. To give it the limits, put a `.aider.model.metadata.json` in your home folder or the repository:

```json
{
  "openai/claude-sonnet-5-5": {
    "max_tokens": 128000,
    "max_input_tokens": 1000000,
    "max_output_tokens": 128000,
    "litellm_provider": "openai",
    "mode": "chat"
  }
}
```

For aider's diff edits, add `.aider.model.settings.yml`:

```yaml
- name: openai/claude-sonnet-5-5
  edit_format: diff
  use_repo_map: true
```

**Limits:** aider sends no output limit (see [the hold](#your-balance-and-the-hold)).

### Anything else

If a tool has a "base URL" box next to its OpenAI key box, set those two values.

## Playground

The [Playground](https://demo.routerplus.com/playground) runs in the browser. Compare any catalog models' answers, costs and providers side by side. Sending a prompt needs an account. See [Playground](/docs/playground).

## SDKs inside the project your agent is editing

| Client | Change |
|---|---|
| OpenAI SDK (any language) | `base_url = "https://demo-api.routerplus.com/v1"`, `api_key = TM_API_KEY` |
| Anthropic SDK (any language) | `base_url = "https://demo-api.routerplus.com"`, `api_key = TM_API_KEY` |
| Raw HTTP | host → `https://demo-api.routerplus.com`, key in `Authorization: Bearer` or `x-api-key` |

- Streaming examples for both surfaces: [Quickstart](/docs/quickstart).
- The Responses API: [POST /v1/responses](/docs/api-responses).
- Migrating a codebase in one prompt: [Migrate](/docs/migration).

## The installation runbook

A coding agent can run this section. Give it to your agent (`read https://demo.routerplus.com/docs/install.md and get me set up`), or follow it yourself.

**Objective:** this environment makes a successful, streamed, billed model call through the gateway.

**Done when:** a streamed completion ends with a usage object that has a `cost` field, and `https://demo-api.routerplus.com/v1/usage` shows the request.

1. **Get a key** (skip if `TM_API_KEY` is already set).
   - The user opens [Sign up](https://demo.routerplus.com/signup), completes Clerk authentication and email verification, and saves the first key shown after sign-in.
   - Signup does not grant free credit. The user adds paid credits in [Billing](https://demo.routerplus.com/console/billing).
   - Existing customers sign in at [https://demo.routerplus.com/login](https://demo.routerplus.com/login) with the same verified email. They create a key at [API keys](https://demo.routerplus.com/console/keys), or copy an existing one from its row.
   - Account creation has no terminal API. Signup and sign-in use Clerk in the browser, and a public signup response has no `dev_verify_link`. Do not call the retired `POST /v1/signup` or local `POST /login` routes, or the retired magic-link submission. Continue client setup when the key is available.

2. **Export the key**, and write it to the project's `.env` if one exists (never commit it):

   ```bash
   export TM_API_KEY=tm_vk_...
   ```

3. **Discover the models.** Any id in this file works in step 4:

   ```bash
   curl -s https://demo.routerplus.com/api/models.json
   ```

4. **Make the test call** on the OpenAI-compatible surface. It works for every chat model, including Claude models, because the gateway translates:

   ```bash
   curl -N https://demo-api.routerplus.com/v1/chat/completions \
     -H "Authorization: Bearer $TM_API_KEY" \
     -H "content-type: application/json" \
     -d '{"model":"claude-haiku-5-5","stream":true,"max_tokens":40,"messages":[{"role":"user","content":"Reply with exactly: MARKETPLACE OK"}]}'
   ```

   Expected: an SSE stream that ends with a usage chunk with a `cost` field, then `data: [DONE]`.

   On HTTP `429` with `error_type: insufficient_quota`, stop. Ask the user to check their balance and spend caps in [Billing](https://demo.routerplus.com/console/billing). Retry **once** after they fix the cause, never in a loop. [Errors](/docs/errors) gives the retry rules for each error class.

5. **Wire the user's stack.** Apply every branch that matches this environment:
   - the OpenAI SDK, Anthropic SDK and raw-HTTP changes from the table above
   - each coding tool the user has, as its section on this page shows (Claude Code, Codex, Cursor, Cline, Kilo Code, OpenCode, the others)

   Edit a tool's settings file only with the user's consent. Never write the key into a file that is committed.

6. **Verify billing.** This is the done-when condition:

   ```bash
   curl -s -H "Authorization: Bearer $TM_API_KEY" https://demo-api.routerplus.com/v1/usage
   ```

   The test call must appear in `recent_attempts` with its token counts and `cost_usd`.

7. **Optional, recommended:** add attribution headers to the user's app: `HTTP-Referer: https://the-users-app.example` and `X-Title: The App Name`. They identify the app for per-app analytics and carry no content.

Finally, report to the user: the org id, where the key is stored, which stacks and tools were wired, and the cost of the test call.

## /llms.txt: start here if you are an agent

```bash
curl -s https://demo.routerplus.com/llms.txt
```

`/llms.txt` has the gateway base URL, browser signup instructions, a raw-markdown link to every doc page, and the model catalog. Its first link is the [Agent integration guide](https://demo.routerplus.com/docs/agent-integration.md). [https://demo.routerplus.com/llms-full.txt](https://demo.routerplus.com/llms-full.txt) has every page in one fetch.

## Next steps

- [Quickstart](/docs/quickstart): the same flow by hand, with streaming examples on both surfaces
- [Migrate](/docs/migration): the one-prompt migration for an existing codebase
- [Apps from Replit, Lovable, Bolt and v0](/docs/app-builders): the prompt that wires an app you build there
- [Errors](/docs/errors): every `error_type`, and the retry rules agents should follow
- [Rate limits & spend caps](/docs/limits): trial RPM, monthly caps, and the 429 with a reset time

---

Source: https://demo.routerplus.com/docs/cli.md

# CLI

The RouterPlus CLI connects your coding tools to the gateway. You approve a key in the browser
once. The CLI stores the key and writes it into each tool's settings. The CLI also has a
coding agent of its own.

The CLI needs Node 24 or later. It has no other dependencies. Run it from a checkout of the
RouterPlus repository:

```bash
node tools/routerplus-cli/main.ts <command> [options]
```

## Commands

| Command | What it does |
|---|---|
| `login` | Opens the consent page, receives the approved key and stores it. |
| `setup <tool>` | Writes a tool's settings to use RouterPlus with the stored key. Tools: `claude-code`, `codex`, `cursor`, `opencode`, `cline`. |
| `status` | Shows the key's prefix, its usage and limit, and the tools that use RouterPlus. |
| `logout` | Deletes the stored key. With a management key, it also revokes the key. |
| `code [prompt]` | Starts a coding agent in the current directory. See [Coding agent](#coding-agent). |

## Log in

```bash
node tools/routerplus-cli/main.ts login
```

1. The CLI starts a server on `127.0.0.1`, on a random port, to receive the approval.
2. The CLI opens the consent page in your browser and prints its URL. If the browser does not
   open, copy the URL.
3. Sign in, select a workspace and click **Approve**. You can set a spending limit on the same
   page.
4. The browser returns to the CLI. The CLI exchanges the one-time code for a key with
   [OAuth PKCE](/docs/oauth).
5. The CLI stores the key in `~/.config/routerplus/credentials.json` with mode `0600`.

The approval must arrive within 5 minutes. The CLI never prints the key. It shows only the
first 12 characters.

| Option | Meaning |
|---|---|
| `--no-browser` | Print the URL and do not open a browser. Use this on a remote machine with port forwarding. |
| `--limit USD` | Ask for a spending limit, for example `25`. You can change it on the consent page. |

## Set up a tool

```bash
node tools/routerplus-cli/main.ts setup claude-code --dry-run
node tools/routerplus-cli/main.ts setup claude-code
```

`--dry-run` shows the lines that would change and writes nothing. Without `--dry-run`, the CLI
does these steps for each file it changes:

1. It copies the file to `<file>.bak`.
2. It writes the new file with mode `0600`.
3. It shows the lines it changed. The key shows as its prefix.

A second `setup` for the same tool changes nothing. The settings are the same as in
[Coding agents](/docs/install):

| Tool | What `setup` writes |
|---|---|
| `claude-code` | `~/.claude/settings.json`: `env.ANTHROPIC_BASE_URL` (`https://demo-api.routerplus.com`, without `/v1`) and `env.ANTHROPIC_AUTH_TOKEN`. It removes `env.ANTHROPIC_API_KEY`, because that variable makes Claude Code open its login screen. Other settings stay. |
| `codex` | `~/.codex/config.toml`: `model_provider = "routerplus"` and a `[model_providers.routerplus]` table with `base_url` `https://demo-api.routerplus.com/v1`, `wire_api = "responses"` and the key in `experimental_bearer_token`. It sets `model = "gpt-6-luna"` only when the file has no model. Other tables stay. |
| `opencode` | `~/.config/opencode/opencode.json`: `provider.routerplus` with `@ai-sdk/openai-compatible`, `baseURL` `https://demo-api.routerplus.com/v1`, the key in `apiKey`, and `claude-sonnet-5-5`, `claude-opus-5-5` and `gpt-6-luna` with image input. It sets `model` only when the file has none. |
| `cline` | Runs `cline auth --provider openai-compatible --modelid claude-sonnet-5-5 --baseurl https://demo-api.routerplus.com/v1` with the key. Cline writes its own files in `~/.cline`. The CLI backs up those files first and shows what changed. If the Cline CLI is not on `PATH`, the CLI shows the steps for the extension. |
| `cursor` | Writes no file. Cursor keeps the key in its settings screen, so the CLI shows the steps. `--copy-key` puts the key on the clipboard for you to paste. |

After `setup`, the Codex and OpenCode files hold your key. Do not commit or share them.

## Status

```bash
node tools/routerplus-cli/main.ts status
```

```
Key: tm_vk_1a2b3c... (approved by you@example.com)
Stored in: /Users/you/.config/routerplus/credentials.json
Gateway: https://demo-api.routerplus.com
Usage: $1.50 (limit $25.00, $23.50 left)
Rate limit: 300 requests per 1m
Tools:
  claude-code  configured
  codex        not configured
  cursor       set by hand
  opencode     configured with another key
  cline        not configured
```

- Usage and the limit come from `GET https://demo-api.routerplus.com/v1/key`.
- "configured with another key" means that the tool uses RouterPlus with a different key. Run
  `setup` for that tool to use the stored key.

## Log out

```bash
node tools/routerplus-cli/main.ts logout
```

`logout` deletes the stored key. The key still works until you disable it in the console, under
Access > [API keys](https://demo.routerplus.com/console/keys).

To revoke the key from the CLI, give a management key. Use `--management-key tm_mk_...` or the
`ROUTERPLUS_MANAGEMENT_KEY` variable. The CLI finds the key by its prefix in `GET /v1/keys` and
calls `DELETE /v1/keys/<id>`.

The tools that you set up keep the key in their own settings. Run `setup` again after the next
`login`, or remove the settings.

## Coding agent

`code` starts a coding agent in your terminal. The agent reads, searches and changes the files
in the current directory, and runs commands there. It uses the key in `ROUTERPLUS_API_KEY` when
that variable is set. Otherwise it uses the stored key.

```bash
cd my-project
node /path/to/routerplus/tools/routerplus-cli/main.ts code
node /path/to/routerplus/tools/routerplus-cli/main.ts code --model claude-sonnet-5-5 --max-cost 2
```

Type a task at the `>` prompt. The agent streams its answer and asks you before each change. It
keeps the conversation until you type `/clear` or `/exit`.

| Command | What it does |
|---|---|
| `/cost` | Shows the cost of the session in USD. |
| `/clear` | Starts a new conversation. |
| `/exit` | Quits and shows the session cost. |

### One-shot mode

Give the task as an argument and pipe stdin. The agent does the task, prints the cost and exits.

```bash
node tools/routerplus-cli/main.ts code --yes "Add a test for parseDate" < /dev/null
```

With no argument, the agent reads the task from stdin. In this mode the agent cannot ask you
for approval:

- Without `--yes`, it refuses writes, edits and commands.
- Without `--auto-approve-reads` or `--yes`, it also refuses reads.

The exit code is `0` when the agent finishes and `1` when it stops early.

### Tools

| Tool | What it does | Asks you |
|---|---|---|
| `read_file` | Reads a text file, with line numbers. Takes `offset` and `limit` in lines. | Unless `--auto-approve-reads` or `--yes` |
| `list_dir` | Lists a directory. | Unless `--auto-approve-reads` or `--yes` |
| `search` | Searches file contents for a regular expression. Skips `.git`, `node_modules`, binary files and files over 1 MB. Returns at most 100 matches. | Unless `--auto-approve-reads` or `--yes` |
| `write_file` | Creates or replaces a file. Shows a diff first. | Unless `--yes` |
| `edit_file` | Replaces one string in a file. The string must occur once. Shows a diff first. | Unless `--yes` |
| `run` | Runs a shell command. Stops the command after 120 s and keeps the first 20,000 characters of output. | Unless `--yes` |

### Safety

- Every path must be inside the current directory. The agent refuses other paths, also through
  a symbolic link.
- Before each write or edit, the agent shows a unified diff.
- These files can hold secrets: `.env`, `.env.*`, `credentials.json`, `*.pem` and names that
  contain `key`. The agent sends their contents only when you type `y`, also with `--yes`.
  `search` does not look in these files.
- `--yes` lets the model change files and run any command. Use it only in a directory you can
  restore, for example a clean git checkout.

### Options

| Option | Default | Meaning |
|---|---|---|
| `--model ID` | `routerplus/auto` | The model. `routerplus/auto` lets the gateway select a model for each request. See [Auto router](/docs/auto-router). |
| `--yes`, `-y` | off | Run writes, edits and commands without asking. |
| `--auto-approve-reads` | off | Read, list and search without asking. |
| `--max-cost USD` | none | Stop when the session cost reaches this amount. The cost is the sum of `usage.cost` in the answers. |

Each request sends these headers:

- `x-session-id`, with one id for the conversation. The gateway keeps the conversation on one
  provider, so the prompt cache stays warm. `/clear` starts a new id.
- `x-tm-labels: client=routerplus-code`, so you can find the agent's requests in your
  usage.

## Options

| Option | Default | Meaning |
|---|---|---|
| `--api URL` | `https://api.routerplus.com` | The gateway. `login` stores it, and later commands use the stored value. `code` also reads `ROUTERPLUS_API_URL`. |
| `--console URL` | `https://app.routerplus.com` | The console with the consent page. |

Both URLs must use https. Plain http works only on `localhost`, `127.0.0.1` and `[::1]`.

---

Source: https://demo.routerplus.com/docs/app-builders.md

# Apps from Replit, Lovable, Bolt and v0

Use this page when an app that you build in Replit, Lovable, Bolt or v0 has its own AI
features, for example a chat box, a summary button or an agent. The app calls RouterPlus with
your key. You select the model, and RouterPlus bills every call to one balance.

The builder's own agent cannot use RouterPlus. Replit Agent, Lovable, Bolt and v0 run on the
builder's models, and the builder bills for them.

## Before you start

1. Create an account at [Sign up](https://demo.routerplus.com/signup). Signup does not add credit. Buy
   credits in the console, under Billing > Credits ([Billing](https://demo.routerplus.com/console/billing)).
2. Create a key for the app in the console, under Access > API keys
   ([API keys](https://demo.routerplus.com/console/keys)). A key looks like `tm_vk_...`.
3. Set a monthly spend cap on the key with **Set cap**. Then a bug or a loop in the app cannot
   spend all of your balance.
4. Select a model id from [`/api/models.json`](https://demo.routerplus.com/api/models.json).
   `claude-sonnet-5-5` is a good default. `gpt-6-luna` and `claude-haiku-5-5` cost less.

## The prompt

Paste this text into the builder's chat:

```text
Use my own API key. Call AI through RouterPlus: OpenAI-compatible, base URL https://demo-api.routerplus.com/v1, model claude-sonnet-5-5, key in a secret named TM_API_KEY.
```

The builder writes code that calls the gateway with the OpenAI SDK. Then it asks you for the
secret. If the builder offers its own AI service instead, say no. The builder bills that
service.

For an app with more AI features, add these rules:

```text
Rules for every AI call:
- Use the official OpenAI SDK with baseURL https://demo-api.routerplus.com/v1 and apiKey from the TM_API_KEY secret.
- Call it only from server-side code (an API route, a server action or a backend function). Never put the key in browser code.
- Send max_tokens on every request. Stream answers that a person reads as they arrive.
- Copy model ids from https://demo.routerplus.com/api/models.json. Never invent one.
- On an error, read error.metadata.error_type. On rate_limit, wait retry-after seconds, then retry, at most twice. On insufficient_quota, show "AI is unavailable" and do not retry.
- Docs for the gateway: https://demo.routerplus.com/llms.txt
```

## Store the key as a secret

Keep the key in the builder's secret store, with the name `TM_API_KEY`. Do not paste the
key into the chat or into code.

| Builder | Where the secret goes |
|---|---|
| Replit | **Tools → Setup → Secrets → New Secret**, key `TM_API_KEY`, then **Add Secret**. The app reads it as an environment variable. A published app does not get workspace secrets: add the secret again under **Publishing → Adjust settings → Production app secrets**. |
| Lovable | **More → Cloud → Secrets → Add secret**, or the secure form that Lovable shows in the chat when it asks for the key. Only the app's server code (its backend functions) can read it. |
| Bolt | The **Secrets** page of Bolt Cloud (the database icon at the top): name `TM_API_KEY`, the key as its value, then **Create secret**. The app's server functions read it. |
| v0 | The project's **Settings → Environment Variables** (the **Vars** panel), for the environments that need it. The v0 preview reads the Development value. |

Keep the key on the server. The gateway accepts calls from any web page. Thus anyone who opens
the app can copy a key from browser code. Some builders put a variable into browser code when
its name starts with `VITE_` or `NEXT_PUBLIC_`. Do not give the key such a name.

## What the code looks like

The builder writes code like this. Use it to check the builder's work.

TypeScript, a server route with the official `openai` package:

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://demo-api.routerplus.com/v1",
  apiKey: process.env.TM_API_KEY,
});

export async function POST(req: Request) {
  const { question } = await req.json();
  const reply = await client.chat.completions.create({
    model: "claude-sonnet-5-5",
    max_tokens: 1024,
    messages: [{ role: "user", content: question }],
  });
  return Response.json({ answer: reply.choices[0]?.message.content });
}
```

Python, a server with the official `openai` package:

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

reply = client.chat.completions.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Summarize this note in one line."}],
)
print(reply.choices[0].message.content)
```

Every chat model in the catalog, Claude and GPT, answers on `/v1/chat/completions`. GPT models
also work with the OpenAI SDK's Responses API (`client.responses.create`, with
`model: "gpt-6-luna"`). See [POST /v1/responses](/docs/api-responses). Claude models are a
preview on that API, so call Claude on `/v1/chat/completions`.

## Limits

- The builder's agent keeps its own models and pricing. Your RouterPlus balance pays only
  for the calls that your app makes.
- Each call bills at the model's price. `usage.cost` in each answer gives the amount. On
  the Responses API, it is in `response.usage`. [GET /v1/usage](/docs/api-usage) and the console
  show what the app spent.
- Rate limits and spend caps apply to the app's key. See
  [Rate limits & spend caps](/docs/limits).

## Where next

- [Coding agents](/docs/install): Claude Code, Codex, Cursor, Cline and more, on your own machine
- [Agent integration guide](/docs/agent-integration): what a coding agent needs to build the gateway into an app
- [POST /v1/chat/completions](/docs/api-chat-completions): the request and the answer, field by field
- [Errors](/docs/errors): each error class, and what the app should do

---

Source: https://demo.routerplus.com/docs/for-providers.md

# List your models

Sell your inference on RouterPlus. You send one model document, your models pass a published conformance bar, and RouterPlus pays you on the schedule you choose.

- **Pricing is pass-through.** Buyers pay your listed price with no markup.
- **Routing follows live health and published metrics.** You see the same numbers. Routing is not negotiated.

**To apply:** email **providers@evo-hq.com** with your model document attached. A person at RouterPlus and the automated suite handle onboarding. It usually completes in the same week.

## The model document

The model document is one JSON document with your endpoint, models, prices, capacity and data policy, pushed to RouterPlus as a versioned manifest. RouterPlus does not poll your `/v1/models`, so nothing changes between your updates. RouterPlus adds new models. A self-serve listing changes its own prices, limits and on/off state through the API (see [Changing your prices yourself](#changing-your-prices-yourself)).

Validate the document against the JSON Schema before you send it: [`/docs/provider-manifest.schema.json`](/docs/provider-manifest.schema.json).

```json
{
  "manifest_version": "1",
  "provider": {
    "id": "acme",
    "name": "Acme Inference",
    "privacy_policy_url": "https://acme.example/privacy",
    "terms_of_service_url": "https://acme.example/terms",
    "status_page_url": "https://status.acme.example",
    "prompt_logging": "none"
  },
  "endpoint": {
    "base_url": "https://api.acme.example/v1",
    "dialect": "openai",
    "api_key_env": "ACME_API_KEY"
  },
  "settlement": {
    "mode": "arrears",
    "interval": "weekly",
    "billing_contact": "billing@acme.example"
  },
  "models": [
    {
      "id": "acme-large-1",
      "display_name": "Acme Large 1",
      "context_length": 128000,
      "max_output_tokens": 16384,
      "streaming": true,
      "is_ready": false,
      "pricing": [
        { "type": "prompt", "unit": "token", "cost_usd_per_million": "0.90" },
        { "type": "completion", "unit": "token", "cost_usd_per_million": "3.60" }
      ],
      "capacity": [
        { "type": "completion", "unit": "token", "per": "minute", "value": 2000000 }
      ]
    }
  ]
}
```

### Identity

- `id` is the identifier RouterPlus sends when it calls your API. It is never an alias.
- `display_name` is the name buyers see.
- A listing is text (chat completions) unless `output_modality` is `"image"` or `"decisions"`.
- A text listing declares `streaming`. Use `true` when your endpoint streams SSE. Use `false` (OpenAI dialect only) when it answers in one JSON body.

**Image listings** are served on `POST /v1/images/generations` only. They need no `context_length` or `streaming`. They must declare `max_output_tokens` (the per-image ceiling used for the reservation) and `supported_parameters` for `n` and a `size` or `aspect_ratio` enum.

**Decisions listings** return typed answers to questions about a state. They are served on `POST /v1/decisions` only, and need a `context_length` and no `streaming`.

Your endpoint declares `decisions_path`, the path RouterPlus POSTs the request to. If your API picks the model by its URL, each decisions model
declares its own `decisions_path` instead (Cloudflare Workers AI: `/clef` and `/clef-flash`), and the endpoint needs none. The body still carries the model's `upstream_id` (or `id`) as `model`.

The model declares `decisions_wire`, the question format your API speaks. Buyers send questions in that format, and the gateway sends them to you in it:

- `"systemone"`: TypeSafe's System One API, which Jev, the Bespoke
Nimble models, Clef, Clef-flash, Decider 2B, Kev 4B and Perplexity Decider v1 27B use. This is the default when the field is absent.
- `"gliner"`: Fastino's GLiNER schema, sent in an OpenAI chat-completions envelope. `decisions_path` can then be `/chat/completions`.

Rules for decisions listings:

- Chat traffic never reaches a decisions listing, even on a chat path.
- Two listings of one model id must declare the same wire.
- RouterPlus refuses `decisions_wire` on a listing that is not `"decisions"`.

### Decisions endpoint options

**`endpoint.response_envelope`.** Declare this if your API wraps every answer and error in its own transport envelope. The gateway opens it before it reads the answer. Only a manifest of decisions models may declare it. The one value today is `"cloudflare_v4"`, Cloudflare's REST envelope:

- A 200 is `{"result": <answer>, "success": true, "errors": [], "messages": []}`.
- An error is `{"errors": [{"code": ..., "message": ...}], "success": false, "result": {}}`.
- A 200 without `success: true` is a failed answer, and RouterPlus bills nothing for it.

**`endpoint.decisions_format`.** Declare this if your API asks System One's kinds of questions (a choice, a score, a yes/no) in its own body format. The one value today is `"openai"`: OpenAI's `POST /v1/decisions` format, with `input` and an ordered `questions` array of `predicate`, `choice` and `score` questions, answered by an ordered `answers` array. Your models still
declare `decisions_wire: "systemone"`, and buyers still write System One. The gateway translates both ways. Only a manifest of System One decisions models may declare it. OpenAI's GPT-6 Luna Decisions uses it.

**System One body options.** Two booleans for a decisions model whose `decisions_wire` is `"systemone"` (or absent). RouterPlus refuses them on any other listing. Absent means `false`.

| Option | Meaning |
|---|---|
| `decisions_state_per_question` | Your API runs one prompt per question, each with the whole state, and bills the state once per question (as Perplexity's decider does). The gateway then reserves money and tokens for the state once per question. |
| `decisions_image_parts` | Your API reads an object whose `type` is `"image_url"`, at any depth of the state or a question, as an image. A decisions listing takes text and JSON, so the gateway refuses such a body with a typed 400 before it reserves money. The body never reaches you. |

### Chat listing options

**`input_modalities`** lists what a message to the model may carry: always `"text"`, plus any of `"image"`, `"file"` (a document such as a PDF), `"audio"` and `"video"`. Absent means text only. RouterPlus refuses the field on an image or decisions listing.

- Buyers see it on the model page and filter by it. `/v1/models` lists it. The model page shows an input only when every listing of the model declares it.
- The gateway sends a part only to a listing that declares its kind. A part that no listing declares gets a typed 400.
- A declared part reaches you as sent from the surface that speaks your dialect. Across dialects, the gateway refuses it with a typed 400.

**`"chat_api": "responses"`** (OpenAI-dialect endpoints only) makes the gateway call your Responses API (`POST /responses` under `base_url`). Absent means Chat Completions. Declare it when the model takes function tools only there, as OpenAI's GPT-6 and GPT-5.6 do. Buyers still call either chat surface. Parameters with no Responses form on these models (`stop`, `n`, `seed`, `logprobs`, `logit_bias`, the penalties) are dropped and recorded. An output limit under 16 tokens is sent as 16.

**`"fixed_sampling": true`** declares that the model takes only its default sampling, and you refuse `temperature` (other than 1), `top_p` and `top_k` with a 400, as Anthropic does for the Claude 5 models and OpenAI for GPT-6 and GPT-5.6. The gateway then leaves these parameters out on every route of the model and records the drop. Absent means you take them.

### Serving a model the marketplace already lists

If another provider already lists the model (for example, the marketplace id `claude-sonnet-4-5`), list it under that id. Set `upstream_id` to the name your endpoint expects:

```json
{ "id": "claude-sonnet-4-5", "upstream_id": "anthropic/claude-sonnet-4.5", "...": "..." }
```

- Requests to you carry `upstream_id`. Conformance check C7 expects you to echo it.
- Your prices must match the other providers of that id, or the catalog build rejects them. Two exceptions:
  - You set your own prices (see [Changing your prices yourself](#changing-your-prices-yourself)).
  - Your listing is opt-in (`"routing": "opt_in"`). Then you serve only requests that name you (`provider.only` or `provider.order`), at your own price.
- `provider.priority` orders the providers of one model: `0` for a first-party lab, `1` for an aggregator fallback. Routing tries the lower number first.

### Endpoint

- `dialect` is `"openai"` (Chat Completions) or `"anthropic"` (Messages). Buyers reach your models from both surfaces, and the gateway translates.
- `api_key_env` names the secret slot for the key you issue to RouterPlus. The key never appears in a manifest, page or log.

### Pricing

Entries are `{type, unit, cost_usd_per_million}`. Costs are decimal strings in USD, never floats. Types are `prompt`, `cached_prompt`, `cache_write`, `completion` and `internal_reasoning`. The unit is `token`.

- **Declare what you charge.** A SKU you leave out bills its tokens at your base `prompt` or `completion` rate. A missing SKU is never free.
- **Price changes are append-only.** Each change is a new effective-dated row. A request bills at the price in force when it arrived, so your statement and the buyer's bill agree.
- Pricing is flat per token. Conditional overrides, time-of-day windows and per-request SKUs are not supported yet.

### Changing your prices yourself

RouterPlus can make your listing self-serve. You then change its prices and some limits through this API.

| | |
|---|---|
| Endpoint | `https://app.routerplus.com/v1/provider/manifest` |
| Auth | `Authorization: Bearer tm_pk_…`, the provider key that RouterPlus issues you. One key speaks for one provider. Keep it secret. |
| Methods | `GET`, `POST` (replace), `PATCH` (change named fields) |

**What you can change:**

- `provider.privacy_policy_url`, `provider.terms_of_service_url` and `provider.status_page_url`.
- For each model you list: `pricing`, `provider_pricing`, `enabled`, `context_length` and `max_output_tokens`.

RouterPlus owns all other fields. Leave them out or send them unchanged. A write that changes one is refused with the field's name.

- You cannot add or remove a model.
- `"enabled": false` takes a model off the marketplace routes and the public lists. It does not stop a buyer who sends their own key for your service (BYOK).
- A limit (`context_length`, `max_output_tokens`) can only go down: at most the listed value, and `max_output_tokens` at most `context_length`.
- For an image or video model, RouterPlus owns `max_output_tokens`.
- For a decisions model, RouterPlus owns `context_length` and `max_output_tokens`.

#### GET

```
GET /v1/provider/manifest
```

GET returns the state in force (the RouterPlus listing with your edits on it) and these fields:

| Field | Meaning |
|---|---|
| `revision` | The latest revision. |
| `in_force` | The revision in force. |
| `frozen` | Whether RouterPlus froze your listing. |
| `price_feed` | Your price feed, if you have one. |
| `pending` | Scheduled changes: every one that will take effect, and the 20 latest that never will. A change with `"shadowed": true` never takes effect, because a later write takes effect first. `shadowed_omitted` counts the rest. |
| `pushed_provider_fields` | On `in_force` and on each pending change: the URLs that you set. The other URLs follow the RouterPlus listing, also when RouterPlus changes them later. |

**How a write treats a URL or a limit.** A write changes a URL or a limit only when it sends `null` or a value different from the state. Leaving the field out of a PATCH, or sending the value in the state, keeps your earlier value, even when the RouterPlus listing hides it. For example:

1. You lower `context_length` to 6000.
2. RouterPlus lowers its listing to 4000. GET shows 4000.
3. A PATCH of a price only, `PATCH {}`, or a POST of your GET body keeps your 6000.
4. RouterPlus raises its listing to 8192. Your 6000 applies again.

Send `null` to follow the RouterPlus listing again. This is a change even when only storage changes: its `changes` show `pushed_provider_fields` (before and after) or `lowered_context_length` / `lowered_max_output_tokens` (your value before, `null` after).

#### POST

POST replaces all of your editable state, from its `effective_from`. Send every model you list. An editable field you leave out goes back to the RouterPlus value:

- No `provider_pricing`: RouterPlus pays you the buyer price.
- No `enabled`: the model is on.
- No limit: the listed limit.
- No URL: the listed URL.

When the state already shows the listed value for a limit or URL, leaving it out keeps what you set before (see above). The safe sequence is GET, edit, POST. A POST replaces every pending change that starts at or after its `effective_from`, and lists them in `superseded`. To keep a pending change, use PATCH.

```
POST /v1/provider/manifest
Idempotency-Key: 6f1c…

{ "manifest": { "provider": { "id": "your-id" },
                "models": [ { "id": "your-model",
                              "pricing": [ { "type": "prompt", "unit": "token", "cost_usd_per_million": "0.40" },
                                           { "type": "completion", "unit": "token", "cost_usd_per_million": "1.60" } ] } ] },
  "effective_from": "2026-11-01T00:00:00Z", "expect_revision": 3 }
```

#### PATCH

PATCH changes only the fields you name. `null` goes back to the RouterPlus value (`"provider_pricing": null` means RouterPlus pays you the buyer price).

```
PATCH /v1/provider/manifest

{ "models": [ { "id": "your-model", "context_length": 65536 },
              { "id": "your-old-model", "enabled": false } ] }
```

A PATCH never cancels a change that you or your price feed scheduled.

- It applies to the state in force at its `effective_from` (default: the next 5-minute mark), and to each later pending change, which RouterPlus writes again at its own time with your PATCH on it.
- The answer's `revisions` lists every revision written, each with `effective_from`, `diff`, `changes` and `reissues` (the pending change it replaces). `superseded` lists the old pending changes. `head` is the latest revision: send it as the next `expect_revision`.

Example: you scheduled a price increase for tomorrow, and today you PATCH `"enabled": false` on a model. RouterPlus writes two revisions: the model is off now, and tomorrow the increase starts with the model still off, with no new notice.

RouterPlus checks each revision against what buyers would pay at its time without your write. If one is refused, nothing is written, and the 422 names its time.

#### The rules

**Timing**

- **Every change starts at a 5-minute mark** (UTC 10:00, 10:05, 10:10), the first one at least 2 minutes after your write. A write at 10:02:30 starts at 10:05, and one at 10:03:30 at 10:10.
- A set `effective_from` moves to the first mark at or after it: 10:07:10 becomes 10:10, and 11:00 stays 11:00. The answer's `effective_from` gives the start.
- A write cannot change what starts less than 2 minutes after it.
- **A stop applies at once.** A write with no future `effective_from` whose only changes turn a model off or lower a limit applies when written (`"timing": "at_once"`). If it also changes a price, what RouterPlus pays you, a URL, turns a model on, or replaces a pending change with a POST, all of it waits for the mark.
- `effective_from` is ISO 8601 with a zone, for example `2026-11-01T00:00:00Z`. It must be a real time (`2026-02-30` or hour `24` is refused), not in the past, and at most 90 days ahead.

**Notice**

- **Increases need notice: 1 hour** for every provider, unless RouterPlus tells you otherwise. Set `effective_from` that far ahead. Decreases, turning a model off or on, and lowering a limit need none.
- The notice is checked against the time you set. The price is compared with what buyers pay at that time, and with every change you already scheduled.
- A model that is off has no buyer price. When you turn it on, RouterPlus compares its price with the price buyers last paid before it went off, or the listed price if it was never on. A higher price needs notice. The price you set while it is off is not compared.

**Price limits**

- **Margin.** On each SKU, your price minus `provider_pricing` (what RouterPlus pays you) may not be below zero, or below the negative margin the RouterPlus listing already accepts.
- **Floor.** Your key and feed may not set a price below one tenth of the listed price. A SKU listed at 0 has no floor. RouterPlus operators can set a lower price if you ask.
- A SKU listed at 0 may stay 0. To make a priced SKU free, ask RouterPlus first.
- These three rules apply only to SKUs that your write changes (price, or for the margin what RouterPlus pays you), including by replacing a pending change. You can keep a low or free price that operators set, but not extend it past the end they scheduled. A later change to the listing's margin does not block a write that keeps a SKU as it is.
- At most $1,000 per million tokens per SKU.
- A change of more than 10× needs `"confirm_large_change": true`. A change to or from 0 falls under the free-price rule instead.
- A self-serve listing is outside the identical-price rule in [Serving a model the marketplace already lists](#serving-a-model-the-marketplace-already-lists). A buyer pays the price of the provider that served them.

**Pending changes**

- At most 20 pending changes per listing, and 200 scheduled revisions in all, counting `shadowed` ones. A write that would leave more is a 422 `too_many_pending`.
- A POST with no `effective_from` ends every pending change from its mark on, so it is always possible.
- A pending change that your write makes equal to the state before it is superseded.

**Body**

- A policy URL must be `https://` or `http://`, at most 2,048 characters, with no spaces or control characters.
- A price entry has only `type`, `unit` and `cost_usd_per_million`. Another key is refused with its path.
- A body is a 400 if a key or string contains U+0000 (NUL) or a lone UTF-16 surrogate (such as `\ud800` without its pair), or if it is nested more than 32 levels deep (the top-level object is level 1). This check runs after the key, rate, listing and query checks, and before the write checks.

#### Answers

- `?dry_run=1` (or `true`) checks a POST or PATCH and writes nothing. `?dry_run=0` (or `false`) writes. Any other value or query parameter is a 400.
- A write answers 201 with these fields. A dry run gives the same times.

| Field | Meaning |
|---|---|
| `revision` | The first revision written. |
| `head` | The latest revision after the write. |
| `revisions` | Every revision written. |
| `diff` | Each price SKU of the first revision: `increase`, `decrease` or `new`. |
| `changes` | The other fields that changed. |
| `superseded` | The pending changes that the write replaces. |
| `effective_from` | When the change starts: a 5-minute mark, or the time of the write for a stop. |
| `requested_effective_from` | The time you sent, or the time of the write. |
| `timing` | `"next_mark"` or `"at_once"`. |

- A write that changes nothing answers 200 with `"unchanged": true`. It counts toward your limit.
- **Idempotency.** The same `Idempotency-Key` with the same body returns the first answer, also when that write changed nothing or the listing is frozen. With another body it is a 409 (a 423 while frozen). The key covers the whole write and all its revisions. A refused write (any 4xx) does not use up its key.

#### Errors

Every error has `error.code`, `error.message` and `error.errors` (each with its field path).

| Status | Code | Why |
|---|---|---|
| 400 | `invalid_request` | The body is not a write, a field has the wrong type, the body has a NUL or a lone surrogate or is nested too deep, or the query is not `dry_run` |
| 401 | `unauthorized` | The key is missing, unknown or revoked |
| 403 | `forbidden` | The listing is not self-serve |
| 403 | `prices_from_feed` | RouterPlus reads your prices from your price feed |
| 404 | `not_found` | RouterPlus has no listing for your key's provider |
| 405 | `method_not_allowed` | Use GET, POST or PATCH |
| 409 | `conflict` | `expect_revision` is stale, or the Idempotency-Key was used for another body (while your listing is frozen: 423) |
| 413 | `payload_too_large` | The body is more than 256 KB |
| 422 | `manifest_refused` | A rule above, or the database refused a value of the write; see below |
| 422 | `too_many_pending` | The write would leave more than 20 pending changes, or more than 200 scheduled revisions in all |
| 423 | `frozen` | RouterPlus froze your listing. Writes wait until it is unfrozen |
| 429 | `rate_limited` | More than 30 writes, or 600 requests, an hour per key |
| 503 | `unavailable` | The API cannot reach its database. Try again after `retry-after` seconds |

A 422 lists every problem found in `errors`, with a `note`. RouterPlus checks in stages:

1. The fields you may change.
2. The manifest rules.
3. The prices.
4. The notice and 10× rules.

A stage runs only when the stages before it pass, so a fixed write can show new problems. Use `?dry_run=1` first. An answer lists at most 100 errors, and `errors_omitted` counts the rest.

#### Limits

- At most 30 writes an hour per key. These answers count as a write:
  - 201 (accepted)
  - 200 with `"unchanged": true` (no change)
  - 403 `prices_from_feed`, 409 `conflict` and 422 (`manifest_refused`, `too_many_pending`) (refused)
- No other answer counts: not a 400, 401, 403 `forbidden`, 404, 405, 413, 423, 429 or 503, not a dry run, and not the replay of an `Idempotency-Key`.
- At most 600 requests an hour per key, of every kind.
- A body is at most 256 KB.

#### Price feed

You can publish your prices instead of writing them. RouterPlus sets up the feed in its listing (`endpoint.price_feed`) and reads it every hour by default. The feed must use https, and RouterPlus does not follow redirects. A read stops after 20 seconds or 4 MiB. A UTF-8 byte order mark is fine. Two formats:

| Format | Shape |
|---|---|
| `openai_models` | `{"data": [{"id", "pricing": {"prompt", "completion", "input_cache_read", "input_cache_write", "internal_reasoning"}}]}`. USD per token, as decimal strings. Models match by the id you serve them under. A `"0"` cache or reasoning price means "not billed separately" and leaves that price as it is. RouterPlus pays you the price you publish, so there is no `provider_pricing`. |
| `routerplus` | `{"models": [{"id", "pricing", "provider_pricing"}]}`, in this document's price form, with RouterPlus model ids. No `provider_pricing` means RouterPlus pays you the buyer price. A model entry has only these three keys, and a price entry only `type`, `unit` and `cost_usd_per_million`. Another key (a `currency`, for example) makes it malformed. |

**What a read changes.**

- Only listed models change. A model your feed leaves out keeps its price.
- With `openai_models`, only the prices you publish change. With `routerplus`, an entry is the model's full price list: a SKU it leaves out bills like its base (cached prompt and cache write like the prompt, reasoning like the completion).
- One malformed entry for a listed model voids the whole read.

**When a fed price starts.**

- A **decrease starts at the next 5-minute mark**, at least 2 minutes after the read.
- An **increase starts at the first mark after the notice period plus 5 minutes**, counted from the read that first saw it. Until then, the old price stays on the SKUs that rise.
- A later read does not move a scheduled increase, and a read with the same prices writes nothing. A decrease on another SKU or model starts at the next mark without postponing the increase.
- For a price that keeps rising, RouterPlus keeps at most two scheduled increases: the first announced and the newest. A higher read replaces the ones between, and the earlier price stays until the newest starts. A higher price only ever starts later, and does not block your other changes.
- RouterPlus replaces only increases your feed scheduled, never ones you or operators scheduled, even if a feed read wrote them again (for example, to carry a decrease of another model). A feed increase stays the feed's when you PATCH another field of it.
- A read back at the price in force cancels the increase an earlier read scheduled, and writes the prices in force again from the next mark. A price that goes up and down leaves at most one scheduled increase.
- Your other scheduled changes stay. A model you scheduled to turn off still turns off at its time.
- A lower price that operators scheduled stays until your increase can start (the notice period plus 5 minutes).

**When a read is refused.**

- RouterPlus refuses a change of more than 10x against the price in force, a price of zero (unless free prices are allowed for your listing), or a price below the floor (one tenth of the listed price). Operators then look at it.
- The 10x rule compares only with the price in force, and only for prices the read changes. A later read with the right price corrects a wrong scheduled increase, even if it is more than 10x lower.
- If operators scheduled a change of more than 10x and your feed publishes that price, the read is not refused, and the price starts at the operators' time.
- With `openai_models`, a read is also refused while a revision in force or pending pays you your own cost (`provider_pricing`, from before the feed). Operators must clear that cost first. RouterPlus never drops it without a decision.
- While your listing is frozen, the feed is not read. A freeze during a read stops its write.

**Your writes while a feed is on.** Your writes cannot change prices (403 `prices_from_feed`). Other fields can change at any time. A PATCH keeps a pending fed increase at its time. A POST that replaces one changes prices, so it is a 403.

#### When gateways apply a change

- Gateways read your revisions, including future ones, at least every 30 seconds, so all gateways switch at the mark.
- `/api/models.json` lists a scheduled change within about 2.5 minutes. The public pages show only the price in force.
- A request bills at the price in force when it arrived, even if the gateway plans it after the change.
- A stop that applies at once reaches every gateway within about 30 seconds.

#### Freezes

While RouterPlus freezes your listing:

- Buyers pay the listed price, and RouterPlus pays you the listed cost.
- Only your prices are held back. A model you turned off stays off, and a lowered limit stays lowered.
- Your key and your price feed cannot write.
- A freeze and an unfreeze reach every gateway within about 30 seconds, without waiting for a mark.

On unfreeze, your revision in force applies again, and so does each pending change that starts within the notice period. Operators must confirm any of them above the listed price before the unfreeze, including one where a model is off: after the unfreeze you can turn it on with no notice at the price buyers last paid while it was on.

If RouterPlus turns self-serve off for your listing, the gateways ignore all your revisions, prices and model states, and a model you turned off is on again at the listed limits. RouterPlus also freezes the listing when it turns self-serve off, removes the listing, or finds the listing's file in the RouterPlus repository not valid. When self-serve is on again, RouterPlus unfreezes it with the same check.

### Capacity

Entries are `{type, unit, per, value}`:

- `type` is `request`, `prompt` or `completion`.
- `per` is `minute`, `hour` or `day`.
- `unit: "token"` is required for the two token types.
- Identity is type plus window. Validation rejects duplicates.
- Absent capacity means undeclared. It does not mean zero.

Capacity is information for us, not a setting. Validation checks it. Routing and admission do not read it. Admission enforces a house pool for your deployment, set by hand by RouterPlus operators from what you declare and what you tell RouterPlus (see [Limits and capacity](/docs/admission)). Above that pool, your own 429 is the backstop. It pauses the pool for its `retry-after` (1 to 60 seconds).

### Parameters

You may declare `supported_parameters` per model.

- On the image route it is enforced. A value outside a declared enum is a 400 that names the field.
- An image listing must declare `n` and a `size` or `aspect_ratio` enum, or it fails validation. `quality` and `resolution` are optional, but must be enums when declared.
- On chat routes, the gateway applies its own per-dialect allowlists and records every parameter it removes (the buyer sees them in `x-tm-dropped-params` and their audit endpoint). Declare parameters accurately: chat enforcement will use them when it ships.

### Data policy

`prompt_logging` is `"none"` or `"retained"`. Your model pages publish it next to your privacy policy and terms URLs, and buyers filter on it, so it must be true. See [Data policy](/docs/data-policy).

### Lifecycle

- **`is_ready: false`** stages a model: validated and conformance-testable, but not listed or routed. Set it to `true` when ready.
- **`listed: false`** accepts a model by name only. It is routed, priced and billed like any other model, but no list shows it (`GET /v1/models`, the Models page, `/api/models.json`, the Playground). Use it for a name that you move from model to model, such as Bespoke Labs' `nimble-latest`. Give it its own id, its own `upstream_id` and the price you charge for that name. Two listings of one model id must agree on it.
- **`deprecation_date`** (ISO date): after this date the model leaves the catalog.
- **`released`** (ISO date): the lab's release date, or the day the model first listed publicly. The Models page and the Playground sort newest first, after a few featured models.
- Date-pinned requests (`acme-large-1-20260901`) resolve to your listed model for routing and billing. Your API gets the buyer's original id and decides whether to serve the snapshot.

## The conformance bar

RouterPlus runs seven checks against every model, on the stream and non-stream paths, at listing time and on every catalog change. RouterPlus emails you the report.

| Check | Requirement |
|---|---|
| C1 | A non-stream completion returns provider-reported usage tokens. Usage is the billable record, so a model without it cannot be listed. |
| C2 | Streamed responses are valid SSE, and every frame parses. |
| C3 | Streams end with their dialect's terminal marker (`[DONE]` / `message_stop`). |
| C4 | Usage tokens are reported in the stream. |
| C5 | An unknown model returns a parseable 4xx JSON error, never a 200 or HTML. |
| C6 | A billed response contains assistant content. RouterPlus never passes usage counters without an answer. |
| C7 | The response reports the model that was asked for. Resolving an undated id to your own dated snapshot is fine. Declare any other equivalence in `resolves_to`. Substituting a different model without saying so disqualifies the listing. |

An **image** listing (`output_modality: "image"`) is non-stream only, so it runs C5 and these checks instead of C2 to C4. Each check buys one real render at the model's top declared quality.

| Check | Requirement |
|---|---|
| I1 | A generation returns decodable base64 and provider-reported usage tokens. |
| I2 | The tokens billed for that render fit the declared per-image ceiling (`max_output_tokens`). An under-declared ceiling loses RouterPlus money on every request. |
| I3 | An undeclared size returns a parseable 4xx JSON error. |
| I4 | Opt-in (`--image-ceiling`): I2 at every declared size and at `size: auto`. |

A **decisions** listing (`output_modality: "decisions"`) runs two checks:

- **D1**: one call with three questions comes back as typed answers, with usage.
  - On the `systemone` wire: one Choice, one Score and one Noul question, and `usage.input_tokens`.
  - On the `gliner` wire: one yes/no task and one three-way task in one schema (sent with `store: false`). The message content must parse to one JSON object with a labelled answer under each task name, and the usage must have `prompt_tokens`.
  - D1 calls each model at its own `decisions_path` when the model declares one, else at the endpoint's.
  - With `endpoint.response_envelope: "cloudflare_v4"`, the 200 must say `success: true`, and D1 reads the answer and the usage inside `result`.
  - With `endpoint.decisions_format: "openai"`, D1 sends the same three questions in OpenAI's form (a `choice`, a `score` with levels and a `predicate`). It checks each answer under its name in its own type, with `usage.input_tokens`.
  - When D1 gets another status, the report shows the first 300 characters of your body.
- **C5**: an unknown model gets a parseable 4xx error. FastAPI's `{"detail": ...}` envelope counts as a described error, and so does Cloudflare's `{"errors": [{"code": ..., "message": ...}]}` (or one such object in place of the list) when an entry has a message. With `endpoint.decisions_format: "openai"`, C5 sends the
  unknown model in OpenAI's form (OpenAI answers 404 `model_not_found`).

For every model, the report also lists any `Ratelimit`, `Ratelimit-Policy` or `Retry-After` header your endpoint sent, or says that it sent none. These headers are reported, never
judged.

## Wire requirements

- An HTTPS endpoint that speaks OpenAI Chat Completions or Anthropic Messages, with streaming.
- An OpenAI-dialect endpoint that cannot stream declares `streaming: false`. RouterPlus asks it for JSON and streams the finished answer, so the buyer's first byte waits for the full answer.
- Provider-reported usage tokens on both paths (C1, C4).
- Errors: return 429 early when you are at capacity. Queueing raises your published latency. Return 4xx JSON for bad requests.
- **Streaming:** send response headers within 20 seconds, or the attempt fails over. Send SSE keep-alive comments through long gaps before the first byte, and stream tokens as soon as they exist. Non-stream requests get a long generation budget.
- After the first output, RouterPlus never switches providers, so buyers see a stall as yours.

## How uptime is counted

Routing uses the same numbers:

```text
uptime = completed ÷ (completed + counted failures), trailing 24h
```

- **Counted against you:** provider 401, 402 and 403, all 5xx, streams that die mid-stream, and incomplete streams.
- **Never counted:** requests the buyer caused: 400 and 413 (bad input), 429 (the buyer's rate limit) and client cancellations mid-stream.
- **Published after 100 or more requests.** Below that, the pages show a dash.
- Failures open a per-deployment **health circuit**: a cooldown, then one half-open probe. Recovery closes it. While every deployment of a model cools down, RouterPlus still sends a last-resort attempt.
- Content-policy refusals are never retried and never failed over.

## Performance

RouterPlus publishes **TTFB p50/p95** (from merged t-digests) per model, and the router uses the same figures. Return early 429s instead of queueing, and start streaming at once.

## How you get paid

Pick a settlement mode in your manifest:

- **`prepaid`**: RouterPlus deposits with you ahead of traffic, and usage draws it down. RouterPlus watches balance against burn so routing never hits an empty account.
- **`arrears`**: RouterPlus routes first and pays on your interval: `half_daily`, `daily`, `weekly` or `monthly`.

Your statement comes from the append-only ledger that bills buyers: token counts and USD per period, auditable to each attempt. Each attempt also records byte counts, a tokenizer-independent bound for disputes. Statements and conformance reports come by email. There is no provider dashboard yet.

## What you get

- Buyer traffic from both wire surfaces on the day your models go live.
- A public page for each model: prices, context, your data-policy flag and live metrics.
- Settlement statements per period that reconcile to the token.
- Circuits and time to recover when your service degrades. RouterPlus does not delist a provider without notice.

## Apply

Email **providers@evo-hq.com** with your manifest attached. Validate it against [the schema](/docs/provider-manifest.schema.json) first. RouterPlus runs the suite against your endpoint, shares the report, stages your models and makes them live with you.

---

Source: https://demo.routerplus.com/docs/api-chat-completions.md

# POST /v1/chat/completions

The OpenAI-compatible chat endpoint. Point any OpenAI SDK or client at `https://demo-api.routerplus.com/v1` with your RouterPlus key, and call any chat model in the catalog, Claude models included. The gateway translates requests, streams and errors between provider dialects.

Image models answer on [POST /v1/images/generations](/docs/api-images) only. The Anthropic-native endpoint is [POST /v1/messages](/docs/api-messages).

```
POST https://demo-api.routerplus.com/v1/chat/completions
```

## Examples

```bash
curl -N https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-haiku-4-5","stream":true,"max_tokens":100,"messages":[{"role":"user","content":"Say hello."}]}'
```

Python:

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

resp = client.chat.completions.create(
    model="claude-sonnet-4-5",  # a Claude model over the OpenAI wire: the gateway translates
    max_tokens=200,
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(resp.choices[0].message.content)
print(resp.usage)  # prompt/completion tokens, cache splits, cost (USD)
```

TypeScript:

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://demo-api.routerplus.com/v1",
  apiKey: process.env.TM_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "gpt-4o-mini",
  stream: true,
  messages: [{ role: "user", content: "Count to five." }],
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  if (chunk.usage) console.log("\n", chunk.usage); // final chunk: tokens, cache splits, cost
}
```

The Python and TypeScript examples use the official `openai` SDK with only the base URL changed.

## Authentication

| Header | Format |
|---|---|
| `Authorization` | `Bearer tm_vk_...` (the standard OpenAI header) |
| `x-api-key` | `tm_vk_...` (also accepted, same key) |

A missing or invalid key returns 401 with `error_type: "auth"`. To identify your app, send `HTTP-Referer` and `X-Title`. They carry no content.

## Request

The body is JSON, at most 10 MB (413 `request_too_large` above that).

| Parameter | Type | Behavior |
|---|---|---|
| `model` | string, required | A catalog id. See [Model discovery](#model-discovery). An unknown id is a 404. The gateway never substitutes another model. Each Claude model also answers to one `rp-` name, for clients such as Cursor that do not send `claude-` names to another base URL: `rp-sonnet-5-5` is `claude-sonnet-5-5` (see [Models & catalog](/docs/models#claude-names-for-cursor)). This route also serves a deployed endpoint from [Optimize](/docs/model-search) (`tm/<name>-v<n>`), with a key of the organization that owns it, and a [dedicated endpoint](/docs/dedicated-endpoints) (`<your-namespace>/<name>`). |
| `messages` | array, required | Roles `system`, `developer`, `user`, `assistant`, `tool`. Content is a string or `text` parts. A model that takes them also accepts `image_url`, `file`, `input_audio` and `video_url` parts (see `architecture.input_modalities` in [`GET /v1/models`](/docs/api-models)). A part the model does not take is a typed 400 that names the part. See [Content parts](#content-parts). |
| `stream` | boolean | Server-sent events. See [Streamed response](#streamed-response). |
| `max_tokens` | number | Output cap, from 1 to the model's own maximum (`max_output_tokens` in the catalog). The maximum is 128,000 on the Claude 5, GPT-6 and GPT-5.6 models, and can be below 32,768 (GPT-4o: 16,384). **If you send neither `max_tokens` nor `max_completion_tokens`, the gateway sets the model's own maximum** on the forwarded request, on every provider. It lowers the value when needed so that your input and the output fit the context window. If the catalog declares no maximum, or does not list the model (on your own connection), the value is 32,768. `null` counts as not set. A value under 1 (`0`, `-1`) or above the maximum is a 400. |
| `max_completion_tokens` | number | An alias of `max_tokens`. Both with different values is a 400. |
| `temperature` | number | Passed through. A value above 1 cannot be translated to an Anthropic-dialect provider. See [Wire compatibility](/docs/compat). |
| `top_p` | number | Passed through. |
| `stop` | string \| string[] | Becomes Anthropic `stop_sequences` on cross-dialect routes. |
| `tools` | array | `type: "function"` tools work on every model. A tool without `parameters` gets an empty object schema on Anthropic-dialect routes. Custom (free-form) tools work on the GPT-6 and GPT-5.6 models (all seven call OpenAI's Responses API): `{"type":"custom","custom":{"name":"…","description":"…","format":{"type":"text"}}}`, or a `grammar` format in `lark` or `regex` syntax. The call comes back as a `tool_calls` entry `{id, type: "custom", custom: {name, input}}`. Claude models take function tools only: a custom tool to a Claude model is a 400 that names it. See [Wire compatibility](/docs/compat#gpt-6-and-gpt-56-on-the-chat-surface). |
| `tool_choice` | string \| object | `"auto"`, `"none"`, `"required"`, `{"type":"function","function":{"name":"..."}}`, or, on the models above, `{"type":"custom","custom":{"name":"..."}}`. |
| `parallel_tool_calls` | boolean | `false` becomes `disable_parallel_tool_use` on Anthropic-dialect routes. |
| `n` | number | **A value above 1 is a 400 `invalid_request`.** See [Why `n > 1` is rejected](#why-n-1-is-rejected). |
| `stream_options` | object | Usage is always on. An explicit `{"include_usage": false}` is honored: you are billed the same, but you get no usage chunk. Otherwise an OpenAI-dialect provider gets the object with `include_usage` set. On a translated route the gateway acts on it itself, and does not forward it or name it in `x-tm-dropped-params`. |
| `provider` | object | Routing controls: `require_parameters`, `order`, `only`, `ignore`, `allow_fallbacks` and the rest. The gateway reads them and never forwards them. An unknown control is a 400. See [Routing policies](/docs/routing-policies). |
| `session_id` | string | A session id. The gateway sends it to OpenAI and Azure as `prompt_cache_key`, and to Morph as the header `x-session-id`. The headers `x-session-id`, `x-session-affinity` and `x-claude-code-session-id` work too. See [Sessions and prompt caching](/docs/compat#sessions-and-prompt-caching). |
| `prompt_cache_key` | string | OpenAI's cache key, sent unchanged to OpenAI and Azure. |
| `prompt_cache_retention` | string | `"in_memory"` or `"24h"`. Sent unchanged to OpenAI and Azure, dropped and recorded elsewhere. |
| `safety_identifier` | string | Your end-user id. On a house route the gateway sends a hash of it with your organization. See [Sessions and prompt caching](/docs/compat#sessions-and-prompt-caching). |
| `cache_control` | object | Automatic prompt caching, as on the Anthropic API. Sent to Anthropic, dropped and recorded on OpenAI and Azure. A field that Anthropic would refuse is dropped and recorded on every route (see [Wire compatibility](/docs/compat)). A `cache_control` mark on a `text` part is kept for Anthropic and passes unchanged to OpenAI and Azure, which ignore it. |

A top-level `api_key`, `base_url`, `connection_id` or other credential or destination field is a 400. You set keys and endpoints in the console, never in a request.

### Everything else

The table above is the **portable** set: it works on any provider. Other parameters depend on the dialect of the provider that serves the request:

| Serving provider | What happens |
|---|---|
| OpenAI dialect | The OpenAI chat parameters (`response_format`, `reasoning_effort`, `seed`, `user`, `logprobs`, `top_logprobs`, `frequency_penalty`, `presence_penalty`, `logit_bias`, `metadata`, `store`, `prediction`) reach it as you sent them. |
| Translated to the Anthropic dialect | `response_format`, `prediction`, `audio` and `modalities` are a typed 400, because they change what the model produces. `reasoning_effort` becomes Claude's own effort (`output_config.effort`). A Claude model that takes no effort drops it and records the drop. Other unknown parameters are dropped, never changed into something else. |

- Any other top-level key is dropped and recorded on every route.
- The `x-tm-dropped-params` header names every drop.
- `"provider": {"require_parameters": true}` turns a drop into a 400.
- The full tables are on [Wire compatibility](/docs/compat).

### Content parts

The gateway never drops content, because the provider would still bill for it. A part the model does not take is a typed 400 that names the field (for example `messages[0].content[1]`), on every route, before anything is billed or sent. An image to a text-only model is one such case.

- On an OpenAI-dialect provider, a part that the model takes passes through as sent.
- On a route translated to the Anthropic dialect (Claude), the gateway translates an `image_url` part (a JPEG, PNG, GIF or WebP data URL, or an `http(s)` URL) and a `file` part with a PDF data URL.
- On those routes, audio parts, video parts, files given by id, other file types and unknown part types are a typed 400 that names the field.

### Why `n > 1` is rejected

The gateway returns one choice. It refuses `n` above 1 before dispatch, so you never pay for fewer choices than you asked for:

```json
{
  "error": {
    "code": "invalid_request",
    "message": "n>1 is not supported in v1; request a single choice",
    "type": "invalid_request_error",
    "metadata": { "error_type": "invalid_request" }
  },
  "request_id": "..."
}
```

## Non-streamed response

A standard `chat.completion` object.

- On a translated response, `id` is `chatcmpl-<request-id>`.
- On an OpenAI-dialect provider, the body passes through with the provider's own `id`, keeping OpenAI's fields and the extensions below only. Fields a provider adds of its own (its cost, its token accounting, its host's name) are dropped. The message passes as the model wrote it ([what an answer keeps](/docs/compat#what-an-answer-from-an-openai-dialect-provider-keeps)).
- `model` names the model you asked for, also when the provider knows it under another id.
- For the `/v1/generation?id=` audit, use the `x-request-id` response header. It carries the bare request id on every route.
- On a [closed dedicated endpoint](/docs/dedicated-endpoints#closed-endpoints), the body keeps a fixed set of fields on every route. `model` is the endpoint id, `id` is `chatcmpl-` and the request id without dashes, and provider fields such as `system_fingerprint` and `native_finish_reason` are dropped.

```json
{
  "id": "chatcmpl-1c9a7b2e-...",
  "object": "chat.completion",
  "created": 1757000000,
  "model": "claude-sonnet-4-5",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Hello!" },
      "finish_reason": "stop",
      "native_finish_reason": "end_turn"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 5,
    "total_tokens": 17,
    "prompt_tokens_details": { "cached_tokens": 0, "cache_write_tokens": 0 },
    "cost": 0.000111
  }
}
```

Usage fields (the billing contract, see [Wire compatibility](/docs/compat)):

| Field | Meaning |
|---|---|
| `prompt_tokens` | All billed input tokens, **cache included**: uncached input + cache reads + cache writes. |
| `prompt_tokens_details.cached_tokens` | Cache reads, billed at the cached rate. |
| `prompt_tokens_details.cache_write_tokens` | Cache writes, billed at their own rate. Always present on billed responses, 0 when none. |
| `completion_tokens` | Output tokens. Reasoning tokens are a subset, in `completion_tokens_details.reasoning_tokens` when the provider reports them (streams always carry the field). |
| `cost` | Settled cost in USD, on billed keys only. The gateway computes it with the same integer micro-USD math as the ledger. It covers the **full request** debit, including attempts that failed over before your answer started, so you can recompute your bill from the response. |

`finish_reason` is one of `stop`, `length`, `tool_calls`, `content_filter`. On a response translated from an Anthropic-dialect provider, `choices[0].native_finish_reason` keeps the provider's raw stop reason (for example `end_turn`, `stop_sequence`). When the provider returns the model's thinking, it is in `message.reasoning_content`.

## Streamed response

With `stream: true`, the gateway sends SSE frames in `chat.completion.chunk` shape, in this order:

1. A role delta (`{"role":"assistant","content":""}`) as soon as the provider answers.
2. Content deltas: `delta.content`, `delta.reasoning_content` (thinking models), and `delta.tool_calls` fragments addressed by `index`. The id and `function.name` arrive on each call's first fragment.
3. A finish chunk with `finish_reason` set, plus `native_finish_reason` on translated streams.
4. A usage chunk with empty `choices`. It is always sent, without `stream_options`:

```text
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1757000000,"model":"claude-haiku-4-5","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":5,"total_tokens":17,"prompt_tokens_details":{"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"reasoning_tokens":0},"cost":0.000037}}

data: [DONE]
```

5. `data: [DONE]`.

- These are the shapes of a translated stream.
- On an OpenAI-dialect provider, the gateway relays the stream record for record and adds `usage.cost` to its usage chunk. Each chunk keeps OpenAI's fields only and names the model you asked for.
- On a closed dedicated endpoint, each chunk keeps only the fixed set of fields, and the provider's own SSE comment lines are dropped.
- When a started stream is silent for 15 s or more (slow reasoning models), the gateway sends an SSE comment (`: processing`) so that proxies keep the stream open. SDKs ignore it.
- See [Streaming](/docs/streaming) for the full detail.

> [!WARNING]
> If the serving provider fails **after** your answer started, the stream ends with one error chunk. It is a full chunk envelope with `finish_reason: "error"` and a top-level `error` object with `metadata.error_type`. **No `[DONE]` follows an error.** The gateway never restarts an answer on another provider mid-stream. Failover happens only before the first byte of output, and you do not see it. The same chunk ends a stream that reaches the request deadline (450 s from dispatch, or a routing policy's `timeout_ms`).

## Errors

Every error uses the OpenAI envelope with a stable `error.metadata.error_type` (`auth`, `rate_limit`, `insufficient_quota`, `model_unavailable`, `invalid_request`, `context_overflow`, `content_policy`, `upstream_error`, `gateway_error`, ...). [Errors](/docs/errors) has the full table with fixes.

- The gateway never relays a provider's raw error body. When a provider rejected the request, the class says so. The message has the provider's own words (up to 1,000 characters, without keys, account ids and host names), and the `x-tm-upstream-status` header has its status.
- A prompt too long for the model is OpenAI's standard 400. `error.code` is `context_length_exceeded`, with OpenAI's wording ("This model's maximum context length is M tokens. However, your messages resulted in N tokens. …"), so coding agents compact and retry.
- Out of credits is the OpenAI-standard **429 `insufficient_quota`**, which your SDK recognizes. It covers an empty balance or a monthly spend cap. On a cap breach, the message and the `x-tm-cap-reset` header give the UTC reset time.
- First trial keys, shown after verified sign-in, have a 20 RPM limit. Until an organization buys credit, all its keys together run at 20 RPM. The 429 `rate_limit` carries `retry-after`.

> [!TIP]
> A billed request reserves an estimated cost before dispatch: one token per byte of input, plus 65,536 per image part, and all of `max_tokens` (at most 32,768 tokens when you omit it). If your balance cannot cover it, with your organization's other in-flight reservations, you get `insufficient_quota` before any provider is called. You pay the observed usage only, so a large `max_tokens` never costs extra. The gateway holds it in full, though, so on a small balance set `max_tokens` to what you need, or leave it out. See [Reserve, then settle](/docs/pricing#reserve-then-settle).

## Response headers

| Header | Meaning |
|---|---|
| `x-request-id` | Gateway request id. Joins `/v1/usage` and `/v1/generation?id=`. |
| `x-tm-provider` | Deployment that served the request. `routerplus` on a closed dedicated endpoint. |
| `x-tm-attempts` | Dispatches made, failovers included. |
| `x-tm-upstream-status` | The provider's own HTTP status. |
| `x-tm-error-code` | Canonical error class, on failures. `x-tm-error-origin` says where the failure came from, and `x-tm-limit-scope` / `x-tm-limit-kind` name a limit that refused the request. See [Errors](/docs/errors). |
| `x-tm-upstream-model` | The id the gateway sent, when the provider knows the model under its own id. The response's `model` field still names the model you asked for. Absent on a closed dedicated endpoint. |
| `x-tm-served-by` | The provider that ran the request, when the route reports one. Absent on a closed dedicated endpoint. |
| `x-tm-dedicated-endpoint`, `x-tm-route-role` | On a [dedicated endpoint](/docs/dedicated-endpoints): its id, and `primary` or `fallback`. |
| `x-tm-queue-ms` | On a dedicated endpoint with a [paced limit](/docs/dedicated-endpoints#the-paced-limit): how long the request waited for its turn, in milliseconds. |
| `x-tm-auto-cache` | On a Claude route: `applied` when the gateway turned on Anthropic's automatic caching, else `skipped:` and the reason. Send the request header `x-tm-auto-cache: off` to turn it off. Absent on a closed dedicated endpoint. See [Automatic caching on Claude](/docs/compat#automatic-caching-on-claude). |
| `x-tm-dropped-params` | Comma-joined field paths of every parameter the gateway removed for this route. Absent when nothing was dropped. Send `"provider": {"require_parameters": true}` to get a typed 400 instead. |
| `x-tm-route-plan-id` | On billed requests: the id of the route plan that chose the deployments. |
| `x-tm-admission-mode`, `x-tm-remaining-rpm`, `x-tm-remaining-tpm` | On billed requests: how the request was admitted, and the smallest allowance left across the limits it claimed. See [Limits and capacity](/docs/admission). |

## Model discovery

`GET https://demo-api.routerplus.com/v1/models` (authenticated) returns the catalog in the OpenAI list shape, one row per model id, plus your organization's dedicated endpoints. The public directory with prices and context lengths is `https://demo.routerplus.com/api/models.json`.

- A model id that is not listed returns 404 `model_unavailable` with the id you asked for.
- If every deployment of a listed model is cooling down after failures, you get a 503 `gateway_error` with `retry-after: 5` and `x-tm-limit-kind: health`. Retry the same request.

See also: [POST /v1/messages](/docs/api-messages) · [Errors](/docs/errors) · [Wire compatibility & billing contract](/docs/compat) · [Quickstart](/docs/quickstart)

---

Source: https://demo.routerplus.com/docs/api-responses.md

# POST /v1/responses

OpenAI's Responses API. Point an OpenAI SDK at `https://demo-api.routerplus.com/v1` and call `client.responses.create` with your RouterPlus key. Codex (the CLI, the desktop app and the IDE extension), the OpenAI Agents SDK and other Responses API tools call this route.

OpenAI's models are a passthrough. Claude and the other chat models are a **preview**: the gateway translates them through its chat surface. The chat endpoints are [POST /v1/chat/completions](/docs/api-chat-completions) and [POST /v1/messages](/docs/api-messages).

```
POST https://demo-api.routerplus.com/v1/responses
POST https://demo-api.routerplus.com/v1/responses/compact
```

## Examples

```bash
curl -N https://demo-api.routerplus.com/v1/responses \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"gpt-6-luna","stream":true,"store":false,"max_output_tokens":200,"input":"Say hello in one sentence."}'
```

Python:

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

resp = client.responses.create(
    model="gpt-6-luna",
    store=False,
    max_output_tokens=200,
    input="Say hello in one sentence.",
)
print(resp.output_text)
print(resp.usage)  # input, cached, output and reasoning tokens, and cost (USD)
```

TypeScript:

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://demo-api.routerplus.com/v1",
  apiKey: process.env.TM_API_KEY,
});

const stream = await client.responses.create({
  model: "gpt-5.6-terra",
  store: false,
  stream: true,
  input: "Count to five.",
});

for await (const event of stream) {
  if (event.type === "response.output_text.delta") process.stdout.write(event.delta);
  if (event.type === "response.completed") console.log("\n", event.response.usage);
}
```

The examples use the official `openai` SDK with only the base URL changed. To set up Codex, see [Coding agents](/docs/install#codex).

## Which models

| Models | Status | How the gateway serves them |
|---|---|---|
| `gpt-6-astra`, `gpt-6.1-sol`, `gpt-6-sol`, `gpt-6-luna`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-4o`, `gpt-4o-mini`, `gpt-4.1-mini` | Supported | Passthrough to OpenAI's own Responses API, on the marketplace route or on your own OpenAI connection. The body goes as you sent it, except for the fields in [Request](#request). The answer and its stream events come back as OpenAI sent them, with `usage.cost` added. |
| Claude models (`claude-sonnet-5-5`, `claude-opus-5-5`, …, and their `rp-` names) and the other chat models | Preview | Translated to a chat request and served with every chat rule. The answer is translated back. See [Preview: Claude and other models](#preview-claude-and-other-models). |

- A deployed model from [Optimize](/docs/model-search) (`tm/<name>-v<n>`) and a [dedicated endpoint](/docs/dedicated-endpoints) are not served here. They are a 400 with code `model_not_supported_on_responses`, whose message points to `/v1/chat/completions`.
- Image models answer on [POST /v1/images/generations](/docs/api-images) only, and decision models on [POST /v1/decisions](/docs/api-decisions) only.
- An id that is not in the catalog is a 404. The gateway never substitutes another model.

## Authentication

| Header | Format |
|---|---|
| `Authorization` | `Bearer tm_vk_...` (what the OpenAI SDK and Codex send) |
| `x-api-key` | `tm_vk_...` (also accepted, same key) |

A missing or invalid key returns 401 with `error_type: "auth"`. None of your request headers reach the provider. The gateway sends its own key, or your connection's.

## Request

The body is OpenAI's Responses API body, at most 10 MB. The gateway changes these fields:

| Field | Handling |
|---|---|
| `model` | A catalog id. The provider gets its own id for the model. |
| `store` | Always `false`. The gateway keeps no responses, and OpenAI does not keep the response for later retrieval. OpenAI's own retention policy still applies (see [Data policy](/docs/data-policy)). A `store: true` is dropped and named in `x-tm-dropped-params`. Leave `store` out or send `false`. |
| `previous_response_id`, `conversation` | Not supported: a 400 `invalid_request`. Send the whole conversation in `input` on every turn. To keep the model's reasoning between turns, ask for `include: ["reasoning.encrypted_content"]` and send the reasoning items back (Codex does both). |
| `background` | `true` is a 400 `invalid_request`. Stream long answers instead. |
| `max_output_tokens` | Optional. A value you send must be a whole number from 1 to the model's own maximum, or to 32,768 if the model's maximum is lower. OpenAI's Responses API takes a limit above GPT-4o's 16,384, and the model stops at its own maximum. When you send none, the gateway adds none: OpenAI uses the model's own maximum, and the balance hold counts at most 32,768 output tokens of it. A limit that you send is held in full, up to the model's own maximum. |
| `prompt_cache_key` | Your value, else your session id (`session_id`, or the `x-session-id`, `x-session-affinity` or `x-claude-code-session-id` header). |
| `safety_identifier`, `user` | On the marketplace route, one hash of your organization and your end-user id goes as `safety_identifier`, never your raw value (see [Sessions and prompt caching](/docs/compat#sessions-and-prompt-caching)). |
| `service_tier` | On the marketplace route, a tier other than `auto` or `default` is dropped and named, because the catalog prices the default tier. |
| `temperature`, `top_p` | Dropped and named on GPT-6 and GPT-5.6, which take only their default sampling (`1` passes). |
| `tools` | Every type passes: function, custom, namespace, and OpenAI's hosted `web_search`. On the marketplace route, the hosted tools `code_interpreter`, `image_generation`, `file_search` and `mcp`, and a stored `prompt` template, are a 400 that names them. They are billed outside the token price, or they read data kept in an OpenAI account. Your own OpenAI connection takes them all. |
| `provider` | The routing controls of [Routing policies](/docs/routing-policies). The gateway reads them and never forwards them. |

Every other field passes to OpenAI as you sent it: input items of every type (reasoning items with encrypted content, custom tool calls), `instructions`, `tool_choice`, `parallel_tool_calls`, `reasoning`, `text`, `include`, `metadata`, `max_tool_calls` and `truncation`.

**Web search** is OpenAI's hosted `web_search` tool, billed at OpenAI's price: $10 per 1,000 calls, plus the tokens of the search results. The older `web_search_preview` tool costs $25 per 1,000 calls on GPT-4o, GPT-4o mini and GPT-4.1 mini. The fee is part of `usage.cost`. See [Pricing & billing](/docs/pricing#web-search).

`POST /v1/responses/compact` passes to OpenAI the same way and is billed from its usage. It never streams. It takes only the fields OpenAI's compact endpoint takes, and the gateway drops and names the others.

`GET /v1/responses/{id}` and the other stored-response endpoints are a 404. The gateway does not store responses.

## Response

On the supported models, the response is OpenAI's `response` object as OpenAI sent it, plus `usage.cost`: the exact charge in USD (see [Pricing & billing](/docs/pricing)).

- Usage is OpenAI's: `input_tokens` (cache reads and cache writes included), `input_tokens_details.cached_tokens`, `input_tokens_details.cache_write_tokens`, `output_tokens` and `output_tokens_details.reasoning_tokens`. They are billed with the same math as every other route.
- The `x-request-id` response header gives the id for `GET https://demo-api.routerplus.com/v1/generation?id=<x-request-id>`.
- Every call appears in `GET https://demo-api.routerplus.com/v1/usage`.

With `stream: true`, the events are OpenAI's server-sent events, record for record, in OpenAI's order: `response.created`, `response.in_progress`, `response.output_item.added`, `response.output_text.delta`, …, and `response.completed` (or `response.incomplete`). Its `response.usage` has the billed usage and `cost`. This API sends no `[DONE]`.

- The first events wait until the provider sends output. If a provider fails before that, the next route replaces it and you do not see it.
- If the provider fails after output started, the stream ends with the gateway's own `response.failed` event (code `server_error`, the canonical class in `metadata.error_type`), and nothing follows it. Codex retries it.
- If you close the stream early, you pay for the input and the output streamed to you so far, as on the chat surfaces (see [Pricing & billing](/docs/pricing#reserve-then-settle)).

The other response headers are the same as on [POST /v1/chat/completions](/docs/api-chat-completions#response-headers): `x-request-id`, `x-tm-provider`, `x-tm-attempts`, `x-tm-dropped-params` and the rest.

## Preview: Claude and other models

For a model that OpenAI does not serve (Claude, by its id or its `rp-` name, and the other chat models), the gateway translates the request into a chat request. It serves it with the chat surface's rules: routing and failover, the output default and its hold, automatic caching on Claude, billing and the ledger. The answer comes back as a `response` object or as Responses stream events. Codex works this way with Claude models.

| Responses input | Chat form |
|---|---|
| `instructions` and `developer` messages | System messages |
| Message items with `input_text`, `output_text`, `input_image`, `input_file` and `input_audio` parts | Chat messages with `text`, `image_url`, `file` and `input_audio` parts |
| `function_call` and `function_call_output` items | An assistant tool call and its tool message. A tool output with an image (Codex's `view_image`) goes into a user message right after the tool messages |
| `custom` tools, `custom_tool_call` and its output | A function tool with one string parameter, `input`. The call comes back as a `custom_tool_call` |
| `namespace` tools | Function tools named `<namespace>__<name>` (with a short hash when that name is taken or too long). The call comes back under its own namespace and name |
| `max_output_tokens`, `reasoning.effort`, `text.format` | `max_tokens`, `reasoning_effort`, `response_format` |
| `tool_choice`, `parallel_tool_calls`, `prompt_cache_key`, `temperature`, `top_p` | Their chat forms, or as sent |
| Hosted tools (`web_search`, `file_search`, `code_interpreter`, `image_generation`, `computer_use_preview`, `mcp`, `shell`, `apply_patch` and others), and `reasoning` items you send back | Dropped and named in `x-tm-dropped-params`, for example `tools[3].web_search` and `input[*].reasoning`. Web search is not mapped to Claude's own search |

- **Also dropped and named:** `reasoning.summary` (Codex asks for one on every request), a `text.verbosity` other than `medium`, `store: true`, and a `truncation`, `max_tool_calls`, `service_tier` or `top_logprobs` value that asks for something.
- **Refused with a 400 that names the field:** `previous_response_id`, `conversation`, `prompt`, `background: true`, `item_reference` items, an uploaded `file_id`, a `file_url`, and a part or item type that a chat body cannot carry.
- With `"provider": {"require_parameters": true}`, a drop is a 400 too.

The answer:

- Its `model` is the name you sent. Its `id` is `resp_` and the request id. The `x-request-id` header has the request id for `GET /v1/generation`.
- Text comes back as a `message` item, a tool call as a `function_call` or `custom_tool_call` item, and the model's reasoning as a `reasoning` item with its text.
- `status` is `incomplete` when the answer reached `max_output_tokens` or was filtered, and `completed` otherwise.
- Usage: `input_tokens` counts cache reads and writes, as on the chat surface, and `cost` is inside it.

Limits of the preview:

- Claude routes refuse `text.format` with a 400 that names it. `reasoning.effort` reaches Claude as its own effort (`output_config.effort`, see [Wire compatibility](/docs/compat)), so Codex's `model_reasoning_effort` works on Claude.
- The Claude 5 models refuse a forced tool choice (`tool_choice: "required"`, or a named tool) with a 400. Codex sends `auto`.
- A reasoning item that you send back is dropped, so the model's reasoning does not carry from turn to turn. The exception is Morph's and Mistral's routes: a reasoning item with the model's reasoning text, as this surface returns it, goes back to them as that turn's reasoning.
- Annotations and logprobs are not carried. A custom tool's input arrives whole when its call ends.
- `POST /v1/responses/compact` takes OpenAI's models only. For any other model it is a 400 with code `model_not_supported_on_responses`.

For production traffic to Claude, use `/v1/chat/completions` or `/v1/messages`.

## Errors

Every error uses OpenAI's error envelope, with the canonical class in `error.metadata.error_type`, as on [POST /v1/chat/completions](/docs/api-chat-completions#errors). A provider's refusal has the provider's own words in the message.

- On OpenAI's models, `error.code` is the canonical class, as on the chat surface. A prompt too long for the model has `code: "context_length_exceeded"`.
- On the preview path, `error.code` uses OpenAI's own names where clients act on them: `rate_limit_exceeded` (class `rate_limit`), `context_length_exceeded` (`context_overflow`), `model_not_found` (`model_unavailable`) and `invalid_prompt` (`content_policy`).
- On the preview path, a failure in the middle of a stream is a `response.failed` event. Any other code there is `server_error`, which Codex retries.
- A 400 for a field that the translation wrote names your field: `reasoning.effort`, `text.format` or `max_output_tokens`.

Codex compacts a conversation when it sees `context_length_exceeded`. The full table is on [Errors](/docs/errors).

See also: [POST /v1/chat/completions](/docs/api-chat-completions) · [Errors](/docs/errors) · [Wire compatibility](/docs/compat) · [Coding agents](/docs/install)

---

Source: https://demo.routerplus.com/docs/api-messages.md

# POST /v1/messages

The Anthropic-compatible endpoint. Point the Anthropic SDK or Claude Code at `https://demo-api.routerplus.com` with your RouterPlus key, and call any chat model in the catalog, GPT models included. The gateway translates requests, streams and errors between provider dialects.

Image models answer on [POST /v1/images/generations](/docs/api-images) only. The OpenAI-native endpoint is [POST /v1/chat/completions](/docs/api-chat-completions).

```
POST https://demo-api.routerplus.com/v1/messages
```

The gateway accepts a query string. Claude Code sends `/v1/messages?beta=true`, which routes normally.

## Examples

A GPT model over the Anthropic wire:

```bash
curl -N https://demo-api.routerplus.com/v1/messages \
  -H "x-api-key: $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"gpt-4o-mini","stream":true,"max_tokens":100,"messages":[{"role":"user","content":"Say hello."}]}'
```

Python:

```python
import os
from anthropic import Anthropic

client = Anthropic(base_url="https://demo-api.routerplus.com", api_key=os.environ["TM_API_KEY"])

with client.messages.stream(
    model="claude-sonnet-4-5",
    max_tokens=300,
    messages=[{"role": "user", "content": "Count to five."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="")
    print("\n", stream.get_final_message().usage)  # tokens, cache splits, cost
```

TypeScript:

```typescript
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  baseURL: "https://demo-api.routerplus.com",
  apiKey: process.env.TM_API_KEY,
});

const stream = client.messages.stream({
  model: "claude-haiku-4-5",
  max_tokens: 200,
  messages: [{ role: "user", content: "Say hello." }],
});
stream.on("text", (t) => process.stdout.write(t));
console.log("\n", (await stream.finalMessage()).usage);
```

The examples use the official `anthropic` and `@anthropic-ai/sdk` SDKs with only the base URL changed. The SDK appends `/v1/messages` itself.

Claude Code works unmodified:

```bash
ANTHROPIC_BASE_URL=https://demo-api.routerplus.com ANTHROPIC_AUTH_TOKEN=$TM_API_KEY claude
```

## Headers

| Header | Required | Behavior |
|---|---|---|
| `x-api-key` | one of these two | Your RouterPlus key, `tm_vk_...`. The Anthropic-native header. |
| `Authorization` | | `Bearer tm_vk_...`, also accepted, same key. |
| `content-type` | yes | `application/json`. |
| `anthropic-version` | no | Accepted for SDK compatibility. The gateway sends `anthropic-version: 2023-06-01` to Anthropic providers, whatever you send. |
| `x-tm-auto-cache` | no | `off` turns off the automatic prompt caching that the gateway may add for an agent's conversation on Claude (see [Automatic caching on Claude](/docs/compat#automatic-caching-on-claude)). The response header of the same name says whether it was applied. |
| `anthropic-beta` | no | See below. No effect on other routes. |

On an Anthropic-dialect provider, the gateway forwards the `anthropic-beta` values that do not change how a token is billed:

- On every Anthropic-dialect provider: the `prompt-caching`, `token-efficient-tools`, `fine-grained-tool-streaming`, `interleaved-thinking`, `claude-code`, `oauth` and `computer-use` prefixes.
- On Anthropic's own API also: `per-turn-control`, `mid-conversation-output-config`, `context-management`, `mid-conversation-tool-changes` and `inline-tools` (what Claude Code needs on the Claude 5 models).
- Any other value is dropped and recorded in `x-tm-dropped-params` as `header:anthropic-beta:<value>`, for example `advisor-tool` (a second model that bills at its own rates).

A missing or invalid key returns 401 in the Anthropic error shape with `error_type: "auth"`. To identify your app, send `HTTP-Referer` and `X-Title`. They carry no content.

## Request

The body is JSON, at most 10 MB (413 `request_too_large` above that).

| Parameter | Type | Behavior |
|---|---|---|
| `model` | string, required | A catalog id: any chat model, Claude or not. The `rp-` names of the Claude models (`rp-sonnet-5-5`) work here too. An unknown id is a 404. An image model is a 400 that names `/v1/images/generations`. A deployed endpoint id (`tm/<name>-v<n>`) is served on `POST /v1/chat/completions` only, and is a 400 here. |
| `max_tokens` | number | Output cap, from 1 to the model's own maximum (`max_output_tokens` in the catalog). The maximum is 128,000 on the Claude 5 models, and can be below 32,768 (GPT-4o: 16,384). If the catalog declares no maximum, the value is 32,768. **When you omit it, the gateway sets the model's own maximum** on the forwarded request, on every provider. It lowers the value when needed so that your input and the output fit the context window. `null` counts as not set. A value under 1 or above the maximum is a 400. `max_tokens: 0`, which on Anthropic's API only warms the cache, is not served. |
| `messages` | array, required | Roles `user`, `assistant` and `system`. Content is a string or blocks: `text`, `tool_use` (assistant), `tool_result` (user), `thinking` / `redacted_thinking` (assistant). `image` and `document` blocks work on a model that takes them (see `architecture.input_modalities` in [`GET /v1/models`](/docs/api-models)). A block the model does not take is a typed 400 that names it, also inside a `tool_result`. On Anthropic-dialect routes the messages pass through as sent. See [Messages on an OpenAI-dialect route](#messages-on-an-openai-dialect-route). |
| `system` | string \| array | A string or text blocks. |
| `stream` | boolean | SSE streaming. See [Streamed response](#streamed-response). |
| `temperature`, `top_p` | number | Passed through. |
| `stop_sequences` | string[] | Becomes OpenAI `stop` on cross-dialect routes. |
| `tools` | array | `{name, description?, input_schema}`. Anthropic server tools (computer use, web search, bash, text editor) pass through on Anthropic-dialect routes, and are a typed 400 when an OpenAI-dialect provider would serve the request. Web search is billed at Anthropic's price per search, inside `usage.cost`. See [Pricing & billing](/docs/pricing#web-search). An empty list goes to an OpenAI-dialect provider as no list. |
| `tool_choice` | object | `{"type":"auto"}`, `{"type":"any"}`, `{"type":"none"}`, or `{"type":"tool","name":"..."}`. `disable_parallel_tool_use: true` becomes `parallel_tool_calls: false` on OpenAI-dialect routes. |
| `thinking` | object | Passes through on Anthropic-dialect routes. For OpenAI-dialect routes, see [Thinking on an OpenAI-dialect route](#thinking-on-an-openai-dialect-route). |
| `output_config` | object | Passes through on Anthropic-dialect routes. On an OpenAI-dialect route, `effort` becomes `reasoning_effort` (the nearest level the provider declares). `format` with `type: "json_schema"` becomes a strict json_schema `response_format` with the whole schema, and only providers that take json_schema serve it. When none does, the request is a typed 400 that names `output_config.format`. Another `format` type is a typed 400. |
| `context_management` | object | Context editing. Passes to Anthropic's own API with the `context-management` beta. Dropped and recorded on Bedrock and on OpenAI-dialect routes. |
| `top_k`, `metadata` | | Pass through on Anthropic-dialect routes. Dropped and recorded on OpenAI-dialect routes. On a house route, `metadata.user_id` becomes a hash with your organization. See [Sessions and prompt caching](/docs/compat#sessions-and-prompt-caching). |
| `cache_control` | object | Automatic prompt caching, on the request or on blocks. Passes to Anthropic. On Bedrock, the request-level field becomes a mark on the last block. Dropped and recorded on OpenAI and Azure. A request-level field that Anthropic would refuse is dropped and recorded on every route (see [Wire compatibility](/docs/compat)). On a house route, `ttl: "1h"` is removed (so the TTL is 5 minutes) and recorded. When you send no `cache_control`, the gateway may add the request-level field for an agent's conversation. See [Automatic caching on Claude](/docs/compat#automatic-caching-on-claude). |
| `session_id` | string | A session id, also accepted as the header `x-session-id`, `x-session-affinity` or `x-claude-code-session-id`. It reaches OpenAI or Azure as `prompt_cache_key`, and Morph as the header `x-session-id`. Anthropic has no session input. See [Sessions and prompt caching](/docs/compat#sessions-and-prompt-caching). |
| `mcp_servers`, `container` | | A typed 400 on every route, because removing them would change who runs what. |
| `provider` | object | Routing controls: `require_parameters`, `order`, `only`, `ignore`, `allow_fallbacks` and the rest. The gateway reads them and never forwards them. An unknown control is a 400. See [Routing policies](/docs/routing-policies). |

- Any other top-level key is dropped and recorded on every route, and the `x-tm-dropped-params` header names it. The gateway never changes it into something else.
- A top-level `api_key`, `base_url` or `connection_id` is a 400.
- The gateway never drops **content**, because the provider would still bill for it. A block the model does not take, or one a translation cannot carry, is a typed 400 that names the field.
- The gateway decides a translation 400 per deployment. When another deployment of the model speaks the Anthropic dialect, the request goes there instead. See [Wire compatibility](/docs/compat).

### Messages on an OpenAI-dialect route

- A `system` message inside the conversation applies from its place onward, as on Anthropic's API. On an OpenAI-dialect provider it goes as a system message at the same place, with its `output_config.effort` as `reasoning_effort`.
- A turn-scoped system message (`clear_at: "next_user_message"`) is left out once a user message follows it. A tool added or removed inside one that still shows is a typed 400.
- `image` blocks (base64 or URL) and base64 PDF `document` blocks in user messages are translated.
- A document given by URL or as text, an image given by file id, and an image in an assistant message are a typed 400.

### Thinking on an OpenAI-dialect route

- `adaptive` and `enabled` leave the model's own reasoning on. `budget_tokens` has no counterpart, and is dropped and recorded.
- `disabled` becomes `reasoning_effort: "none"`.
- `display` sets how the reasoning comes back. With `"omitted"` or `"updates"`, it comes back as `thinking` blocks with an empty `thinking` field, and the text travels in the block's `signature` for the next turn. Otherwise it comes back as readable `thinking` blocks.
- A route that serves the model with thinking off, or a model that does not reason (Mistral's Codestral, Large 3 and Ministral 3), drops and records `thinking`.
- See [Wire compatibility](/docs/compat#reasoning-and-structured-output-on-openai-dialect-routes).

> [!NOTE]
> Transcripts with `thinking` blocks round-trip safely. When a model served over the OpenAI dialect returns reasoning, this endpoint encodes it as `thinking` blocks: in a stream, and as a leading block in a reply that is not streamed. When you send that transcript back to an OpenAI-dialect provider, the gateway drops assistant `thinking`/`redacted_thinking` blocks instead of rejecting them. The gateway's own blocks (unsigned, or with the gateway's signature when `thinking.display` hid their text) are not named in `x-tm-dropped-params`, and Morph's and Mistral's routes read them back as the model's reasoning. The gateway also leaves them out of the history an Anthropic-dialect route gets, because Anthropic verifies every thinking signature. A conversation that moved to another provider while Anthropic was unavailable therefore continues on Anthropic. The route that produced your transcript never returns a 400 on it.

## Streamed response

With `stream: true`, you get the Anthropic Messages event sequence, in the same encoding on every provider. On a translated stream the message `id` is `msg_<request-id>`, which joins with the `x-request-id` header and `/v1/generation?id=`.

```text
event: message_start
data: {"type":"message_start","message":{"id":"msg_1c9a7b2e-...","type":"message","role":"assistant","model":"claude-sonnet-4-5","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":0,"output_tokens":0}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"input_tokens":12,"cache_read_input_tokens":0,"cache_creation_input_tokens":0,"output_tokens":5,"cost":0.000111}}

event: message_stop
data: {"type":"message_stop"}
```

- Block types: `text` (`text_delta`), `thinking` (`thinking_delta`, plus `signature_delta` for multi-turn thinking with tools), `redacted_thinking`, and `tool_use` (`input_json_delta` fragments).
- Streamed `stop_reason` is one of `end_turn`, `max_tokens`, `tool_use`, `refusal`.
- Usage fields follow the [billing contract](/docs/compat). `input_tokens` **excludes** cache reads and writes, which are in `cache_read_input_tokens` and `cache_creation_input_tokens`. `usage.cost` (USD, billed keys only) in the final `message_delta` is the full request debit, failed-over attempts included, computed with the same math as the ledger.

> [!NOTE]
> Read usage from the final `message_delta`. On same-dialect routes (a Claude model on this endpoint), the gateway relays the stream record for record, so `message_start` has the provider's real token counts and its own message id. On cross-dialect routes (an OpenAI-dialect provider behind this endpoint), the gateway re-encodes the stream, and the counts in `message_start` are zeros. The final `message_delta` has the real numbers on every route.

- When a started stream is silent for 15 s or more, the gateway sends native `ping` events (`{"type":"ping"}`) so that proxies keep the stream open.
- If the serving provider fails **after** output started, you get one `error` event inside the 200 stream, and nothing after it. It has the Anthropic-native `type` plus a stable `error_type`. The gateway never restarts the answer on another provider.
- The same event ends a stream that reaches the request deadline (450 s from dispatch, or a routing policy's `timeout_ms`).
- The Anthropic wire has no place for usage outside `message_delta`. For a billed stream that fails mid-answer, get the request's cost from `/v1/generation?id=`.

## Non-streamed response

```json
{
  "id": "msg_1c9a7b2e-...",
  "type": "message",
  "role": "assistant",
  "model": "gpt-4o-mini",
  "content": [{ "type": "text", "text": "Hello!" }],
  "stop_reason": "end_turn",
  "stop_sequence": null,
  "usage": {
    "input_tokens": 12,
    "output_tokens": 5,
    "cache_read_input_tokens": 0,
    "cache_creation_input_tokens": 0,
    "cost": 0.000004
  }
}
```

On a translated response, `stop_reason` is `end_turn`, `max_tokens`, `tool_use`, or `refusal`. On an Anthropic-dialect provider, the body passes through with its native values (including `stop_sequence`), plus the added `cost`.

## Errors

Errors use the Anthropic error shape, with the native `type` string your SDK reads (`authentication_error`, `rate_limit_error`, `billing_error`, `not_found_error`, ...), plus a stable `error_type` with the canonical class:

```json
{
  "type": "error",
  "error": {
    "type": "not_found_error",
    "message": "model \"claude-9\" is not in the catalog; GET /v1/models lists what this key can serve",
    "error_type": "model_unavailable"
  },
  "request_id": "..."
}
```

| Case | Status and type |
|---|---|
| Out of credits, or a spend-cap breach | 429, native type `billing_error`, `error_type: "insufficient_quota"`. On a cap breach, the message and the `x-tm-cap-reset` header give the UTC reset time. |
| A key whose signup email is not verified yet | 403, native type `permission_error`, `error_type: "email_not_verified"`. |
| An organization that has not added credit | Its keys and the organization as a whole run at 20 RPM. Above that: 429 `rate_limit` with `retry-after`. |
| A prompt too long for the model | Anthropic's standard 400 `invalid_request_error`, `prompt is too long: N tokens > M maximum`, with `error_type: "context_overflow"`. |

The gateway never relays a provider's raw error body. When a provider rejected the request, the class says so. The message has the provider's own words (up to 1,000 characters, without keys, account ids and host names), and `x-tm-upstream-status` has its status. Full table: [Errors](/docs/errors).

Headers:

- Every response carries `x-request-id`.
- Served requests add `x-tm-provider` and `x-tm-attempts`.
- Failures add `x-tm-error-code`, and `x-tm-error-origin` when a limit, a provider or RouterPlus's infrastructure refused the request.
- `x-tm-upstream-status` appears whenever a provider responded.
- The other headers are the same as on [POST /v1/chat/completions](/docs/api-chat-completions#response-headers).

## Model discovery

`GET https://demo-api.routerplus.com/v1/models` returns the catalog. Send an `anthropic-version` header (the Anthropic SDK does) to get the Anthropic list shape instead of the OpenAI one. That shape lists chat models only. The public directory with prices and context lengths is `https://demo.routerplus.com/api/models.json`.

See also: [POST /v1/chat/completions](/docs/api-chat-completions) · [Errors](/docs/errors) · [Wire compatibility & billing contract](/docs/compat) · [Migrate in one prompt](/docs/migration)

## POST /v1/messages/count_tokens

An unbilled passthrough to Anthropic-dialect deployments. It takes the same request shape as `/v1/messages`, and the provider's own tokenizer answers:

```bash
curl -s https://demo-api.routerplus.com/v1/messages/count_tokens \
  -H "x-api-key: $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model": "claude-sonnet-5", "messages": [{"role": "user", "content": "hello"}]}'
```

```json
{ "input_tokens": 8 }
```

- The answer is the count only.
- The call creates no ledger attempt and costs nothing. It counts against your key's rate limit and the provider's pool like any request.
- An unknown model is a 404.
- If only OpenAI-dialect deployments (or a Bedrock connection) serve the model, the response is a typed 400. The gateway does not estimate a count.

---

Source: https://demo.routerplus.com/docs/api-images.md

# POST /v1/images/generations

Generate images from a text prompt. The route follows the OpenAI Images API, so
`client.images.generate` works unchanged with the base URL `https://demo-api.routerplus.com/v1`.
`https://demo-api.routerplus.com/images/generations` is an alias for SDKs that omit the `/v1` prefix.

To try it without code, pick an image model in the [playground](/docs/playground).

- The route returns one JSON response by default. Existing clients need no change.
- On the development API, `stream: true` is optional. It returns keep-alives and then one
  final image event. See [Final-image streaming](#final-image-streaming).
- Images always come back as base64, with no partial previews.
- OpenAI serves the GPT Image models directly. Replicate, Krea and Recraft serve the other
  models. See [Krea image models](#krea-image-models),
  [Models served by Replicate](#models-served-by-replicate) and [Recraft models](#recraft-models).

## Examples

Each example generates one image and decodes it to `out.png`.

curl:

```bash
curl -s https://demo-api.routerplus.com/v1/images/generations \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"gpt-image-1","prompt":"a red bicycle on a white background","size":"1024x1024","quality":"medium"}' \
  | jq -r '.data[0].b64_json' | base64 -d > out.png
```

Python:

```python
import base64
import os
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

r = client.images.generate(
    model="gpt-image-1",
    prompt="a red bicycle on a white background",
    size="1024x1024",
    quality="medium",
)
with open("out.png", "wb") as f:
    f.write(base64.b64decode(r.data[0].b64_json))
print(r.usage.model_dump()["cost"])  # the exact ledger debit, in USD
```

TypeScript:

```typescript
import { writeFileSync } from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://demo-api.routerplus.com/v1", apiKey: process.env.TM_API_KEY });

const r = await client.images.generate({
  model: "gpt-image-1",
  prompt: "a red bicycle on a white background",
  size: "1024x1024",
  quality: "medium",
});

const b64 = r.data?.[0]?.b64_json ?? "";
writeFileSync("out.png", Buffer.from(b64, "base64"));
console.log(r.usage); // input/output tokens and cost (USD)
```

Send your key as `Authorization: Bearer tm_vk_...` or as `x-api-key: tm_vk_...`. A missing or
invalid key returns 401 with `error_type: "auth"`. You can send `HTTP-Referer` and `X-Title` to
identify your app. They carry attribution only, no content.

## Request

The body is JSON, at most 10 MB. A larger body is a 413 `request_too_large`. RouterPlus checks
every rule below before it reserves money or dispatches. A refused request has no ledger row,
no `x-tm-attempts` header and no charge.

| Parameter | Type | Behavior |
|---|---|---|
| `model` | string, required | An image model id. See [Model discovery](#model-discovery). A dated pin (`-20260421`, `-2026-04-21`, `-latest`) resolves to its catalog family for routing and pricing, and your exact string goes upstream. A chat model is a 400 that names the chat routes. A decision model is a 400 that names `/v1/decisions`. An id that is not an image model anywhere is a 404. Omitted or empty is a 400 that names `model`. |
| `prompt` | string, required | Non-empty, at most 32 000 characters. |
| `n` | integer | 1 to 4 on the GPT Image models. Defaults to 1. The SDK allows 10, but four buffered images already approach the response cap. |
| `size` | string | `auto`, `1024x1024`, `1536x1024` or `1024x1536` on the GPT Image models. |
| `quality` | string | `auto`, `low`, `medium`, `high`. Also `xhigh` and `max` on `gpt-image-2.5-flare` and `gpt-image-2.5-sunburst`. |
| `background` | string | `auto`, `transparent`, `opaque` on the GPT Image models. |
| `moderation` | string | `auto` or `low` on the GPT Image models. |
| `output_format` | string | `png`, `jpeg` or `webp` on the GPT Image models. |
| `output_compression` | integer | 0 to 100 on the GPT Image models. |
| `user` | string | Forwarded as sent. Any type other than a string is a 400. |
| `response_format` | string | `b64_json` only, and it has no effect. `url` is a 400. See [Always base64](#always-base64). |
| `stream` | boolean | `true` turns on final-image SSE and requires `n: 1`. Absent, `null` or `false` returns the ordinary JSON response. |
| `partial_images` | integer | Absent, `null` or `0` only. RouterPlus does not support partial previews. Other values are a 400 before dispatch. |
| `provider` | object | Routing controls, including `require_parameters`. RouterPlus reads them and never forwards them. See [Routing policies](/docs/routing-policies). |

Each model publishes its accepted values for `size`, `quality`, `background`, `moderation`,
`output_format` and `output_compression` in `/api/models.json` under `supported_parameters`.

- A value outside the declared set of a model is a 400. The message starts with the field
  name and lists what that model accepts.
- A parameter from this list that a model does not declare is dropped and recorded, like any
  unknown key.
- An explicit `null` means "not provided" for every optional parameter above. The OpenAI SDKs
  send an unset optional as `null`, so
  `client.images.generate(model=…, prompt=…, n=None, size=None)` works. RouterPlus never
  forwards a `null` and never records it as a drop.
- `model` and `prompt` are required, so `null` there is a 400.

### Everything else

RouterPlus validates and forwards the fields that the selected model declares in
`supported_parameters`. This includes the native Krea reference and editing parameters.

RouterPlus drops and records undeclared keys, and never forwards them. The dropped names are:

- comma-joined in the `x-tm-dropped-params` response header,
- stored on the ledger row of the attempt,
- visible at `GET /v1/generation?id=`.

To get a 400 instead of a drop, send `"provider": {"require_parameters": true}`. The 400 comes
before RouterPlus reserves money.

### Always base64

RouterPlus always returns the image bytes inline as `b64_json`. There is no hosted URL: storing
images would put your content on RouterPlus disks, which the
[data policy](/docs/data-policy) forbids. `response_format: "url"` is a typed 400. Decode the
base64 yourself, as the [examples](#examples) do.

## Response

Without `stream: true`, the answer is HTTP 200, `application/json`. The body is OpenAI's
`ImagesResponse`, relayed unchanged except for one added field, `usage.cost`:

```json
{
  "created": 1790000000,
  "data": [{ "b64_json": "iVBORw0KGgoAAAANSUhEUg..." }],
  "background": "opaque",
  "output_format": "png",
  "quality": "medium",
  "size": "1024x1024",
  "usage": {
    "input_tokens": 12,
    "input_tokens_details": { "text_tokens": 12, "image_tokens": 0 },
    "output_tokens": 1056,
    "total_tokens": 1068,
    "cost": 0.0423
  }
}
```

| Field | Meaning |
|---|---|
| `data[].b64_json` | The image, base64-encoded. One entry per image the provider rendered, normally `n`. RouterPlus relays what it received and bills what the provider reported. |
| `data[].revised_prompt` | The provider's rewritten prompt, when it sends one. See [Data policy](/docs/data-policy). |
| `usage.input_tokens` | Prompt tokens, billed at the `prompt` rate of the model. |
| `usage.output_tokens` | Image output tokens, billed at the `completion` rate of the model. |
| `usage.input_tokens_details` | For information only. The bill uses the two counts above. |
| `usage.total_tokens` | `input_tokens + output_tokens`. |
| `usage.cost` | USD, the full request debit, including any attempt that failed over before this one. `0` on BYOK. Absent on static dev keys. |

RouterPlus never adds the chat field `prompt_tokens_details` to an images response.

> [!NOTE]
> The provider can return a 200 without a usage object, or with `input_tokens` and
> `output_tokens` both 0. Then RouterPlus bills `n` × the per-image ceiling of the model, with
> provenance `estimated`. That is the amount the reservation held. RouterPlus writes those
> numbers into `usage`, so your copy of the bill still adds up. See the provenance at
> [GET /v1/generation](/docs/api-usage).

### Long-running JSON requests

Image generation can take several minutes. You do not need to change the request or the SDK.
An ordinary request keeps its final HTTP status, complete JSON body, headers and billing. Every
image count the model supports still works.

While the image is pending, the edge relay keeps its origin connection active, then returns the
original result. This avoids
[Cloudflare's default 125-second proxy read timeout](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-524/).
The relay stores no image content and starts no second generation.

The generation budget is 450 seconds. Set the overall timeout of your client to at least that.
Do not retry a lost response automatically: the provider may already have accepted the render.

### Final-image streaming

Send `"stream": true` with `"n": 1` for a render that may take several minutes.

1. After validation, the response starts as HTTP 200 with `Content-Type: text/event-stream`.
2. RouterPlus sends a comment at once, then a keep-alive comment every 15 seconds while the
   image is pending. Comments keep the connection active. They are not partial images or
   progress estimates.
3. The final event follows the
   [OpenAI image completion event](https://developers.openai.com/api/reference/resources/images/generation-streaming-events).
4. The connection closes.

```text
: rendering

event: image_generation.completed
data: {"type":"image_generation.completed","created_at":1790000000,"b64_json":"iVBORw0KGgoAAAANSUhEUg...","usage":{"cost":0.04},"request_id":"gen_example","response_headers":{"x-tm-provider":"krea","x-tm-attempts":"1"}}

```

| Field | Meaning |
|---|---|
| `b64_json` | The final image. |
| `created_at` | The response timestamp. |
| `size`, `quality`, `background`, `output_format` | Present only when known from the result. RouterPlus does not report requested values as measured output. |
| `usage` | The same accounting as the JSON response, including `usage.cost` where available. |
| `request_id` | RouterPlus extension. |
| `response_headers` | RouterPlus extension. The HTTP headers are already sent, so this carries the final routing headers, such as `x-tm-provider`, `x-tm-attempts` and `x-tm-upstream-status`. |

There is one completed event, with no partial-image events and no `[DONE]` event. The
`x-request-id` HTTP header identifies the request from the start.

An error found before the stream starts keeps its normal HTTP status and JSON body. After the
stream starts, a failed request ends with an `error` event in place of the completion:

```text
event: error
data: {"type":"error","error":{"code":"upstream_error","message":"Image generation failed.","type":"api_error","metadata":{"error_type":"upstream_error"}},"request_id":"gen_example","status":502,"response_headers":{"x-tm-error-code":"upstream_error"}}

```

- `status` is the HTTP-equivalent status of the failure. The open SSE response stays HTTP 200.
- `error` and `response_headers` carry the error details and the final routing headers.
- The OpenAI SDK raises streaming errors.
- Do not resubmit a disconnected stream or a timed-out request automatically. The provider may
  already have accepted the render. Use the request ID to read
  [generation usage](/docs/api-usage).

SSE is optional on the development API. Plain JSON uses the edge relay in
[Long-running JSON requests](#long-running-json-requests). Neither SSE nor plain JSON extends the generation budget or guarantees that a provider finishes
in time.

These examples keep a slow Qwen render connected and print its final event. The Python example
turns automatic retries off, so a lost response does not submit another render.

curl:

```bash
curl -N --max-time 480 https://demo-api.routerplus.com/v1/images/generations \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"krea/alibaba/qwen-image-3","prompt":"a red bicycle on a white background","width":1024,"height":1024,"n":1,"stream":true,"partial_images":0}'
```

Python:

```python
import base64
import os
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    base_url="https://demo-api.routerplus.com/v1",
    api_key=os.environ["TM_API_KEY"],
    timeout=480.0,
    max_retries=0,
)

stream = client.images.generate(
    model="krea/alibaba/qwen-image-3",
    prompt="a red bicycle on a white background",
    n=1,
    stream=True,
    partial_images=0,
    extra_body={"width": 1024, "height": 1024},
)
for event in stream:
    if event.type == "image_generation.completed":
        Path("out.image").write_bytes(base64.b64decode(event.b64_json))
        usage = event.usage.model_dump() if event.usage is not None else {}
        print(usage.get("cost"))
```

The file holds the provider's image bytes. Use `event.output_format` for the file extension
when it is present. Otherwise inspect the file type.

## Billing

Images use the same two SKUs and the same integer micro-USD math as chat
([Pricing & billing](/docs/pricing)). Text input bills at `prompt`, and image output bills at
`completion`.

- **Reserve**, before RouterPlus contacts a provider: prompt bytes ÷ 4 at the `prompt` rate,
  plus `n` × the per-image ceiling of the model at the `completion` rate, rounded up. The chat
  `max_tokens` default plays no part.
- **Settle** on the counts the provider reported, rounded down. RouterPlus releases the hold in
  full.

Example on `gpt-image-1` (ceiling 6240 tokens, $5 / $40 per million):

| Case | Amount |
|---|---|
| Reserve, `n: 1` | 6240 × $40/M ≈ $0.2496, plus the prompt estimate |
| Reserve, `n: 4` | 4 × 6240 × $40/M ≈ $0.9984, plus the prompt estimate |
| Settle, one medium 1024×1024 image | 1056 × $40/M = $0.04224, plus the prompt |

**A failed generation costs nothing.** A failed attempt, a response over the 32 MiB cap and a
200 with no decodable image all settle at zero. After RouterPlus receives a 200, it never
dispatches again, so one request never pays for two renders.

**A cancelled render is billed at what the provider reported.** If you close the connection
mid-render, RouterPlus does not abort the upstream call. The render finishes, RouterPlus reads
the usage, and the attempt settles at the provider's numbers. It is never the reservation and
never $0. RouterPlus writes nothing to your closed socket.

> [!WARNING]
> Reservations are large next to a trial balance. `n: 4` at a high quality holds about a
> dollar, and your balance must cover every reservation your organization has in flight. A
> `429 insufficient_quota` on `n: 4` is expected. Lower `n`, or add credits.

### Krea image models

Krea models have provider-qualified IDs such as `krea/krea-3` and
`krea/google/nano-banana-2.1`. `/api/models.json` lists the current models with the
`supported_parameters`, `required_parameters` and `request_defaults` of each. Structured
parameters carry their JSON schema under `schema`.

Each request renders one image (`n: 1`). RouterPlus runs Krea's asynchronous submit, poll and
download flow. Your request returns the completed OpenAI-style base64 image.

```json
{
  "model": "krea/krea-3",
  "prompt": "Change the mug to blue, keeping everything else the same.",
  "image_urls": ["data:image/png;base64,..."],
  "resolution": "2k",
  "thinking_mode": "fast",
  "regional_edit": true
}
```

Krea 3 parameters:

- `image_urls`: up to eight.
- `negative_prompt`: optional, up to 1,024 characters.
- `seed`.
- `style_urls` (style references): not in Krea's API yet. The field is a 400 before dispatch.
- Regional edits are experimental.
- `thinking_mode`: `fast` or `high`. `ultra` requires `resolution: "2k"` and cannot be
  combined with regional editing.
- Custom canvas: send both `width` and `height` (positive integers up to 4096), and omit
  `resolution` and `aspect_ratio`. Krea rounds the request to a supported canvas, so exact
  pixel dimensions are not guaranteed.
- Otherwise the default is 2K with automatic framing. The model lists its optional
  `aspect_ratio` values.

A Krea 3 reference is a Krea asset URL or an inline PNG, JPEG or WebP data URI. RouterPlus
uploads inline images to Krea's assets endpoint before it submits the job. The playground also
accepts local files. References count toward the 10 MB request limit. Other Krea models accept
the external reference URLs their schemas declare.

Pricing:

- Krea 3 is free during early access.
- Other enabled Krea models charge the fixed USD price for the selected settings and
  references, shown on their model pages.
- RouterPlus holds the published maximum of the model, then settles the selected price only
  after a successful render. It does not convert Krea's in-app compute units into API dollars.
- A model without verified price rules is unavailable until those rules exist.

Qwen Image 3 (`krea/alibaba/qwen-image-3`) and Seedream 5 Pro
(`krea/bytedance/seedream-5-pro`) price the requested total pixels, `width × height`:

| Model | Lower tier, inclusive | Above that area | References |
|---|---|---|---|
| Qwen Image 3 | ≤1,048,576 pixels: $0.04 | $0.075 | Up to 3; included in the price |
| Seedream 5 Pro | ≤2,359,296 pixels: $0.045 | $0.09 | Up to 10; first included, then $0.003 each |

- A Qwen 2048×512 image costs $0.04, the same as 1024×1024.
- A Seedream 3072×768 image costs $0.045 before additional references.
- Crossing the area threshold raises the base price, even when only one dimension changes.
- Qwen defaults to 1024×1024. Seedream defaults to 1536×1536. For another size, send positive
  whole-pixel `width` and `height`. The parameter schema of each model has its dimension bounds.
- The reservation ceilings are $0.075 (Qwen) and $0.117 (Seedream).

### Models served by Replicate

Replicate serves Nano Banana Pro, FLUX.2 [max] and [pro], Seedream 5.0 Lite, Grok Imagine Image
and Wan 2.7 Image Pro. RouterPlus bills them at Replicate's published price for the render,
with no margin ([Pricing & billing](/docs/pricing#models-billed-at-the-providers-reported-cost)).
The reservation is `n` × the per-image ceiling of the model (**Per image ≤**). The charge is
the listed price. These models take `aspect_ratio` (or `size`) and `resolution` in place of
`quality`.

| Model id | Made by | Shape | Listed price | Per image ≤ |
|---|---|---|---|---|
| `nano-banana-pro` | Google | `aspect_ratio`; `resolution` 1K, 2K, 4K | $0.15 at 1K or 2K, $0.30 at 4K | $0.30 |
| `flux-2-max` | Black Forest Labs | `aspect_ratio`; `resolution` 0.5MP, 1MP, 2MP, 4MP; `seed` | $0.04 a render + $0.03 per output megapixel | $0.19 |
| `flux-2-pro` | Black Forest Labs | as `flux-2-max` | $0.015 a render + $0.015 per output megapixel | $0.09 |
| `seedream-5-lite` | ByteDance | `aspect_ratio`; `resolution` 2K, 3K | $0.035 | $0.035 |
| `grok-imagine-image` | xAI | `aspect_ratio` | $0.02 | $0.02 |
| `wan-2.7-image-pro` | Alibaba | `size` (10 exact sizes up to 2048x2048, or `auto`); `n` up to 4, at most two 2048x2048 images' pixels per answer; `seed` | $0.03 | $0.03 |

- Replicate counts FLUX.2 output megapixels in whole megapixels, rounded up. A 0.5MP render
  bills as 1.
- Prices were read from Replicate on 2026-10-12.
- Wan 2.7 Image Pro answers in PNG only, so one answer holds at most 8,388,608 pixels. `n` can
  be up to 4 at the 1K sizes, 3 at 2048x1152 and 1152x2048, and 2 at 2048x1536, 1536x2048,
  2048x2048 and `auto`. A larger `n` is a 400 before anything renders, so it costs nothing.

### Recraft models

Recraft serves its models directly, with IDs of the form `recraft/<model>`, such as
`recraft/recraftv4_1` or `recraft/recraftv2_vector`. RouterPlus bills them at Recraft's
published price, passed through. The charge is the cost Recraft reports for the request.

| Family | Model ids | Raster | Vector | Pro raster (2K) | Pro vector |
|---|---|---|---|---|---|
| V4.1 Flash | `recraft/recraftv4_1_flash` | $0.007 | n/a | n/a | n/a |
| V4.1 | `recraft/recraftv4_1`, `…_vector`, `…_pro`, `…_pro_vector` | $0.035 | $0.08 | $0.21 | $0.30 |
| V4.1 Utility | `recraft/recraftv4_1_utility`, `…_vector`, `…_pro`, `…_pro_vector` | $0.035 | $0.08 | $0.21 | $0.30 |
| V4 Styles | `recraft/recraftv4_styles`, `…_vector`, `…_pro`, `…_pro_vector` | $0.035 | $0.05 | $0.10 | $0.12 |
| V4 | `recraft/recraftv4`, `…_vector`, `…_pro`, `…_pro_vector` | $0.04 | $0.08 | $0.25 | $0.30 |
| V3 | `recraft/recraftv3`, `recraft/recraftv3_vector` | $0.04 | $0.08 | n/a | n/a |
| V2 | `recraft/recraftv2`, `recraft/recraftv2_vector` | $0.022 | $0.044 | n/a | n/a |

Prices are per image. A request with `style_reference_urls` adds $0.005 once, because Recraft
creates a style from the references. The per-image ceiling includes that fee. Prices were read
from Recraft on 2026-10-16.

- **Output:**
  - `_vector` models return SVG. `data[].b64_json` holds the SVG text, and `output_format` is `svg`.
  - Raster models return WebP by default. Send `image_format: "png"` for PNG.
- **Images per request:** `n` takes 1 to 6.
- **`size`:** every model accepts the 14 aspect ratios (`1:1`, `16:9`, `6:10`, …). Raster models also accept their exact `WxH` sizes. Pro models use the 2K ones. Omit `size` to let Recraft choose from the prompt.
- **Styles:**
  - Every model except V4.1 Flash takes `style_id` (a Recraft style) or `style_reference_urls`: one to ten PNG, JPEG or WebP URLs or data URIs. Never both.
  - You can reuse the `style_id` from the response.
  - V4 Styles models require one of the two.
  - `style_match` is `flexible` or `precise` on V4 and newer, and `regular` on V2 and V3.
- **V2 and V3 only:** `style` (a curated style name such as `Photorealism`) and `negative_prompt`. Prompts on V2 and V3 are limited to 1,000 characters. On newer models the limit is 10,000.
- **V3 only:** `text_layout`. `controls` takes `colors` and `background_color` on every model, plus `artistic_level` and `no_text` on V3.
- **Seed:** RouterPlus passes `seed` to Recraft as its `random_seed`.

```json
{
  "model": "recraft/recraftv4_styles_vector",
  "prompt": "a lighthouse on a cliff",
  "size": "16:9",
  "style_reference_urls": ["https://example.com/brand-reference.png"]
}
```

## Errors

| Case | Status | `error_type` | Billed |
|---|---|---|---|
| A field outside the model's accepted values, `n` outside the model's limits, `stream: true` with `n` other than 1, nonzero `partial_images`, `response_format: "url"`, a chat or decision model on this route | 400 | `invalid_request` | no row |
| The model is listed but has no price row | 400 | `model_not_priced` | no row |
| Not an image model in the catalog | 404 | `model_unavailable` | no row |
| Request body over 10 MB | 413 | `request_too_large` | no row |
| Key RPM or a shared limit, balance too low, or a monthly spend cap | 429 | `rate_limit` / `insufficient_quota` | no row |
| OpenAI moderation refusal (`moderation_blocked`) | the provider's, usually 400 | `content_policy` | $0; never rerouted, never a health strike |
| Any other provider 4xx | the provider's | `upstream_error` | $0; no failover |
| Provider 5xx, a failed connection, or the 450 s budget | 5xx relayed / 502 / 504 | `upstream_error` / `model_unavailable` | $0 per failed attempt; fails over like chat |
| Response over the 32 MiB cap | 502 | `gateway_error` | $0; no failover |
| A 200 that is unparseable or has no `b64_json` | 502 | `upstream_error` | $0; no failover |
| The budget ran out while reading the image body | 504 | `upstream_error` | $0; no failover |
| Every deployment serving the model is cooling down | 503 | `gateway_error` | nothing dispatched; `retry-after: 5` |
| The ledger journal, the shared limit store, or the admission lease is unavailable | 503 | `gateway_error` | nothing charged |

- Error bodies use the OpenAI error envelope, with the canonical class in
  `error.metadata.error_type`. Every failure also has the class in `x-tm-error-code`.
- RouterPlus never relays a provider's own error body. The provider's status is in
  `x-tm-upstream-status`.
- After SSE begins, the terminal `error` event carries these details, and the HTTP status can
  no longer change. See [Final-image streaming](#final-image-streaming).
- An image model sent to `POST /v1/chat/completions`, `POST /v1/messages` or
  `POST /v1/decisions` is a 400 `invalid_request` that names `/v1/images/generations`.

[Errors](/docs/errors) lists every error class and its fix.

## Limits

| Limit | Value |
|---|---|
| Images per request | JSON: model limit up to `n` = 4; SSE: `n` = 1 |
| Prompt length | 32 000 characters |
| Request body | 10 MB → 413 `request_too_large` |
| Upstream response | 32 MiB → 502 `gateway_error`, nothing billed. |
| Generation budget | 450 s, covering the body read as well as the render |
| Streaming | Opt-in final-image SSE; no partial previews |
| Admission | one slot held for the whole render (typically 10 to 60 s) |
| Providers | OpenAI direct for the GPT Image models, never Azure, Bedrock or an Anthropic deployment |

## Response headers

| Header | When | Meaning |
|---|---|---|
| `x-request-id` | every call | the id `GET /v1/generation?id=` audits |
| `x-tm-provider` | served requests | deployment that rendered the image |
| `x-tm-attempts` | after ≥ 1 dispatch | physical dispatches, failovers included |
| `x-tm-upstream-status` | when a provider answered | the provider's own HTTP status |
| `x-tm-dropped-params` | when non-empty | comma-joined names of parameters RouterPlus stripped |
| `x-tm-upstream-model` | when the route knows the model by another id | the id RouterPlus sent |
| `x-tm-served-by` | when a route reports one | the provider that ran the request |
| `x-tm-route-plan-id` | billed requests | the route plan that chose the deployment |
| `x-tm-admission-mode`, `x-tm-remaining-rpm`, `x-tm-remaining-tpm` | billed requests | how the request was admitted and the headroom left. See [Limits and capacity](/docs/admission). |
| `x-tm-error-code` | every failure | the canonical error class |
| `x-tm-error-origin`, `x-tm-limit-scope`, `x-tm-limit-kind` | limit, provider and infrastructure failures | where it came from, and the limit that refused it |

There is no image-specific header. Browsers can read all of these except `x-tm-error-code`
and `x-tm-route-plan-id`, which are not CORS-exposed. For SSE, the `response_headers` extension
of the completed event also carries the final routing headers.

## Request data

Prompts, image bytes and any `revised_prompt` follow the [Data policy](/docs/data-policy).
Process logs, telemetry and the billing ledger contain metadata only. Telemetry records the
response byte size, to measure throughput.

## Model discovery

```bash
curl -s "https://demo-api.routerplus.com/v1/models?output_modalities=image" \
  -H "Authorization: Bearer $TM_API_KEY"
```

Every row of `GET /v1/models` carries `architecture.output_modalities`, for example
`["image"]`. The public feed `https://demo.routerplus.com/api/models.json` carries, for image models:

- `output_modalities`
- the full `supported_parameters` descriptor map
- `max_output_tokens`, the per-image ceiling

See [GET /v1/models](/docs/api-models) and [Models & catalog](/docs/models).

## BYOK

Image generation works through an `openai` provider connection.

- List on the connection the exact image ids you intend to send. Matching is literal. An id
  the connection does not carry is a `404 model_unavailable`.
- The model's metadata (ceiling, descriptors) still comes from the RouterPlus catalog.
- `usage.cost` is `0` and RouterPlus charges nothing. OpenAI bills your own account for the
  render.

See [Bring your own key](/docs/byok).

See also: [Models & catalog](/docs/models) · [Pricing & billing](/docs/pricing) ·
[Errors](/docs/errors) · [Wire compatibility](/docs/compat)

---

Source: https://demo.routerplus.com/docs/api-videos.md

# POST /v1/videos

Generate a video from a text prompt. The route follows the OpenAI Videos API, so
`client.videos.create` works unchanged with the base URL `https://demo-api.routerplus.com/v1`.

A video takes from ten seconds to several minutes to make. That is longer than any edge keeps
a request open, so each video is a job:

1. `POST /v1/videos` answers at once with the job.
2. `GET /v1/videos/{id}` reads the job. Repeat until it is `completed`.
3. `GET /v1/videos/{id}/content` downloads the MP4.

To try it without code, pick a video model in the [playground](/docs/playground).

Limits:

- Replicate serves every video model, on the RouterPlus account. A key bound to your own
  provider connection cannot make videos.
- Text-to-video only. RouterPlus refuses a request that sends an image input or reference
  images.

## Examples

Each example creates a job, polls it and downloads the MP4.

curl:

```bash
curl -s https://demo-api.routerplus.com/v1/videos \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"seedance-2.0-fast","prompt":"a paper boat drifting down a rainy street","seconds":"5"}'
# → {"id":"video_…","status":"queued",…}
curl -s https://demo-api.routerplus.com/v1/videos/video_… -H "Authorization: Bearer $TM_API_KEY"
curl -s https://demo-api.routerplus.com/v1/videos/video_…/content -H "Authorization: Bearer $TM_API_KEY" -o video.mp4
```

Python:

```python
import os
import time
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

video = client.videos.create(model="wan-3", prompt="a lighthouse at dusk", seconds="5")
while video.status in ("queued", "in_progress"):
    time.sleep(5)
    video = client.videos.retrieve(video.id)
if video.status == "completed":
    client.videos.download_content(video.id).write_to_file("video.mp4")
```

TypeScript:

```typescript
import { writeFileSync } from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://demo-api.routerplus.com/v1", apiKey: process.env.TM_API_KEY });

let video = await client.videos.create({ model: "veo-3.1-fast", prompt: "a fox leaping into snow", seconds: "4" });
while (video.status === "queued" || video.status === "in_progress") {
  await new Promise((r) => setTimeout(r, 5000));
  video = await client.videos.retrieve(video.id);
}
if (video.status === "completed") {
  const file = await client.videos.downloadContent(video.id);
  writeFileSync("video.mp4", Buffer.from(await file.arrayBuffer()));
}
```

The SDK types list OpenAI's own models and lengths. RouterPlus accepts any model id, and any
whole number of seconds that the model accepts.

Send your key as `Authorization: Bearer tm_vk_...` or as `x-api-key: tm_vk_...`. A missing or
invalid key returns 401 with `error_type: "auth"`.

## Create a video

`POST /v1/videos` takes a JSON body of at most 1 MB. RouterPlus checks every rule below before
it reserves money or sends anything to a provider. A refused request has no ledger row and no
charge.

| Parameter | Type | Behavior |
|---|---|---|
| `model` | string, required | A video model id. See [Model discovery](#model-discovery). A chat model is a 400 that names the chat routes. An image model is a 400 that names `/v1/images/generations`. An id that is not a video model anywhere is a 404. |
| `prompt` | string, required | Non-empty, at most 32 000 characters. |
| `seconds` | integer, or a string of digits | The length. OpenAI's SDKs send a string (`"8"`). RouterPlus reads both forms the same way. Each model accepts a range (see [Models](#models)). If you omit it, RouterPlus sends the model's default length, and the job is billed for the length that was reserved. |
| `duration` | integer | Another name for `seconds`. Send one of the two. Both with different values is a 400. |
| `resolution` | string | Per model, for example `480p`, `720p`, `1080p`. |
| `aspect_ratio` | string | Per model, for example `16:9`, `9:16`, `1:1`. |
| `size` | string | Exact pixels, for example `1280x720`, on the models that declare sizes (Seedance). Other models drop and record it. |
| `generate_audio` | boolean | On the models that make sound. |
| `seed` | integer | On the models that declare one. |
| `user` | string | Your end-user id, at most 256 characters. Kept with the job and returned on it. Never sent to the provider. |
| `provider` | object | Routing controls, including `require_parameters`. RouterPlus reads them and never forwards them. See [Routing policies](/docs/routing-policies). |

Each model publishes its accepted values in `/api/models.json` under `supported_parameters`.
A value outside the set of a model is a 400. The message starts with the field name and lists
what that model accepts. An explicit `null` means "not provided", as on the other routes.

**Refused, never dropped:** `input_reference`, `frame_images` and `input_references`. A dropped
image would make, and bill, a different video from the one you asked for.

**Dropped and recorded:** any other key, including `callback_url`. RouterPlus does not send
provider webhooks to your URL. The `x-tm-dropped-params` header and the ledger row name the
dropped fields. Send `"provider": {"require_parameters": true}` to get a 400 instead.

The answer is HTTP 200 with the video object and `status: "queued"`. It carries the same
headers as an image response: `x-request-id`, `x-tm-provider`, `x-tm-attempts`,
`x-tm-upstream-status`, and `x-tm-dropped-params` when a field was dropped. The provider has
60 seconds to accept the job. After that, the request is a 504 `upstream_error`.

## The video object

```json
{
  "id": "video_7f3c2a9b1e4d4c6a8b0f2e1d3c5a7b9e",
  "object": "video",
  "model": "wan-3",
  "status": "completed",
  "progress": 100,
  "created_at": 1790000000,
  "completed_at": 1790000094,
  "expires_at": null,
  "seconds": "5",
  "size": null,
  "resolution": "720p",
  "aspect_ratio": "16:9",
  "error": null,
  "usage": { "cost": 0.5 }
}
```

| Field | Meaning |
|---|---|
| `id` | The RouterPlus id: `video_` and 32 hex digits. Never the provider's id. |
| `status` | `queued` → `in_progress` → `completed` or `failed`. |
| `progress` | `0` until the video is done, then `100`. The providers report no progress, and RouterPlus does not estimate one. |
| `seconds` | A string, as in OpenAI's object. |
| `error` | On a failed job: `{code, message}`. The code is one of `video_generation_failed`, `content_policy_violation`, `video_cancelled`, `video_expired`, `video_timeout`. RouterPlus never keeps the provider's own words. |
| `usage.cost` | The charge in USD, once the job has ended. Absent while the job runs, and on static dev keys. |
| `user` | Your `user`, when you sent one. |

## Read, list and download

| Call | Answer |
|---|---|
| `GET /v1/videos/{id}` | The video object. Poll every few seconds. A job takes minutes. |
| `GET /v1/videos` | Your organization's videos, newest first: `{object: "list", data, first_id, last_id, has_more}`. Query: `limit` (1 to 100, default 20), `order` (`desc` or `asc`), `after` (a video id). |
| `GET /v1/videos/{id}/content` | The MP4, streamed from the provider. `?variant=video` is the only variant. Any other value is a 400. RouterPlus passes a `Range` header on, so a player can seek. A range the provider cannot satisfy is a 416. Before the job is `completed`, the call is a 409. |

- RouterPlus polls the provider for you: every 5 seconds for the first two minutes, then every
  10 seconds, then every 20. A read between two polls shows the last status RouterPlus has.
- A video of another organization is a 404, the same as a video that does not exist.
- The file streams through as it arrives, with the provider's `content-type` (else
  `video/mp4`), `content-length`, `content-range` and `accept-ranges`.
- RouterPlus never stores the file. Save it when you download it.

## Billing

RouterPlus bills a video at the provider's listed price for the job: Replicate's published
price per second of video, for the seconds Replicate says it billed. That is never more than 5%
past the length you asked for. The price passes through with no margin. See
[Pricing & billing](/docs/pricing#models-billed-at-the-providers-reported-cost).

| Case | Charge |
|---|---|
| Before the job is submitted | RouterPlus reserves `seconds` × the per-second ceiling of the model. The models page shows that ceiling as **Per second ≤**. Your balance must cover it. |
| The job ends | The reported cost. RouterPlus releases the rest of the hold. |
| A failed, cancelled or expired job | Nothing. You get no video. |
| A completed job without a reported cost | The reservation, marked `estimated` in [GET /v1/generation](/docs/api-usage). |
| The gateway restarts while a job runs | The reservation, as for any request a crash leaves open. The job continues, and `GET /v1/videos/{id}` still answers for it. |
| No end after two hours of polling | The reservation. The job ends `failed` with the code `video_timeout`. |

Example on `wan-3` (per-second ceiling $0.105): a 5-second request reserves $0.525
(5 × $0.105). At 480p, Replicate bills $0.025 a second, so the charge is $0.125. RouterPlus
releases the rest of the hold.

## Models

| Model id | Made by | Length | Resolutions | Sound | Listed price | Per second ≤ |
|---|---|---|---|---|---|---|
| `wan-3` | Alibaba | 2 to 30 s (default 5) | 480p, 720p, 1080p | provider default; no audio toggle | $0.025/s at 480p, $0.05/s at 720p, $0.10/s at 1080p | $0.105 |
| `grok-imagine-video` | xAI | 1 to 15 s (default 5) | 480p, 720p | provider default; no audio toggle | $0.05/s | $0.0525 |
| `seedance-2.0` | ByteDance | 4 to 15 s (default 5) | 480p, 720p, 1080p | yes, at no extra cost | $0.08/s at 480p, $0.18/s at 720p, $0.45/s at 1080p | $0.48 |
| `seedance-2.0-fast` | ByteDance | 4 to 15 s (default 5) | 480p, 720p | yes, at no extra cost | $0.07/s at 480p, $0.15/s at 720p | $0.16 |
| `kling-v3-video` | Kuaishou | 3 to 15 s (default 5) | 720p, 1080p, 4k | `generate_audio`, off by default | $0.168/s at 720p ($0.252 with sound), $0.224/s at 1080p ($0.336 with sound), $0.42/s at 4k | $0.441 |
| `veo-3.1-fast` | Google | 4, 6 or 8 s (default 8) | 720p, 1080p | on by default; `generate_audio: false` turns it off | $0.15/s with sound, $0.10/s without | $0.158 |

- [Wan 3](https://replicate.com/alibaba/wan-3) and
  [Grok Imagine Video](https://replicate.com/xai/grok-imagine-video) follow the provider's
  audio defaults. This API has no audio toggle for them.
- Prices are Replicate's published prices for each model, read on 2026-10-12.
- Replicate bills the video it returned. That can run a fraction of a second past the length
  you asked for (5.04 s for 5 s). The fraction is billed, up to 5% of the length.
- Sora 2 was retired on September 24, 2026. Existing video and usage records stay readable. See
  [OpenAI’s discontinuation notice](https://help.openai.com/en/articles/20001152-what-to-know-about-the-sora-discontinuation).

## Errors

| Case | Status | `error_type` | Billed |
|---|---|---|---|
| A field outside the model's accepted values, an image input, a chat or image model on this route | 400 | `invalid_request` | no row |
| The model is listed but has no price row | 400 | `model_not_priced` | no row |
| Not a video model in the catalog | 404 | `model_unavailable` | no row |
| Request body over 1 MB | 413 | `request_too_large` | no row |
| Unknown id, or another organization's video | 404 | `not_found` | n/a |
| The file before the job is completed | 409 | `invalid_request` | n/a |
| Key RPM or a shared limit, balance too low for the reservation, or a monthly spend cap | 429 | `rate_limit` / `insufficient_quota` | no row |
| The provider refused the submit | the provider's | `upstream_error` | $0 |
| The provider did not accept the job within 60 s | 504 | `upstream_error` | $0 |
| The provider accepted the job but named no job id, or its answer could not be read | 502 | `upstream_error` | $0 |
| Every deployment serving the model is cooling down | 503 | `gateway_error` | nothing dispatched; `retry-after: 5` |
| The provider did not send the file on a download | 502 | `upstream_error` | n/a |
| The provider accepted the job but it failed | 200 on the read; `status: "failed"` | n/a | $0 |

Error bodies use the OpenAI error envelope, with the canonical class in
`error.metadata.error_type`. RouterPlus never relays a provider's own error body. A video model
sent to `POST /v1/chat/completions`, `POST /v1/messages` or `POST /v1/images/generations` is a
400 that names `/v1/videos`.

## Data policy

RouterPlus keeps the id, model, length, shape, status and charge of a job, so you can retrieve
it later. The video file streams from the provider when you download it. See
[Data policy](/docs/data-policy).

## Model discovery

```bash
curl -s "https://demo-api.routerplus.com/v1/models?output_modalities=video" \
  -H "Authorization: Bearer $TM_API_KEY"
```

Every row of `GET /v1/models` carries `architecture.output_modalities`, for example
`["video"]`. For each video model, the public feed `https://demo.routerplus.com/api/models.json` carries:

- `supported_parameters`
- `max_output_tokens`: the per-second ceiling in cost units (one micro-dollar each)
- `price_card`: the provider's listed prices

See also: [POST /v1/images/generations](/docs/api-images) ·
[Pricing & billing](/docs/pricing) · [Errors](/docs/errors)

---

Source: https://demo.routerplus.com/docs/api-audio.md

# POST /v1/audio

Turn speech into text, and text into speech. These routes follow the OpenAI Audio API, so
`client.audio.transcriptions.create` and `client.audio.speech.create` work unchanged with the
base URL `https://demo-api.routerplus.com/v1`.

| Route | Alias | What it does |
|---|---|---|
| `POST /v1/audio/transcriptions` | `POST /audio/transcriptions` | Returns the text of an audio file |
| `POST /v1/audio/speech` | `POST /audio/speech` | Returns spoken audio for a text, streamed as it is made |

## Examples

```bash
curl https://demo-api.routerplus.com/v1/audio/transcriptions \
  -H "Authorization: Bearer $ROUTERPLUS_API_KEY" \
  -F model=whisper-1 \
  -F response_format=srt \
  -F file=@meeting.mp3
```

```bash
curl https://demo-api.routerplus.com/v1/audio/speech \
  -H "Authorization: Bearer $ROUTERPLUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o-mini-tts", "input": "Your order has shipped.", "voice": "coral"}' \
  --output speech.mp3
```

```python
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key="tm_vk_...")

with open("meeting.mp3", "rb") as f:
    transcript = client.audio.transcriptions.create(model="gpt-4o-transcribe", file=f)
print(transcript.text)

with client.audio.speech.with_streaming_response.create(
    model="tts-1", voice="alloy", input="Your order has shipped."
) as response:
    response.stream_to_file("speech.mp3")
```

Send your key as `Authorization: Bearer tm_vk_...` or as `x-api-key: tm_vk_...`. A missing or
invalid key returns 401.

## Models

| Model | Route | Price | Billed on |
|---|---|---|---|
| `gpt-4o-transcribe` | transcriptions | $2.50 per 1M input tokens, $10.00 per 1M output tokens | `usage.input_tokens` and `usage.output_tokens` |
| `gpt-4o-mini-transcribe` | transcriptions | $1.25 per 1M input tokens, $5.00 per 1M output tokens | `usage.input_tokens` and `usage.output_tokens` |
| `whisper-1` | transcriptions | $0.006 per minute ($0.0001 per second) | seconds of audio, rounded up |
| `gpt-4o-mini-tts` | speech | $0.60 per 1M input tokens, $12.00 per 1M audio output tokens | the tokens the provider reports |
| `tts-1` | speech | $15.00 per 1M characters | characters of `input` |
| `tts-1-hd` | speech | $30.00 per 1M characters | characters of `input` |

OpenAI serves all six models. `GET /v1/models?output_modalities=transcription` lists the
transcription models, and `?output_modalities=audio` lists the speech models.

## POST /v1/audio/transcriptions

### Request

The body is `multipart/form-data`. The audio file is at most 25 MiB. A larger file is a 413. RouterPlus checks every rule below before it reserves money or calls a provider. A
refused request has no ledger row and no charge.

| Field | Type | Behavior |
|---|---|---|
| `file` | file, required | The audio: flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav or webm. |
| `model` | string, required | A transcription model id. A speech, chat, image, video, decision or embeddings model is a 400 that names its own route. An id that is not in the catalog is a 404. |
| `response_format` | string | `json` (the default), `text`, `verbose_json`, `srt` or `vtt`. `gpt-4o-transcribe` and `gpt-4o-mini-transcribe` take `json` and `text` only. |
| `language` | string | The language of the audio as an ISO-639-1 code, for example `en`. |
| `prompt` | string | Text that guides the style or spelling of the transcript. |
| `temperature` | number | From 0 to 1. |
| `timestamp_granularities[]` | string | `segment` or `word`. With `whisper-1` and `verbose_json` only. |
| `provider` | JSON string | Routing controls. See [Provider routing](/docs/provider-routing). |

RouterPlus drops any other field and names it in the `x-tm-dropped-params` header. With
`provider.require_parameters: true`, a dropped field is a 400. `stream=true` is a 400: the
route returns one response.

### Response

`json`:

```json
{
  "text": "Hello there.",
  "usage": { "type": "tokens", "input_tokens": 1500, "output_tokens": 40, "total_tokens": 1540, "cost": 0.00415 }
}
```

`whisper-1` reports the billed seconds:

```json
{
  "text": "Hello there.",
  "usage": { "type": "duration", "seconds": 62, "cost": 0.0062 }
}
```

| Format | Content type | Body |
|---|---|---|
| `json` | `application/json` | `text` and `usage` |
| `verbose_json` | `application/json` | `text`, `language`, `duration`, `segments` (and `words` when asked for), and `usage` |
| `text` | `text/plain` | The transcript |
| `srt` | `text/plain` | SubRip subtitles, one cue per segment |
| `vtt` | `text/vtt` | WebVTT subtitles, one cue per segment |

`usage.cost` is the charge in USD for the request. The `text`, `srt` and `vtt` formats do not
carry it. `GET /v1/generation` shows it for every request.

## POST /v1/audio/speech

### Request

The body is JSON, at most 1 MB.

| Parameter | Type | Behavior |
|---|---|---|
| `model` | string, required | A speech model id. |
| `input` | string, required | The text to speak. At most 4,096 characters. |
| `voice` | string, required | A voice name, for example `alloy`, `coral` or `nova`. |
| `response_format` | string | `mp3` (the default), `opus`, `aac`, `flac`, `wav` or `pcm`. |
| `speed` | number | From 0.25 to 4. The default is 1. |
| `instructions` | string | How to speak, for example "calm and slow". `gpt-4o-mini-tts` only. The tokens count as input. Another model drops the field. |
| `stream_format` | string | `audio` only. Any other value is a 400. |
| `provider` | object | Routing controls. See [Provider routing](/docs/provider-routing). |

### Response

The body is the audio file in `response_format`. RouterPlus sends each part of the audio when
it arrives from the provider, so playback can start before the speech is complete.

| Format | Content type |
|---|---|
| `mp3` | `audio/mpeg` |
| `opus` | `audio/opus` |
| `aac` | `audio/aac` |
| `flac` | `audio/flac` |
| `wav` | `audio/wav` |
| `pcm` | `audio/pcm` (raw 24 kHz, 16-bit, mono samples) |

The response does not carry the cost. `GET /v1/generation` shows it.

## Billing

1. Before the call, RouterPlus holds an estimate of the cost on your balance.
2. After the call, it charges the billed units from the [Models](#models) table and releases the rest of the hold.
3. If the provider reports no usage, RouterPlus charges the estimate it held.

The hold for each model:

| Model | Hold |
|---|---|
| `gpt-4o-transcribe`, `gpt-4o-mini-transcribe` | 16,000 input tokens and 2,000 output tokens |
| `whisper-1` | The length of the audio. For a wav or mp3 file, RouterPlus reads the length from the file. For another format, it assumes 1,000 bytes per second. |
| `gpt-4o-mini-tts` | The input text at 3 bytes per token, and 2 audio tokens per character divided by `speed` |
| `tts-1`, `tts-1-hd` | The characters of `input` |

A failed attempt costs nothing.

## Routing and failover

RouterPlus picks the provider with the same rules as chat. [Guardrails](/docs/guardrails)
apply: a model your guardrail does not allow is a 403 with `error_type: "guardrail_blocked"`.

If a provider fails before it answers (a 5xx, a 429 or a timeout), RouterPlus tries the next
provider of the model. For speech, failover stops at the first byte of audio. After that byte,
the answer comes from that provider. `x-tm-attempts` gives the number of attempts.
`x-tm-provider` names the provider that answered.

---

Source: https://demo.routerplus.com/docs/api-embeddings.md

# POST /v1/embeddings

Turn text into vectors. The route follows the OpenAI Embeddings API, so
`client.embeddings.create` works unchanged with the base URL `https://demo-api.routerplus.com/v1`.
`https://demo-api.routerplus.com/embeddings` is an alias. The route returns one JSON response and does not
stream.

## Examples

```bash
curl https://demo-api.routerplus.com/v1/embeddings \
  -H "Authorization: Bearer $ROUTERPLUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "text-embedding-3-small", "input": ["first text", "second text"]}'
```

```python
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key="tm_vk_...")
result = client.embeddings.create(model="text-embedding-3-small", input="hello world")
print(len(result.data[0].embedding), result.usage.prompt_tokens)
```

Send your key as `Authorization: Bearer tm_vk_...` or as `x-api-key: tm_vk_...`. A missing or
invalid key returns 401.

## Models

| Model | Provider | Price per 1M input tokens | Max tokens per input |
|---|---|---|---|
| `text-embedding-3-small` | OpenAI | $0.02 | 8,191 |
| `text-embedding-3-large` | OpenAI | $0.13 | 8,191 |
| `baai/bge-m3` | DeepInfra | $0.01 | 8,192 |
| `qwen/qwen3-embedding-8b` | DeepInfra | $0.01 | 32,768 |
| `intfloat/multilingual-e5-large` | DeepInfra | $0.01 | 512 |

`GET /v1/models?output_modalities=embeddings` lists the embeddings models your key can use.
You pay for input tokens only. Output tokens are free.

## Request

The body is JSON, at most 10 MB. RouterPlus checks every rule below before it reserves money
or calls a provider. A refused request has no ledger row and no charge.

| Parameter | Type | Behavior |
|---|---|---|
| `model` | string, required | An embeddings model id. A chat, image, video or decision model is a 400 that names its own route. An id that is not in the catalog is a 404. |
| `input` | required | A string, an array of strings, a token array (integers), or an array of token arrays. At most 2,048 inputs. An empty string, an empty array or an empty token array is a 400. |
| `encoding_format` | string | `float` (the default) or `base64`. |
| `dimensions` | integer | A positive integer. Only models that support it shorten their vectors (`text-embedding-3-*` and some open models). |
| `user` | string | Sent to the provider as is. |
| `provider` | object | Routing controls. RouterPlus reads them and does not send them to the provider. See [Provider routing](/docs/provider-routing). |

RouterPlus drops any other field and names it in the `x-tm-dropped-params` header. With
`provider.require_parameters: true`, a dropped field is a 400.

A 400 message starts with the field name, for example `input[1]: must not be an empty string`.

## Response

```json
{
  "object": "list",
  "data": [
    { "object": "embedding", "index": 0, "embedding": [0.0123, -0.0456] }
  ],
  "model": "text-embedding-3-small",
  "usage": { "prompt_tokens": 5, "total_tokens": 5, "cost": 0.0000001 }
}
```

| Field | Meaning |
|---|---|
| `data` | One item per input, in the order you sent them. |
| `embedding` | An array of floats. With `encoding_format: "base64"`, a base64 string of little-endian float32 values. |
| `model` | The id you sent. |
| `usage.prompt_tokens` | The count the provider reports. You are billed on it. |
| `usage.cost` | The charge in USD for this request. |

## Billing

1. Before the call, RouterPlus holds the estimated cost of your input on your balance.
2. After the call, it charges `usage.prompt_tokens` at the model's input price and releases the rest of the hold.
3. If the provider reports no usage, RouterPlus charges the estimate it held.

A failed attempt costs nothing. `GET /v1/generation` shows each attempt.

## Routing and failover

RouterPlus picks the provider with the same rules as chat. [Guardrails](/docs/guardrails)
apply: a model your guardrail does not allow is a 403 with `error_type: "guardrail_blocked"`.

If a provider fails before it answers (a 5xx, a 429 or a timeout), RouterPlus tries the next
provider of the model. `x-tm-attempts` gives the number of attempts. `x-tm-provider` names the
provider that answered.

A key with an OpenAI connection sends OpenAI embeddings requests on your own OpenAI key. See
[Bring your own key](/docs/byok).

---

Source: https://demo.routerplus.com/docs/api-decisions.md

# POST /v1/decisions

Ask a decision model typed questions about a **state** (a message, a ticket, a record) and get one typed answer per question. A decision model does not write text. Each answer is a choice, a score, a yes/no, an order, a set of tags or extracted fields, with the probability or confidence behind it. Use it for routing, classification, moderation and other decision points in your code.

```bash
curl https://demo-api.routerplus.com/v1/decisions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"typesafe/jev-1.13",
       "state":"Help! My payouts have been failing for 3 days.",
       "questions":{"urgent":{"type":"noul","instructions":"Is this urgent?"}}}'
```

No SDK has a decisions method. Send plain HTTP.

Python:

```python
import os, httpx

r = httpx.post(
    "https://demo-api.routerplus.com/v1/decisions",
    headers={"Authorization": f"Bearer {os.environ['TM_API_KEY']}"},
    json={
        "model": "typesafe/jev-1.13",
        "state": "Help! My payouts have been failing for 3 days.",
        "questions": {"team": {"type": "choice", "instructions": "Which team?",
                               "criteria": {"billing": "Payments", "technical": "Bugs"}}},
    },
)
answer = r.json()["answers"]["team"]
print(answer["choice"], answer["confidence"])
```

Node:

```js
const r = await fetch("https://demo-api.routerplus.com/v1/decisions", {
  method: "POST",
  headers: { authorization: `Bearer ${process.env.TM_API_KEY}`, "content-type": "application/json" },
  body: JSON.stringify({
    model: "typesafe/jev-1.13",
    state: { ticket: "Help! My payouts have been failing for 3 days." },
    questions: { urgent: { type: "noul", instructions: "Is `ticket` urgent?" } },
  }),
});
const { answers, usage } = await r.json();
console.log(answers.urgent.noul, usage.cost);
```

## Models

The catalog has twelve decision models:

| Model | Catalog id | Made by | Providers |
|---|---|---|---|
| Jev 1.13 | `typesafe/jev-1.13` | TypeSafe | `typesafe` |
| Bespoke Nimble v4 | `bespokelabs/nimble-v4` | Bespoke Labs | `bespokelabs` |
| Bespoke Nimble v4 Max | `bespokelabs/nimble-v4-max` | Bespoke Labs | `bespokelabs` |
| Clef | `cloudflare/clef` | Cloudflare | `workers-ai` |
| Clef-flash | `cloudflare/clef-flash` | Cloudflare | `workers-ai` |
| Decider 2B | `routerplus/decider-2b` | RouterPlus | `routerplus` |
| Kev 4B | `routerplus/kev-4b` | RouterPlus | `routerplus` |
| Perplexity Decider v1 27B | `perplexity/pplx-decider-v1-27b` | Perplexity | `perplexity-decisions` |
| GPT-6 Luna Decisions | `openai/gpt-6-luna-decisions` | OpenAI | `openai-decisions` |
| Sage 1.2 | `levanto/sage-1.2` | Levanto | `levanto` |
| ScaleDown Classify-1 | `scaledown/classify-1` | ScaleDown | `scaledown` |
| GLiNER-2.5-Decide | `fastino/gliner-2.5-decide` | Fastino | `fastino` |

The gateway also accepts `bespokelabs/nimble-latest`, which no list shows. Bespoke picks the model that answers it. See [Bespoke Nimble](#bespoke-nimble).

Every call has the same envelope: `model` and `state` in, `answers` and `usage` out. The questions use the model's own format:

- Eleven models share one format, **System One** (TypeSafe's): Jev, the two Nimble models, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider, GPT-6 Luna Decisions, Sage and ScaleDown Classify-1. They take a `questions` map. A body written for one of them runs on the others with only `model` changed, within each model's limits. ScaleDown Classify-1 bills each question as its own call.
- GLiNER takes a `schema` object: `classifications`, `entities`, `structures`, `relations`.
- A question in another model's format is a 400.
- The twelve are different models, so the gateway never fails over from one to another.

The gateway calls these endpoints:

| Model | Order | Provider | Endpoint the gateway calls | Model id sent upstream |
|---|---|---|---|---|
| Jev | 1 | TypeSafe (direct API) | `POST https://api.typesafe.ai/v1/systemone` | `jev-1.13.0` |
| Nimble v4, Nimble v4 Max | 1 | Bespoke Labs (direct API) | `POST https://api.bespokelabs.ai/v1/nimble/systemone` | `nimble-v4`, `nimble-v4-max` (and `nimble-latest` for `bespokelabs/nimble-latest`) |
| Clef | 1 | Cloudflare Workers AI (direct API) | `POST https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/run/@cf/cloudflare/clef` | `clef` |
| Clef-flash | 1 | Cloudflare Workers AI (direct API) | `POST https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/run/@cf/cloudflare/clef-flash` | `clef-flash` |
| Decider 2B | 1 | RouterPlus (our own service, direct API) | `POST /v1/decisions` on RouterPlus's endpoint | `decider-2b` |
| Kev 4B | 1 | RouterPlus (our own service, direct API) | `POST /v1/decisions` on RouterPlus's endpoint | `kev-4b` |
| Perplexity Decider | 1 | Perplexity (direct API) | `POST https://api.perplexity.ai/v1/decisions` | `pplx-decider-v1-27b` |
| GPT-6 Luna Decisions | 1 | OpenAI (direct API) | `POST https://api.openai.com/v1/decisions` | `gpt-6-luna` |
| ScaleDown Classify-1 | 1 | ScaleDown (direct API) | `POST https://api.scaledown.xyz/v1/scaledown` | `classify-1` |
| Sage | 1 | Levanto (direct API) | `POST https://sage.levanto.ai/v1/systemone` | `sage-latest` |
| GLiNER | 1 | Fastino (direct API) | `POST https://api.fastino.ai/v1/chat/completions` | `fastino/GLiNER-2.5-Decide` |

Each decision model has one provider, so a request has no second provider to try. The Nimble models share one Bespoke
deployment: one pool and one health circuit serve all of them.

- Decider 2B and Kev 4B share one RouterPlus deployment: one pool and one health circuit serve both.
- On Workers AI, `{account_id}` is our Cloudflare account, and the path names the model.
- OpenAI's API does not speak System One. The gateway translates your body into OpenAI's format, and OpenAI's answers back into Jev's shape (see [GPT-6 Luna Decisions](#gpt-6-luna-decisions)).
- The `x-tm-provider` header names the provider that answered.

To compare the models without code, use **Decision** in the [playground](/docs/playground). It translates one set of questions for each model.

## Authentication

| Header | Format |
|---|---|
| `Authorization` | `Bearer tm_vk_...` |
| `x-api-key` | `tm_vk_...` (also accepted, same key) |

A missing or invalid key returns 401 with `error_type: "auth"`.

## Request

The body is JSON, at most 1 MB (413 `request_too_large` above that). The gateway checks every rule on this page **before** it reserves money or calls a provider. A refused request has no ledger row, no `x-tm-attempts` header and no charge.

| Parameter | Type | Behavior |
|---|---|---|
| `model` | string, required | A decision model id. A dated id of a listed decision model routes as that model: `typesafe/jev-1.13-20260917` as `typesafe/jev-1.13`. Providers' own names are not ids (see the next table). A chat or image model is a 400 that names its route. An unknown id is a 404. |
| `state` | required | What the model evaluates. Its forms depend on the model: see [System One questions](#system-one-questions) and [GLiNER schema](#gliner-schema). |
| `questions` | object, the System One models, required | A map of question id to question. You choose the ids, and each answer comes back under its id. At least one question. On GLiNER, `questions` is a 400 that says the model takes a GLiNER `schema`. |
| `schema` | object, GLiNER only, required | GLiNER's own schema. See [GLiNER schema](#gliner-schema). On Jev, Nimble, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider, GPT-6 Luna Decisions, Sage or ScaleDown Classify-1, `schema` is a 400 that says the model takes `questions`. |
| `threshold`, `include_confidence`, `include_spans` | GLiNER only | See [GLiNER options](#gliner-options). |
| `provider` | object | Routing controls: `only`, `order`, `ignore`, `require_parameters` and the rest. The gateway reads them and never forwards them. See [Routing policies](/docs/routing-policies). The provider ids are in the next table. |

Model names that are not ids, and provider ids:

| Model | Send as `model` | Not an id here | `provider` id |
|---|---|---|---|
| Jev | `typesafe/jev-1.13` | | `typesafe` |
| Nimble | `bespokelabs/nimble-v4`, `bespokelabs/nimble-v4-max` | Bespoke's names `nimble-v4`, `nimble-v4-max`, `nimble-latest`. `bespokelabs/nimble-latest` is accepted but not listed: the gateway sends it to Bespoke as `nimble-latest`, and Bespoke picks the model (see [Bespoke Nimble](#bespoke-nimble)). | `bespokelabs` |
| Clef, Clef-flash | `cloudflare/clef`, `cloudflare/clef-flash` | `clef`, `clef-flash`, `@cf/cloudflare/clef`, `@cf/cloudflare/clef-flash` | `workers-ai` (not `cloudflare`, which matches no route) |
| Decider 2B, Kev 4B | `routerplus/decider-2b`, `routerplus/kev-4b` | RouterPlus's own short names (`decider-2b`, `kev-4b`) are not ids here | `routerplus` |
| Perplexity Decider | `perplexity/pplx-decider-v1-27b` | `pplx-decider-v1-27b` | `perplexity-decisions` (not `perplexity`, which matches no route) |
| GPT-6 Luna Decisions | `openai/gpt-6-luna-decisions` | OpenAI's own name `gpt-6-luna` is our id for OpenAI's chat model GPT-6 Luna, so here it is a 400 that names the chat routes. `openai/gpt-6-luna` and `gpt-6-luna-decisions` without `openai/` are a 404. | `openai-decisions` for GPT-6 Luna Decisions (not `openai`, which is our OpenAI chat route) |
| Sage | `levanto/sage-1.2` | `sage-latest`, `levanto-sage-v1.3` | `levanto` |
| GLiNER | `fastino/gliner-2.5-decide` | | `fastino` |

### System One questions

Nine models take the same question format, TypeSafe's **System One**: Jev, the two Nimble models, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider and GPT-6 Luna Decisions. This section holds for each of the nine. Some models add limits of their own: see [Bespoke Nimble](#bespoke-nimble), [Clef and Clef-flash](#clef-and-clef-flash), [Decider 2B and Kev 4B](#decider-2b-and-kev-4b), [Perplexity Decider v1 27B](#perplexity-decider-v1-27b), [GPT-6 Luna Decisions](#gpt-6-luna-decisions) and [Sage 1.2](#sage-12).

The examples use Jev. To run them on another System One model, change `model`.

**State.** A non-empty string, or any JSON object or array (a chat log, a record, application state). A question can point at a field by name: `` "Is `ticket.body` urgent?" ``.

**Questions.** Every question has a `type`, `instructions` and, for most types, `criteria`. `instructions` is a non-empty string, or a JSON object or array: put the question in one field and the data it refers to in others.

| `type` | `criteria` | The answer |
|---|---|---|
| `choice` | Required. An object of option name to description, 1 to 255 options. A description can be a string, JSON, or `null` when the name says enough. | `choice` (the most likely option), `probabilities` (every option), `confidence` |
| `score` | Required. An ordered array of level descriptions, low to high, 2 to 10 levels. | `score` (probability-weighted, can fall between levels), `legend`, `probabilities`, `confidence` |
| `noul` | Optional. `{"true": "...", "false": "..."}`: what yes and no mean. | `noul`: the probability that the answer is yes, 0 to 1 |

- A question takes only `type`, `instructions` and `criteria`. Any other field is a 400 that names it.
- A question with `kind` (the format of Levanto's own `/decide` API) is a 400 that says the model takes System One questions.
- Ask many questions in one call. The model reads the state once and answers every question against it, so one call with ten questions costs less and returns sooner than ten calls.
- Exceptions: Perplexity Decider and Sage read and bill the state once per question. GPT-6 Luna Decisions bills the state once per call, but each question adds about 150 input tokens.
- `confidence` is not the chance that the answer is right. Some models compute it their own way (see each model's Numbers row). Tune a threshold on the model you use, or use the probabilities.

```json
{
  "model": "typesafe/jev-1.13",
  "state": { "ticket": "Help! My payouts have been failing for 3 days." },
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team should handle `ticket`?",
      "criteria": { "billing": "Payments, invoices, refunds", "technical": "Bugs, outages, integrations", "sales": null }
    },
    "frustration": {
      "type": "score",
      "instructions": "How frustrated is the customer?",
      "criteria": ["Calm", "Annoyed", "Angry"]
    },
    "urgent": {
      "type": "noul",
      "instructions": "Is this urgent?",
      "criteria": { "true": "Explicitly time-sensitive", "false": "No urgency expressed" }
    }
  }
}
```

The sections below list the limits and behavior of each System One model. The gateway checks the rules above. It does not check each model's limits. The Playground's Decision mode does check them, before a run. If a body breaks a model's limits, the provider refuses it, and the gateway relays the refusal as an `upstream_error` with the provider's message, at no cost. [Limits and failures](#limits-and-failures) covers the rules for every provider: pool limits, relayed 429s, the health circuit and refusals of our account.

#### Bespoke Nimble

The gateway calls Bespoke Labs' own API.

| | Bespoke Nimble v4 and Nimble v4 Max |
|---|---|
| Catalog ids | `bespokelabs/nimble-v4` and `bespokelabs/nimble-v4-max`. Bespoke no longer serves Nimble v3: `bespokelabs/nimble-v3` is a 404. |
| Models | Nimble v4 is the faster, lower-cost model. Nimble v4 Max is Bespoke's most accurate model. It is larger and slower, and it costs more. The two return different probabilities for the same request, so check your thresholds when you change models. |
| `bespokelabs/nimble-latest` | Accepted, but no list shows it. Bespoke picks the model: Nimble v4 today, and a newer model when Bespoke releases one. It costs $0.04 per million input tokens, Bespoke's price for `nimble-latest`. The answer's `model` is `bespokelabs/nimble-latest`. To stay on one model, send that model's id. |
| Route | Bespoke Labs' API only, as `nimble-v4`, `nimble-v4-max` or `nimble-latest`. One deployment serves all of them. |
| Questions | 1 to 64 questions per call. A `choice` takes 2 to 255 options, so a one-option choice (which Jev takes) is refused. A `score` takes 2 to 10 levels through the gateway (Nimble itself takes up to 255). Bespoke refuses a body above these limits with a 422. |
| Structured content | `instructions` and a `choice` option's description may be a JSON object or array, as on Jev. Bespoke's published schema types them as strings, but a probe with both answered 200 (2026-10-13 on Nimble v4 and v4 Max). |
| Context | 262,144 tokens for each question's prompt on Nimble v4, Nimble v4 Max and `bespokelabs/nimble-latest`, the state included. Nimble refuses a longer prompt and never cuts one. Bespoke counts the state and the questions once per call. |
| Busy | Bespoke answers 503 with `Retry-After` while Nimble starts, 529 when it is busy (Bespoke says to retry after about one second) and 502 when the model fails. Each is a retryable `upstream_error` that counts against the health circuit, and one circuit covers all the Nimble models. Retry after a second or two, with backoff. The gateway does not relay Bespoke's `Retry-After` on a 503. |
| Numbers | Full precision. Jev's numbers have 2 decimals, and Nimble's are not rounded. A very small probability can come back in exponent form, for example `3.7e-06`. `confidence` is 1 when one option has all the probability and 0 when every option is equally likely. |
| Our limit | The pool takes 7 requests in flight at once, and each organization at most 5 of them. Bespoke allows our account 8, and our test environment uses one. The pool also takes 600 requests a minute (450 per organization) and 15,000,000 tokens a minute (11,250,000 per organization). The Nimble models share these limits. |
| Bespoke's limit | At most 8 requests at once for our whole account, over all the Nimble models. Above that Bespoke answers 429 with `Retry-After`, which the gateway relays (see [Limits and failures](#limits-and-failures)). |
| Retention | Bespoke keeps request content for up to 30 days, to debug the service and check for abuse, and then deletes it. The deployment declares `prompt_logging: "retained"`. Bespoke does not train on API content unless an organization opts in, and ours does not. See [Data policy](/docs/data-policy). |

#### Clef and Clef-flash

The gateway calls Cloudflare's own API, Workers AI. Clef-flash is the smaller model (9 billion parameters, against Clef's 27 billion). It costs $0.09 per million input tokens, against $0.24 for Clef.

| | Clef and Clef-flash |
|---|---|
| Route | Workers AI's REST API only, as `clef` and `clef-flash`. The path picks the model, and the gateway sets the same model in the body. It removes Cloudflare's envelope (`result`, `success`, `errors`, `messages`), so you get Jev's answer shape. A 200 whose envelope does not say `success: true` is a 502 `upstream_error` and costs nothing. |
| Questions | 1 to 64 questions per call. Each question id (a key of `questions`) must match `^[A-Za-z0-9_.-]{1,100}$`. A `choice` takes 2 to 255 options, so a one-option choice (which Jev takes) is refused. A `score` takes 2 to 10 levels. Cloudflare refuses a body outside these limits with a 422 (or 400). |
| Structured content | `instructions`, a `score` level and a `choice` option's description may be a JSON object or array, as on Jev. An option's description may be `null`. Cloudflare's schema allows both, and a probe with JSON `instructions` and JSON option descriptions answered 200 (2026-10-02). |
| Context | 65,536 tokens, the state and every question together. Before the model runs, Cloudflare estimates the request's tokens at about 4 characters a token. It refuses a request estimated above 65,536 tokens (about 262,000 characters) with a 413. The gateway returns it as 413 `context_overflow`, and it costs nothing. Cloudflare's schema says a long state is cut to fit, but in our test (2026-10-02) a request of 522,286 characters was refused, not cut. |
| Images | Not accepted through the gateway. Clef's `images` field, Cloudflare's addition to System One, is dropped and recorded in `x-tm-dropped-params`. With `"provider": {"require_parameters": true}` it is a 400. |
| `request_id` | Do not send a top-level `request_id`. The gateway drops and records it. Workers AI reads that field as a lookup in its queue of async requests, and answers 404. |
| Numbers | 4 decimals. Cloudflare reports `output_tokens: 0` on every answer. `confidence` is on Clef's own scale, which Cloudflare describes only as derived from the probabilities. On one live answer (three options, the top one at 0.8274), Clef's `confidence` was 0.5654, where Jev's formula gives 0.7411. A threshold that you tuned on Jev's `confidence` does not transfer to Clef. |
| Busy | When Workers AI is busy, Cloudflare answers 429 with code 3040, "Capacity temporarily exceeded, please try again." The gateway relays it with Cloudflare's words. Without a `Retry-After`, the relayed 429 has no `retry-after` either, and the pool pauses for 1 s. A 5xx (for example Cloudflare's 500, "Model execution failed") counts against the health circuit. A 408, Cloudflare's own timeout, is a retryable 504 `upstream_error` that does not count. |
| Our limit | Clef and Clef-flash share one pool. It takes 200 requests a minute (150 per organization), at most 8 requests at a time (6 per organization) and 2,000,000 tokens a minute (1,500,000 per organization). |
| Cloudflare's limit | Cloudflare publishes 300 requests a minute for Clef's task type, Text Generation. It does not say whether Clef and Clef-flash share that limit, so our pool stays below it. Cloudflare's 429 with code 3036 means that our account's free daily allocation is used up. The gateway returns 503 `model_unavailable` with `metadata.retryable: false`. This is our account, never yours, and it never counts against the health circuit. |
| Retention | Cloudflare says that it does not read, store or train on requests to Clef, or on their responses. The deployment declares `prompt_logging: "none"`. The gateway calls Workers AI directly, never through Cloudflare's AI Gateway, which logs prompts and responses by default. See [Data policy](/docs/data-policy). |

#### Decider 2B and Kev 4B

RouterPlus runs its own decision-model service. Both models run on our own GPUs in the United States, and each reads English.

| | Decider 2B | Kev 4B |
|---|---|---|
| Catalog id | `routerplus/decider-2b` | `routerplus/kev-4b` |
| Name at RouterPlus | `decider-2b` | `kev-4b` |
| Input, per million tokens | $0 | $0 |
| Context | 25,600 tokens. A longer input is cut (see below). | 8,192 tokens. A longer input is refused. |

Rules for both:

| | Decider 2B and Kev 4B |
|---|---|
| Route | RouterPlus's endpoint only. One deployment serves both models, so one pool, one health circuit and one pause after a 429 cover both. |
| Price | Free: input and output are $0, and `usage.cost` is `0` on every call. RouterPlus runs these models itself and charges nothing for them. See [Billing](#billing). |
| Questions | The gateway checks the System One rules above. Both models take every form that Jev takes: `instructions` as a string or as a JSON object or array, `score` levels as strings or as JSON objects (the answer's `legend` gives the levels as you sent them), and a `choice` option's description as a string, JSON or `null`. In probes on 2026-10-02, each model also answered a `choice` with one option, a `choice` of 255 options, a `score` of 2 and of 10 levels, a yes/no with described `true` and `false`, a JSON object or array as the state, and 200 questions in one request. A question that RouterPlus calls malformed is its 400, relayed with RouterPlus's reason for each question id (see [Errors](#errors)). It costs nothing. |
| Long input | Decider 2B reads up to 25,600 tokens and cuts a longer input. It keeps the question and its options first, then the start of the state, and drops the rest without an error. Kev 4B never cuts: above 8,192 tokens it refuses the request with a 400. The gateway bills RouterPlus's `usage.input_tokens`, which counts only the tokens the model read: at most the model's context. In a test on 2026-10-02, a Decider 2B request of 30,053 tokens billed 25,600. RouterPlus can refuse a long input instead of cutting it (its `truncate` field), but the gateway drops that field. Through the gateway, a long input on Decider 2B is always cut. Put what matters at the start of the state. Use Kev 4B for an input up to 8,192 tokens and Decider 2B up to 25,600. |
| Response cache | RouterPlus keeps each answer, with its usage, for 600 s (10 minutes). The cache key is the RouterPlus API key, the model and the exact request body, byte for byte. The gateway sends every request with our one RouterPlus key, so an identical request from any customer within 600 s can get the cached answer: the same `answers` and the same input count in `usage`. Your response keeps its own `id`. RouterPlus marks a hit with `usage.cached: true` and the header `X-Cache: HIT` in its own response. A cached answer costs $0: the listing prices cached input at $0. The gateway's response marks it with `usage.cached_input_tokens`, equal to `usage.input_tokens`. RouterPlus skips its cache for a request with `"cache": false` or a `Cache-Control: no-cache` header. The gateway forwards neither, so you cannot skip the cache through the gateway. |
| Request id | RouterPlus sends `X-Request-Id` (`req_…`) on every answer. The gateway stores it with the request's record as the provider's id. Your response's `id` and its `x-request-id` header are the gateway's own. |
| Cold start | After a quiet period, the first request starts the model on a GPU. This takes 15 to 20 s. The gateway waits up to 60 s for a decision, so a cold start is a slow 200. |
| Our limit | The pool takes 11,400 requests a minute, 15,000,000 tokens a minute and 59 requests in flight at once. Each organization may use 75 % of each: 8,550 requests a minute, 11,250,000 tokens a minute, 44 at once. The 59 in flight is the burst limit, since a pool has no limit per second. At RouterPlus's typical 0.31 s a call, 59 in flight carry about 190 a second, just above the pool's 11,400 a minute. With our test environment's 10 a second, that stays within RouterPlus's own limit. Your organization's own limits bind first. |
| RouterPlus's limit | About 200 requests a second for the whole endpoint, shared with our test environment. A burst of repeated requests can reach it, because cached answers return sooner. Above it RouterPlus answers 429. The gateway relays it and pauses the pool for both models. |
| Busy or failed | A 5xx from RouterPlus counts against the health circuit: two in a row open it for 30 s for both models. A RouterPlus 401, 402 or 403 refuses our key, never yours: you get 503 `model_unavailable`. |
| Retention | The RouterPlus model service writes no request logs. Its answer cache keeps each answer and its usage for 600 s, under a key made from our RouterPlus key and the request. The cache is all it keeps, so the deployment declares `prompt_logging: "none"`. Nothing is used for training. See [Data policy](/docs/data-policy). |

#### Perplexity Decider v1 27B

The gateway calls Perplexity's own API. This model reads and bills the state once per question, so the cost grows with the number of questions.

| | Perplexity Decider v1 27B |
|---|---|
| Route | Perplexity's API only, as `pplx-decider-v1-27b`. Perplexity answers in Jev's shape, with no envelope. The gateway stores Perplexity's `x-request-id` with the request's record as the provider's id. |
| The state, once per question | Each question runs its own prompt, which holds and bills the whole state. In a test on 2026-10-02, a state of 1,843 tokens billed 1,843 tokens with one question and 9,215 with five. One call with ten questions saves round trips, not tokens: it costs about what ten calls cost. The question ids are not billed. `output_tokens` is one per question, at $0. A `choice` with one option runs no prompt and costs nothing: a request of only such questions returns 0 tokens and costs $0. |
| Questions | 1 to 128 questions per call. Above 128 is Perplexity's 400, "Each request needs between 1 and 128 questions". A `choice` takes 1 to 255 options, as on Jev. A `score` takes 2 to 10 levels. A question id is any key the gateway takes. |
| Structured content | `instructions`, a `score` level and a `choice` option's description may be a JSON object or array, as on Jev. An option's description may be `null`. A probe with JSON `instructions`, JSON option descriptions and JSON score levels answered 200 (2026-10-02). |
| Context | 262,144 tokens for each question's prompt: the state and that one question. A request with several questions can bill more than that in all: a state of 52,348 tokens with six questions billed 314,088 tokens (2026-10-02). Above the limit, Perplexity refuses the request with a 400, "Input length (262144) exceeds or equals model's maximum context length (262144)". The gateway returns it as 400 `context_overflow`, with `metadata.retryable: false`, and it costs nothing. Perplexity does not cut a long state. |
| Image parts | Decision models take text and JSON. An object with `"type": "image_url"`, anywhere in the state or in a question, is a 400 from the gateway before any money is reserved, for example `state[1]: an object with "type": "image_url" is an image part, and model "perplexity/pplx-decider-v1-27b" takes text and JSON only`. Other JSON, including a field named `type` with another value, goes as you sent it. |
| Numbers | Full precision, as Nimble's. On a `choice`, `confidence` is Jev's: `(p_max − 1/n) / (1 − 1/n)`, so a threshold that you tuned on Jev's choice confidence transfers. On a `score` it does not. Perplexity's score confidence is `1 − Σ pᵢ·|i − m| / ((1/L)·Σⱼ |j − m|)`, where `m` is the most likely level and `L` the number of levels. Jev measures the second sum about the central level, so the two agree only when the most likely level is the central one. On one live answer (three levels at 0.756, 0.193 and 0.051), Perplexity's `confidence` was 0.7051, where Jev's formula gives 0.5577. Tune a score threshold on Perplexity Decider. |
| Busy | Perplexity answers 504 when the model does not answer in about a minute. The body is an HTML page. The gateway returns a retryable 504 `upstream_error` that counts against the health circuit. In Perplexity's tests, a few hundred input tokens answered in under 2 s, and a prompt near the context in 23 s. |
| Our limit | The pool takes 540 requests a minute, at most 8 requests at a time and 15,000,000 tokens a minute. Each organization may use 75 % of each: 405 requests a minute, 6 at a time, 11,250,000 tokens a minute. Tokens are held at the gateway's estimate while a request runs, with the state counted once per question, and corrected to Perplexity's count when it settles. A long state with many questions can be larger than a token limit on its own. Such a request is a 429 `rate_limit` with `x-tm-limit-kind: tpm` before anything reaches Perplexity, and it cannot pass as it is. Ask fewer questions per call, or send a shorter state. |
| Perplexity's limit | Perplexity allows our organization 10 requests a second, over a rolling second, shared with our test environment. Our pools keep to that as a minute budget (600 a minute in all) and do not count seconds, so a burst inside one second can get Perplexity's 429. Perplexity also limits large bursts of tokens, with no published number. Above a limit Perplexity answers 429, "Request rate limit exceeded, please try again later.", with `Retry-After: 1`, which the gateway relays. |
| Retention | Perplexity says that it does not retain query data sent through its API and does not train on it: its API has zero day retention of prompt data by default. Its compute runs on AWS in North America. The deployment declares `prompt_logging: "none"`. See [Data policy](/docs/data-policy). |

#### GPT-6 Luna Decisions

The gateway calls OpenAI's own API. OpenAI's API has its own format, so the gateway translates your System One body on the way out and OpenAI's answers on the way back.

| | GPT-6 Luna Decisions |
|---|---|
| Catalog id | `openai/gpt-6-luna-decisions`. Released 2026-10-06, the day OpenAI's Decisions API opened to every developer as a public beta. |
| Route | OpenAI's API only, as `gpt-6-luna`. One provider, no failover. The gateway stores OpenAI's `x-request-id` (`req_…`) with the request's record as the provider's id. |
| Each question's prompt | OpenAI bills the state once per call, but it runs a prompt for each question, and each question adds about 150 input tokens. In tests on 2026-10-07, one yes/no about a two-character state billed 156 tokens, two billed 307 and five billed 760. With a state of 2,795 tokens, each added question cost 151 tokens more. The question ids are not billed. One call with many questions saves round trips and the state's tokens, but each question still costs about 150 tokens. The gateway holds 192 tokens for each question (see Our limit below). |
| Questions | 1 to 200 questions per call. A `choice` takes 2 to 255 options, so a one-option choice (which Jev takes) is refused. A `score` takes 2 to 10 levels. A question id is any key the gateway takes: a 300-character id and an id with a space and an accent answered 200 (2026-10-07). OpenAI refuses a body outside these limits with a 400. |
| Translation | The state and `instructions`: a string goes as you sent it, and a JSON object or array goes as compact JSON text. A `choice`: each option becomes one of OpenAI's choices, with the option name as its value and the description beside it (a JSON description as JSON text, and no description for `null`). A `score`: each level becomes one of OpenAI's levels, with the level's text as its label (a JSON level as JSON text). The answer's `legend` gives the levels as you sent them. A `noul`: OpenAI's predicate, the probability that the statement is true. A predicate has no criteria, so the gateway writes your `true` and `false` descriptions under the instructions, as two lines: `true: …` and `false: …`. In a live test the descriptions moved the probability the way they say: from 0.82 to 0.90 on an urgent ticket, and from 0.10 to 0.02 on a calm one. |
| Error names | OpenAI names a question in an error by its place in OpenAI's list. The gateway puts your question id (at most its first 64 characters) and System One's field name in its place: OpenAI's `choices` and `levels` are both `criteria`. |
| Refusal | OpenAI can decline one question and answer the rest. That question's answer is `{"type": "refusal"}`, under its id, with no value. No other System One model sends this type, so check each answer's `type` before you read `choice`, `score` or `noul`. The call is still billed: OpenAI bills the input of every question, the declined one too. In our tests (2026-10-07), OpenAI declined a question with empty instructions, and a `choice` of 255 options whose names meant nothing. The Playground shows such an answer as **Refused**. |
| Numbers | 2 decimals, as Jev's: OpenAI rounds them. `confidence` is OpenAI's own, and OpenAI does not document it as Jev's formula. A threshold that you tuned on Jev's `confidence` does not transfer to GPT-6 Luna Decisions. |
| Context | 272,000 tokens, the state and every question together. OpenAI's listed price holds up to there. OpenAI bills input above 272,000 tokens at twice the rate. The model itself reads up to 1,050,000 tokens. The request body is at most 1 MB. |
| Images | Not through the gateway. The state goes to OpenAI as text, so an object such as `{"type": "image_url", ...}` in the state is read as text, not as an image. Decision models take text and JSON. |
| Busy | In our probes a call took about 0.1 to 0.3 s. A 5xx from OpenAI counts against the health circuit. OpenAI's 404 `model_not_found` means that our access to the model was withdrawn. You get 404 `model_unavailable` with OpenAI's words, ``The model `gpt-6-luna` does not exist or you do not have access to it.`` |
| Our limit | The pool takes 24,000 requests a minute, 150,000,000 tokens a minute and 64 requests in flight at once. Each organization may use 75 % of each: 18,000 requests a minute, 112,500,000 tokens a minute, 48 at once. Tokens are held at the gateway's estimate while a request runs, with 192 tokens for each question, so a request with many questions is held near what OpenAI bills. The hold is corrected to OpenAI's count when the request settles. |
| OpenAI's limit | OpenAI allows our organization 30,000 requests and 180,000,000 tokens a minute on the model. Our test environment uses the same OpenAI key, and its pool takes 600 requests a minute, 6,000,000 tokens a minute and 8 in flight. Above OpenAI's limit, OpenAI answers 429. The gateway relays it with OpenAI's `retry-after` when it sends one. |
| Retention | OpenAI keeps abuse-monitoring logs of `/v1/decisions` requests for up to 30 days. Zero Data Retention needs an eligible account and an agreement with OpenAI, and our organization does not have it. So the deployment declares `prompt_logging: "retained"`. OpenAI does not train on API data by default. See [Data policy](/docs/data-policy). |

#### Sage 1.2

The gateway calls Levanto's System One API. Like Perplexity Decider, Sage reads and bills the state once per question.

| | Sage 1.2 |
|---|---|
| Route | Levanto's API only, as `sage-latest`. Levanto answers in Jev's shape. |
| The state, once per question | Levanto reads and bills the state once for each question, so one call with ten questions saves round trips, not tokens. The gateway holds the state once per question too. |
| Questions | A `choice` takes 2 to 120 options and a `score` 2 to 26 levels. Every question gets an answer, never `null`. Levanto refuses a request outside these limits. |
| Not offered | Levanto's own `/decide` format (`kind`, sort, tags), images, grounding and `reasoning`. A question with `kind` is a 400. `reasoning` and `latency_mode` are dropped and recorded. |
| Retention | The deployment declares `prompt_logging: "retained"`. See [Data policy](/docs/data-policy). |

### GLiNER schema

GLiNER-2.5-Decide is a small encoder model. It reads one text and fills in a schema. It classifies the text against your labels, and it can extract entities, structured fields and relations.

**State.** Text only: a non-empty string, or `{"kind": "text", "value": "..."}`. A list, an image or any other state is a 400 that names the field. To judge a record, send it as text, for example the JSON as a string.

**Schema.** GLiNER takes no `questions`. It takes `schema`, Fastino's own schema object, which the gateway checks and forwards unchanged. `schema` has only these four keys. Send at least one, and each key you send must be non-empty (an empty list or object is a 400 that names the key). A flat list in place of the object is a 400 (Fastino has deprecated it).

| Key | Form | Limits |
|---|---|---|
| `classifications` | A list of `{"task", "labels", "multi_label"?, "top_k"?, "cls_threshold"?}` | 1 to 50 tasks. `task` is 1 to 256 characters, unique in the schema, and not `__proto__`. `labels` is 1 to 100 unique non-empty strings. `multi_label` is a boolean. `top_k` is an integer, 1 or more. `cls_threshold` is a number from 0 to 1. Any other key is a 400 that names it. |
| `entities` | A list of entity names, or of `{"name", "description"?}` | 1 to 50 entities. Each name is a non-empty string, unique in the list. |
| `structures` | An object of structure name to a list of fields, each `"field::type::description"` | 1 to 50 structures. A name is 1 to 256 characters and not `__proto__`. 1 to 50 non-empty field strings per structure. |
| `relations` | A list of relation names, or an object of name to `{"description"?, "threshold"?}` | 1 to 50 unique non-empty names. `threshold` is a number from 0 to 1. A `head` or `tail` key is a 400. |

- **Labels are plain strings.** A label cannot be an object with a description. To describe a label, put the description in the label itself: `"credit: store credit"`. Descriptions change results. On the same refund ticket, the labels `refund`, `credit`, `deny` chose `refund`, and the same labels with descriptions chose `credit`. The answer carries the label text as you sent it.
- **The task name is the question.** A classification has no `instructions` field. GLiNER reads the task name, so a task named `"Does the customer ask for money back?"` with the labels `["yes", "no"]` is a yes/no question.
- **Result names share one namespace.** GLiNER puts every result at the top level of its answer: each task under its task name, each structure under its structure name, the entities under `entities` and the relations under `relation_extraction`. At Fastino, two results with one name overwrite each other without an error. The gateway refuses such a collision with a 400 that names both, for example a task named `entities` in a request that also asks for entities.

```json
{
  "model": "fastino/gliner-2.5-decide",
  "state": "I was charged twice for my March invoice. Please refund the order from Acme Corp. Signed, Jane Doe.",
  "schema": {
    "classifications": [
      { "task": "action", "labels": ["refund", "credit", "deny"] },
      { "task": "issues", "labels": ["double_charge", "angry_customer", "fraud"], "multi_label": true }
    ],
    "entities": ["person", { "name": "organization", "description": "business or institution name" }]
  }
}
```

#### GLiNER options

Three optional top-level fields go to Fastino unchanged. Leave them out to use Fastino's defaults.

| Field | Values | What it does |
|---|---|---|
| `threshold` | A number from 0 to 1. Default 0.5. | The confidence a result needs to be returned. A task's own `cls_threshold` wins for that task. Lower values return more results. Higher values return fewer, surer ones. |
| `include_confidence` | Boolean. Default `true`. | With `false`, a single-label answer is the bare label string, and a multi-label answer is a list of label strings. |
| `include_spans` | Boolean. Default `true`. | With `true`, each entity carries `start` and `end`: its character offsets in the text. |

- A single-label task returns its top label only. `top_k` does not add more labels to the answer.
- When no label reaches the threshold, the task's answer is `null`. Set `"cls_threshold": 0` on the task to always get the top label and its confidence.
- A task with `multi_label: true` returns every label at or above the threshold, so `"cls_threshold": 0` returns every label with its confidence.

#### Store

Fastino keeps each inference unless the request says `store: false`. **The gateway always sends `store: false`.** A `store` field that you send is dropped and recorded, never forwarded. Our Fastino account also has Zero Data Retention on, so Fastino does not train on your text. See [Data policy](/docs/data-policy).

#### Not offered yet

The gateway does not reach these parts of Fastino's API:

- Fine-tuned GLiNER models (Fastino's training-job ids).
- Fastino's own batch and async routes.
- Several messages. The gateway sends Fastino one user message: the state. GLiNER does not read a system message.

### Parameters that do not apply

Decision models take no sampling or output controls. The gateway forwards no `temperature`, `max_tokens`, `stream`, `seed`, `user` or `metadata` to them. None offers prompt caching, so `cache_control` does nothing. RouterPlus's answer cache (see [Decider 2B and Kev 4B](#decider-2b-and-kev-4b)) is not prompt caching, and `cache_control` does not reach it.

A top-level field that the model does not take is **dropped and recorded**, never forwarded. RouterPlus's own `truncate` and `cache` fields are dropped in this way:

| Model | Top-level fields it takes | Examples of fields dropped |
|---|---|---|
| Jev, Nimble, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider, GPT-6 Luna Decisions, Sage, ScaleDown Classify-1 | `model`, `state`, `questions`, `provider` | `temperature`, `reasoning`, `latency_mode`, `images`, `request_id`, `truncate`, `cache` |
| GLiNER | `model`, `state`, `schema`, `provider`, `threshold`, `include_confidence`, `include_spans` | `store`, `temperature`, `messages` |

The dropped names are in the `x-tm-dropped-params` response header and on the attempt's ledger row. To refuse instead, send `"provider": {"require_parameters": true}`. A field that would be dropped is then a 400 before any money is reserved.

## Response

HTTP 200, `application/json`. Every decision model uses the same envelope: `id`, `object`, `model`, `answers` and `usage`.

### System One answers

Every System One model returns this shape. The differences:

| Model | `model` | Numbers | `output_tokens` |
|---|---|---|---|
| Jev | `typesafe/jev-1.13` | 2 decimals | As reported |
| Nimble | its catalog id, for example `"model": "bespokelabs/nimble-v4"` | full precision | As reported |
| Clef, Clef-flash | `"model": "cloudflare/clef"` or `"model": "cloudflare/clef-flash"` | 4 decimals | `"output_tokens": 0` |
| Decider 2B, Kev 4B | their catalog ids | As reported | As reported |
| Perplexity Decider | `"model": "perplexity/pplx-decider-v1-27b"` | full precision | one per question |
| GPT-6 Luna Decisions | `"model": "openai/gpt-6-luna-decisions"` | 2 decimals | `"output_tokens": 0` |

```json
{
  "id": "5d0c1c4e-3f7a-4c55-9d1b-2f0e7f6f2a10",
  "object": "decision",
  "model": "typesafe/jev-1.13",
  "answers": {
    "team": { "type": "choice", "choice": "billing",
              "probabilities": { "billing": 0.88, "technical": 0.12, "sales": 0 }, "confidence": 0.81 },
    "frustration": { "type": "score", "score": 1.05,
                     "legend": { "0": "Calm", "1": "Annoyed", "2": "Angry" },
                     "probabilities": { "0": 0, "1": 0.95, "2": 0.05 }, "confidence": 0.92 },
    "urgent": { "type": "noul", "noul": 0.95 }
  },
  "usage": { "input_tokens": 436, "output_tokens": 71, "cost": 0.000018 }
}
```

`answers` holds one typed answer per question, under the ids you sent, as the provider returned them. GPT-6 Luna Decisions can also return `{"type": "refusal"}` for a question that OpenAI declined (see [GPT-6 Luna Decisions](#gpt-6-luna-decisions)). Check each answer's `type` before you read its value.

### GLiNER answers

`answers` is GLiNER's result, unchanged. Fastino returns it as a JSON string in `choices[0].message.content`, and the gateway parses it into an object. For the request in [GLiNER schema](#gliner-schema):

```json
{
  "id": "0b7e4c2a-6f1d-4a39-9c85-3e2d1f7a8b64",
  "object": "decision",
  "model": "fastino/gliner-2.5-decide",
  "answers": {
    "action": { "label": "refund", "confidence": 0.9327635765075684 },
    "issues": [ { "label": "double_charge", "confidence": 0.872347354888916 } ],
    "entities": {
      "person": [ { "text": "Jane Doe", "confidence": 0.99609375, "start": 90, "end": 98 } ],
      "organization": [ { "text": "Acme Corp", "confidence": 0.98828125, "start": 71, "end": 80 } ]
    }
  },
  "usage": { "input_tokens": 33, "output_tokens": 91, "cost": 0 }
}
```

| You asked | Key in `answers` | Value |
|---|---|---|
| A single-label task | The task name | `{"label", "confidence"}`: the top label. `null` when no label reaches the threshold (0.5 unless you set one; `"cls_threshold": 0` always returns the top label). With `include_confidence: false`, the bare label string. |
| A multi-label task | The task name | `[{"label", "confidence"}, ...]`: every label at or above the threshold, possibly none. With `include_confidence: false`, a list of label strings. |
| Entities | `entities` | An object of entity name to `[{"text", "confidence", "start", "end"}, ...]`. `start` and `end` are there when `include_spans` is on. |
| A structure | The structure name | A list of records, each field as `{"text", "confidence"}`, for example `{"order": [{"id": {"text": "4411", "confidence": 1.0}}]}`. |
| Relations | `relation_extraction` | `{"relations": [...]}`. |

- **A confidence is not a calibrated probability.** It is GLiNER's score for that label or span, from 0 to 1. Multi-label confidences are independent and do not sum to 1. Compare them within one model, not with System One's probabilities.
- GLiNER has no "not sure" answer. A low confidence shows that it is unsure.
- The gateway accepts a 200 only when the content parses to a JSON object with every task and structure name you asked for, plus `entities` and `relation_extraction` when you asked for them. Anything else is an `upstream_error` and costs nothing.

### Usage and headers

| Field | Meaning |
|---|---|
| `id` | The gateway's request id, also in the `x-request-id` header. Look the request up with `GET /v1/generation?id=`. |
| `model` | The catalog id that was billed. The provider's own name for the model is in the `x-tm-upstream-model` header. |
| `usage.input_tokens` | Input tokens, billed at the model's input rate. The provider's count (Fastino's `prompt_tokens` for GLiNER). On Decider 2B, at most the model's context. On Perplexity Decider and Sage, the state once per question. |
| `usage.output_tokens` | Output tokens, billed at the model's output rate. Every decision model prices output at $0. Sage, Clef, Clef-flash and GPT-6 Luna Decisions report 0, and Perplexity Decider one per question. For GLiNER, Fastino's `completion_tokens`. |
| `usage.cached_input_tokens` | Present only on an answer from RouterPlus's cache (Decider 2B, Kev 4B). Equal to `usage.input_tokens`, billed at the cached-input price, $0. |
| `usage.cost` | USD, the full debit for the request after any discount, including any attempt that failed over before this one. `0` on BYOK. Absent on static dev keys. |
| `usage.cost_before_discount` | Present only when a discount applied. What the request costs at the list price. |
| `usage.discount_percent` | Present only when a discount applied. The percent taken off. |

| Header | Value |
|---|---|
| `x-request-id` | The gateway's request id. |
| `x-tm-provider` | `typesafe`, `bespokelabs`, `workers-ai`, `routerplus`, `perplexity-decisions`, `openai-decisions`, `levanto` or `fastino` |
| `x-tm-upstream-model` | The provider's own name for the model, for example `clef` or `clef-flash` from `workers-ai`, `decider-2b` from `routerplus`, `pplx-decider-v1-27b` from `perplexity-decisions`, `gpt-6-luna` from `openai-decisions` |
| `x-tm-attempts` | The attempts made. |
| `x-tm-upstream-status` | The provider's status, when a provider answered. |
| `x-tm-discount-percent` | When a discount applied. |
| `x-tm-dropped-params` | When something was dropped. |

## Billing

A decision is metered like any other request: key limits, the reservation, the ledger and `usage.cost` all work as on the chat routes. Every decision model is priced per million tokens. Each charge rounds down to the micro-dollar.

Decider 2B and Kev 4B are free. The other models charge for input only. On Perplexity Decider and Sage the cost grows with the number of questions as well as with the state.

| Model | Input, per million tokens | Cached input, per million tokens | Output, per million tokens | What one call costs |
|---|---|---|---|---|
| Jev | $0.042 | – | $0 | A three-question call of about 400 input tokens costs about $0.000017. |
| Nimble v4 | $0.04 | – | $0 | A three-question call of 178 input tokens costs $0.000007. Bespoke's own price, with no markup. `bespokelabs/nimble-latest` costs the same: $0.04 is Bespoke's price for `nimble-latest`. |
| Nimble v4 Max | $0.09 | – | $0 | The same call costs $0.000016. Bespoke's own price, with no markup. |
| Clef | $0.24 | – | $0 | A call of 400 input tokens costs $0.000096. Cloudflare's own price, with no markup. |
| Clef-flash | $0.09 | – | $0 | A call of 400 input tokens costs $0.000036. Cloudflare's own price, with no markup. |
| Decider 2B | $0 | $0 | $0 | Nothing. The call is still metered, limited and recorded in `GET /v1/generation`, and it still needs an organization with credit. |
| Kev 4B | $0 | $0 | $0 | Nothing, as on Decider 2B. |
| Perplexity Decider | $0.04 | – | $0 | Five questions about a state of 1,843 tokens bill 9,215 input tokens: $0.000368. Perplexity's own price, with no markup. |
| GPT-6 Luna Decisions | $0.10 | – | $0 | A three-question call (a choice, a score and a yes/no about a short complaint) billed 396 input tokens: $0.000039. OpenAI's own price, with no markup. |
| Sage | $0.05 | – | $0 | Three questions about a short ticket read 204 input tokens: $0.00001. Levanto's own price, with no markup. |
| GLiNER | $0.03 | – | $0 | A call of 1,000 input tokens costs $0.00003. A call of 33 input tokens, as in the example in [GLiNER answers](#gliner-answers), rounds down to $0. |

A dash means the model has no cached-input price.

If a provider answers 200 without a usage object, the gateway bills its own estimate (the amount it reserved) and marks the attempt `estimated`:

- For GLiNER, the estimate comes from the size of the text and from the tasks, labels and extraction keys in the schema.
- For Perplexity Decider and Sage, it counts the state once per question.
- For GPT-6 Luna Decisions, it counts 192 tokens for each question.

These 200s are an `upstream_error` and cost nothing:

- A 200 that does not answer every question.
- A GLiNER 200 whose content is not the object described in [GLiNER answers](#gliner-answers).
- A GPT-6 Luna Decisions 200 that answers a question in another type, or with an option or a level that was not asked.

After a provider answers 200, the gateway never sends the same request to a second provider.

### Discount

A discount may apply to some or all decision models, each at its own percent, from every key, including the playground's. The list price does not change. The discount can change or end, so read it from each response. A discounted model's row in `/api/models.json` carries `discount_percent`, and its page shows the percent beside the struck list price.

- `usage.cost` is what you pay, after the discount.
- `usage.cost_before_discount` is the list price of the request.
- `usage.discount_percent` and the `x-tm-discount-percent` header give the percent.
- When no discount applies, these three are absent.

A discounted charge is the list charge × (100 − percent) / 100, rounded down to the micro-dollar. At 100 % it is 0, and the request is still metered, limited and recorded in `GET /v1/generation`. The discount is for organizations with credit. A balance of $0 is refused with `429 insufficient_quota` whatever the discount. A new account must complete verified browser sign-in and buy credits before its first decision.

On Decider 2B and Kev 4B the list price is already $0, so the discount changes nothing: `usage.cost` and `usage.cost_before_discount` are both `0`. When a discount covers one of them, the percent is still reported, so your code can read one shape for all twelve.

## Limits and failures

These rules hold for every decision model. Each model's section gives its own numbers.

**Pool limits.** Each deployment has a pool with limits on requests a minute, tokens a minute and requests in flight. Each organization may use 75 % of each. Your organization's own limits bind first: by default 300 requests a minute, 1,000,000 tokens a minute and 8 in flight (see [Rate limits & spend caps](/docs/limits)). A burst above any limit is a 429 `rate_limit` from the gateway, before anything reaches the provider. It carries `retry-after` and the `x-tm-limit-*` headers. `x-tm-limit-kind` names the limit: `rpm`, `tpm`, `concurrency`, or `cooldown` for the pause after a relayed provider 429.

**A provider's 429.** The gateway relays it as 429 `rate_limit` with `x-tm-upstream-status: 429`, the provider's words, and its `Retry-After` when it sends one. The gateway then pauses the deployment's pool for that `retry-after`, at least 1 s and at most 60 s. A request inside the pause gets the gateway's own 429 `rate_limit`, with `x-tm-limit-kind: cooldown`, `x-tm-limit-id: pool:…` and no `x-tm-upstream-status`. A relayed 429 never counts against the deployment's health circuit.

**The health circuit.** A provider 5xx is a retryable `upstream_error` that counts against the deployment's health circuit: two in a row open it for 30 s, and in that window every request but one probe gets 503 `gateway_error`, "all deployments cooling down". Retry after a few seconds, with backoff. A 408, a provider's own timeout, is a retryable 504 `upstream_error` that does not count.

**Our account.** A 401, 402 or 403 from any System One provider refuses our account, never yours. You get 503 `model_unavailable`, "out of capacity at the provider". Each refusal counts against the circuit, so after two of them most requests get the 503 `gateway_error` above.

**A provider's refusal.** A 4xx that says the request is wrong is relayed as an `upstream_error` with the provider's words and `x-tm-upstream-status`. It is not tried on a second provider, and it costs nothing.

### Context

| Model | Context | A longer request |
|---|---|---|
| Jev | 64 000 tokens per request through TypeSafe (32 000 for the state plus the longest question) | Refused by TypeSafe |
| Nimble | 262,144 tokens per question's prompt, state included | Refused by Bespoke |
| Clef, Clef-flash | 65,536 tokens per request, as Cloudflare estimates them | Refused with a 413 `context_overflow` |
| Decider 2B | 25,600 tokens | Cut, and only the tokens read are billed |
| Kev 4B | 8,192 tokens | Refused with a 400 |
| Perplexity Decider | 262,144 tokens per question's prompt, state included | Refused with a 400 `context_overflow` |
| GPT-6 Luna Decisions | 272,000 tokens per request at the listed price, up to 1,050,000 at twice the rate | The request body is at most 1 MB |
| Sage | 32 768 tokens per request | Refused by Levanto |
| GLiNER-2.5-Decide | 8,192 tokens | Refused by Fastino |

## Errors

Errors use the gateway's normal shape. See [Errors](/docs/errors).

| Status | `error.code` | When |
|---|---|---|
| 200 | none | GPT-6 Luna Decisions: OpenAI declined one question and answered the rest. That answer is `{"type": "refusal"}`, and the call is billed. |
| 400 | `invalid_request` | The body failed a rule above, such as a question in another model's format or a field a question does not take. The message names the field, for example `questions.team.criteria: at most 255 options`. |
| 400 | `invalid_request` | The request does not fit the model's grammar: `schema` on Jev, Nimble, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider, GPT-6 Luna Decisions, Sage or ScaleDown Classify-1 (the message says the model takes `questions`), or `questions` on GLiNER (the message says the model takes a GLiNER `schema`). |
| 400 | `invalid_request` | GLiNER only: a state that is not text, a flat-list schema, a schema key or classification key GLiNER does not take, a limit above, two tasks with one name, or two results with one name (the message names both). |
| 400 | `invalid_request` | The model is not a decision model. The message names the route that serves it. |
| 400 | `invalid_request` | Perplexity Decider: an object with `"type": "image_url"` anywhere in the state or in a question. The message names where. |
| 400 | `context_overflow` | Perplexity Decider: one question's prompt (the state and that question) is above 262,144 tokens. The message is Perplexity's, prefixed `upstream perplexity-decisions returned 400:`. `metadata.retryable` is `false`: make the state shorter. |
| 400 | `upstream_error` | GPT-6 Luna Decisions: the request is outside OpenAI's limits, for example more than 200 questions, or a `choice` with one option. OpenAI's 400, with `x-tm-upstream-status: 400` and your question id in place of OpenAI's index. For example `upstream openai-decisions returned 400: Invalid 'questions.team.criteria': array too short. Expected an array with minimum length 2, but got an array with length 1 instead.` or `upstream openai-decisions returned 400: Invalid 'questions': array too long. Expected an array with maximum length 200, but got an array with length 201 instead.` It does not count against the circuit. |
| 404 | `model_unavailable` | No decision model has this id. `gpt-6-luna-decisions` without `openai/` is such an id: send `openai/gpt-6-luna-decisions` (`gpt-6-luna` is the chat model, a 400). |
| 404 | `model_unavailable` | GPT-6 Luna Decisions: OpenAI answered 404 `model_not_found`, because our access to the model was withdrawn. It is our account, never yours. |
| 413 | `request_too_large` | The body is larger than 1 MB. |
| 413 | `context_overflow` | Clef and Clef-flash: Cloudflare estimated the request above 65,536 tokens (its 413, code 5021), for example `upstream workers-ai returned 413: The estimated number of input and maximum output tokens (130569) exceeded this model context window limit (65536).` `metadata.retryable` is `false`: make the state shorter. |
| 422 and other 4xx | `upstream_error` | The provider refused the request as wrong. The message has the provider's reason, and `x-tm-upstream-status` its status. Cases by model follow this table. |
| 429 | `rate_limit` | Nimble: more than its pool allows, or Bespoke already runs 8 requests for our account, over all the Nimble models (Bespoke's 429). |
| 429 | `rate_limit` | Clef and Clef-flash: more than their shared pool allows, or Workers AI is busy (Cloudflare's 429 with code 3040). The pool pauses for at most 60 s (1 s when Cloudflare sends no `Retry-After`). Every other Cloudflare 429 is relayed the same way, except code 3036 (a 503). |
| 429 | `rate_limit` | Decider 2B and Kev 4B: more than their shared pool allows (59 in flight at once, 44 per organization), or RouterPlus's 429 at about 200 requests a second. |
| 429 | `rate_limit` | Perplexity Decider: more than its pool allows (a long state with many questions can exceed the token limit on its own, `x-tm-limit-kind: tpm`), or Perplexity's limit of 10 requests a second for our organization. |
| 429 | `rate_limit` | GPT-6 Luna Decisions: more than its pool allows, or OpenAI's limit for our organization (30,000 requests and 180,000,000 tokens a minute). |
| 429 | `rate_limit` | GLiNER: a Fastino 429. |
| 503 | `model_unavailable` | The provider refused our account. The message says the model is out of capacity at the provider. It is our account, never yours. See the cases after this table. |
| 503 | `gateway_error` | "all deployments cooling down", with `retry-after: 5`: the deployment's health circuit is open. While a provider refuses our account, only one request in each 30 s reaches it and gets the 503 `model_unavailable`. A provider's 429 never counts, so Nimble's limit of 8 requests at once, Cloudflare's busy answer on Clef, RouterPlus's limit of about 200 a second, Perplexity's limit of 10 a second and OpenAI's limit of 30,000 a minute stay 429s, and Cloudflare's code 3036 stays a 503 `model_unavailable`. |
| 504 | `upstream_error` | A System One provider answered 408, its own timeout, for example `upstream workers-ai returned 408: Request timeout` on Clef. Retry after a few seconds. A 408 does not count against the circuit. The gateway's own wait for a decision (60 s) also ends in a 504. |
| 5xx (and 529) | `upstream_error` | The provider failed. Retryable. Cases by model follow this table. |
| 401, 402, 403 | `auth` | The provider refused our own account. This is never your key's fault. For GLiNER a 402 or 403, and for every System One model (Jev, Nimble, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider, GPT-6 Luna Decisions, Sage and ScaleDown Classify-1) a 401, 402 or 403, is the 503 above. |

**Provider refusals (422 and other 4xx), by model.** Each of these costs nothing.

- GLiNER. A Fastino 404 (an unknown upstream model id, which is our misconfiguration, never yours) is a 502 `upstream_error`, with `x-tm-upstream-status: 404`.
- Nimble: a request above Nimble's own limits (more than 64 questions, a `choice` with one option, or a question's prompt with the state above 262,144 tokens on Nimble v4, Nimble v4 Max or `bespokelabs/nimble-latest`) is Bespoke's 422, relayed with Bespoke's words.
- Clef and Clef-flash: a request outside Clef's own limits (more than 64 questions, a question id that does not match `^[A-Za-z0-9_.-]{1,100}$`, or a `choice` with one option) is Cloudflare's 422 (code 5012) or 400 (code 5006), relayed with Cloudflare's words, for example `upstream workers-ai returned 422: Request body failed validation: questions: Dictionary should have at most 64 items after validation, not 65`.
- Decider 2B and Kev 4B: a request that RouterPlus refuses is RouterPlus's 400, relayed here with RouterPlus's reason and `x-tm-upstream-status: 400`. RouterPlus refuses a malformed question (`invalid questions`, then the reason for each question id, for example `score needs criteria: a list of 2 to 10 levels`) and a Kev 4B request above 8,192 tokens (the question's id, then `ContextOverflow: branch too long`).
- Perplexity Decider: a request with more than 128 questions is Perplexity's 400, relayed with Perplexity's words (`Each request needs between 1 and 128 questions`).

**Refusals of our account (503 `model_unavailable`), by model:**

- Sage: Levanto refused our account (its 401, 402 or 403, for example when our month's usage is used up).
- GLiNER: Fastino refused our account for credit or billing (its 402 or 403).
- Nimble: Bespoke refused our account for credit (its 402). `metadata.retryable` is `false`.
- Clef and Clef-flash: Cloudflare refused our account, for our token or plan (its 401 or 403), or because the account's free daily allocation is used up (its 429 with code 3036, until 00:00 UTC). `metadata.retryable` is `false`.
- A 401, 402 or 403 from any System One provider is handled the same way. That covers TypeSafe (Jev), RouterPlus (Decider 2B and Kev 4B), Perplexity (Perplexity Decider) and OpenAI (GPT-6 Luna Decisions).

On a connection with your own provider key, that provider's 401, 402 or 403 is your account, and it keeps its status. Other models are not affected.

**Provider failures (5xx and 529), by model.** Each 5xx counts against the circuit, as in [Limits and failures](#limits-and-failures).

- Sage: Levanto answers 503 while Sage loads. Retry after a few seconds.
- GLiNER: Fastino answers 425 or 503 while GLiNER starts (a cold start). A cold start can also outlast the gateway's wait for a decision (60 s, a 504). Both are a retryable `upstream_error`, so retry after a few seconds.
- Nimble, Clef, Decider 2B and Kev 4B, Perplexity Decider and GPT-6 Luna Decisions: see the Busy row in each model's section. A Perplexity 504 is an HTML page, so the message has no words of Perplexity's.

## Examples

The examples at the top of this page use Jev. The examples below send the same System One body to other models, then a GLiNER schema.

### Nimble

Send the Jev body under a Nimble model's id, or `bespokelabs/nimble-latest` to let Bespoke pick. Keep to 1 to 64 questions, and 2 or more options in a choice.

```bash
curl https://demo-api.routerplus.com/v1/decisions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"bespokelabs/nimble-v4",
       "state":"Help! My payouts have been failing for 3 days.",
       "questions":{"urgent":{"type":"noul","instructions":"Is this urgent?"}}}'
```

Python:

```python
import os, httpx

r = httpx.post(
    "https://demo-api.routerplus.com/v1/decisions",
    headers={"Authorization": f"Bearer {os.environ['TM_API_KEY']}"},
    json={
        "model": "bespokelabs/nimble-v4",
        "state": "Help! My payouts have been failing for 3 days.",
        "questions": {
            "team": {"type": "choice", "instructions": "Which team?",
                     "criteria": {"billing": "Payments", "technical": "Bugs"}},
            "urgent": {"type": "noul", "instructions": "Is this urgent?"},
        },
    },
)
answers = r.json()["answers"]
print(answers["team"]["choice"], answers["urgent"]["noul"])  # e.g. billing 0.997817283712868
```

Node:

```js
const r = await fetch("https://demo-api.routerplus.com/v1/decisions", {
  method: "POST",
  headers: { authorization: `Bearer ${process.env.TM_API_KEY}`, "content-type": "application/json" },
  body: JSON.stringify({
    model: "bespokelabs/nimble-v4",
    state: { ticket: "Help! My payouts have been failing for 3 days." },
    questions: { urgent: { type: "noul", instructions: "Is `ticket` urgent?" } },
  }),
});
const { answers, usage } = await r.json();
console.log(answers.urgent.noul, usage.input_tokens, usage.cost);
```

### Clef

Send the Jev body under Clef's or Clef-flash's id, within [Clef's limits](#clef-and-clef-flash). The Node example uses Clef-flash.

```bash
curl https://demo-api.routerplus.com/v1/decisions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"cloudflare/clef",
       "state":"Checkout has been failing for every customer for the last hour.",
       "questions":{"urgent":{"type":"noul","instructions":"Is this support request urgent?"}}}'
```

Python:

```python
import os, httpx

r = httpx.post(
    "https://demo-api.routerplus.com/v1/decisions",
    headers={"Authorization": f"Bearer {os.environ['TM_API_KEY']}"},
    json={
        "model": "cloudflare/clef",
        "state": "Checkout has been failing for every customer for the last hour.",
        "questions": {
            "urgent": {"type": "noul", "instructions": "Is this support request urgent?"},
            "team": {"type": "choice", "instructions": "Which team should handle this request?",
                     "criteria": {"billing": "Payments and invoices", "technical": "Bugs and outages",
                                  "sales": "Plans and upgrades"}},
            "severity": {"type": "score", "instructions": "How severe is the customer impact?",
                         "criteria": ["No impact", "Minor", "Major", "Critical"]},
        },
    },
)
answers = r.json()["answers"]
team = answers["team"]
print(team["choice"], team["probabilities"][team["choice"]], team["confidence"])  # e.g. technical 0.8274 0.5654
```

Node:

```js
const r = await fetch("https://demo-api.routerplus.com/v1/decisions", {
  method: "POST",
  headers: { authorization: `Bearer ${process.env.TM_API_KEY}`, "content-type": "application/json" },
  body: JSON.stringify({
    model: "cloudflare/clef-flash",
    state: { ticket: "Checkout has been failing for every customer for the last hour." },
    questions: { urgent: { type: "noul", instructions: "Is `ticket` urgent?" } },
  }),
});
const { answers, usage } = await r.json();
console.log(answers.urgent.noul, usage.input_tokens, usage.output_tokens); // output_tokens is always 0
```

### RouterPlus models

Send the Jev body under Decider 2B's or Kev 4B's id. A cold start can take 15 to 20 s, so give your client a timeout above 60 s, the gateway's own wait.

```bash
curl https://demo-api.routerplus.com/v1/decisions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"routerplus/decider-2b",
       "state":"Help! My payouts have been failing for 3 days.",
       "questions":{"urgent":{"type":"noul","instructions":"Is this urgent?"}}}'
```

Python:

```python
import os, httpx

r = httpx.post(
    "https://demo-api.routerplus.com/v1/decisions",
    headers={"Authorization": f"Bearer {os.environ['TM_API_KEY']}"},
    json={
        "model": "routerplus/decider-2b",
        "state": "Help! My payouts have been failing for 3 days.",
        "questions": {
            "team": {"type": "choice", "instructions": "Which team?",
                     "criteria": {"billing": "Payments", "technical": "Bugs"}},
            "urgent": {"type": "noul", "instructions": "Is this urgent?"},
        },
    },
    timeout=70,
)
answers = r.json()["answers"]
print(answers["team"]["choice"], answers["urgent"]["noul"], r.headers["x-tm-upstream-model"])  # e.g. billing 0.97 decider-2b
```

Node:

```js
const r = await fetch("https://demo-api.routerplus.com/v1/decisions", {
  method: "POST",
  headers: { authorization: `Bearer ${process.env.TM_API_KEY}`, "content-type": "application/json" },
  body: JSON.stringify({
    model: "routerplus/decider-2b",
    state: { ticket: "Help! My payouts have been failing for 3 days." },
    questions: { urgent: { type: "noul", instructions: "Is `ticket` urgent?" } },
  }),
  signal: AbortSignal.timeout(70_000),
});
const { answers, usage } = await r.json();
console.log(answers.urgent.noul, usage.input_tokens, usage.cost);
```

### Perplexity

Send the Jev body under Perplexity Decider's id, with 1 to 128 questions. Each question is billed with the whole state, so ask only the questions you need.

```bash
curl https://demo-api.routerplus.com/v1/decisions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"perplexity/pplx-decider-v1-27b",
       "state":"Help! My payouts have been failing for 3 days.",
       "questions":{"urgent":{"type":"noul","instructions":"Is this urgent?"}}}'
```

Python:

```python
import os, httpx

r = httpx.post(
    "https://demo-api.routerplus.com/v1/decisions",
    headers={"Authorization": f"Bearer {os.environ['TM_API_KEY']}"},
    json={
        "model": "perplexity/pplx-decider-v1-27b",
        "state": "Help! My payouts have been failing for 3 days.",
        "questions": {"urgent": {"type": "noul", "instructions": "Is this urgent?"}},
    },
)
body = r.json()
print(body["answers"]["urgent"]["noul"], body["usage"]["input_tokens"])  # e.g. 0.9297849172276498 95
```

Node:

```js
const r = await fetch("https://demo-api.routerplus.com/v1/decisions", {
  method: "POST",
  headers: { authorization: `Bearer ${process.env.TM_API_KEY}`, "content-type": "application/json" },
  body: JSON.stringify({
    model: "perplexity/pplx-decider-v1-27b",
    state: { ticket: "Help! My payouts have been failing for 3 days." },
    questions: {
      team: { type: "choice", instructions: "Which team should handle `ticket`?", criteria: { billing: "Payments", technical: "Bugs" } },
      urgent: { type: "noul", instructions: "Is `ticket` urgent?" },
    },
  }),
});
const { answers, usage } = await r.json();
// Two questions: Perplexity bills the state twice.
console.log(answers.team.choice, answers.urgent.noul, usage.input_tokens, usage.cost);
```

### GPT-6 Luna Decisions

Send the Jev body under GPT-6 Luna Decisions' id, with 1 to 200 questions and 2 or more options in a choice. Check each answer's `type`, because OpenAI can decline one question.

```bash
curl https://demo-api.routerplus.com/v1/decisions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"openai/gpt-6-luna-decisions",
       "state":"Help! My payouts have been failing for 3 days.",
       "questions":{"urgent":{"type":"noul","instructions":"Is this urgent?"}}}'
```

Python:

```python
import os, httpx

r = httpx.post(
    "https://demo-api.routerplus.com/v1/decisions",
    headers={"Authorization": f"Bearer {os.environ['TM_API_KEY']}"},
    json={
        "model": "openai/gpt-6-luna-decisions",
        "state": {"ticket": "Help! My payouts have been failing for 3 days."},
        "questions": {
            "team": {"type": "choice", "instructions": "Which team should handle `ticket`?",
                     "criteria": {"billing": "Payments, invoices, refunds",
                                  "technical": "Bugs, outages, integrations", "sales": None}},
            "frustration": {"type": "score", "instructions": "How frustrated is the customer?",
                            "criteria": ["Calm", "Annoyed", "Angry"]},
            "urgent": {"type": "noul", "instructions": "Is this urgent?",
                       "criteria": {"true": "Explicitly time-sensitive", "false": "No urgency expressed"}},
            "business": {"type": "noul", "instructions": "Is the customer a business?"},
        },
    },
)
for qid, answer in r.json()["answers"].items():
    if answer["type"] == "refusal":
        print(qid, "declined by OpenAI")  # billed, but no value
    else:
        print(qid, answer)
```

Node:

```js
const r = await fetch("https://demo-api.routerplus.com/v1/decisions", {
  method: "POST",
  headers: { authorization: `Bearer ${process.env.TM_API_KEY}`, "content-type": "application/json" },
  body: JSON.stringify({
    model: "openai/gpt-6-luna-decisions",
    state: { ticket: "Help! My payouts have been failing for 3 days." },
    questions: {
      team: { type: "choice", instructions: "Which team should handle `ticket`?", criteria: { billing: "Payments", technical: "Bugs" } },
      urgent: { type: "noul", instructions: "Is `ticket` urgent?" },
    },
  }),
});
const { answers, usage } = await r.json();
// A declined question comes back as { type: "refusal" }, with no value.
for (const [id, a] of Object.entries(answers)) console.log(id, a.type === "refusal" ? "declined" : a);
// Two questions: about 150 input tokens each, on top of the state.
console.log(usage.input_tokens, usage.output_tokens, usage.cost); // output_tokens is always 0
```

A sample answer to the Python request:

```json
{
  "id": "8c1f4b2e-7d3a-4e90-a6b5-1f2e3d4c5b6a",
  "object": "decision",
  "model": "openai/gpt-6-luna-decisions",
  "answers": {
    "team": { "type": "choice", "choice": "billing",
              "probabilities": { "billing": 1, "technical": 0, "sales": 0 }, "confidence": 1 },
    "frustration": { "type": "score", "score": 1.02,
                     "legend": { "0": "Calm", "1": "Annoyed", "2": "Angry" },
                     "probabilities": { "0": 0.03, "1": 0.92, "2": 0.05 }, "confidence": 0.88 },
    "urgent": { "type": "noul", "noul": 0.96 },
    "business": { "type": "noul", "noul": 0.49 }
  },
  "usage": { "input_tokens": 561, "output_tokens": 0, "cost": 0.000056 }
}
```

### Sage

Send the Jev body under Sage's id. Each question is billed with the whole state, so ask only the questions you need.

```bash
curl https://demo-api.routerplus.com/v1/decisions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"levanto/sage-1.2",
       "state":"Help! My payouts have been failing for 3 days.",
       "questions":{"urgent":{"type":"noul","instructions":"Is this urgent?"}}}'
```

### GLiNER

Each answer sits under its task name.

```bash
curl https://demo-api.routerplus.com/v1/decisions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"fastino/gliner-2.5-decide",
       "state":"Help! My payouts have been failing for 3 days.",
       "schema":{"classifications":[{"task":"Is this urgent?","labels":["yes","no"]}]}}'
```

Python:

```python
import os, httpx

r = httpx.post(
    "https://demo-api.routerplus.com/v1/decisions",
    headers={"Authorization": f"Bearer {os.environ['TM_API_KEY']}"},
    json={
        "model": "fastino/gliner-2.5-decide",
        "state": "Help! My payouts have been failing for 3 days.",
        "schema": {"classifications": [{"task": "team",
                                        "labels": ["billing: payments", "technical: bugs"]}]},
    },
)
answer = r.json()["answers"]["team"]
print(answer["label"], answer["confidence"])  # a confidence, not a calibrated probability
```

Node:

```js
const r = await fetch("https://demo-api.routerplus.com/v1/decisions", {
  method: "POST",
  headers: { authorization: `Bearer ${process.env.TM_API_KEY}`, "content-type": "application/json" },
  body: JSON.stringify({
    model: "fastino/gliner-2.5-decide",
    state: "Help! My payouts have been failing for 3 days.",
    schema: {
      classifications: [
        { task: "topics", labels: ["billing", "outage", "fraud"], multi_label: true, cls_threshold: 0 },
      ],
    },
  }),
});
const { answers, usage } = await r.json();
console.log(answers.topics, usage.input_tokens, usage.output_tokens, usage.cost);
```

## Model discovery

```bash
curl -s "https://demo-api.routerplus.com/v1/models?output_modalities=decisions" \
  -H "Authorization: Bearer $TM_API_KEY"
```

Decision models carry `architecture.output_modalities: ["decisions"]` in `GET /v1/models`, and `output_modalities: ["decisions"]` in the public feed `https://demo.routerplus.com/api/models.json`. The Anthropic shape of `GET /v1/models` never lists them.

---

Source: https://demo.routerplus.com/docs/api-models.md

# GET /v1/models

`GET /v1/models` lists every model id RouterPlus can serve on your key, one row per id.
`GET /v1/models/<id>` returns one row of that list. SDK `models.list()` and `models.retrieve()`
calls resolve to these endpoints, so both answer in two shapes:

| Request | Response shape |
|---|---|
| No `anthropic-version` header | OpenAI |
| `anthropic-version` header present (any value) | Anthropic |

The Anthropic SDK sends `anthropic-version` on every request, so each SDK gets its own shape
with no configuration.

Send your key as `Authorization: Bearer $TM_API_KEY` or as `x-api-key: $TM_API_KEY`. A missing
or invalid key is a 401 with `error_type: "auth"` (see [Errors](/docs/errors)).

## List models: OpenAI shape (default)

```bash
curl -s https://demo-api.routerplus.com/v1/models \
  -H "Authorization: Bearer $TM_API_KEY"
```

```json
{
  "object": "list",
  "data": [
    { "id": "claude-haiku-4-5", "object": "model", "created": 1756900000, "owned_by": "anthropic",
      "context_length": 200000,
      "architecture": { "input_modalities": ["text", "image", "file"], "output_modalities": ["text"] } },
    { "id": "gpt-image-1", "object": "model", "created": 1756900000, "owned_by": "openai",
      "architecture": { "input_modalities": ["text"], "output_modalities": ["image"] } },
    { "id": "typesafe/jev-1.13", "object": "model", "created": 1756900000, "owned_by": "typesafe",
      "architecture": { "input_modalities": ["text"], "output_modalities": ["decisions"] } }
  ]
}
```

| Field | Type | Meaning |
|---|---|---|
| `id` | string | The exact string for the `model` field of your request. |
| `object` | `"model"` | Constant. |
| `created` | integer | Unix seconds. See the note under [Anthropic shape](#anthropic-shape). |
| `owned_by` | string | The provider id of the first deployment routing would try for this model. `routerplus` for a dedicated endpoint. |
| `context_length` | integer | Chat models only. The largest context window, in tokens, among the routes of your key for this model. Absent when no route declares a window. |
| `architecture.input_modalities` | array | What a message to the model may carry: `"text"`, plus each of `"image"`, `"file"`, `"audio"` and `"video"` that a route of this key takes. A chat request with a part that no route takes is a 400 (see [Wire compatibility](/docs/compat#content-is-never-silently-dropped)). Image and decision models take `["text"]`. A route on your own provider key declares no inputs, so it adds none here. RouterPlus sends it every part as you sent it. |
| `architecture.output_modalities` | array | `["text"]`, `["image"]`, `["video"]`, `["decisions"]`, `["embeddings"]`, `["transcription"]` or `["audio"]`. A transcription model takes `["audio"]` and a speech model takes `["text"]`. |

OpenAI SDKs ignore `context_length` and `architecture`. Clients that size their context window
or pick a route by modality read them.

## What the list contains

- A key bound to your own provider connection lists the models of that connection.
- A key under a routing policy lists the models of the policy.
- Your organization's active [dedicated endpoints](/docs/dedicated-endpoints)
  (`<your-namespace>/<name>`) are listed with `owned_by` `routerplus`, on the base URL they are
  served from.
- Deployed endpoints from [Optimize](/docs/model-search) (`tm/<name>-v<n>`) are not listed.
- The OpenAI shape lists the `rp-` name of each Claude model (`rp-sonnet-5-5` for
  `claude-sonnet-5-5`) right after its model. That row has the same `owned_by`,
  `context_length` and `architecture`, and a `canonical_slug` that names the model. Model
  pickers that read this list, such as Cursor's, show these names. See
  [Claude names for Cursor](/docs/models#claude-names-for-cursor). The Anthropic shape does
  not list them.

## Filtering

```bash
curl -s "https://demo-api.routerplus.com/v1/models?output_modalities=image" \
  -H "Authorization: Bearer $TM_API_KEY"
```

`output_modalities` takes a comma-separated list of these values:

| Value | Models for |
|---|---|
| `text` | Chat |
| `image` | Image generation |
| `video` | Video generation |
| `decisions` | `POST /v1/decisions` |
| `embeddings` | `POST /v1/embeddings` |
| `transcription` | `POST /v1/audio/transcriptions` |
| `audio` | `POST /v1/audio/speech` |

- The list is a filter. `?output_modalities=text,image,video,decisions,embeddings,transcription,audio`
  is the same as no filter.
- A kind with no model in the catalog gives an empty list.
- Any other value is a 400 `invalid_request` ("invalid output_modalities filter").
- The Anthropic shape lists chat models only. The Anthropic Messages API has no route for image
  or decision models.

A `provider` query parameter takes the same JSON object as the request-level `provider`
routing controls, for example `{"only":["openai"]}`. The response lists the models those
controls let through. See [Routing policies](/docs/routing-policies).

## Anthropic shape

```bash
curl -s https://demo-api.routerplus.com/v1/models \
  -H "x-api-key: $TM_API_KEY" \
  -H "anthropic-version: 2023-06-01"
```

```json
{
  "data": [
    { "type": "model", "id": "claude-haiku-4-5", "display_name": "claude-haiku-4-5", "created_at": "2026-09-03T12:00:00.000Z" }
  ],
  "has_more": false,
  "first_id": "claude-haiku-4-5",
  "last_id": "gpt-4o-mini"
}
```

`has_more` is always `false`. The full list arrives in one response, with no pagination.

> [!NOTE]
> `created` and `created_at` are the time the gateway process loaded its catalog. They are not
> the release date of the model, so do not build logic on them. For prices and retention
> policy, use the public [`/api/models.json`](https://demo.routerplus.com/api/models.json), documented in
> [Models & catalog](/docs/models).

## Retrieve one model

`GET /v1/models/<id>` returns the row that `GET /v1/models` gives your key for that id, in the
same shape rules. An id with a slash, such as `typesafe/jev-1.13`, works as it is or
URL-encoded.

```bash
curl -s https://demo-api.routerplus.com/v1/models/claude-haiku-4-5 \
  -H "Authorization: Bearer $TM_API_KEY"
```

```json
{ "id": "claude-haiku-4-5", "object": "model", "created": 1756900000, "owned_by": "anthropic",
  "context_length": 200000,
  "architecture": { "input_modalities": ["text", "image", "file"], "output_modalities": ["text"] } }
```

With `anthropic-version`, the answer is one Anthropic row, for example
`{"type": "model", "id": "claude-haiku-4-5", "display_name": "claude-haiku-4-5", "created_at": "2026-09-03T12:00:00.000Z"}`.

An id that the list does not give your key is a 404 `model_unavailable`, with the same body as
in [Unknown models](#unknown-models-on-completion-calls). In the Anthropic shape, this includes
every model that is not a chat model.

`GET /v1/models/<id>/endpoints` is a separate route. It lists each provider of one model with
its price, data policy, status and measurements. See [Provider routing](/docs/provider-routing).

## SDK usage

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])
for m in client.models.list():
    print(m.id, m.owned_by)
```

```python
import os
import anthropic

# The SDK sends anthropic-version itself, so this gets the Anthropic shape.
client = anthropic.Anthropic(base_url="https://demo-api.routerplus.com", api_key=os.environ["TM_API_KEY"])
for m in client.models.list():
    print(m.id)
```

```typescript
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://demo-api.routerplus.com/v1", apiKey: process.env.TM_API_KEY });
for await (const m of client.models.list()) {
  console.log(m.id, m.owned_by);
}
```

## Unknown models on completion calls

A completion request (`/v1/chat/completions` or `/v1/messages`) for an id that is not in this
list fails with a 404. The error echoes the string you sent. RouterPlus contacts no provider
and bills nothing. One exception: a dated pin of a listed model (`claude-haiku-4-5-20251001`,
`gpt-4o-2024-08-06`, `-latest`) routes as its family. See
[Models & catalog](/docs/models#date-pinned-ids).

OpenAI surface:

```json
{
  "error": {
    "code": "model_unavailable",
    "message": "model \"gpt-5o\" is not in the catalog; GET /v1/models lists what this key can serve",
    "type": "invalid_request_error",
    "metadata": { "error_type": "model_unavailable" }
  },
  "request_id": "6f3c…"
}
```

Anthropic surface:

```json
{
  "type": "error",
  "error": {
    "type": "not_found_error",
    "message": "model \"gpt-5o\" is not in the catalog; GET /v1/models lists what this key can serve",
    "error_type": "model_unavailable"
  },
  "request_id": "6f3c…"
}
```

Both carry the `x-tm-error-code: model_unavailable` header and `x-request-id`.

> [!WARNING]
> RouterPlus does no fuzzy matching and never falls back to a similar model. A typo in the
> model id is a 404. Every unknown id gets the same 404, so the error does not reveal which
> ids exist.

## Errors

| Status | `error_type` | Meaning |
|---|---|---|
| 404 | `model_unavailable` | The id is not in the catalog. Fix the id. |
| 503 | `gateway_error` | The id is in the catalog, but every deployment serving it is cooling down after failures. Comes with `retry-after: 5` and `x-tm-limit-kind: health`. Retry the request. |
| 400 | `model_not_priced` | Billed key, and the ledger has no price row for this model. RouterPlus does not serve a model it cannot meter. |
| 400 | `invalid_request` | A listed image model sent to a chat route. The message names `POST /v1/images/generations`. A chat model sent to the images route gets the same 400, pointing at the chat routes. See [POST /v1/images/generations](/docs/api-images). |

[Errors](/docs/errors) has the full list.

---

Source: https://demo.routerplus.com/docs/api-usage.md

# GET /v1/usage & /v1/generation

Two read-only gateway endpoints report your spend. `GET /v1/usage` returns your balance and
recent attempts. `GET /v1/generation` returns the cost of one request, attempt by attempt. Both
read the ledger that your balance comes from, so they match your bill.

Both return **metadata only**: token counts, outcomes and costs. They never return prompts or
completions. See the [Data policy](/docs/data-policy).

In the console, open Usage or Logs for the same data across your organization.

## Authentication

Both endpoints need a billed key: the first key shown after verified sign-in, or one created in
the console. Results cover that key's org, workspace and principal. Send the key in either
header:

```bash
curl -s https://demo-api.routerplus.com/v1/usage -H "Authorization: Bearer $TM_API_KEY"
curl -s https://demo-api.routerplus.com/v1/usage -H "x-api-key: $TM_API_KEY"
```

> [!NOTE]
> A static development key (a gateway running without the money plane) has no
> ledger to read. On these routes it gets the same 404 as any unknown path.

## GET /v1/usage

Returns the 20 most recent attempts made with the keys of your workspace and principal, newest
first.

```bash
curl -s https://demo-api.routerplus.com/v1/usage -H "Authorization: Bearer $TM_API_KEY" | jq
```

```json
{
  "balance_usd": 0.98971,
  "credited_usd": 1.0,
  "spent_usd": 0.01029,
  "recent_attempts": [
    {
      "request_id": "0b6f6a4e-6a01-4c5e-9d5a-1c9f2b7e8a33",
      "deployment": "provider-b",
      "model": "claude-sonnet-4-5",
      "outcome": "completed",
      "usage_provenance": "observed",
      "input_tokens": 1200,
      "output_tokens": 350,
      "cost_usd": 0.00669,
      "billing_source": "house",
      "inference_cost_usd": 0.00669,
      "inference_price_source": "platform_list",
      "price_snapshot_id": "5d2c…",
      "fee_policy_version": "house-pass-through-v1",
      "at": "2026-09-04T09:12:33.104Z"
    },
    {
      "request_id": "0b6f6a4e-6a01-4c5e-9d5a-1c9f2b7e8a33",
      "deployment": "provider-a",
      "model": "claude-sonnet-4-5",
      "outcome": "failed",
      "usage_provenance": "observed",
      "input_tokens": 1200,
      "output_tokens": 0,
      "cost_usd": 0.0036,
      "billing_source": "house",
      "inference_cost_usd": 0.0036,
      "inference_price_source": "platform_list",
      "price_snapshot_id": "5d2c…",
      "fee_policy_version": "house-pass-through-v1",
      "at": "2026-09-04T09:12:31.512Z"
    }
  ]
}
```

The org balance fields are filled for a key in the organization's default workspace and default
principal. Every first key shown after verified sign-in, and every key made in the console, is
one. A key bound to an explicit workspace or principal gets `null` (see
[Workspaces and principals](/docs/identity)).

| Field | Type | Meaning |
|---|---|---|
| `balance_usd` | number \| null | `credited_usd − spent_usd`, computed from ledger rows on each read. Never a cached counter. |
| `credited_usd` | number \| null | The sum of all credit grants to the org. |
| `spent_usd` | number \| null | The sum of every settled attempt. |
| `recent_attempts[].request_id` | string | The request's `x-request-id`. Pass it to `/v1/generation`. |
| `recent_attempts[].deployment` | string | The deployment that served the attempt. It matches the `x-tm-provider` response header. It is `routerplus` on a closed [dedicated endpoint](/docs/dedicated-endpoints#closed-endpoints) and on any endpoint's dedicated capacity. |
| `recent_attempts[].model` | string | The model id you requested (for an `rp-` name, the Claude model's own id). On a closed dedicated endpoint, the endpoint id. |
| `recent_attempts[].outcome` | string | See [Attempt outcomes](#attempt-outcomes). |
| `recent_attempts[].usage_provenance` | string \| null | `observed`, `estimated` or `unknown`. `null` before settlement. |
| `recent_attempts[].input_tokens` / `output_tokens` | number | Provider-reported counts. Input includes cache tokens. On a model billed at the provider's reported cost, `output_tokens` is the charge in micro-dollars. |
| `recent_attempts[].cost_usd` | number \| null | The settled cost, any web search fee included. `null` until settlement is written (in batches, about every 100 ms). |
| `recent_attempts[].server_tool_use` | object \| null | The web searches and web fetches the provider ran for the model: `web_search_requests`, `web_fetch_requests`. `null` on attempts from before these were recorded. |
| `recent_attempts[].server_tool_fee_usd` | number \| null | The part of `cost_usd` that is web search fees (see [Pricing & billing](/docs/pricing#web-search)). |
| `recent_attempts[].at` | string | The dispatch time. |

Each row is one **attempt**. A request that failed over has one row per dispatch, all with the
same `request_id`.

### Accounting fields

Both endpoints return these fields on each attempt:

| Field | Meaning |
|---|---|
| `billing_source` | `house` for inference paid from RouterPlus credits. `byok` for inference on your own provider connection. |
| `inference_cost_usd` | The cost from the pinned rates and the usage. `null` when either is unknown. |
| `inference_price_source` | `platform_list`, `customer_rate`, `unknown`, or `legacy_platform_list` for old records. |
| `price_snapshot_id` | The id of the pinned rate snapshot. `null` when unknown or not kept for old records. |
| `fee_policy_version` | The pricing policy for this attempt: `house-pass-through-v1` or `byok-free-v1`. |

- `cost_usd` is the RouterPlus charge. Spend caps apply to it.
- `inference_cost_usd` is a separate cost estimate, not a provider invoice. A list-price or
  estimated-usage figure can differ from your provider bill.
- BYOK attempts have a RouterPlus charge of zero and an unknown (`null`) inference cost. See
  [Bring your own key](/docs/byok) and [routing policies](/docs/routing-policies).

## GET /v1/generation?id=

Returns the full per-attempt audit for one request.

| Parameter | In | Required | Meaning |
|---|---|---|---|
| `id` | query | yes | The request id: the `x-request-id` header on every response, or `request_id` in `/v1/usage` rows. |

- An `id` that is not a UUID returns `404 not_found`.
- An unknown id, or one from another org, workspace or principal, returns `404` with
  `"attempts": []` and `"admission_events": []`.

This example is a request that failed over once. The deployment ids are examples. Yours match
your `x-tm-provider` headers.

```bash
curl -s "https://demo-api.routerplus.com/v1/generation?id=0b6f6a4e-6a01-4c5e-9d5a-1c9f2b7e8a33" \
  -H "Authorization: Bearer $TM_API_KEY" | jq
```

```json
{
  "request_id": "0b6f6a4e-6a01-4c5e-9d5a-1c9f2b7e8a33",
  "admission_events": [],
  "attempts": [
    {
      "attempt": 1,
      "deployment": "provider-a",
      "model": "claude-sonnet-4-5",
      "outcome": "failed",
      "usage_provenance": "observed",
      "tokens": { "input": 1200, "cached": 0, "cache_write": 0, "output": 0, "reasoning": 0 },
      "reserved_max_usd": 0.009,
      "cost_usd": 0.0036,
      "billing_source": "house",
      "inference_cost_usd": 0.0036,
      "inference_price_source": "platform_list",
      "price_snapshot_id": "5d2c…",
      "fee_policy_version": "house-pass-through-v1",
      "error_code": "upstream_error",
      "error_origin": "upstream",
      "admission_context": { "…": "…" },
      "route_context": { "…": "…" },
      "dropped_params": [],
      "dispatched_at": "2026-09-04T09:12:31.512Z"
    },
    {
      "attempt": 2,
      "deployment": "provider-b",
      "model": "claude-sonnet-4-5",
      "outcome": "completed",
      "usage_provenance": "observed",
      "tokens": { "input": 1200, "cached": 800, "cache_write": 0, "output": 350, "reasoning": 0 },
      "reserved_max_usd": 0.009,
      "cost_usd": 0.00669,
      "billing_source": "house",
      "inference_cost_usd": 0.00669,
      "inference_price_source": "platform_list",
      "price_snapshot_id": "5d2c…",
      "fee_policy_version": "house-pass-through-v1",
      "error_code": "",
      "error_origin": null,
      "admission_context": { "…": "…" },
      "route_context": { "…": "…" },
      "dropped_params": [],
      "dispatched_at": "2026-09-04T09:12:33.104Z"
    }
  ]
}
```

| Field | Type | Meaning |
|---|---|---|
| `attempt` | number | The dispatch number, starting at 1. |
| `deployment` | string | The deployment that served this attempt. `routerplus` on a closed dedicated endpoint. |
| `outcome` | string | See [Attempt outcomes](#attempt-outcomes). |
| `usage_provenance` | string | `observed` (provider-reported tokens, or a completed attempt), `unknown` (no usage seen), or `estimated`: the provider reported nothing and the gateway priced the attempt itself. On [`/v1/images/generations`](/docs/api-images), a 200 without usage bills `n` × the model's per-image ceiling, the amount the reservation held. On a chat attempt cancelled or cut off before usage arrived, it bills the input (the gateway's estimate, when the provider sent no count) plus the output streamed to you. |
| `tokens` | object | `input` (includes cache tokens), `cached` (reads), `cache_write`, `output`, `reasoning` (part of output). On a model billed at the provider's reported cost, `output` is the charge in micro-dollars. |
| `reserved_max_usd` | number | The worst-case hold taken **before** dispatch (ceiling math). |
| `cost_usd` | number \| null | What settled (floor math), any web search fee included. `null` only before settlement. |
| `server_tool_use` | object \| null | `web_search_requests` and `web_fetch_requests`: the server-tool calls the provider ran for the model. `null` on attempts from before these were recorded. |
| `server_tool_fee_usd` | number \| null | The part of `cost_usd` that is web search fees. |
| `error_code` | string | The error class on failure. Empty on success. |
| `error_origin` | string \| null | Where a failure came from: `upstream`, `gateway_admission`, `upstream_quota`, `gateway_infrastructure` or `authorization`. `null` on success. |
| `admission_context` | object \| null | The limits this attempt claimed, recorded at dispatch. See [Limits and capacity](/docs/admission). |
| `route_context` | object \| null | The route plan behind this attempt: plan id, the deployments planned and excluded, and the policy revisions in force. On a dedicated endpoint, `dedicated` (the endpoint, its revision and the role). |
| `dropped_params` | array | The parameter paths the gateway dropped for this deployment, with the same names as the `x-tm-dropped-params` header. |

`admission_events` lists the limit or balance refusals recorded for this request before any
attempt (`origin`, `scope`, `scope_id`, `limit_kind`, `reason`, `status`, `occurred_at`). It is
empty when nothing was refused.

On a closed [dedicated endpoint](/docs/dedicated-endpoints#closed-endpoints):

- `deployment` and `provider` are `routerplus`, and `model` is the endpoint id.
- `route_context` keeps only `dedicated`, `admission_context` is null and `dropped_params` is
  empty.
- `admission_events` carry no pool `scope_id`.

On any dedicated endpoint, its dedicated capacity reads as `routerplus`.

`reserved_max_usd` is the hold, released in full at settlement. `cost_usd` is the settled
amount, from provider-reported usage, rounded down. See [Pricing & billing](/docs/pricing).

### Attempt outcomes

| `outcome` | Meaning | Billing |
|---|---|---|
| `completed` | The full response was delivered. | Settled on reported usage. |
| `failed` | The attempt failed. A later attempt may have served you. | Settled on any usage the provider reported before it failed (often 0). |
| `cancelled` | You disconnected mid-request. | Settled on the usage the provider reported up to the disconnect. When it reported none: the input plus the output streamed to you before the disconnect, never the hold. If you disconnected before the attempt left the gateway, it settles at zero with provenance `observed`. On `/v1/images/generations` the render is never aborted: it finishes upstream and the attempt settles the provider's reported usage. |
| `incomplete` | The provider failed after your stream started. You received a terminal error event. | Settled on usage observed so far. When there was none: the input plus the output streamed to you. |
| `unknown_after_crash` | The gateway stopped mid-attempt and could not observe the outcome. The attempt is recovered at restart from the gateway's ledger journal, or by the operator reconciler. | Settles **at the reservation**, because the provider may have billed the call. |

> [!WARNING]
> You are debited for **every settled attempt** of a request, including ones
> discarded by failover before your answer started. A provider that fails after reading
> your prompt has still billed prompt tokens. The response's `usage.cost` is the
> request total. In the example above it was `"cost": 0.01029`:
> `0.0036 + 0.00669`, the sum of both attempts.

## Recomputing your bill

The sum of attempt costs equals the response's `usage.cost`:

```bash
curl -s "https://demo-api.routerplus.com/v1/generation?id=$REQ" -H "Authorization: Bearer $TM_API_KEY" \
  | jq '[.attempts[].cost_usd] | add'
```

You can recompute each attempt's cost from its token counts and the published prices. This
script uses the settlement math, including floor division:

```python
import os
import requests
from decimal import Decimal

GATEWAY = "https://demo-api.routerplus.com"
H = {"Authorization": f"Bearer {os.environ['TM_API_KEY']}"}

prices = {m["id"]: m["pricing_usd_per_million"]
          for m in requests.get("https://demo.routerplus.com/api/models.json").json()["models"]}

def per_m(p, sku, fallback):
    # absent SKUs bill at the base rate, never $0
    return int(Decimal(p.get(sku, p[fallback])) * 1_000_000)  # micro-USD per M

def settle_micro(t, p):
    non_cached = max(0, t["input"] - t["cached"] - t["cache_write"])
    reasoning = min(max(0, t["reasoning"]), t["output"])
    plain_out = t["output"] - reasoning
    total = (non_cached        * per_m(p, "prompt", "prompt")
           + t["cached"]       * per_m(p, "cached_prompt", "prompt")
           + t["cache_write"]  * per_m(p, "cache_write", "prompt")
           + plain_out         * per_m(p, "completion", "completion")
           + reasoning         * per_m(p, "internal_reasoning", "completion"))
    return total // 1_000_000  # floor: settlement rounds down

gen = requests.get(f"{GATEWAY}/v1/generation",
                   params={"id": os.environ["REQ"]}, headers=H).json()
micro = sum(settle_micro(a["tokens"], prices[a["model"]])
            for a in gen["attempts"] if a["outcome"] != "unknown_after_crash")
print(micro / 1e6)  # equals the wire's usage.cost
```

- The script leaves out `unknown_after_crash` attempts. They settle at the reservation, not from
  token counts. Compare their `reserved_max_usd`.
- The same math holds for a model billed at the provider's reported cost. Its prices are `"0"`
  in and `"1"` out per million, and `output` is the charge in micro-dollars.
- An attempt with `inference_price_source` `customer_rate` was billed at a
  [dedicated endpoint's](/docs/dedicated-endpoints#prices) contract rate card. Recompute it with
  those prices.
- Attempts settle against **frozen price snapshots** pinned when the request was reserved. A
  recomputation at today's catalog differs only if the price changed since dispatch.

## Who can read what

- Usage and generation reads cover the authenticated workspace and principal, including that
  principal's other keys.
- A request made under another workspace or principal is a 404, like one that never existed.
- Keys bound to an explicit workspace or principal get `null` for the org balance, credit and
  spend totals. A key in the default workspace and principal gets them.

Members of the organization see the org-wide accounting in the console at
[https://demo.routerplus.com/console/usage](https://demo.routerplus.com/console/usage) and
[https://demo.routerplus.com/console/logs](https://demo.routerplus.com/console/logs).

In the console, open Analytics for spend, requests and tokens by model, provider, workspace,
member or key:

![Console Analytics page, grouped by model and then by provider](media/console-analytics.gif)

## Related

- [Pricing & billing](/docs/pricing): the reserve and settle math behind these numbers
- [Errors & remediation](/docs/errors): what each `error_code` means and what to do
- [Quickstart](/docs/quickstart): signup to first billed call

---

Source: https://demo.routerplus.com/docs/api-management.md

# Management API

Use the management API to control your organization's keys, guardrails, grants, presets,
workspaces, notifications and exports. It also reports usage, spend forecasts, savings, security
and audit data.

Every route is served under `/v1/...` and under `/api/v1/...`. The two paths act the same.

```bash
curl https://demo-api.routerplus.com/v1/keys \
  -H "Authorization: Bearer $TM_MANAGEMENT_KEY"
```

## Authentication

Send a management key as a Bearer token:

```
Authorization: Bearer tm_mk_...
```

- A management key belongs to one organization. Every route acts on that organization only. An
  id from another organization returns `404`.
- A management key never authenticates an inference request. An API key (`tm_vk_...`) never
  authenticates a management route, with two exceptions:
  - [`GET /v1/key`](#key-status-api-key) returns the calling API key's own status.
  - An API key can request grants for itself and read or cancel them. See [Grants](#grants).
- A missing, disabled or expired management key returns `401`.

## Requests and responses

- Send JSON with `content-type: application/json`. A request body is at most 256 KiB.
- One object comes back as `{"data": {...}}`. A list comes back as `{"data": [...]}`.
- Money is in USD. Times are ISO 8601 in UTC. Ids are UUIDs.
- The [audit log](#audit-log) records every write, with the management key's prefix as the actor.

### Errors

```json
{"error": {"code": 400, "message": "limit: must be a number of USD from 0 to 10,000,000", "type": "invalid_request"}}
```

`code` is the HTTP status. `type` is a stable string you can match on.

| Status | Meaning |
|---|---|
| 400 | The input is not valid. `message` names the field. |
| 401 | No valid key. |
| 403 | The key is valid but cannot do this. For example, an API key cannot approve a grant. |
| 404 | No such record in your organization, or no such route. |
| 405 | The route does not take this method. |
| 409 | A conflict: a record exists, a version is stale, or the record is in the wrong state. |
| 413 | The body is larger than 256 KiB. |

## Key status (API key)

| Route | Does |
|---|---|
| `GET /v1/key` | Returns the calling API key's own status. Authenticate with `Authorization: Bearer tm_vk_...`. |

```json
{"data": {"label": "ci", "usage": 1.25, "limit": 10, "limit_remaining": 8.75, "limit_reset": "monthly",
  "expires_at": null, "is_free_tier": false, "rate_limit": {"requests": 300, "interval": "1m"}, "include_byok_in_limit": false}}
```

- `usage` is the spend in the current limit period. With no `limit_reset`, it is the spend over
  the key's life.
- While an active [grant](#grants) raises the key's limit, `limit` is the granted value. The
  answer then also has `base_limit` (the key's own limit) and `grant_id`.
- An expired key returns `401 key_expired`.

## API keys

| Route | Does |
|---|---|
| `GET /v1/keys` | Lists active keys. Query: `workspace_id`, `include_disabled=true`. |
| `POST /v1/keys` | Creates a key. Returns `201` with `key` (the raw key) and `data`. |
| `GET /v1/keys/:id` | Reads one key. |
| `PATCH /v1/keys/:id` | Changes the fields you send. |
| `DELETE /v1/keys/:id` | Disables the key. It stops working at once. |
| `POST /v1/keys/:id/rotate` | Creates a new key with the same settings. |

The create and rotate responses show the raw key once. Store it then. You cannot read it again.

### Create fields

| Field | Type | Notes |
|---|---|---|
| `name` | string | Required. 1 to 80 characters. |
| `workspace_id` | UUID | Default: the default workspace. |
| `principal_id` | UUID | Default: the workspace's service principal. |
| `limit` | number or null | USD per `limit_reset` period. |
| `limit_reset` | string or null | `daily`, `weekly`, `monthly`, or null for no reset. |
| `include_byok_in_limit` | boolean | Count BYOK inference cost in the limit. |
| `expires_at` | ISO time or null | Must be in the future. The key stops working then. |
| `ip_allowlist` | array or null | CIDR ranges or addresses. Null allows any address. |
| `labels` | object | At most 20. Keys: `a-z 0-9 _ . -`, 1 to 40 characters. Values: at most 200 characters. |
| `finance_category` | string or null | 1 to 64 characters. |

### Update

`PATCH` takes `name`, `disabled`, and every create field except `workspace_id` and
`principal_id`. Send `null` to clear `limit`, `expires_at`, `ip_allowlist` or
`finance_category`.

### Rotate

Rotate takes `{"grace_seconds": 0..604800}` (default 0).

- With 0, the old key is disabled at once. Otherwise the old key expires after the grace period.
- The new key has `rotated_from` set to the old key's id.
- A disabled or expired key cannot be rotated (`409`).

### Key object

```json
{"key_id": "...", "name": "ci", "key_prefix": "tm_vk_3f9a1c", "workspace_id": "...", "principal_id": "...",
 "disabled": false, "limit": 10, "base_limit": 10, "grant_id": null, "limit_reset": "monthly", "limit_remaining": 8.75,
 "usage": 1.25, "include_byok_in_limit": false, "expires_at": null, "ip_allowlist": ["10.0.0.0/8"], "labels": {"team": "core"},
 "finance_category": null, "rpm_limit": 300, "created_at": "...", "created_via": "management_api",
 "rotated_from": null, "last_used_at": "..."}
```

While an active grant raises the key's limit, `limit` and `limit_remaining` use the granted
value. `base_limit` is the key's own limit and `grant_id` names the grant. With no grant,
`base_limit` equals `limit` and `grant_id` is null.

## Management keys

| Route | Does |
|---|---|
| `GET /v1/management-keys` | Lists management keys. |
| `POST /v1/management-keys` | `{name, expires_at?}`. Returns `201` with `key` (shown once). |
| `DELETE /v1/management-keys/:id` | Disables the key. A key cannot delete itself (`400 cannot_delete_self`). |

## Workload identities

A workload identity lets a CI job or cloud workload trade its OIDC token for a short-lived API
key.

| Route | Does |
|---|---|
| `GET /v1/workload-identities` | Lists identities. |
| `POST /v1/workload-identities` | Creates one. Returns `201`. |
| `DELETE /v1/workload-identities/:id` | Deletes one. |

| Field | Notes |
|---|---|
| `name` | Required. 1 to 100 characters. |
| `workspace_id` | Required. |
| `principal_id` | Default: the workspace's service principal. |
| `issuer` | An https URL, such as `https://token.actions.githubusercontent.com`. |
| `audience` | The token's `aud`. |
| `subject_pattern` | The token's `sub`. `*` matches any run of characters. |
| `required_claims` | Claim names and the exact values they must have. At most 20. |
| `key_ttl_seconds` | 300 to 86,400. Default 3,600. |

## Guardrails

A guardrail is a set of rules. It applies only when you assign it to the org, a workspace, a
member or a key. When several guardrails apply to a request, the strictest rule wins.

| Route | Does |
|---|---|
| `GET /v1/guardrails` | Lists guardrails with their assignments. |
| `POST /v1/guardrails` | `{name, description?, ...rules}`. Returns `201`. |
| `GET /v1/guardrails/:id` | Reads one guardrail. |
| `PATCH /v1/guardrails/:id` | Replaces the rules you send. `version` goes up by 1. |
| `DELETE /v1/guardrails/:id` | Deletes the guardrail and its assignments. |
| `POST /v1/guardrails/:id/assignments` | `{target_kind, target_id}`. Returns `201`. |
| `DELETE /v1/guardrails/:id/assignments/:kind/:target` | Removes one assignment. URL-encode an email. |
| `GET /v1/guardrails/:id/status` | Spend of each budget in its current period. |
| `GET /v1/guardrails/events` | Requests a guardrail blocked, redacted or flagged. |

### Rules

| Field | Type |
|---|---|
| `allowed_models`, `blocked_models` | Model patterns. `*` matches any run of characters. |
| `allowed_providers`, `blocked_providers` | Provider names. |
| `zdr` | boolean. Only providers that keep no data. |
| `data_collection` | `"deny"`: only providers that do not train on data. |
| `allowed_regions` | Region codes such as `us`, `eu`, `in`. |
| `blocked_infrastructure` | Clouds such as `aws`, `gcp`. |
| `budgets` | Up to 20 of `{id?, limit_usd, period, model?, provider?, include_byok?, shared?}`. `period`: `day`, `week`, `month`, `lifetime`. `shared: true` makes one budget for every target of the guardrail. |
| `limit_usd`, `reset_interval`, `include_byok_in_budgets` | One budget, in the short form. |
| `rpm`, `tpm`, `max_output_tokens` | Integers. |
| `server_tools` | `allow` or `block`. |
| `content_rules` | Up to 50 of `{id?, kind, preset?, pattern?, flags?, action, label?}`. `kind`: `pii`, `regex`, `injection`. `action`: `redact`, `block`, `flag`. |
| `on_scan_limit` | `allow` or `block`, for text too long to scan in full. |
| `overridable` | boolean. Default `true`. `false` makes the guardrail a hard limit. See [Hard limits](#hard-limits). |

An unknown field returns `400 invalid_guardrail`. A regex that can run too long returns
`400 invalid_regex_pattern`.

### Assignments

| `target_kind` | `target_id` |
|---|---|
| `org` | Your org id, or leave it out. |
| `workspace` | A workspace id. |
| `member` | A member's email. |
| `key` | An API key id. |

A target has at most one guardrail. A new assignment on a target replaces the old one. The
response names the replaced guardrail in `replaced_guardrail_id`.

### Status

A status entry:

```json
{"budget_id": "b0", "budget_key": "...", "target_kind": "key", "target_id": "...", "shared": false, "model": null, "provider": null,
 "include_byok": false, "limit_usd": 5, "base_limit_usd": 5, "grant_id": null, "spent_usd": 1.25, "remaining_usd": 3.75,
 "period": "day", "period_start": "2026-10-31T00:00:00.000Z", "resets_at": "2026-11-01T00:00:00.000Z"}
```

`limit_usd` is the limit RouterPlus enforces now. `base_limit_usd` is the guardrail's own limit.
While an active grant raises the budget, `grant_id` names it and `limit_usd` is the granted value.

### Events

| Query | Notes |
|---|---|
| `since`, `until` | Time range. |
| `action` | `blocked`, `redacted` or `flagged`. |
| `key_id` | One key. |
| `limit` | At most 500. Default 100. |

Events come newest first. They never hold the matched text.

## Grants

A grant is a time-boxed exception to a guardrail budget, an API key's limit, or a guardrail's
model rules. A member or an API key requests a grant. An admin approves or rejects it. An
approved grant applies from `starts_at` until `expires_at`. After that, the base value applies
again.

In the console, open Access > Grants.

```bash
curl https://demo-api.routerplus.com/v1/grants \
  -H "Authorization: Bearer $TM_MANAGEMENT_KEY" \
  -H "content-type: application/json" \
  -d '{"target": {"type": "key_limit", "key_id": "8c0d..."},
       "change": {"limit_usd": 50},
       "expires_at": "2026-11-09T00:00:00Z",
       "reason": "Load test for the launch"}'
```

| Route | Does | Who may call |
|---|---|---|
| `GET /v1/grants` | Lists grants, newest first. | Management key. An API key sees only the grants it requested. |
| `POST /v1/grants` | Requests a grant. Returns `201` with the grant in `pending` status. | Management key, or an API key for itself. |
| `GET /v1/grants/:id` | Reads one grant with its `events`. | Management key, or the API key that requested it. |
| `POST /v1/grants/:id/approve` | `{note?}`. Approves a pending grant. | Management key only. An API key gets `403`. |
| `POST /v1/grants/:id/reject` | `{note?}`. Rejects a pending grant. | Management key only. An API key gets `403`. |
| `POST /v1/grants/:id/cancel` | `{note?}`. Ends a pending or approved grant before `expires_at`. | Management key, or the API key that requested it. |

`note` is optional, at most 500 characters. An approve or cancel applies to the next request on
every gateway.

### Request fields

| Field | Notes |
|---|---|
| `target` | Required. What the grant changes. See the table below. |
| `change` | Required. `{"limit_usd": n}` for `budget` and `key_limit` grants. `{"allow_model": "<id or pattern>"}` for `guardrail` grants. |
| `starts_at` | ISO time. Default: now. It cannot be in the past (5 minutes of clock skew is allowed). |
| `expires_at` | Required. ISO time after now and after `starts_at`. At most 90 days after `starts_at`. |
| `reason` | Required. 1 to 500 characters. |
| `requested_by` | Management key only. The email of an active member to record as the requester. |

`target`:

| `type` | Fields | Changes |
|---|---|---|
| `budget` | `guardrail_id`, `budget_id`, and optionally `target_kind` and `target_id` | Raises one budget of a guardrail to `change.limit_usd`. With `target_kind` and `target_id`, the grant raises the budget for that assignment only. Without them, it raises it for every assignment. A shared budget takes no target. |
| `key_limit` | `key_id` | Raises an API key's limit to `change.limit_usd`. The key must have a limit. |
| `guardrail` | `guardrail_id`, and optionally `target_kind` and `target_id` | Lets `change.allow_model` pass the guardrail's allowed and blocked model lists. With a target, it covers only the keys under that target. Every other guardrail still applies. |

`target_kind` is `org`, `workspace`, `member` or `key`, as for [assignments](#assignments).

Rules for limits:

- `change.limit_usd` is a number of USD from 0 to 10,000,000. It must be more than the current
  limit.
- When several active grants raise the same limit, the highest one applies.
- A granted limit never goes below the base limit and never above the org's
  [hard limit](#hard-limits) for that period.

What an API key may request:

- A `key_limit` grant for its own key only.
- A `budget` grant only on a guardrail that applies to the key. Without a target, the grant
  raises the budget of the most specific assignment that covers the key.
- A `guardrail` grant only on a guardrail that applies to the key. The grant covers that key only.
- An API key cannot set `requested_by`.

### Grant object

```json
{"data": {"id": "...", "status": "approved", "state": "active",
  "target": {"type": "budget", "guardrail_id": "...", "budget_id": "b0", "target_kind": "workspace", "target_id": "..."},
  "change": {"limit_usd": 50}, "base_limit_usd": 20, "workspace_id": "...",
  "starts_at": "...", "expires_at": "...", "reason": "Load test for the launch",
  "requested_by": "key:...", "approved_by": "tm_mk_...", "decided_by": "tm_mk_...", "decided_at": "...",
  "decision_note": null, "cancelled_by": null, "cancelled_at": null, "created_at": "...", "updated_at": "..."}}
```

- `status` is `pending`, `approved`, `rejected` or `cancelled`.
- `state` adds the time window: `pending`, `scheduled` (approved, not started), `active`,
  `expired`, `rejected` or `cancelled`.
- `base_limit_usd` is the limit when the grant was requested. It is null for `guardrail` grants.
- `GET /v1/grants/:id` adds `events`: each state change with `action`, `actor`, `detail` and
  `created_at`.

### List query

| Query | Notes |
|---|---|
| `status` | `pending`, `approved`, `rejected` or `cancelled`. |
| `state` | `pending`, `scheduled`, `active`, `expired`, `rejected` or `cancelled`. |
| `target_type` | `budget`, `key_limit` or `guardrail`. |
| `key_id` | Grants on that key's limit, or guardrail grants scoped to that key. |
| `guardrail_id` | Grants on one guardrail. |
| `limit` | At most 500. Default 100. |

### Where grants show

- A response to a request under active grants has the header `x-tm-grants`, a comma-separated
  list of grant ids. A refusal under active grants has it too.
- [Guardrail status](#status) shows `limit_usd`, `base_limit_usd` and `grant_id` per budget.
- [Key reads](#key-object) and [`GET /v1/key`](#key-status-api-key) show the granted `limit`,
  `base_limit` and `grant_id`.
- Budget alerts use the granted limits.

### Notices

RouterPlus sends a notice to the org's enabled [notification channels](#notification-channels)
and by email when:

- a grant is requested (to org owners and admins, and admins of the grant's workspace),
- a grant is approved, rejected or cancelled (to the requester),
- an approved grant ends in less than a day (to the requester and the approver).

### Hard limits

A guardrail with `"overridable": false` is a hard limit.

- It takes no grant. A request for a grant on it returns `409 hard_limit`.
- When it is assigned to the org, each of its budgets with no `model` or `provider` caps every
  grant that raises a budget or key limit with the same period. A raise above the cap returns
  `409 hard_limit`.
- Approval checks the target again. A grant whose guardrail became a hard limit, or whose raise
  is now above the cap, is refused with `409 hard_limit`.

### Grant errors

| Status and type | When |
|---|---|
| `400 invalid_expiry` | `expires_at` is not after now and after `starts_at`. |
| `400 grant_too_long` | The grant lasts more than 90 days. |
| `400 grant_not_raise` | `change.limit_usd` is not more than the current limit. |
| `403 forbidden` | An API key tried to approve or reject, set `requested_by`, or request a grant outside its own key. |
| `404` | The guardrail, budget, assignment, key, member or grant is not in your org. |
| `409 hard_limit` | The guardrail is a hard limit, or the raise is above the org's hard limit. |
| `409 no_limit` | The key has no limit to raise. |
| `409 grant_not_pending` | Approve or reject on a grant that is not pending. |
| `409 grant_not_open` | Cancel on a grant that is rejected or cancelled. |
| `409 grant_expired` | The grant's `expires_at` has passed. |

## Presets

A preset is a versioned request template. Call it with `"model": "@preset/<slug>"` or
`"@preset/<slug>@<version>"`.

| Route | Does |
|---|---|
| `GET /v1/presets` | Lists the latest version of each preset. Query: `workspace_id`. |
| `POST /v1/presets` | `{slug, description?, workspace_id?, template}`. Creates version 1. |
| `GET /v1/presets/:slug` | The latest version. Query `version=N` for an older one. |
| `PUT /v1/presets/:slug` | `{template, description?, expected_version}`. Creates the next version. |
| `DELETE /v1/presets/:slug` | Deletes every version. |

- A slug is 1 to 80 of `a-z 0-9 _ -`, starting with a letter or digit.
- `POST` returns `409 preset_exists` when the slug is in use.
- `PUT` returns `409 version_conflict` when `expected_version` is not the latest version.

### Template

| Field | Type |
|---|---|
| `model` or `models` | Send one, not both. `models` lists 1 to 5 models, tried in order. |
| `provider` | A provider routing object, as in a request. |
| `defaults` | Parameters used when the request does not set them. |
| `overrides` | Parameters that replace the request's values. |
| `system_prompt` | Up to 20,000 characters. Put before the request's own system text. |
| `guardrail_id` | A guardrail in your org. |

Parameters in `defaults` and `overrides`:

| Parameter | Range |
|---|---|
| `temperature` | 0 to 2 |
| `top_p` | 0 to 1 |
| `top_k` | 1 to 32768 |
| `max_tokens`, `max_completion_tokens` | 1 to 1,000,000 |
| `frequency_penalty`, `presence_penalty` | -2 to 2 |
| `seed` | integer |
| `stop` | a string or up to 4 strings |
| `reasoning_effort` | `minimal`, `low`, `medium`, `high`, `xhigh` |
| `response_format` | object |
| `parallel_tool_calls`, `stream` | boolean |

### How a preset combines with a request

1. `overrides` beat the request. The request beats `defaults`.
2. The template's `model` or `models` replace the request's.
3. A request's `provider` object can only narrow the template's. `only` keeps the providers in
   both. `ignore` adds. `allow_fallbacks: false`, `zdr: true` and `data_collection: "deny"`
   always win. The request's `sort` and `order` apply only when the template sets none.
4. On `/v1/messages`, a preset sets `temperature`, `top_p`, `top_k`, `max_tokens`, `stop` (as
   `stop_sequences`) and `stream`. The other parameters apply on the OpenAI surface only.

## Workspaces

| Route | Does |
|---|---|
| `GET /v1/workspaces` | Lists workspaces. |
| `POST /v1/workspaces` | `{name, finance_category?, default_sort?}`. Returns `201`. |
| `PATCH /v1/workspaces/:id` | `name`, `disabled`, `finance_category`, `default_sort`, `route_policy_id`. |
| `POST /v1/workspaces/:id/clone` | `{name, include}`. Returns `201`. |
| `GET /v1/workspaces/:id/members` | Lists workspace members. |
| `PUT /v1/workspaces/:id/members` | `{email, role}`. Adds a member or changes the role. |
| `GET /v1/workspaces/:id/members/:email` | Reads one member. |
| `PUT /v1/workspaces/:id/members/:email` | `{role}`. |
| `DELETE /v1/workspaces/:id/members/:email` | Removes a member. |

- `default_sort`: `routing_cost`, `price`, `throughput` or `latency`.
- `role`: `admin`, `member` or `viewer`.
- The email must belong to a member of the organization.

### Clone

A clone gets the source's route policy, finance category and default sort. `include` can list:

- `guardrails`: the source workspace's guardrail assignment.
- `presets`: the presets scoped to the source. Each copy gets the slug `<slug>-copy`.
- `members`: the workspace members.

## Organization settings

| Route | Does |
|---|---|
| `GET /v1/org` | Returns `org_id`, `name`, `default_sort`, `request_log`, `request_log_until`, `retention_days` and `hipaa_mode`. |
| `PATCH /v1/org` | Changes the fields below. |

| Field | Notes |
|---|---|
| `default_sort` | As for workspaces, or null. |
| `request_log` | `on` or `off`. With `on`, RouterPlus logs full requests for `request_log_days` (1 to 365, default 30). In HIPAA mode, `on` returns `409 hipaa_mode`. |
| `retention_days` | 1 to 3650, or null for the defaults. See [Retention](#retention). |
| `hipaa_mode` | `true` turns HIPAA mode on at once. `false` turns it off only with `"confirm": "turn off hipaa mode"`. Without it the answer is `409 confirmation_required`. See [HIPAA mode](#hipaa-mode). |

Every change writes an audit row (`GET /v1/audit`, action `org.update`). Audit rows cannot be
changed or deleted.

### Retention

RouterPlus deletes stored records past their limit. A job runs every 30 minutes and deletes at
most 5,000 rows per table per run.

| Records | Default limit |
|---|---|
| Request labels and sessions | 400 days |
| Guardrail events | 400 days |
| MCP tool call records | 400 days |
| Eval answers and judge results | 90 days after the eval finished |
| Batch files | At their `expires_at` |
| Slack agent thread memory | Emptied 7 days after the last message, deleted 7 days later |
| OAuth codes and SSO sign-in states | 1 day after they expire |
| Sign-in log | 400 days |

`retention_days` can only make your org's limits shorter. RouterPlus uses the smaller of the
default and `retention_days`.

- For eval results, it counts from when the eval finished.
- For batch files, it counts from their creation.
- For Slack threads, it counts from their last message.
- The sign-in log, OAuth codes and SSO states keep the defaults.

### HIPAA mode

With `hipaa_mode` on:

- Every request gets an implicit org guardrail: `zdr: true`, `data_collection: "deny"`,
  `server_tools: "block"`. Its id in `x-tm-guardrails` is `00000000-0000-4000-8000-000000000074`.
  RouterPlus refuses a model with no route that meets it.
- RouterPlus does not log requests. The org's `request_log` setting and any batch setting do not
  apply.
- The response cache and the semantic cache are off. A request that asks for the cache gets an
  answer with `x-tm-cache: skip`.
- A broadcast destination with `privacy.include_content: true` returns `409 hipaa_mode`.
  RouterPlus does not send to a content destination made before HIPAA mode.
- Slack agents keep no thread memory. Each answer uses only the current message. Turning HIPAA
  mode on empties the stored memory.

HIPAA mode is a set of data rules. It is not a business associate agreement.

## Notification channels

Channels receive budget alerts when spend crosses a threshold, and [grant notices](#notices).

| Route | Does |
|---|---|
| `GET /v1/notifications/channels` | Lists channels. |
| `POST /v1/notifications/channels` | `{kind, config, enabled?, thresholds?}`. Returns `201`. |
| `PATCH /v1/notifications/channels/:id` | Changes the fields you send. `kind` cannot change. |
| `DELETE /v1/notifications/channels/:id` | Deletes the channel. |
| `POST /v1/notifications/channels/:id/test` | Sends a sample alert. Returns `{ok, status?, error?}`. |

| `kind` | `config` |
|---|---|
| `email` | `{to: [1 to 10 addresses]}` |
| `slack` | `{webhook_url: "https://hooks.slack.com/..."}` |
| `webhook` | `{url: "https://...", secret?}`. With a secret, RouterPlus signs each request in the `x-routerplus-signature` header. |

- `thresholds` are percents of a budget, integers from 1 to 100. Default: `[50, 80, 100]`.
- Each threshold is sent once per budget period. A channel gets one alert per budget, for the
  highest threshold crossed.
- Alerts cover guardrail budgets and API key limits.
- The Slack URL and the webhook secret come back as `"__secret__"`. Send `"__secret__"` to keep
  the stored value.

## Broadcast destinations

Destinations receive a trace of each request.

| Route | Does |
|---|---|
| `GET /v1/broadcast/destinations` | Lists destinations. |
| `POST /v1/broadcast/destinations` | `{name?, kind, config, enabled?, sampling?, privacy?, filter?}`. Returns `201`. |
| `PATCH /v1/broadcast/destinations/:id` | Changes the fields you send. `kind` cannot change. |
| `DELETE /v1/broadcast/destinations/:id` | Deletes the destination. |
| `POST /v1/broadcast/destinations/:id/test` | Sends one sample trace. Returns `{ok, status}`. |

| `kind` | `config` |
|---|---|
| `webhook` | `{url, secret?, headers?}` |
| `otlp` | `{endpoint, headers?, service_name?}` |
| `langfuse` | `{public_key, secret_key, host?}` |
| `datadog` | `{api_key, site?, service?}` |
| `clickhouse` | `{url, table, database?, user?, password?}` |
| `s3` | `{bucket, region, access_key_id, secret_access_key, prefix?, endpoint?, session_token?, gzip?}`. Also R2 and MinIO. |
| `azure_blob` | `{account, container, sas_token or account_key, prefix?, gzip?}` |
| `bigquery` | `{project_id, dataset, table, service_account_json}` |
| `snowflake` | `{account, user, private_key_pem, database, schema, table, warehouse?, role?}` |

[Broadcast to storage and warehouses](broadcast.md) gives the object layout and the table DDL for
the storage and warehouse kinds.

| Field | Notes |
|---|---|
| `sampling` | 0 to 1. Default 1. |
| `privacy` | `{include_content, omit_fields, hash_fields, redact_paths}`. Content is left out by default. In HIPAA mode, `include_content: true` is refused. |
| `filter` | `{workspace_ids, key_ids, models, errors_only}`. |

Secrets and header values never come back. They show as `"__secret__"`. Send `"__secret__"` to
keep the stored value.

## BYOK connections

| Route | Does |
|---|---|
| `PATCH /v1/connections/:id` | `{"negotiated_discount_percent": 0..100}` sets the discount your provider gives you on a BYOK connection. Send `null` to clear it. |

Cost reports use the discount (see [`GET /v1/activity/byok`](#activity)).

## Activity

| Route | Does |
|---|---|
| `GET /v1/activity` | Usage per group. |
| `GET /v1/activity/labels?label=<key>` | Requests grouped by the value of one request label. |
| `GET /v1/activity/sessions` | Requests grouped by session id. |
| `GET /v1/activity/cache` | Prompt caching per model and provider. |
| `GET /v1/activity/cache/advice` | What in your traffic costs cache reuse, with the numbers and the fix. |
| `GET /v1/activity/byok` | Requests on your own provider connections, per connection and model. |
| `GET /v1/activity/forecast` | Projected spend to the end of the month or quarter. See [Spend forecast](#spend-forecast). |
| `POST /v1/activity/forecast` | The same forecast, with the parameters in the body. |
| `GET /v1/activity/savings` | What your traffic would cost on cheaper routes or models. See [Savings](#savings). |

`GET /v1/activity` takes:

| Query | Notes |
|---|---|
| `from`, `to` | UTC dates, both included. Default: the last 30 days. At most 366 days. |
| `group_by` | `day` (default), `model`, `provider`, `key`, `workspace`, `member`, `finance_category`. |
| `key_id`, `workspace_id`, `model`, `provider` | Filters. |

The label, session, cache and BYOK reports take `from` and `to` too.

Response rows:

| Report | Each row has |
|---|---|
| `/v1/activity` | The group field (`date`, `model`, `provider`, `key_id`, `workspace_id`, `member` or `finance_category`), `requests`, `usage_usd`, `byok_usage_usd`, `input_tokens`, `cached_input_tokens`, `cache_write_tokens`, `output_tokens`, `reasoning_tokens`, `cache_hit_rate`. |
| `/v1/activity/labels` | `value`, `requests`, `attempts`, `usage_usd` and the token counts. |
| `/v1/activity/sessions` | `session_id`, `requests`, `usage_usd`, `models`, `first_seen`, `last_seen`. Query `limit`: at most 500. |
| `/v1/activity/cache` | `input_tokens`, `cached_input_tokens`, `cache_write_tokens`, `cache_hit_rate`, `estimated_savings_usd`. |
| `/v1/activity/byok` | `requests`, `list_cost_usd`, `negotiated_discount_percent`, `negotiated_cost_usd` (the list cost less the discount set with `PATCH /v1/connections/<id>`). |

- `cache_hit_rate` is cached input tokens divided by input tokens.
- `estimated_savings_usd` is the cached tokens times the difference between the input price and
  the cached input price.

`/v1/activity/cache/advice` returns these findings:

| Finding | When |
|---|---|
| `low_hit_rate` | A model with 1M+ input tokens and a hit rate under 20%, with the most it could save. |
| `cache_writes_unread` | Cache writes that no later request read. |
| `no_session_ids` | Over 30% of requests carry no session id. |

### Spend forecast

`GET /v1/activity/forecast` projects your org's spend on RouterPlus credits to the end of the
current UTC month, per group. It also gives the first day each budget or key limit is projected
to run out. BYOK spend is not projected.

```bash
curl "https://demo-api.routerplus.com/v1/activity/forecast?group_by=workspace&horizon=quarter" \
  -H "Authorization: Bearer $TM_MANAGEMENT_KEY"
```

| Parameter | Notes |
|---|---|
| `group_by` | `model` (default), `workspace`, `key` or `label`. |
| `label` | Required with `group_by=label`. A label key: 1 to 40 of `a-z 0-9 _ . -`. |
| `horizon` | `month` (default): to the end of the current UTC month. `quarter`: to the end of the third month after it. |
| `scenario` | Up to 10 changes. See [Scenarios](#scenarios). With `GET`, send it as URL-encoded JSON. |

`POST` takes the same four fields as a JSON body. An unknown field returns `400`.

How RouterPlus projects each group's daily spend:

| `method` | When | How |
|---|---|---|
| `weekday_trend_14d` | 14 or more days of history | A least-squares line through the last 14 days, times each weekday's factor from the last 28 days, clipped at zero. |
| `trailing_mean` | 3 to 13 days of history | The mean daily spend of those days. No weekday pattern, no trend. |
| `insufficient_history` | Fewer than 3 days | No projection. `projected_usd`, `low_usd` and `high_usd` are null. |

`low_usd` and `high_usd` are about a 90% range. History counts from the first day with spend.

Response:

| Field | Notes |
|---|---|
| `group_by`, `label`, `horizon` | The parameters used. |
| `as_of` | Today, UTC date. |
| `periods` | The months of the horizon: `{period, start, end}`. |
| `method` | `{trend_days: 14, pattern_days: 28, min_history_days: 3, band: "90%", descriptions}`. |
| `total` | The org's projection. See the projection fields below. |
| `data` | One row per group, largest first, at most 200. Each row has the group field (`model`, `workspace_id`, `key_id` or `value`), `name` for workspaces and keys, and the projection fields. |
| `crossings` | Budgets and key limits projected to run out before the end of the horizon, earliest first. |
| `scenario` | Null, or the scenario's result. |
| `daily` | `{start, usd}`: the org's actual spend per day, from 42 days before today. |

Projection fields: `spent_so_far_usd` (this month, today included), `projected_usd`, `low_usd`,
`high_usd`, `method`, `history_days`, `history_too_short`, and `periods` (per month:
`period`, `start`, `end`, `spent_usd`, `projected_usd`, `low_usd`, `high_usd`).

A crossing:

| Field | Notes |
|---|---|
| `kind` | `guardrail_budget` or `key_limit`. |
| Budget fields | `guardrail_id`, `budget_id`, `target_kind` (`group` for a shared budget), `target_id`, `model`, `provider`, `spent_known`. |
| Key limit fields | `key_id`, `name`, `key_prefix`, `limit_kind` (`limit` or `monthly_cap`). |
| `period`, `limit_usd`, `spent_usd` | The limit and its spend in the current period. |
| `crosses_on` | The first UTC date the projection reaches the limit. Spend resets at each period start. |
| `already_exceeded` | `true` when spend has already reached the limit. `crosses_on` is then today. |
| `method`, `history_days` | How that scope's spend was projected. |

#### Scenarios

A scenario changes the history that RouterPlus fits the projection to. It never changes spend so
far.

| Change | Does |
|---|---|
| `{"scale": n}` | Multiplies all traffic by `n` (0 to 100). |
| `{"scale": n, "workspace_id": "..."}` | Multiplies one workspace's traffic. Use `model` or `key_id` in place of `workspace_id` for one model or key. Name at most one. |
| `{"swap": {"from": "a", "to": "b"}}` | Moves model `a`'s traffic to model `b`. It prices the last 14 days of `a`'s tokens on `b`'s cheapest catalog route that your org guardrails allow. |

```bash
curl https://demo-api.routerplus.com/v1/activity/forecast \
  -H "Authorization: Bearer $TM_MANAGEMENT_KEY" \
  -H "content-type: application/json" \
  -d '{"horizon": "month", "scenario": [{"scale": 2}, {"swap": {"from": "google/gemini-3.1-pro", "to": "google/gemini-3.5-flash"}}]}'
```

`scenario` in the response has:

- `changes`: each change. A swap adds `provider`, `cost_ratio` and `tokens_basis_days` (14), and a
  `note` when `a` had no spend in the last 14 days.
- `total`: the projection fields and `delta_usd`, the change from the base projection.
- `data`: per group, `projected_usd`, `low_usd`, `high_usd`, `delta_usd` and `method`.
- `crossings`: the crossings under the scenario.

A swap returns `400` when the target model is not allowed by your org guardrails, has no catalog
price on an allowed route, or when one model is swapped twice.

### Savings

`GET /v1/activity/savings` reprices your traffic on RouterPlus credits for a window. For each
model and provider, it shows what the same tokens would cost on a cheaper route of the same model,
and on cheaper models of a similar quality tier. BYOK traffic is left out.

```bash
curl "https://demo-api.routerplus.com/v1/activity/savings?days=30" \
  -H "Authorization: Bearer $TM_MANAGEMENT_KEY"
```

| Query | Notes |
|---|---|
| `days` | An integer from 1 to 90. Default 30. The window ends today. |

The rules:

- A suggestion must save at least 1% of the baseline.
- A suggestion meets the guardrails of the keys that sent the traffic.
- A model of a similar tier has an Artificial Analysis intelligence index at most 3 points below
  the current model's. A model with no score is not compared. The answer counts those models.
- Each row lists at most 3 other models.
- The figures reprice token counts. They do not show that answers would be as good. Each row
  with other models includes the body of an eval to test them.

Response:

| Field | Notes |
|---|---|
| `days`, `from`, `to` | The window. |
| `scores_status` | Whether benchmark scores are loaded. `unavailable` when the server has no model catalog. |
| `tier_points` | 3. |
| `summary` | `spend_usd`, `route_saving_usd`, `model_saving_usd`. |
| `data` | One row per model and provider, largest saving first. |
| `notes` | Plain-text notes on what the figures mean. |
| `source`, `source_url` | `Artificial Analysis`, `https://artificialanalysis.ai/`. |

A `data` row:

| Field | Notes |
|---|---|
| `model`, `provider`, `requests`, `spend_usd` | The traffic. |
| `tokens` | `input`, `cached_input`, `cache_write`, `output`, `reasoning`. |
| `baseline_usd`, `baseline_basis` | The cost the savings compare against. `catalog_price` when the current route has a catalog price, else `spend` (what you were charged). |
| `intelligence_index` | The model's score, or null. |
| `cheapest_route` | Null, or `{provider, cost_usd, saving_usd, saving_percent, confidence, note}` for the same model on another provider. `confidence` is `high`, or `medium` against `spend`. |
| `route_note` | Why there is no `cheapest_route`, or null. |
| `alternative_models` | Up to 3 of `{model, provider, intelligence_index, index_difference, cost_usd, saving_usd, saving_percent, confidence, note}`. `confidence` is `medium` for an equal or higher score, `low` for a lower one. |
| `alternatives_note` | Why there are no alternatives, or how many models had no score. |
| `eval` | Null, or `{method: "POST", path: "/v1/evals", body, note}`: an eval of the current and other models, with a pairwise judge. Replace `cases` with 20 or more real prompts. |
| `best_saving_usd` | The larger of the route saving and the best model saving. |

### Labels

Label a request in one of three ways:

- The header `x-tm-labels: team=search,feature=autocomplete` (also `x-routerplus-labels`).
- The body field `trace`: an object of string values. RouterPlus removes it before the request
  reaches a provider.
- Labels set on the key (`PATCH /v1/keys/<id> {labels}`).

Rules:

- For one name, a request's own label wins over the key's.
- At most 20 labels. Names are lowercase letters, digits, `_`, `.` and `-`, up to 40 characters.
  Values are up to 200 characters.
- RouterPlus adds `task` (the kind of request).
- `x-tm-finance-category` sets the request's finance category.

## Security report

| Route | Does |
|---|---|
| `GET /v1/security/keys` | Lists every active API key with flags. `summary` counts each flag. |

| Flag | True when |
|---|---|
| `no_limit` | The key has no spend limit. |
| `no_expiry` | The key never expires. |
| `idle` | No request in 30 days, or no request ever and the key is older than 7 days. |
| `expiring_soon` | The key expires within 7 days. |
| `no_ip_allowlist` | The key works from any address. |
| `rotated` | A rotation replaced the key, and it still works in its grace period. |

Each key also has `last_used_at`, `created_at`, `expires_at` and `usage_30d_usd`.

## Audit log

| Route | Does |
|---|---|
| `GET /v1/audit` | Returns `{source, at, actor, action, subject}` rows, newest first. |

- `source` is `governance`, `identity`, `connection` or `routing_policy`.
- Query: `since`, `until`, `action`, `actor`, `limit` (at most 500, default 100).
- Grant changes write the actions `grant.request`, `grant.approve`, `grant.reject` and
  `grant.cancel`.

## SLA report

| Route | Does |
|---|---|
| `GET /v1/sla/report?month=YYYY-MM` | Your org's availability in one UTC month. |

Without `month`, it reports the current month up to now. A month in the future returns `400`.

The report counts requests. A request's result is its last attempt:

| Class | The last attempt |
|---|---|
| `good` | Completed. |
| `buyer_caused` | Was cancelled by the client, ended with HTTP 400, 413 or 429, or was refused while the org's configuration changed. |
| `upstream_quota` | Hit a provider pool's quota. |
| `failed` | Any other failure. |
| `pending` | Is not settled yet. Left out of the other counts. |

```json
{"data": {"month": "2026-09", "period_start": "2026-09-01T00:00:00.000Z", "period_end": "2026-10-01T00:00:00.000Z", "complete": true,
  "requests": {"total": 75, "good": 57, "buyer_caused": 4, "upstream_quota": 1, "failed": 13, "pending": 1},
  "availability_percent": 82.6667,
  "degraded": {"bucket_minutes": 5, "min_requests": 20, "failed_percent_over": 5, "minutes": 5, "period_minutes": 43200,
    "minutes_percent_available": 99.9884, "buckets": [{"start": "2026-09-10T12:05:00.000Z", "requests": 20, "failed": 2}]},
  "credit": null}}
```

| Field | Meaning |
|---|---|
| `availability_percent` | (`total` - `failed`) / `total` x 100. Null with no requests. |
| `degraded.minutes` | 5 minutes for each 5-minute bucket with at least 20 requests and more than 5% `failed`. |
| `credit` | Null unless your agreement sets credit tiers. Then it is `{tiers, tier}`, where `tier` is the tier for this month, or null. |

## Configuration backup

| Route | Does |
|---|---|
| `GET /v1/config/export` | Exports the configuration. Add `?include_secrets=true` to include secrets. |
| `POST /v1/config/import` | `{mode, dry_run?, config}`, where `config` is an export. |

The export:

```json
{"version": 1, "exported_at": "...", "guardrails": [...], "presets": [...], "workspaces": [...],
 "notification_channels": [...], "broadcast_destinations": [...]}
```

Secrets are written as `"__secret__"` unless you add `?include_secrets=true`.

Import modes:

| Value | Does |
|---|---|
| `mode: "merge"` | Creates what is missing and updates what matches. Workspaces and guardrails match by name, presets by slug, channels and destinations by id, then by address or name. |
| `mode: "replace"` | Does what `merge` does, then deletes the guardrails and presets that the file does not list. |
| `dry_run: true` | Returns the same counts and changes nothing. |

A `"__secret__"` value keeps the stored secret. RouterPlus skips a record that needs a secret you
did not send.

The response:

```json
{"data": {"mode": "merge", "dry_run": false,
  "counts": {"workspaces": {"created": 1, "updated": 0, "deleted": 0}, "guardrails": {...}, "assignments": {...},
             "presets": {...}, "notification_channels": {...}, "broadcast_destinations": {...}},
  "skipped": [{"kind": "notification_channels", "ref": "slack:...", "reason": "..."}]}}
```

---

Source: https://demo.routerplus.com/docs/api-signup.md

# Retired signup API

`POST https://demo.routerplus.com/v1/signup` is removed and returns HTTP `404`. `POST https://demo.routerplus.com/login`, the old magic-link route, also returns HTTP `404`. Create an account in the browser through Clerk:

1. Open [Sign up](https://demo.routerplus.com/signup).
2. Complete Clerk authentication and email verification.
3. Save the first API key shown after sign-in. RouterPlus shows each raw key once.

Create more keys at [API keys](https://demo.routerplus.com/console/keys). If you have an account, [sign in](https://demo.routerplus.com/login) with the same verified email. Your account, balance and keys stay.

Signup does not grant free credit. Paid credit purchases start at **$10**.

If purchase matching is offered for your account, Billing shows the cap and remaining matching credit.

See [Pricing](/docs/pricing), and [Billing](https://demo.routerplus.com/console/billing) for the offer on your account.

Existing `tm_vk_` API keys and the authentication headers do not change. Continue with [Quickstart](/docs/quickstart) or [Authentication](/docs/authentication).

---

Source: https://demo.routerplus.com/docs/errors.md

# Errors

Every failure carries one canonical error class, a short stable string such as `rate_limit` or `insufficient_quota`. The class is in the body and in the `x-tm-error-code` response header. Switch on the class in your code. Do not parse the message.

The body keeps the native shape of the surface you called (OpenAI or Anthropic), so your SDK's error handling keeps working.

A failure after RouterPlus accepted your request also has an origin, in the `x-tm-error-origin` header and in `origin` in the body's metadata:

| Origin | Meaning |
|---|---|
| `gateway_admission` | A limit or balance of yours |
| `upstream_quota` | A provider's declared capacity |
| `upstream` | The provider answered |
| `gateway_infrastructure` | RouterPlus's side |
| `authorization` | Your key or connection |

When a limit refused the request, `x-tm-limit-scope`, `x-tm-limit-kind` and, where known, `x-tm-limit-id` name it. See [Limits and capacity](/docs/admission).

## Where the class lives

OpenAI surface (`POST /v1/chat/completions`):

```json
{
  "error": {
    "code": "model_unavailable",
    "message": "model \"gpt-5-nano\" is not in the catalog; GET /v1/models lists what this key can serve",
    "type": "invalid_request_error",
    "metadata": { "error_type": "model_unavailable" }
  },
  "request_id": "6f8f57b2-..."
}
```

Anthropic surface (`POST /v1/messages`):

```json
{
  "type": "error",
  "error": {
    "type": "not_found_error",
    "message": "model \"gpt-5-nano\" is not in the catalog; GET /v1/models lists what this key can serve",
    "error_type": "model_unavailable"
  },
  "request_id": "6f8f57b2-..."
}
```

- The canonical class is always in `error.metadata.error_type` (OpenAI surface) or `error.error_type` (Anthropic surface).
- RouterPlus generates every error body and never relays a provider's raw error body. When a provider answered, its HTTP status is in `x-tm-upstream-status`.
- On the Anthropic surface, `error.type` is the native Anthropic type string (`rate_limit_error`, `billing_error`, ...), so the Anthropic SDK's error classification and retries work unmodified.
- On the OpenAI surface, `error.code` carries the canonical class, and `error.type` is a generic string. Switch on `metadata.error_type`, `error.code` or the HTTP status.
- Three OpenAI-surface errors match OpenAI's own wire byte for byte, so OpenAI SDKs and coding agents recognize them:
  - Out of credits: `code` and `type` are both `insufficient_quota`.
  - A prompt too long for the model: `code` is `context_length_exceeded`. See [Context overflow](#context-overflow).
  - A rate limit on `/v1/chat/completions` and `/v1/responses`: `code` is `rate_limit_exceeded`, and `type` is `requests`, or `tokens` for a token limit.
- An error before routing (bad key, unknown route, a crash in a RouterPlus handler) takes its shape from the path: Anthropic on `/v1/messages`, OpenAI on every other path.
- On `POST /v1/responses`, a model served through the chat translation (Claude and the other models OpenAI does not serve) gets OpenAI's own codes where clients act on them: `rate_limit_exceeded`, `context_length_exceeded`, `model_not_found` and `invalid_prompt` in `error.code`. The canonical class stays in `error.metadata.error_type`. See [POST /v1/responses](/docs/api-responses#errors).

### The provider's own words

When a provider refused the request, the message carries the provider's own words: `upstream <route> returned <status>: <words>`. Example: `upstream anthropic returned 400: messages.1.output_config: Extra inputs are not permitted`. Coding agents read these words to drop the refused feature and retry.

- The words are the `message` of the provider's JSON error, on one line, at most 1,000 characters.
- A marker replaces links, host names, keys, and account, organization and project ids.
- The message is `upstream <route> returned <status>` with no words when:
  - the provider's error is not JSON,
  - the words would repeat RouterPlus's key,
  - on a RouterPlus route, the error is about RouterPlus's provider account (a 401, 402, 403 or 429, credit, billing or quota).
- On your own connection (BYOK), you also get the words about your account.
- A [closed dedicated endpoint](/docs/dedicated-endpoints#closed-endpoints) never shows a provider's words. Dedicated capacity is named `routerplus`.

### Limit metadata

When a limit or a balance refused the request, `error.metadata` on both surfaces also has `origin`, `limit_scope`, `limit_kind`, `limit_id`, `retryable`, and `retry_at` or `reset_at` when known. On the OpenAI surface they sit beside `error_type`.

> [!NOTE]
> A 502 or a 504 does not reach you as the JSON body on this page. Cloudflare carries all
> traffic to the gateway and replaces these two statuses with its own short error page (plain
> text such as `error code: 502`, or an HTML page), without `x-request-id`. Treat both as
> retryable. When the request reached a provider, `GET /v1/usage` lists it with its request id.

## The canonical classes

| `error_type` | HTTP | OpenAI `error.code` | Anthropic `error.type` | Meaning |
|---|---|---|---|---|
| `auth` | 401 (or a relayed 401/402/403) | `auth` | `authentication_error` | Your key is missing, wrong or disabled. Rarely, RouterPlus's own provider account failed on every candidate (see [Failover](#failover-and-what-the-error-you-see-means)). |
| `invalid_request` | 400 | `invalid_request` | `invalid_request_error` | The request cannot be served as sent. See [invalid_request](#invalidrequest). |
| `model_not_priced` | 400 | `model_not_priced` | `invalid_request_error` | The model is listed but has no price row. RouterPlus refuses the request and does not guess $0. |
| `context_overflow` | 400 (on `/v1/decisions`, the provider's status: 413 from Cloudflare on Clef and Clef-flash) | `context_length_exceeded` (`context_overflow` on `/v1/decisions`) | `invalid_request_error` | The prompt exceeds the model's context window. See [Context overflow](#context-overflow). |
| `content_policy` | provider's status (usually 400) | `content_policy` | `invalid_request_error` | The provider refused the content. An OpenAI image moderation refusal (`moderation_blocked`) is relayed at the provider's own status (usually 400). Never billed, never rerouted, never a health strike. |
| `model_unavailable` | 404 / 502 / 503 | `model_unavailable` | `not_found_error` | The model cannot be served. See [model_unavailable](#modelunavailable). |
| `upstream_error` (timeout) | 504 | `upstream_error` | `api_error` | The provider exceeded RouterPlus's wait. See [Timeouts](#timeouts). Retryable. A single timeout never trips the health circuit. |
| `not_found` | 404 | `not_found` | `not_found_error` | No such route. On `/v1/generation`, an id that is not a request id. |
| `request_too_large` | 413 | `request_too_large` | `request_too_large` | Body over 10 MB (1 MB on `POST /v1/decisions`). |
| `rate_limit` | 429 | `rate_limit_exceeded` (`rate_limit` on `/v1/decisions`, images and video) | `rate_limit_error` | A rate limit refused the request. See [rate_limit](#ratelimit). |
| `email_not_verified` | 403 | `email_not_verified` | `permission_error` | The key's account has not opened its email verify link. RouterPlus refuses every request until it does, even a free one. |
| `insufficient_quota` | 429 | `insufficient_quota` | `billing_error` | Balance too low, or a monthly spend cap hit. No `retry-after`. A spend cap also carries `x-should-retry: false`. |
| `usage_limited` | 429 | `usage_limited` | `rate_limit_error` | RouterPlus detected unusual usage on an organization that has only trial credit, and limited this request. `retry-after` says when to try again. |
| `usage_restricted` | 403 | `usage_restricted` | `permission_error` | RouterPlus detected unusual usage on an organization that has only trial credit. It refuses this kind of request until support reviews the account. |
| `upstream_error` | 5xx, or provider's 4xx | `upstream_error` | `api_error` | The provider failed and no alternate could serve. At 4xx, the provider rejected the request itself. On `/v1/decisions`, OpenAI's 400 on GPT-6 Luna Decisions names your question id and System One's field, for example `questions.team.criteria`. On the images route, a 200 that is unparseable or has no image is a 502 that bills nothing. |
| `gateway_error` | 400 / 500 / 502 / 503 / 504 | `gateway_error` | `api_error` | RouterPlus refused or could not route the request. See [gateway_error](#gatewayerror). |

When the status comes from a provider, `x-tm-upstream-status` also has the provider's own HTTP status.

On `POST /v1/decisions`, GPT-6 Luna Decisions can decline one question and answer the rest.
That is not an error: the response is a 200, the declined question's answer is
`{"type": "refusal"}`, and the call is billed. Check each answer's `type`.

### invalid_request

The message names the field. Causes:

- The body is not valid JSON, or `n>1`.
- No deployment can serve the request as sent:
  - `temperature > 1`.
  - `response_format` or `thinking` across a dialect boundary.
  - Content a translation cannot carry across a dialect boundary: audio, a file id, a file other than a PDF, or an image in a format other than JPEG, PNG, GIF or WebP.
  - A content part the model does not take: an image, file, audio or video part that no route of the model declares.
- A request-level credential or destination (`api_key`, `base_url`, `connection_id`, …), or an unknown `provider` control.
- On `/v1/responses`: `previous_response_id` or `background: true`. RouterPlus keeps no response state.
- On `/v1/images/generations`: a value outside the model's accepted values, `n` outside 1 to 4, `stream`/`partial_images`, `response_format: url`, or a model on the wrong route. An image model on a chat route is also this class, and the message names the right route.
- On `/v1/decisions`:
  - A question in another decision model's format: `schema` on Jev, the Bespoke Nimble models, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider v1 27B, GPT-6 Luna Decisions, Sage or ScaleDown Classify-1, or `questions` on GLiNER-2.5-Decide.
  - A question kind that does not fit the state.
  - A field the model does not take.
  - Two GLiNER results with one name.
  - On Perplexity Decider v1 27B, an object with `"type": "image_url"` in the state or a question. Decision models take text and JSON.

### model_unavailable

- **404:** the id is not in the catalog (the message echoes your id), or your key's connection does not serve it. On `/v1/decisions`, also when OpenAI no longer gives us access to GPT-6 Luna Decisions (its 404 `model_not_found`, relayed with OpenAI's words).
- **502:** the connection to the provider failed on every candidate.
- **503, on `/v1/decisions`:** the provider refuses RouterPlus's account for capacity. The message says the model is out of capacity at the provider. Cases:
  - Levanto's monthly allowance for Sage is used up. The message says Sage is out of capacity until the provider's next billing period.
  - Fastino refuses GLiNER-2.5-Decide for credit or billing.
  - Bespoke Labs refuses a Bespoke Nimble model for credit (its 402).
  - Cloudflare refuses Clef or Clef-flash for our token or plan (its 401 or 403), or because our account's free daily allocation is used up (its 429 with code 3036).
  - Also a 401, 402 or 403 from any System One provider on our key: TypeSafe (Jev), RouterPlus (Decider 2B and Kev 4B), Perplexity (Perplexity Decider v1 27B) and OpenAI (GPT-6 Luna Decisions).
  - On a connection with your own key, the provider's 401, 402 or 403 keeps its status.

### Timeouts

The provider exceeded RouterPlus's wait (504 `upstream_error`). RouterPlus waits 20 s for the first response headers on streams, the generation budget (450 s) on non-stream calls, or 60 s for an answer on `/v1/decisions`.

- Cold starts on `/v1/decisions`: a RouterPlus cold start, 15 to 20 s, fits inside it. A GLiNER-2.5-Decide cold start can take longer than 60 s.
- On `/v1/decisions`, a System One provider's own 408 (Cloudflare's timeout on Clef and Clef-flash) is this class too.
- So is Perplexity's own 504 on Perplexity Decider v1 27B. It counts against the circuit, as every 5xx does.

### rate_limit

A rate limit refused the request. The limit is one of:

- the key's RPM,
- one of the organization's shared limits,
- a provider pool's declared capacity, or its cooldown after a provider 429,
- the provider rate-limited every candidate.

`x-tm-limit-scope` and `x-tm-limit-kind` say which. `x-tm-limit-id` says whether a pool limit was the pool's (`pool:…`) or your organization's share of it (`pool-share:…`).

On `/v1/decisions`, a provider's 429 never opens the deployment's health circuit. Pools per model:

| Models | Pool | Provider's own 429 |
|---|---|---|
| Bespoke Nimble models | the Bespoke Nimble models share one pool, which takes 7 requests at once, 5 per organization. Bespoke allows 8 at once for our whole account, and one is kept for our test environment. | Comes back as `rate_limit`. |
| Clef and Clef-flash | One shared pool: 200 requests a minute, 150 per organization; at most 8 at a time, 6 per organization. | Cloudflare's 429 when Workers AI is busy (code 3040) comes back as `rate_limit`. Its code 3036 is the 503 `model_unavailable` above. |
| Decider 2B and Kev 4B | Decider 2B and Kev 4B share one RouterPlus pool (11,400 requests a minute, 59 in flight, 75 % per organization), so your organization's own limits bind first. | RouterPlus's own 429 (about 200 requests a second for the whole endpoint) comes back as `rate_limit` and pauses the pool for both models. |
| Perplexity Decider v1 27B | 540 requests a minute (405 per organization) and 8 at once (6 per organization). Its hold counts the state once per question, so a long state with many questions can pass a token limit by itself (`x-tm-limit-kind: tpm`). Ask fewer questions per call. | Perplexity's own 429 (10 requests a second for our organization) comes back as `rate_limit`. |
| GPT-6 Luna Decisions | GPT-6 Luna Decisions' pool takes 24,000 requests a minute (18,000 per organization) and 64 in flight (48 per organization), so your organization's own limits bind first. | OpenAI's own 429 (30,000 requests a minute for our organization) comes back as `rate_limit`. |

### gateway_error

| Status | Cause |
|---|---|
| 400 | RouterPlus refused an output bound (`x-tm-limit-kind: output_bound`): `max_tokens` outside 1 to the model's own maximum, or not a whole number. The message names the maximum. It is 32,768 only when no maximum is declared. `0` and `-1` are refused too, on both surfaces. So are `max_tokens` and `max_completion_tokens` both present and different. `null` counts as not set, and RouterPlus sets its default. |
| 500 | A RouterPlus bug. |
| 502 | The provider's image response exceeded the 32 MiB cap, or a 2xx that RouterPlus's translator cannot represent. Nothing billed. |
| 503 | No capacity now: every deployment serving the model is cooling down (`x-tm-limit-kind: health`, `retry-after: 5`), the platform or this worker is at capacity, or the shared limit store or the ledger journal is unavailable. Nothing was sent to a provider or charged. |
| 504 | The request's routing deadline (450 s, or the policy's `timeout_ms`) passed before RouterPlus could try a provider. |

## Context overflow

A prompt longer than the model's context window is a 400 in the standard shape of the surface you called. Coding agents (Claude Code, Codex, Cursor, Cline and others) look for these words, then compact the conversation and retry. M is the model's context window in tokens, and N is the size of your prompt. The status is 400 whatever the provider sent. `x-tm-upstream-status` keeps the provider's status.

OpenAI surface (`POST /v1/chat/completions`):

```json
{
  "error": {
    "code": "context_length_exceeded",
    "param": "messages",
    "message": "This model's maximum context length is M tokens. However, your messages resulted in N tokens. Please reduce the length of the messages.",
    "type": "invalid_request_error",
    "metadata": { "error_type": "context_overflow" }
  },
  "request_id": "..."
}
```

Anthropic surface (`POST /v1/messages`):

```json
{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "prompt is too long: N tokens > M maximum",
    "error_type": "context_overflow"
  },
  "request_id": "..."
}
```

When the input fits but the input and the output limit together do not, the message says so:

- OpenAI surface: `… However, you requested T tokens (N in the messages, K in the completion). Please reduce the length of the messages or completion.`
- Anthropic surface: ``input length and `max_tokens` exceed context limit: N + K > M, decrease input length or `max_tokens` and try again``

The numbers come only from the provider's own words. When the provider gave none, the message is `This model's maximum context length was exceeded. Please reduce the length of the messages.` or `prompt is too long for the model's context window`. `POST /v1/responses` uses the same `code`, `context_length_exceeded`.

On `/v1/decisions`, the message carries the provider's words:

- Cloudflare refuses a Clef or Clef-flash request that it estimates above 65,536 tokens (its 413, code 5021).
- Perplexity refuses a Perplexity Decider v1 27B question whose prompt (the state and that question) is above 262,144 tokens (its 400).

The class stays `context_overflow` in `error_type` and in `x-tm-error-code`. RouterPlus does not send the request to another provider and bills nothing. `context_length` in [`/api/models.json`](https://demo.routerplus.com/api/models.json) gives each model's window.

## What to do, per class

Coding agents can follow this table as written. "Never blind-retry" means: change the request before you send it again.

| Class | Retry? | Remediation |
|---|---|---|
| `auth` (401 from RouterPlus) | no | Send the key as `Authorization: Bearer tm_vk_...` or `x-api-key: tm_vk_...`. Both work on every endpoint. Complete browser signup and create a key in the console at [https://demo.routerplus.com/console/keys](https://demo.routerplus.com/console/keys). |
| `auth` (relayed 401/402/403) | no | Nothing to fix on your side. This is RouterPlus's provider account. See the note under [Failover](#failover-and-what-the-error-you-see-means). |
| `invalid_request` | never blind-retry | Fix what the message names, then resend. |
| `model_not_priced` | no | Report it with the `x-request-id`. A listed model without a price is a RouterPlus data bug. |
| `context_overflow` | never blind-retry | Shorten the input (a coding agent compacts its conversation) or pick a model with a larger context. RouterPlus does not fail it over. |
| `content_policy` | never blind-retry | Change the prompt. RouterPlus never reroutes it to another provider. |
| `model_unavailable` (404) | no | Pick an id from `GET /v1/models` (or the public list at `https://demo.routerplus.com/api/models.json`). |
| `model_unavailable` (502) | yes | One retry after about 2 s. Every candidate's connection failed. |
| `model_unavailable` (503, decisions) | not soon | See [Decision capacity](#decision-capacity). `metadata.retryable` is `false`. |
| `not_found` | no | Endpoints: `/v1/chat/completions`, `/v1/responses`, `/v1/messages`, `/v1/messages/count_tokens`, `/v1/images/generations` (alias `/images/generations`), `/v1/decisions`, `/v1/models`, `/v1/usage`, `/v1/generation`, `/v1/limits`, `/v1/route`. |
| `request_too_large` | no | Trim the body under 10 MB (1 MB for a decisions request). The images route also caps the provider response at 32 MiB, but that is a 502 `gateway_error`. |
| `rate_limit` | yes | Wait `retry-after` (when the limit has room again), then retry. A capped wait can need a second try. Coding agents do this themselves ([how each one does](/docs/install#when-you-hit-a-rate-limit)). See [Rate limits & spend caps](/docs/limits) and [Limits and capacity](/docs/admission). |
| `email_not_verified` | **no** | Do not loop. Open the verify link in the signup email (an agent asks its human to). The key works within seconds of the click. |
| `usage_limited` | after `retry-after` | Honor `retry-after` once. Do not loop. If it happens again, contact support. |
| `usage_restricted` | **no** | Contact support to have the account reviewed. |
| `insufficient_quota` | **no** | Do not loop: a human must add money. Add credits, or raise the cap at [https://demo.routerplus.com/console/billing](https://demo.routerplus.com/console/billing). On a cap breach, the exact reset time is in the message and in `x-tm-cap-reset`. |
| `upstream_error` (5xx) | yes | One retry after about 2 s, then report with the `x-request-id`. RouterPlus already failed over if it could. |
| `upstream_error` (4xx) | never blind-retry | The provider rejected the request itself. The message names the provider and its status, and `x-tm-upstream-status` carries the status. |
| `gateway_error` (400) | never blind-retry | Send one output limit, a whole number from 1 to the maximum the message names (32,768 only when no maximum is declared), or send none. |
| `gateway_error` (500) | no | Report it with the `x-request-id`. |
| `gateway_error` (502, images) | yes, once | Nothing was billed. Lower `n` or the quality. |
| `gateway_error` (503) | yes | Wait `retry-after` (or about 2 s) and resend. The request never reached a provider, so a retry cannot double-bill. |
| `gateway_error` (504) | yes | Resend after about 2 s. The deadline passed while RouterPlus was trying candidates. Nothing was billed. |

### Decision capacity

A 503 `model_unavailable` on `/v1/decisions` will not clear soon. The provider's capacity comes back with its allowance or credit:

- Sage: at Levanto's next billing period.
- Clef and Clef-flash after Cloudflare's code 3036: at 00:00 UTC.

Meanwhile, use another decision model, in its own question format.

While the provider refuses, most requests to that model get `gateway_error` (503, "all deployments cooling down") instead, because each refusal counts against the provider's circuit. Treat that the same way on Sage, GLiNER-2.5-Decide, the Bespoke Nimble models, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider v1 27B and GPT-6 Luna Decisions. Cloudflare's code 3036 is a 429, and a 429 never counts.

## Failure headers

| Header | When | Meaning |
|---|---|---|
| `x-request-id` | always | Quote it in any report. Also the id for `GET /v1/generation?id=`. |
| `x-tm-error-code` | every failure | The canonical class. Not CORS-exposed: browser code reads the class from the body. |
| `x-tm-error-origin` | limit, provider and infrastructure failures | One of the origins at the top of this page. A request refused for its own shape (bad JSON, `n>1`, an unknown model) carries none. |
| `x-tm-limit-scope`, `x-tm-limit-kind`, `x-tm-limit-id` | a refused limit | The scope (`platform`, `org`, `workspace`, `principal`, `key`, `model`, `pool`, `endpoint`), the kind (`rpm`, `rps`, `rps_queue`, `tpm`, `concurrency`, `spend`, `health`, …), and the limit's id when one exists. |
| `x-tm-attempts` | after ≥1 dispatch | Physical provider attempts, failovers included. |
| `x-tm-upstream-status` | when a provider answered | The provider's own HTTP status. |
| `retry-after` | 429 / 503 | Whole seconds to wait. See [retry-after values](#retry-after-values). |
| `x-should-retry` | spend-cap 429 only | `false`. The cap frees only at `x-tm-cap-reset`. OpenAI's and Anthropic's SDKs and Claude Code obey it. A low balance does not carry it, because a balance held by your own requests in flight frees when they end. |
| `x-tm-cap-reset` | spend-cap 429 only | The exact ISO instant the cap resets (the first instant of the next UTC month). |

### retry-after values

`retry-after` is at most 59 seconds, because SDKs and coding agents ignore or stop on a longer wait.

| Case | Value |
|---|---|
| A rate limit | When its one-minute window has room again |
| A full limit on requests at once | 10 |
| A provider's own value | Honored, capped at 59 |
| A provider's 429 without a value | 10 |
| The all-cooling-down 503 | 5 |
| The ledger-journal 503 | 2 |
| `usage_limited` | 60 or 3,600 |
| `insufficient_quota` | none |

## Deliberate refusals

Five refusals that people often hit are deliberate. Each one stops RouterPlus from billing you for a wrong answer.

**`n>1` is a typed 400.** The normalizer returns one choice, so RouterPlus does not accept `n: 4` and bill a garbled single-choice response.

```bash
curl -s https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"claude-haiku-4-5","n":4,"messages":[{"role":"user","content":"hi"}]}'
```

```json
{
  "error": {
    "code": "invalid_request",
    "message": "n>1 is not supported in v1; request a single choice",
    "type": "invalid_request_error",
    "metadata": { "error_type": "invalid_request" }
  },
  "request_id": "..."
}
```

**Content a model does not take is a typed 400.** A provider that does not read images can drop the image, answer the rest and still bill. So RouterPlus sends an image, file, audio or video part only to a route whose listing takes that kind. It refuses a part that no route of the model takes, before it reserves or sends anything. `GET /v1/models` lists what each model takes in `architecture.input_modalities`.

```bash
curl -s https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" -H "content-type: application/json" \
  -d '{"model":"gpt-4.1-mini","messages":[{"role":"user","content":[{"type":"text","text":"What is said in this recording?"},{"type":"input_audio","input_audio":{"data":"UklGRiQAAABXQVZF","format":"wav"}}]}]}'
```

```json
{
  "error": {
    "code": "invalid_request",
    "message": "messages[0].content[1]: model \"gpt-4.1-mini\" does not take audio input; it takes text, image and file",
    "type": "invalid_request_error",
    "metadata": { "error_type": "invalid_request" }
  },
  "request_id": "..."
}
```

**No model substitution.** An unknown model id is a 404 that echoes the string you sent. RouterPlus never swaps in a similar model. The 404 is the same for an id that never existed and one that is not served, so it does not reveal which ids exist.

**A content-policy refusal never reroutes.** RouterPlus never sends a refused request to a more permissive provider. If you get `content_policy`, change the prompt.

**A failed image generation bills nothing.** A failed, over-cap or malformed generation returns a 502. `n` above 4 on the images route is a typed 400, like `n > 1` on chat. See [POST /v1/images/generations](/docs/api-images).

## Failover, and what the error you see means

Before any output reaches you, RouterPlus fails over between deployments with zero backoff, and you see no error.

| Fails over | Never fails over |
|---|---|
| `rate_limit` | `content_policy` |
| `auth` (upstream) | `context_overflow` (reshape the request) |
| `model_unavailable` | provider 4xx rejections (the request is the problem) |
| 5xx `upstream_error` | |

If a class from the first column reaches you, RouterPlus tried every healthy candidate. `x-tm-attempts` says how many.

> [!NOTE]
> A relayed 402 (class `auth`) is never your fault. An upstream 401, 402 or 403 means RouterPlus's account with that provider has a credential or payment problem. RouterPlus treats it as a health signal: the error is failover-eligible and counts against that provider's uptime, so traffic moves away. You see it only when every deployment serving the model has the same problem. On `/v1/decisions`, most of these refusals come back as 503 `model_unavailable` instead, with `metadata.retryable: false` (see [model_unavailable](#modelunavailable)).

## Mid-stream failures (HTTP 200 already committed)

After your answer starts streaming, RouterPlus never switches providers. If the provider fails mid-answer, you get one terminal error event inside the 200 stream, and nothing after it. The OpenAI surface sends a chunk with `finish_reason: "error"` and `metadata.error_type`. The Anthropic surface sends a native `error` event with `error_type`. You decide whether to retry. [Streaming](/docs/streaming#when-a-provider-dies-mid-stream) shows the frames and where to read the usage.

## Reading the class in code

```python
import os
from openai import OpenAI, APIStatusError

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

try:
    out = client.chat.completions.create(
        model="claude-haiku-4-5",
        messages=[{"role": "user", "content": "hello"}],
    )
except APIStatusError as e:
    err = e.response.json().get("error") or {}
    kind = (err.get("metadata") or {}).get("error_type") \
        or e.response.headers.get("x-tm-error-code")
    origin = e.response.headers.get("x-tm-error-origin")
    request_id = e.response.headers.get("x-request-id")
    # kind is one of the canonical classes in the table above
```

---

Source: https://demo.routerplus.com/docs/compat.md

# Wire compatibility

This page defines how the gateway forwards, translates and drops parameters between the OpenAI and Anthropic formats, and how it counts the tokens you are billed for. A change to these rules is a change to your billing contract.

## Two surfaces, one catalog

| Surface | Endpoint | Auth |
|---|---|---|
| OpenAI-compatible | `POST https://demo-api.routerplus.com/v1/chat/completions` | `Authorization: Bearer` or `x-api-key` |
| Anthropic-compatible | `POST https://demo-api.routerplus.com/v1/messages` | `Authorization: Bearer` or `x-api-key` |
| OpenAI Responses | `POST https://demo-api.routerplus.com/v1/responses` | same |
| OpenAI Images | `POST https://demo-api.routerplus.com/v1/images/generations` | same |

- Every chat model in the catalog answers on both chat surfaces.
- Image models answer on the Images surface only.
- The Responses surface passes OpenAI's models through. Other chat models are a preview there ([POST /v1/responses](/docs/api-responses)).
- `GET https://demo-api.routerplus.com/v1/models` lists what your key can serve, in the OpenAI shape, or the Anthropic shape with an `anthropic-version` header.

Each request takes one of two routes:

- **Passthrough route:** your format matches the provider's dialect. The gateway forwards your dialect's parameters as sent, and adds `stream_options.include_usage` to a streamed OpenAI-dialect dispatch. The provider decides what it accepts, including Anthropic prompt caching (`cache_control`), `thinking`, `top_k` and the `anthropic-beta` values below. A top-level key outside your dialect's parameter set is dropped and recorded. Session ids and cache marks follow [Sessions and prompt caching](#sessions-and-prompt-caching).
- **Translated route:** the dialects differ. The gateway translates the request, the stream and the errors, by the rules below.

A "typed 400" below is decided per deployment. If another deployment of the model can serve the request (it speaks your dialect, or takes a content part), the request goes there. You get the 400 only when no deployment can. See [Content is never silently dropped](#content-is-never-silently-dropped).

## Parameter handling on passthrough routes

Any other top-level key is dropped and recorded.

| Dialect | Forwarded as sent |
|---|---|
| OpenAI → OpenAI-dialect provider | `model`, `messages`, `stream`, `stream_options`, `temperature`, `top_p`, `max_tokens`, `max_completion_tokens`, `stop`, `n`, `tools`, `tool_choice`, `parallel_tool_calls`, `response_format`, `reasoning_effort`, `seed`, `user`, `logprobs`, `top_logprobs`, `frequency_penalty`, `presence_penalty`, `logit_bias`, `metadata`, `store`, `prediction` |
| Anthropic → Anthropic-dialect provider | `model`, `messages`, `system`, `max_tokens`, `stream`, `temperature`, `top_p`, `top_k`, `stop_sequences`, `tools`, `tool_choice`, `metadata`, `thinking`, `output_config`, `cache_control`, `context_management` (to Anthropic's own API; on Bedrock it is dropped and recorded) |

The GPT-6 and GPT-5.6 models are translated to OpenAI's Responses API (see [GPT-6 and GPT-5.6 on the chat surface](#gpt-6-and-gpt-56-on-the-chat-surface)).

### What an answer from an OpenAI-dialect provider keeps

A chat completion or stream chunk from an OpenAI-dialect provider keeps only OpenAI's fields and the gateway's documented extensions. Other fields a provider adds at the top level, on a choice or in `usage` (its own cost, token accounting or host name) are dropped.

| Where | Kept |
|---|---|
| Top level | `id`, `object`, `created`, `model`, `choices`, `usage`, `system_fingerprint`, `service_tier`, `obfuscation`, `error` |
| Each choice | `index`, `message` or `delta`, `finish_reason`, `logprobs`, `native_finish_reason`, `error` |
| `usage` | `prompt_tokens`, `completion_tokens`, `total_tokens`, `prompt_tokens_details` (`cached_tokens`, `audio_tokens`, `cache_write_tokens`), `completion_tokens_details` (`reasoning_tokens`, `audio_tokens`, `accepted_prediction_tokens`, `rejected_prediction_tokens`), and `cost`: the gateway's, never the provider's |

- The message and delta pass as written: content, reasoning (`reasoning_content`, `reasoning`, `reasoning_details`), tool calls and annotations.
- `model` names the model you asked for. The provider's own id, if different, is in the `x-tm-upstream-model` header. A provider sent your own id can answer with a dated snapshot (`gpt-4o-mini-2024-07-18`).
- Through your own provider key ([BYOK](/docs/byok)), the answer passes as sent.
- A [closed dedicated endpoint](/docs/dedicated-endpoints#closed-endpoints) keeps a smaller set of fields.

### Provider-specific keys

- The `reasoning` object (`{"enabled": false}`, `{"effort": "low"}`) is dropped and recorded. Send `reasoning_effort`. DeepInfra takes `"none"` (its `"enabled": false`), `"low"`, `"medium"` and `"high"`.
- `prompt_cache_retention` and `safety_identifier` go to OpenAI and Azure only.
- A `cache_control` mark on a content part passes unchanged to every OpenAI-dialect provider. OpenAI and Azure ignore it.
- On Bedrock, a top-level `cache_control` becomes a mark on the last block that can carry one.
- On every route, a top-level `cache_control` that Anthropic would refuse is dropped and recorded as `cache_control`. Anthropic refuses:
  - a field other than `{"type": "ephemeral"}` with an optional `ttl` of `"5m"` or `"1h"`
  - a fifth mark
  - a 1-hour field after a 5-minute mark
  - a field whose TTL differs from the mark on the last block
- `mcp_servers` and `container` are a typed 400 on every route.

### The `anthropic-beta` header

The gateway forwards to Anthropic-dialect providers only the `anthropic-beta` values that do not change how a token is billed:

- On every Anthropic-dialect provider: the prefixes `prompt-caching`, `token-efficient-tools`, `fine-grained-tool-streaming`, `interleaved-thinking`, `claude-code`, `oauth` and `computer-use`.
- On Anthropic's own API, also these values for Claude Code on the Claude 5 models:
  - the prefixes `per-turn-control` and `mid-conversation-output-config` (effort inside a message)
  - `context-management` (context editing; compaction's own beta is not forwarded)
  - `mid-conversation-tool-changes` and `inline-tools` (tools added during a conversation)
  - `thinking-display-updates` (readable progress updates)

Any other value is dropped and recorded as `header:anthropic-beta:<value>`. Examples: `advisor-tool` (a second model at its own rates), and the auto-mode values `dangerous-tool-use` and `afk-mode` with the `safeguards` field (their result field is not passed back).

On Anthropic's direct API, Claude Code's `thinking: {"type":"adaptive","display":"updates"}` goes with its `thinking-display-updates-2026-08-18` beta, and progress updates and signatures survive streaming. Bedrock drops `thinking.display: "updates"` and that beta, and keeps the thinking mode. Both drops are reported, and `require_parameters` refuses the request.

## Parameter handling on translated routes

### OpenAI surface → Anthropic-dialect provider

| Parameter | Handling |
|---|---|
| `model`, `messages`, `stream`, `temperature`, `top_p` | Translated or copied as sent |
| `max_tokens`, `max_completion_tokens` | Translated. With neither, the gateway sets the model's declared maximum output (32,768 when none is declared), fitted to its context window. See [POST /v1/chat/completions](/docs/api-chat-completions) |
| `stop` (string or array) | → `stop_sequences` |
| `system` / `developer` messages | → the `system` string, joined with blank lines. If a part has a `cache_control` mark, `system` becomes text blocks, and each mark ends a block |
| `cache_control` on a text part | Kept on the Anthropic block. A `role:"tool"` message's mark goes on its `tool_result` |
| Other keys of a text part | Not forwarded, not recorded |
| `cache_control` (top-level) | → the Anthropic top-level field (automatic caching). On Bedrock it becomes a mark on the last block |
| `tools`, `tool_choice` | `auto`→`{type:"auto"}`, `required`→`{type:"any"}`, `{function:{name}}`→`{type:"tool",name}`, `none`→`{type:"none"}`. With `none` and no tools in the history, tools are left out, so you do not pay for them. A tool without `parameters` gets an empty object schema. The Claude 5 models refuse `any` and `tool` (a forced tool choice) with a 400 |
| `role:"tool"` messages | → `tool_result` blocks. Consecutive results merge into one Anthropic user message |
| Tool-call ids | Kept when valid for Anthropic (letters, digits, `_`, `-`, at most 64 characters). Others become `toolu_` and a hash of the id, the same in the call, its result and every turn, so pairs match and the cached prompt start is stable. Answers carry the provider's own ids |
| `image_url` parts in user messages | → `image` blocks. A base64 data URL (JPEG, PNG, GIF or WebP, typed by its bytes) becomes a base64 source. An `http(s)` URL becomes a URL source that Anthropic fetches. A host that refuses Anthropic's fetcher gives a 400, with no failover. A `cache_control` mark is kept. A `detail` other than `auto` is dropped and recorded. Another image type is a typed 400 naming the part |
| `file` parts in user messages | A PDF sent as a data URL → a `document` block, with the file's `filename` as its title. A `file_id`, or a file of another type, is a typed 400 naming the part |
| An assistant message with `content: null` and no `tool_calls` (OpenAI's refusal shape) | Dropped and recorded |
| `stream_options` | Usage is on. An explicit `include_usage: false` removes the usage chunk, and you are still billed. It is not forwarded and not named in `x-tm-dropped-params` |
| `n` | `n > 1` is a 400 `invalid_request` on `/v1/chat/completions` and `/v1/messages`. The images route takes `n` up to 4 (see [/docs/api-images](/docs/api-images)) |
| `temperature` | Forwarded when ≤ 1. `temperature > 1` is a typed 400, never clamped (Anthropic's range is 0 to 1) |
| `top_p` | Forwarded only without `temperature` (Anthropic takes one of the two). Otherwise dropped and recorded |
| `parallel_tool_calls: false` | → `tool_choice.disable_parallel_tool_use: true` |
| `tools[].function.strict` | Mapped 1:1 |
| `response_format`, `prediction`, `audio`, `modalities` | **Typed 400** naming the parameter. Not mapped across dialects yet |
| `reasoning_effort` | → `output_config.effort` at the same level (`low` to `max`; `minimal` → `low`). `none` → `thinking: {"type": "disabled"}` on Claude Haiku 5.5, Sonnet 5 and Opus 5, and `low` on the other Claude 5 models. Without it, thinking is the model's default. On Claude Sonnet 4.5 and Haiku 4.5 it is **dropped and recorded** (`none` is not recorded). A value outside OpenAI's levels is a **typed 400** |
| Trailing `assistant` message | **Typed 400** (an assistant prefill, which current Claude models reject) |
| Everything else (`logprobs`, `seed`, penalties, …) | **Dropped and recorded**, never forwarded |

### Anthropic surface → OpenAI-dialect provider

| Parameter | Handling |
|---|---|
| `model`, `stream`, `temperature`, `top_p` | Copied as sent |
| `max_tokens` | → `max_completion_tokens` (o-series and GPT-5 reasoning models reject `max_tokens`). When absent, set to the model's maximum, as above |
| `system` (string or text blocks) | → one leading `system` message |
| `stop_sequences` | → `stop` |
| `messages` | `tool_use` ↔ `tool_calls`, `tool_result` blocks → `role:"tool"` messages (one per result). A `system` message inside the conversation stays at its place. See [below](#reasoning-and-structured-output-on-openai-dialect-routes) |
| `image` blocks in user messages | → `image_url` parts, from a base64 source or a URL source. An image in an assistant message or a tool result, or an image given by a file id, is a typed 400 naming the block |
| `document` blocks in user messages | A base64 PDF → a `file` part. The block's `title`, `context` and `citations` are dropped and recorded. A document given by URL or as text is a typed 400 naming the block |
| `tools`, `tool_choice` | `auto`→`"auto"`, `any`→`"required"`, `none`→`"none"`, `{type:"tool",name}`→`{function:{name}}`. `disable_parallel_tool_use: true` → `parallel_tool_calls: false`. `tools[].strict` maps 1:1. An empty `tools` list is sent as no list, without `auto`, `none` or the parallel flag. Server tools (computer use, web search, bash, text editor) are a **typed 400** |
| `thinking` | `adaptive`, `enabled` → the model's own reasoning stays on (`budget_tokens` **dropped and recorded**). `disabled` → `reasoning_effort: "none"`. A `type` or `display` value the gateway does not know is a **typed 400**. See [below](#reasoning-and-structured-output-on-openai-dialect-routes) |
| `output_config` | `effort` → `reasoning_effort`. `format` `json_schema` → a strict json_schema `response_format`. Another `format` type is a **typed 400** |
| `top_k`, `metadata` | **Dropped and recorded**. On a house route, `metadata.user_id` becomes the end-user id and is not recorded (see [Sessions and prompt caching](#sessions-and-prompt-caching)) |
| `cache_control` (nested and top-level) | **Dropped and recorded with the full path**, for example `messages[0].content[0].cache_control` |
| `tool_result.is_error` (nested) | **Dropped and recorded with its full path** |

> [!NOTE]
> Every drop is **recorded** per request, with its full path, in the `x-tm-dropped-params`
> header, in each attempt's `dropped_params` in [`GET /v1/generation`](/docs/api-usage),
> and in the ledger (which holds no content). With `"provider": {"require_parameters":
> true}`, the first drop is a typed 400 instead. Only parameters are dropped, never content.

### Reasoning and structured output on OpenAI-dialect routes

When an OpenAI-dialect provider serves an Anthropic-format request (GLM 5.3, Kimi K3, DeepSeek and the other open models), reasoning and output controls map to these OpenAI fields:

| You send | The provider gets |
|---|---|
| `thinking: {"type": "adaptive"}` or `{"type": "enabled", ...}` | Nothing: the model reasons by default. `budget_tokens` is dropped and recorded |
| `thinking: {"type": "disabled"}` | `reasoning_effort: "none"`. An `output_config.effort` beside it is dropped and recorded |
| `output_config.effort` | `reasoning_effort` at the same level, or the nearest level a provider declares. A system message's own effort applies from its place on, so the last one wins |
| `output_config.format` with `type: "json_schema"` | `response_format: {"type": "json_schema", "json_schema": {"name": "output", "schema": <your schema>, "strict": true}}`, only on providers that take json_schema. If none does, a typed 400 names `output_config.format` before anything is held or sent |
| A `system` message inside `messages` | A `system` message at the same place. A turn-scoped one (`clear_at: "next_user_message"`) is left out once a user message follows it. One with no content only sets effort. A tool added or removed in one that still shows is a typed 400 |

`thinking.display` sets how reasoning comes back. An open model's reasoning is its raw chain of thought.

- `"omitted"` and `"updates"`: each `thinking` block has an empty `thinking` field, as on Anthropic's API, and the text travels in its `signature`. Send the block back unchanged: Morph's and Mistral's routes read the reasoning from it. Under `"updates"`, Claude Code shows such a block as a progress line, without the raw reasoning.
- `"summarized"` or no `display`: readable `thinking` blocks with the reasoning text.

Dropped and recorded:

- `thinking` and the effort, when a provider serves the model with thinking off (for example Qwen3.8 Max on DeepInfra) or the model does not reason (Mistral's Codestral, Large 3 and Ministral 3).
- `context_management` (context editing) and `safeguards` (Claude Code's auto-mode classifier context).

### GPT-6 and GPT-5.6 on the chat surface

The GPT-6 and GPT-5.6 models (GPT-6 Astra, GPT-6.1 Sol, GPT-6 Sol, GPT-6 Luna, GPT-5.6 Sol, GPT-5.6 Terra and GPT-5.6 Luna) take tools only on OpenAI's Responses API. You call them on `/v1/chat/completions` or `/v1/messages`, and the gateway translates. This applies on the marketplace route and on your own OpenAI connection.

| Chat field | Handling |
|---|---|
| `messages` | → Responses input items: text, image and file parts; an assistant's tool calls as `function_call` or `custom_tool_call` items; a tool message as the matching output item. Audio parts are a typed 400 |
| `max_tokens`, `max_completion_tokens` | → `max_output_tokens`. A limit under 16 is sent as 16 (OpenAI's floor) and recorded as dropped |
| `tools` with `type: "function"` | → function tools. A tool without `strict` is sent with `strict: false`, the Chat Completions default |
| `tools` with `type: "custom"` | → custom (free-form) tools, with a `format` of `text` or a `grammar` in `lark` or `regex` syntax. Both the chat form (`{type: "custom", custom: {name, description, format}}`) and Cursor's flat form work. A call comes back as a `tool_calls` entry `{id, type: "custom", custom: {name, input}}`. Send its result as a `role:"tool"` message. `input` must be a string |
| `tool_choice` | → the Responses form. It may name a custom tool |
| Tool-call ids | An id over 64 characters becomes `call_` and a hash of it, the same in the call and its result |
| `response_format`, `reasoning_effort` | → their Responses forms |
| `store` | Sent as `false` unless you set it |
| `n`, `stop`, `seed`, `logprobs`, `top_logprobs`, `logit_bias`, `prediction`, the penalties | Dropped and recorded. A value equal to the default (`n: 1`, a penalty of 0) is dropped without a record |
| `temperature`, `top_p` | Dropped and recorded, except `temperature: 1` and `top_p: 1`. These models take only their default sampling |

The answer is a chat completion, or `chat.completion.chunk` records on a stream. `finish_reason` is `tool_calls` after a tool call and `length` at `max_output_tokens`. A short GPT-6 commentary before the answer is joined to it with a blank line. Reasoning does not carry from one turn to the next.

### Content is never silently dropped

A provider that drops a content part still bills the request, so the gateway never drops content.

**A part goes only to a model that takes it.**

- Each listing declares what a message may carry: text, plus any of image, file (a document such as a PDF), audio and video. `GET /v1/models` shows it for your key in `architecture.input_modalities`. A model's page lists only the inputs that every listing takes.
- The gateway sends a part only to a deployment whose listing takes its kind. A part that no deployment takes is a **typed 400 naming the exact field**, on both surfaces, before anything is reserved or sent: `messages[0].content[1]: model "gpt-4.1-mini" does not take audio input; it takes text, image and file`.
- Checked parts: `image_url`, `file`, `input_audio` and `video_url` (or `input_video`) on the OpenAI surface; `image` and `document` blocks on the Anthropic surface, also inside a `tool_result`.
- A BYOK route declares nothing and gets every part as sent.

**A translation names what it cannot carry.** Images (JPEG, PNG, GIF and WebP) and PDFs in user messages cross between the formats, as the tables show. On a translated route, this content is a **typed 400 naming the exact field** (`messages[2].content[0]` and so on):

- audio parts
- an image or a file given by file id
- a file other than a PDF
- an image in an assistant message or a tool result
- unknown block types

**Exception: echoed thinking blocks.** On the Anthropic surface, assistant `thinking` and `redacted_thinking` blocks that you echo back are dropped, not rejected. The gateway itself emits them for an OpenAI-dialect model's reasoning (on a stream, and as a leading block on a non-streamed reply).

- Thinking is never forwarded to an OpenAI-dialect provider.
- A signed `thinking` block (Claude's own) and a `redacted_thinking` block are named in `x-tm-dropped-params`.
- An unsigned block, or one with the gateway's own signature (text hidden by `thinking.display`), is dropped without a record, and `require_parameters` does not refuse the next turn for it.
- On an Anthropic-dialect provider, the gateway's own blocks are left out without a record, because Anthropic verifies every signature. A conversation served elsewhere while Anthropic was unavailable can continue on Anthropic.

## Sessions and prompt caching

A provider keeps a prompt's cache on one machine. The gateway passes on what each provider needs to route a conversation back to it.

**Session id.** The gateway takes the first valid value of these inputs, in this order:

| Order | Input | Where |
|---|---|---|
| 1 | `session_id` | Body |
| 2 | `x-session-id` | Header |
| 3 | `x-session-affinity` | Header |
| 4 | `x-claude-code-session-id` | Header |

- A valid value has 1 to 256 printable ASCII characters. The body field `prompt_cache_key` follows the same rule.
- An invalid value is ignored, with no error. The gateway uses the next input and records the ignored one as `session_id (invalid)`, `prompt_cache_key (invalid)` or `header:<name> (invalid)`. With `provider.require_parameters: true`, it is a 400.
- A valid session input is never recorded as dropped.

Each route gets its own field:

| Route | Field sent | Value |
|---|---|---|
| OpenAI (house or BYOK), Azure | `prompt_cache_key` | Your `prompt_cache_key`, else the session id. Nothing if you sent neither |
| Morph | Header `x-session-id` | The session id, else your `prompt_cache_key`. Nothing if you sent neither. Morph picks region and worker by it |
| Anthropic, Bedrock | Nothing | Its cache matches the prompt prefix |
| DeepInfra | Nothing | Its cache matches the prompt prefix without a key |

- Values go unchanged. A value longer than the provider accepts is replaced by `tm-` followed by 43 base64url characters: a hash of the value.
- `prompt_cache_retention` (`"in_memory"` or `"24h"`) goes unchanged to OpenAI and Azure. Other routes drop and record it.

**End-user ids on house routes.** A house route runs on the marketplace's provider account, shared by all customers. It gets one end-user id: a hash of your organization and your `safety_identifier` (else `user`, else, on `/v1/messages`, `metadata.user_id`), or of your organization alone.

| House route | Field sent | Not sent |
|---|---|---|
| `openai` | `safety_identifier` | `user` |
| `anthropic` | `metadata.user_id` | `user`, `safety_identifier` |

BYOK routes get your values unchanged: `user` and `metadata`, and `safety_identifier` on OpenAI and Azure from `/v1/chat/completions`. Anthropic and Bedrock BYOK routes, and any translation from `/v1/messages`, drop and record `safety_identifier`.

**Cache marks.**

- The gateway never adds a `cache_control` mark to a block or a part.
- On a request with no `cache_control` at all, the gateway may add the top-level field ([Automatic caching on Claude](#automatic-caching-on-claude)).
- The ledger prices every cache write at the 5-minute rate. On a house route, `ttl: "1h"` is removed from every mark and recorded as `<path>.cache_control.ttl`, for example `system[0].cache_control.ttl`. `ttl: "5m"` passes. BYOK routes keep `ttl: "1h"`.

**Staying on the first route.** When the gateway's quota pool for your first route is full, a follow-up request (an assistant or tool message after a user message) waits up to 5 seconds for that route, so it does not move to a cold cache. `x-tm-affinity-wait-ms` gives the milliseconds waited.

### Automatic caching on Claude

Coding agents that call Claude through an OpenAI-compatible setting (Cline, Cursor, Kilo Code, OpenCode and others) send no cache marks. The gateway turns on Anthropic's automatic caching when all of these are true:

1. Anthropic's own API serves it: the marketplace's `anthropic` route, or your own Anthropic connection. Not Bedrock or dedicated endpoints.
2. The request has no `cache_control` anywhere (top level, system prompt, tool, message, or a block inside a tool result).
3. It has tools, or an assistant turn before its last message. A single question without tools is left alone.
4. The prompt (system prompt, tools and messages, at four bytes a token) reaches the model's minimum cacheable length:
   - 512 tokens on Claude Opus 5.5, Opus 5, Fable 5.1, Fable 5, Sonnet 5.5 and Haiku 5.5
   - 1,024 on Claude Sonnet 5 and Sonnet 4.5
   - 4,096 on Claude Haiku 4.5
5. You did not turn it off with the request header `x-tm-auto-cache: off` (any letter case).

The gateway then adds only the top-level `cache_control: {"type": "ephemeral"}`. Anthropic puts one cache point on the last cacheable block and moves it forward as the conversation grows. The cache lasts 5 minutes, and the answer does not change. The first turn pays a cache write (1.25 times the input price). Later turns read at the cache-read price, a tenth of the input price or less.

A response from a Claude route carries `x-tm-auto-cache`: `applied`, or `skipped:` and the reason (`opted-out`, `client-markers`, `single-turn`, `short-prompt`, `bedrock`, `dedicated` and a few more). A closed dedicated endpoint never sends it. `x-tm-dropped-params` does not change.

**Cache privacy.** Marketplace requests reach Anthropic from one RouterPlus account, and Anthropic keeps caches per account. A cache read needs a prompt start that is byte for byte the same as an earlier one, which can be another customer's. So someone who knows your exact prompt start could tell from their own cache reads that it was sent in the last few minutes. This applies also to requests with their own marks. To avoid it, send `x-tm-auto-cache: off` and no marks.

## Usage accounting (how tokens are counted)

| Concept | OpenAI surface | Anthropic surface |
|---|---|---|
| Billable input | `prompt_tokens`, cache-inclusive: input + cache reads + cache writes | `input_tokens`, which excludes cache reads and writes |
| Cache reads | `prompt_tokens_details.cached_tokens` | `cache_read_input_tokens` |
| Cache writes | `prompt_tokens_details.cache_write_tokens` | `cache_creation_input_tokens` |
| Output | `completion_tokens` | `output_tokens` |
| Reasoning | `completion_tokens_details.reasoning_tokens` (a subset of output) | not reported separately |

Translation converts by this table. `total_tokens = prompt_tokens + completion_tokens`.

- **Usage is always on**, streamed or not. Only an explicit `include_usage: false` removes the usage chunk (see `stream_options` above).
- **Cache writes** bill at their own rate. On billed OpenAI-surface responses, `prompt_tokens_details.cache_write_tokens` is always present (0 when the provider reports none).
- **Every billed response carries `usage.cost` (USD)**, in the final usage chunk on a stream, else in the body. It uses the ledger's integer micro-USD math and covers the full debit, including attempts that failed over before your answer started.
- **Rounding.** Prices are micro-USD per million tokens. The reservation rounds up, and settlement rounds down.
- **Web search has a fee per search**, at the provider's price, on top of the tokens: Anthropic's web search server tool, and OpenAI's hosted web search on `POST /v1/responses`. The fee is inside `usage.cost`. See [Pricing & billing](/docs/pricing#web-search).

> [!WARNING]
> An Anthropic-surface stream that dies mid-answer never sends `message_delta`, its only
> usage slot. Get those requests from `GET https://demo-api.routerplus.com/v1/generation?id=REQUEST_ID`,
> which returns every attempt with tokens and settled cost. The OpenAI surface emits known
> usage and cost *before* the terminal error event.

## Streams

Event order on the OpenAI surface:

1. A role-priming delta, sent as soon as the upstream responds.
2. Content deltas.
3. The finish chunk.
4. The usage chunk.
5. `data: [DONE]`.

The gateway never sends `[DONE]` after an error. Nothing follows a terminal event.

- **Keep-alives.** After output starts, each silence of 15 seconds or more gets a keep-alive frame, so proxies do not close a slow stream: an SSE comment (`: processing`) on the OpenAI surface, a native `ping` event on the Anthropic surface. SDKs skip both.
- **The commit boundary.** Before output starts, the gateway fails over provider failures with zero backoff, and you see one response with one event sequence. `x-tm-attempts` counts the dispatches. After output starts, the gateway **never** switches providers. A failure becomes one terminal error event inside the 200 stream, and you decide whether to retry. A refusal is never rerouted.
- **The request deadline.** 450 seconds from dispatch, or a routing policy's `timeout_ms`. A stream still open then gets the same terminal error event.
- **Mid-stream terminal shape.** On the OpenAI surface: a full chunk envelope with `finish_reason: "error"` and a top-level `error` object. On the Anthropic surface: an `event: error` frame.

## `finish_reason` and `native_finish_reason`

Translated responses map finish reasons both ways:

| Anthropic `stop_reason` | OpenAI `finish_reason` |
|---|---|
| `end_turn` | `stop` |
| `stop_sequence` | `stop` |
| `max_tokens` | `length` |
| `tool_use` | `tool_calls` |
| `refusal` | `content_filter` |

Two stop reasons both become `stop`. Translated responses on the OpenAI surface carry the raw value in `choices[0].native_finish_reason`, in the body and in the stream's finish chunk. Use it for analytics. Passthrough responses are relayed as sent.

## Errors

The gateway writes every error body in the dialect you called with.

- Each body carries a stable `error_type` with the canonical class: `error.metadata.error_type` on the OpenAI surface, `error.error_type` on the Anthropic surface.
- Each body also carries the dialect's native fields, so SDK error handling and retries work. On the Anthropic surface these are the native type strings (`billing_error`, `rate_limit_error`, `authentication_error`, `not_found_error`, `api_error`, `invalid_request_error`).
- The gateway never relays a provider's raw error body. The message carries the provider's own message, up to 1,000 characters, with keys, account ids and host names removed (never on a closed dedicated endpoint).
- `x-tm-upstream-status` carries the provider's HTTP status.
- The body's `metadata` names where the failure came from (`origin`, and the limit that refused it when one did). See [Errors](/docs/errors).

A prompt too long for the model is a 400 in the surface's standard shape, which coding agents compact on: code `context_length_exceeded` with OpenAI's wording, or `prompt is too long: N tokens > M maximum` on the Anthropic surface. The class is `context_overflow` (see [Context overflow](/docs/errors#context-overflow)).

Canonical classes and how routing treats them:

| Upstream signal | `error_type` | Failover |
|---|---|---|
| 429 (Retry-After honored, capped at 59 s; 10 s when the provider names none) | `rate_limit` | yes |
| 401/402/403 from a provider (a problem with the RouterPlus account, not yours) | `auth` | yes |
| Context/length errors | `context_overflow` | no: fix the request or pick a bigger window |
| Content policy / refusal | `content_policy` | **never**: a refusal is not rerouted |
| 404 / model not found upstream | `model_unavailable` | yes |
| 5xx / 529 / overloaded | `upstream_error` | yes |
| Other 4xx | `upstream_error` | no |

On `POST /v1/decisions`, some signals map differently. The gateway reads them by their status and the provider's own error code, never by the provider's name:

- These are a 503 `model_unavailable` with `metadata.retryable: false`: Levanto's 402, Fastino's 402 or 403, a System One provider's 401, 402 or 403 on the RouterPlus account, and Cloudflare's 429 with code 3036 (the free daily allocation is used up).
- Cloudflare's 413 with code 5021 is a 413 `context_overflow`.
- On Perplexity Decider v1 27B, Perplexity's 400 that names the model's maximum context length is a 400 `context_overflow`, by the context rule above.
- Timeouts: a System One provider's 408 is a retryable 504 `upstream_error`.
- OpenAI's 404 `model_not_found` on GPT-6 Luna Decisions means RouterPlus's access to the model was withdrawn. It is a 404 `model_unavailable` by the 404 rule above, with no second provider to try.
- A decisions provider's 429 never counts against the deployment's health circuit.

On GPT-6 Luna Decisions, OpenAI can decline one question and answer the rest. That is not a `content_policy`
error. The response is a 200, that question's answer is `{"type": "refusal"}`, and the call is billed. See [POST /v1/decisions](/docs/api-decisions).

Out of credits is a 429 `insufficient_quota`, which OpenAI's SDKs back off on. Error bodies never echo flagged input content. [Errors](/docs/errors) lists the fix for each status.

## Response headers

| Header | When | Meaning |
|---|---|---|
| `x-request-id` | every call | the id that `GET /v1/generation?id=` audits |
| `x-tm-provider` | served requests | the deployment that served the answer |
| `x-tm-attempts` | served or failed dispatches | physical dispatches, failovers included |
| `x-tm-upstream-status` | when an upstream responded | the provider's own HTTP status |
| `x-tm-error-code` | failures | canonical class |
| `x-tm-error-origin` | limit, provider and infrastructure failures | where it came from: `gateway_admission`, `upstream_quota`, `upstream`, `gateway_infrastructure` or `authorization` |
| `x-tm-limit-scope`, `x-tm-limit-kind`, `x-tm-limit-id` | a refused limit | which limit refused the request (see [Limits and capacity](/docs/admission)) |
| `x-tm-cap-reset` | spend-cap 429s | when the monthly cap resets (the first of next UTC month) |
| `x-should-retry` | spend-cap 429s | `false`: do not retry before `x-tm-cap-reset`. OpenAI's and Anthropic's SDKs and Claude Code obey it |
| `x-tm-auto-cache` | a Claude route served the request (never on a closed dedicated endpoint) | `applied`, or `skipped:` and the reason (see [Automatic caching on Claude](#automatic-caching-on-claude)) |
| `x-tm-dropped-params` | when not empty | comma-joined names of the parameters the gateway removed. Every drop is recorded |
| `x-tm-upstream-model` | when the route knows the model by another id | the id the gateway sent upstream |
| `x-tm-served-by` | when a route reports one | the provider that ran the request |
| `x-tm-route-plan-id` | billed requests | the id of the route plan that chose the deployments |
| `x-tm-admission-mode` | billed requests | `shared`, or `bounded_local` while the shared limit store is unreachable |
| `x-tm-remaining-rpm`, `x-tm-remaining-tpm` | admitted requests | the smallest remaining allowance across the limits the request claimed |
| `retry-after` | 429/503 | whole seconds to wait, at most 59 (`usage_limited` sends 60 or 3,600). A provider's value is honored, capped at 59. A provider's 429 without one gets 10 |

Browsers can read every header above except `x-tm-error-code` and `x-tm-route-plan-id`, which are not CORS-exposed. In browser code, use the body's `error_type`.

Optional: send `HTTP-Referer` and `X-Title` to identify your app in per-app analytics. They carry no content.

## Not supported

- Audio and video input, and files other than PDFs, across dialects (see [Content is never silently dropped](#content-is-never-silently-dropped)).
- Image edits, variations and streaming (see [POST /v1/images/generations](/docs/api-images)).
- `n > 1` on the chat surfaces.
- Model id variants. An id that contains `:` is not in the catalog and returns 404. The gateway never substitutes another model.
- On passthrough routes, the provider decides how to handle forwarded parameters. RouterPlus guarantees only what this page documents.

See also: [Migrate](/docs/migration) · [Quickstart](/docs/quickstart) · [Errors](/docs/errors)