# Routing & failover

You name a model. The gateway picks a deployment that serves it, and fails over to another
deployment when a provider fails. Three rules apply to every request:

1. **You get the model you name.** An unknown model is a 404 that names the id. The gateway
   never substitutes another model.
2. **Failover happens only before your answer starts.** Once the first token of real output
   reaches you, the request stays with that provider.
3. **The gateway never reroutes a content-policy refusal** to another provider.

To change the provider order or filter providers, see [Provider routing](/docs/provider-routing).

## How a request picks a provider

A **deployment** is a provider endpoint with a wire dialect, the model ids it serves and a
routing `priority`. Today each Claude and GPT model has one deployment: the lab's own API
(`anthropic` for Claude, `openai` for GPT). Routing runs in memory and makes no remote calls.
Billed requests still check the ledger and the shared admission store for authentication,
limits and the reserve check.

1. **Model filter.** Keep deployments that list the requested id. The match is exact, with two
   exceptions:
   - A dated snapshot of a listed id (`claude-haiku-4-5-20251001`) routes and bills as its
     family. The provider still gets your original id.
   - The `rp-` name of a Claude model (`rp-sonnet-5-5`, for Cursor) routes, bills and goes to
     the provider as that model. See [Models & catalog](/docs/models#claude-names-for-cursor).
2. **Health filter.** Drop deployments whose circuit is open.
3. **Order.** The base order is cheapest first: input plus output price per million tokens, at
   each provider's live price.
   - Equal prices keep ascending `priority`, then catalog order.
   - A route billed at the cost its provider reports (some image and video models) has no price
     to compare. It comes after the priced routes.
   - The providers that `provider.order` names go first, in its order.
   - The [routing strategy](/docs/provider-routing) then sets the final order.
   - A key under a [routing policy](/docs/routing-policies) keeps the policy's saved order.

The gateway tries the list in order. The first deployment to produce real output wins. A
failure that can fail over moves to the next deployment with no backoff. The gateway translates
requests, streams and errors between dialects, so you can call any listed model from either
surface.

A key with a BYOK connection or a routing policy runs the same steps over its own routes, with
the policy's order, funding rule and attempt limit on top. `POST /v1/route` shows the plan
without sending the request.

A [dedicated endpoint](/docs/dedicated-endpoints) (`<your-namespace>/<name>`) routes first to
capacity reserved for your organization, then to public deployments of the same model that meet
its constraints. It never routes to a different model. A
[closed](/docs/dedicated-endpoints#closed-endpoints) endpoint names RouterPlus as the provider,
whichever route served.

## The commit boundary

**A request commits to a deployment at its first semantic output:** the first text delta,
reasoning delta, tool call, refusal or usage report that reaches you.

**Before commit**, failover is possible only while the gateway has written nothing to you.

- A provider that fails before its first frame is skipped, and the next one is tried at once.
  This covers a refused connection, a 5xx, a failure before any bytes, and a miss of the
  20-second headers deadline.
- The gateway sends your response headers only when a live attempt starts writing.
- After the headers and (on the OpenAI surface) the role-priming delta go out, the gateway does
  not switch providers. A later failure becomes the terminal in-stream error in
  [Streaming](/docs/streaming), even with no semantic output.

**After commit**, the gateway never switches providers. If the provider fails mid-answer, you
get the tokens it produced plus one terminal error event inside the 200 stream. You decide
whether to retry. A retry starts fresh.

> [!NOTE]
> You pay for every settled attempt, including attempts that failover discarded before commit.
> A provider that fails after it reports prompt usage still bills RouterPlus for that prompt,
> and pass-through pricing passes the charge on. The inline `usage.cost` is the **full request
> debit** across all attempts, so it matches the ledger. `x-tm-attempts` gives the number of
> physical dispatches.

## No model substitution

A model that is not in the catalog returns a 404 that names it:

```json
{
  "error": {
    "code": "model_unavailable",
    "message": "model \"gpt-5-ultra\" is not in the catalog; GET /v1/models lists what this key can serve",
    "type": "invalid_request_error",
    "metadata": { "error_type": "model_unavailable" }
  },
  "request_id": "…"
}
```

There is no closest-match fallback and no default model. `n>1` returns a typed 400, because the
gateway sends every stream as a single choice.

## What never fails over

The gateway decides whether an error can fail over once, when it normalizes the error. Routing
never reads raw provider errors.

- **`content_policy`** never retries and **never** goes to another provider. This check runs
  before every status-code rule. Error metadata on this path never echoes your flagged input.
  To try a different provider, send a new request.
- **`context_overflow`** never fails over, because every deployment of a model has the same
  context window. Shorten the request or choose a model with a larger context.
- Other buyer-fault 4xx errors, such as a malformed request, return directly.

## Failover eligibility by error class

| Upstream signal | `error_type` | Retryable | Fails over |
|---|---|---|---|
| 429 (Retry-After honored, capped 60s) | `rate_limit` | yes | yes |
| 401 / 402 / 403 from the provider | `auth` | no | **yes** |
| Context / length errors | `context_overflow` | no | no |
| Content policy / refusal shapes | `content_policy` | no | **never** |
| 404 / model_not_found upstream | `model_unavailable` | no | yes |
| 5xx / 529 / overloaded | `upstream_error` | yes | yes |
| Other 4xx | `upstream_error` | no | no |

> [!NOTE]
> A 401, 402 or 403 **from a provider** is a problem with RouterPlus's account at that
> provider. The gateway routes around the deployment and counts the error against that
> provider's health. You see an `auth` error only if every candidate failed.

## Health circuits and cooldowns

Each deployment has its own circuit breaker:

- **Two failures in a row open the circuit** for a 30-second cooldown. Routing skips the
  deployment while the circuit is open.
- **After the cooldown the circuit goes half-open.** One live request goes through as a probe.
  Success closes the circuit. Failure opens it again with a new cooldown.
- **Only failures that can fail over count.** Buyer-fault errors (400/413-class, content
  policy, context overflow) never count.
- An upstream 429 counts against the circuit. Uptime scoring excludes it as buyer-caused. It
  also pauses that provider's admission pool for the provider's `retry-after`, 1 to 60 seconds.
- A client cancellation does not count.
- **A hanging provider counts as a failure.** This applies when the provider sends no response
  headers for 60 seconds (20 seconds on a stream) and the request then ends, by the gateway's
  deadline or your cancellation.
  - The attempt counts on the deployment's circuit and on a second circuit for that model at
    that deployment.
  - Two such attempts in a row pause the provider for that model for 30 seconds. Its other
    models keep answering. Then one request probes it.

Public uptime numbers use the same rules. A model page shows no uptime percentage for a
provider RouterPlus calls directly until it has 100 counted requests in the window.

## When everything is cooling down

If every deployment of your model has an open circuit, the gateway refuses the request at once,
with no queue. The response is a `503` with:

- `retry-after: 5`
- `x-tm-error-code: gateway_error`
- `x-tm-limit-kind: health`
- the message `all deployments cooling down`

A cooldown lasts 30 seconds, then one request goes through as the probe. Retry after a few
seconds.

## Reading what happened

Every response says how it was served:

| Header | Meaning |
|---|---|
| `x-request-id` | your handle for audit and support (always present) |
| `x-tm-provider` | the deployment that served the request; `routerplus` on a closed dedicated endpoint |
| `x-tm-served-by` | the provider that ran the request, when a route reports one; absent on a closed dedicated endpoint |
| `x-tm-upstream-model` | the id sent to the provider, when it differs from the one you sent; absent on a closed dedicated endpoint |
| `x-tm-attempts` | physical dispatches, failovers included |
| `x-tm-upstream-status` | the HTTP status the provider returned |
| `x-tm-dropped-params` | request fields the gateway removed before dispatch, by name |
| `x-tm-route-plan-id` | the plan id, for correlation with `/v1/route` |
| `x-tm-dedicated-endpoint`, `x-tm-route-role` | on a dedicated endpoint: its id, and whether your dedicated capacity (`primary`) or the shared pool (`fallback`) served |
| `x-tm-queue-ms` | on a dedicated endpoint with a [paced limit](/docs/dedicated-endpoints#the-paced-limit): how long the request waited for its turn, in milliseconds |
| `x-tm-remaining-rpm`, `x-tm-remaining-tpm` | what your rate limits had left at admission |
| `x-tm-error-code` | canonical error class, on failures |
| `x-tm-error-origin` | on failures: `upstream`, `gateway_admission`, `upstream_quota`, `gateway_infrastructure` or `authorization` |
| `retry-after` | on rate limits and cooldowns; whole seconds, at most 59 (`usage_limited` sends 60 or 3,600) |

On failures the body carries the canonical class: `error.metadata.error_type` on the OpenAI
surface, `error.error_type` on the Anthropic surface. Each dialect's native type string is there
too, so your SDK's retry rules keep working. The gateway never relays a provider's raw error
body. See [Errors & remediation](/docs/errors) for the full table.

### Per-attempt audit

The generation audit endpoint returns every physical attempt for a request id: which
deployments the gateway tried, in what order, and what each one cost.

```bash
curl -s -H "Authorization: Bearer $TM_API_KEY" \
  "https://demo-api.routerplus.com/v1/generation?id=<x-request-id>"
```

In this trimmed example, a Claude request under a `byok_with_house_fallback` policy failed on
your own Anthropic connection, then succeeded on the marketplace `anthropic` deployment. Each
attempt also carries its billing source, price snapshot, error origin, admission and route
context, and dropped parameters.

```json
{
  "request_id": "2f6d8e3b-…",
  "admission_events": [],
  "attempts": [
    {
      "attempt": 1,
      "deployment": "byok:5f3a9c1e-…:0b7d42e8-…",
      "model": "claude-sonnet-5",
      "outcome": "failed",
      "usage_provenance": "unknown",
      "tokens": { "input": 0, "cached": 0, "cache_write": 0, "output": 0, "reasoning": 0 },
      "reserved_max_usd": 0,
      "cost_usd": 0,
      "error_code": "upstream_error",
      "dispatched_at": "2026-09-04T09:12:44.120Z"
    },
    {
      "attempt": 2,
      "deployment": "anthropic",
      "model": "claude-sonnet-5",
      "outcome": "completed",
      "usage_provenance": "observed",
      "tokens": { "input": 412, "cached": 0, "cache_write": 0, "output": 638, "reasoning": 0 },
      "reserved_max_usd": 0.04,
      "cost_usd": 0.007204,
      "error_code": "",
      "dispatched_at": "2026-09-04T09:12:45.010Z"
    }
  ]
}
```

The response holds metadata only. See [Data policy](/docs/data-policy).

| Field | Values |
|---|---|
| `outcome` | `completed`, `failed`, `cancelled`, `incomplete` (failed after your answer started) or `unknown_after_crash` (settled at the reserved maximum until reconciled) |
| `admission_events` | Refusals before any attempt, such as a rate limit |

Use this endpoint to recompute an Anthropic-surface stream that failed mid-answer (see
[Streaming](/docs/streaming)).