Routing & failover

You name a model. The gateway picks a deployment that serves it, and fails over to another deployment when a provider fails. Three rules apply to every request:

1. 01
   You get the model you name. An unknown model is a 404 that names the id. The gateway never substitutes another model.
2. 02
   Failover happens only before your answer starts. Once the first token of real output reaches you, the request stays with that provider.
3. 03
   The gateway never reroutes a content-policy refusal to another provider.

To change the provider order or filter providers, see Provider routing.

How a request picks a provider

A deployment is a provider endpoint with a wire dialect, the model ids it serves and a routing priority. Today each Claude and GPT model has one deployment: the lab's own API (anthropic for Claude, openai for GPT). Routing runs in memory and makes no remote calls. Billed requests still check the ledger and the shared admission store for authentication, limits and the reserve check.

1. 01
   Model filter. Keep deployments that list the requested id. The match is exact, with two exceptions:

   • A dated snapshot of a listed id (claude-haiku-4-5-20251001) routes and bills as its family. The provider still gets your original id.
   • The rp- name of a Claude model (rp-sonnet-5-5, for Cursor) routes, bills and goes to the provider as that model. See Models & catalog.
2. 02
   Health filter. Drop deployments whose circuit is open.
3. 03
   Order. The base order is cheapest first: input plus output price per million tokens, at each provider's live price.

   • Equal prices keep ascending priority, then catalog order.
   • A route billed at the cost its provider reports (some image and video models) has no price to compare. It comes after the priced routes.
   • The providers that provider.order names go first, in its order.
   • The routing strategy then sets the final order.
   • A key under a routing policy keeps the policy's saved order.

The gateway tries the list in order. The first deployment to produce real output wins. A failure that can fail over moves to the next deployment with no backoff. The gateway translates requests, streams and errors between dialects, so you can call any listed model from either surface.

A key with a BYOK connection or a routing policy runs the same steps over its own routes, with the policy's order, funding rule and attempt limit on top. POST /v1/route shows the plan without sending the request.

A dedicated endpoint (<your-namespace>/<name>) routes first to capacity reserved for your organization, then to public deployments of the same model that meet its constraints. It never routes to a different model. A closed endpoint names RouterPlus as the provider, whichever route served.

The commit boundary

A request commits to a deployment at its first semantic output: the first text delta, reasoning delta, tool call, refusal or usage report that reaches you.

Before commit, failover is possible only while the gateway has written nothing to you.

• A provider that fails before its first frame is skipped, and the next one is tried at once. This covers a refused connection, a 5xx, a failure before any bytes, and a miss of the 20-second headers deadline.
• The gateway sends your response headers only when a live attempt starts writing.
• After the headers and (on the OpenAI surface) the role-priming delta go out, the gateway does not switch providers. A later failure becomes the terminal in-stream error in Streaming, even with no semantic output.

After commit, the gateway never switches providers. If the provider fails mid-answer, you get the tokens it produced plus one terminal error event inside the 200 stream. You decide whether to retry. A retry starts fresh.

No model substitution

A model that is not in the catalog returns a 404 that names it:

{
  "error": {
    "code": "model_unavailable",
    "message": "model \"gpt-5-ultra\" is not in the catalog; GET /v1/models lists what this key can serve",
    "type": "invalid_request_error",
    "metadata": { "error_type": "model_unavailable" }
  },
  "request_id": "…"
}

There is no closest-match fallback and no default model. n>1 returns a typed 400, because the gateway sends every stream as a single choice.

What never fails over

The gateway decides whether an error can fail over once, when it normalizes the error. Routing never reads raw provider errors.

• content_policy never retries and never goes to another provider. This check runs before every status-code rule. Error metadata on this path never echoes your flagged input. To try a different provider, send a new request.
• context_overflow never fails over, because every deployment of a model has the same context window. Shorten the request or choose a model with a larger context.
• Other buyer-fault 4xx errors, such as a malformed request, return directly.

Failover eligibility by error class

Health circuits and cooldowns

Each deployment has its own circuit breaker:

• Two failures in a row open the circuit for a 30-second cooldown. Routing skips the deployment while the circuit is open.
• After the cooldown the circuit goes half-open. One live request goes through as a probe. Success closes the circuit. Failure opens it again with a new cooldown.
• Only failures that can fail over count. Buyer-fault errors (400/413-class, content policy, context overflow) never count.
• An upstream 429 counts against the circuit. Uptime scoring excludes it as buyer-caused. It also pauses that provider's admission pool for the provider's retry-after, 1 to 60 seconds.
• A client cancellation does not count.
• A hanging provider counts as a failure. This applies when the provider sends no response headers for 60 seconds (20 seconds on a stream) and the request then ends, by the gateway's deadline or your cancellation.
  • The attempt counts on the deployment's circuit and on a second circuit for that model at that deployment.
  • Two such attempts in a row pause the provider for that model for 30 seconds. Its other models keep answering. Then one request probes it.

Public uptime numbers use the same rules. A model page shows no uptime percentage for a provider RouterPlus calls directly until it has 100 counted requests in the window.

When everything is cooling down

If every deployment of your model has an open circuit, the gateway refuses the request at once, with no queue. The response is a 503 with:

• retry-after: 5
• x-tm-error-code: gateway_error
• x-tm-limit-kind: health
• the message all deployments cooling down

A cooldown lasts 30 seconds, then one request goes through as the probe. Retry after a few seconds.

Reading what happened

Every response says how it was served:

On failures the body carries the canonical class: error.metadata.error_type on the OpenAI surface, error.error_type on the Anthropic surface. Each dialect's native type string is there too, so your SDK's retry rules keep working. The gateway never relays a provider's raw error body. See Errors & remediation for the full table.

Per-attempt audit

The generation audit endpoint returns every physical attempt for a request id: which deployments the gateway tried, in what order, and what each one cost.

curl -s -H "Authorization: Bearer $TM_API_KEY" \
  "https://demo-api.routerplus.com/v1/generation?id=<x-request-id>"

In this trimmed example, a Claude request under a byok_with_house_fallback policy failed on your own Anthropic connection, then succeeded on the marketplace anthropic deployment. Each attempt also carries its billing source, price snapshot, error origin, admission and route context, and dropped parameters.

{
  "request_id": "2f6d8e3b-…",
  "admission_events": [],
  "attempts": [
    {
      "attempt": 1,
      "deployment": "byok:5f3a9c1e-…:0b7d42e8-…",
      "model": "claude-sonnet-5",
      "outcome": "failed",
      "usage_provenance": "unknown",
      "tokens": { "input": 0, "cached": 0, "cache_write": 0, "output": 0, "reasoning": 0 },
      "reserved_max_usd": 0,
      "cost_usd": 0,
      "error_code": "upstream_error",
      "dispatched_at": "2026-09-04T09:12:44.120Z"
    },
    {
      "attempt": 2,
      "deployment": "anthropic",
      "model": "claude-sonnet-5",
      "outcome": "completed",
      "usage_provenance": "observed",
      "tokens": { "input": 412, "cached": 0, "cache_write": 0, "output": 638, "reasoning": 0 },
      "reserved_max_usd": 0.04,
      "cost_usd": 0.007204,
      "error_code": "",
      "dispatched_at": "2026-09-04T09:12:45.010Z"
    }
  ]
}

The response holds metadata only. See Data policy.

Use this endpoint to recompute an Anthropic-surface stream that failed mid-answer (see Streaming).