Rate limits, capacity and outage behavior

This page is the full reference for admission: limit scopes, provider pools, fairness, and what happens when a store is down. For per-key RPM, the org defaults and spend caps, see Rate limits & spend caps. Pools attach to BYOK connections and routing policies. See BYOK setup and routing policies.

All gateway instances count every limit on this page in one shared store. Adding instances does not raise any limit.

Manage limits

In the console, open Billing > Rate limits (/console/limits). There you view or set gateway limits, create provider pools and assign connections.

Gateway limits overlap. The strictest platform, org, workspace, principal, key, model and provider-account limit that applies wins.

The limits API uses the site origin (https://demo.routerplus.com), a session cookie, a JSON content type and same-origin protection on writes. Inference bearer keys cannot call it. Owners and admins can write. Viewers can read.

Example limit body:

{"scope":"key","subject":"VIRTUAL_KEY_UUID","rpm":100,"tpm":200000,"concurrency":4,"outage_mode":"bounded_open"}

A numeric value that is null or left out inherits the default or the parent's limit. A key's original RPM ceiling still applies: a larger key limit does not raise it.

Defaults

• Until an org adds credit, the org and each of its keys are held to 20 RPM.
• The operator sets platform limits separately.
• Under a contract, the operator can replace the defaults and the keys' RPM ceiling for one org. This override can add a per-second limit on the org. It also replaces the 20 RPM hold. Limits you set still apply and can only lower an override.
• Traffic to a dedicated endpoint keeps the endpoint's own limits.

Declare provider account capacity

{
  "provider":"openai",
  "rpm":500,
  "tpm":1000000,
  "concurrency":20,
  "headroom_percent":90,
  "parent_pool_id":null
}

• provider is openai, anthropic, azure or bedrock for customer pools.
• Send finite positive values.
• headroom_percent is the usable percentage, 1 to 100. Default 90.
• Fractional limits round down, with a minimum of one unit.
• A parent pool must belong to the same org and use the same provider.
• Pools nest up to four levels, the leaf included. Cycles are not allowed.
• An update replaces the limits and parent. Send the complete configuration.

To bind a connection, send {"connection_id":"CONNECTION_UUID"} to /api/quota-pools/{id}/bind. The connection's profile must match the pool's provider.

Keys for the same upstream account should share a pool, or use child pools under a shared parent. Rotating credentials does not reset capacity. RouterPlus checks parent and child claims together. RouterPlus cannot tell your real provider account from a key, so your pool declarations must be accurate.

The operator owns the house pools and maps them to catalog deployment IDs. You cannot change them. A route with no declared pool is unavailable (503, x-tm-limit-kind: capacity_unknown). RouterPlus does not treat it as unlimited.

RouterPlus does not see provider usage outside the gateway. A provider 429 pauses the affected leaf pool for the provider's retry-after (1 to 60 seconds), across all workers. An error of unknown scope does not mark the whole provider as down.

Counting and token estimates

• One admitted logical request uses gateway RPM once.
• Every physical attempt, an eligible fallback included, uses upstream RPM and its own token reservation.
• A request sent to a provider is not refunded. A request proven not dispatched can release its provider claim.
• Concurrency uses renewable 10-second leases, so a crashed worker does not hold slots.

Input estimate

The input reservation is the UTF-8 byte size plus 65,536 per image or document part. The base64 of an inline image is left out of the byte size, so the image counts only its 65,536. This is an estimate, not the vendor's tokenizer count.

Output bound

• RouterPlus sends an output bound of up to the model's own max_output_tokens (16,384 on GPT-4o, 128,000 on the Claude 5 models, 32,768 when the model declares none).
• A request without an output limit gets the model's own maximum, written into the request. It is lowered when needed so that the input and output fit the model's context window.
• The token estimate counts at most 32,768 of the output limit, so a large default does not crowd out other requests.
• The balance hold counts all of an output limit that the request sends, and at most 32,768 tokens of the default.
• A larger or conflicting output limit returns 400.

Settlement

• Observed input and output usage of a completed attempt replaces the estimate. Cache and reasoning tokens are not counted twice.
• Partial prefill is not a final bill. Cancelled, incomplete, unknown or estimated usage keeps its reservation until the rate window expires. Work proven never dispatched is the exception.
• The rolling minute includes a partial second, so a charge can stay for up to 61 seconds.

Anthropic token counting

Anthropic token counting uses the same authorization and shared limits. It uses gateway and provider RPM, reserves estimated input capacity and writes a request audit before it calls the provider. It creates no inference debit and no inference-attempt row. Its answer is {"input_tokens": N} and nothing else.

Inspect limits and errors

GET /v1/limits?model=MODEL_ID on the gateway returns the effective gateway limits for the key, inherited limits included. model is optional. The answer also has:

On a dedicated endpoint, RouterPlus caps a configured default_max_output_tokens at the model's maximum, and /v1/limits reports the capped value. RouterPlus still rejects an output limit that a client sends above the model maximum.

A successful admission returns x-tm-remaining-rpm and x-tm-remaining-tpm: the lowest remaining amount across the claimed scopes at that moment. They do not promise capacity for a later request. On a dedicated endpoint with a paced limit, x-tm-queue-ms gives how long the request waited before admission.

Failures keep SDK-compatible statuses and include metadata and CORS-readable headers:

Headers:

• x-tm-error-origin
• x-tm-limit-scope
• x-tm-limit-kind (rpm, rps, rps_queue, tpm, concurrency, cooldown, spend, health, capacity_unknown, …)
• x-tm-limit-id (optional)
• retry-after

The metadata says whether to retry and, where known, the retry or cap-reset time. A rolling-window retry time is advisory. It does not guarantee capacity.

RouterPlus records quota and admission refusals after authentication, also when no attempt was made. The record is best effort: when it cannot be written, you still get the 429, not a 503. Read them with GET /v1/generation?id=REQUEST_ID. admission_events come with the attempts, and each attempt includes its receipt context and error origin. This audit cannot tie unauthenticated requests or HTTP-parser protections to an org.

Availability and fairness

• Healthy shared store: limits are shared across workers. Pilot fairness protects idle tenant headroom. It caps each org at a share of a shared house pool's usable capacity, with a one-unit minimum. The share is half by default, and 75% on the decision-model pools (typesafe, bespokelabs, workers-ai, levanto, fastino, routerplus, perplexity-decisions, openai-decisions). RouterPlus rejects excess work at once. There is no waiting queue, except on a dedicated endpoint with a paced limit. There is no general no-starvation guarantee for any number of tenants.
• Redis unavailable: the default local allowance is at most 5 logical requests a minute, 2 concurrent local requests and 25,000 estimated TPM per scope per worker, for at most 30 seconds. A dedicated endpoint's traffic instead gets the endpoint's own limits divided among the gateways that serve it, for at most 10 minutes by default. A paced endpoint paces each gateway at its share of the rate. x-tm-admission-mode: bounded_local marks this path. Limits and balances can be approximate across workers during this time. Strict mode refuses new work. Every provider call still needs a durable journal intent.
• Postgres unavailable: verified snapshots can serve for a bounded time (5 minutes by default, up to 1 hour if the operator sets it for an outage) while the live Redis authority fence still matches. A single failed read falls back to a snapshot at most 5 minutes old. After reads have failed for 5 seconds with none succeeding, the gateway answers from the snapshot at once, up to 15 seconds after the last failure. It does not wait on Postgres again in that time. Cold, evicted, expired or changed authority fails closed. x-tm-authorization-mode: bounded_snapshot marks requests that used this fallback. If both stores are unavailable, RouterPlus does not use cached authority.
• Management during Redis failure: RouterPlus refuses changes, so revocation stays coordinated. A failed publication can leave a fence that needs operator recovery. Provider calls in progress can finish on their pinned credential version.
• Redis replaced or state lost: shared counters need explicit recovery. RouterPlus does not treat an empty store as a full unused allowance.

These are pilot controls. They give no provider capacity commitments, exact vendor TPM compliance, private connectivity, regional processing or ZDR. Under a routing policy, automatic fallback stays within one provider label. See routing policies.

Workspace and principal scopes

workspace and principal limits add up across the keys bound to that scope. RouterPlus claims them in the same atomic step as the org, key, model, platform and pool limits. Bindings come from server-managed keys, never from request headers. See identity management.