# Rate limits, capacity and outage behavior

This page is the full reference for admission: limit scopes, provider pools, fairness, and what
happens when a store is down. For per-key RPM, the org defaults and spend caps, see
[Rate limits & spend caps](/docs/limits). Pools attach to BYOK connections and routing policies.
See [BYOK setup](/docs/byok) and [routing policies](/docs/routing-policies).

All gateway instances count every limit on this page in one shared store. Adding instances
does not raise any limit.

## Manage limits

In the console, open Billing > Rate limits
([`/console/limits`](https://demo.routerplus.com/console/limits)). There you view or set gateway limits,
create provider pools and assign connections.

Gateway limits overlap. The strictest platform, org, workspace, principal, key, model and
provider-account limit that applies wins.

The limits API uses the site origin (`https://demo.routerplus.com`), a session cookie, a JSON content type
and same-origin protection on writes. Inference bearer keys cannot call it. Owners and admins
can write. Viewers can read.

| Endpoint | Purpose |
|---|---|
| `GET /api/admission-limits` | Lists your org's configured limits, every scope. |
| `POST /api/admission-limits` | Replaces one scope's limits. |
| `GET /api/quota-pools` | Lists the newest 100 org-owned provider pools. |
| `GET /api/quota-pools/{id}` | Reads one pool. |
| `POST /api/quota-pools` | Creates a pool. |
| `POST /api/quota-pools/{id}` | Replaces its limits and parent. |
| `POST /api/quota-pools/{id}/bind` | Assigns an owned connection to that pool. |

Example limit body:

```json
{"scope":"key","subject":"VIRTUAL_KEY_UUID","rpm":100,"tpm":200000,"concurrency":4,"outage_mode":"bounded_open"}
```

| Field | Rule |
|---|---|
| `scope` | `org`, `workspace`, `principal`, `key` or `model`. |
| `subject` | `org` for the org scope. The UUID for the workspace, principal and key scopes. The exact model ID for the model scope. |
| `rpm` | 1 to 1,000,000. |
| `tpm` | 1 to 1,000,000,000. |
| `concurrency` | 1 to 10,000. |
| `outage_mode` | `bounded_open` (the default) or `closed`. `closed` refuses new work while the counter store is down. A closed mode at any scope that applies wins. |

A numeric value that is null or left out inherits the default or the parent's limit. A key's
original RPM ceiling still applies: a larger key limit does not raise it.

### Defaults

| Scope | Default |
|---|---|
| Org, workspace, principal | 300 RPM, 1M TPM, 8 concurrent requests |
| Key | Its RPM ceiling, 1M TPM, 8 concurrent requests |
| Model | None. Set one to add a limit for one model. |

- Until an org adds credit, the org and each of its keys are held to 20 RPM.
- The operator sets platform limits separately.
- Under a contract, the operator can replace the defaults and the keys' RPM ceiling for one org.
  This override can add a per-second limit on the org. It also replaces the 20 RPM hold. Limits
  you set still apply and can only lower an override.
- Traffic to a [dedicated endpoint](/docs/dedicated-endpoints#limits) keeps the endpoint's own
  limits.

## Declare provider account capacity

```json
{
  "provider":"openai",
  "rpm":500,
  "tpm":1000000,
  "concurrency":20,
  "headroom_percent":90,
  "parent_pool_id":null
}
```

- `provider` is `openai`, `anthropic`, `azure` or `bedrock` for customer pools.
- Send finite positive values.
- `headroom_percent` is the usable percentage, 1 to 100. Default 90.
- Fractional limits round down, with a minimum of one unit.
- A parent pool must belong to the same org and use the same provider.
- Pools nest up to four levels, the leaf included. Cycles are not allowed.
- An update replaces the limits and parent. Send the complete configuration.

To bind a connection, send `{"connection_id":"CONNECTION_UUID"}` to
`/api/quota-pools/{id}/bind`. The connection's profile must match the pool's provider.

Keys for the same upstream account should share a pool, or use child pools under a shared
parent. Rotating credentials does not reset capacity. RouterPlus checks parent and child claims
together. RouterPlus cannot tell your real provider account from a key, so your pool
declarations must be accurate.

The operator owns the house pools and maps them to catalog deployment IDs. You cannot change
them. A route with no declared pool is unavailable (`503`, `x-tm-limit-kind:
capacity_unknown`). RouterPlus does not treat it as unlimited.

RouterPlus does not see provider usage outside the gateway. A provider 429 pauses the affected
leaf pool for the provider's `retry-after` (1 to 60 seconds), across all workers. An error of
unknown scope does not mark the whole provider as down.

## Counting and token estimates

- One admitted logical request uses gateway RPM once.
- Every physical attempt, an eligible fallback included, uses upstream RPM and its own token
  reservation.
- A request sent to a provider is not refunded. A request proven not dispatched can release its
  provider claim.
- Concurrency uses renewable 10-second leases, so a crashed worker does not hold slots.

### Input estimate

The input reservation is the UTF-8 byte size plus 65,536 per image or document part. The base64
of an inline image is left out of the byte size, so the image counts only its 65,536. This is an
estimate, not the vendor's tokenizer count.

### Output bound

- RouterPlus sends an output bound of up to the model's own `max_output_tokens` (16,384 on
  GPT-4o, 128,000 on the Claude 5 models, 32,768 when the model declares none).
- A request without an output limit gets the model's own maximum, written into the request. It
  is lowered when needed so that the input and output fit the model's context window.
- The token estimate counts at most 32,768 of the output limit, so a large default does not
  crowd out other requests.
- The balance hold counts all of an output limit that the request sends, and at most 32,768
  tokens of the default.
- A larger or conflicting output limit returns 400.

### Settlement

- Observed input and output usage of a completed attempt replaces the estimate. Cache and
  reasoning tokens are not counted twice.
- Partial prefill is not a final bill. Cancelled, incomplete, unknown or estimated usage keeps
  its reservation until the rate window expires. Work proven never dispatched is the exception.
- The rolling minute includes a partial second, so a charge can stay for up to 61 seconds.

### Anthropic token counting

Anthropic token counting uses the same authorization and shared limits. It uses gateway and
provider RPM, reserves estimated input capacity and writes a request audit before it calls the
provider. It creates no inference debit and no inference-attempt row. Its answer is
`{"input_tokens": N}` and nothing else.

## Inspect limits and errors

`GET /v1/limits?model=MODEL_ID` on the gateway returns the effective gateway limits for the key,
inherited limits included. `model` is optional. The answer also has:

| Field | Meaning |
|---|---|
| `max_output_tokens` | The output bound: the model's own maximum, or 32,768 without a model. |
| `default_max_output_tokens` | The limit a request without one gets, for a short prompt. A long prompt gets less. |
| `quota_max_output_tokens` | The most output the token estimate counts: 32,768. |
| outage mode | The outage mode that applies. |

On a dedicated endpoint, RouterPlus caps a configured `default_max_output_tokens` at the model's
maximum, and `/v1/limits` reports the capped value. RouterPlus still rejects an output limit
that a client sends above the model maximum.

A successful admission returns `x-tm-remaining-rpm` and `x-tm-remaining-tpm`: the lowest
remaining amount across the claimed scopes at that moment. They do not promise capacity for a
later request. On a dedicated endpoint with a
[paced limit](/docs/dedicated-endpoints#the-paced-limit), `x-tm-queue-ms` gives how long the
request waited before admission.

Failures keep SDK-compatible statuses and include metadata and CORS-readable headers:

| Origin | Typical status | Meaning |
|---|---:|---|
| `gateway_admission` | 429 or 503 | A customer quota or spend limit, fairness headroom, or worker capacity. |
| `upstream_quota` | 429, or 503 if unconfigured | A declared account pool is used up or cooling down, or its capacity is unknown. |
| `upstream` | The provider's status | A provider response, including its 429. |
| `gateway_infrastructure` | 503 | A counter, journal or recovery failure. |
| `authorization` | 401, 404 or 503 | Authority is invalid or unavailable, or a snapshot changed or expired. |

Headers:

- `x-tm-error-origin`
- `x-tm-limit-scope`
- `x-tm-limit-kind` (`rpm`, `rps`, `rps_queue`, `tpm`, `concurrency`, `cooldown`, `spend`,
  `health`, `capacity_unknown`, …)
- `x-tm-limit-id` (optional)
- `retry-after`

The metadata says whether to retry and, where known, the retry or cap-reset time. A
rolling-window retry time is advisory. It does not guarantee capacity.

RouterPlus records quota and admission refusals after authentication, also when no attempt was
made. The record is best effort: when it cannot be written, you still get the 429, not a 503.
Read them with `GET /v1/generation?id=REQUEST_ID`. `admission_events` come with the attempts,
and each attempt includes its receipt context and error origin. This audit cannot tie
unauthenticated requests or HTTP-parser protections to an org.

## Availability and fairness

- **Healthy shared store:** limits are shared across workers. Pilot fairness protects idle
  tenant headroom. It caps each org at a share of a shared house pool's usable capacity, with a
  one-unit minimum. The share is half by default, and 75% on the decision-model pools
  (`typesafe`, `bespokelabs`,
  `workers-ai`, `levanto`, `fastino`, `routerplus`, `perplexity-decisions`, `openai-decisions`).
  RouterPlus rejects excess work at once. There is no waiting queue, except on a dedicated
  endpoint with a [paced limit](/docs/dedicated-endpoints#the-paced-limit). There is no general
  no-starvation guarantee for any number of tenants.
- **Redis unavailable:** the default local allowance is at most 5 logical requests a minute, 2
  concurrent local requests and 25,000 estimated TPM per scope per worker, for at most 30
  seconds. A dedicated endpoint's traffic instead gets the endpoint's own limits divided among
  the gateways that serve it, for at most 10 minutes by default. A paced endpoint paces each
  gateway at its share of the rate. `x-tm-admission-mode: bounded_local` marks this path.
  Limits and balances can be approximate across workers during this time. Strict mode refuses
  new work. Every provider call still needs a durable journal intent.
- **Postgres unavailable:** verified snapshots can serve for a bounded time (5 minutes by
  default, up to 1 hour if the operator sets it for an outage) while the live Redis authority
  fence still matches. A single failed read falls back to a snapshot at most 5 minutes old.
  After reads have failed for 5 seconds with none succeeding, the gateway answers from the
  snapshot at once, up to 15 seconds after the last failure. It does not wait on Postgres again
  in that time. Cold, evicted, expired or changed authority fails closed.
  `x-tm-authorization-mode: bounded_snapshot` marks requests that used this fallback. If both
  stores are unavailable, RouterPlus does not use cached authority.
- **Management during Redis failure:** RouterPlus refuses changes, so revocation stays
  coordinated. A failed publication can leave a fence that needs operator recovery. Provider
  calls in progress can finish on their pinned credential version.
- **Redis replaced or state lost:** shared counters need explicit recovery. RouterPlus does not
  treat an empty store as a full unused allowance.

These are pilot controls. They give no provider capacity commitments, exact vendor TPM
compliance, private connectivity, regional processing or ZDR. Under a routing policy, automatic
fallback stays within one provider label. See [routing policies](/docs/routing-policies).

## Workspace and principal scopes

`workspace` and `principal` limits add up across the keys bound to that scope. RouterPlus claims
them in the same atomic step as the org, key, model, platform and pool limits. Bindings come
from server-managed keys, never from request headers. See [identity management](identity.md).