# Streaming

Set `"stream": true` to receive the answer as Server-Sent Events (SSE). Both chat surfaces
stream: `POST https://demo-api.routerplus.com/v1/chat/completions` (OpenAI) and
`POST https://demo-api.routerplus.com/v1/messages` (Anthropic).

```bash
curl -N https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-5","stream":true,"max_tokens":100,
       "messages":[{"role":"user","content":"Say hello"}]}'
```

The images route (`POST https://demo-api.routerplus.com/v1/images/generations`) does not stream. There,
`stream: true` and `partial_images` are typed 400s.

## Relayed and translated streams

RouterPlus handles a stream in one of two ways. The provider's dialect decides which.

**Same dialect: relayed.** The provider speaks the dialect of your surface, for example a Claude
model on `/v1/messages` or a GPT model on `/v1/chat/completions`. RouterPlus relays the
provider's stream record for record, with the payloads unchanged.

- Provider features reach you as they do from the provider: server tool use, web search
  results, citations, annotations, `pause_turn` and the matched `stop_sequence`.
- RouterPlus reads the stream for billing and failover. It adds `usage.cost` to the provider's
  usage record and makes sure the stream ends with a well-formed terminator.
- On the OpenAI surface, each chunk keeps only OpenAI's fields and names the model you asked
  for. RouterPlus drops fields the provider adds (its cost, its accounting, its host's name).
  The delta passes as the provider sent it. See
  [what an answer keeps](/docs/compat#what-an-answer-from-an-openai-dialect-provider-keeps).
- An explicit `stream_options: {"include_usage": false}` turns the relay off. RouterPlus must
  then remove the usage chunk it asked the provider for.

**Other dialect: translated.** RouterPlus translates the provider's stream into the framing of
your surface. The frame shapes below apply.

## Deadlines

| Deadline | Value |
|---|---|
| First byte (headers) | 20 seconds. A provider that misses it is failed over, and you get no error. See [Routing & failover](/docs/routing). |
| After the first byte | None. RouterPlus does not cut a slow stream. |
| Whole request | 450 seconds from dispatch, or the `timeout_ms` of a [routing policy](/docs/routing-policies) (1 to 600 s). |

A stream that is still open at the request deadline ends with the terminal error event in
[When a provider dies mid-stream](#when-a-provider-dies-mid-stream).

## Response headers

Every stream response carries `x-request-id`, `x-tm-provider` (the deployment that served you),
`x-tm-attempts` and `x-tm-upstream-status`. It carries `x-tm-dropped-params`,
`x-tm-upstream-model` and `x-tm-served-by` when they apply. See
[Wire compatibility](/docs/compat#response-headers).

## Frame order: OpenAI surface

Each frame is a complete `data: {json}\n\n` line in the `chat.completion.chunk` shape. The
order is fixed: role-priming delta, content deltas, finish chunk, usage chunk, `data: [DONE]`.

```text
data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":" there."},"finish_reason":null}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{},"finish_reason":"stop","native_finish_reason":"end_turn"}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":9,"total_tokens":21,"prompt_tokens_details":{"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"reasoning_tokens":0},"cost":0.000114}}

data: [DONE]
```

- RouterPlus sends the role-priming delta (`{"role":"assistant","content":""}`) when the
  upstream responds. It shows that the stream is alive and carries no content.
- Tool calls stream as `delta.tool_calls` entries with stable, ascending `index` values.
  Concatenate the argument fragments. A tool called with no arguments gets one `"{}"` fragment.
- Reasoning models stream their thinking as `delta.reasoning_content`.
- Refusal text streams as `delta.refusal`.
- On a translated stream, the finish chunk also carries `native_finish_reason`, the provider's
  own stop reason (here `end_turn`).
- The usage chunk is always sent, zero-filled if the provider reported nothing. The exception
  is a request with `stream_options.include_usage: false`.
- `[DONE]` comes only after a successful finish. It never follows an error.

## Frame order: Anthropic surface

Frames are named SSE events (`event: <name>\ndata: {json}\n\n`) in the Anthropic Messages API
order: `message_start`, `content_block_start`, `content_block_delta`, `content_block_stop`,
`message_delta`, `message_stop`.

```text
event: message_start
data: {"type":"message_start","message":{"id":"msg_2f6d8e3b","type":"message","role":"assistant","model":"gpt-4o-mini","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":0,"output_tokens":0}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello there."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"input_tokens":12,"cache_read_input_tokens":0,"cache_creation_input_tokens":0,"output_tokens":9,"cost":0.000007}}

event: message_stop
data: {"type":"message_stop"}
```

- On a translated stream, the token counts in `message_start` are zeros. Read the final usage
  and `cost` in the last `message_delta`. On a relayed stream, `message_start` carries the
  provider's own counts.
- Tool calls stream as `tool_use` content blocks with `input_json_delta` fragments.
- Thinking streams as `thinking` blocks with `thinking_delta`, plus `signature_delta` where the
  provider signs it.
- `input_tokens` excludes cache reads and writes, per the [billing contract](/docs/compat).
  `cache_read_input_tokens` and `cache_creation_input_tokens` report cache traffic.

## Usage is always in the stream

You do not opt in to stream usage. RouterPlus adds `stream_options.include_usage` to every
OpenAI-dialect upstream request, because OpenAI streams carry no usage without it.

- RouterPlus honors an explicit `stream_options: {"include_usage": false}`. You are billed the
  same, and you get no usage chunk.
- Every other stream ends with the token counts the provider reported.
- Every billed stream carries `usage.cost` in USD. RouterPlus computes it with the integer
  micro-USD math the ledger settles with. It covers the full request debit, including attempts
  that failed over before your answer started.

> [!TIP]
> `cost` on the wire is the amount you are charged. You can recompute it from the token
> counts and the public prices at
> [https://demo.routerplus.com/api/models.json](https://demo.routerplus.com/api/models.json). Settlement rounds down.

## Keep-alive frames

Proxies, load balancers and some HTTP clients close idle TCP connections. Slow reasoning models
can be silent for a long time mid-answer. After your stream starts, RouterPlus sends a
keep-alive frame after any silence of 15 seconds or more.

| Surface | Frame | Who sees it |
|---|---|---|
| OpenAI | The SSE comment `: processing` | The SSE spec requires clients to ignore it. SDKs never see it. |
| Anthropic | `event: ping` / `data: {"type":"ping"}` | The same frame Anthropic's API sends. Anthropic SDKs skip it. |

RouterPlus measures the silence from the last real write, so a gap is at most about 15 s.
Nothing is sent after a terminal event, keep-alives included.

## Cancelling a stream

To cancel, close the connection (abort the HTTP request). RouterPlus then aborts the upstream
request, and the provider stops generating.

Billing:

- You pay for the input and for the output streamed to you before the abort. The attempt has
  the outcome `cancelled`, visible at `GET /v1/generation?id=<x-request-id>`.
- If the provider reported no final usage, you pay for the input and the output you received,
  never the full hold. The input count is the provider's if it sent one, else RouterPlus's
  estimate.
- A cancel before the first token costs the input only.

Uptime:

- A cancel does not count against the provider's uptime. The public uptime numbers and the
  health circuits exclude cancelled attempts.
- Exception: if the provider sent no response headers for 60 seconds before you closed the
  connection, the cancel counts as a provider failure. See
  [Routing & failover](/docs/routing#health-circuits-and-cooldowns).

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://demo-api.routerplus.com/v1",
  apiKey: process.env.TM_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "claude-sonnet-5",
  max_tokens: 4096,
  messages: [{ role: "user", content: "Explain SSE, thoroughly." }],
  stream: true,
});

let chars = 0;
for await (const chunk of stream) {
  chars += chunk.choices[0]?.delta?.content?.length ?? 0;
  if (chars > 2000) {
    stream.controller.abort(); // upstream generation stops; billed to here
    break;
  }
}
```

## When a provider dies mid-stream

If the serving provider fails after your answer started, RouterPlus does not restart the
request on another provider (see the commit boundary in [Routing & failover](/docs/routing)).
Instead, you get one terminal error event inside the open 200 stream.

**OpenAI surface.** RouterPlus first sends any known usage and its `cost`, so you can recompute
the billed partial attempt. Then it sends a final chunk with `finish_reason: "error"` and a
top-level `error`:

```text
data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}],"error":{"code":"upstream_error","message":"upstream failure","type":"gateway_error","metadata":{"error_type":"upstream_error"}}}
```

**Anthropic surface.** RouterPlus sends a native `error` event with the Anthropic type string
and the stable `error_type`:

```text
event: error
data: {"type":"error","error":{"type":"api_error","message":"upstream failure","error_type":"upstream_error"}}
```

Nothing follows a terminal event: no `[DONE]` and no other frames. The same event ends a
stream that reaches the request deadline. It also ends a stream whose provider closed the
connection without a proper terminator.

> [!NOTE]
> The Anthropic wire format has no usage slot outside `message_delta`. So an Anthropic-surface
> stream that fails mid-answer cannot carry its partial usage. Read the settled numbers from
> `GET /v1/generation?id=<x-request-id>`.

[Errors & remediation](/docs/errors) lists every `error_type` and what to do about it.

## SDK streaming examples

Both official SDKs work unmodified. Change the base URL and the key.

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

stream = client.chat.completions.create(
    model="claude-sonnet-5",
    max_tokens=200,
    messages=[{"role": "user", "content": "Write a haiku about failover."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:  # the final usage chunk has an empty choices list
        print(f"\ncost: ${chunk.usage.cost}")
```

```python
import os
import anthropic

client = anthropic.Anthropic(base_url="https://demo-api.routerplus.com", api_key=os.environ["TM_API_KEY"])

with client.messages.stream(
    model="gpt-4o-mini",  # a GPT model over the Anthropic wire
    max_tokens=200,
    messages=[{"role": "user", "content": "Write a haiku about failover."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()
    print(f"\nstop_reason: {final.stop_reason}")
```