Streaming

Set "stream": true to receive the answer as Server-Sent Events (SSE). Both chat surfaces stream: POST https://demo-api.routerplus.com/v1/chat/completions (OpenAI) and POST https://demo-api.routerplus.com/v1/messages (Anthropic).

curl -N https://dev2-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-5","stream":true,"max_tokens":100,
       "messages":[{"role":"user","content":"Say hello"}]}'

The images route (POST https://demo-api.routerplus.com/v1/images/generations) does not stream. There, stream: true and partial_images are typed 400s.

Relayed and translated streams

RouterPlus handles a stream in one of two ways. The provider's dialect decides which.

Same dialect: relayed. The provider speaks the dialect of your surface, for example a Claude model on /v1/messages or a GPT model on /v1/chat/completions. RouterPlus relays the provider's stream record for record, with the payloads unchanged.

• Provider features reach you as they do from the provider: server tool use, web search results, citations, annotations, pause_turn and the matched stop_sequence.
• RouterPlus reads the stream for billing and failover. It adds usage.cost to the provider's usage record and makes sure the stream ends with a well-formed terminator.
• On the OpenAI surface, each chunk keeps only OpenAI's fields and names the model you asked for. RouterPlus drops fields the provider adds (its cost, its accounting, its host's name). The delta passes as the provider sent it. See what an answer keeps.
• An explicit stream_options: {"include_usage": false} turns the relay off. RouterPlus must then remove the usage chunk it asked the provider for.

Other dialect: translated. RouterPlus translates the provider's stream into the framing of your surface. The frame shapes below apply.

Deadlines

A stream that is still open at the request deadline ends with the terminal error event in When a provider dies mid-stream.

Response headers

Every stream response carries x-request-id, x-tm-provider (the deployment that served you), x-tm-attempts and x-tm-upstream-status. It carries x-tm-dropped-params, x-tm-upstream-model and x-tm-served-by when they apply. See Wire compatibility.

Frame order: OpenAI surface

Each frame is a complete data: {json}\n\n line in the chat.completion.chunk shape. The order is fixed: role-priming delta, content deltas, finish chunk, usage chunk, data: [DONE].

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":" there."},"finish_reason":null}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{},"finish_reason":"stop","native_finish_reason":"end_turn"}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":9,"total_tokens":21,"prompt_tokens_details":{"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"reasoning_tokens":0},"cost":0.000114}}

data: [DONE]

• RouterPlus sends the role-priming delta ({"role":"assistant","content":""}) when the upstream responds. It shows that the stream is alive and carries no content.
• Tool calls stream as delta.tool_calls entries with stable, ascending index values. Concatenate the argument fragments. A tool called with no arguments gets one "{}" fragment.
• Reasoning models stream their thinking as delta.reasoning_content.
• Refusal text streams as delta.refusal.
• On a translated stream, the finish chunk also carries native_finish_reason, the provider's own stop reason (here end_turn).
• The usage chunk is always sent, zero-filled if the provider reported nothing. The exception is a request with stream_options.include_usage: false.
• [DONE] comes only after a successful finish. It never follows an error.

Frame order: Anthropic surface

Frames are named SSE events (event: <name>\ndata: {json}\n\n) in the Anthropic Messages API order: message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop.

event: message_start
data: {"type":"message_start","message":{"id":"msg_2f6d8e3b","type":"message","role":"assistant","model":"gpt-4o-mini","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":0,"output_tokens":0}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello there."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"input_tokens":12,"cache_read_input_tokens":0,"cache_creation_input_tokens":0,"output_tokens":9,"cost":0.000007}}

event: message_stop
data: {"type":"message_stop"}

• On a translated stream, the token counts in message_start are zeros. Read the final usage and cost in the last message_delta. On a relayed stream, message_start carries the provider's own counts.
• Tool calls stream as tool_use content blocks with input_json_delta fragments.
• Thinking streams as thinking blocks with thinking_delta, plus signature_delta where the provider signs it.
• input_tokens excludes cache reads and writes, per the billing contract. cache_read_input_tokens and cache_creation_input_tokens report cache traffic.

Usage is always in the stream

You do not opt in to stream usage. RouterPlus adds stream_options.include_usage to every OpenAI-dialect upstream request, because OpenAI streams carry no usage without it.

• RouterPlus honors an explicit stream_options: {"include_usage": false}. You are billed the same, and you get no usage chunk.
• Every other stream ends with the token counts the provider reported.
• Every billed stream carries usage.cost in USD. RouterPlus computes it with the integer micro-USD math the ledger settles with. It covers the full request debit, including attempts that failed over before your answer started.

Keep-alive frames

Proxies, load balancers and some HTTP clients close idle TCP connections. Slow reasoning models can be silent for a long time mid-answer. After your stream starts, RouterPlus sends a keep-alive frame after any silence of 15 seconds or more.

RouterPlus measures the silence from the last real write, so a gap is at most about 15 s. Nothing is sent after a terminal event, keep-alives included.

Cancelling a stream

To cancel, close the connection (abort the HTTP request). RouterPlus then aborts the upstream request, and the provider stops generating.

Billing:

• You pay for the input and for the output streamed to you before the abort. The attempt has the outcome cancelled, visible at GET /v1/generation?id=<x-request-id>.
• If the provider reported no final usage, you pay for the input and the output you received, never the full hold. The input count is the provider's if it sent one, else RouterPlus's estimate.
• A cancel before the first token costs the input only.

Uptime:

• A cancel does not count against the provider's uptime. The public uptime numbers and the health circuits exclude cancelled attempts.
• Exception: if the provider sent no response headers for 60 seconds before you closed the connection, the cancel counts as a provider failure. See Routing & failover.

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://demo-api.routerplus.com/v1",
  apiKey: process.env.TM_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "claude-sonnet-5",
  max_tokens: 4096,
  messages: [{ role: "user", content: "Explain SSE, thoroughly." }],
  stream: true,
});

let chars = 0;
for await (const chunk of stream) {
  chars += chunk.choices[0]?.delta?.content?.length ?? 0;
  if (chars > 2000) {
    stream.controller.abort(); // upstream generation stops; billed to here
    break;
  }
}

When a provider dies mid-stream

If the serving provider fails after your answer started, RouterPlus does not restart the request on another provider (see the commit boundary in Routing & failover). Instead, you get one terminal error event inside the open 200 stream.

OpenAI surface. RouterPlus first sends any known usage and its cost, so you can recompute the billed partial attempt. Then it sends a final chunk with finish_reason: "error" and a top-level error:

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}],"error":{"code":"upstream_error","message":"upstream failure","type":"gateway_error","metadata":{"error_type":"upstream_error"}}}

Anthropic surface. RouterPlus sends a native error event with the Anthropic type string and the stable error_type:

event: error
data: {"type":"error","error":{"type":"api_error","message":"upstream failure","error_type":"upstream_error"}}

Nothing follows a terminal event: no [DONE] and no other frames. The same event ends a stream that reaches the request deadline. It also ends a stream whose provider closed the connection without a proper terminator.

Errors & remediation lists every error_type and what to do about it.

SDK streaming examples

Both official SDKs work unmodified. Change the base URL and the key.

import os
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

stream = client.chat.completions.create(
    model="claude-sonnet-5",
    max_tokens=200,
    messages=[{"role": "user", "content": "Write a haiku about failover."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:  # the final usage chunk has an empty choices list
        print(f"\ncost: ${chunk.usage.cost}")

import os
import anthropic

client = anthropic.Anthropic(base_url="https://demo-api.routerplus.com", api_key=os.environ["TM_API_KEY"])

with client.messages.stream(
    model="gpt-4o-mini",  # a GPT model over the Anthropic wire
    max_tokens=200,
    messages=[{"role": "user", "content": "Write a haiku about failover."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()
    print(f"\nstop_reason: {final.stop_reason}")