Features

Batches

Run a file of requests and get the results as a file.

The Batches API runs many requests from one file and returns the answers in result files. It uses the shapes of the OpenAI Files and Batches API. The OpenAI SDKs work when you change base_url and the key.

A batch has four steps:

01
Upload a JSONL file of requests: POST /v1/files.
02
Create a batch from the file: POST /v1/batches.
03
Poll the batch until it ends: GET /v1/batches/{batch_id}.
04
Download the output and error files: GET /v1/files/{file_id}/content.

Every route also works under /api/v1/.

Example

python

import os, time
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

while batch.status not in ("completed", "failed", "cancelled", "expired"):
    time.sleep(30)
    batch = client.batches.retrieve(batch.id)

print(batch.request_counts)
if batch.output_file_id:
    print(client.files.content(batch.output_file_id).text)
if batch.error_file_id:
    print(client.files.content(batch.error_file_id).text)

To cancel:

python

client.batches.cancel(batch.id)

Authentication, billing and governance

Send your inference key as Authorization: Bearer <key>. Files and batches belong to the key's org. A file or batch of another org returns 404.

Each line of a batch runs as an ordinary request to the gateway:

RouterPlus bills it to your org at the normal price of the model and provider that serve it.
Your org, workspace, member and key guardrails apply to it.
It has the same rate limit as your key.
Routing, provider failover and a models fallback list work as on a direct request.
It shows in your usage and activity, with your key's labels and finance category.

When you create a batch, the gateway makes a key for that batch. The batch key copies these settings from the key that created the batch: workspace, member, labels, finance category, rate limit, routing and key guardrail. The batch key stops working when the batch ends.

Note

Your key's spending limit and IP allowlist do not apply to the batch key. To limit batch spend, use a guardrail budget.

The input file

Put one request on each line, as JSON:

json

{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-6-luna", "messages": [{"role": "user", "content": "Hello"}]}}

Field	Rule
`custom_id`	1 to 64 characters. Unique in the file. The answer line has the same `custom_id`.
`method`	`"POST"`.
`url`	`/v1/chat/completions`, `/v1/messages` or `/v1/responses`. The same on every line.
`body`	The request body, as you send it to that endpoint.

Limits:

A file has at most 50 MiB.
A file has at most 10,000 requests.
The gateway always sends stream as false. Each line gets one complete answer.

The upload checks every line. A bad line returns 400. The message names the first bad line:

json

{"error": {"message": "Line 2: method must be \"POST\".", "type": "invalid_request_error", "param": "file", "code": "invalid_file"}}

Files

Upload

Send POST /v1/files as multipart/form-data with the fields purpose=batch and file:

bash

curl https://dev2-api.routerplus.com/v1/files \
  -H "Authorization: Bearer $TM_API_KEY" \
  -F purpose=batch \
  -F [email protected]

Or send the file as the body, with content-type: application/jsonl:

bash

curl "https://demo-api.routerplus.com/v1/files?purpose=batch&filename=requests.jsonl" \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/jsonl" \
  --data-binary @requests.jsonl

The response is a file object:

json

{"id": "file-5b2e0c7d9a1f4e3b8c6d0a2f", "object": "file", "bytes": 1832, "created_at": 1793577600,
 "expires_at": 1796169600, "filename": "requests.jsonl", "purpose": "batch", "status": "processed", "status_details": null}

RouterPlus keeps a file for 30 days.

Other file routes

Route	Response
`GET /v1/files/{file_id}`	The file object.
`GET /v1/files/{file_id}/content`	The file's JSONL content.
`DELETE /v1/files/{file_id}`	`{"id": "...", "object": "file", "deleted": true}`. You cannot delete the input file of a batch that has not ended (400).
`GET /v1/files?purpose=&limit=&after=&order=`	A list of file objects, newest first. `purpose` is `batch`, `batch_output` or `batch_error`.

Batches

Create

POST /v1/batches:

Field	Rule
`input_file_id`	A file with purpose `batch`.
`endpoint`	The `url` on the file's lines.
`completion_window`	`"24h"`.
`metadata`	Optional. Up to 16 pairs. Keys have up to 64 characters. Values are strings of up to 512 characters.

bash

curl https://dev2-api.routerplus.com/v1/batches \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"input_file_id": "file-5b2e0c7d9a1f4e3b8c6d0a2f", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'

The response is a batch object:

json

{"id": "batch_8f3a1c2e4b5d6f708192a3b4", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null,
 "input_file_id": "file-5b2e0c7d9a1f4e3b8c6d0a2f", "completion_window": "24h", "status": "validating",
 "output_file_id": null, "error_file_id": null, "created_at": 1793577600, "in_progress_at": null,
 "expires_at": 1793664000, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null,
 "cancelling_at": null, "cancelled_at": null, "request_counts": {"total": 3, "completed": 0, "failed": 0}, "metadata": null}

Other batch routes

Route	Response
`GET /v1/batches/{batch_id}`	The batch object. `request_counts` grows while the batch runs.
`POST /v1/batches/{batch_id}/cancel`	The batch object, with status `cancelling`.
`GET /v1/batches?limit=&after=`	A list of batch objects, newest first. `limit` is 1 to 100, default 20. `after` is a batch id.

Statuses

Status	Meaning
`validating`	The batch is waiting to start.
`in_progress`	Lines are running.
`finalizing`	All lines ran. The gateway is writing the output and error files.
`completed`	The output and error files are ready.
`failed`	The batch could not run. `errors` gives the reason. No line ran after the failure.
`cancelling`	You cancelled. Lines that already started finish. No new lines start.
`cancelled`	The cancel is done. The files hold the lines that ran.
`expired`	The batch did not end within 24 hours of creation. The files hold the lines that ran. Every line that did not run is in the error file with code `batch_expired`.

request_counts.completed counts the lines with a 2xx answer. request_counts.failed counts all other lines.

Retries

The gateway retries a line that gets 429 or 5xx, or that it cannot send. It makes up to 3 attempts in total. Before each retry it waits for the answer's retry-after, up to 60 seconds. With no retry-after, it waits 1 second, then 2 seconds. The batch records the last answer.

Output and error files

When a batch ends, output_file_id and error_file_id name its result files.

A file id is null when the file has no lines.
The lines follow the order of the input file. Match them to your requests by custom_id.

A line with a 2xx answer goes to the output file:

json

{"id": "batch_req_1c9e7a0b3d5f2e4a6c8b0d1f", "custom_id": "req-1", "response": {"status_code": 200, "request_id": "8d1c0f6e-2b7a-4e39-9a51-3c0d2e4f6a7b", "body": {"id": "chatcmpl-1", "object": "chat.completion", "choices": []}}, "error": null}

A line with any other answer goes to the error file, with that answer's status and body:

json

{"id": "batch_req_7e2d4c6a8b0f1e3d5c7a9b2e", "custom_id": "req-2", "response": {"status_code": 400, "request_id": "4f2a...", "body": {"error": {"message": "..."}}}, "error": null}

A line that got no answer has response: null and an error with a code and message.

request_id is the gateway's id for that request. Use it to find the request in your activity.

Errors

Errors use the OpenAI shape:

json

{"error": {"message": "No batch found with id 'batch_...'.", "type": "invalid_request_error", "param": "batch_id", "code": "batch_not_found"}}

Status	When
400	A bad file, field or value. `param` names it.
401	No key, or a key that is not valid.
404	No such file or batch in your org.
409	The batch cannot be cancelled in its status.
413	The upload is larger than 50 MiB.

For agents: /docs/api-batches.md and /llms.txt