# Batches

The Batches API runs many requests from one file and returns the answers in result files. It
uses the shapes of the OpenAI Files and Batches API. The OpenAI SDKs work when you change
`base_url` and the key.

A batch has four steps:

1. Upload a JSONL file of requests: `POST /v1/files`.
2. Create a batch from the file: `POST /v1/batches`.
3. Poll the batch until it ends: `GET /v1/batches/{batch_id}`.
4. Download the output and error files: `GET /v1/files/{file_id}/content`.

Every route also works under `/api/v1/`.

## Example

```python
import os, time
from openai import OpenAI

client = OpenAI(base_url="https://demo-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

while batch.status not in ("completed", "failed", "cancelled", "expired"):
    time.sleep(30)
    batch = client.batches.retrieve(batch.id)

print(batch.request_counts)
if batch.output_file_id:
    print(client.files.content(batch.output_file_id).text)
if batch.error_file_id:
    print(client.files.content(batch.error_file_id).text)
```

To cancel:

```python
client.batches.cancel(batch.id)
```

## Authentication, billing and governance

Send your inference key as `Authorization: Bearer <key>`. Files and batches belong to the
key's org. A file or batch of another org returns 404.

Each line of a batch runs as an ordinary request to the gateway:

- RouterPlus bills it to your org at the normal price of the model and provider that serve it.
- Your org, workspace, member and key guardrails apply to it.
- It has the same rate limit as your key.
- Routing, provider failover and a `models` fallback list work as on a direct request.
- It shows in your usage and activity, with your key's labels and finance category.

When you create a batch, the gateway makes a key for that batch. The batch key copies these
settings from the key that created the batch: workspace, member, labels, finance category, rate
limit, routing and key guardrail. The batch key stops working when the batch ends.

> [!NOTE]
> Your key's spending limit and IP allowlist do not apply to the batch key. To limit batch
> spend, use a guardrail budget.

## The input file

Put one request on each line, as JSON:

```json
{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-6-luna", "messages": [{"role": "user", "content": "Hello"}]}}
```

| Field | Rule |
|---|---|
| `custom_id` | 1 to 64 characters. Unique in the file. The answer line has the same `custom_id`. |
| `method` | `"POST"`. |
| `url` | `/v1/chat/completions`, `/v1/messages` or `/v1/responses`. The same on every line. |
| `body` | The request body, as you send it to that endpoint. |

Limits:

- A file has at most 50 MiB.
- A file has at most 10,000 requests.
- The gateway always sends `stream` as `false`. Each line gets one complete answer.

The upload checks every line. A bad line returns 400. The message names the first bad line:

```json
{"error": {"message": "Line 2: method must be \"POST\".", "type": "invalid_request_error", "param": "file", "code": "invalid_file"}}
```

## Files

### Upload

Send `POST /v1/files` as `multipart/form-data` with the fields `purpose=batch` and `file`:

```bash
curl https://demo-api.routerplus.com/v1/files \
  -H "Authorization: Bearer $TM_API_KEY" \
  -F purpose=batch \
  -F file=@requests.jsonl
```

Or send the file as the body, with `content-type: application/jsonl`:

```bash
curl "https://demo-api.routerplus.com/v1/files?purpose=batch&filename=requests.jsonl" \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/jsonl" \
  --data-binary @requests.jsonl
```

The response is a file object:

```json
{"id": "file-5b2e0c7d9a1f4e3b8c6d0a2f", "object": "file", "bytes": 1832, "created_at": 1793577600,
 "expires_at": 1796169600, "filename": "requests.jsonl", "purpose": "batch", "status": "processed", "status_details": null}
```

RouterPlus keeps a file for 30 days.

### Other file routes

| Route | Response |
|---|---|
| `GET /v1/files/{file_id}` | The file object. |
| `GET /v1/files/{file_id}/content` | The file's JSONL content. |
| `DELETE /v1/files/{file_id}` | `{"id": "...", "object": "file", "deleted": true}`. You cannot delete the input file of a batch that has not ended (400). |
| `GET /v1/files?purpose=&limit=&after=&order=` | A list of file objects, newest first. `purpose` is `batch`, `batch_output` or `batch_error`. |

## Batches

### Create

`POST /v1/batches`:

| Field | Rule |
|---|---|
| `input_file_id` | A file with purpose `batch`. |
| `endpoint` | The `url` on the file's lines. |
| `completion_window` | `"24h"`. |
| `metadata` | Optional. Up to 16 pairs. Keys have up to 64 characters. Values are strings of up to 512 characters. |

```bash
curl https://demo-api.routerplus.com/v1/batches \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"input_file_id": "file-5b2e0c7d9a1f4e3b8c6d0a2f", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'
```

The response is a batch object:

```json
{"id": "batch_8f3a1c2e4b5d6f708192a3b4", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null,
 "input_file_id": "file-5b2e0c7d9a1f4e3b8c6d0a2f", "completion_window": "24h", "status": "validating",
 "output_file_id": null, "error_file_id": null, "created_at": 1793577600, "in_progress_at": null,
 "expires_at": 1793664000, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null,
 "cancelling_at": null, "cancelled_at": null, "request_counts": {"total": 3, "completed": 0, "failed": 0}, "metadata": null}
```

### Other batch routes

| Route | Response |
|---|---|
| `GET /v1/batches/{batch_id}` | The batch object. `request_counts` grows while the batch runs. |
| `POST /v1/batches/{batch_id}/cancel` | The batch object, with status `cancelling`. |
| `GET /v1/batches?limit=&after=` | A list of batch objects, newest first. `limit` is 1 to 100, default 20. `after` is a batch id. |

### Statuses

| Status | Meaning |
|---|---|
| `validating` | The batch is waiting to start. |
| `in_progress` | Lines are running. |
| `finalizing` | All lines ran. The gateway is writing the output and error files. |
| `completed` | The output and error files are ready. |
| `failed` | The batch could not run. `errors` gives the reason. No line ran after the failure. |
| `cancelling` | You cancelled. Lines that already started finish. No new lines start. |
| `cancelled` | The cancel is done. The files hold the lines that ran. |
| `expired` | The batch did not end within 24 hours of creation. The files hold the lines that ran. Every line that did not run is in the error file with code `batch_expired`. |

`request_counts.completed` counts the lines with a 2xx answer. `request_counts.failed` counts
all other lines.

### Retries

The gateway retries a line that gets 429 or 5xx, or that it cannot send. It makes up to 3
attempts in total. Before each retry it waits for the answer's `retry-after`, up to 60
seconds. With no `retry-after`, it waits 1 second, then 2 seconds. The batch records the last
answer.

## Output and error files

When a batch ends, `output_file_id` and `error_file_id` name its result files.

- A file id is `null` when the file has no lines.
- The lines follow the order of the input file. Match them to your requests by `custom_id`.

A line with a 2xx answer goes to the output file:

```json
{"id": "batch_req_1c9e7a0b3d5f2e4a6c8b0d1f", "custom_id": "req-1", "response": {"status_code": 200, "request_id": "8d1c0f6e-2b7a-4e39-9a51-3c0d2e4f6a7b", "body": {"id": "chatcmpl-1", "object": "chat.completion", "choices": []}}, "error": null}
```

A line with any other answer goes to the error file, with that answer's status and body:

```json
{"id": "batch_req_7e2d4c6a8b0f1e3d5c7a9b2e", "custom_id": "req-2", "response": {"status_code": 400, "request_id": "4f2a...", "body": {"error": {"message": "..."}}}, "error": null}
```

A line that got no answer has `response: null` and an `error` with a `code` and `message`.

`request_id` is the gateway's id for that request. Use it to find the request in your activity.

## Errors

Errors use the OpenAI shape:

```json
{"error": {"message": "No batch found with id 'batch_...'.", "type": "invalid_request_error", "param": "batch_id", "code": "batch_not_found"}}
```

| Status | When |
|---|---|
| 400 | A bad file, field or value. `param` names it. |
| 401 | No key, or a key that is not valid. |
| 404 | No such file or batch in your org. |
| 409 | The batch cannot be cancelled in its status. |
| 413 | The upload is larger than 50 MiB. |