# Evals

An eval compares candidate models on your own cases. RouterPlus sends each case to each model.
Then a judge can score the answers, compare them with an expected answer, or select the best
answer for each case. When the eval ends, you get a summary for each model: quality, cost and
latency.

1. Create the eval with your cases and models: `POST /v1/evals`.
2. Poll the eval until it ends: `GET /v1/evals/{eval_id}`.
3. Read the answers for each case: `GET /v1/evals/{eval_id}/results`.

Every route also works under `/api/v1/`. In the console, open Build > Evals.

## Authentication, billing and governance

Send your inference key as `Authorization: Bearer <key>`. An eval belongs to the key's
org. An eval of another org returns 404.

Each candidate call and each judge call runs as an ordinary chat completion:

- RouterPlus bills it to your org at the normal price of the model and provider that serve it.
- Your org, workspace, member and key guardrails apply. Your key's rate limit applies.
- Routing and provider failover work as on a direct request.
- It shows in your usage and activity, with your key's labels and finance category.

When you create an eval, the gateway makes a key for that eval. The eval key copies these
settings from your key: workspace, member, labels, finance category, rate limit, routing and
key guardrail. The eval key stops working when the eval ends.

Your key's spending limit and IP allowlist do not apply to the eval key. To limit eval
spend, use a guardrail budget.

## Create an eval

`POST /v1/evals`:

| Field | Rule |
|---|---|
| `name` | 1 to 200 characters. |
| `cases` | An array of 1 to 2,000 cases. Send one of `cases` and `file_id`. |
| `file_id` | A file with purpose `evals`. See [Eval files](#eval-files). |
| `models` | 1 to 10 candidate model ids. |
| `judge` | Optional. See [Judge modes](#judge-modes). Without a judge, the summary has cost, latency, tokens and errors only. |
| `params` | Optional. `temperature` (0 to 2) and `max_tokens` (1 to 128,000). RouterPlus sends them on every candidate call. |

A case:

| Field | Rule |
|---|---|
| `id` | Optional. 1 to 64 characters, unique in the eval. Default `case-1`, `case-2` and so on. |
| `input` | A string of up to 200,000 characters, sent as one user message. Or an array of chat messages. |
| `expected` | Optional. The reference answer, up to 20,000 characters. Mode `match` needs it on every case. The other modes show it to the judge. |
| `criteria` | Optional. What a good answer to this case does, up to 4,000 characters. The judge reads it. |

```bash
curl https://demo-api.routerplus.com/v1/evals \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "name": "support answers",
    "cases": [
      {"input": "How do I reset my password?", "criteria": "Names the settings page."},
      {"input": [{"role": "system", "content": "Answer in one word."}, {"role": "user", "content": "Capital of France?"}], "expected": "Paris"}
    ],
    "models": ["gpt-6-luna", "deepseek/deepseek-v4.1-flash", "moonshotai/kimi-k3"],
    "judge": {"model": "gpt-6-luna", "mode": "score", "rubric": "Correct, short and polite."},
    "params": {"temperature": 0, "max_tokens": 400}
  }'
```

The response is the eval object, with status `queued`.

## Judge modes

| Mode | What happens | Needs `judge.model` |
|---|---|---|
| `score` | The judge reads each answer and gives a score from 1 to 10. One judge call per answer. | Yes |
| `match` | RouterPlus compares each answer to `expected`. No judge call. | No |
| `pairwise` | The judge reads all answers to a case and picks the best one. One judge call per case. Needs 2 or more models. | Yes |

- `rubric` is optional, up to 4,000 characters. The judge reads it on every call.
- The judge sees the answers with the labels A, B, C. It never sees the model ids. In
  `pairwise` mode the label order changes from case to case.
- In `pairwise` mode, when only one model answered a case, that model wins with no judge call.
  When no model answered a case, the case has no winner.
- In `match` mode, an answer is an `exact` match when it equals `expected` after trimming. It is
  a `normalized` match when both are equal after these changes: lower case, each run of
  whitespace as one space, and no quotes, brackets or end punctuation.
- A judge reply with no score from 1 to 10, or with no named answer, is a judge error. That
  answer or case has no score or winner.

## Eval files

For many cases, upload a JSONL file with one case on each line. Then send its id as `file_id`:

```json
{"id": "q1", "input": "Capital of France?", "expected": "Paris"}
{"id": "q2", "input": [{"role": "user", "content": "2 + 2?"}], "expected": "4", "criteria": "A number only."}
```

```bash
curl https://demo-api.routerplus.com/v1/files \
  -H "Authorization: Bearer $TM_API_KEY" \
  -F purpose=evals \
  -F file=@cases.jsonl
```

- The upload checks every line and names the first bad line.
- A file holds at most 2,000 cases.
- The other file routes are the same as for [batches](api-batches.md).
- The eval keeps its own copy of the cases. You can delete the file after you create the eval.

## Read an eval

`GET /v1/evals/{eval_id}`:

```json
{
  "id": "eval_3f9a0c1e2b4d6f8091a2b3c4", "object": "eval", "name": "support answers", "status": "completed",
  "models": ["gpt-6-luna", "deepseek/deepseek-v4.1-flash", "moonshotai/kimi-k3"],
  "judge": {"model": "gpt-6-luna", "rubric": "Correct, short and polite.", "mode": "score"},
  "params": {"temperature": 0, "max_tokens": 400}, "file_id": null,
  "progress": {"cases_total": 2, "cases_done": 2, "calls_done": 12, "call_errors": 0},
  "created_at": 1793577600, "started_at": 1793577601, "finished_at": 1793577640, "error": null,
  "summary": {
    "mode": "score",
    "models": [
      {"model": "deepseek/deepseek-v4.1-flash", "cases": 2, "answered": 2, "errors": 0, "mean_score": 8.5, "scored": 2,
       "match_rate": null, "exact_match_rate": null, "win_rate": null, "wins": 0, "cost_usd": 0.000412,
       "mean_latency_ms": 1840, "p50_latency_ms": 1620, "p95_latency_ms": 2060,
       "prompt_tokens": 64, "completion_tokens": 190, "total_tokens": 254}
    ],
    "judge": {"model": "gpt-6-luna", "calls": 6, "errors": 0, "cost_usd": 0.000951, "prompt_tokens": 2310, "completion_tokens": 140},
    "total_cost_usd": 0.002104
  }
}
```

`summary` is `null` until the eval ends. A cancelled eval has a summary of the calls that ran.

Summary fields for each model:

| Field | Meaning |
|---|---|
| `cases`, `answered`, `errors` | Cases sent to the model, answers with text, and calls that failed or came back empty. |
| `mean_score`, `scored` | Mode `score`: the mean judge score and the number of scored answers. |
| `match_rate`, `exact_match_rate` | Mode `match`: matched cases over all cases. A failed call counts as no match. |
| `win_rate`, `wins` | Mode `pairwise`: cases the model won over cases the judge decided. |
| `cost_usd` | The billed cost of the model's calls, in USD. Judge calls are in `judge.cost_usd`. |
| `mean_latency_ms`, `p50_latency_ms`, `p95_latency_ms` | Time to the full answer, over answered calls. |
| `prompt_tokens`, `completion_tokens`, `total_tokens` | Tokens of the model's calls. |

`total_cost_usd` is the cost of all candidate and judge calls. It matches your usage.

### Statuses

| Status | Meaning |
|---|---|
| `queued` | The eval is waiting to start. |
| `running` | Cases are running. `progress` grows. |
| `completed` | All cases ran. `summary` is ready. |
| `failed` | The eval could not run. `error` has a `code` and `message`. |
| `cancelled` | You cancelled. Cases that already started finish. No new cases start. |

An eval that has not ended 7 days after creation fails with code `eval_expired`.

## Results per case

`GET /v1/evals/{eval_id}/results?limit=&offset=` lists the cases in order. `limit` is 1 to 500,
default 100. `offset` is the first case's index, default 0.

```json
{
  "object": "list", "eval_id": "eval_3f9a0c1e2b4d6f8091a2b3c4", "status": "completed", "offset": 0, "limit": 100, "total": 2, "has_more": false,
  "data": [
    {"case_index": 1, "case_id": "case-2", "expected": "Paris", "results": [
      {"model": "deepseek/deepseek-v4.1-flash", "status": "ok", "answer": "Paris", "answer_truncated": false, "error": null,
       "score": 10, "matched": null, "won": null, "judge_reason": "Correct and short.",
       "cost_usd": 0.000021, "judge_cost_usd": 0.00016, "latency_ms": 980,
       "prompt_tokens": 22, "completion_tokens": 2, "request_id": "8d1c0f6e-2b7a-4e39-9a51-3c0d2e4f6a7b"}
    ]}
  ]
}
```

- `status` is `ok`, `error` or `pending` (not run yet).
- `answer` holds the first 4,000 characters. `answer_truncated` is `true` when the answer was
  longer.
- `score` is the judge score in mode `score`, and 1 or 0 in mode `match`.
- In mode `pairwise`, `won` tells if the model won the case. Each case has a `judge` object with
  `best_model`, `reason` and `cost_usd`.
- `request_id` is the gateway's id for the call. Use it to find the call in your activity.

## Other routes

| Route | Response |
|---|---|
| `POST /v1/evals/{eval_id}/cancel` | The eval object, with status `cancelled`. An eval that already ended returns 409. |
| `GET /v1/evals?limit=&after=` | A list of eval objects, newest first. `limit` is 1 to 100, default 20. `after` is an eval id. |

## Retries

RouterPlus retries a call that gets 429 or 5xx, or that it cannot send. It makes up to 3 attempts
in total. Before each retry it waits for the response's `retry-after`, up to 60 seconds.
With no `retry-after`, it waits 1 second, then 2 seconds. The eval records the last response.

## Errors

Errors use this shape:

```json
{"error": {"message": "No eval found with id 'eval_...'.", "type": "invalid_request_error", "param": "eval_id", "code": "eval_not_found"}}
```

| Status | When |
|---|---|
| 400 | A bad field or value, or a bad eval file. `param` names it. |
| 401 | No key, or a key that is not valid. |
| 404 | No such eval or file in your org. |
| 409 | The eval cannot be cancelled in its status. |
| 413 | The request body is larger than 32 MiB. Use an eval file. |