Evals

An eval compares candidate models on your own cases. RouterPlus sends each case to each model. Then a judge can score the answers, compare them with an expected answer, or select the best answer for each case. When the eval ends, you get a summary for each model: quality, cost and latency.

1. 01
   Create the eval with your cases and models: POST /v1/evals.
2. 02
   Poll the eval until it ends: GET /v1/evals/{eval_id}.
3. 03
   Read the answers for each case: GET /v1/evals/{eval_id}/results.

Every route also works under /api/v1/. In the console, open Build > Evals.

Authentication, billing and governance

Send your inference key as Authorization: Bearer <key>. An eval belongs to the key's org. An eval of another org returns 404.

Each candidate call and each judge call runs as an ordinary chat completion:

• RouterPlus bills it to your org at the normal price of the model and provider that serve it.
• Your org, workspace, member and key guardrails apply. Your key's rate limit applies.
• Routing and provider failover work as on a direct request.
• It shows in your usage and activity, with your key's labels and finance category.

When you create an eval, the gateway makes a key for that eval. The eval key copies these settings from your key: workspace, member, labels, finance category, rate limit, routing and key guardrail. The eval key stops working when the eval ends.

Your key's spending limit and IP allowlist do not apply to the eval key. To limit eval spend, use a guardrail budget.

Create an eval

POST /v1/evals:

A case:

curl https://dev2-api.routerplus.com/v1/evals \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "name": "support answers",
    "cases": [
      {"input": "How do I reset my password?", "criteria": "Names the settings page."},
      {"input": [{"role": "system", "content": "Answer in one word."}, {"role": "user", "content": "Capital of France?"}], "expected": "Paris"}
    ],
    "models": ["gpt-6-luna", "deepseek/deepseek-v4.1-flash", "moonshotai/kimi-k3"],
    "judge": {"model": "gpt-6-luna", "mode": "score", "rubric": "Correct, short and polite."},
    "params": {"temperature": 0, "max_tokens": 400}
  }'

The response is the eval object, with status queued.

Judge modes

• rubric is optional, up to 4,000 characters. The judge reads it on every call.
• The judge sees the answers with the labels A, B, C. It never sees the model ids. In pairwise mode the label order changes from case to case.
• In pairwise mode, when only one model answered a case, that model wins with no judge call. When no model answered a case, the case has no winner.
• In match mode, an answer is an exact match when it equals expected after trimming. It is a normalized match when both are equal after these changes: lower case, each run of whitespace as one space, and no quotes, brackets or end punctuation.
• A judge reply with no score from 1 to 10, or with no named answer, is a judge error. That answer or case has no score or winner.

Eval files

For many cases, upload a JSONL file with one case on each line. Then send its id as file_id:

{"id": "q1", "input": "Capital of France?", "expected": "Paris"}
{"id": "q2", "input": [{"role": "user", "content": "2 + 2?"}], "expected": "4", "criteria": "A number only."}

curl https://dev2-api.routerplus.com/v1/files \
  -H "Authorization: Bearer $TM_API_KEY" \
  -F purpose=evals \
  -F [email protected]

• The upload checks every line and names the first bad line.
• A file holds at most 2,000 cases.
• The other file routes are the same as for batches.
• The eval keeps its own copy of the cases. You can delete the file after you create the eval.

Read an eval

GET /v1/evals/{eval_id}:

{
  "id": "eval_3f9a0c1e2b4d6f8091a2b3c4", "object": "eval", "name": "support answers", "status": "completed",
  "models": ["gpt-6-luna", "deepseek/deepseek-v4.1-flash", "moonshotai/kimi-k3"],
  "judge": {"model": "gpt-6-luna", "rubric": "Correct, short and polite.", "mode": "score"},
  "params": {"temperature": 0, "max_tokens": 400}, "file_id": null,
  "progress": {"cases_total": 2, "cases_done": 2, "calls_done": 12, "call_errors": 0},
  "created_at": 1793577600, "started_at": 1793577601, "finished_at": 1793577640, "error": null,
  "summary": {
    "mode": "score",
    "models": [
      {"model": "deepseek/deepseek-v4.1-flash", "cases": 2, "answered": 2, "errors": 0, "mean_score": 8.5, "scored": 2,
       "match_rate": null, "exact_match_rate": null, "win_rate": null, "wins": 0, "cost_usd": 0.000412,
       "mean_latency_ms": 1840, "p50_latency_ms": 1620, "p95_latency_ms": 2060,
       "prompt_tokens": 64, "completion_tokens": 190, "total_tokens": 254}
    ],
    "judge": {"model": "gpt-6-luna", "calls": 6, "errors": 0, "cost_usd": 0.000951, "prompt_tokens": 2310, "completion_tokens": 140},
    "total_cost_usd": 0.002104
  }
}

summary is null until the eval ends. A cancelled eval has a summary of the calls that ran.

Summary fields for each model:

total_cost_usd is the cost of all candidate and judge calls. It matches your usage.

Statuses

An eval that has not ended 7 days after creation fails with code eval_expired.

Results per case

GET /v1/evals/{eval_id}/results?limit=&offset= lists the cases in order. limit is 1 to 500, default 100. offset is the first case's index, default 0.

{
  "object": "list", "eval_id": "eval_3f9a0c1e2b4d6f8091a2b3c4", "status": "completed", "offset": 0, "limit": 100, "total": 2, "has_more": false,
  "data": [
    {"case_index": 1, "case_id": "case-2", "expected": "Paris", "results": [
      {"model": "deepseek/deepseek-v4.1-flash", "status": "ok", "answer": "Paris", "answer_truncated": false, "error": null,
       "score": 10, "matched": null, "won": null, "judge_reason": "Correct and short.",
       "cost_usd": 0.000021, "judge_cost_usd": 0.00016, "latency_ms": 980,
       "prompt_tokens": 22, "completion_tokens": 2, "request_id": "8d1c0f6e-2b7a-4e39-9a51-3c0d2e4f6a7b"}
    ]}
  ]
}

• status is ok, error or pending (not run yet).
• answer holds the first 4,000 characters. answer_truncated is true when the answer was longer.
• score is the judge score in mode score, and 1 or 0 in mode match.
• In mode pairwise, won tells if the model won the case. Each case has a judge object with best_model, reason and cost_usd.
• request_id is the gateway's id for the call. Use it to find the call in your activity.

Other routes

Retries

RouterPlus retries a call that gets 429 or 5xx, or that it cannot send. It makes up to 3 attempts in total. Before each retry it waits for the response's retry-after, up to 60 seconds. With no retry-after, it waits 1 second, then 2 seconds. The eval records the last response.

Errors

Errors use this shape:

{"error": {"message": "No eval found with id 'eval_...'.", "type": "invalid_request_error", "param": "eval_id", "code": "eval_not_found"}}