# Optimize

[Optimize](https://demo.routerplus.com/model-search) finds a cheaper model or mix that scores as well as
your model on your own Braintrust evals. It deploys the result as an endpoint: one model id that
you call like any other.

> [!NOTE]
> Optimize is paused. Searches cannot start, continue or deploy for now. Past results stay
> available. A deployed endpoint answers only while every model it calls is in the catalog.

## Before you start

1. In the console, open [Access > Integrations](https://demo.routerplus.com/console/integrations) and
   connect Braintrust with a Braintrust API key.
   - RouterPlus checks the key with Braintrust and stores it encrypted. The page shows only the
     last four characters.
   - Owners and admins can connect, replace or disconnect the key. Viewers can only look.
   - Langfuse is on the same page, under **Request access**. The RouterPlus team sets it up
     with you.
2. In Braintrust, make sure that you have one of these sources to search on:

| Source | What the search uses |
|---|---|
| An **experiment** that ran on a dataset | The dataset is the test cases. The experiment's `metadata.model` is the baseline. The prompt comes from `metadata.prompt_id` or `metadata.prompt_slug`, or is the project's only prompt. |
| A **dataset** | The rows are the test cases. You select the prompt and, optionally, a baseline model. Without a baseline, the strongest candidate becomes the baseline. |
| **Production logs** | The logged calls of the last 1, 7 or 30 days. Each call is a test case, with the production reply as the expected turn. A candidate scores on whether its reply is as good as the production reply. The baseline is the model that most logged calls used, unless you select one. |

The baseline must be a catalog model or one of your own deployed endpoints (`tm/...`). A search
takes up to 2,000 cases.

For an experiment or a dataset, these parts of the rows are used:

| Braintrust | Used as |
|---|---|
| A row's `input.messages` | One test of the next assistant turn. |
| The project's scorers | Quality. A case's score is the mean of its scorers. |
| A row's `tags` | The slices in a result's detail view. |
| A row's `metadata.conversation_id` and `metadata.goal` | Keep the turns of one conversation together. Simulated customers use the goal. |

Code scorers run in Braintrust. Prompt scorers (LLM graders) run through the gateway and bill
your credits.

## Start a search

1. Click **New search** at the bottom of the searches list.
2. Select the source and name the search.
3. Set the **search budget**: the most that the search spends on model calls, from $5 to $100.

Model calls bill your credits while the search runs. They use a managed key that RouterPlus
makes for the search. To see the key, turn on **Show playground keys** on `/console/keys`. The
key has a **managed** tag. Viewers cannot start or change a search.

The search runs in stages. The budget pays for them in this order:

1. **Shortlist.** Public benchmarks select the open models to try: the Pareto frontier of
   benchmark score against price. The search tries each model on up to three hosts: the
   cheapest host, the host with the highest precision, and the model maker's own host.
2. **Pilot.** Every candidate answers a sample of the conversations.
3. **Dev.** The leaders answer all the dev conversations.
4. **Mixes.** The search builds cascades, routers and votes from the saved answers. A search
   agent proposes mixes to try. Mixes cost nothing extra.
5. **Held-out.** The finalists, and a fusion of the best models, answer conversations that the
   search never saw. The winner comes from these.
6. **Customers.** The finalists talk to a simulated customer who has the goal of a held-out
   conversation.

| Action | Effect |
|---|---|
| Budget reached | The search stops and keeps its results. |
| **Add $25** | Continues the search from where it stopped. |
| **Stop** | Ends the search and keeps its results. |

If a Braintrust read fails while the search loads its cases, RouterPlus tries 3 more times,
after 2 s, 10 s and 30 s. Then it stops the search. A stopped search shows the reason. Fix the
cause (for example, connect the key again) and add budget to continue.

## Read the results

The chart plots each candidate's cost against its score. A line shows the best-value
frontier. The table below the chart has these columns:

| Column | Meaning |
|---|---|
| Model | The model, or `tm-mix-N` for a mix. Open a row to see what the mix does. |
| Provider | The host that served the model, with its precision when the host states it. |
| Score | The mean scorer score, in percent. |
| $ / 1k req | The measured cost of 1,000 requests like your cases. |
| p50 | The median time to a full answer. |
| TTFT | The median time to the first token. |
| Out tok | Output tokens per answer. |
| Status | The stage that scored the row: `pilot`, `dev` or held-out. |

The ★ row is the winner. It is the cheapest finalist that meets all three rules:

- It costs less than the baseline.
- It is not worse than the baseline on a paired test over the same held-out cases.
- It scores within 2 points of the baseline.

When two rows cost the same, the row with the lower median latency wins. Rows marked `dev` or
`pilot` scored on fewer conversations and cannot win. If no row meets the rules, no row has a
star.

Open a row to see the cases where it did worse than the baseline, side by side, and its score
for each tag.

## Deploy the result

**Deploy** makes a result callable as `tm/<name>-v1`. The name has lowercase letters, digits
and dashes, 40 characters at most:

```bash
curl https://demo-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model": "tm/billing-agent-v1", "messages": [{"role": "user", "content": "Hi"}]}'
```

- Only keys of your organization can call it.
- RouterPlus bills each model call that the endpoint makes as a direct call to that model.
  `usage.cost` in the response is the sum.
- Send it on `POST /v1/chat/completions`. With `"stream": true`, it returns the finished answer
  as one chunk, then usage.
- A second deploy makes a new version (`-v2`). Earlier versions continue to work.

[Endpoints](https://demo.routerplus.com/endpoints) lists everything that your organization deployed, with
the search that each result came from. **Try it** sends one held-out case of that search to the
endpoint, and bills your credits like any call.

## What is stored

A search stores its cases and the candidates' answers, for your organization only, until
you delete the search. A deployed model id continues to work after you delete its search.