Optimize

Optimize finds a cheaper model or mix that scores as well as your model on your own Braintrust evals. It deploys the result as an endpoint: one model id that you call like any other.

Before you start

1. 01
   In the console, open Access > Integrations and connect Braintrust with a Braintrust API key.

   • RouterPlus checks the key with Braintrust and stores it encrypted. The page shows only the last four characters.
   • Owners and admins can connect, replace or disconnect the key. Viewers can only look.
   • Langfuse is on the same page, under Request access. The RouterPlus team sets it up with you.
2. 02
   In Braintrust, make sure that you have one of these sources to search on:

The baseline must be a catalog model or one of your own deployed endpoints (tm/...). A search takes up to 2,000 cases.

For an experiment or a dataset, these parts of the rows are used:

Code scorers run in Braintrust. Prompt scorers (LLM graders) run through the gateway and bill your credits.

Start a search

1. 01
   Click New search at the bottom of the searches list.
2. 02
   Select the source and name the search.
3. 03
   Set the search budget: the most that the search spends on model calls, from $5 to $100.

Model calls bill your credits while the search runs. They use a managed key that RouterPlus makes for the search. To see the key, turn on Show playground keys on /console/keys. The key has a managed tag. Viewers cannot start or change a search.

The search runs in stages. The budget pays for them in this order:

1. 01
   Shortlist. Public benchmarks select the open models to try: the Pareto frontier of benchmark score against price. The search tries each model on up to three hosts: the cheapest host, the host with the highest precision, and the model maker's own host.
2. 02
   Pilot. Every candidate answers a sample of the conversations.
3. 03
   Dev. The leaders answer all the dev conversations.
4. 04
   Mixes. The search builds cascades, routers and votes from the saved answers. A search agent proposes mixes to try. Mixes cost nothing extra.
5. 05
   Held-out. The finalists, and a fusion of the best models, answer conversations that the search never saw. The winner comes from these.
6. 06
   Customers. The finalists talk to a simulated customer who has the goal of a held-out conversation.

If a Braintrust read fails while the search loads its cases, RouterPlus tries 3 more times, after 2 s, 10 s and 30 s. Then it stops the search. A stopped search shows the reason. Fix the cause (for example, connect the key again) and add budget to continue.

Read the results

The chart plots each candidate's cost against its score. A line shows the best-value frontier. The table below the chart has these columns:

The ★ row is the winner. It is the cheapest finalist that meets all three rules:

• It costs less than the baseline.
• It is not worse than the baseline on a paired test over the same held-out cases.
• It scores within 2 points of the baseline.

When two rows cost the same, the row with the lower median latency wins. Rows marked dev or pilot scored on fewer conversations and cannot win. If no row meets the rules, no row has a star.

Open a row to see the cases where it did worse than the baseline, side by side, and its score for each tag.

Deploy the result

Deploy makes a result callable as tm/<name>-v1. The name has lowercase letters, digits and dashes, 40 characters at most:

curl https://dev2-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model": "tm/billing-agent-v1", "messages": [{"role": "user", "content": "Hi"}]}'

• Only keys of your organization can call it.
• RouterPlus bills each model call that the endpoint makes as a direct call to that model. usage.cost in the response is the sum.
• Send it on POST /v1/chat/completions. With "stream": true, it returns the finished answer as one chunk, then usage.
• A second deploy makes a new version (-v2). Earlier versions continue to work.

Endpoints lists everything that your organization deployed, with the search that each result came from. Try it sends one held-out case of that search to the endpoint, and bills your credits like any call.

What is stored

A search stores its cases and the candidates' answers, for your organization only, until you delete the search. A deployed model id continues to work after you delete its search.