Model selection and routing

Vendors sell models in tiers, and the cheaper tiers cost a fraction of the large one per token. The temptation is to route by price: send the easy requests to the small tier, keep the large one for the hard ones, and report the saving. The saving is real; so is the drop in quality on the cases the router got wrong, and whether the trade was worth it is a question only the eval can answer — before the switch, not after a customer notices.

Tiers against cost

A tier is a point on two axes: what it passes on your eval, and what it charges. The price list gives the second axis for free. The first axis is specific to your feature, and a tier that scores well on a public benchmark can fail your billing-dispute slice while passing everything else. Treat the published capability ranking as a prior and the eval as the measurement.

For the arithmetic in this lesson, assume the small tier charges one-fifth of the large on both P_in and P_out. Substitute the ratio from your own price list; the shape of the decision does not change.

The decision is never "which tier is best". It is "which is the cheapest tier that passes the eval at the gate", per feature, and sometimes per step within a feature: the three tool-calling turns of an agent and its final answer are different tasks with different passing tiers.

Run the cheaper model on your eval first

The mistake is the demo: ten prompts tried by hand on the small tier, the answers look fine, the switch ships. Ten prompts cannot see a five-point regression, and the hand-picked ten are the easy ones. The method is the one evals as the unit test of LLM work sets out — the golden set, run on both tiers, scored per slice, with the paired diff rather than two totals.

Three numbers come out of that run, and the decision needs all three:

• the overall pass rate on each tier, with its standard error — on 200 items, about 2 points
• the per-slice pass rates, because a tier that drops 2 points overall may drop 15 on one slice
• the paired diff: which items flipped from pass to fail and which the other way

A tier that passes everything within the gate's tolerance is a routing candidate. One that fails a slice is a routing candidate for every other slice, with the router sending that slice to the large tier — which is where the router stops being a config flag and becomes a classifier.

One model at lower effort versus a cascade

Two architectures get the same headline saving. A cascade sends each request to the small tier first and escalates to the large one when the small tier's answer fails a check — a confidence score, a schema, a judge. One model at lower effort keeps a single tier and turns down its reasoning budget where the vendor exposes that knob, so routine requests spend fewer tokens on the same model.

Expand

A two-tier cascade: every request goes to the small tier, a check escalates the answers it rejects to the large tier, and each tier keeps its own cache100%−+FitSVG (opens in a new tab)CloseEsc

Pinch to zoom · drag to panScroll to zoom · drag or arrow keys to pan · 0 fits · Esc closes

Cost the cascade with an escalation rate e — the fraction of requests the small tier hands up:

cascade   = 0.2 + e × 1.0                      (in units of the large tier's price)
e = 0.1  → 0.30      e = 0.3 → 0.50      e = 0.5 → 0.70

At e = 0.3 the cascade costs half of large-only. Now the two costs it hides. Escalated requests are paid twice, which e × 1.0 already counts, but they are also slower by the whole small-tier call — an agent with a 10 s p95 budget from latency budgets and streaming loses several seconds on 30% of requests. And caches are per model: the large tier now sees 30% of the traffic, so its prefix entries go cold more often, and a request that escalates writes a fresh prefix into a second cache. Compare with large-only and a working cache from prompt caching, where 80% of input tokens are reads:

large only, cached = 0.2 + 0.8 × C_r                             (input, same units)
cascade, cached    = 0.2 × (0.2 + 0.8 × C_r) + 0.3 × (1 − h × (1 − C_r))

The small tier sees every request, so it keeps the 80%; h is the large tier's hit rate on the escalated 30%, with misses at list price. If h held at 0.8, the cascade would still cost half of large-only: caching both sides does not change the ratio. The cascade loses through h. At h = 0, every escalation a cold miss, it costs 0.34 + 0.16 × C_r: more than the single cached model for any C_r below about 0.22, before the write surcharge on those misses is added, and with two prompts to maintain instead of one and the escalation latency on top. Lower effort on the same model keeps the one cache and the one prompt and cuts output tokens on the routine cases, which is where the single-model saving comes from. The cascade wins only when e is small, the escalated traffic is heavy enough to keep the large tier's cache warm, and the small tier's pass rate on the non-escalated traffic is at the gate — all of which the eval and the counters, not the price list, have to show.

The router is a classifier with its own eval

Whatever decides where a request goes is a classifier: a rule on input length, a keyword list, a small model asked "is this hard", a confidence threshold on the small tier's own answer. It has a precision and a recall like any classifier, and its errors are not symmetric. A hard request sent to the small tier is a quality failure the user sees; an easy request sent to the large tier is a few tokens of waste nobody notices. So the router is tuned for recall on the hard class, and that tuning needs labelled data: a few hundred requests marked hard or easy by the same process that built the golden set, with the router scored on them the same way a model is.

Length is the common bad router. Short messages are often the hard ones — "cancel everything" carries more risk in three words than a 300-word description of a shipping delay — and routing by length sends exactly the wrong cases down. A router that was never evaluated is the usual reason a 40% saving arrives with a 5-point drop, and the critique exercise in this module takes one apart.

Routing also has a maintenance cost the sheet should carry. Two tiers means two prompts tuned separately, two sets of eval runs on every change, and a vendor model update on either tier re-opening the question, as sampling and determinism warns for a single tier. The operator's checklist for the same decision is in cost control: token budgets, caching and routing.

In an interview

The question is "would you use a cheaper model for this?", and what is tested is whether your first move is an eval run or a price comparison. Say: "I'd run the golden set on the small tier and look at the per-slice pass rates and the paired diff before deciding anything". Then the architectures — cascade against one model at lower effort — with the escalation rate, the doubled cost on escalated requests, and the per-model cache named. Then the router as a classifier with its own labelled set, tuned for recall on the hard cases.

The mistake that loses points is routing by price before running the eval, and reporting the saving without the quality number beside it. A close second is a cascade justified by list price alone, with the lost cache reuse and the escalation latency left out.

Check yourself

1. The small tier passes 89% of the golden set against the large tier's 92%, overall, on 200 items. Switch?

Not on that number: the standard error on 200 items is about 2 points, so 89 and 92 are not clearly different. Look at the slices and the paired diff. If the 3-point gap is spread thinly, route and keep measuring; if it is 12 points on one slice, route everything except that slice and send it to the large tier — which means building and evaluating a router for it.

2. A cascade escalates 40% of requests. Each escalated request costs a small call plus a large one and adds 2 s. Against large-only with an 80% cache hit rate, which is cheaper on input?

Uncached, the cascade is 0.2 + 0.4 = 0.6 of large price. Cached like for like it is 0.6 × (0.2 + 0.8 × C_r), cheaper than large-only's 0.2 + 0.8 × C_r — but only if the large tier keeps 80% on the 40% it sees, so read that tier's hit rate before deciding. If its cache is cold, the cascade is 0.04 + 0.16 × C_r + 0.4 = 0.44 + 0.16 × C_r, more than large-only for any C_r under 0.375, and 2 s slower on 40% of traffic. At e nearer 0.1 the cascade wins even with a cold large-tier cache, which means a much better router or an easier traffic mix.

3. A router sends messages under 40 tokens to the small tier. The eval's "account closure" slice drops from 90% to 70% after the switch. Which component gets fixed?

The router, not the small tier. Closure requests are short and high-stakes, so a length rule routes them down by construction. Label 200 requests hard or easy, score the router's recall on the hard class, and replace the length rule with a classifier that reaches an agreed recall — or route the closure slice to the large tier by category while the classifier is built.