Model selection as a product decision
Choose a model by capability class, price tier and measured latency, route the traffic the eval clears to a small model, and defend the split in numbers.
On this page
The demo picked the model that read best on three tickets. The product decision is a row in a table with four columns — capability class, price tier, measured latency, golden-set score — and the model's name is not one of them. Names change every few months; the four columns do not, and a decision made on them re-runs in an afternoon when a vendor publishes a new price.
Capability classes, not model names
Four capabilities decide whether a model can do the job at all.
Context length is how much the model can read in one call: system prompt, ticket, retrieved chunks, history. The three biggest vendors' current models sit in one class: Anthropic's take 1,000,000 tokens (source, read 18 Oct 2026) and return up to 128,000 tokens (source, read 18 Oct 2026), the current GPT models 1.05M tokens (source, read 18 Oct 2026), Google's flagship Gemini model 1M tokens (source, read 18 Oct 2026). At 1.3 tokens per word, a million tokens is roughly 770,000 words, a thousand help articles. The drafting feature's 1,500-token prompt and 300-token reply come nowhere near. Once every candidate clears your prompt with margin, context length stops being a selection criterion and becomes a price question (below).
Tool use means the model can ask your system to run a function, such as an order lookup, and read the result. Streaming means tokens arrive as produced, the 400 ms first token from reading an architecture. Structured output means the model returns a shape your code can parse: a draft, a confidence class, cited article ids. All three are table stakes on every current line; a model missing one is disqualified, and a model with all four is not yet chosen.
Price tiers
Every vendor sells a ladder of tiers, and the rungs line up across vendors more closely than the names suggest.
| Tier | Anthropic | OpenAI | |
|---|---|---|---|
| Top | $10 per million input tokens and $50 per million output tokens (source, read 18 Oct 2026) | $10.00 per million input tokens and $50.00 per million output tokens (source, read 18 Oct 2026) | — |
| Large | $4 per million input tokens and $20 per million output tokens (source, read 18 Oct 2026) | — | $2.00 per million input tokens and $12.00 per million output tokens (source, read 18 Oct 2026) |
| Mid | $2 per million input tokens and $10 per million output tokens (source, read 18 Oct 2026) | $2.00 per million input tokens and $10.00 per million output tokens (source, read 18 Oct 2026) | — |
| Small | $0.10 per million input tokens and $0.50 per million output tokens (source, read 18 Oct 2026) | $0.10 per million input tokens and $0.50 per million output tokens (source, read 18 Oct 2026) | $0.30 per million input tokens and $2.50 per million output tokens (source, read 18 Oct 2026) |
On input, top to small is 100× and mid to small 20× on the two ladders with both rungs; the third's large-to-small gap is under 7×. This module assumes a tenth, inside that spread. At a tenth, $54,000 a month on the mid tier is $5,400 on the small tier, if the small tier can do the job; that "if" is the whole decision, and the eval settles it.
A third thing the table hides: the tier can change with prompt length. Anthropic's small model charges $0.50 per million input tokens and $2.50 per million output tokens on prompts over 100,000 tokens (source, read 18 Oct 2026), five times the short-prompt rate. Google's large model moves to $4.00 per million input tokens and $18.00 per million output tokens on prompts over 200k tokens (source, read 18 Oct 2026). OpenAI's short-context rates stop at 272K input tokens (source, read 18 Oct 2026). "Long context" is therefore a price tier with a cliff in it, not a free capability, and the engineer who proposes putting the whole help centre in every prompt has proposed a price change.
Latency is measured, not published
No vendor publishes a latency number you can put in a PRD. Anthropic's Models overview (opens in a new tab) ranks its tiers from fastest to slower and says actual latency depends on prompt length, output length and thinking effort: honest, and useless for a requirement. Measure it. Run 200 real prompts against each candidate at a working-day hour and record first-token and last-token times at p50 and p95. Those are your numbers; the vendor's adjective is not.
The mechanism to expect: smaller models produce tokens faster. A 300-token reply at an assumed 50 tokens a second takes 6 s; at 150 tokens a second it takes 2 s. So routing to a small tier is a latency lever as well as a cost lever, usually with room to spare under the 4 s p95 the design module needs.
Routing 70% to a small model
The drafting feature's eval says 70% of tickets need a short reply — a password reset, an order status, a "yes, that is covered". The remaining 30% need a policy read or an escalation. Nothing says both halves need the same model.
The arithmetic. Input cost on the mid tier at the assumed $3 per million is $900 a day (200,000 × 1,500 tokens). Route 70% of that to a model at a tenth of the price and the input line becomes $900 × (0.3 + 0.7 × 0.1) = $900 × 0.37 = $333 a day. Output on the routed tickets is also short, so the output line falls too; tokens as a cost model carries the full bill from $54,000 to about $27,500 a month.
The condition. The router sends a ticket to the small model only when the eval, run on the short-reply slice alone, puts the small model within 2 points of the mid tier. Why 2: a wrong draft the agent catches costs about 20 seconds to read and discard. On 140,000 routed drafts a day, each point of pass rate is 1,400 wrong drafts, 7.8 agent-hours, about $7,000 a month at an assumed $30 an hour. Two points cost $14,000 a month against the $17,000 the routed input line saves; three points cost $21,000 and the lever is underwater. A wrong draft that is sent costs a 20-minute escalation, which is why the factual-error class is gated separately in evals as the feature's test suite and never traded against a saving. A small-model draft flagged as low-confidence takes the dashed edge to the mid tier, and the eval scores the combined path.
The eval decides, not the demo
A demo is three tickets the engineer chose. An eval is 200 tickets the support lead labelled before anyone chose a model, scored by a rubric. The decision doc line reads: "Mid tier for policy and escalation tickets; small tier for short replies while the routed path scores within 2 points of the mid tier on that slice and 0 factual errors; revisit on any price change over 20% or any model update." The vendor is not in that sentence; the price table is in the appendix, dated.
In an interview
What is being tested is whether you can separate a product decision from a vendor's marketing. State the four columns before any name: "capability class, price tier, measured p95, golden-set score." Give the ratios with their date (100× top to small, 20× mid to small, a tenth in the plan) and state the routed split as a condition, not a number: "the share the eval clears." Then price a wrong draft in agent time; that number makes a tenfold saving look small.
The mistake that loses points is choosing by demo, or by "the newest model", which is the same mistake with a date on it. The second is quoting a context window as a reason — a million tokens does not help a 1,500-token prompt. The third is proposing a cheap model without stating the eval threshold that gates it, which an interviewer reads as a bill arriving later in agent time.
Check yourself
1. The small tier scores 89% on the short-reply slice; the mid tier scores 92%. Routing saves about $17,000 a month on the input line. Do you route?
No. 3 points on 140,000 routed drafts a day is 4,200 extra wrong drafts at 20 seconds each, 23 agent-hours a day, about $21,000 a month at $30 an hour — more than the $17,000 it saves, before counting any that get sent. Route only a sub-slice the eval puts within 2 points (order-status tickets, say; route or not works one), and send the rest to the mid tier.
2. An engineer wants to put the entire 3,000-article help centre, about 2.4 million words, into every prompt instead of running retrieval. What do you ask before answering?
Two things. Does it fit — 2.4 million words × 1.3 is about 3.1 million tokens, three times the largest window above, so no. And if a tenth of it fit, a 300,000-token prompt crosses every long-prompt threshold above, at two to five times the short-prompt input rate, on 200,000 calls a day. Retrieval stays.
3. A vendor cuts its mid-tier price by half overnight. Which lines in the decision doc change?
The price-tier column and the bill: $54,000 becomes about $27,000 on the mid tier alone, close to what routing achieved. The routing condition changes too, because it was derived in dollars: the routed input line now saves about $7,500 a month, and at $7,000 a month a point, routing pays only within about one point. That is the 20% revisit trigger firing as designed.