The tech a PM needs · Models and cost

Route or not

Decide which ticket classes go to a small model from per-class eval scores, design the router's default and its gate, and net the saving against agent time.

designmediumExercise · 4 min · 34 of 59

On this page

Model selection routed 70% of tickets to a small model on one aggregate number. The eval now scores each ticket class on its own, and one class inside that 70% loses money when it moves.

The situation

The drafting feature serves 200,000 requests a day on the mid-tier model. The output cap is already in, so the bill is $44,550 a month against a $30,000 budget. Assume $3 per million input tokens and $15 per million output on the mid tier, a tenth of both on the small model, 1,500 input tokens a call, 150 output tokens on the three short classes and 300 on the other two. On the routed path the small model flags about 10% of its drafts as low-confidence, and those are re-run on the mid tier, which pays for both calls. A wrong draft the agent catches costs 20 seconds at an assumed $30 an hour, and the launch gate blocks any path above 0.5% factual errors. The support lead labelled 1,000 tickets by class; the eval scored the mid tier and the whole routed path, router included:

Class	Share	Mid tier	Routed	Routed factual errors
Order status	35%	95%	94%	0.2%
Account and login	15%	95%	93%	0.3%
Is this covered	20%	92%	86%	1.2%
Policy question	20%	89%	74%	2.5%
Escalation	10%	85%	60%	4.0%

The question

Decide which classes go to the small model, what the router does with a ticket it cannot place, and what keeps the split honest after a model update. State the monthly saving net of agent time.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Price a routed short call first. On the mid tier it is (1,500 × $3 + 150 × $15) ÷ 1M = $0.00675. The small model costs a tenth, $0.000675, and the 10% escalated pay the mid tier again, another $0.000675, so routing saves $0.0054 a call, 80% of its cost. A lost point costs 1% × 20 s at $30 an hour, $0.0017 a draft, so the break-even is 0.0054 ÷ 0.0017 ≈ 3.2 points. The rule is 2, which keeps a point of margin for noise in a few hundred labelled cases.

The routing table:

Class	Goes to	Why
Order status, −1	small	saves $11,340 a month, costs $3,500 of agent time
Account and login, −2	small	saves $4,860, costs $3,000
Is this covered, −6	mid	would save $6,480 and cost $12,000; fails the factual gate
Policy, escalation	mid	−15 and −25

Net: $16,200 saved, $6,500 spent in agent time, $9,700 a month, and the bill falls to $28,350, inside budget.

The mechanism. The router labels the class, and a ticket it cannot place goes to the mid tier, so the default is the expensive path, not the cheap one. The low-confidence flag sends a draft to the mid tier before the agent sees it. The eval runs per class, end to end with the router in place, on every model update and weekly; a routed class that falls more than 2 points below its mid-tier score reverts to the mid tier without a meeting.

What would change it: account and login sliding to −3, past the margin, or a mid-tier price cut, which shrinks every saving in the table.

Rubric

Prices a routed call including the escalated re-runs, prices a lost point in agent time, and states the break-even.
Routes by class from the per-class scores, not by the 70% aggregate.
Keeps "is this covered" on the mid tier and gives both reasons: a net loss and a factual-error rate above the gate.
Makes the mid tier the default for any ticket the router cannot place, and scores the routed path end to end with the router included.
Names the per-class re-run on every model update and the rule that reverts a class.
States the net monthly saving and the bill against the budget.

Where answers go wrong

"Route the 70% of short tickets, as the lesson did." The aggregate hides "is this covered": 6 points down, $12,000 a month of agent time to save $6,480, and 1.2% factual errors that fail the gate on their own. Blanket routing nets $4,180; routing by class nets $9,700.
"Do not route; any drop in pass rate is a regression." The bill stays at $44,550, 1.5× the budget, to protect order status from a 1-point drop that costs $3,500 a month against $11,340 saved. A regression has a price, and here it is the cheaper side.
"Send everything small and let the low-confidence flag catch the hard ones." The flag is already inside the routed scores, and escalations still land 25 points down: on 20,000 drafts a day that is about $25,000 a month of agent time to save $4,320, before counting the 4.0% factual errors.