Cost, latency and routing · Routing

The router that saved 40% and cost 5 points

Take apart a length-based router's reported 40% saving using the cache counters and the per-slice eval, then say what should replace it.

critiquehardExercise · 4 min · 30 of 77

On this page

A routing change shipped with a slide that said 40% cheaper. The eval and the cache counters tell a different story, and the critique has to put a number on it.

The situation

A support assistant ran entirely on the large tier, reading 10M input tokens a day with a measured cache hit rate of 85%. Last month a router went live: any message of 300 tokens or fewer goes to the small tier, everything else to the large one. After the change, half of all input tokens go to each tier. The launch slide computed the saving from the price list — a blend of half at the small rate and half at the large rate against all at the large rate — and reported 40%.

Two weeks later the golden set (200 items) reads 87% against 92% before the change. The "billing and account closure" slice, 50 items, fell from 90% to 74%. The dashboard shows the cache hit rate at 60% on both tiers; the team had not looked at it.

Assume P_in = $3 per million input tokens on the large tier and $0.60 on the small tier (one-fifth), a cache read at C_r × P_in, with C_r = 0.1, on each tier, cache writes small enough to ignore, and output tokens unchanged by the router and left out of the sums.

The question

Decide whether the router should stay, with the real daily input cost before and after it rather than the slide's figure — and say what the team should do instead.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Before the router, the effective input rate with an 85% hit rate was 0.15 × $3 + 0.85 × $0.30 = $0.705 per million, so 10M tokens cost $7.05 a day.

After it, each tier sees 5M tokens at a 60% hit rate. Small: 0.4 × $0.60 + 0.6 × $0.06 = $0.276 per million, so $1.38. Large: 0.4 × $3 + 0.6 × $0.30 = $1.38 per million, so $6.90. Total $8.28 a day — 17% more than before. The slide's 40% compared list prices, $1.80 blended against $3.00, and never priced the cache; splitting traffic cut each cache's hit rate from 85% to 60%, and the lost reads cost more than the cheaper tier saved. Model selection and routing names this as the cost a cascade hides.

Five points overall on 200 items is about two standard errors, so it is probably real; sixteen points on the closure slice is unambiguous, and closure requests are the short ones — a length rule sends the highest-stakes messages to the small tier by construction. The router was never evaluated as a classifier, and the eval was read as one total.

Revert the router today: it costs more and scores less. Then, in order: run the large tier at lower effort on the golden set, the simpler candidate: one prompt, one cache; label 300 requests hard or easy and build a router scored on recall for the hard class; gate any routing on per-slice pass rates and on cost per task from the counters, not the price list. A router comes back only with hard-class recall above 95% and a hit rate back at 85% on the tier that matters.

Rubric

recomputes the daily cost before and after with the cache multiplier applied to each tier
shows that the slide's 40% came from list prices and names the lost cache reads as the cause of the increase
reads the 5-point total against its standard error and the 16-point slice as the real finding
identifies the length rule as a router that was never evaluated, and why it fails on short high-stakes messages
states the decision to revert and names the simpler alternative to compare first
names the evidence that would justify reinstating a router

Where answers go wrong

"Keep it — 40% is a lot, and 5 points can be tuned back with prompt work." The 40% does not exist; with the cache counted, the bill went up 17%. A critique that accepts the slide's number has not checked it, and prompt work cannot fix a router that sends closure requests to the wrong tier.

"Revert because quality dropped." Right decision, half the reasoning. Without the cost recomputation the team keeps believing the router was a trade of quality for money, and will try it again with a smaller quality drop. The finding is that it was a loss on both axes.

"Raise the length threshold to 500 tokens so fewer requests go to the small tier." Still a length router. Closure requests are short at any threshold, and the hit-rate loss comes from splitting traffic at all, not from where the split falls.