A support-triage classifier runs at temperature 0. CI has a test that sends one real ticket — "I was charged twice and now I can't log in" — and asserts that the label is billing. About one run in fifteen it fails, returning account. A pull request proposes retries: 3 on the test so that a failing run re-executes until it passes, with the comment "temperature 0 is deterministic, so this is an API glitch". Reviewers have approved it because the red build is blocking releases.
The team has a 200-item labelled set it ran once, four months ago, when the prompt scored 182 of 200. It has not been run since; the prompt has changed twice. Each classifier call is ~800 tokens in and 5 out, and finishes in under two seconds. Assume ¾ of a word per token; the price does not matter for this one — 20 calls of 800 tokens is 16k tokens, whatever P_in is. The relevant lesson is sampling and determinism.
Temperature 0 narrows the sampler to the top token; it does not make the top token the same on every call. Batched inference changes which kernels run and in what order they add, floating-point addition is not associative, and the model behind the name is updated. A ticket about a double charge and a failed login is genuinely both billing and account, so its top two labels sit within that noise, and one flip in fifteen is the expected behaviour of a near-tie item, not an API glitch. The test measures one item's noise and calls it correctness. Retrying until green hides the agreement rate, the information the failure carried, and makes the next model update invisible.
Replace it in three steps. First, run the ticket 20 times at the shipped settings — 16k tokens, under a minute — and record agreement; say 18 of 20. That number says the item is a boundary case. Second, write the tie-break rule the boundary needs — "a ticket that mentions a charge, refund or invoice is billing even if it also mentions logging in" — re-run the 200-item set, and read the paired diff: which tickets flipped and which way. Third, make CI gate on the set, not the item: a pass rate at or above 90% on 200 with the paired diff against the last run, on every prompt or model change. Keep single-item assertions only for items at 20 of 20, asserting on the parsed label.
The evidence that would change this: 100 runs with no account, which a one-in-fifteen item manages about once in a thousand. Then the failure is in what CI does differently — its prompt version, model or parser — a different investigation.
"Add the retries; it is an API glitch." Three retries are four attempts, which turn a 1-in-15 failure into roughly 1-in-50,000, so the build goes green and the team stops seeing a prompt that cannot decide between its two most common labels. The reviewer is scoring whether the candidate knows the failure was data.
"Set a seed and the problem goes away." A seed reproduces a run on the same serving configuration; it does nothing when the batch, the backend or the model changes, and the test would still assert one item's noise.
"Delete the test — model output cannot be unit-tested." The output can be measured: an agreement rate per item and a pass rate per set. Deleting the test replaces a noisy measurement with none.
