Sampling and determinism
Set temperature and top-p for the job, measure agreement with a 20-run repeat test, and put determinism in the harness instead of expecting it from temperature 0.
On this page
A model does not return a token; it returns a probability for every token in its vocabulary, and a sampler picks one. Temperature and top-p are the sampler's settings, chosen per request, and they decide how far two calls with the same prompt can drift apart. The test that "fails one run in fifteen" is almost always a sampling fact being mistaken for a bug.
The distribution and two knobs
At each step the model produces a score per vocabulary token. The sampler divides the scores by the temperature, turns them into probabilities, and draws one token. Temperature 1 samples the distribution as trained. Below 1 the distribution sharpens — mass moves to the top tokens — and as it approaches 0 the draw approaches "always the top token". Top-p truncates instead of sharpening: sort the tokens by probability, keep the smallest set whose mass reaches p, renormalise, and draw from that set. Top-k keeps a fixed count. Set one of them deliberately and leave the other at its default; stacking all three makes a result nobody can explain.
Where to set it follows from the job. Classification, extraction, code edits and anything a parser reads: low temperature, 0 to 0.3, because you want the most likely answer and nothing else. Drafting, naming, brainstorming: 0.7 to 1, because the second-most-likely phrasing is often the better one and you want the spread. The two mistakes are mirror images. A classifier run at temperature 1 flips 12 of 200 labels between runs for no reason anyone asked for; an idea generator run at 0 returns the same five ideas to every user.
Why temperature 0 is not a guarantee
At temperature 0 the sampler takes the highest-probability token. Three things change which token is highest between two calls with identical input.
Batched inference. Your request is computed alongside other people's. The batch's shape chooses which kernels run, and different kernels add the same numbers in a different order. Floating-point addition is not associative, so a sum over thousands of terms lands on slightly different last bits depending on the order. When two tokens are within those last bits of each other — the 34% and 31% in the diagram, sharpened to 61% and 38% — the top one flips. One flipped token is enough: every token after it is conditioned on a different prefix, and the two outputs diverge from that point on, not by a word but entirely.
The model behind the name. A model name is an alias for whatever the provider serves under it today. Weights are updated, the serving stack changes, a tokenizer is revised, and the distribution behind the same name moves. Pin a dated version where the API offers one and record it in every trace, so that a change in behaviour has a suspect.
Expect, then, the practical shape: identical output most of the time, and a different output on the items where two continuations are close. Those items are not random; they are the boundary cases of your task.
Seeds, where the API offers them
Some APIs accept a seed and return a fingerprint of the serving configuration. The same seed with the same fingerprint gives the same output on a best-effort basis; when the fingerprint changes, so may the output. Treat the seed as a tool for reproducing a bug — same input, same seed, same trace — not as a correctness mechanism. It does nothing against the model-behind-the-name problem, and it holds only as long as the backend does.
The 20-run repeat test
Measure agreement instead of assuming it. Take one input, run it 20 times at the settings you ship, and compare the field that matters — the parsed label, the extracted amount — not the raw string. Agreement is the share of runs that match the majority: 18 of 20 is 90%.
The cost is small. Twenty runs of a 1.5k-token prompt is 30k input tokens, 30k × P_in, and a few hundred output tokens; it finishes in under a minute with runs in flight together. The reading is what matters. Twenty of twenty: the item is stable at these settings. Seventeen to nineteen: a boundary item — the prompt needs a tie-break rule for the pair it wavers between, or the item is genuinely ambiguous and belongs in a slice of its own. Sixteen or fewer: the prompt is not doing the job on this kind of input. Run the test on the items where your golden set disagrees with the model, because that is where the boundaries are.
Statistics keep the reading honest. If the true per-run agreement is 95%, 20 runs show one disagreement on average, so seeing zero proves little and seeing five says the true rate is well below 95. A single-item assertion in CI at temperature 0 measures one item's noise and calls it correctness; the flaky classifier test exercise takes that test apart. The fix is never a retry-until-green: a retry hides the agreement rate, which was the information.
Determinism from the harness, not the sampler
Where the feature needs the same answer every time, put the guarantee in code. Validate the output against a schema and retry on failure, so a divergent run that breaks the format is caught before anyone reads it. Make side effects idempotent, so a retried call is not two refunds. Cache the result keyed by a hash of the input, so a repeated question gets the identical answer at no cost. And where a classification must be stable, take three samples and a majority vote.
The vote costs three times the call, so apply it where it pays. An extraction feature at 10,000 calls a day with a vote on every call is 30,000 calls; a vote on the 5% of calls that fail an invariant check or land on a known-ambiguous slice is 11,000 — 10,000 plus 2 × 500 — for most of the agreement gain. The decision is which items are worth 3×, not whether to vote.
In an interview
What is being tested is whether you know what temperature 0 buys and what it does not. The phrasing that scores: "temperature 0 narrows the sampler to the top token; it does not fix which token is top when two are close, and batching, floating-point order and model updates move that. The harness gives determinism — schema checks, idempotent side effects, a vote where it pays — and a 20-run repeat test gives the number." Name the agreement rate as a metric you would track.
The mistake that loses points is "set temperature to 0 and the output is deterministic". The second is not knowing that temperature and top-p are alternatives — sharpening versus truncation — and setting both to a value copied from somewhere.
Check yourself
1. A classifier runs at temperature 0. A CI test sends one real ticket and asserts the label; it fails one run in fifteen. What do you do?
Run the 20-run repeat on that ticket; one failure in fifteen predicts about 19 of 20. At 17 to 19 of 20 it is a near-tie item: write a tie-break rule for the pair it wavers between, or move it to a known-ambiguous slice. Then replace the single-item assertion with a set-level gate — a pass rate over 200 items with a paired diff against the last run — and keep single-item assertions only for items at 20 of 20, on the parsed field. Retrying the test to green is the one option that destroys the information.
2. An extraction feature wants 99% agreement on the amount field at 10,000 calls a day. A three-way vote everywhere costs 3×. Where do you spend it?
Not everywhere. Run the 20-run test on 50 sampled invoices to find where agreement is already at 20 of 20 — most of them — and vote only on calls that fail the arithmetic invariant or come from the supplier slices that showed disagreement. If 5% qualify, that is 10,000 + 1,000 = 11,000 calls a day, a 10% increase, against 30,000 for a blanket vote.
3. Product says the idea generator "always says the same thing". It runs at temperature 0. What changes, and how do you know it worked?
Temperature 0.8 to 1 with top-p around 0.95, and a different metric: the number of distinct ideas across 20 runs, not agreement. The same 20-run test measures spread as well as stability; here the target is high spread with every idea still passing the format check.