One distribution, three dials, and a temperature zero that still wobbles

Barakaeli Lawuo, Jun 26, 2026

Five requests, five different programs

Shuyin Ouyang and colleagues at King's College London, UCL and Bristol took 829 programming problems from three benchmarks and sent each one to ChatGPT through the API five times, with the identical prompt every time.¹ Then they ran the five programs they got back against the benchmark tests and compared the outputs. On CodeContests, a set of 165 competitive programming problems, 75.76% of the problems had an output equivalence rate of zero: across the five programs, not a single test produced the same output twice.¹ On APPS the figure was 51.00%, and on HumanEval, 47.56%.¹

Correctness moved too. On 39.63% of HumanEval problems, one of the five programs passed every test and another passed none, the largest swing the metric allows.¹ These runs used GPT-3.5 at the API's default temperature of 1, with everything else left at its default except the model name and the prompt.¹ The authors also counted how often papers account for this. Out of 76 recent papers on LLM code generation, 21.1% considered non-determinism in their experiments.¹

None of this is a malfunction. A language model does not produce an answer. It produces a probability for every possible next token, and a separate piece of code, the sampler, picks one. The authors put it plainly: the randomness usually comes from the sampling method, so identical prompts "can yield completely different responses."¹ Three settings control that pick, and they all act on the same distribution. What follows explains each one using the paper that measured it, then covers the part that surprises people: setting the temperature to 0 does not make the output repeatable either.^1,5,6

Token: a word or piece of a word. The model generates text one token at a time.
Logit: the raw score the model gives each token in its vocabulary before the scores are turned into probabilities.
Decoding: the rule that turns the model's probabilities into an actual next token, step after step.
Greedy decoding: always taking the single most probable token. This is what temperature 0 is meant to do.
Beam search: keeping the b most probable partial sequences at each step and returning the most probable complete one.

The distribution, and the first dial

At each step the model outputs one logit per vocabulary entry. The softmax turns those logits into probabilities, and temperature is a single number that divides every logit before the softmax runs. Holtzman et al. write it this way:²

\begin{gathered} p(x = V_l \mid x_{1:i-1}) \\[4pt] = \frac{\exp(u_l / t)}{\sum_{l'} \exp(u_{l'} / t)} \end{gathered}

Softmax with temperature, equation 4 of Holtzman et al., 2020.² Atil et al. use the same form.⁵

Read it term by term. $x_{1:i-1}$ is everything generated so far, prompt included. $V_l$ is the $l$ -th token in the vocabulary, and $u_l$ is its logit. $t$ is the temperature. The top of the fraction scores one token and the bottom adds up the scores of every token, so the results are positive and sum to 1. That is a probability distribution, and the sampler draws the next token from it.²

Dividing by $t$ is what makes it a dial. At $t = 1$ the model's own probabilities come through unchanged. A $t$ between 0 and 1 stretches the gaps between logits, which, in the paper's words, "skews the distribution towards high probability events" and lowers the mass in the tail.² A $t$ above 1 flattens the distribution.⁵ As $t$ shrinks toward 0, nearly all the probability piles onto the top token. The formula itself cannot be evaluated at exactly 0, since that would divide by zero, so APIs treat temperature 0 as a rule: always take the highest-probability token, which is greedy decoding.^1,6

Ouyang et al. reran their CodeContests experiment at three temperatures. Lower temperature did make the five programs agree more often, but it never got close to full agreement.¹

Redrawn from Table 5 of Ouyang et al.¹ 165 problems, five separate API requests each, with GPT-3.5. OER is the share of test outputs that are identical across the five programs. The mean test pass rate stayed almost flat, between 0.15 and 0.16, at all three temperatures.

That flat pass rate led the authors to suggest low temperature for code, since it gave comparable correctness with less variation.¹ Temperature has a cost in open-ended writing, though, and that cost is where the other two dials come from.

Why the most likely text is bad text

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi studied decoding for open-ended generation, meaning tasks like continuing a story where many continuations are acceptable.² They used GPT-2 Large and generated 5,000 passages of up to 200 tokens, each continuing the opening of a held-out WebText document.² The obvious approach is to ask for the most probable text, through greedy decoding or beam search. The paper shows that this approach degenerates: the output is "bland, incoherent, or gets stuck in repetitive loops."²

The numbers are stark. The authors counted a generation as stuck when a phrase of at least two tokens repeated at least three times at its end. Human text did this 0.28% of the time. Greedy decoding did it 73.66% of the time, and beam search with a beam of 16 did it 28.94% of the time.² Perplexity tells the same story. Perplexity measures how surprised the model is by a text, with lower meaning more predictable. The model scored human continuations at 12.38 and its own greedy output at 1.50.² So the greedy text was far more predictable to the model than anything a person actually writes. In the paper's first example, a beam of 32 repeats a university's name over and over, and the caption notes that at beam widths of 64 or more GPT-2 Large and XL prefer to stop generating right after the prompt.²

The paper gives two reasons. First, repetition feeds itself: each time a phrase repeats, the model assigns it a higher probability of repeating again, a positive feedback loop the authors found for the vast majority of phrases they tested.² Second, people do not write the most probable next word. Natural text rarely stays in a high-probability zone for several tokens in a row and keeps veering into less likely, more informative words. The authors conjecture this is a property of human language and link it to Grice's maxims, since people avoid stating the obvious.² Fan et al. had seen the same thing in story generation two years earlier: beam search stories "tend to be short and generic."³

Sampling from the full distribution, which the paper calls pure sampling, fails the other way. Its perplexity was 22.73, worse than human text, and its output wanders off topic. In one sample it made up a word, "umidauda," apparently a bird species.² The authors blame the unreliable tail: tens of thousands of low-probability tokens that each have a tiny chance, but together are sampled far too often.² Lowering the temperature shrinks the tail, but it also brings back repetition. The paper reports that temperatures below 0.9 "severely increase repetition."²

Second dial: keep the top k

The blunt fix is to remove the tail. Top-k sampling keeps the $k$ most probable tokens, sets every other probability to zero, rescales the survivors so they sum to 1, and samples from that. Fan et al. used it with $k = 10$ , noting that completely random sampling "can introduce very unlikely words" and that restricting to the top 10 reduces that risk.³ Holtzman et al. note that the best-known GPT-2 samples, such as OpenAI's unicorn article, were also produced with top-k sampling.²

Holtzman et al. argue that no single $k$ is right, because the shape of the distribution changes from one step to the next. Sometimes the head is flat, with tens or hundreds of reasonable tokens, as with nouns or verbs in a generic context. Sometimes one token holds almost all the mass. A small $k$ makes flat steps bland, and a large $k$ lets bad candidates into peaked steps, where rescaling actually increases their chance of being picked.²

Third dial: keep the top p of the probability

Nucleus sampling, the paper's proposal, cuts by probability mass instead of by count. At each step it takes the smallest set of tokens whose probabilities add up to at least $p$ , then rescales and samples within that set.²

\begin{gathered} V^{(p)} = \text{smallest set with} \\ \sum_{x \in V^{(p)}} P(x \mid x_{1:i-1}) \ge p \\[6pt] P'(x \mid x_{1:i-1}) = \frac{P(x \mid x_{1:i-1})}{p'} \;\text{ if } x \in V^{(p)}, \text{ else } 0 \end{gathered}

Nucleus (top-p) sampling, equations 2 and 3 of Holtzman et al.²

p'

is the total probability of the tokens that were kept.

$V^{(p)}$ is the nucleus. When the model is confident, the nucleus might be one token. When it is unsure, the nucleus grows. The authors report it usually holds between one and a thousand candidates.² Top-k uses the same rescaling step. The only difference is where the cut falls.² One setting, like $p = 0.95$ , adapts to both flat and peaked steps, which a fixed $k$ cannot do.

To judge quality and diversity together, the paper used HUSE, a measure from Hashimoto, Zhang and Liang. It is twice the error rate of a classifier that tries to tell human text from model text using two signals: the model's probability for the text and human ratings of how typical it reads.⁴ If the classifier cannot tell them apart, HUSE approaches 1. Holtzman et al. collected 20 human ratings for each of 200 generations per method, 4,000 ratings per method.²

Redrawn from Table 1 of Holtzman et al., 2020.² Greedy and beam search have no HUSE score in the table. For top-k and nucleus the authors computed HUSE on a distribution mixed with 0.1 of the original, because the truncated versions scored near 0 despite humans favoring them.

Nucleus sampling scored highest and top-k came second.² Its perplexity was 13.13, against 12.38 for human text, and its repetition rate was 0.36%.² The pairing common at the time, top-k 40 with temperature 0.7, did worst. It scored 0.08, and its repetition rate was 8.86%.² The caveat in the caption matters: the truncated methods had to be scored on a smoothed version of themselves, so the HUSE comparison is not perfectly like for like.²

This is why hosted APIs expose temperature and top_p as separate settings. Ouyang et al. kept top_p at its default of 1, which switches nucleus truncation off, and changed only the temperature.¹

Temperature 0, and the answers still change

Greedy decoding has no randomness in it. Given the same logits, it picks the same token every time. Ouyang et al. point out that the logits should be "a pure function of the input sequence and the model weights."¹ Yet at temperature 0, 43.64% of their CodeContests problems still had five programs with no shared test output. The figures were 27.40% on APPS and 18.29% on HumanEval.¹ The authors call this "contrary to many people's belief."¹

Berk Atil and colleagues from Penn State and Comcast went further. They tested five models, GPT-3.5 Turbo, GPT-4o, Llama-3-70B-Instruct, Llama-3-8B-Instruct and Mixtral-8x7B-Instruct, on eight multiple-choice tasks from BBH and MMLU. Each ran ten times with temperature 0, top-p 1 and a fixed seed.⁵ Accuracy varied by up to 15% between runs, and the gap between best possible and worst possible accuracy reached 70%.⁵ Best possible accuracy counts a question as right if any run got it right. Worst possible counts it wrong if any run got it wrong.⁵

Redrawn from Table 2 of Atil et al.⁵ 100 questions, 5-shot prompts, ten runs per model with identical inputs and settings. With deterministic decoding all three bars would be the same height.

Raw strings were less stable than answers. Atil et al. define TARr@10 as the share of questions where all ten raw responses match character for character, and TARa@10 as the share where the parsed answer matches.⁵ For GPT-4o on college math, TARa@10 was 50.0% and TARr@10 was 0.0%. For none of the 100 questions did it write the same response ten times.⁵ Longer outputs were less stable, and the authors suggest capping maximum output length for that reason.⁵ Their engineering note is simple arithmetic: a dialog system built from four classifiers that are each 95% stable is $0.95^4 \approx 0.814$ stable overall, before accuracy on new inputs is even counted.⁵

Floating point, and the batch you share

Ouyang et al. describe the cause as "still controversial." The hypotheses they list are floating point, unreliable GPU calculations, and a sparse mixture-of-experts architecture that fails to enforce per-sequence determinism.¹ The basic fact behind the first one is that floating point addition is not associative. $(a + b) + c$ can differ from $a + (b + c)$ , because every addition rounds to a fixed number of significant digits. The Thinking Machines report by Horace He gives the example that $(0.1 + 10^{20}) - 10^{20}$ comes out as 0 while $0.1 + (10^{20} - 10^{20})$ comes out as 0.1.⁶ If a sum is computed in a different order, the result can change in the last bits. At temperature 0, a change that small is enough to flip which of two nearly tied tokens ranks first, and after that the whole continuation differs. (That last step is my reading of how the pieces connect. The report shows the divergence without stating it this way.)

The common explanation is that parallel GPU threads finish in random order. The report argues that this is not the main culprit for inference. It says a typical LLM forward pass contains no atomic adds, the operations whose order depends on which core finishes first, so the forward pass is run-to-run deterministic: the same batch in gives bitwise the same result out.⁶ The problem is that the kernels are not batch invariant. To keep the GPU busy, a kernel picks its strategy by batch size: with a small batch it may split each sum across cores (a split-K matmul) or use different tensor-core instructions, and either choice changes the order in which a row's numbers are added.⁶ The report demonstrates this in PyTorch: multiplying one row alone and multiplying it as part of a 2,048-row batch gives results for that row that differ by up to 1669.25.⁶ On a shared server, the batch size depends on how many other users' requests are in flight. For any single user, that is outside their control.⁶

The report measured it. It sampled 1,000 completions of "Tell me about Richard Feynman" from Qwen3-235B-A22B-Instruct-2507 at temperature 0, 1,000 tokens each. There were 80 unique completions, and the most common appeared 78 times.⁶ All 1,000 matched for the first 102 tokens. At token 103, after "Feynman was born on May 11, 1918, in," 992 continued with "Queens, New York" and 8 with "New York City."⁶ With batch-invariant kernels enabled, all 1,000 were identical. That determinism had a cost: on Qwen-3-8B, 1,000 requests took 26 seconds with default vLLM, 55 seconds with the unoptimized deterministic version, and 42 seconds with an improved attention kernel.⁶ The report is a technical blog post from a company, not a peer-reviewed paper, and should be read that way.

Atil et al. arrived at the same suspect from the other side. They ran Llama-3-8B on their own GPUs without serving optimizations and got deterministic results. They name continuous batching, chunked prefill and prefix caching as optimizations that might introduce non-determinism.⁵ Continuous batching is exactly the practice of mixing different users' requests into one batch.

The hosted-model half of this story is the least certain part, and the paper says so. Atil et al. write that because GPT-3.5 and GPT-4o are closed source, and all five models were "hosted behind APIs we don't control, we can only speculate about the reason for this behavior."⁵ The batching explanation is tested on open engines and local GPUs. For the hosted APIs where the variation was measured, it is still an inference.