Context Rot: Three Ways a Longer Prompt Loses the Answer

Barakaeli Lawuo, Jul 18, 2026

In 2023 a team led by Nelson Liu at Stanford gave gpt-3.5-turbo-0613 a question and 20 Wikipedia passages, about 4,000 tokens in all. Exactly one passage held the answer. When that passage came first, the model answered 75.8% of questions correctly. When it came tenth, accuracy fell to 53.8%. When it came last, accuracy recovered to 63.2%.¹ The same model with no passages at all, answering from memory, scored 56.1%.¹ So with the answer sitting in the middle of its prompt, the model did worse than it did with no prompt documents whatsoever.

The authors called this shape a U-shaped performance curve. Models were best with relevant information at the very start of the input, which they call primacy bias, or at the very end, recency bias, and worst in between.¹ The paper is titled "Lost in the Middle," and the name stuck.

Position is one of three separate ways a longer prompt can lose an answer that is plainly inside it. The second is length itself: accuracy falls as the input grows, even when the relevant text is easy to find and the filler is blank space. The third is semantic distance: when the question and the passage that answers it share no words, the drop comes much sooner. Each has its own paper and its own measurements. "Context rot" is a popular name for the overall effect rather than a term from these papers; I use it here as shorthand for all three.

Context window

The largest number of tokens a model accepts in one request: instructions, pasted documents, conversation history and the question together.

Needle in a haystack (NIAH)

A test that hides one fact (the needle) inside long irrelevant text (the haystack) and asks the model to find it.

Distractor

Text in the prompt that looks related to the question but does not contain the answer.

Closed-book

Asking the model with no documents, so it can only answer from what it learned in training.

Literal match

The question and the passage holding the answer share the same words, so the model can find the passage by matching text.

Position: the tenth of twenty documents

The Lost in the Middle setup is close to how retrieval-augmented generation works in practice. Questions come from NaturalQuestions-Open, real Google searches with answers marked in Wikipedia; the team used 2,655 of them. Each prompt holds the one paragraph containing the answer plus $k-1$ distractor passages of at most 100 tokens, chosen by a retriever (Contriever) as the most relevant passages that do not contain the answer.¹ Moving the answer passage around changes where the information sits but not what the correct output is. Adding distractors changes the length but not the answer.¹

The U showed up in every model they tested, open and closed: MPT-30B-Instruct, LongChat-13B (16K), GPT-3.5-Turbo and Claude-1.3.¹ It also got wider with more documents. With 30 documents, GPT-3.5-Turbo (16K) went from 73.4% at position 1 to 50.5% at position 10.¹ The paper states it directly: in the 20- and 30-document settings, the worst case was below closed-book performance.¹

Bigger windows did not help. GPT-3.5-Turbo has a 4K window and GPT-3.5-Turbo (16K) a 16K one; when the prompt fit in both, their curves were "nearly superimposed." Claude-1.3 and Claude-1.3 (100K) matched the same way.¹ A bigger window lets a model accept more text. It says nothing about how well the model reads it.

To rule out reading comprehension as the cause, the authors built a stripped-down test: a JSON object of 75, 140 or 300 key-value pairs, every key and value a random 128-bit UUID, and a request for the value of one key. The task only needs an exact string match. Claude-1.3 did it almost perfectly at every length. GPT-3.5-Turbo, GPT-3.5-Turbo (16K) and MPT-30B-Instruct did worst when the key sat in the middle.¹

Where the bias comes from is less settled. Encoder-decoder models (Flan-T5-XXL, Flan-UL2), whose encoder reads the whole input in both directions, were fairly flat across positions as long as the input stayed within their training length. Flan-UL2 varied by only 1.9 points between best and worst case. Past that length, the U came back.¹ Instruction tuning was not the cause: base MPT-30B, before any instruction tuning, showed the U too.¹ In Llama-2, the 7B models were only recency-biased, while the 13B and 70B models showed the full U.¹ The authors point out that humans do something similar when recalling a list, called the serial-position effect, and that it is surprising in a Transformer, since self-attention can in principle retrieve any token equally well.¹

Length: the drop that happens even with nothing in the way

By 2024, long-context models were routinely scoring almost perfectly on the standard needle-in-a-haystack test. An NVIDIA team led by Cheng-Ping Hsieh argued that this only checks a shallow form of understanding, and built RULER: 13 synthetic tasks at lengths from 4K to 128K tokens, including needles of different types, several needles at once, tracking a variable through a chain of assignments, counting word frequencies, and question answering with distractor paragraphs.² They tested 17 models that all claimed windows of 32K tokens or more.²

That test gave the field a precise term. RULER defines effective context length as the longest tested input at which a model's average score across all 13 tasks still beats a fixed threshold: 85.6%, the score of Llama2-7B at 4K tokens. The claimed length is whatever the model's makers advertise.² Measured this way, only half of the models held up at 32K tokens, and almost all fell below the threshold before reaching their claimed length.²

Some drops were steep. DBRX claims 32K and scored 95.1 at 4K but 63.1 at 32K. Mistral-v0.2 7B, also claiming 32K, went from 93.6 to 75.4 at 32K.² Gemini-1.5-Pro was the one model whose effective length exceeded 128K, the longest length they tested.² A closer look at Yi-34B, which claims 200K, found concrete failure modes as inputs grew. Filling the haystack with lookalike needles cut its score by about 40 points at 256K, and it often returned a value from near the target instead of the target itself. On question answering, its accuracy sank toward what it scored with no context at all.² Training on longer sequences did not reliably fix this: a model from the same family trained to 1M tokens was worse at 256K than one trained to 512K.²

RULER's haystacks are still made of text, though, so any drop could be blamed on noise. A 2025 paper by Yufeng Du and colleagues removed that excuse. They split GSM8K math problems, MMLU questions, HumanEval coding tasks and a simple variable-sum task into evidence and question, and padded the gap between them to a set length. First they checked retrieval as strictly as possible: the model had to recite the evidence word for word. Llama-3.1-8B-Instruct recited it exactly for 970 of 1,000 MMLU problems padded to 30K tokens, as often as it did on the short versions, yet its accuracy dropped by 24.2%.³

Then they swapped the essay padding for whitespace, which carries almost no information. Performance still fell, by at least 7% for both Llama and Mistral on every task at 30K space tokens, by 48% for Llama on the variable sum, and by 30% for Mistral on GSM8K.³ Moving the evidence to sit right before the question, so the distance never changed, still produced drops of up to 20% for Llama and 17% for Mistral.³ Finally they masked the padding so the model's attention could not see it at all. Llama-3's variable-sum accuracy still went from 97.0 to 47.0, and its HumanEval score from 57.3 to 7.3, at 30K masked tokens.³ The authors' conclusion is that the sheer length of the input alone can hurt performance, independent of retrieval quality and without any distraction.³ The closed models they tried (GPT-4o, Gemini-2.0, a Claude model) lost less, and GPT-4o and Gemini-2.0 held a perfect score on the variable sum throughout.³

Semantic distance: when the question shares no words with the answer

Ali Modarressi and colleagues at LMU Munich and Adobe Research noticed something about the tests above. In almost all of them, the question repeats words from the passage that answers it. They measured this with ROUGE-1 precision, the share of the question's words that also appear in the relevant text: 0.905 for the vanilla NIAH test and 0.571 for RULER's single-needle task.⁴ If the answer can be found by matching text, the benchmark mostly tests matching.

Their benchmark, NoLiMa (short for No Literal Matching), got that overlap down to 0.069.⁴ A needle reads "Actually, Yuki lives next to the Semper Opera House." The question is "Which character has been to Dresden?" To answer, the model has to know that the Semper Opera is in Dresden; no word links the two. Two-hop versions ask about the state of Saxony instead.⁴ The haystacks are snippets of books, filtered to remove words similar to the question and any passage that could be read as a wrong answer.⁴ Each model's base score comes from its best results at the shortest lengths (250, 500 and 1K tokens), and NoLiMa counts a length as effective while the score stays above 85% of that base.⁴

All 13 models tested claim at least 128K tokens. At 32K, 11 of them scored half or less of their base score.⁴ GPT-4o went from 99.3 to 69.7.⁴ Llama 3.1 70B, with an effective length of 64K in RULER's table,² has an effective length of only 2K on NoLiMa, where it scored 42.7% at 32K.⁴ The two benchmarks set their thresholds differently, but most of the gap comes from one change: the question no longer gives away where to look.

The authors checked the literal-match explanation directly with Llama 3.3 70B. When the question named the needle's own keyword, accuracy at 32K was 98.5%. The one-hop version scored 56.2%. Keeping the one-hop question but offering four names to choose from, one of them correct, raised it to 93.1%.⁴ They also looked at position. For one-hop questions they saw a lost-in-the-middle dip at 32K. For two-hop questions, longer inputs pulled the whole curve down toward zero, including at the start and end of the context.⁴ So as questions get harder, length matters more than position.

Overlap can also point the wrong way. NoLiMa inserted a sentence that contained the question's keyword but had nothing to do with the answer; GPT-4o's effective length fell to 1K.⁴ Freda Shi and colleagues found the same thing at a much smaller scale in 2023. Adding one irrelevant sentence to grade-school math problems dropped code-davinci-002 with chain-of-thought from 95.0% on the clean problems to 72.4%, and only 6.0% of base problems were solved under every distractor variant.⁵ The hardest distractors were on-topic and reused a character's name from the problem.⁵ That is what a retriever tends to return: passages that look like the question.

What the papers found helps

Shorter prompts. Du et al. had the model first recite the evidence it needed, then answer from a fresh prompt containing only that evidence and the question. On GSM8K padded with essays to 26,250 tokens, Mistral-7B went from 35.5% to 66.7%. On RULER's second QA task, GPT-4o gained up to 4 points, from 68.4 to 72.4 at 32K.³ Liu et al. saw the retrieval side of the same point: going from 20 to 50 retrieved documents raised accuracy by only about 1.5 points for GPT-3.5-Turbo and 1 point for Claude-1.3, while making the input much longer.¹ They suggest reranking, so relevant passages move toward the start, and ranked list truncation, meaning retrieving fewer documents when that is enough. They propose both as promising directions rather than testing them.¹

Telling the model about distractors. In Shi et al., adding "Feel free to ignore irrelevant information given in the questions" raised chain-of-thought accuracy from 72.4% to 77.8%. Using worked examples that themselves contained irrelevant sentences raised it to 76.8%, without hurting accuracy on the original clean problems.⁵ Sampling 20 answers and taking the majority vote (self-consistency) raised it to 88.1%.⁵

Placing the question before and after the data. For the UUID key-value task this fixed nearly everything: GPT-3.5-Turbo (16K) went to perfect accuracy on 300 pairs, where its worst case had been 45.6%.¹ RULER found that larger models degraded less. Yi-34B beat Yi-6B at the same 200K training length, both at 4K and in how much it lost as length grew.²

What the papers found does not help, or not enough

Placing the question on both sides barely changed the multi-document QA curve, raising the first position slightly and lowering the others.¹ Extended-context versions of the same model read no better.¹ A longer training context did not reliably help on RULER.² Chain-of-thought helped on NoLiMa but did not close the gap: two-hop accuracy for Llama 3.3 70B at 32K went from 25.9% to 34.3%.⁴ On the ten hardest NoLiMa pairs, reasoning models such as GPT-o1 scored near 100 at short lengths and fell below half of that at 32K; GPT-o1 went from 99.9 to 31.1.⁴ Even the best combination in Shi et al., least-to-most prompting with self-consistency, consistently solved only 45% of base problems across all distractor variants.⁵ And the recite-then-answer trick has a condition its authors spell out: it requires near-perfect retrieval, which is already hard in many real tasks.³

The number RULER leaves out

RULER's effective context length is now the usual way to say how much of a window a model can use. Its authors list what it does not measure. RULER reports one number per input length with no breakdown by depth, so it cannot show the lost-in-the-middle effect, and the paper says they plan to add control over where the relevant information goes.² Its tasks were chosen because most models do well on them at 4K, which the authors warn should not be read as proof that models are fine at 4K; other work has found drops at inputs of just a few thousand tokens.² And they did not verify that their synthetic tasks track realistic long-context work, so they recommend treating RULER as a convenient behavioral check, not as a replacement for more realistic evaluations.²