What RAG is, and why it helps with obscure facts more than famous ones

Barakaeli Lawuo, Feb 4, 2026

In December 2022 a team from the University of Washington, Johns Hopkins and the Allen Institute for AI released a question set designed to find where language models stop knowing things.¹ PopQA has about 14,000 questions. Each one comes from a (subject, relation, object) fact sampled from Wikidata across 16 relation types, turned into a question by a hand-written template: "What is the capital of Louisiana?" or "What is Kathy Saltzman's occupation?" Every question also carries a popularity score, the monthly Wikipedia page views of its subject.¹ Louisiana gets a lot of views. Kathy Saltzman, a politician, gets few.

The models did badly on the obscure end, and size barely helped there. On the 4,000 least popular questions, GPT-NeoX 20B answered 16% correctly and GPT-3 davinci-003 answered 19%.¹ Most of the gain from scaling came from questions whose subject had more than about 10,000 monthly views; below that, accuracy stayed roughly flat as models grew.¹ Then the authors pasted one retrieved Wikipedia paragraph in front of each question. With that single paragraph, a 2.7 billion parameter GPT-Neo beat unassisted davinci-003 on those same 4,000 long-tail questions.¹

The surprise was at the other end. For popular subjects, retrieval "doesn't help much or even hurts the performance," the paper reports, because the model already knew the answer and the retrieved text could mislead it.¹ So the authors built a rule that retrieves only for questions below a popularity threshold. With davinci-003 it reached 46.5% accuracy on PopQA, which they report as 5.3% higher than any non-adaptive method, and it cut GPT-3 API costs by about half.¹

That chart is the condition this post is about. Retrieval is a patch for what a model never memorized, and for what it did memorize the patch can do harm. To see why both halves follow from how the method works, it helps to go back to the two 2020 papers that defined it.

Two places a model can keep a fact

Parametric memory

knowledge stored in the model's weights during training. You cannot read it directly or edit one fact at a time.

Non-parametric memory

knowledge stored outside the model as text, usually an index of passages, that the model reads at answer time. You can inspect it and swap it out.

Retriever

the component that scores passages against the input and returns the best few. A dense retriever compares embedding vectors; BM25 counts overlapping words.

Marginalize

sum a probability over every value of a hidden choice, weighted by how likely each choice is. In RAG the hidden choice is which passage to read.

Long tail

the many facts that each appear rarely, such as details about entities few people look up.

Both founding papers start from the same complaint. Pre-trained models store a surprising amount of world knowledge, but it sits implicitly in their parameters, so it is hard to tell what the network knows or where, and covering more facts means training ever larger networks.² The RAG paper adds that such models cannot easily revise their memory, cannot show where an answer came from, and may hallucinate.³

REALM put a retriever inside pre-training

REALM, from Google Research in February 2020, attached a learned retriever to a BERT-style masked language model.² Before predicting a masked word, the model retrieves documents from a corpus such as Wikipedia and reads them alongside the input. The paper writes the whole thing as a two-step generative process, retrieve then predict, and treats the retrieved document $z$ as a hidden variable to sum out:²

The retriever scores a document by the inner product of two BERT embeddings, one for the input and one for the document, and turns the scores into probabilities with a softmax.² The part I find most useful is the paper's gradient analysis. For each document, training nudges the retriever's score up if the document makes the correct answer more likely than the average document does, and down if it makes it less likely. In the paper's words, a document "receives a positive update whenever it performs better than expected."² Nobody labels which document is relevant. The prediction loss decides.

That cost engineering. The document embeddings go stale every time the retriever's weights change, so REALM re-embedded and re-indexed its roughly 13 million Wikipedia chunks asynchronously every few hundred training steps.² It also added a null document to the retrieved set so that masked words needing no world knowledge had somewhere to put their credit.² Fine-tuned for open-domain question answering, REALM scored 40.4 exact match on Natural Questions, above the 34.5 of T5 with 11 billion parameters, while being about 30 times smaller.² But REALM extracts its answers as spans copied from a retrieved document. It cannot write new text.

RAG made the passage a hidden variable for a text generator

Three months later Patrick Lewis and colleagues at Facebook AI Research, UCL and NYU named the idea retrieval-augmented generation and applied it to sequence-to-sequence models, which generate free text.³ Their parametric memory is BART-large, a 400 million parameter pre-trained encoder-decoder.^3,6 Their non-parametric memory is the December 2018 Wikipedia split into 100-word chunks, 21 million passages in all, embedded by the dense passage retriever DPR and searched with a FAISS index.^3,4 The retrieved passage is simply concatenated to the input before BART reads it.³

The paper gives two ways to sum out the passage. The first, RAG-Sequence, assumes one passage is responsible for the whole answer:³

Read it term by term. $x$ is the input, a question for example, and $y$ is the output of $N$ tokens. $p_\eta(z \mid x)$ is the retriever's probability for passage $z$, with $\eta$ its weights. It is proportional to $\exp(d(z)^\top q(x))$, where $d(z)$ is a BERT embedding of the passage and $q(x)$ a BERT embedding of the query.³ "top-$k$" means the sum only visits the $k$ highest scoring passages, found by maximum inner product search, instead of all 21 million; the paper used 5 or 10 during training.³ $p_\theta(y \mid x, z)$ is the generator's probability of the whole output when it reads $x$ with passage $z$ attached, and the product on the second line spells it out as one token at a time, each conditioned on the tokens before it.³

Put together, each retrieved passage produces its own complete answer, and the final probability is a weighted vote across passages, weighted by how much the retriever trusts each one. Training minimizes the negative log of this summed probability, with no signal about which passage was right.³ Unlike REALM, RAG keeps the document encoder and index frozen and fine-tunes only the query encoder and BART; the authors found index refreshing unnecessary for strong results.³ The second variant, RAG-Token, moves the sum inside the product so that each generated token can lean on a different passage:³

The difference shows up at decoding time. RAG-Token behaves like an ordinary generator and works with a standard beam search. RAG-Sequence does not break into per-token probabilities, so it runs a beam search per passage and then needs extra forward passes to score candidates that appeared in some beams but not others; the paper offers a cheaper approximation that skips those passes.³

What RAG measured, including the human raters

On open-domain question answering RAG-Sequence reached 44.5 exact match on Natural Questions, against 41.5 for the DPR pipeline, 40.4 for REALM and 34.5 for closed-book T5-11B.³ Generating answers instead of extracting them had a measurable side effect: on Natural Questions, RAG answered 11.8% of questions correctly even when the answer appeared in none of the retrieved passages, where an extractive model scores zero.³

The generation results carry more weight for this post. The authors asked RAG and a plain BART model to write Jeopardy clues for a given answer, then showed human raters 452 pairs of generations from BART and RAG-Token.³ Factuality meant the statement could be backed by trusted external sources; specificity meant the output depended closely on the input.³

Raters picked RAG as more factual in 42.7% of pairs and BART in 7.1%.³ RAG also wrote more varied text without any decoding tricks for diversity: the share of distinct trigrams in Jeopardy generations was 53.8% for RAG-Sequence against 32.4% for BART.³

The cleanest demonstration of non-parametric memory was an index swap. The authors built a second index from a December 2016 Wikipedia dump and asked the model "Who is {position}?" for 82 world leaders who had changed between 2016 and 2018. With the index matched to the year, RAG answered 70% (2016) and 68% (2018) correctly. With the mismatched index it fell to 12% and 4%.³ The model's weights never changed. Only the text it read did.

The paper also caught parametric and non-parametric memory working together. Asked to write a Jeopardy clue for "Hemingway," RAG-Token leaned on one retrieved passage while writing "The Sun Also Rises" and on another for "A Farewell to Arms," and the reliance on any passage flattened after the first word of each title.³ The authors checked: plain BART, given the partial text, completes the titles on its own. Retrieval steered the generation, and the model's weights supplied the rest.³

Why the famous facts did not need the search

Back to Mallen and colleagues. Their retrieval setup was deliberately simple: an off-the-shelf retriever pulls the single top paragraph from the December 2018 Wikipedia, and it is concatenated with the question, no training involved.¹ They tried BM25, a term-matching retriever with no training, and Contriever, a dense retriever pre-trained with contrastive learning on unlabeled text and then fine-tuned on MS MARCO.^1,5 They ran it across ten models from 1.3 billion parameters up to GPT-3.¹

Popularity predicted what the models had memorized. For almost every relation type, accuracy rose with the subject's page views, and the correlation was strongest for the largest model, roughly 0.4 for davinci-003 against about 0.1 for GPT-Neo 1.3B.¹ This agrees with a concurrent study that tied a model's accuracy on a factual question to how many pre-training documents contain the entities of the question and its answer.^1,7 The authors conclude that scaling lowers the popularity a fact needs before a model reliably memorizes it, but is not projected to push that line far into the long tail at practical model sizes.¹

To see why retrieval hurt on popular questions, they sorted every PopQA question by two outcomes for davinci-003: right or wrong without retrieval, and right or wrong with Contriever. Then they measured recall@1, whether the single retrieved paragraph contained a gold answer.¹

• Right alone, wrong with retrieval: 10% of questions, recall@1 of 0.14. The passage usually did not contain the answer, and the model followed it anyway.¹
• Wrong alone, right with retrieval: 17% of questions, recall@1 of 0.88.¹
• Right both ways: 24%, recall@1 of 0.83. Wrong both ways: 49%, recall@1 of 0.11.¹

Overall recall@1 was 0.42.¹ So the harm has a concrete mechanism. When retrieval misses, a model that knew the answer can be talked out of it by a passage that is about the wrong thing. RAG's human raters saw retrieval improve factuality on average; Mallen's table shows the same move costing answers on the questions where the model had them already.

Adaptive retrieval: when to skip it

The rule the authors propose uses only information in the question. Retrieve when the subject's popularity is below a threshold, and answer from the model's own memory otherwise. They pick a separate threshold for each relation type on a development set, choosing whatever maximizes accuracy.¹

The threshold moves with model size. Small models ended up retrieving for almost every question, because there were few questions where their own memory beat a retrieved paragraph, and the gain from the adaptive rule was much smaller below 10 billion parameters.¹ Larger models retrieved far less: davinci-003 paired with BM25 retrieved for only 40% of questions, and GPT-NeoX 20B skipped retrieval on more than 20%.¹ Skipping also saves time. Attaching the retrieved passage almost doubled GPT-J 6B's inference latency, and the adaptive rule cut inference time by up to 9% compared with always retrieving.¹ On EntityQuestions, where most subjects are obscure and the models retrieve much more, it still cut GPT-3 API costs by 15% with the same accuracy as always retrieving.¹

When the retriever learns to return the same page

Both founding papers ran into the same trap from the other side. REALM describes it at initialization: if the retriever starts with poor embeddings, the documents it returns are unrelated to the input, the encoder learns to ignore them, and the retriever then receives no meaningful gradient and cannot improve, a vicious cycle REALM avoided by warm-starting both embedding models.² The RAG paper records a version of it in its appendix. In preliminary experiments on tasks such as story generation, the retriever "collapsed": it learned to return the same documents no matter the input.³ Once that happened the generator learned to ignore the documents, and the RAG model performed the same as plain BART.³ The authors suggest the collapse could come from a less explicit need for factual knowledge in those tasks, or from longer target sequences giving the retriever less informative gradients.³