Fine-tuning or retrieval for new knowledge: what controlled tests found

Barakaeli Lawuo, Feb 18, 2026

In a paper first posted in December 2023, Oded Ovadia and three colleagues at Microsoft in Israel tested 910 multiple-choice questions about events in the United States from August to November 2023, built from Wikipedia's pages on that year's events.¹ The dates were chosen so the events fell after the training cutoff of the models they tested: Llama2-7B, Mistral-7B and Orca2-7B. Each question had four options, so guessing gives 25%. Base Mistral-7B scored 48.1%. With retrieval, where the relevant Wikipedia text was pasted into the prompt, it scored 87.5%. After fine-tuning on that same text, it scored 50.4%. Llama2-7B did worse after fine-tuning: it fell from 35.3% to 21.9%, below chance.¹

That study is a direct head-to-head on one question: which method gets new facts into a model. Two later papers explain why it came out that way and where the result changes. Gekhman and colleagues at Google Research and the Technion measured what happens inside fine-tuning when a training example states a fact the model does not know.² Soudani, Kanoulas and Hasibi, at Radboud University and the University of Amsterdam, repeated the comparison on twelve models and split the results by how famous the fact is.³ This post takes them in that order.

Knowledge injection

Giving a pre-trained model facts it did not have, or had only weakly, so it can answer questions about them. Ovadia et al. frame it as finding a change to the model that raises its accuracy on a set of factual questions.¹

Fine-tuning

Continuing to train the model's weights on new data. In Ovadia et al. this was unsupervised: plain next-token prediction on raw text, also called continued pre-training. In the other two papers it was supervised, on question and answer pairs.^1,2,3

Retrieval-augmented generation (RAG)

Searching a document store for passages related to the question and adding them to the prompt. Lewis et al. describe this as pairing the model's parametric memory, its weights, with a non-parametric memory, a searchable index of text.⁵

Hallucination

A confident answer that is factually wrong. In Gekhman et al. it is measured as lost accuracy on held-out questions the model has to answer from what it already knows.²

The Microsoft comparison, number by number

Ovadia's team ran two kinds of test. The first used five subjects from the MMLU benchmark: anatomy, astronomy, college biology, college chemistry and prehistory. They picked subjects where the questions are short and lean on facts more than reasoning. For each subject they scraped the relevant Wikipedia articles, cleaned them into chunks, and used that text both as the fine-tuning corpus and as the retrieval store.¹ The second was the current events set from the opening, which the models could not have seen during pre-training.¹

Retrieval used bge-large-en embeddings in a FAISS index, and they tried adding from zero to five chunks to the prompt. Fine-tuning ran on 256-token chunks for up to five epochs on four A100 GPUs, with learning rates between 1e-6 and 5e-5 found by search. Scoring did not depend on generated text. The model read each question with each answer option attached, and whichever option got the highest log probability counted as its choice.¹

On MMLU, the paper reports that RAG beat the base model in every case, and that RAG on the base model was always better than fine-tuning alone.¹ The gaps were moderate; the paper notes that for these subjects the models had already seen related material in pre-training. Mistral-7B on zero-shot anatomy is typical: 55.6% base, 57.0% fine-tuned, 68.1% with retrieval. Fine-tuning plus retrieval sometimes helped a little more and sometimes did not, which the authors take as a sign that fine-tuning is unstable.¹

The current events results are much starker. Retrieval nearly doubled Mistral and Orca2. Fine-tuning on the raw articles moved them by a few points and pushed Llama2 below chance. Retrieval also got an advantage from how the test was built: every question came from a specific chunk of the corpus, so the matching chunk was always there to be found.¹ And combining fine-tuning with retrieval did worse than retrieval alone for all three models.¹

One change did help fine-tuning. The team had GPT-4 write ten paraphrases of each chunk and trained on those too. Mistral's fine-tuned score rose from 50.4% to 58.8% and Llama2's from 21.9% to 39.2%. Across all three models, accuracy rose steadily as they added more paraphrases.¹ From this they propose that a model has to see a fact stated in many different ways to learn it through fine-tuning, which is already known to hold for pre-training. Even so, paraphrased fine-tuning stayed about 30 points behind retrieval on Mistral and Orca2.¹ As for why retrieval won, the authors offer possible reasons rather than tested ones: retrieval gives the model text that matches the exact question, and fine-tuning may erase some of what the model already knew, which is called catastrophic forgetting.¹

What an unknown fact does to the model during fine-tuning

Zorik Gekhman and colleagues tested a common conjecture: fine-tuning a model on facts it does not know teaches it to state things it has no grounds for.² To test it, they needed to know, for every training example, whether the model already knew the answer. They built a method called SliCK for this. For each question, they prompted PaLM 2-S with ten different sets of four examples, and for each prompt took one greedy answer plus 16 answers sampled at temperature 0.5. How often the model got the answer right sorted each fact into one of four categories.²

• HighlyKnown: greedy decoding always gives the right answer.
• MaybeKnown: greedy decoding sometimes gives it.
• WeaklyKnown: greedy decoding never gives it, but sampling sometimes does.
• Unknown: the model never produces the right answer, greedy or sampled.

The data came from EntityQuestions, which turns Wikidata facts into questions such as "Where is Benedict located?" They used 12 relations for training and testing and held back 7 more as an out-of-distribution test.² Of the questions, 24% came out HighlyKnown, 23% MaybeKnown, 17% WeaklyKnown and 36% Unknown. To sort the whole dataset they ran 170 inference steps per example, more than 15 million in total.² Then they kept the fine-tuning set at a fixed size and varied only the share of Unknown examples. Test questions were disjoint from training questions, so the model had to answer them from knowledge it already had. A drop in test accuracy therefore means the model got worse at using facts it knew, which is how the paper measures hallucination.²

The first finding is about speed. Unknown examples were fitted much more slowly than Known ones. At the point where development accuracy peaked, the model had fitted most of the Known examples and only a small fraction of the Unknown ones. The paper reads this as a sign that fine-tuning mostly teaches a model to use what it learned in pre-training, and adds little that is new.² That point came after 5 to 10 epochs. The authors also trained each run to 50 epochs, by which time the model fitted 100% of its training set.²

The second finding is about harm, and it appears late in training. With early stopping, removing the Unknown examples from the training set barely changed the results, so at that point they were neutral. Trained to convergence, they did real damage, and the damage grew with the share of Unknown examples.² The authors fitted a linear model of test accuracy against how many examples of each kind the model had fitted by that point:

Each Unknown example the model learned cost about as much accuracy as each Known example gained. On the held-out relations the coefficients were smaller, +3.2 and -3.0, but the fit was tighter, with an R squared of 0.95.² In the paper's words, training on questions like "Where is [E1] located?" can encourage hallucinations on seemingly unrelated questions such as "Who founded [E2]?". The authors take this as evidence that the model is learning a behavior, giving answers not grounded in what it knows, rather than just picking up bad facts.²

The per-category results in the paper's Table 2 show where the damage lands. A model fine-tuned only on Unknown examples and trained to convergence fell from 95.6% to 55.8% on HighlyKnown test questions, facts it had answered reliably before.² The obvious training set was not the best one. With early stopping, training only on HighlyKnown examples reached 40.5% overall, while training only on MaybeKnown examples reached 43.6%, because the model then handled its partly known facts better.² The paper tested two fixes. Early stopping avoided most of the harm. So did replacing the answers on Unknown training examples with "I don't know": on a set that was half Unknown, accuracy on the questions the model chose to answer stayed at 61.8% from early stopping to convergence, while it answered fewer of them (58.7% down to 55.6%). The unmodified set dropped from 43.0% to 38.8%.²

How popular the fact is changes the gap

Soudani's team asked the question a company would ask: what if the facts are about obscure things, such as internal products or niche entities with little written about them?³ Their tests used three question sets about long-tail entities, PopQA and WiTQA with about 14,000 questions each and EntityQuestions with 17,300, and measured how popular each entity was by its Wikipedia page views, split into five buckets.³ They tested twelve models from 80 million to 11.3 billion parameters. Fine-tuning data was question and answer pairs generated from the entities' Wikipedia summary sections. Retrieval had to find those same summaries, and the "ideal" retriever was one that always returned the right summary first.³

This setup builds on Mallen et al., the paper that introduced PopQA. They found that models struggle with less popular facts and that making models bigger mostly helps with popular ones. On the 4,000 least popular questions, GPT-neo 2.7B with a dense retriever beat GPT-3 davinci-003 without one. For popular entities, retrieval sometimes hurt large models, because the retrieved text could mislead them.⁴

The paper does not print bucket-level numbers, so the ones here are read off the figure. Without retrieval, StableLM2 got about 23% right in the least popular bucket and about 10% in the next one up, rising to about 65% for the most popular entities. Fine-tuning moved those to roughly 30%, 14% and 75%. With DPR retrieval, the base model was near 67% in the least popular bucket. The authors themselves report that retrieval raised accuracy most for the least popular entities, and that fine-tuning helped at every popularity level, most in the least popular buckets for Mistral and Llama3.³

The most popular bucket behaves differently. Retrievers found the right document more often for obscure entities than for famous ones, probably because fewer distracting documents mention an obscure entity. On PopQA, accuracy with retrieval fell from the least popular bucket to the fourth, then rose again in the top one. The authors explain the rise by the model ignoring noisy context and relying on what it already knows about famous entities.³ In my reading of Figure 1, fine-tuning without retrieval sits at or slightly above the retrieval lines in that top bucket, the only place in the figure where that happens.

Overall scores, with the ideal retriever, show how model size matters. On PopQA, StableLM2 (1.6B) went from 17.01% to 21.75% with fine-tuning, 76.14% with retrieval, and 82.09% with both. Mistral 7B went from 21.47% to 30.70% with fine-tuning and 80.25% with retrieval, but fell to 78.44% with both. Llama3-chat 8B barely moved with fine-tuning (32.52% to 32.75%), and adding it to retrieval changed little (81.29% to 81.54%).³ The paper concludes that combining the two gave the best results for models up to 3B parameters but hurt models from 7B to 11.3B, and that fine-tuning seems to weaken the larger models' ability to reason over retrieved text.³

The way you fine-tune also matters, not just whether you do. QLoRA, which trains a small set of added weights, gained less than full fine-tuning when the model answered without retrieval, but it kept the model able to use retrieved text.³ Full fine-tuning of StableLM2 on the question pairs from a fine-tuned T5-large generator dropped its PopQA score with retrieval to 5.87%, against 76.14% for the untouched model with retrieval. That generator produced over twelve times as many pairs as prompting Zephyr did, yet the Zephyr pairs trained better models.³ The authors also built Stimulus RAG, which puts the most relevant retrieved sentence at the top of the prompt. With no fine-tuning, it beat every fine-tuned model that used the same top three documents.³

My reading of when each one won

What follows is my interpretation, limited to conditions these papers actually tested: multiple-choice or short-answer factual questions, mostly from Wikipedia, on models from 80 million to about 11 billion parameters.

• Facts from after the training cutoff: retrieval beat fine-tuning by about 37 points on Mistral and Orca2. Fine-tuning on raw text gave a few points at best and made Llama2 worse; paraphrasing narrowed the gap but did not close it.¹
• Obscure facts the model has barely seen: this is where retrieval helped most. Fine-tuning helped too, and for models of 3B parameters or fewer, QLoRA fine-tuning on top of retrieval scored higher than retrieval alone.³
• Facts the model half knows: this is where fine-tuning did its job. Training on MaybeKnown examples gave the best test accuracy, by getting the model to use what it already had.²
• Training sets full of facts the model does not know: fitting them raised hallucination on facts it did know, including on unrelated relations. Early stopping or relabeling them as "I don't know" contained most of the damage in that setup.²
• Very famous facts: in the most popular bucket, StableLM2 answering from its fine-tuned weights did about as well as with retrieval, and Mallen et al. found retrieval could mislead large models on popular entities. Separately, adding fine-tuning to ideal retrieval did not help the 7B to 11.3B models.^3,4

None of these papers tested tone, output format or long-form writing, so they say nothing about fine-tuning for behavior. Their conclusions apply to facts only.

Where the evidence stops

Gekhman et al. ran every experiment on one model, PaLM 2-S, and say it is unclear whether the results hold for others. Their fixes, such as filtering out Unknown examples, have not been checked on long-form generation.² Soudani et al. assumed each answer sits in the Wikipedia summary paragraph, and say that assumption is not entirely accurate.³ All three drew their facts from Wikipedia or Wikidata, so in my reading the limit Ovadia et al. state about their own study applies to the whole set: "all of our sources came from Wikipedia. Other datasets may yield different results, and must be evaluated carefully."¹