Same Retriever, Different Driver: Fixed RAG Pipelines vs Models That Search

Barakaeli Lawuo, Jul 21, 2026

In March 2025 a team from the University of Illinois, UMass Amherst and Google Cloud AI Research ran a comparison where almost everything was held still. Both systems used Qwen2.5-7B. Both searched the same 2018 Wikipedia dump with the same retriever, E5, and both got exactly 3 passages back per retrieval.¹ E5 is a text embedding model: it turns a query and each passage into a single vector, so the nearest vectors can be looked up as the most relevant passages.² The baseline was plain retrieval-augmented generation (RAG). It took the question, retrieved once, pasted the passages in front of the question, and let the model answer. The other system, Search-R1, let the model write its own search queries in the middle of its reasoning, as many as its budget allowed, after training it with reinforcement learning to do so.¹

Averaged over seven question answering datasets, the RAG baseline scored 0.304 exact match. Search-R1 trained from the base 7B model scored 0.431.¹ Exact match (EM) is the share of answers that equal the gold answer string, so this is 30.4% of questions right against 43.1%. By my arithmetic that is a 42% relative gain over RAG. The abstract reports 24%, which matches the gap to the paper's strongest baseline, a rejection sampling method at 0.348, rather than to RAG.¹ The number worth keeping is the raw pair. Same retriever, same corpus, same passage count. The one thing that changed was who decided what went into the context window.

What exactly differs between the two designs

The phrase "agentic RAG" gets used loosely, so it helps to pin both designs down by three questions: what goes in, who writes the query, and how many retrieval calls happen.

Fixed pipeline (naive RAG)

The engineer decides everything ahead of time. The user question is the query, one retrieval runs, a fixed number of passages is added to the prompt, and the model generates once. Lewis et al. introduced RAG as a model that pairs a pretrained generator with a dense vector index of Wikipedia reached through a neural retriever.³

Enhanced RAG

Still a fixed sequence, but with extra modules bolted on: a router that decides whether to retrieve at all, a rewriter that reformulates the query, and a reranker that re-sorts retrieved chunks before generation.⁵

Agentic search

The model decides, token by token, whether to search, what to search for, and when to stop. Retrieval results are appended to its running text and it keeps reasoning over them.^1,4

Reinforcement learning (RL) here

Training where the model tries whole question-to-answer episodes, including its searches, and is updated toward the episodes that earned a higher reward. In Search-R1 the reward is only whether the final answer matches the gold answer.¹

In Search-R1, the RAG baseline's input is the question plus the top 3 passages for that question: one call, with the query written by nobody but the user. Search-R1's input starts as a short template telling the model it can wrap a query in <search> and </search> tags. When the system sees a closing search tag it pauses generation, runs the query, and inserts the top 3 results between <information> tags. The loop ends when the model writes an <answer>, or when it hits the maximum action budget, which the authors set to 4.¹ So the agent makes somewhere between zero and four calls, and it writes every query itself.

There is a second difference that is easy to miss. The RAG baseline ran on the instruction-tuned model with no task training, while Search-R1 was trained with PPO on the merged NQ and HotpotQA training sets.¹ PPO (proximal policy optimization) is a standard RL algorithm; the paper also tried GRPO. The authors did train other baselines on the same data, including supervised fine-tuning and rejection sampling, to separate the effect of training from the effect of search.¹ My reading is that the fair comparison to RAG is really "trained agent" against "untrained pipeline", and the fair test of agency alone is the untrained agents, which come up below.

Search-o1, from Renmin University of China, is the untrained version of the idea. It gives a reasoning model, QwQ-32B-Preview, instructions to emit a query between special search tokens when it is unsure, retrieves the top 10 results from the Bing Web Search API, and runs a separate pass the authors call Reason-in-Documents that condenses the retrieved pages before they go back into the main reasoning chain.⁴ Its fixed baseline retrieves the top 10 documents for the original question once and puts them next to the question.⁴

The third paper frames the question for production systems. Pietro Ferrazzi and colleagues compared an Enhanced RAG pipeline against a single-tool agent built on the PocketFlow framework, where the agent's only choices are to call the RAG tool or to answer. They limited the agent to one tool on purpose, so it would have no abilities the pipeline lacked.⁵ Both used the same embedder, OpenAI's text-embedding-3-small with cosine similarity, and the same pgvector database, so retrieval time and cost were identical in both settings.⁵

Where search paid and where it did not

Search-R1 reports per-dataset scores. Three of the seven datasets (NQ, TriviaQA, PopQA) are general questions that usually need one fact. The other four (HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle) are multi-hop, meaning the answer needs two or more facts found in sequence. NQ and HotpotQA were the training sets; the other five are out of domain.¹

The gaps are largest on the multi-hop sets. On Musique, RAG got 5.8% and the base Search-R1 model 19.6%; on Bamboogle, 20.8% against 43.2%. On TriviaQA the gap is about five points, and the instruct model barely beat RAG on PopQA (39.7% against 39.2%).¹ The paper's own case study shows why multiple calls help. Asked which city and state the singer behind the fragrance Curious was born in, the model first searched for the fragrance, learned it was Britney Spears's, then searched for her birthplace and answered McComb, Mississippi. A version trained without search guessed Beyoncé and answered Houston.¹ A single retrieval on the original question cannot issue that second query, because the name it needs is not in the question.

Search-o1 reached the same split without any training. With QwQ-32B, its agentic retrieval baseline (the model issues its own queries, no Reason-in-Documents) beat single-shot RAG by 23.2% in average EM on the four multi-hop sets. On the two single-hop sets there was no significant change: 47.8 against 47.6 average EM. The authors' explanation is that those questions need one piece of knowledge and no second retrieval.⁴ The full Search-o1 system beat single-shot RAG with QwQ-32B by 29.6% on multi-hop average EM.⁴

Ferrazzi et al. looked at the retrieval steps one at a time. For query rewriting, the agent, which rewrote queries when it chose to, averaged 55.6 NDCG@10 across four datasets against 52.8 for the pipeline that always applied HyDE rewriting.⁵ NDCG@10 scores how well the ten retrieved documents are ranked against the labeled relevant ones, from 0 to 100 here. For picking the final documents, the result flipped. The pipeline with a reranker reached 49.5 NDCG@10 averaged over FiQA and CQADupStack-English, and the agent reached 43.9, below the naive pipeline's 45.5.⁵ The agent chose to retrieve a second time in only 10% of cases, and when it did, 53% of the documents it got back were the same as the first time. The authors conclude that once the model has made a decision it is not likely to reconsider it.⁵

The bill: calls, tokens, latency and training

A fixed pipeline makes one retrieval call per question, by design. The agent makes more. SearchAgent-X, an inference system paper that served Search-R1 models, reports an average of 2.717 retrieval calls per question across six datasets, ranging from 2.288 on NQ to 3.247 on Musique.⁶ Search-R1's own training curves show the count of valid searches rising as training goes on; the model learns to call search more.¹

Those extra calls cost more than their retrieval time. In a fixed pipeline all retrieval happens before generation, and retrieval takes milliseconds against seconds for the whole request, so the SearchAgent-X authors found naive RAG insensitive to retrieval latency.⁶ In an agent, retrieval interrupts generation. The paused request can lose its cached attention state (its KV cache) to other requests and must then recompute its whole prefix. When average retrieval time rose from 0.6 to 4.4 seconds, the agents' prefix cache hit rate fell from over 30% to under 21%, while RAG's end-to-end latency stayed largely stable. Under a standard first-come-first-served scheduler, 55.9% of tokens in affected requests were recomputed without need, more than doubling computation time per request.⁶

Ferrazzi et al. measured the bill directly, with the retrieval side identical. Averaged over their datasets, the agent used 3.3 times the input tokens, 1.9 times the output tokens and 1.5 times the end-to-end time of the Enhanced pipeline, and up to 3.6 times the input tokens on CQADupStack-English.⁵ The latency gap depended heavily on the model:

Training is a separate bill that only the RL approach pays. Search-R1's PPO runs took 500 steps on a single node of 8 H100 GPUs with a batch size of 512, and every rollout during training calls the search engine live.¹ The method also needed a fix that plain fine-tuning does not: the retrieved passages sit inside the model's own output sequence, so the authors masked them out of the loss. Without that mask the 7B base model averaged 0.343 EM instead of 0.431.¹ GRPO, the other RL algorithm they tried, converged faster but showed reward collapse after many steps. When training diverged, they evaluated the most recent stable checkpoint instead of the final one.¹

When the fixed pipeline still wins

The papers report several conditions where the engineer's pipeline did as well or better. I list only what they measured.

The clearest case is single-fact questions. Search-o1 found no significant gain from agentic retrieval on single-hop QA, and with QwQ-32B the fixed RAG setup scored higher on TriviaQA (65.6 EM against 62.0).⁴ In Search-R1 with Qwen2.5-3B, RAG beat the trained instruct model on NQ (0.348 against 0.341) and PopQA (0.387 against 0.378).¹

Model size matters too. Search-R1's 7B model opened a much wider gap over RAG than its 3B model did, which the authors read as larger models being better at learning to search.¹ On Bamboogle and Musique, the 3B base model's gains over RAG were under one point (0.088 against 0.080, and 0.049 against 0.047).¹ Search-o1 found that a model not trained for long reasoning, Qwen2.5-32B-Instruct, did about as well with agentic retrieval as with standard RAG on the GPQA science questions, and worse on math and code. The authors conclude that ordinary LLMs cannot effectively use search as a tool for complex reasoning tasks.⁴

Deciding whether to retrieve at all went to the pipeline when the domain was broad. Ferrazzi et al. tested whether each system could tell in-scope questions from out-of-scope ones. On the finance and grammar-forum datasets the agent was slightly better. On FEVER, a fact-checking set with no clear domain boundary, the pipeline's semantic router scored 87.9 F1 and the agent 64.6, because the agent often retrieved when it should not have.⁵

For choosing the final documents, as noted above, a dedicated reranker in the pipeline beat the agent's own retrieval choices, 49.5 against 43.9 NDCG@10.⁵

Then there is the budget. Every measurement above puts the agent at more tokens and more time for the same retriever.^5,6 Ferrazzi et al. write that a well-optimized Enhanced RAG can match or exceed Agentic performance while remaining more efficient, and they suggest adding an explicit reranking step to agentic pipelines.⁵

What the comparisons did not measure

Search-R1's appendix includes a failure it labels as such. Asked for the title of Weezer's debut album, the model sometimes failed to break the question into parts and was misled by irrelevant retrieved passages.¹ Ferrazzi et al. state the limit of their own cost numbers in a table note: their agent always performed a maximum of 3 turns, and in scenarios requiring more turns, the tokens it consumed would increase. They also note that giving the agent a single tool restricts what their study can say about agents that do more than retrieval.⁵