RAG-Fusion is presented as a practical answer to a common weakness in standard RAG: a single query often captures only part of a user’s intent. When that happens, relevant documents can be missed, top-ranked results can be incomplete, and the final context can leave the LLM without enough evidence to respond well. Instead of simply retrieving more documents from one query, RAG-Fusion expands the search laterally by generating several query variants, each aimed at a different facet of the same request.
The method has two core steps. First, an LLM produces diverse query perspectives rather than simple paraphrases, using its prior knowledge to decompose ambiguous questions into sub-intents. Second, the system retrieves results for each query in parallel and combines the ranked lists with Reciprocal Rank Fusion, which ignores raw scores and rewards documents that appear consistently across multiple lists. This makes the process robust when scores from different queries are not directly comparable and helps surface a broader, more useful set of documents before answer generation.
The chapter also shows how to implement the idea in real systems and how to tune it. It walks through LangChain and LlamaIndex examples, a Python RRF implementation, and a full toy pipeline, then compares the approach with standard RAG in an e-commerce scenario where broader query coverage improves completeness. At the same time, it stresses the trade-offs: more queries increase cost, latency, and noise, so the number of variants and the RRF constant must be chosen carefully. RAG-Fusion is most valuable when coverage matters more than speed, and it works especially well as part of a modular pipeline that can later be paired with reranking.
The coverage problem in single-query retrieval.
How Reciprocal Rank Fusion calculates scores.
RAG-Fusion complete data flow.
Summary
One query vector is one guess at what the user meant: A single phrasing captures an average of a complex intent and misses documents that answer a part of it.
Generating several query variants widens coverage: An LLM rewrites the question from different angles, and each variant reaches documents the others miss.
The variants have to shift perspective, not just wording: Rephrasings retrieve the same documents, so the generator must target distinct sub-intents.
Reciprocal Rank Fusion merges the ranked lists without trusting their scores: It rewards documents that several retrievers rank highly and ignores the uncalibrated numbers, which is why it works across retrievers that score on different scales.
RRF is older than RAG: It comes from a 2009 metasearch paper and needs no tuning to be useful.
More variants is not better: Each one adds latency, cost, and a chance of dragging in noise. Three to five queries and an RRF constant of 60 is the place to start.
The wins are on discovery questions: Knowledge assistants, support bots, and product search, where the user cannot phrase the query the way the corpus does.
The LLM has moved into the retriever: Query expansion here, hypothetical documents in chapter 7, self-critique in chapter 9. The same model now orchestrates retrieval as well as writing the answer.
FAQ
What problem does RAG-Fusion solve in standard RAG systems?It solves the coverage problem: a single query often captures only part of a user’s intent, so standard RAG can miss relevant documents, return incomplete context, or rank the wrong documents too low to retrieve.Why is a single retrieval query often insufficient?A single query is usually only an incomplete snapshot of intent. Different phrasings can point to different semantic clusters, so one query may miss documents that would be found by another wording.How does RAG-Fusion improve retrieval coverage?It broadens retrieval by generating multiple perspective-shifted query variants, running them in parallel, and then merging the results so the final context covers more facets of the user’s intent.What is the difference between query rephrasing and query expansion in RAG-Fusion?Rephrasing creates near-duplicates of the same query, while query expansion in RAG-Fusion generates distinct sub-intents or facets of the original question to retrieve a more diverse set of documents.What role does the LLM play in RAG-Fusion?The LLM acts as a query generator. It decomposes the user’s question into multiple alternative search queries that explore different angles before retrieval happens.Why does RAG-Fusion use Reciprocal Rank Fusion instead of averaging retrieval scores?Because retrieval scores from different queries are not calibrated or directly comparable. RRF avoids this problem by ignoring raw scores and combining results using only document rank.How does Reciprocal Rank Fusion decide which documents should rank highest?It gives each document a score based on its rank in each query result list. Documents that appear consistently across multiple lists tend to score higher than documents that are rank 1 in only one list.What is the typical value of the RRF k constant, and what does it control?The k constant is commonly set around 60. It dampens the influence of top ranks and controls how strongly the fusion favors consistency across lists versus one very high rank in a single list.What are the main trade-offs of using RAG-Fusion?RAG-Fusion improves recall and completeness, but it increases cost, latency, and the risk of noise from overly broad or spurious query variants.When should RAG-Fusion be used, and when should it be avoided?It is best for complex or ambiguous queries where coverage matters, such as enterprise search, technical support, and research. It should be avoided for narrow lookups, latency-sensitive applications, expert users with precise queries, or cases where the corpus is already small and well-covered by a single query.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Retrieval Augmented Generation, The Foundational Ideas ebook for free
