5 Fusion-in-Decoder for multi-document processing
Fusion-in-Decoder is presented as a practical answer to a central RAG problem: retrieved information often exists, but it is scattered across many passages and cannot be reliably combined by extractive readers or by prompt stuffing alone. The chapter contrasts earlier retrieval-extractive systems and closed-book generators with FiD’s hybrid design, showing why multi-document questions need a model that can both preserve evidence and synthesize it. FiD’s main lesson is that the retriever should prioritize recall, while the reader should take responsibility for precision and reasoning.
The architecture works by encoding each retrieved passage independently alongside the question, which keeps computation linear and avoids the quadratic costs of letting every passage interact with every other passage during encoding. Only after each passage has been represented separately are the results fused into one long memory for the decoder. The decoder then uses cross-attention to scan all encoded passages at once, selecting useful details from different sources and ignoring distractors, which makes it possible to answer multi-hop questions that no single passage can fully support.
The chapter also shows how FiD was implemented with encoder-decoder models such as T5 and why fine-tuning matters: the model must learn to combine evidence rather than simply repeat the strongest passage. Experiments demonstrated that FiD improved as more passages were added, especially on demanding QA benchmarks, and that it could outperform extractive readers when answers were distributed across documents. At the same time, the chapter notes that later successors improved FiD by reducing memory bottlenecks and by filtering or reweighting retrieved evidence, but the lasting contribution of FiD was to make separate encoding and late fusion the core pattern for scalable multi-document reasoning.
DrQA Retrieval Extractive Pipeline.
The FiD architecture: Independent Encoding, Fused Decoding.
RAG Pipeline: Conventional vs. FiD.
Encoder-Decoder Architecture.
Summary
- FiD solves the multi-document bottleneck. By encoding passages independently and in parallel before fusing them in the decoder, FiD avoids the quadratic complexity that prevented earlier models from processing many documents at once.
- The architecture enables true synthesis. The decoder attends to all retrieved evidence simultaneously at each step, allowing it to generate novel, coherent answers by aggregating information scattered across multiple sources.
- FiD shifts the retriever's goal from precision to recall. Because the decoder filters noise so well, the retriever's job becomes getting the right information somewhere into the top-k documents, not necessarily at the top rank.
- FiD's main trade-off is inference latency. The decoder is the slow half, since it attends to the whole concatenated sequence of passage representations.
- A lesson in system design. Efficient AI systems must be co-designed with hardware in mind, addressing memory bandwidth and not only theoretical computational cost.
FAQ
What is Fusion-in-Decoder (FiD) in RAG?
FiD is an encoder-decoder architecture for retrieval-augmented generation that encodes each retrieved passage independently and then fuses all passage representations only during decoding. This lets the model synthesize information from many documents into one answer.
Why was FiD needed when standard RAG systems already retrieved documents?
Standard RAG systems often processed retrieved passages too separately or relied too much on the top-ranked documents. FiD was introduced to better combine evidence scattered across many passages, especially when no single document contains the full answer.
What does “independent encoding, fused decoding” mean?
It means each retrieved passage is encoded on its own, without interacting with other passages. Then, during generation, the decoder attends to all encoded passages together and merges their evidence into a coherent response.
How does FiD improve scalability for many retrieved documents?
FiD keeps encoding cost linear with the number of documents because passages are processed independently in parallel. This avoids the quadratic cost of self-attention over one massive concatenated context.
Why does FiD help with multi-document questions?
FiD allows the decoder to combine facts from different passages, which is useful for questions that require synthesis or multi-hop reasoning. It can pull one fact from one document and another supporting fact from a different document to form a complete answer.
How is FiD different from extractive readers like DrQA?
DrQA-style systems retrieve passages and extract a literal answer span from each one. FiD instead uses a generative decoder, so it can produce an abstractive answer that synthesizes information across multiple passages.
Why can FiD work better with recall-oriented retrieval?
Because the decoder can ignore irrelevant passages, the retriever does not need to rank the single best passage first. It is enough that the relevant evidence appears somewhere in the top-k results, which shifts retrieval from precision to recall.
How is FiD implemented with a Transformer model like T5?
In practice, the question is paired with each retrieved passage, the encoder processes all pairs independently, and the resulting hidden states are concatenated. The decoder then generates the answer using cross-attention over this fused encoder output.
What are FiD’s main limitations in production?
Classic FiD can be expensive at inference time because the decoder must repeatedly attend over a large fused context. It also depends on retrieval quality, so if the needed document never appears in the retrieved set, FiD cannot recover the answer.
What replaced FiD in production settings?
FiD was largely superseded by long-context decoder-only models and optimized successors such as FiDO. These approaches are often easier to deploy and faster in practice, while still aiming to preserve FiD’s ability to use multiple documents for grounding.
Retrieval Augmented Generation, The Foundational Ideas ebook for free
