The chapter explains why open-domain question answering is hard for standard retrieval pipelines and why the field moved from sparse keyword search to dense, learned representations. Early systems could find relevant passages, but the retriever was usually frozen and optimized for similarity rather than usefulness, so it often surfaced documents that echoed the question instead of answering it. Models like SBERT improved semantic matching for symmetric tasks, while DPR addressed the asymmetric nature of question answering by training separate encoders for questions and passages. Even so, the retriever and reader still operated as separate parts, which left a gap between finding text that looked relevant and finding text that actually helped produce the correct answer.
REALM closes that gap by training retrieval and reading together during pretraining. Its core idea is to treat retrieval as a latent choice and optimize the language model’s masked-token objective over retrieved documents, so the retriever gets rewarded only when the fetched passage improves prediction. This makes retrieval performance-based rather than similarity-based: documents are pulled closer in vector space because they are useful for the task, not because they merely resemble the query. The model uses a dual-encoder retriever for efficient search and a cross-encoder reader for contextual reasoning, and it relies on approximate marginalization over top-k documents plus asynchronous index refreshes to make the system computationally feasible at scale.
The chapter also shows why this mattered in practice. REALM outperformed prior systems on open-domain QA benchmarks and even surpassed much larger parametric models, demonstrating that external knowledge plus end-to-end training could beat brute-force scaling. The examples make the benefit tangible: rare or specific facts that a standalone language model cannot reliably recall become predictable once retrieval supplies supporting evidence. The chapter closes by emphasizing REALM’s legacy as the conceptual origin of modern RAG: knowledge can live outside the model, retrieval can be interpretable and updatable, and the real bottleneck is often not architecture alone but the quality of the training signal that teaches the retriever what is truly useful.
REALM's End-to-End Architecture. This diagram shows REALM's two-part system, which trains a retriever and a reader jointly.
The Two-Stage Training Process of REALM. This diagram illustrates the two distinct phases of training a REALM model.
Prediction with and without retrieval, on REALM's own qualitative example.
Summary
REALM’s novelty was its fully unsupervised pretraining method, which used the Masked Language Modeling (MLM) objective as a performance-based signal to train the retriever. Instead of defining vector similarity solely by semantic closeness, REALM reinterprets it in terms of usefulness for prediction. Documents are considered similar if they help the model fill in masked tokens correctly.
This is an example of self-supervised learning: the model generates its own training targets by masking tokens in the input. In Hugging Face’s MLM tasks, these targets appear as the labels in the dataset, the y that the model tries to predict. In REALM, these predictions also guide the retriever, so the model learns to fetch documents that are useful for predicting the masked tokens, all without external supervision.
This new paradigm was validated by achieving state-of-the-art results on popular open-domain question answering benchmarks. REALM beat previous methods and much larger parametric-only models such as T5-11B, despite being more than 30 times smaller.
REALM established several key principles foundational to RAG: knowledge can be explicit, modular, and interpretable, and the system’s downstream performance provides a powerful, implicit training signal for the retriever.
REALM embedded this training as a computationally expensive pretraining step, where the whole system was trained from scratch. This required clever engineering solutions, such as asynchronous MIPS refreshes, to be tractable, but it remained impractical for most applications.
FAQ
What problem in open-domain question answering does REALM try to solve?REALM addresses the challenge of answering questions across any topic using a large corpus. Traditional systems could retrieve documents, but they left the hard work of finding the answer to the user or used rigid pipelines. REALM aims to make retrieval and answering work together so the system can find useful evidence and use it directly.How is open-domain QA different from open-book QA?Open-domain QA means the system must answer questions about many topics, not a narrow domain. Open-book QA means the system is allowed to consult an external knowledge source, like Wikipedia or a database, instead of relying only on memorized parameters. REALM combines both ideas by working over an external corpus while handling broad questions.Why were traditional retrievers not enough for ODQA?Classic retrievers such as TF-IDF or BM25 mainly use keyword matching, so they can miss relevant passages when the question and answer use different wording. They also do not know whether a retrieved document actually helps answer the question. This causes failures like retrieving near-duplicates of the question instead of the passage with the answer.What is the difference between symmetric and asymmetric search?Symmetric search compares texts of similar type and length, such as paraphrases or duplicate sentences. Asymmetric search matches a short question to a long answer-bearing passage. ODQA is asymmetric, so a model trained for “sentence twins” like SBERT may struggle unless it is adapted for question-to-passage retrieval.What was DPR, and how did it improve retrieval?DPR, or Dense Passage Retrieval, used two separate encoders: one for questions and one for passages. It was trained on question, correct passage, and incorrect passage triplets so it could learn an embedding space tailored to QA. This made retrieval more suitable for ODQA than general semantic similarity models.What is REALM’s main innovation?REALM’s key innovation was joint pre-training of retrieval and language understanding. Instead of training the retriever separately, REALM connected a retriever to a language model and used the model’s own prediction performance as the training signal. This let the retriever learn which documents were actually useful, not just similar in wording.How does REALM train the retriever without document labels?REALM uses masked language modeling and treats the retrieved document as a latent variable. If a document helps the model predict the masked word better, it receives positive training signal through the loss. In this way, the masked word itself provides the supervision, and the retriever is updated based on utility rather than explicit labels.What do the retriever and reader do in REALM’s architecture?The retriever is a dual-encoder that embeds the query and documents separately and scores them with a dot product. The reader is a cross-encoder that takes the query and retrieved document together and predicts the masked token. The whole system is trained end-to-end so both parts improve together.Why is marginal log-likelihood important in REALM?Marginal log-likelihood lets REALM sum over possible retrieved documents and optimize the probability of the correct masked word overall. Since the correct document is unknown, the model considers multiple candidates and rewards the ones that improve prediction. This is how retrieval becomes tied to the language model’s objective.What practical lesson did REALM and REALM++ leave for modern RAG systems?REALM showed that retrieval can be trained for instrumental usefulness, not just similarity. REALM++ later showed that training process improvements, like larger batches, better supervision, and stronger inference, can yield big gains without changing the architecture. The broader lesson is that good training often matters as much as model design in RAG systems.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Retrieval Augmented Generation, The Foundational Ideas ebook for free
