15 Production RAG: Metrics, agentic systems, and continuous improvement
This chapter shifts from RAG as a collection of techniques to RAG as a production system that must be measured, tuned, and improved over time. It highlights the idea that the generator, not just the retriever, often determines whether retrieval helps or hurts, especially at scale. Using RAGGED as a key reference point, it shows that some models are robust to extra context while others become more fragile as more passages are added, so production teams need metrics that capture stability and scalability rather than assuming more retrieval is always better.
The chapter then describes agentic RAG as the next step in the field: systems that decide at runtime whether to retrieve, reformulate, plan, or stop instead of following a fixed pipeline. Reflection, planning, tool use, and multi-agent collaboration are presented as recurring design patterns, along with practical pipeline structures such as routing and orchestrator-worker setups. It also shows how fine-tuning can be folded into the RAG lifecycle, especially by unifying ranking and generation in a single model and by adapting either the retriever or the reader to reduce noise and improve domain performance.
Finally, the chapter emphasizes continuous improvement in production through logging, evaluation, versioning, and targeted remediation based on failure modes. It frames RAG tuning as a diagnostic process: if content is missing, expand retrieval; if the right passage is present but poorly ranked, rerank; if the generator ignores useful context or hallucinates, improve reader robustness or use self-reflection; if the domain is specialized, fine-tune with synthetic or labeled examples. The chapter closes by stressing that real-world RAG systems should be routed by query complexity, monitored like living software, and extended with image-based retrieval or page-level answering when text extraction is unreliable.
Two distinct reader scaling behaviors from RAGGED.
The orchestrator-workers pattern for agentic RAG.
Three stages of self-improvement in RAG systems.
Diagnostic decision tree for RAG technique selection.
Summary
- RAGGED showed that some reader models degrade with more retrieved passages (peak-then-decline) while others improve and plateau. The RAG Stability Score (RSS) and RAG Scalability Coefficient (RSC) capture these dimensions separately, enabling targeted optimization of retrieval depth per model.
- Agentic RAG generalizes adaptive retrieval into autonomous pipelines. Four design patterns (reflection, planning, tool use, and multi-agent collaboration) compose into workflow architectures that decide at runtime which retrieval techniques to apply. These systems improve quality on complex queries but add latency in proportion to the number of extra model calls, which is what query-based routing is for.
- Fine-tuning and RAG are complementary, not competing. RankRAG demonstrated that instruction-tuning a single LLM for both context ranking and answer generation outperforms separate specialist models, with a small fraction of ranking data producing outsized improvements. Domain-specific adapter training further improves noise robustness and retrieval precision.
- Self-improving RAG systems learn progressively deeper forms of trust in retrieval. Adaptive kNN-MT learns when to trust retrieval through a lightweight gate network. SKILL-RAG learns which retrieved sentences to trust by modeling the LLM's self-knowledge boundaries. SimRAG teaches the model to internalize domain-specific retrieval skills through self-generated training data with round-trip consistency filtering.
- When the parser fails, send the page. Figure text in PDFs routinely decodes to garbage, and no retriever recovers what was lost before retrieval ran. Rendering the page region at 200 dpi and passing it as an image block lets the model read what the parser destroyed, with no change to the index. A full page-image retrieval index (ColPali and its descendants) wins in a single stage and loses to text-plus-reranker once a reranker is in play, at 2,312 times the storage. Concatenating the top results from a text and a visual retriever beats either alone.
- Technique selection should be driven by failure point diagnosis. Map your system's measured failures (FP1-FP7) to specific techniques from this book, starting with low-complexity interventions (hybrid search, reranking) and adding high-complexity ones (Self-RAG, agentic orchestration) only when simpler approaches fall short.
- Production RAG requires pipeline versioning, continuous evaluation, and a three-level testing pyramid. Log full retrieval contexts, track pipeline configurations as versioned artifacts, and run component, integration, and regression tests at appropriate frequencies. Framework choice (LangChain, LlamaIndex, Haystack, DSPy, or direct implementation) should match your abstraction preferences and pipeline complexity.
- Domain-specific agentic RAG demonstrates measurable gains with latency costs. The fintech case study showed retrieval accuracy rising from 54.12% to 62.35% at Hit@5, about 8 points, with average latency rising from 0.79 to 5.02 seconds for the same queries. That is a 6.4x slowdown bought with 8 points, on 85 questions, which is the trade you are actually being offered. The authors name sub-query generation as the largest contributor from qualitative inspection rather than an ablation.
Retrieval Augmented Generation, The Foundational Ideas ebook for free
