6 Retrieval Augmented Generation (RAG)

Retrieval Augmented Generation (RAG) combines semantic retrieval with large language model generation to ground answers in factual, up-to-date sources and curb hallucinations. This chapter shows how to turn plain search into conversational, personalized assistance by uniting encoders (for retrieval) and decoders (for fluent responses). It walks through two end-to-end applications: upgrading the Travelle hotel search into a recommendation assistant and building a research agent that synthesizes insights from academic papers with clear source attribution. It also compares industrial and open-source LLMs and demonstrates using a unified gateway to experiment with multiple models and balance cost, quality, and reliability.

Under the hood, the RAG pipeline spans query processing, embeddings, a vector-backed knowledge base, retrieval, prompt construction, and guided generation. The chapter emphasizes why long documents must be chunked: fixed context windows and information dilution make single-shot embeddings ineffective. It introduces practical chunking strategies—fixed-size, sentence/paragraph, semantic, and especially recursive splitting with overlaps—to preserve meaning while fitting model limits. For production-scale retrieval, it replaces FAISS with a vector database, highlighting Qdrant for efficient similarity search, robust metadata filtering (for example, city-scoped hotel queries), hybrid search, and scalable operations. With structured prompts and streaming, raw ranked results become coherent narratives tailored to the user’s intent.

Because RAG quality depends on both what is retrieved and what is generated, the chapter establishes an evaluation toolkit. Retrieval is assessed with precision@K, recall, MRR, NDCG, and context relevance, while generation is judged on faithfulness to sources, answer relevancy, context adherence, and completeness. The RAGAS framework provides a reference-free, LLM-assisted suite of core metrics and a composite score, complemented by techniques like LLM-as-judge, synthetic test sets, and production-aligned testing. By the end, readers have built two complete RAG systems, understand the necessity and methods of chunking, know when and how to adopt vector databases, can select appropriate LLMs, and have a clear path to measuring and improving reliability as they move toward enterprise- and agentic-grade RAG.

Architecture of a Retrieval Augmented Generation (RAG) system showcasing the integration of retrieval and generative components for enhanced natural language processing tasks.

OpenRouter.ai landing page, allowing access to potentially any LLM Endpoint

Computational Cost of Context Window Expansion

Comparison of LLM Text Chunking Strategies

Metrics for Evaluating Retrieval Engine

Metrics for Evaluating LLM Generation

RAGAS Core Metric extensively covers LLM Grounding and Response (Source)

• Retrieval Augmented Generation combines semantic search capabilities with language model generation to eliminate hallucinations and provide accurate, contextually relevant responses grounded in factual information, representing a fundamental shift from standalone LLMs to knowledge-grounded AI systems.
• Context window limitations in embedding models (8,192 tokens for Nomic Embed) require recursive chunking techniques that break down large documents into semantically coherent segments while preserving context through overlap, enabling effective processing of research papers and comprehensive hotel descriptions without losing critical information.
• OpenRouter's unified API enables seamless integration of multiple language models with retrieval results, producing coherent, well-cited responses that synthesize information from multiple sources while maintaining factual accuracy and providing flexibility in model selection for different use cases.
• Production-ready vector databases like Qdrant provide essential capabilities over experimental FAISS setups, including persistent storage, concurrent access, and optimized retrieval algorithms necessary for enterprise-scale RAG applications that require reliability and performance at scale.
• RAG's versatility spans diverse domains from personalized travel recommendations (Travelle hotel search) to academic literature analysis (research paper agents), demonstrating consistent accuracy and source attribution capabilities across different content types and user requirements.
• Dense vector representations through models like Nomic Embedding enable sophisticated semantic understanding, while metadata filtering capabilities allow targeted searches (city-specific hotels, year-filtered papers) that significantly improve retrieval precision and computational efficiency.
• Systematic RAG evaluation through frameworks like RAGAS requires measuring both retrieval quality and generation accuracy using metrics including faithfulness, context precision, context recall, and answer relevancy to ensure production system reliability and continuous improvement.

FAQ