7 Search and Retrieval-Augmented Generation
Many organizations struggle to unlock value from scattered, fast-changing knowledge spread across wikis and files. This chapter shows how Retrieval-Augmented Generation (RAG) weaves that proprietary knowledge directly into language model outputs to make responses relevant, current, and trustworthy. Framed through Alex’s product journey and his users’ needs, it lays out a practical roadmap: start with solid semantic search, combine it with careful prompting for generation, and then iterate with measurement and optimization so the system consistently “gets” the domain rather than producing generic or outdated content.
The retrieval foundation centers on semantic embeddings and a vector database, with careful attention to document chunking so each piece is focused and comparable in length. The chapter details how queries are embedded to fetch top-k candidates, and it emphasizes rigorous evaluation—qualitative “vibe checks,” quantitative metrics like precision, recall, and mean reciprocal rank, and production monitoring with CTR and engagement signals. It then walks through high-impact optimizations: smarter chunking (semantic and hierarchical), contextual retrieval that enriches fragments with summaries, domain-adapted embedding models via fine-tuning, hybrid retrieval that blends semantic and lexical signals for exact matches, metadata filtering with time bias for freshness, and reranking to recover meaning lost during embedding compression.
On the generation side, a basic RAG pipeline augments prompts with retrieved context and selects long-context LLMs suited to open-ended tasks. Evaluation spans component-level checks and end-to-end “LLM-as-a-judge” scoring for groundedness and answer relevance. The system is further strengthened by query analysis (rewriting, expansion, and classification), prompt constraints that dial between strict source-grounding and broader synthesis, structured reasoning techniques (such as stepwise decomposition and reflection), context fusion to remove redundancy, and multi-turn retrieval to fill gaps. For advanced scenarios, the chapter introduces GraphRAG to leverage entity relationships, and it closes with the option to fine-tune the LLM for deep domain expertise—tying together a lifecycle that turns proprietary data into a durable product advantage.
From semantic search to a full RAG system
The lifecycle of a RAG system
Semantically similar words are closer to each other in embedding space.
Sentences arranged in an embedding space
Constructing the embedding database
The process of semantic search
Recap of the search system and optimization potentials
Combining semantic and lexical search can increase overall retrieval accuracy.
Schema of a RAG system. Users prompt the system with a query, the results of which are fed into the embedding database. Those results generate an augmented prompt fed to an LLM, which generates a final response to the user query.
Optimization potentials in the RAG system
Summary
- A Retrieval Augmented Generation (RAG) system combines semantic search with large language models (LLMs) to retrieve relevant documents and use them to generate specific, up-to-date, and contextually relevant content.
- RAG allows product builders to bridge the gap between the amazing capabilities of generative AI, and their users' specific domain and data.
- Semantic embeddings are a key representation in modern NLP, accurately reflecting the semantics of words and texts.
- Semantic embeddings allow you to efficiently retrieve relevant documents for a given user query in the context of search - whether stand-alone or as part of a RAG system.
- Providing the retrieved documents as additional context in the prompt ensures the LM responses are factually grounded and domain-specific.
- Evaluate your RAG system by measuring context relevance, groundedness, and answer relevance to ensure high-quality, reliable outputs.
- RAG is simple, but not easy. After setting up your initial end-to-end system as a baseline, product teams must experiment with many optimization parameters to achieve high output quality.
- To optimize semantic search, you can work with advanced chunking methods, improve the embedding model, and add lexical search and metadata filters for more precise retrieval.
- To optimize response generation, you can fine-tune the prompt and the language model used for content generation.
FAQ
What problems does RAG solve that prompt engineering alone cannot?
RAG grounds model outputs in the user’s up-to-date, domain-specific knowledge. It reduces generic or outdated responses, cuts post-editing time, and helps the AI “get” the user’s domain by retrieving relevant internal and external sources at generation time. This directly addresses trust issues caused by undifferentiated content.
How does semantic search with embeddings differ from keyword search?
Semantic search represents texts and queries as vectors and retrieves results by meaning, not just exact word matches. It surfaces relevant content even when wording differs (e.g., “vehicle” vs. “car”), improving recall and relevance across sentences, paragraphs, and documents.
How do I build an embedding database for retrieval?
- Ingest data from key repositories (e.g., Confluence, Google Drive).
- Chunk documents into moderately sized segments (often 200–300 tokens to start).
- Embed chunks with a suitable model (e.g., MiniLM, SentenceBERT).
- Store vectors and metadata in a vector-capable database (e.g., Weaviate, Pinecone, FAISS, Milvus, PostgreSQL with pgvector, IBM watsonx.data).
- Expose similarity search APIs and tune parameters (e.g., top-k).
What’s the right way to chunk documents?
- Baseline: fixed-size chunking for consistency and speed.
- By structure: split by sentences/paragraphs for shorter docs; hierarchical chunking for long reports.
- By meaning: semantic chunking to align segments with distinct topics.
- Trade-offs: smaller chunks for precise facts; larger chunks for richer context in open-ended generation.
How should I evaluate and monitor retrieval quality?
- Qualitative: vibe checks, user studies, task-based reviews.
- Quantitative: precision, recall, Mean Reciprocal Rank (MRR) on a labeled test set.
- In production: CTR on results, dwell time, bounce rate, and query log analysis.
- Tuning: adjust top-k to balance recall (too small misses info) and noise/cost (too large).
Which techniques improve retrieval beyond basic embeddings?
- Contextual retrieval: prepend a 50–100 token chunk summary to restore context before embedding.
- Hybrid search: combine semantic with lexical (e.g., BM25) for exact matches on technical terms.
- Metadata and recency: filter and boost by author, type, date; time-bias to promote fresh content.
- Reranking: apply a supervised reranker (e.g., Cohere Rerank) over a generous candidate set to recover information lost in dense vectors.
How do I pick an LLM and design a basic RAG prompt?
- Model choice: favor LLMs with strong general knowledge, conversational tuning, and long-context handling; watch input token costs and “lost in the middle” issues.
- Prompt structure: system role, instruction, and retrieved context chunks.
- Constraints: for factual tasks, require use of provided sources and citations; for exploratory tasks, allow combining with general knowledge.
How do I evaluate a RAG system end-to-end?
- Component-level: measure retrieval (precision/recall/MRR) and generation (hold retrieval fixed while testing prompts).
- LLM-as-a-judge: use strong models to score outputs for groundedness (uses retrieved sources) and answer relevance (fully addresses the query).
- Iterate with real user data to reflect true use cases.
How can I improve user queries and context curation?
- Query improvement: apply query expansion/transformation; classify queries to route to domain-specific pipelines and sources.
- Context curation: fuse redundant snippets, enforce coherent structure, and use multi-turn retrieval to fill gaps for complex tasks.
When should I fine-tune models or consider GraphRAG?
- Embedding model fine-tuning: capture domain-specific terminology and nuances for better retrieval.
- LLM fine-tuning: when generation quality depends on deep domain knowledge and source data is noisy or biased.
- GraphRAG: add a knowledge graph to leverage entity relationships for complex, multi-step queries in technical, legal, or research-heavy domains.
The Art of AI Product Development ebook for free
