This chapter explains how embeddings and vector similarity search turn unstructured data into machine-usable vectors and how these pieces power retrieval-augmented generation (RAG). It frames a RAG system around two roles: a retriever that finds relevant information and a generator (typically an LLM) that composes the final answer. By using vector similarity—most effectively cosine similarity for text—the retriever supplies focused evidence, enabling smaller, cheaper models to generate grounded responses with fewer hallucinations.
After outlining the concepts, the chapter walks through a practical RAG build: preparing a text corpus, chunking it thoughtfully (size, overlap, and split strategy) to sharpen semantics, and embedding all chunks with a consistent model and dimension. It highlights why a vector index (ANN) trades a bit of accuracy for speed and shows a simple data model where chunks and their embeddings are stored (here in Neo4j) and queried to return top-k matches. The retriever pipeline embeds the user query, searches the vector space, and passes selected chunks to the generator, which uses a constrained prompt (system and user messages) to produce an answer grounded in the retrieved text.
To improve retrieval quality further, the chapter adds fulltext search for a hybrid approach that blends keyword and semantic results. It demonstrates running both searches, normalizing scores, merging and deduplicating candidates, and then selecting the best k results—often boosting recall and precision versus either method alone. The chapter closes by noting remaining limits when working purely with unstructured text (missed references and insufficient context) and foreshadows later techniques that add structure, iteratively strengthening both retriever and generator performance.
The data flow for this RAG application using vector similarity search.
The data model
The data model
Summary
A RAG application consists of a retriever and a generator. The retriever finds relevant information, and the generator uses that information to create a response.
Text embeddings capture the meaning of text in a vector space, which allows us to use vector similarity search to find similar text.
By adding fulltext search to the RAG application, we can enable hybrid search to improve the performance of the retriever.
Vector similarity search and hybrid search can work well in certain situations, but their quality, accuracy, and performance is still quite limited as the data complexity grows.
FAQ
What are embeddings and why are they important for RAG?Embeddings are numeric vectors that represent complex data (like text) in a uniform vector space. In RAG, they let the retriever compare a user query to document chunks by semantic similarity, enabling the system to find meaningfully related content rather than exact keyword matches.How do the retriever and generator work together in a RAG application?The retriever finds relevant information (usually via vector similarity search) and passes it to the generator. The generator (often an LLM) then crafts the final answer using only that retrieved context, which reduces hallucinations and allows smaller, faster models.Why use a vector index (ANN) instead of brute-force similarity search?A vector index enables approximate nearest neighbor (ANN) search, which trades a small amount of accuracy for significant speed. It’s highly recommended for scalability compared to brute-force, especially as your corpus grows.Which similarity metric should I use for text embeddings?Cosine similarity is typically best for text because it measures the angle between vectors, aligning well with semantic similarity. Euclidean distance focuses on magnitude and is usually less helpful for text matching in this context.Why must I use the same embedding model throughout the pipeline?The retriever and your stored vectors must live in the same embedding space. If you switch models (especially if the dimension changes), you need to regenerate embeddings and recreate the vector index to avoid mismatches and degraded search quality.How should I chunk text for better retrieval?Split documents into smaller, meaningful chunks to sharpen each embedding’s focus. Use a sliding window with overlap to capture context spanning boundaries, and consider splitting on whitespace to avoid broken words. Optimal chunk size and overlap depend on your data and use case—experiment.What does the retriever pipeline do at query time?It embeds the user query with the same model used for documents, runs a vector similarity search against the index, and returns the most relevant chunks. Often, it applies post-processing (e.g., re-ranking or deduplication) before passing chunks to the generator.How do I store and search embeddings in a database?Use a database that supports vector indexes. For example, in Neo4j you store chunks as nodes with text and embedding properties, create a vector index with the correct dimension and similarity function (e.g., cosine), then query it to retrieve nearest neighbors.What is hybrid search and when should I use it?Hybrid search combines vector similarity search with fulltext (keyword) search, then merges and normalizes scores. It improves recall and robustness—useful when exact terms matter (keywords) and when semantic meaning matters (vectors).How do I combine scores from vector and fulltext search?Run both searches, normalize each set of scores by dividing by its own maximum, merge results, deduplicate by node, and select the top-k by the normalized score. Ensure index names and settings match your creation queries to avoid configuration mistakes.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Essential GraphRAG ebook for free
