This chapter motivates and delivers improvements to retrieval-augmented generation when basic embedding-and-similarity search returns incomplete or off-target results. It explains why user queries and document embeddings often misalign—because of differing terminology, specificity, or context—and positions two levers to close that gap: rewriting the query to better match the corpus and changing what gets embedded to better represent the information space. The goal is a retrieval stage that surfaces more relevant context so downstream generation becomes more accurate and reliable.
The first lever is query rewriting, with a focus on step-back prompting: reformulating a specific question into a broader, higher-level version that is more likely to align with diverse passages about the same topic. This is implemented with an LLM using concise task instructions and few-shot examples, avoiding fine-tuning while yielding robust paraphrases that widen the search net without losing intent. The chapter also introduces advanced embedding strategies such as hypothetical question embeddings, where one embeds the questions a document can answer rather than the document text itself, thereby directly matching user intent. It briefly notes other accuracy boosters—embedding-model finetuning and reranking—as complementary options outside the chapter’s scope.
The second lever is the parent document retriever, which splits large sources into parent sections and smaller child chunks. Only child chunks are embedded and searched, but retrieval returns the full parent section to preserve context and avoid concept “averaging” in long-document embeddings. The chapter shows how to store this hierarchy in a graph database, index child embeddings for vector search, query top child matches, deduplicate to parents, and supply the complete parent texts to the generator. Finally, it integrates both ideas into a pipeline: generate a step-back version of the user query, retrieve with the parent–child strategy, and answer using the original question plus the retrieved parents—delivering more precise matches and richer context for better answers.
Query rewritting by using the stepback technique to increase the vector retrieval accuracy
Hypothetical question and parent document retriever strategies
Rewriting queries using the step-back approach with an LLM
Parent document graph representation
Graph visualization of part of the imported data in Neo4j Browser
Summary
Query rewriting can enhance the accuracy of document retrieval by aligning user queries more closely with the language and context of target documents.
Techniques like hypothetical document retriever and step-back prompting effectively bridge the gap between the user’s intent and the document’s content, reducing the chances of missing relevant information.
The effectiveness of a retrieval system can be improved by embedding not just the exact text but also contextually relevant summaries or paraphrases, capturing the essence of documents.
Implementing strategies like hypothetical question embedding and parent document retrieval can lead to more precise matching between queries and documents, enhancing the relevance and accuracy of retrieved information.
Splitting documents into smaller, more manageable chunks for embedding purposes allows for a more granular approach to information retrieval, ensuring that specific queries find the most relevant document sections.
FAQ
Why can basic vector retrieval underperform in a RAG application?Because the embedding of a user’s query may not align with the embeddings of relevant documents due to differences in terminology or context. This mismatch can cause highly relevant documents to be overlooked, leading to incomplete or irrelevant results.What is step-back prompting and how does it improve retrieval?Step-back prompting is a query rewriting technique that paraphrases a specific question into a broader, higher-level question. Broadening the query helps align it with more documents, improving the chances that vector search retrieves passages containing the needed facts and thus increasing answer accuracy.How do I implement step-back prompting with an LLM?Provide the LLM with a short system instruction to paraphrase the input question into a more generic “step-back” question and include a few example input–output pairs (few-shot prompting). Then, send the user’s original question and use the LLM’s rewritten output as the retrieval query.What is the hypothetical question embedding strategy?Instead of embedding document text, you embed hypothetical questions that the document can answer (generated by an LLM or mined from chat history). At query time, you embed the user’s question and find the nearest hypothetical questions; you then retrieve the documents associated with those questions.What is the parent document retriever and why embed child chunks?The parent document retriever splits a large document into smaller child chunks, embeds the children, and uses those embeddings for matching. When a child chunk matches a query, the system retrieves the entire parent document for richer context. This avoids long-document embeddings that average out concepts and improves specific query matching.How should I structure data for the parent document strategy in a graph?Create a PDF node connected to multiple Parent nodes (e.g., ~2000-character chunks), each connected to multiple Child nodes (e.g., ~500-character chunks). Store the text and embedding on Child nodes. This PDF → Parent → Child hierarchy supports granular matching with full parent context retrieval.How do vector indexes fit into this approach?Create a vector index on the Child nodes’ embedding property (e.g., dimension 1536 with cosine similarity). The index enables efficient nearest-neighbor search over child embeddings when matching a query vector.How does the parent document retrieval Cypher query work?It queries the vector index to get the top k*4 most similar Child nodes, traverses to their Parent nodes, aggregates scores per Parent (e.g., max score), orders by score, deduplicates parents naturally via grouping, and finally limits to k unique Parent documents.How do I assemble the complete pipeline combining step-back and parent retrieval?1) Generate a step-back version of the user’s question using an LLM. 2) Retrieve top Parent documents by searching over Child embeddings and aggregating to Parents. 3) Pass the original question and the retrieved Parent texts to an LLM to generate the final answer.What other techniques can improve retrieval accuracy?Finetune the embedding model to better capture your domain’s semantics, and apply reranking strategies to reorder retrieved documents by relevance to user intent. These can further boost accuracy, though detailed coverage is beyond the scope of this chapter.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Essential GraphRAG ebook for free
