9 Question transformations

This chapter argues that many Retrieval-Augmented Generation failures stem less from missing data and more from how users phrase their questions. It presents a set of question-transformation strategies that better align queries with embeddings and retrievers, boost recall and precision, and provide richer, more relevant context to the model. The overarching pattern is to transform the user’s question for retrieval while preserving the original wording for answer synthesis, so the final response remains faithful to the user’s intent.

The first family of techniques reframes or multiplies the query. In Rewrite-Retrieve-Read, an LLM rewrites a vague or noisy question into a clearer, retrieval-friendly form before searching a vector store, then the original question is used to synthesize the answer. When a question is well-formed but multifaceted, the chapter recommends generating multiple explicit sub-questions, retrieving for each, and fusing the resulting contexts; this broadens coverage and mitigates similarity-search blind spots. These methods can be composed into a single RAG chain so that rewriting or multi-query expansion happens automatically and transparently prior to retrieval.

The second family broadens or enriches context beyond the literal query. Step-back questions convert a detailed prompt into a more abstract version to fetch higher-level context that complements fine-grained snippets, improving completeness. HyDE generates a short, hypothetical answer document from the user’s question and uses it as the retrieval query, often increasing semantic alignment with indexed chunks. Finally, the chapter distinguishes between single-step decomposition (independent sub-questions) and multi-step decomposition (interdependent, parameterized steps executed sequentially across sources), noting that complex tasks may involve routing, structured-query generation, and iterative retrieval. The core takeaway: treat the question as a transformable object—rewrite it, expand it, generalize it, or decompose it—to systematically improve RAG retrieval and final answer quality.

In the standard Retrieve-and-Read setup, the retriever processes the user question directly, delivering results to the LLM for synthesis. In the “Rewrite-Retrieve-Read” approach, an initial Rewrite step uses an LLM to rephrase the query before it reaches the retriever, enhancing the clarity of the retrieval process (diagram adapted from the paper “Query Rewriting for Retrieval-Augmented Large Language Models”, https://arxiv.org/pdf/2305.14283.pdf).

Using query rewriting to generate an optimized vector store query, while preserving the original question for answer synthesis via the chatbot prompt.

Workflow for Multiple Query Generation: The LLM application reformulates the original question into multiple explicit questions, which are executed against the vector store to produce the context, and synthesizes the answer from a prompt with the original question and the produced context.

Step-Back Question Workflow: The LLM application first sends the detailed question (Q_D) to the vector store to retrieve a detailed context (C_D). It then prompts the LLM to generate a more abstract question (Q_A) based on Q_D, which is also executed on the vector store to obtain an abstract context (C_A). Finally, the LLM application combines Q_D, C_D, and C_A into a single prompt, enabling the LLM to synthesize a comprehensive answer.

Hypothetical Document Embeddings (HyDE): In this approach, the LLM generates hypothetical documents that would answer the user’s question. Rather than querying the document store with the user’s original question, these generated documents are used. Because these hypothetical documents are semantically closer to the document chunk text, they improve the relevance of retrieved content.

Multi-Step Decomposition Workflow: In this workflow, the original complex question is sent to the LLM, which generates a sequence of parameterized questions. Each question is then executed on the vector store, using answers from previous steps as parameters. Once all questions are answered, the LLM synthesizes a final response based on the collected information.

• The quality of LLM answers depends on well-structured questions, as vague or complex queries can mislead the vector store and LLM, resulting in weaker results.
• Poorly phrased questions lead to less accurate retrievals and answers, making clear and concise questions essential.
• The "Rewrite-Retrieve-Read" approach improves answers by refining the original question, helping retrieve better context and enhancing LLM responses.
• Splitting questions into explicit sub-questions, executed separately, ensures better retrieval, and LangChain’s MultiQueryRetriever can combine these answers into a comprehensive response.
• Highly detailed queries can return overly specific results, missing the broader context needed for a complete answer.
• Using coarse chunks for synthesis and fine-grained chunks for detail, or employing “step-back questions” to broaden queries, helps capture high-level context.
• Generating hypothetical documents with HyDE retrieves semantically aligned content based on user questions without altering chunk embeddings.
• Decomposing questions into independent sub-questions allows for individual retrieval and synthesis.
• Generating a sequence of parameterized questions for interdependent queries ensures that each step builds on the previous answer, systematically resolving dependencies for a final answer.

FAQ