8 Advanced indexing

Modern RAG systems often underperform not because of weak models but because of naïve indexing and retrieval design. This chapter reframes accuracy as a pipeline problem across ingestion and Q&A: vague queries, mismatched chunking, and thin metadata commonly yield irrelevant or incomplete context. The remedy is iterative design with LangChain—tuning chunking, enriching indexes, transforming questions, routing to the right stores, and post-filtering—so the retriever consistently surfaces the most useful evidence for synthesis.

The core advances center on indexing. First, choose a split strategy aligned to your questions: hierarchical splits preserve semantics for topic-centric docs, while fixed-size splits offer consistency; balance granular chunks (precision) and coarse chunks (context), and adjust overlaps as needed. Then layer multi-vector techniques to make coarse chunks discoverable by fine-grained signals: embed child chunks but return parent chunks (ParentDocumentRetriever) for rich context; add summary embeddings to densify coarse representations; generate hypothetical question embeddings so user phrasing better matches stored vectors; and expand granular chunks at retrieval time with neighboring sentences to restore missing context. Together, these methods raise recall and precision without sacrificing synthesis quality.

On the query side, transform unclear or broad questions into sharper, retrieval-ready variants, decompose them when necessary, and route to the right content store. Blend vector search with structured sources (relational or graph), generating source-specific queries (e.g., SQL) to unlock tables and facts, and filter noisy results before synthesis. Finally, extend the same principles to semi-structured and multi-modal content: summarize and embed tables or images, store the raw artifacts separately, and use a multi-vector retriever to return the full objects when their summaries hit. The result is a robust, ensemble-style RAG pipeline that consistently retrieves focused, context-rich evidence across text, structured data, and media.

Common accuracy issues in the ingestion stage of a simple RAG architecture are often due to inadequate indexing that only uses basic embeddings for each text chunk. Advanced indexing techniques involve generating multiple embeddings for each chunk, enhancing searchability.

Common Issues in the Q&A stage of the naive RAG architecture and their solutions:1) Handle poorly formulated questions with question transformation; 2) Enhance retrieval accuracy by transforming the original user questions into more suitable vector DB search queries; 3) Include relevant data sources by adding structured data content stores, such as relational DBs; 4) Generate DB queries for structured data content stores; 5) Filter out irrelevant context retrieved from the content stores.

Impact of Chunk Size on Answer Accuracy. Coarse chunks provide more context but less focus, and they are suitable for broader questions. Granular chunks provide less context but more focus and they are suitable for more detailed questions.

Child Chunk embeddings—A coarse document chunk is indexed with its own embedding and the embeddings generated from its smaller child chunks. This allows it to match effectively with both broad and detailed queries.

Chunk Summary Embedding—A coarse chunk is indexed with its own embedding and an additional embedding from its summary. This allows for more accurate retrieval when answering detailed questions.

Hypothetical Question embeddings—A document chunk is indexed with its own embedding and additional embeddings generated from hypothetical questions that it can answer. This allows for more accurate matching with user queries.

Sentence Expansion—A granular chunk can be enhanced by including the content from its preceding and following chunks to provide additional context.

Embedding Structured and Unstructured Content—Structured data, such as tables, should be summarized and embedded just like unstructured text chunks. This ensures that the embeddings match detailed questions as effectively as text embeddings. Use the MultiVectorRetriever to manage both types of content.

Multi-Modal RAG Workflow—1) Data Ingestion: Use a multi-modal LLM to generate an image summary, store the summary embeddings in a vector store, and keep the raw image in a document store. 2) Retrieval: If the summary embeddings match a query, the raw image is returned by the MultiVectorRetriever and fed into the LLM for synthesis.

• A basic RAG implementation often produces low-quality results, making it unsuitable for real-world applications.
• You can improve answer quality using techniques like advanced indexing, multiple query generation, query transformation, content-specific queries, retrieval post-processing (e.g., result reranking), and ensemble methods.
• Advanced indexing may include refined document splitting strategies, multiple embeddings for coarse chunks, and expanded context for granular chunks.
• Choose a splitting strategy based on content type: by size (e.g., paragraphs, sentences, words, or characters) or by structure (e.g., chapters or sections).
• Structured formats like HTML or Markdown benefit from splitting based on document hierarchy.
• For coarse chunks, enhance retrieval accuracy by using multiple embeddings generated from child chunks, summaries, or hypothetical questions about the content.
• Embed child chunks using the ParentDocumentRetriever. Use the MultiVectorRetriever to embed chunk summaries, hypothetical questions, or child chunks.
• Since it’s hard to predict which multi-vector retrieval technique will perform best, experiment with each and compare results to find the optimal approach.
• For granular chunks, expand retrieval context by including adjacent chunks for a fuller context.
• For semi-structured content (e.g., text with tables) or multi-modal content (e.g., images and audio), use specialized indexing techniques for optimal results.

FAQ