10 Query generation, routing and retrieval post-processing

This chapter deepens Retrieval-Augmented Generation by teaching models to ask the right questions of the right systems and to pass only the best evidence to the LLM. It focuses on three pillars: generating store-specific queries (for vector, relational, document, and graph databases), routing user questions to the appropriate retrieval pipeline, and post-processing retrieved results to maximize answer quality. The overarching goal is higher precision, better coverage, and lower noise throughout the RAG workflow.

First, the chapter shows how to turn natural language into structured queries tailored to each store. For vector databases, it introduces self-querying: enriching semantic search with inferred metadata filters via explicit filters, SelfQueryRetriever, or function-calling into strongly typed objects. For relational data, it demonstrates text-to-SQL with schema-aware prompting, SQL cleanup, and execution chains; it then extends this to semantic SQL by storing embeddings in the database (e.g., via vector columns) to support similarity search alongside exact matches. For knowledge graphs, it outlines KG-RAG: extracting entities/relations, generating Cypher or SPARQL from questions, executing the queries, and synthesizing natural-language answers from graph results.

Next, it introduces an LLM-based router that selects the right retrieval branch (e.g., vector-store facts vs. SQL offers), then integrates routing and retrieval into end-to-end RAG chains. Finally, it details retrieval post-processing techniques: similarity-score thresholds, keyword inclusion/exclusion, time-weighted ranking to favor recent content, and Reciprocal Rank Fusion (RAG Fusion) to rerank merged results from multiple reformulated queries. Combined, these methods deliver more accurate, explainable, and efficient retrieval pipelines that consistently improve downstream answer synthesis.

Self-querying workflow: the original question turns into a semantic query with an embedded metadata filter. When this enriched query runs on the vector store, it first selects chunks matching the metadata filter, then applies semantic search on this refined set.

Entity-Relationship diagram of the UkBooking DB

RAG with SQL Workflow: The LLM converts the natural language question into a SQL query, which is executed on the SQL database. The database returns records that are then processed by the LLM to generate the final answer.

Graph Representation of data: Nodes represent entities such as "Roberto" and "InterMilan," while edges like `hasOccupation` and `isFanOf` depict their relationships.

Knowledge Graph RAG Architecture (KG-RAG): Similar to vector-store-based RAG setups, but using a SPARQL generator. The generator converts a natural language question into SPARQL, which is executed on the Knowledge Graph database. The retrieved graph data is then provided to the LLM, along with the original question, to synthesize the answer.

Complex RAG architecture with branching pathways, each optimized for specific tasks, such as answering questions about tourist Destinations (from a vector store) or Accommodation offers (from a relational SQL DB).

Retrieval Post-processing: Retrieved chunks from the vector store are filtered to remove irrelevant content, ensuring only high-quality chunks are sent to the LLM for answering the user’s question.

Reciprocal Rank Fusion Workflow: Multiple queries are generated from the initial user question. Each query retrieves a set of results, such as from a vector store. All results are then reranked using RRF scores, with only the top results sent to the LLM for final answer synthesis.

• Use structured data from SQL and document databases to enhance LLM responses, complementing unstructured data in vector stores.
• Structured queries are often required for these databases, and LLMs can generate them from natural language inputs.
• Vector stores support dense searches with embeddings and sparse searches using keyword-based indexing.
• Keywords for sparse searches can be added via metadata, derived with algorithms like TF-IDF, or generated by the LLM.
• LLMs can also generate metadata queries directly from user questions.
• Many LLMs can translate user questions into SQL queries, enabling direct access to relational databases.
• SQL-based retrievals rely on exact matching, but semantic SQL expands searches to find approximate matches aligned with user intent.
• Graph databases represent data as nodes and relationships, making them ideal for complex, interconnected information.
• Graph databases excel in LLM use cases requiring advanced reasoning and relationship understanding, as they can be used to create knowledge graphs with the help of LLMs or answer questions based on knowledge graphs through KG-RAG.
• Content for LLM applications often spans vector stores, relational databases, document stores, and knowledge graphs.
• Routing chains ensure queries are directed to the appropriate storage type for optimized retrieval.
• Reciprocal Rank Fusion (RRF) improves query relevance by reranking documents from multiple queries using cumulative scores.
• RRF enhances answer quality by feeding the most relevant results to the LLM for synthesis.

FAQ