7 Enterprise RAG: Agentic Routing, Semantic Caching, and Query Rewriting

This chapter reframes retrieval-augmented generation for enterprise realities, moving beyond single-pipeline “naive” RAG to adaptive, auditable systems that reason about user intent, data domains, and operational constraints. It outlines how large, heterogeneous organizations face challenges that simple vector search cannot solve alone—strict latency SLAs, cost controls, regulatory and provenance needs, and frequent ambiguity in user queries. Within a broader Enterprise RAG landscape that also includes safety guardrails, memory, access control, and observability, the chapter focuses on three high-impact pillars—agentic routing, semantic caching, and query rewriting—as the fastest path from proof-of-concept to production-grade performance and reliability.

Agentic routing sits at the decision-making core, classifying intent and directing queries to purpose-built knowledge partitions and retrieval strategies rather than a monolithic index. By separating “where to look” (routing) from “what to find” (retrieval), it avoids cross-domain contamination, respects authority and provenance, and preserves precision as domains scale. A minimal three-route example demonstrates the pattern: two internal, domain-specific vector collections for technical documentation and financial filings, plus a live web search fallback for out-of-scope or current-event queries. The router’s structured, justified outputs create an audit trail for transparency, while domain-specific retrieval keeps prompts focused, costs contained, and answers grounded—at the known tradeoff of a small added routing latency.

To meet enterprise cost and latency targets, semantic caching reuses results for semantically similar questions via an embedding-based cache (e.g., FAISS), with a conservative similarity threshold for accuracy and a fast keyword filter to bypass the cache for time-sensitive queries. This cuts LLM calls and vector searches on repeat questions, delivering millisecond responses without sacrificing freshness. Query rewriting and sub-query decomposition improve retrieval quality before it begins: a rewriter expands abbreviations, resolves ambiguity using recent context, and preserves user intent, while a decomposer splits compound prompts into focused, independently retrievable sub-queries. Combined into a unified pipeline—cache check, time-sensitivity detection, routing, rewriting, optional decomposition, targeted retrieval, grounded synthesis, then caching—the three pillars yield a scalable, auditable, and high-precision Enterprise RAG system that aligns with real-world compliance and performance demands.

Enterprise RAG: A modular framework that uses a central decision-making hub to route requests through specialized knowledge repositories and safety guardrails for high-precision results.

Agentic Router: A contrast between two frameworks, one that uses central knowledge retrieval (left) vs. a router-based approach to retrieve data from relevant repositories (right)

Naïve Search: A central unified index raises challenges of pulling data from one source, often leading to irrelevant retrieval

illustrates the complete query flow through the three-route architecture, from the initial user query through routing classification, domain-specific retrieval, and final response generation.

The core concept of semantic caching within an Enterprise RAG system to help reduce processing time to look up time for similar queries previously asked

A detailed, step-by-step pipeline diagram illustrating a query processing system

• The first step in building Enterprise RAG is recognizing that naive RAG architectures fail in enterprise environments. These architectures embed all documents into a single index and run every query against the same collection. When document types differ significantly, such as financial filings, technical documentation, and live web content, a shared vector space introduces cross-domain interference that degrades retrieval precision across every query.
• To direct queries to the right knowledge base, we built an LLM-based agentic router that classifies user intent and sends each query to the most appropriate retrieval backend. Unlike keyword matching or naive semantic search across a unified index, the LLM router reasons about the meaning and category of a query as a whole, correctly classifying questions even when they contain no obvious domain-specific keywords.
• The three-route architecture we built covers OpenAI documentation, 10-K financial filings, and live web search, and is deliberately minimal. The same routing pattern scales to any number of domain-specific collections. The key design principle is that routes should map to fundamentally different retrieval strategies, not just different topics. Adding a new route requires only a new collection and an updated router prompt.
• Semantic caching addresses the cost and latency problem that comes up when enterprise users repeatedly ask variations of the same question. By embedding each query and comparing it against previously cached queries using FAISS with an L2 distance threshold, the system can return stored answers for semantically equivalent questions in 10 to 50 milliseconds, which is over 100x faster than running the full routing, retrieval, and generation pipeline.
• Cache correctness depends heavily on the time-sensitivity filter. We use a keyword-based approach rather than an LLM-based classifier to keep cache lookups fast. Queries containing temporal indicators like "today," "current," or "latest" bypass the cache entirely and always pull fresh results from live search, which prevents the system from returning stale answers to time-sensitive questions.
• To handle the ambiguity and imprecision of real-world enterprise queries, we added query rewriting and sub-query decomposition as pre-retrieval transformation steps. The single-query rewriter expands abbreviations, resolves vague references using conversation history, and adds domain specificity. The sub-query decomposer breaks compound questions into focused atomic queries that can each be routed and retrieved independently, giving precise retrieval for every distinct information need within a complex question.
• The complete Enterprise RAG pipeline processes each query through a fixed sequence: semantic cache check, time-sensitivity filtering, LLM-based routing, query rewriting, optional sub-query decomposition, domain-specific retrieval, grounded response generation, and cache storage. Each component targets a specific failure mode of naive RAG, and the most expensive components only run on cache misses, so the system gets faster and cheaper the longer it operates.

FAQ