3 Indexing Pipeline: Creating a Knowledge Base for RAG-based Applications
This chapter explains how to build the indexing pipeline that powers the non-parametric memory of RAG applications. It presents a clear, end-to-end path from raw sources to a searchable knowledge base, organized around four components: Data Loading, Data Splitting (chunking), Data Conversion (embeddings), and Storage (vector databases). With practical, Python-oriented examples using LangChain, the chapter shows how to connect to diverse sources, transform content, generate vector representations, and persist them for fast retrieval—equipping readers to set up the pipeline before real-time interaction with an LLM begins.
The data loading section emphasizes connecting to sources, extracting and parsing text, curating metadata, and cleaning or transforming content (including masking and de-duplication). Chunking is motivated by LLM context limits, the “lost-in-the-middle” effect, and search efficiency. It covers fixed-size approaches (by character or by token, often with recursive splitting and overlap), specialized/adaptive methods that respect structure in formats like HTML, Markdown, JSON, or code, and experimental semantic chunking that groups text by meaning. The chapter recommends choosing a chunking strategy based on content type, expected query length/complexity, the application’s goals (e.g., QA vs summarization), and the embeddings model being used.
Next, the chapter introduces embeddings as high-dimensional vector representations of text that enable semantic similarity search via metrics like cosine similarity or Euclidean distance. It outlines the use of both open-source and proprietary models to embed chunks, then details vector databases as the storage layer for efficient indexing and retrieval. Options span lightweight vector indices, specialized vector stores, search platforms, and SQL/NoSQL/graph systems with vector capabilities. Selection trade-offs include accuracy versus speed, flexibility versus performance, hosting model, feature depth, and cost. The chapter concludes by tying the steps together—loading, chunking, embedding, and vector storage—to demonstrate similarity search that retrieves the most relevant chunks, forming a robust knowledge base for RAG.
Four sub-steps of the Data Loading component of the Indexing Pipeline
Data chunking process
Chunking Methods.
Words in a unidimensional vector
Words in a two-dimensional vector space
The process of embedding transforms data (like text) into vectors, compresses the input information resulting in an embedding space specific to the training data.
Cosine similarity of vectors in two-dimensional vector space.
Embeddings created for chunks of Wikipedia page using all-MiniLM-l6-v2 model.
Creating the knowledge base for RAG enabled system using a combination of strategies.
FAQ
What is the Indexing Pipeline in a RAG system and why is it needed before user interaction?
The Indexing Pipeline builds the application’s knowledge base (non‑parametric memory) before any real‑time LLM interaction. It ingests raw data, splits it into manageable chunks, converts those chunks into embeddings, and stores them in a vector database so relevant information can be retrieved at query time.What are the four components of the Indexing Pipeline?
- Data Loading (connect, extract, parse, clean)- Data Splitting or Chunking (create searchable, contextual segments)
- Data Conversion to Embeddings (numeric vector representations)
- Storage in Vector Databases (for fast semantic search and retrieval)
What does Data Loading involve and what are its sub-steps?
Data Loading sources content from files, web pages, databases, or cloud stores and prepares it for indexing. Sub‑steps:- Connect to the source (e.g., Wikipedia, S3, SQL/NoSQL)
- Extract and parse text (HTML, PDF, DOCX, CSV, etc.)
- Review and update metadata (source, title, language, timestamps, authorship)
- Clean/transform data (remove HTML, special characters, deduplicate, mask PII, optional fact‑checks). Tools shown: LangChain loaders like AsyncHtmlLoader, transformers like Html2TextTransformer.
Why is metadata important and how is it used during retrieval?
Metadata describes the data’s origin, type, purpose, and context (e.g., source URL, title, language, dates). It enables filtering and ranking during retrieval, helps resolve conflicts (chronology, authority), and improves relevance. Many sources provide some metadata automatically; you can and often should add more.Why do we split data into chunks for RAG?
- Respect LLM context-window limits (avoid truncation).- Mitigate “lost in the middle” by sending only the most relevant portions to the LLM.
- Improve search efficiency: smaller, focused chunks rank and retrieve better than long monolithic text.
What chunking methods are available and when should I use them?
- Fixed-size chunking: split by character or token, with overlap; use CharacterTextSplitter or RecursiveCharacterTextSplitter (more robust).- Token-based sizing: align chunk size/overlap to tokenizer (e.g., tiktoken).
- Specialized/adaptive: respect structure for HTML/Markdown/JSON/code using HTMLHeaderTextSplitter, MarkdownHeaderTextSplitter, RecursiveJsonSplitter, etc.
- Semantic chunking (experimental): group sentences by meaning using embeddings (SemanticChunker).
- You can combine methods (e.g., split by HTML headers, then apply fixed-size).
How should I choose chunk size and overlap?
Base it on:- Nature of content (long reports vs short posts; code/HTML benefits from structural splits).
- Expected query length/complexity (long queries may suit larger chunks; short queries often need tighter, focused chunks).
- Use case (Q&A prefers smaller, precise chunks; summarization favors larger chunks).
- Embeddings model behavior (some models perform best within certain chunk sizes). Experiment and iterate.
What are embeddings and how is similarity computed?
Embeddings convert text into high‑dimensional vectors where semantically similar items lie close together. Similarity is commonly measured by:- Cosine similarity (angle between vectors; -1 to 1, higher is more similar).
- Euclidean distance (smaller distance means more similar).
Embeddings power semantic search, clustering, ML features, and recommendations in RAG.
How do I choose an embeddings model, and what are some popular options?
Choose based on use case performance (e.g., retrieval vs clustering), cost, and language/domain needs. Popular options mentioned:- OpenAI: text-embedding-3-small (configurable dims), text-embedding-3-large (3072 dims).
- Google: text-embedding-004 (elastic dims up to 768).
- Voyage AI: voyage-large-2-instruct, voyage-large-2, voyage-law-2, voyage-code-2.
- Mistral: mistral-embed (open-source).
- Cohere: embed-english-v3.0, embed-english-light-v3.0, embed-multilingual-v3.0.
Use benchmarks like MTEB for guidance; also consider token volume costs.
What are vector databases and how do I store/query my knowledge base?
Vector databases index and store embeddings for fast similarity search. Categories include:- Vector indices (FAISS, NMSLIB, ANNOY, ScaNN).
- Specialized vector DBs (Pinecone, ChromaDB, Milvus, Qdrant, Weaviate, etc.).
- Search platforms with vectors (Elasticsearch, OpenSearch, Solr, Lucene).
- SQL/NoSQL/Graph with vector support (Postgres+pgvector, Azure SQL, MongoDB, Neo4j).
Selection factors: accuracy vs speed, flexibility vs performance, local vs cloud, API vs direct libraries, feature complexity, cost. Example flow with LangChain: load → split → embed (e.g., HuggingFaceEmbeddings or OpenAIEmbeddings) → store (e.g., FAISS) → query via similarity_search.
A Simple Guide to Retrieval Augmented Generation ebook for free
