6 RAG fundamentals with Chroma DB

This chapter introduces semantic search and the Retrieval Augmented Generation (RAG) pattern as a practical way to build Q&A systems over documents. It contrasts semantic search with keyword matching, showing how LLMs retrieve meaningfully related passages even when words differ, and highlights the risks of hallucination when models answer beyond the provided context. Through a simple single-document chatbot example, it illustrates why constraining answers to supplied text and using conversational state can improve accuracy and user experience.

The core RAG architecture is presented in two stages: ingestion and question-answering. Ingestion prepares content by extracting text, splitting it into concise chunks, and converting each chunk into embeddings stored in a vector database for efficient semantic lookup. The Q&A stage embeds the user question, performs a similarity search to retrieve the most relevant chunks, and then augments the LLM prompt with this context so the model can synthesize an answer. The chapter explains vector stores, similarity measures, and the evolution from lightweight vector libraries (e.g., FAISS) to full-featured vector databases (e.g., Chroma, Milvus, Pinecone, Weaviate) and extensions in general-purpose databases, emphasizing CRUD support, metadata handling, and scalability.

Building on these concepts, the chapter walks through implementing RAG with Chroma DB and an OpenAI model. It shows how to stand up Chroma, create a collection, add chunked documents with metadata, and run semantic queries that return ranked passages and distances. It then composes a prompt that combines the original question with retrieved context and applies a safer Q&A template to reduce hallucinations, culminating in a minimal chatbot function that orchestrates retrieval and generation. By the end, you understand the moving parts of RAG—chunking, embeddings, retrieval, prompting, and synthesis—and are prepared to re-implement the pattern with higher-level frameworks in subsequent chapters.

A simple Q&A chatbot process: 1) The user sends a prompt containing a document (context) and a question to the chatbot; 2) The chatbot returns an initial answer; 3) The user follows up with a more detailed question; 4) The chatbot provides a more accurate answer.

Hypothetical design for an enterprise Q&A chatbot: the knowledge of the chatbot is expanded with various data sources: the enterprise intranet, knowledge bases and documents

RAG Ingestion Stage: Documents are extracted from sources, split into chunks, and converted into embeddings while being stored in a vector database, which stores a copy of the original chunks and their embeddings (vector form).

RAG Question & Answer Stage: 1) The chatbot uses a retriever to transform the user question into embeddings; 2) then the retriever uses the embedding to perform a similarity search in the vector store. 3) The vector store returns several relevant text chunks; 4) The retrieved content is fed into a prompt as a "context" along with the original user question. 5) The chatbot sends the prompt to the LLM which synthesizes a response and returns it to the user.

RAG Architecture including Chroma DB and GPT-5-nano model

• Implement a minimalistic Q&A chatbot by feeding a question and supporting document to an LLM like ChatGPT or Claude.
• For answering questions over a knowledge base, use RAG, which includes a vector store, a retriever, and an LLM to synthesize responses.
• The RAG design pattern has two stages: ingestion (populating the vector store with text and embeddings) and Q&A (retrieving relevant documents and generating responses).
• A vector store is a specialized database optimized for storing and retrieving text through related embeddings.
• Many vector stores evolved from in-memory libraries to full-fledged persistent databases.
• You can implement a simple RAG system by directly using the APIs of the LLM (such as the OpenAI API) and the vector store (such as the Chroma DB API).

FAQ