Overview

8 Understanding agent memory and knowledge

This chapter explores how retrieval underpins agent memory and knowledge, showing how both are used to augment prompts with relevant context. It distinguishes knowledge (document-derived context) from memory (agent- or user-derived facts and history) while unifying them under the Retrieval Augmented Generation pattern. The core flow—load, split, embed, store, and retrieve—enables agents to ground answers in external, long‑lived stores, making semantic search and prompt engineering central to reliable, contextualized reasoning.

Building from first principles, the text moves from classic TF‑IDF and cosine similarity to modern embedding-based semantic search, storage in vector databases, and practical querying. It demonstrates how document indexing choices affect retrieval quality, emphasizing chunking strategies (size and overlap), token-based splitting, and the trade‑offs between sending whole documents to an LLM versus targeted augmentation. Using LangChain, the chapter assembles end‑to‑end RAG pipelines across loaders, splitters, embeddings, and vector stores (e.g., Chroma) to deliver scalable document Q&A and agent grounding.

The chapter then applies these patterns to agent platforms with knowledge and memory stores, showing how agents can attach a knowledge base to chats and how memories are captured, summarized, embedded, and retrieved. It maps cognitive notions to AI (sensory, short‑term/working, long‑term) and extends memory into semantic, episodic, and procedural forms using LLM functions for extraction and categorization. Finally, it introduces compression—clustering and summarizing memories/knowledge to reduce redundancy and improve retrieval—and offers guidance on when to compress, how often, and how multi‑level compression or multiple stores can further optimize agent behavior.

Memory and retrieval and the augmentation of the prompt using the prompt engineering strategies Use External Tools and Provide Reference Text.
The two phases of RAG: First, documents must be loaded, transformed, embedded, and stored, and then they can be queried using augmented generation.
Memory retrieval for augmented generation uses the same embedding patterns to index items to a vector database.
How cosine similarity is measured
The cosine similarity between selected document and document set
Embeddings in 3D, showing how similar semantic documents are grouped
Load, transform, embed, and store steps in storing documents for later retrieval
How the document would ideally be split into chunks for better semantic and contextual meaning
Adding a new knowledge store and populating it with a document
The embeddings and document query views
Enabling the knowledge store for agent use
Managing the knowledge store splitting and chunking options
How memory is broken down into various forms
Basic memory retrieval and augmentation workflow
Adding memories to a newly created memory store
Conversing with a different agent on the same memory store
How semantic memory augmentation works
Configuration for changing the memory store type to semantic
Comparing memories for the same information given two different memory types
The process of memory and knowledge compression
The interface for compressing memories

Summary

  • Memory in AI Applications differentiates between unstructured and structured memory, highlighting their use in contextualizing prompts for more relevant interactions.
  • Retrieval-augmented generation (RAG) is a mechanism for enhancing prompts with context from external documents, utilizing vector embeddings and similarity search to retrieve relevant content.
  • Semantic Search and Document Indexing converts documents into semantic vectors using TF-IDF and cosine similarity, enhancing the capability to perform semantic searches across indexed documents.
  • Vector Databases and Similarity Search stores document vectors in a vector database, facilitating efficient similarity searches and improving retrieval accuracy.
  • Document Embeddings capture semantic meanings, leveraging models like OpenAI's to generate embeddings that preserve a document's context and facilitate semantic similarity searches.
  • LangChain provides several tools for performing RAG, and it abstracts the retrieval process, allowing for easy implementation of RAG and memory systems across various data sources and vector stores.
  • Short-term and Long-term Memory in LangChain implements conversational memory within LangChain, distinguishing between short-term buffering patterns and long-term storage solutions.
  • Storing document vectors in databases for efficient similarity searches is crucial for implementing scalable retrieval systems in AI applications.
  • Agent knowledge directly relates to the general RAG pattern of performing question and answer on documents or other textual information.
  • Agent memory is a pattern related to RAG that captures the agentic interactions with users, itself, and other systems.
  • Nexus is a platform that implements agentic knowledge and memory systems, including setting up knowledge stores for document retrieval and memory stores for various forms of memory.
  • Semantic memory augmentation (Semantic Memory) differentiates between various types of memories (semantic, episodic, procedural). It implements them through semantic augmentation, enhancing agents' ability to recall and use information relevantly specific to the nature of the memories.
  • Memory and Knowledge Compression are techniques for condensing information stored in memory and knowledge systems, improving retrieval efficiency and relevancy through clustering and summarization.

FAQ

What’s the difference between “knowledge” and “memory” in AI agents?Knowledge is external information (documents, files, code, etc.) used to augment a prompt via retrieval. Memory is agent- or user-specific information (e.g., conversation history, facts, preferences) captured over time. Both rely on retrieval, but they differ in how the content is sourced and updated: knowledge is typically preloaded from documents, while memory is incrementally appended from interactions.
How does Retrieval-Augmented Generation (RAG) work?RAG has two phases. Indexing: load documents, split them into chunks, embed each chunk into vectors, and store them in a vector database. Querying: embed the user’s question, retrieve the most similar chunks, insert them as context into the prompt, and let the LLM generate an answer using that context.
Why is document splitting (chunking) important, and how should I choose chunk size and overlap?Chunking ensures only relevant parts of a document are retrieved and included in prompts. Proper chunk size preserves semantic coherence; overlap helps avoid cutting important context at boundaries. You can use LangChain’s RecursiveCharacterTextSplitter or token-based splitters; the optimal settings depend on the content structure and the LLM’s token limits.
What’s the role of embeddings and vector databases in semantic search?Embeddings convert text into high-dimensional vectors that capture meaning. A vector database stores these vectors and enables similarity search (e.g., cosine similarity/distance) to retrieve semantically related chunks. This outperforms keyword-only methods like TF-IDF for meaning-based retrieval.
What’s the difference between cosine similarity and cosine distance?Cosine similarity measures how aligned two vectors are, ranging from -1 (opposite) to 1 (identical). Cosine distance is 1 − cosine similarity, ranging from 0 (most similar) to 2 (most dissimilar). Many retrieval systems score by distance (lower is better) rather than similarity.
How do I build a basic RAG pipeline with LangChain?Follow the pattern: load documents with a loader (e.g., UnstructuredHTMLLoader), split into chunks (e.g., RecursiveCharacterTextSplitter or token-based splitters), embed chunks (e.g., OpenAI embeddings), store them in a vector DB (e.g., Chroma), then at query time embed the query and run similarity search to fetch top chunks for the LLM.
When should I use token-based splitting instead of character-based splitting?Use token-based splitting to align chunks with how LLMs process text and to remove irrelevant whitespace, improving semantic retrieval. Character-based splitting is simple and effective for many cases, but token-based often yields more consistent chunk sizes and better results for LLM-focused workflows.
How does Nexus support agent knowledge?Nexus provides a Knowledge Store where you upload documents that are then chunked, embedded, and stored in ChromaDB. You can query embeddings, configure splitter type/size/overlap, and connect a single Knowledge Store at a time to an agent for RAG-backed chat.
How is agent memory implemented and what types exist?Memory is captured from interactions and transformed via LLM “memory functions” into semantically relevant entries stored as vectors. Types include sensory (brief, modality-specific inputs), short-term/working (recent conversation context), and long-term (facts, preferences). Nexus supports basic memory stores and semantic memory variants (including episodic and procedural), which use additional semantic categorization for better recall.
Why and how should I compress memory and knowledge stores?Over time, stores can become redundant and noisy. Compression clusters related items (e.g., via K-Means) and summarizes them into concise representations, improving retrieval quality and efficiency. Apply compression when clusters are large/unbalanced or content is repetitive; periodic compression helps for memory, while knowledge often benefits most at initial load. Multiple compression passes can further improve retrieval.