Overview
6 Working with memory and knowledge RAG for agents
The chapter explains how agents can go beyond fixed model knowledge by retrieving relevant information from external stores and adding it to the prompt as context. It frames retrieval as the bridge between raw data and useful augmentation, covering both knowledge sources such as documents, databases, and structured logs, and memory sources such as past conversations, user preferences, and prior agent experiences. The core idea is that agents become more capable when they can search for what matters, bring it into context, and answer in a grounded way.
A major focus is the retrieval augmented generation pattern and the mechanics behind it. The text walks through the two stages of RAG, ingestion and retrieval, and shows how text is chunked, embedded into vectors, stored in a vector database, and later matched against user queries with similarity search. It contrasts simple keyword matching with semantic search, uses TF-IDF and cosine similarity as approachable examples, and then moves to modern embeddings and vector stores such as Chroma to show how semantically related content can be found even when the wording differs.
The chapter also emphasizes that practical systems rarely rely on vector search alone. Because semantic retrieval can miss exact codes, ambiguous terms, fresh facts, or relational questions, real agents often combine vector search with keyword search, relational databases, or graph databases in hybrid designs. It extends the same logic to memory, showing how agents can store and retrieve short-term, long-term, semantic, episodic, and procedural memories, then manage them with MCP tools, compression, and forgetting. Overall, the chapter presents memory and knowledge as complementary retrieval problems and argues that robust agents are built by combining multiple retrieval strategies with strong grounding rules.
The process of retrieval for both memory and knowledge, and how it combines with augmentation.
RAG can be broken up into two phases, ingestion and retrieval: first, documents are loaded, transformed, embedded, and stored into a searchable datastore, and, second, they can be retrieved using query mechanisms as part of context augmentation
Memory retrieval for augmented generation uses the same embedding patterns to index items to a vector database.
Cosine similarity search measures the angle/difference between vectors in any dimensional space
The cosine similarity between selected documents and the document set
Embeddings in 3D, showing how similar semantic documents are grouped
The top of figure shows how a user query could be converted to a hybrid search by decomposing the query into keywords and semantic meaning, The bottom figure shows how an agent could use the same search mechanism but as a set of tools to perform a semantic (vector) and keyword query.
How memory is broken down into various forms, from sensory (visual, auditory, and touch) to short term conversational memory and long-term episodic, semantic and implicit/procedural memories
Basic semantic memory retrieval and augmentation workflow. At the top of the figure memories are remembered by being semantically encoded in a vector database. Bottom of figure we can see the basic RAG process of retrieving and augmenting agent context.
graph diagram examples of the Back to the Future script. Top of the figure explains the relationship between entities/nodes. Below we see a simple node relationship graph between the various entieis from the movie Back to the Future. Where types of entities like people are grouped together and locations are grouped separately.
Semantic memory augmentation works by feeding previous experiences into an LLM that then converts the experience into an augmented memory of questions a user may ask that will activate that memory in a future retrieval step.
Memory and knowledge compression works by semantically clustering similar memories/knowledge into groups, it then summarizes these groups into a new single memory for each group. Then each single group memory is entered into an updated vector database..
Summary
- Retrieval is the bridge between raw data and an agent’s reasoning: RAG pipelines turn external text (or other modalities) into prompt-ready context, enabling agents to answer questions with up-to-date, task-specific knowledge.
- Vector similarity search is the semantic workhorse, but by itself it misses exact matches, jargon, numbers, and fresh updates—plan to combine it with keyword, relational, or graph queries for reliable coverage.
- Vector databases (e.g., Chroma, Pinecone) store high-dimensional embeddings and let agents run millisecond-scale similarity look-ups; they are essential infrastructure for any production RAG or memory system.
- Hybrid search (vector + keyword) offers the best of both worlds: lexical precision and semantic recall. Reciprocal-rank fusion or agent-directed tool chaining merges the two result sets into a single, ranked context.
- Typed tool interfaces (via the OpenAI Agents SDK or MCP) let agents decide when to call semantic, keyword, SQL, or graph tools—giving them true agency while keeping each search cost-controlled and observable.
- Memory and knowledge are siblings, not twins: “knowledge” is usually static document context, while “memory” evolves from conversations and agent actions; both ride on the same retrieval primitives.
- Cognitive analogues map cleanly to implementation layers: sensory memory ≈ short-lived embeddings, working memory ≈ conversation buffer or blackboard, long-term memory ≈ vector/graph/SQL stores.
- Graph memory adds relational reasoning—entities, edges, and observations—so an agent can answer multi-hop questions (“Who reports to Alice’s manager?”) that pure embeddings cannot resolve.
- MCP servers abstract memory back-ends: plug in Chroma for semantic recall, a knowledge-graph server for relationships, or both for a hybrid memory agent without rewriting core agent code.
- Compression and forgetting keep stores lean: cluster similar chunks and summarise them, or prune stale items, to control token costs and prevent retrieval noise as memories grow.
- Grounding is non-negotiable—always require agents to cite retrieved passages; it prevents hallucinations and pins answers to verifiable sources.
- In practice, production-grade knowledge/memory agents mix multiple retrieval modes, strongly type their tool calls, trace every interaction, and apply periodic compression—yielding context-aware systems that stay accurate, efficient, and scalable over time.
AI Agents in Action, Second Edition ebook for free
