7 Memory
In this chapter, the framework is extended with memory so LLM agents can benefit from past task executions instead of starting over with a blank slate each time. The central idea is episodic memory: storing completed experiences, then recalling relevant ones later to improve efficiency and accuracy. This is framed as a practical response to the fact that LLMs are stateless and need memory injected through text inputs if they are to learn from prior runs.
The chapter defines memory narrowly as records derived from an agent’s own experiences, distinguishing it from broader notions such as user preferences, general knowledge, RAG, skills, or model weights. It separates short-term working memory from long-term episodic memory, with working memory already represented by the rollout carried across steps inside a single task, and episodic memory stored persistently across tasks. To support this, the chapter introduces an Episode object as the unit of memory, a BaseMemoryStore interface for persistence and recall, and a concrete vector-backed store using Qdrant for similarity search.
Finally, the chapter wraps memory into a new Memory core type and wires recall and record operations into the agent lifecycle. Memory can enrich episodes with custom metadata before storage, and recipes are provided for common patterns such as recency-based, similarity-based, and reflective memory. The LLMAgent and TaskHandler are then updated so agents load recalled memories at the start of execution and save new episodes at the end, enabling both individual and shared memory designs in multi-agent systems. The result is an agent framework that can reuse prior experience, avoid redundant work, and produce better outcomes over time.
We’ll now give our LLM agents memory so that they can learn from past experiences to perform future tasks more effectively and efficiently.
Memory records and recalls Episode objects through its BaseMemoryStore backend. LLM agents use these operations within its task execution lifecycle.
The UML class diagram of Episode and EpisodeFormatMode
UML class diagram of BaseMemoryStore, our new standard interface for memory data backend
Recording an Episode as a point in the embedding space and recalling the most similar Episode objects to a given query
The UML class diagram of QdrantMemoryStore which integrates into the qdrant-client Python library
UML class diagram for our new core type, Memory
The necessary updates to LLMAgent and TaskHandler to support episodic memory
Summary
- LLM agents need memory of past experiences to perform future tasks more efficiently and accurately
- LLMs are stateless which means memories need to be provided at inference time within prompts or in the LLM’s context window
- Working memory is the short-term memory that is used to maintain the task execution state, which we already implemented in Chapter 4
- Episodic memory is a form of long-term memory that records an LLM agent’s task execution and its result
- The choice of recall method and memory storage are tightly coupled (e.g. recalling with semantic similarity requires a vector-based storage)
- Deciding what gets stored into memory is an important design decision
- Episodes are naturally recalled at the start of a task execution and recorded at the end
- Episode is our new unit of memory
- Memory is our new core type responsible for recalling and recording Episode objects into its backend BaseMemoryStore
Build a Multi-Agent System (from Scratch) ebook for free