6 Adding memory to your agent
Memory turns an agent from a stateless responder into something that can preserve progress, recover from interruptions, and adapt over time. This chapter frames memory as a practical engineering problem: without it, complex runs can overflow the model’s context window, conversations break across separate calls, and useful facts disappear between sessions. To solve that, the chapter separates memory into short-term, session-bound context and long-term, cross-session knowledge, then connects both to broader context-engineering ideas such as reducing, writing, and retrieving information at the right time.
For short-term memory, the chapter focuses on managing context during execution with a hierarchical strategy. It starts with simple sliding windows, then moves to token counting so decisions are based on real prompt size rather than message count. When context still grows too large, compaction replaces bulky tool call inputs and outputs with lightweight references, and summarization compresses older reasoning into a compact synthetic record. These techniques are applied through callbacks before LLM calls, with the general rule being: do nothing if context is small, compact first if possible, and summarize only as a last resort.
The second half extends memory across sessions by introducing a Session and SessionManager for persistent conversation continuity, plus an asynchronous human-in-the-loop flow that can pause on sensitive tool calls and resume later with stored state. Finally, the chapter builds long-term memory using structured extraction and vector search: important execution histories are distilled into typed records, checked for duplicates, stored in ChromaDB, and later retrieved to inform new runs. Together, these patterns let the agent remember recent work, continue interrupted conversations, safely request approval, and accumulate reusable knowledge over time.
Book structure overview: Chapter 6 in focus.
Without Session (left), each run() creates a new ExecutionContext that's discarded after completion. With Session (right), multiple run() calls share the same Session, maintaining conversation continuity.
Long-term memory enables information sharing across sessions. Each run() creates an ExecutionContext, multiple runs form a Session, and important information from sessions is extracted and stored for future use.
Context management strategy decision flow.
Relationship between session and ExecutionContext.
Internal features vs external extensions in agent architecture.
Overall flow of the pause and resume mechanism.
The role of MemoryManager.
Summary
- Memory transforms an agent from a stateless tool into an intelligent assistant that learns and adapts across interactions.
- The three usage patterns address different challenges: Pattern 1 (One-off Execution) manages context within a single run, Pattern 2 (Continuous Execution) maintains conversation across multiple runs, and Pattern 3 (Long-term Memory) preserves knowledge across sessions.
- Context management strategies follow a hierarchical approach: measure tokens first, apply Compaction for tool results, and use Summarization as a last resort. The key principle is separating storage from presentation—ExecutionContext remains immutable while LlmRequest is optimized for each call.
- Session and SessionManager enable multi-turn conversations by persisting events and state across run() calls. The same architecture supports asynchronous Human-in-the-Loop workflows, where execution pauses for user approval and resumes when confirmation arrives.
- Long-term memory combines Write and Retrieval strategies: structured information is extracted from execution history, checked for duplicates, stored in a vector database, and retrieved when relevant to new tasks.
- Memory retrieval can be implemented as an explicit tool the agent calls on demand, or as automatic injection via process_llm_request() that adds relevant context before every LLM call.
Build an AI Agent (From Scratch) ebook for free
