4 Memory: What your agent remembers shapes what it becomes

Memory in an AI agent is not just storage; it is a policy for deciding what information survives boundaries, what returns to active context, and under what checks it may influence action. The chapter frames memory as a governed system that distinguishes state persistence, evidence retrieval, and experience capture, emphasizing that persistence alone does not create learning or authority.

To make this concrete, the chapter develops several memory patterns. Hierarchical Retention manages information across tiers with different scopes, lifetimes, and ownership; RAG Pipeline retrieves current evidence from large or changing sources while preserving provenance and permissions; Progress Tracking preserves the state needed to resume work safely after interruptions; and Failure Journals store failure evidence plus candidate lessons without automatically promoting them to rules. Across all of these, versioning, supersession, permissions, and validation matter more than raw recall or proximity to the model.

The chapter also compares these patterns to real systems and shows that production agents already use partial forms of them through scoped instruction files, project memory, checkpoints, and retrieval tools. Its central lesson is practical: start with the authoritative sources a task already has, add memory only when the boundary truly demands it, and measure success by better downstream decisions, fewer repeated failures, safer recovery, and correct source authority rather than by how much text the agent can keep.

Memory access-cost analogy. Promotion and eviction manage active context; authority, version, and permission remain separate checks.

Hierarchical Retention in two views. The three-tier teaching model shows promotion and eviction; the production view adds scope, function, lifetime, ownership, and admission.

RAG Pipeline from basic retrieval to governed compile and assemble paths. Provenance, version, permission, and source identifiers survive into the evidence package.

Progress Tracking from a checkpoint chain to verified resumption. Each checkpoint records the goal, artifacts, versions, effect receipts, and pending decisions.

Failure Journal from an immutable failure entry to governed reuse. An entry may support a candidate lesson; independent admission controls publication, and later retrieval and effect observation govern continued reuse.

Memory validity checks. A superseded game decision is rejected before active context; a current, authorized item may re-enter.

• Memory is a policy for what survives a boundary and under what authority it may return to a later decision. A separate memory service is optional.
• Hierarchical Retention (Memory × Hierarchy) separates scope, function, access, and lifetime. Recency and frequency can guide retrieval; they do not determine authority.
• RAG Pipeline (Memory × Chain) governs a compile path for sources and an assembly path for decision evidence. Exact, lexical, semantic, structural, relational, and temporal retrieval serve different corpora.
• Progress Tracking (Memory × Orchestrate) preserves the goal, acceptance condition, artifact evidence, authoritative versions, pending decisions, side-effect receipts, and handoff conditions needed to resume safely.
• Failure Journals (Memory × Loop) keep raw failure evidence, candidate diagnoses, and published lessons distinct. Chapter 7 owns admission and replay.
• The four patterns are selected from observed failures; they do not form a mandatory pipeline. A system may use one, several, or none.
• Memory and Collaboration may share infrastructure while retaining different responsibilities. Memory governs persistence and retrieval; Collaboration governs roles, handoffs, and integration.
• Measure memory through downstream decisions: retrieval utility, repeated mistakes, stale reads, duplicated effects, harmful transfer, latency, and token cost within a defined workload.

FAQ