4 The Session Service: Teaching your AI to remember

This chapter explains how to give an AI system durable conversation memory so follow-up questions, references, and tool-driven interactions remain coherent. It begins by framing sessions as structured records that belong to users and contain ordered messages, including not just ordinary user and assistant text but also system instructions and tool-call exchanges. Because a single turn can generate multiple internal messages, the service must preserve the full sequence rather than only the final answer, and it must do so in a way that supports concurrency, retrieval, and later context assembly.

To make this practical, the chapter defines a clean service contract for creating or reusing sessions, listing sessions for a user, adding messages in batches, retrieving history with pagination, and deleting conversations. It then separates persistence concerns behind an abstract storage interface so the platform can swap backends without changing application code. PostgreSQL is used as the concrete example: one table stores session metadata, another stores messages with relational links, indexing, cascading deletes, and JSONB support for tool-call details. The same pattern is carried through the gRPC service and SDK, so developers can call simple Python methods while the platform handles serialization, routing, and database access behind the scenes.

The chapter then expands beyond raw message storage into model-managed memory and context-window management. It introduces structured memories as durable key-value facts, suitable for critical information like allergies, preferences, or medications, and shows how the session service can be extended to save, retrieve, update, and clear those facts. Finally, it addresses the reality that conversation history often exceeds model limits, surveying strategies such as truncation, summarization, hierarchical memory, and retrieval across past sessions. The central idea is that effective AI systems need more than a chat log: they need layered, intentional context management that balances fidelity, compression, long-term memory, and application-specific tradeoffs.

The Session Service exposes five operations organized into two groups: session management handles session lifecycle, while message operations handle conversation content.

The Session Service delegates all persistence to a storage interface, allowing teams to swap backends without changing service logic or client code.

The Session Service architecture. The SessionServiceServicer class receives Protocol Buffer requests from the API gateway and delegates to the abstract SessionStorage interface, enabling different storage backends to be swapped without changing service code.

Model-managed memory flow showing how the assistant extracts and saves facts during conversation. The workflow loads existing memories, the model generates a response and optionally calls save_memory, and the Session Service persists the fact for future turns.

Memory accumulation across sessions. Session 1 captures the allergy and medication. Session 2 (a week later) adds the preferred pharmacy. Session 3 (months later) starts with all three facts already loaded.

Context window budget breakdown showing how a context window gets divided among system prompt, tool definitions, retrieved documents, model-managed memory, reserved response space, and conversation history. The conversation history portion is managed by the Session Service.

Simple truncation drops older conversations from the context window. Messages 1-30 are dropped and only messages 31-50 are kept. Message 1 contains "I have a penicillin allergy", which is now lost, and model loses that information when answering the user’s question.

Summarization compresses older messages while preserving key information. Instead of discarding messages 1-30 entirely, an LLM call compresses them into a summary that captures critical facts: the penicillin allergy, insurance verification, appointment preferences. Messages 31-50 remain verbatim. The model now sees the summary followed by recent messages, retaining the context that simple truncation would have lost.

Hierarchical memory combines three tiers of context. Model-managed memories store critical facts as structured key-value pairs. A rolling summary compresses older conversation history into a compact narrative. Recent messages remain verbatim to preserve nuance in the current exchange.

Retrieval-augmented memory searches past sessions for relevant context. When the patient asks about previous headache recommendations, the question triggers a search across their conversation history. Session #34 from March contains the relevant migraine consultation. That context gets pulled into the assembled context for the current session, clearly marked as coming from a previous conversation, allowing the assistant to answer accurately from months of history.

• A session represents a conversation between a user and an AI assistant, containing metadata about who owns it and when activity occurred, plus a list of messages that capture the full back-and-forth including tool calls, tool results, and assistant responses.
• The OpenAI message format has become an industry standard, representing each message with a role (user, assistant, system, or tool) and content, with additional fields for tool calls when the assistant invokes external functions and a tool call ID to link results back to their originating requests.
• A single user exchange might generate five messages internally (system prompt, user question, assistant tool call, tool result, assistant response), and all must be stored together for the conversation to remain coherent across turns.
• The Session Service exposes 5 gRPC operations that map directly to workflow needs: GetOrCreateSession handles both new and returning users in one call, ListSessions retrieves all sessions for a user so clients can display session history and resume past sessions, AddMessages records conversation turns in batches, GetMessages retrieves history with pagination, and DeleteSession handles cleanup.
• Storage abstraction separates the Session Service from any particular database, allowing teams to choose their storage backend.
• The SessionStorage abstract base class defines the contract any backend must fulfill, making testing straightforward with in-memory implementations, enabling gradual migration between databases, and futureproofing the platform for new storage technologies.
• Relational databases like PostgreSQL map naturally to session data: a sessions table holds metadata while a messages table stores conversation turns, with foreign keys ensuring referential integrity and CASCADE deletes automatically removing messages when sessions are deleted.
• The JSONB column type stores nested tool call structures (IDs, function names, arguments) without requiring additional tables, while the SERIAL primary key on messages provides natural ordering within a session without relying on timestamps that could theoretically collide.
• The SDK's SessionClient follows the same pattern established for other platform services: connect to the gateway, specify the target service in metadata, and implement methods that convert between Python objects and Protocol Buffer messages for gRPC transport.
• Model-managed memory extends the Session Service with the ability to store structured facts separately from conversation history, inspired by the MemGPT concept of treating the context window as RAM and external storage as disk.
• Memories are keyed by user ID (required) with an optional session ID, supporting both session-scoped patterns (short-lived context that expires when the conversation ends) and user-scoped patterns (long-term facts that persist indefinitely), with the platform providing primitives while applications decide policy.
• Context engineering addresses the reality that conversation history competes with system prompts, tool definitions, retrieved documents, and response tokens for limited context window space, often leaving only a fraction of the total budget for messages.
• Simple truncation keeps only recent messages that fit within budget, which works for transactional interactions like password resets but loses critical information from early in longer conversations regardless of its importance.
• Summarization compresses older messages into a condensed narrative using an LLM call, extending effective context by reducing dozens of messages to a few hundred tokens while preserving essential facts that truncation would discard.
• Hierarchical memory combines approaches into layers: model-managed memories for critical facts at the top, rolling summaries of older context in the middle, and verbatim recent messages at the bottom, with assembly order optimized for LLM attention patterns that favor context window beginnings and ends.
• Retrieval-augmented memory treats past conversations as a searchable knowledge base through the Data Service, enabling assistants to answer questions like "What did you recommend for my headaches last time?" by finding relevant context from previous sessions.
• The Session Service supports configurable retrieval strategies through the SDK, letting workflows specify truncation, hierarchical summarization, or retrieval-augmented approaches based on their application's conversation patterns and risk tolerance.

FAQ