By default, most agent frameworks scope memory to a single thread. Your agent can hold context across turns within a conversation, but the moment that session ends, nothing carries forward. If you want it to remember a user's preferences, pick up where a task left off, or learn from a past interaction, you have to explicitly extract that information and persist it to a long-term store. Without that step, every new session starts from scratch.
An agent memory platform is the infrastructure that makes cross-session persistence possible: it gives your agent a place to store what it learns, a way to retrieve the right context at the right time, and the mechanics to scale that across users.
Google's Gemini team found that model performance started degrading around 125K tokens, even with a 500K-token context window, so expanding context alone is not a substitute for real memory architecture.
This article breaks down how AI agent memory works, what types exist, how to build retrieval pipelines, and what to look for when choosing a platform.
What is AI agent memory?
AI agent memory is the infrastructure your agent needs to store information from past interactions and recall it during future ones. LLMs are stateless by default. Each API request is independent, with no built-in mechanism for remembering what happened five minutes ago, let alone five weeks ago.
Memory systems fill that gap. They persist information between interactions, compile relevant history at each step, and connect to external knowledge sources like vector databases and structured stores.
Why stateless agents fall short
If you've built a multi-turn chatbot on a raw LLM, you've already hit this wall. Your agent can't reference a prior conversation, track a user's preferences, or build on earlier reasoning. It treats every input as novel. Even with context windows exceeding a million tokens, you face practical limits: cost scales linearly with tokens, and model performance degrades well before you hit the theoretical ceiling.
125K tokens is roughly where Google's Gemini team observed performance degradation, even with a 500K-token context window. Stuffing everything into context is not a memory strategy.
How memory changes agent behavior
When you add memory to your agent, its behavior shifts from reactive to adaptive. A support agent remembers that a customer already tried the standard fix and skips to advanced troubleshooting. A coding agent recalls your team's naming conventions and applies them without prompting. A research agent builds on findings from previous sessions rather than re-reading the same sources.
Memory is what moves AI agents from low autonomy (stateless, single-turn) to medium and high autonomy, where they maintain context, retry failed tasks, and self-correct across long task horizons.
Types of AI agent memory
You'll likely need multiple memory types in production. Each serves a different purpose, and most agent memory platforms combine several.
Short-term memory (working memory)
Your agent's short-term memory (also called STM or working memory) holds the immediate context needed for the current task. When a user says "Find flights to Paris, then recommend hotels near the Louvre," short-term memory tracks each step's results to inform the next action.
Short-term memory is typically implemented as a rolling buffer or sliding window over the conversation. It provides fast access to current context but resets when the session ends. Think of it as the active thread of conversation: essential for coherence within a session, useless for anything beyond it.
Long-term memory
Long-term memory (LTM) persists across sessions, surviving restarts and letting agents build knowledge over weeks or months. If you want your agent to maintain persistent memory of a customer's preferences, store resolved issues, or accumulate domain knowledge, you need long-term storage.
Implementation usually combines a persistent store (a database or key-value system) with semantic search.
RAG is one common approach: your agent fetches relevant information from a knowledge base to enhance its responses. Vector embeddings make this possible by representing text as arrays of numbers that capture semantic meaning, so retrieval works even when the phrasing differs from what was originally stored.
Episodic, semantic, and procedural memory
Your production system may need more than the short-term and long-term split. Specialized memory types target different retrieval patterns:
| Memory type | What it stores | Typical use |
|---|---|---|
| Episodic memory | Specific past experiences with timestamps and outcomes | Log of what happened in a user session |
| Semantic memory | Factual knowledge independent of specific events | Product specs, domain rules, customer profiles |
| Procedural memory | How to perform tasks: workflow steps and decision logic | Agents updating their own behavior based on past performance |
Session memory and shared memory
Session memory persists within a session but resets between them, which is useful for tasks that span multiple turns but don't need permanent storage. Shared memory lets multiple agents or users share a common profile, so knowledge learned by one is available to all. This is especially powerful in multi-agent pipelines: a coding agent sharing a memory profile across your engineering team can stop making mistakes that have already been corrected by a teammate.
How agent memory works
Under the hood, an agent memory platform handles two core pipelines: ingestion (getting information in) and retrieval (getting it back out).
The ingestion pipeline
When your agent finishes a conversation or compacts its context window, messages flow into the ingestion pipeline. A well-designed pipeline does more than dump raw text into a database. It extracts structured facts, events, instructions, and tasks from the conversation history, deduplicates against existing memories, and stores the result in a format optimized for later retrieval.
The key stages are:
-
Chunking: split the conversation into segments that are small enough to embed meaningfully but large enough to preserve context.
-
Extraction: identify discrete facts, preferences, and decisions from the raw transcript.
-
Embedding: transform extracted text into vector representations using a model like OpenAI's embedding API or an open-source alternative.
-
Indexing: write the vectors and metadata into a store optimized for similarity search.
Idempotency matters here. If the same conversation is ingested twice, you don't want duplicate memories cluttering retrieval results. Content-addressed IDs or deduplication logic prevent this.
The retrieval pipeline
When your agent needs context, the retrieval pipeline finds and surfaces the most relevant memories. No single retrieval method works best for all queries, so production systems typically run several in parallel and merge results.
Common retrieval channels include:
-
Vector similarity search: finds semantically similar memories using cosine similarity.
-
Full-text keyword search: catches exact terms and phrases that embedding-based search might generalize away.
-
Structured key lookup: returns results where the query maps directly to a known topic or entity.
-
Graph traversal: follows entity relationships to surface connected facts that wouldn't match by similarity alone.
Results from each channel get fused using techniques like RRF (Reciprocal Rank Fusion), where each result receives a weighted score based on its rank within each channel. The top candidates then go to the model as additional context.
Knowledge graphs and temporal reasoning
A growing class of agent memory platforms stores memories not just as flat vector embeddings but as nodes and relationships in a knowledge graph. Knowledge graphs allow your agent to trace connections between facts, follow entity relationships across sessions, and reason about how information has changed over time. Temporal knowledge graphs add timestamps to each relationship, enabling queries like "what did this customer believe before their last support ticket?" and making it possible to track which facts are currently valid versus which have been superseded.
Frameworks and platforms for agent memory
Your choice of framework determines how much memory infrastructure you build yourself versus what you get out of the box. The options below take meaningfully different architectural bets, so the right choice depends on what your agents actually need to remember and how much control you want over how that happens.
Zep
Zep is built around a temporal knowledge graph architecture. Its core component, Graphiti, is an open-source engine that continuously ingests conversational and structured business data, extracting entities and relationships and storing them with bi-temporal metadata: when the fact was valid in the world and when it was ingested. In the Deep Memory Retrieval (DMR) benchmark, Zep achieves 94.8% accuracy vs. 93.4% for the prior best system, and on LongMemEval it delivers accuracy improvements of up to 18.5% while reducing response latency by 90% compared to baseline implementations. Retrieval stays under 200ms regardless of graph size.
Zep's hybrid retrieval combines semantic embeddings, BM25 keyword search, and graph traversal into a single context block dropped into the agent's prompt. When new information contradicts existing facts, Zep invalidates the old fact rather than duplicating it, preserving historical accuracy without large-scale recomputation. Zep is deployed in production at enterprise scale and is SOC 2 compliant, with options for cloud, BYOK, and VPC deployment.
Zep is the right choice when your agent needs temporal reasoning: tracking how facts about users or business entities change over time across sessions.
Letta
Letta (formerly MemGPT) takes a fundamentally different approach, modeling agent memory after an operating system. The agent's main context is RAM (what's in the prompt right now), and archival memory is disk (long-term storage the agent queries via tool calls). Letta agents autonomously decide what to page in and out, making the LLM itself the memory controller.
Letta organizes memory into three tiers: core memory (always visible, always in-context), recall memory (searchable conversation history stored outside context), and archival memory (a vector database of long-term observations). This architecture enables agents to operate over unlimited conversation histories without a fixed context budget. Letta's latest MemFS system is git-tracked and supports agent dreaming (asynchronous background memory consolidation). Letta Code, its memory-first coding agent, is ranked the number one model-agnostic open-source agent on Terminal-Bench as of 2026.
Letta is the right choice when you're building agent-native applications from scratch and want agents that genuinely control their own memory rather than relying on an external extraction pipeline.
LangMem
LangMem is the LangChain-native long-term memory SDK. It extracts semantic memory, episodic memory, and procedural memory from conversations, consolidates and deduplicates them over time, and can update agent behavior through prompt optimization, not just retrieve facts. LangMem integrates natively with LangGraph's BaseStore and is the sanctioned long-term memory layer in the LangChain 1.0 stack.
The procedural memory angle is genuinely differentiated: LangMem agents modify their own prompts based on experience, allowing behavior to improve with use. It supports any storage backend and works alongside LangGraph checkpointers for in-session memory. Its core API is framework-agnostic, though the tightest integration is with LangGraph. One production caveat: LangMem's p95 search latency on the LOCOMO benchmark is around 59 seconds, making it a poor fit for interactive agents that need sub-second recall. Use Mem0 or Zep for latency-sensitive workloads. LangMem is best for teams already building on LangGraph who want long-term memory without standing up separate infrastructure.
Cognee
Cognee is an open-source AI memory platform that builds a self-hosted knowledge graph from your data. Its cognify operation runs a six-stage pipeline: classify documents, check permissions, extract chunks, use an LLM to extract entities and relationships, generate summaries, then embed everything. The result is a unified graph combining vector embeddings, graph reasoning, and cognitive-science-grounded ontology generation, making documents both searchable by meaning and connected by relationships that evolve as your knowledge grows.
Cognee grew from roughly 2,000 pipeline runs to over one million in 2025, a 500x increase. Bayer uses Cognee to power scientific research workflows; the University of Wyoming built an evidence graph from scattered policy documents. Its fully local deployment is a genuine differentiator for teams with strict data residency requirements. Cognee raised a $7.5M seed in 2026 and is expanding its cloud platform. It's the right choice when you need an open-source, self-hosted knowledge graph memory that you fully control.
Supermemory
Supermemory is a memory API and context engine that packages fact extraction, user profile building, contradiction resolution, and automatic forgetting behind a single API. Its custom vector graph engine delivers sub-300ms retrieval latency, claiming to be 10x faster than Zep and 25x faster than Mem0 on internal benchmarks. Supermemory claims benchmark leadership on LongMemEval (81.6% vs Zep's 71.2%), LoCoMo, and ConvoMem, though these results are self-reported and have not yet been independently verified at time of writing.
Supermemory's automatic forgetting mechanism is notable: temporary facts expire after their relevant date passes, and contradictions are resolved automatically. It runs both RAG and memory together in the same query, handling knowledge base retrieval and personalized user context in a single call. Supermemory ships native plugins for Claude Code and OpenCode, making it purpose-built for coding agent workflows. Backed by seed investors including Google Chief Scientist Jeff Dean and Cloudflare CTO Dane Knecht, Supermemory is the right choice for teams who want the entire context stack (memory, RAG, user profiles, connectors) behind one API with minimal configuration.
Mastra: typed agent memory in a production TypeScript framework
If you're building AI agents in TypeScript, Mastra gives you typed memory primitives alongside the rest of your agent stack. As an open-source framework (Apache 2.0), it ships agents, workflows, RAG, evals, and observability in a single package. Rather than bolting on a separate memory service, you configure memory directly in your agent definition, keeping your stack cohesive and your types consistent across every layer.
Mastra's memory system handles both the thread-scoped and cross-session cases in one API. Within a session, the message history keeps the current thread coherent. Across sessions, semantic recall uses vector search to surface relevant context from any prior thread for the same user, so preferences, past decisions, and accumulated knowledge carry forward automatically. Observational memory goes further: background compression agents (an Observer and a Reflector) compact long conversation histories into dense, timestamped observations at 5 to 40x compression, achieving 94.87% on the LongMemEval benchmark without requiring a separate vector store. Memory processors compose together. Chain TokenLimiter and ToolCallFilter to control what enters the context window without writing custom middleware. Model routing across hundreds of models across dozens of providers through a single interface lets you run extraction on a smaller model and synthesis on a larger one, tuning cost against quality per stage.
Build your first TypeScript agent with persistent memory on Mastra.
Other options
CrewAI supports shared memory across agent crews with pluggable storage backends. LangGraph's BaseStore provides cross-session long-term memory natively for any LangGraph agent, complementing LangMem's extraction logic. For teams that want a complete agent runtime rather than a memory layer, Letta's agent harness approach offers the most control over what agents remember and forget.
Building agent memory systems
Designing a production agent memory platform means making concrete decisions about data structures, performance targets, and how you scope memory across users and sessions.
Choosing the right data structures
Your data structure choices should match your retrieval patterns. The table below maps each storage type to its role and typical tools.
| Storage type | Best for | Example tools |
|---|---|---|
| Vector database | Semantic similarity search across embeddings | pgvector, Chroma, Turbopuffer, Milvus |
| Key-value store | Fast lookups for known entities and session memory | Redis, DynamoDB |
| Event log | Immutable audit trail for episodic memory | Kafka, append-only SQL tables |
| Knowledge graph database | Entity relationships and graph traversal across connected facts | Neo4j, Kuzu, Amazon Neptune |
| Temporal knowledge graph | Tracking how facts change over time with validity windows | Graphiti (Zep), Cognee |
For most agent workloads, start with a vector database plus a key-value store and add complexity as your retrieval patterns demand it.
Performance targets and the latency/cost trade-off
For voice agents or real-time assistants, you'll typically want sub-100ms end-to-end retrieval. For batch processing or offline research agents, disk-backed storage may be perfectly adequate. In-memory storage delivers microsecond reads but costs more per gigabyte; disk-backed vector databases are cheaper but slower.
Index type matters at scale. HNSW (Hierarchical Navigable Small World) provides fast approximate search with high accuracy but uses more memory. IVF (Inverted File Index) clusters vectors into buckets and searches only the most relevant ones, trading some recall for better memory efficiency at very large scales.
Memory scoping and retention
You need to decide what your agent runtime retains and for whom. Per-user memory keeps each user's profile isolated; per-session memory resets between sessions; shared memory makes team-level context available to all agents. Retention policies that automatically expire old memories reduce your data surface and help with GDPR and SOC 2 compliance. Build deletion capabilities early rather than retrofitting them.
Real-world use cases for agent memory
If your agent can't remember the last conversation, every interaction starts from zero. Memory is what separates a stateless chatbot from a production agent.
Customer support and personalization
Your support agent remembers that a customer already tried the standard troubleshooting steps and escalates directly. It recalls their subscription tier and adjusts its tone and available actions accordingly. Over time, it learns which resolution paths work best for specific issue categories. The result is measurably better support outcomes: Spencer Jones at MedTechVendors reported a 65% improvement in match accuracy after adding persistent memory to their procurement agents.
Long-running research and coding agents
Multi-agent systems where one agent plans and another writes code need memory that persists across compaction events. A coding agent that remembers your project's architecture, the bugs it fixed yesterday, and the coding standards your team agreed on produces far better output than one working from a blank context. Claude Code runs autocompact at 95% of context window capacity, automatically summarizing interactions. The key insight is that compaction without memory is data loss; compaction with memory is context management.
Multi-agent pipelines that share context
When your agents collaborate on a task, shared memory prevents them from producing conflicting output. Without context sharing, a subagent has no idea why a prior step made the choices it did. With shared memory, each agent sees the full reasoning trace and can carry intent forward. The most robust multi-agent architectures share full context at agent boundaries rather than passing compressed summaries.
Observability and debugging agent memory
Your agent's memory system is only as good as your ability to inspect it. When responses degrade, you need to trace whether the problem is in retrieval, in the stored data, or in how the model used the context it received.
Tracing what the agent retrieved and why
You need visibility into which memories your agent retrieved for each response and which retrieval channel surfaced them. A trace should show the query, the retrieval results with scores, and the final context assembled for the model. Standard tracing formats like OpenTelemetry work well here. Each retrieval call becomes a span in your trace tree, showing latency, the query embedding, and the returned results.
Evals for memory relevance and staleness
You can't manually review every retrieval result. Instead, set up evals that measure whether retrieved memories are relevant to the query and whether stale information is being surfaced over fresh data. The LongMemEval and LoCoMo benchmarks are the de facto standards for evaluating memory system accuracy on long conversational histories. An LLM-as-judge pattern also works well for production evals: pass the query, retrieved context, and the agent's response to a second model with a rubric.
Guardrails and data-privacy considerations
Your agent's memory may contain sensitive information: customer data, internal decisions, personal preferences. You need guardrails that control what gets stored, who can access it, and how long it persists. Custom memory processors can filter for prompt injection artifacts and sensitive fields before they enter the memory store. Compliance requirements (GDPR, SOC 2) may dictate that users can request deletion of their memory profile.
Choosing an agent memory platform: key evaluation criteria
When you're evaluating an agent memory platform, focus on the characteristics that determine whether the system holds up under production load. The three dimensions below cut across every option.
| Criterion | What to measure | Why it matters |
|---|---|---|
| Latency, scalability, and consistency | Retrieval latency under realistic load; how performance degrades as the index grows; eventual vs. strong consistency model | Stale reads cause real problems for agents making decisions based on retrieved context |
| Data ownership and compliance | Whether you own the data, can export it, and what happens if you switch platforms; compliance certifications relevant to your industry | Memory lock-in is more costly than compute lock-in. It represents months or years of learned context |
| Integration surface | Language and runtime support (TypeScript, Python, both); compatibility with your existing vector database; deployment model (serverless, containers, edge); trace format for your observability stack | The best platform is the one that disappears into your architecture rather than becoming a separate system you have to manage |
Memory is what separates an agent that answers questions from one that learns. Whether you're building a support agent, a coding assistant, or a multi-agent pipeline, the architecture decisions you make here determine whether your system compounds knowledge or resets on every call. Start with short-term and long-term memory, add retrieval channels that match your query patterns, and instrument everything so you can trace what your agent remembers and why.
