A Guide to Agent Memory: Between RAG, Context, and New Frameworks

In 2024, the conversation around language model memory played out almost entirely on a single axis: RAG versus long context window, as if one of the two had to win. In 2026, that question has become almost naive. Those who design agents today don't choose a single technology—they compose a stack, bringing together external knowledge retrieval, immediate context, skills internalized within the model, real-time operating system access, and some form of continuity between sessions. The question is no longer "which technique will we use?" but "which level of memory does which part of the problem need, and how much does making the wrong choice cost?"

This guide attempts to map the options available today, with their pros, their cons, the cases where they shine, and those where they are a mistake. It is not a ranking—it is more like a toolbox: some tools are screwdrivers, others are hammers, and using a hammer to drive a screw only produces frustration.

Classic RAG: Origins and Cracks

Retrieval-augmented generation is often described as a native invention of the language model era, but that is a simplification. Retrieval and generation, query refinement, and answer verification were already core research topics in information retrieval and question answering long before transformers, as recounted in this historical reconstruction of RAG. Large models added a fluid linguistic interface on top of an architecture that document computing had known for a long time—much like Toy Story added a friendly face to decades of computer graphics research.

The basic mechanism remains simple to describe: a corpus is broken into chunks, chunks become vectors in an embedding space, a query is projected into the same space, the most similar chunks are retrieved and injected into the prompt. Advanced variants add hybrid search, reranking, query decomposition, and especially two techniques that have become almost a de facto standard. The first is Self-RAG or Corrective RAG, where the model evaluates the relevance of what it has retrieved before generating, rather than blindly trusting the similarity score. The second is Contextual Retrieval introduced by Anthropic, which generates a brief contextual summary for each chunk before indexing it, so an isolated piece of text carries information about which document it belongs to and what it is about, significantly reducing retrieval failures when combined with reranking.

However, the limits remain structural. Splitting a document into fixed chunks risks separating a question from its answer or splitting reasoning in half. Vector similarity is not the same thing as relevance for a specific task; two sentences can be semantically close yet completely useless for the question asked. Retrieving top results does not guarantee multi-hop reasoning, and the knowledge base is treated as essentially static—every update requires re-indexing. Above all, classic RAG has no notion of continuity between different sessions; it retrieves knowledge about the world, not memory of the user. It makes sense for technical documentation, company policies, medium-to-large knowledge bases where tracking the source is important. It is not enough for highly relational domains or for agents that need to remember who is in front of them.

Long Context: The Myth of Megatokens

Models with huge context windows have tangibly changed what is possible without retrieval pipelines. By mid-2026, Claude Sonnet 5 offers one million tokens of native context, Gemini 3.1 Pro reaches up to two million tokens in production, and GPT-5.5 sits at one million. Numbers that, just two years ago, sounded like laboratory science fiction.

The advantage is architectural simplicity: no vector store needed, no retrieval pipeline required—you upload the text and query the model. For tasks requiring seeing everything together, like summarizing a long document or reasoning over a compact codebase, long context can beat RAG precisely because it does not introduce the information loss of chunking. The flip side is cost and latency: processing a million tokens on every query costs orders of magnitude more than a targeted retrieval, and inferences on huge contexts can take tens of seconds compared to the near-instantaneous response time of a well-designed RAG. Then there is the phenomenon known as "lost in the middle", where even with a vast context, the model's attention is not uniformly distributed, and information buried in the middle risks being underutilized—a problem thoroughly discussed in this analysis of agent memory architectures. Finally—and this is the point that often gets overlooked—a long context window is a single-turn property; it gives no continuity between different conversations nor does it manage user state over time, as clearly explained in this comparison between RAG and large context windows. It makes sense for small, well-defined corpora, rapid prototyping, and global reasoning over a single document. It becomes an anti-pattern in high-volume enterprise applications, where the bill at the end of the month can be brutal.

Fine-Tuning and Continual Pre-Training: Skills, Not Facts

Adapting the model directly, through full fine-tuning or continual pre-training, serves to internalize style, format, and domain conventions—not to store facts that change every week. A model trained on legal language learns how to write an opinion, not the updated content of every regulation. The advantage is that once trained, no runtime retrieval is needed for those specific skills, and on repetitive tasks with a fixed format—like extracting entities according to a precise schema—fine-tuning often beats few-shot prompt solutions, as illustrated in this overview of RAG alternatives.

The downside is sharp: it is not a dynamic knowledge base, requiring curated datasets, training infrastructure, and continuous evaluation; if the domain evolves, the model must be retrained or risks becoming quietly obsolete. It makes sense for skills that remain stable over time; it makes no sense when the primary value lies in factual knowledge that constantly changes.

Agentic Tool-Use: Memory Lives Elsewhere

There is an entire category of problems where the right answer is not "remember better" but "don't trust memory and go check". An agent calling a ticketing API, querying a CRM, or executing a database query is not reading a static corpus; it is asking the owner of truth in real time, as described in this guide on production context retrieval. The advantage is obvious: data is always up to date, access policies apply at the source system level, and hallucinations about dynamic facts drop because the model isn't guessing—it's reading.

The cost lies in integration complexity: input/output schemas, error handling, rate limits, and authentication are required, and latency depends on systems the agent doesn't control, so a chain of multiple calls can slow everything down significantly. In 2026, much of this integration happens through the Model Context Protocol, a standard that has made connecting agents to external tools without reinventing the interface much simpler, and has also become the way memory frameworks described later expose themselves to agents. Tool-use makes sense for critical knowledge in transactional systems and for tasks requiring actions, not just answers. It is a mistake when the domain is mainly document-based and static, or when external APIs offer no reliability guarantees.

GraphRAG: Knowledge as a Network

Instead of retrieving only text snippets, one can build a graph of entities, relationships, and attributes, performing retrieval over nodes and paths. The practical difference is evident in questions like "which projects used a certain model and had security incidents?", where traversing multiple relational hops is required—something text similarity alone cannot catch, a point elaborated in this guide to advanced RAG techniques in 2026 and in my previous article on building auto-generated wikis for LLMs. Independent research presented at industry events showed substantial gains in factual accuracy when moving from pure vector retrieval to a searchable graph, as recounted in this analysis on agent memory beyond vectors.

The graph makes explicit relationships that text leaves implicit, allowing structured queries beyond semantic similarity. Building and maintaining it, however, is labor-intensive: extracting entities and relationships, cleaning, aligning ontologies, and monitoring a complex pipeline where symbolic and neural components must coexist, as noted in this guide to graph databases for AI agents. Not every domain lends itself to graph modeling; often a hybrid between graph and pure text is required. It works well for corporate organizations, product/component networks, and causal incident chains. It is a waste of resources on predominantly narrative or unstructured corpora.

Context Caching: Speed on Stable Content

When the same corpus is reused repeatedly, rebuilding the context from scratch on every call is an obvious waste. Major platforms have responded with concrete, non-theoretical solutions. Anthropic's prompt caching allows marking stable prompt blocks—like system instructions or large documents—and reusing them across subsequent requests at a fraction of the cost, with cache reads priced as low as one-tenth of normal input tokens and significant latency reductions on long documents. Google offers an analogous mechanism with context caching in its Gemini API, designed to avoid paying repeatedly for processing entire codebases or multi-hour meeting transcripts.

The benefit is concrete on heavily reused contexts: an internal assistant operating on fixed manuals or a stable codebase gains immediate advantages in latency and cost. The limitation is equally clear: this technique offers little help on frequently changing knowledge or highly heterogeneous sessions, and introduces non-trivial cache invalidation and consistency management.

Taxonomy of Agent Memory

Here we enter the specific territory of agents that must act as though they remember something, not just as though they know something. Memory in the immediate context—the last few turns of the current conversation—is instantaneous and has zero infrastructure cost, but lasts only for the ongoing session and easily loses information if the window is small. Episodic memory summarizes past sessions, salient events, and decisions made, granting cross-session continuity at the cost of compression that inevitably loses detail. Semantic memory resembles RAG but is designed as long-term agent memory—great for specific facts, weak on complex relationships unless paired with a graph. Procedural memory, often realized via fine-tuning or explicit agent configuration, captures reusable action patterns and skills, but is unsuitable for changing facts—a framework well mapped in this overview of vector, graph, and episodic memory architectures and in this comparison between personal memory and retrieved knowledge.

The general benefits of this layer are cross-session continuity and personalization, but they carry specific risks—in particular what might be called "toxic memory": obsolete or incorrect information that remains stuck and quietly degrades agent behavior over time. It makes sense for agents interacting repeatedly with the same users; it should be kept minimal or avoided in casual chatbots or under strict privacy constraints.

The Landscape of Real Frameworks

The theoretical taxonomy above takes concrete shape in a handful of projects that anyone working on agents in 2026 knows by name. Letta, direct heir to the MemGPT project, popularized the idea of an agent with a core memory of a few hundred tokens that the model reads and writes itself via tool calls, plus a much larger archival memory outside the context window, paged in and out as needed, as described in this comparison of major agent memory frameworks. Mem0 focuses on integration simplicity—a plug-and-play memory layer oriented toward user preferences and sessions, with retrieval based on multiple combined signals.

Zep, built on top of the open-source Graphiti engine, adopts a different and rather elegant approach: it treats every message as a source of facts about entities and relationships, and every fact as a temporally bounded edge in the graph, with a validity window that allows invalidating outdated information rather than overwriting it or letting it contaminate future responses. Thoughtworks' Technology Radar promoted Graphiti to the trial stage in April 2026, citing benchmarks reporting accuracy improvements of around 18.5 percent and latency reductions near 90 percent compared to traditional GraphRAG, as read in this tech radar profile. Cognee follows a logic closer to pure graphs, with a pipeline for remembering, retrieving, forgetting, and improving designed for documents, entities, and codebases.

Alongside these are more niche yet technically relevant projects. EverMind, with its EverMemOS, proposes a dual-layer architecture between working memory and long-term memory as a dynamic knowledge graph, and published research—which I discussed in a May 2026 article—on a mechanism called sparse attention memory, designed to efficiently manage contexts up to one hundred million tokens. Graphify, meanwhile, focuses on memory for programming assistants, converting code, docs, commits, and even browser screenshots into a single navigable graph, updating only nodes that actually changed instead of re-indexing from scratch, as discussed in my June 2026 article.

Anyone navigating these names would do well to start from the problem rather than project popularity. If you need a runtime where memory is the agent's core, Letta is the natural benchmark. If the problem is rapid personalization with minimal integration, Mem0 remains the pragmatic choice. If changing facts over time and their historical validity matter, Zep with Graphiti is likely the most mature answer available today. If the domain is intrinsically a graph—documents, entities, codebases—Cognee or Graphify warrant serious evaluation.

Hybrid Stacks: The Synthesis That Works

None of these techniques, taken alone, covers the full spectrum of a serious agent's needs. The technical community in 2026 converges on this idea quite solidly: long context should be treated as a complement to RAG and memory, not as their replacement, according to findings in this analysis of hybrid AI agent architectures and this comparison between long-term memory and long-context models. Comparative tests across different architectures—pure RAG, long context, memory files, and hybrid approaches—showed that the combination achieves a very high recall rate at a significantly lower cost than long context used alone, a data point cited in this practical guide to memory architectures, though one should always verify the specific conditions of each benchmark before blindly applying them to a use case.

A reasonable pattern for sizing the stack starts from corpus size. For fifty to five hundred pages, RAG to filter relevant passages plus long context to process them together works well. Beyond five hundred pages, or on continuously growing corpora, RAG (optionally graph-enhanced) becomes almost mandatory, and long context should be reserved solely for retrieval results. In a typical enterprise agent, RAG for docs/policies, tool-use for operational systems, episodic/semantic memory for users, and a graph for complex product/service/incident relationships coexist, as described in this guide on context engineering. A technical copilot might combine RAG on documentation, a graph for component dependencies, and caching on stable sections. A tutoring agent might combine episodic memory for the student, RAG on course material, and light fine-tuning for pedagogical style.

Comparison Table

Practical Guidelines for Agent Designers

Before choosing an architecture, it pays to answer a few simple yet frequently overlooked questions: How large is the corpus? How frequently does knowledge change? Is continuity required across returning users, or can each conversation start from scratch? What are the real constraints on cost, latency, and privacy? Are multi-hop reasonings across interconnected entities required, or are targeted answers sufficient?

From these answers, an initial stack draft emerges almost mechanically: a small, stable corpus suggests long context or light RAG; a medium or large corpus demands RAG (optionally graph-enriched), with long context reserved for retrieval outputs; returning users require an episodic or semantic memory layer; real-time operational data calls for tool-use on external systems; complex relationships justify investing in a graph. None of these choices is permanent—and that is precisely the hardest point to accept for those coming from a world of single, definitive solutions: a serious agent's memory is built in layers, reviewed over time, and must be treated as a piece of infrastructure to be maintained just like the code surrounding it, rather than a feature that can be forgotten once implemented.