Inside agent memory

Four words, one confusion

"Add memory to the agent" is the most common request in agent work and one of the least precise. Depending on who says it, it means a bigger context window, a task list that survives a crash, a file of facts about the user that carries over to next week, or a search across every past conversation. Those are four different things. They live in different places, are written by different parties, cost different amounts, and fail in different ways. Building a good one starts with refusing to call them by the same name.

One fact underneath all four: the model remembers nothing between calls. Every request to a language model is a fresh function of the text you send it, and when the reply comes back the model has kept nothing. Whatever "memory" an agent has is therefore on your side of the API, built out of text you store and text you send. The four words are four different ways of doing that.

The next four sections take one row each. Section 06 puts them back together in a single picture, and the last two sections cover what goes wrong when the rows are merged and the one decision each of them needs.

Working memory

Working memory is whatever is in the context window on this call: the system prompt, the tool definitions, the conversation so far, the tool results, and anything retrieved or recalled for this turn. It is the only thing the model can use. If a fact is not in the window, then for the purposes of this call it does not exist, however carefully it was stored somewhere else. What's in an agent's context? walks through what fills the window and what each part costs; this section is about its role.

Two properties make working memory different from the other three. It is rebuilt every turn: the model does not carry it from call to call, the harness re-sends it, and what gets re-sent is a choice. And it is bounded: the window has a size, everything in it is paid for on every turn, and past a certain fill the model's attention degrades before the hard limit arrives - Anthropic's context engineering guide calls this context rot. Both properties point at the same conclusion: working memory is not a place you keep things. It is a place you show things.

Context is a rendering of state, not the state itself.- the habit this article is trying to install

That framing is the single most useful idea in agent memory design. The harness holds what the run knows, in whatever form suits code; each turn it renders a view of that into text - the recent transcript in full, older turns as a summary, the plan as a checklist, three recalled facts as a short block - and sends the view. Change the rendering and the model sees something different without the run forgetting anything. Confuse the two and every compaction is a lobotomy.

The confusion to clear here is the most common one of all: a bigger window is not more memory of any other kind. A million-token window raises the ceiling on working memory and nothing else. State still has to be kept somewhere exact, long-term memory still has to be stored somewhere durable, and retrieval still has to find the right thing - because you still cannot afford to put everything in, and because the model reads a crowded window worse than a curated one.

State

State is the structured information the application maintains about the current run: the goal, the plan and which steps are done, the turn count, the spend so far, which tools have been approved, the id and locale of the user, the results of expensive calls you do not want to repeat, the last checkpoint. It is typed data owned by code. The model never sees the state object. It sees whatever the harness renders from it.

Three reasons to keep state separate from working memory, even though some of its contents also appear in the window:

• It is exact. A budget check should read a number, not parse a transcript. A "which files have I changed" question should hit a list, not a search over forty tool results. Anything the harness has to branch on belongs in state.
• It survives compaction. When the transcript is summarised to make room, detail is lost by design. The state object is untouched, so the plan, the counters, and the facts the run depends on stay complete no matter how much of the conversation has been folded away.
• It serialises. Persist the state after every turn and a run can be resumed after a crash, paused for a human's approval and picked up hours later, or replayed for a post-mortem. Inside an AI agent covers the recovery side; none of it works if the only record of the run is chat.

State is written mostly by the harness. The model can change it too, but only indirectly: an update_plan or mark_done tool is the model proposing a state change and the harness applying it, with validation in between. Frameworks give the object different names - graph state, session, thread - and the idea is the same in each; AI agent frameworks compares how they expose it.

Its lifetime is the run. When the run ends, state either disappears or is promoted: the parts worth keeping are written into long-term memory by an explicit step - "this user prefers TypeScript", "the deploy target is eu-west-1" - not by dumping the whole object into a store. The mistake this section exists to prevent is treating the transcript as state. The transcript is a log of what was said. State is the current truth. They diverge the moment anything is compacted, and the model cannot tell you which one it is looking at.

Long-term memory

Long-term memory is information persisted across runs and sessions, outside any window, in a store: what the agent should still know next week. Cognitive science lends three loose categories, and they map cleanly onto what agents actually keep.

The store itself is ordinary infrastructure: a Markdown file the agent edits, a key-value table, a database, a vector index over past episodes. The designs worth knowing span a decade of ideas in a few years. Generative Agents kept every observation in a memory stream and scored what to recall by recency, importance, and relevance. MemGPT treated the window as RAM and the store as disk, with the model paging facts in and out through tool calls. Anthropic's memory tool gives Claude a directory of files it reads and writes across conversations. Claude Code's memory files do the same for a codebase. Different shapes, one pattern: text outside the window, brought back in when relevant.

Who writes long-term memory matters more than where it lives, because each writer fails differently:

• The person writes it explicitly - a CLAUDE.md, a "remember that I..." - and forgets to update it. Human-written memory is the most trustworthy and the most stale.
• The model writes it through a memory tool, deciding mid-run what is worth keeping. Selective and timely, but models save trivia, save wrong inferences with full confidence, and never delete.
• The harness writes it automatically after a run - summarise the episode, extract the facts. Complete and consistent, and the fastest way to bury the three useful facts under three hundred harmless ones.

The hard problems follow from that. Selectivity: what is worth a permanent record. Staleness: "the API runs on port 8080" was true in March, and a stale fact is retrieved with the same confidence as a fresh one, so every memory needs a timestamp and a way to be superseded. Scope: memory belongs to a user, a project, or an organisation, and a memory that leaks from one user's runs into another's is a security incident, not a quirk. Trust: a memory is an input the agent wrote to itself, so a wrong one poisons every future run, and a prompt injection that gets the agent to write a memory has found a way to persist. Treat the store as you would any data written by an untrusted process, because that is what it is.

And one property that decides the next section: long-term memory is inert until something reads it. A store nobody queries is a log.

Retrieval

Retrieval is the mechanism that finds relevant stored information and brings it into the window. It is not a store and not a kind of memory; it is a function. Given the current need - the user's message, the goal, the last observation - return the few items worth reading now. Retrieval is the bridge between long-term memory and working memory, and between any external corpus and working memory. Without it, everything in section 04 stays on disk.

It comes in more forms than the word "search" suggests, and picking the cheapest one that works is most of the craft:

• Exact lookup. A key you already know: user id to profile, project to its rules file. Instant, deterministic, never wrong. Use it whenever you can name what you want.
• Structured query. Filters over typed fields: the last five runs for this repository, every memory tagged deploy newer than June. Databases have done this well for fifty years.
• Lexical search. Match the words - full-text indexes, BM25. Excellent for identifiers, error strings, and names, which is much of what agents look for.
• Semantic search. Embed the query, find the nearest stored vectors, match meaning rather than wording. This is what people usually mean by "a vector database", and the Data path covers the machinery. Powerful, fuzzy, and the wrong first choice for a lookup you could have done exactly.
• Navigation. The agent reads a notes file, greps the project, opens the document it needs. An agent with read_file and a memory directory is doing retrieval by hand, and for codebases and small stores it is often the most reliable form of all.

Retrieval also has a trigger, and there are two. In the pipeline shape, the harness retrieves before every model call and pastes the results into the context - the model never knows a search happened. In the tool shape, the model decides it needs something and calls search_memory or recall itself. RAG vs MCP is the full argument over which to choose; the short version is that the harness should retrieve what is always needed, and the model should retrieve what only it knows it needs.

One distinction this article exists to make: retrieval over documents and retrieval over memory use the same machinery and are not the same thing. Retrieval-augmented generation reads a corpus that was given to the system - manuals, tickets, the knowledge base - and those documents are ground truth from outside. Memory retrieval reads what the agent itself stored, and those records are the agent's own past notes, with all the trust problems of section 04. Same index type, different corpus, different confidence, and an agent that cannot tell the reader which one a fact came from is hard to trust.

Retrieval fails in one particular way: it returns something. A search that comes back empty is harmless. A search that comes back with the wrong memory hands the model confidently irrelevant context, and the model will use it. Ranking matters, recency matters, and k - how many items you bring back - matters most, because each recalled item is paid for on every subsequent turn of the run.

One loop, four words

Here are the four in one picture. Read the top row left to right for a single turn, and the bottom row for what connects one turn to the next.

A turn arrives. The harness builds the context: the instructions, a rendering of state (the plan, the counters that matter, the notes), the recent transcript, and whatever retrieval brings back from the store for this particular message. The agent acts - reasons, calls tools, answers - and when something worth keeping surfaces, it writes to the memory store, through a memory tool during the turn or an extraction step after it. State is updated. On the next turn retrieval reaches into the store again, and what it finds becomes working memory. Nothing inside the model changed at any point. The loop moved text between four places.

In code, the whole thing is one function that builds the context and one call that writes to the store. store is whatever backs long-term memory; state is the harness's object from section 03.

Notice the asymmetry. Writes are rare and cheap; reads happen on every turn and cost window space each time. Design the store for the reads: small records, timestamps, a scope on everything, and a search that returns four good items rather than forty plausible ones.

Where builders go wrong

Almost every memory problem in a production agent is one of the four words doing another word's job.

• A bigger window as memory. The window is a desk, not a filing cabinet. Enlarging it lets you spread more out for one turn; it does not keep anything, and past a point it makes the model worse at reading what is there. Bigger windows change what is affordable to render, and nothing else.
• The transcript as state. "The agent forgot which files it changed" is almost always a compaction that summarised away the only record. Anything the harness has to act on - counts, lists, decisions, permissions - goes in the state object, where a summary cannot touch it.
• Memory without retrieval. "We store every conversation" is a compliance feature, not memory, until something selects from it. And the opposite failure is as common: retrieval that dumps thirty recalled items into every turn, so the model reads yesterday's lunch order while debugging a build.
• RAG as memory, memory as RAG. Same index, different corpus, different trust. Documents were given to the system and are ground truth; memories were written by the agent and can be wrong, stale, or planted. Keep them in separate stores, or at least separate scopes, and tell the model which is which.
• Believing the model remembers. "It remembered my name" means a file was read and its contents rendered into the prompt. That matters for debugging - look at what was retrieved, not at the model - and for privacy: the memory is a record that exists, can be inspected, and must be deletable on request.
• One memory for everyone. A store without a scope column mixes users, projects, and organisations. The first symptom is a helpful agent mentioning another customer's deployment; there is no second symptom, because that is the incident.

Four decisions

Separating the words is worth doing because each one comes with exactly one design decision, and the decisions are independent.

• Working memory: what to render, in what order, and what to cut first when the window fills. This is the whole of context engineering, and what falls out first is the part to decide before launch, not after.
• State: what must stay exact and survive compaction - then persist it after every turn, so a run can pause, crash, and resume.
• Long-term memory: who is allowed to write, what is worth keeping, when a record expires or is superseded, and whose it is. Scope and timestamps on day one; they cannot be added to a store already full of unscoped, undated facts.
• Retrieval: who triggers it - harness, model, or both - how many items come back, and how they are ranked. Start with exact lookup and structured queries; add semantic search when a real query cannot be expressed any other way.

Make the four decisions separately and "add memory" stops being a feature request and becomes an architecture. The loop these four places sit inside is the subject of Inside an AI agent: the agent loop; the window they all end up in is What's in an agent's context?; and what happens when one of the four is skipped fills a good part of Why agents fail.