TL;DR
Working memory is the context window itself, free and volatile. Everything called "agent memory" beyond that is external storage plus a read policy: something written down outside the window and pulled back in when relevant. Every vendor selling a memory product agrees on that architecture; where they disagree with each other, and where none of them will tell you, is whether your system needs their product at all. A flat log file with a good retrieval query covers more real cases than the marketing suggests.
This is the memory-lever post of the context engineering series, and the one that asks the question the SERP for this term cannot: not how a memory product works, but whether you need one.
Building an agent that has to remember things across sessions? We design AI agent memory around what the task actually needs to persist, not around whichever product got pitched first.
Key Takeaways
- Working memory is the context window. It costs nothing extra and vanishes when the session ends. "Agent memory" is what happens when something needs to survive past that.
- Persisted memory is always storage plus a read policy: what gets written, where, and what triggers reading it back into context later.
- A single-session task never needs a memory product. The context window already is the memory, for exactly as long as the task needs it to be.
- The three real signals for needing a dedicated memory system: cross-session persistence, semantic retrieval over accumulated history, and multiple agents or users reading the same memory.
- Every major vendor on the "agent memory" SERP sells a memory product. None of them can honestly tell you when you don't need one. That is this post's actual job.
What is agent memory, actually?
Start from what does not need a name: the context window an agent is currently working in is already memory, in the plainest sense, information available to the model right now. It costs nothing beyond the tokens already in play, and it disappears the moment the session ends. Most of what an agent needs to "remember" during a single task is already handled by this, no product required.
"Agent memory" as a category refers to what happens once something needs to survive past that window: past a session ending, past a context reset, past what fits in the budget at all. The mechanism is always the same shape, write something to storage outside the window, and read it back in later under some policy that decides when it is relevant. A markdown file an agent appends to, a row in a database, an entry in a vector store, a summarized fact extracted from a conversation, all of these are the same architecture with different storage and different retrieval logic.
The three questions to ask before you buy a memory product
Does the task actually need to survive past this session? If an agent's work is done when the session ends, the working context already served as memory for the full duration it was needed. A shopping assistant helping with one purchase does not need persisted memory. A support agent that should remember a customer's history across visits does. Confirm which one your task is before evaluating any product.
Is a flat log or file enough, or do you genuinely need semantic search over accumulated history? Most systems need less retrieval sophistication than the pitch implies. If an agent only ever needs "what happened in the last conversation," a timestamped log read in full does the job at zero infrastructure cost. Semantic retrieval, finding the relevant fact buried in thousands of past interactions by meaning rather than exact match, is where a vector store earns its complexity, and it is a smaller slice of real use cases than the SERP for this term suggests.
Who reads the memory back, and does it need to be shared? A single agent recalling its own past sessions is a simpler problem than multiple agents or users reading and writing the same memory concurrently. Shared, concurrent memory is where the infrastructure a managed product provides, consistency, conflict handling, access control, starts paying for itself. A solo agent's private notes rarely need any of that.
When a memory product actually earns its keep
None of this is an argument that memory products are unnecessary in general, only that most pitches skip the qualifying questions above. A product earns its complexity when at least one of three things is true: memory needs to be queried semantically across a corpus too large for a linear scan to stay fast, multiple agents or users need consistent concurrent access to the same memory, or the system needs automatic summarization and consolidation of memory over time, compressing what accumulated into what is still worth keeping, which is genuinely hard to build well from scratch.
Recent research pushes on that last point specifically. Newer approaches to agent memory, sometimes described under the name A-Mem, propose organizing memory dynamically, letting the agent link new notes to related past ones as they arrive rather than storing entries in a fixed, predetermined schema, closer to how a person's own notes accumulate structure over time than to a rigid database table. Whether that sophistication is worth adopting still comes back to the same three questions: most tasks do not need it, some genuinely do.
Agent memory vs RAG: are they the same thing?
Related, not identical. RAG retrieves from a corpus of documents the agent did not create, product manuals, a knowledge base, a codebase. Agent memory retrieves from what the agent itself has done or observed, past conversations, extracted facts, prior decisions. The retrieval mechanism underneath can be identical, embeddings, a vector store, a similarity search, but the content and the write path differ: RAG's corpus is authored elsewhere and indexed once; agent memory is authored by the agent's own activity and grows continuously. A system can use both, RAG for external knowledge, memory for what the agent has personally accumulated, without either one substituting for the other.
Do you need a memory product? A quick check
| Signal | Points to plain storage | Points to a memory product |
|---|---|---|
| Session scope | Task ends when the session does | Must survive across sessions |
| Retrieval need | Exact match or "everything from last time" | Semantic search across large history |
| Access pattern | One agent, its own notes | Multiple agents or users, concurrent |
| Growth over time | Small, bounded, manually reviewable | Large, needs automatic consolidation |
| What you have today | A file or a simple database already | Nothing, and the above signals are real |
Frequently asked questions
What is agent memory?
Agent memory is information an AI agent needs that persists beyond its current context window, written to external storage and read back in later under some policy. The context window itself already functions as memory for the duration of one session; agent memory specifically refers to what survives past that.
Do I need a vector database for agent memory?
Only if you need semantic search across a large volume of accumulated history. Many systems only need "recall the last session" or "look up a specific fact," both of which a flat log or a simple structured store handles at far lower cost and complexity than a vector database.
What is the difference between agent memory and RAG?
RAG retrieves from a corpus authored elsewhere, documents, a knowledge base. Agent memory retrieves from what the agent itself observed or decided. Both can use the same retrieval mechanism underneath, but memory's content is generated by the agent's own activity and grows continuously rather than being indexed once from a fixed source.
Is working memory the same as long-term memory?
No. Working memory is the current context window, available for the duration of one session and free beyond the tokens already in use. Long-term memory is anything written to storage outside that window specifically so it can be read back in a future session, which always costs a retrieval call to recover.
What is A-Mem?
A-Mem refers to a newer approach to agent memory that organizes entries dynamically, letting the agent link new memory notes to related past ones as they accumulate rather than storing everything in a fixed schema. It is one answer to the memory-consolidation problem, not a requirement for every system that needs persistence.
