AI News · industry
AI Agent Memory in 2026: What the New Benchmarks Show
Three September 2026 tests measured what agents actually remember. The best score was 70.67%, compressed notes broke on a model swap, and memory sometimes cost more than the agent itself.
AI agent memory is the layer that lets an agent use something a customer told it weeks ago, and as of September 2026 it is measurably unfinished. On DolphinBench, released 22 September 2026, the highest-scoring configuration completed 70.67% of 600 memory-dependent tasks. Nearly a third failed. A separate controlled study, Does Your Agent's Memory Survive a Model Upgrade?, published 4 September, found that when a memory store built by one model was handed to another, accuracy moved by as much as 13.28 percentage points in the wrong direction, without anyone touching the stored data.
Three things landed in September that are worth your attention if you run AI that talks to customers: a benchmark that grades memory by actions instead of quiz answers, a study on whether memory survives a model upgrade, and a Chinese-hosted challenge that puts every memory system under one rulebook. Here is what each measured, and what it changes.
What is AI agent memory, and how is it different from a context window?
A context window is the working space a model has during one run. It empties when the run ends. Agent memory is durable state: what gets written down after a conversation, and retrieved before the next one.
The distinction matters because the industry keeps trying to solve memory by making the window bigger. The evidence says that does not work on its own. The CueMem paper (11 September 2026), from Mashang Consumer Finance and the Harbin Institute of Technology in Shenzhen, found that feeding a whole dialogue history to the model degrades as the input approaches the context limit, an effect the authors attribute partly to irrelevant context and the lost-in-the-middle problem. Their retrieval approach used roughly 10% of the input tokens of the full-history setting on the LoCoMo benchmark and still scored higher.
OpenAI's 21 September write-up on V7, which builds agent context for finance and insurance teams, says the same thing from production: V7's graph is "an order of magnitude cheaper and faster to traverse than long-context approaches," with recent exchanges kept in active context and older material left in the graph until something asks for it.
So memory is not a bigger window. It is a decision about what to write down, and what to keep.
Why did the benchmarks change in September 2026?
Until now, most memory benchmarks asked questions. That flatters the systems being tested.
DolphinBench's argument is blunt. Ask an agent "what messaging platform does the team use?" and you have already told it a platform fact exists and that you want it. The hardest step, noticing that retrieval is needed at all, has been done for it. So DolphinBench removes the question. It gives the agent three simulated knowledge-worker histories (about 500,000 tokens each, 13,539 messages spanning 2023 to 2027), then issues 600 tasks against simulated apps including Notion, Gmail, GitHub and Discord, where a wrong action leaves a trace the grader can see.
One worked example: a CEO states a channel rule once, in April 2023, that release notes go in #eng-releases until a deploy is green. Nothing repeats it. Three and a half years later, buried in a 3,400-message history, a request arrives to post a rollout update, and no channel is named. Post to the wrong one and you fail. Knowing the fact is not enough; the agent has to apply it unprompted.
Every test is also certified solvable: it must pass when the relevant history is supplied and fail when it is withheld. That rule kills the usual complaint about synthetic benchmarks, which is that nobody knows whether a failed test was ever answerable.
Which memory systems performed best, and what did they cost?
Here are DolphinBench's published results for the Hermes harness running GPT-5.6 Luna, as of 22 September 2026. Accuracy is the percentage of 600 tasks completed. The cost columns are the benchmark's own published run costs for the full suite, in US dollars, useful as a ratio between systems on the same test rather than as a price you would pay.
| Memory system | Accuracy | Agent run cost | Memory run cost | Median latency |
|---|---|---|---|---|
| Mem0 | 70.67% | 61.54 | 34.68 | 37.69s |
| Hindsight | 69.50% | 57.99 | 26.66 | 55.31s |
| Honcho | 68.50% | 96.56 | 46.30 | 44.66s |
| Model's built-in memory | 65.67% | 61.48 | 0.00 | 44.35s |
| Supermemory | 59.17% | 63.86 | 281.79 | 52.68s |
Two findings deserve more attention than the winner.
The memory layer can cost more than the agent. Supermemory's memory spend on this run was roughly four and a half times its agent spend, and it finished last on accuracy. Any vendor who quotes you an accuracy number without a cost number is showing you half the result. DolphinBench's position is that cost and latency belong on the same line as accuracy, which is why they call it a Pareto frontier rather than a leaderboard.
How much memory helps depends heavily on the model underneath. With GPT-5.6 Luna, a dedicated memory layer added about five points over the model's built-in memory (70.67% vs 65.67%). Swap in MiniMax M3, the open-weights Chinese model, and built-in memory collapsed to 26.50% while Mem0 reached 47.83%, a gap of over twenty points. The weaker the model's native handling of long history, the more the external memory layer carries. If you are running an open-weights model to control cost, that finding is the one to act on.
Does agent memory survive a model upgrade?
Usually not, and this is the finding most teams have not priced in.
The portability study cited above ran 48 synthetic histories through four memory designs, then swapped the model that wrote the memory. Results by design:
- Fixed-schema knowledge graph: accuracy changed by +0.0004 points. Effectively immune.
- Compressed natural-language notes: accuracy shifted by +9.91 or −13.28 points depending on which direction you migrated. Same notes, different reader, different answer.
- RAG with a half-migrated embedding index: a 50/50 mixed index captured only 4.96 of the 11.90-point gain available from re-embedding everything. Half a migration buys you well under half the benefit.
The repair result is the one to remember. When compressed notes went wrong, fixing them from the store alone failed to reach a 90% recovery target in all 48 cases. Keeping the raw source history recovered 34 of 48. If you throw away the original conversations and keep only the summary, you have thrown away your ability to fix the summary.
This connects directly to something we argued in why you shouldn't build your business on a single AI model: models get retired on 12-to-18-month clocks. What this study adds is that being model-agnostic is not only about swapping an API. Your accumulated memory is coupled to the model that wrote it, and nobody sends you a migration notice.
Who is building agent memory, and where?
The field is genuinely split between the US and China, and between open and closed, in a way most English-language coverage misses.
| System | Origin | License | What it is |
|---|---|---|---|
| Mem0 | US | Open core + managed | Drop-in memory layer; authored DolphinBench |
| Letta (formerly MemGPT) | US | Apache 2.0 | Stateful agent server with explicit memory blocks |
| Honcho, Hindsight, Supermemory | US | Commercial | Managed memory APIs, scored on the DolphinBench leaderboard |
| ReMe | China (Alibaba Tongyi Lab) | Open source | Memory kit in the AgentScope stack, file and vector based |
| MemoryOS | China (Beijing Univ. of Posts and Telecommunications) | Apache 2.0 | Hierarchical memory OS; the team reports 49.11% average F1 gain on LoCoMo |
| MemOS | China | Research | Treats memory as a schedulable system resource across plaintext, activation and parameter levels |
The clearest signal of where this is heading is the second Agent Memory Challenge, which opened on 20 September 2026. It is hosted by the China Society of Image and Graphics and organized with Nanjing University, Zhejiang University and Datawhale. The scale is serious: over 6,000 evaluation instances and roughly 300 million tokens in the textual track alone, plus separate coding and multimodal tracks. Prize pool is ¥150,000, and it is reserved for open-source methods. Commercial products can enter and get scored, but they run on their own leaderboard and win nothing.
Read that design choice carefully. The people setting the rules are rewarding open systems while still holding closed products to the same evidence on a separate board. Evaluations close 4 November, results mid-November.
One of the challenge's textual dimensions is the one nobody markets: memory governance, meaning updates, conflict resolution, deletion and forgetting. Another is epistemic safety and privacy, scored on abstention and minimal disclosure. Those matter most to a regulated business, and until this year almost nothing measured them.
What does this mean for a business running AI on customer conversations?
Five conclusions, drawn from the results above rather than from vendor decks.
- Keep the raw conversations, not only the summary. Store-only repair failed in all 48 cases; retaining source history recovered 34. A summary is a cache, not a system of record.
- Ask for accuracy and cost together. A memory layer that multiplies your cost and loses accuracy is a benchmarked outcome, not a hypothetical.
- Test memory by action, not recall. Does the agent apply the preference your customer stated in March without being reminded?
- Plan the migration before you need it. Half-migrated embedding indexes underperform badly. Decide up front whether a model change means a full re-embed.
- Check deletion reaches derived memory, not just the contact row, and ask your vendor to demonstrate it rather than describe it.
Where Entagl sits on this
Entagl runs one agent brain across channels rather than a separate bot per channel, and the Coordinator voice agent reads a summary of the customer's prior chats before it dials rather than opening cold. Business knowledge (services, products, FAQs, hours, policies) lives in a searchable knowledge base the agents query, which is closer to the fixed-schema approach that survived the model swap in the portability study than to free-text notes.
Carrying content from a WhatsApp thread into that same customer's Instagram thread is a separate cross-channel memory layer, and it is in staged rollout rather than finished. It writes a structured summary per conversation (what has been answered, what is pending, what was promised) while keeping the original messages. It stays off until the rollout reaches a workspace, the owner can switch it off at any time, it requires a verified identity link before it discloses anything (a customer simply claiming a handle is not enough), derived memory is erased when a contact is deleted, and stored memory ages out on a fixed retention window.
On governance, the principle is the one we set out in what's new in AI agents in 2026: protocols and guardrails: AI acts, humans govern. For the buyer-side checklist on linked customer records across channels, see cross-channel customer conversations without the confusion.
What these results do not prove
Worth being straight about the limits.
DolphinBench uses synthetic personas and simulated apps, so the 70.67% ceiling is specific to that task set and those harnesses. The portability study ran on two open-weight models under 10 billion parameters, so its exact point swings will not transfer to a frontier model. The Agent Memory Challenge has not published cycle-two results yet; what we have is its protocol. And Mem0 authored the benchmark it tops, which is disclosed openly and is normal here, but it is a reason to want independent replication before treating 70.67% as settled.
None of that changes the direction. Memory is being measured properly for the first time, and the measurements are less flattering than the marketing.
FAQ
What is the difference between AI agent memory and RAG?
RAG retrieves from a document corpus to answer a question. Agent memory manages evolving state about a specific user or account across sessions: what changed, what was superseded, what was promised, what should be forgotten. RAG is often a component inside a memory system, but a memory system also has to handle updates, conflicts and deletion, which a document index does not.
How accurate is AI agent memory in 2026?
On DolphinBench, the action-based benchmark published 22 September 2026, the top configuration completed 70.67% of 600 memory-dependent tasks. Results varied widely by underlying model: with the open-weights MiniMax M3, built-in model memory scored 26.50% while a dedicated memory layer reached 47.83%.
Does an AI agent forget when you change the model?
It can, without any data being deleted. The September 2026 memory-portability study found compressed natural-language notes shifted accuracy by up to 13.28 percentage points after a model swap, while a fixed-schema knowledge graph was essentially unaffected. How you store memory determines whether it survives.
Can a customer ask an AI system to delete what it remembers?
They can and should be able to, but deletion has to reach derived memory too, not only the original messages. This is now an explicitly scored dimension in the Agent Memory Challenge's textual track, alongside updates, conflict resolution and forgetting. Ask any vendor to demonstrate it rather than describe it.
Is a bigger context window a substitute for memory?
No. The CueMem paper found long-context models degrade as input approaches the window limit, and its retrieval method matched or beat full-history at roughly 10% of the input tokens. OpenAI's V7 write-up reports the same trade-off in production, where a graph is an order of magnitude cheaper to traverse than long context.
The one thing to take away
Ask any vendor one question: what happens to everything the agent knows about my customers when you change the model underneath it? The September 2026 research says the honest answers are either "we keep the raw history and re-derive" or "some of it degrades and we cannot fully repair it." There is no third answer that survives the benchmarks.
To see how a shared-brain setup handles that across DMs, web chat and calls, book a 30-minute demo and bring your hardest continuity question.
Sources, as of 23 September 2026: DolphinBench (Mem0, 22 Sep 2026; paper); Does Your Agent's Memory Survive a Model Upgrade? (Goyal and Ray, 4 Sep 2026); CueMem (11 Sep 2026); Agent Memory Challenge cycle 2 (CSIG, opened 20 Sep 2026); How V7 gives AI agents institutional memory (OpenAI, 21 Sep 2026); MemoryOS; MemOS; ReMe. Model versions and rankings move monthly; figures are dated for that reason.