Mastra’s Memory is SotA on BEAM

Tell an AI agent something early in a long chat. A million tokens later, ask about it. Until now, the best published score at that length was 75%.

Mastra’s memory scores 80.6%.

Observational Memory, Mastra’s open-source agent memory, has the top published scores on BEAM, a long-term memory benchmark from ICLR 2026, at both 100K tokens (83.3%) and 1M tokens (80.6%). That puts it ahead of Exabase M-1, Hindsight and Honcho. At 10M tokens it scores 66.4%, inside the margin of error of the best published score (68.0%).

We ran it with default settings and no benchmark tuning. Claude Opus 5.5 answered the questions; Exabase used the cheaper Gemini 3 Flash. With Gemini 3.8 Flash answering instead, we still lead at 100K (78.4%) and drop to 72.8% at 1M, just behind Exabase and Hindsight. The appendix has the full comparison.

Below: how it works, what we learned from the failures (like the agent passing off its own advice as yours), and where 10M tokens finally breaks it.

Score on each size of BEAM. Mastra’s score is the average of two runs. Other scores are as each provider reported them, not independently verified by Mastra.

How Observational Memory works

The three places OM keeps information: notes the agent always sees, an archive it can search, and the raw messages behind each note.

Observational Memory keeps a set of running notes about what happened in the conversation. These notes are generated by an observer process and give the agent context on what’s happening.

When the notes get too long, we run a reflector which rewrites the notes to be more compact, prioritizing more important/recent details.

The agent is also given access to a recall tool. This tool allows the agent to revisit older notes in the archive and read raw messages behind these notes.

For a detailed look at how to set up Observational Memory or how it works, check out our docs or our announcement post.

How BEAM tests memory

BEAM (which stands for Beyond a Million Tokens) is a long-term memory benchmark which provides a simulated chat between a user and an assistant and asks 20 questions about each conversation. These chats come in multiple sizes, ranging from smaller, 100K token chats, to chats that span 10M tokens.

Each question tests one of ten abilities. For a closer look at how BEAM works, check out the GitHub repository.

Results

Observational Memory has the top published scores on BEAM-100K and BEAM-1M

Mastra’s Observational Memory scores state of the art scores across multiple tiers on the BEAM benchmark.

• BEAM-100K: 83.3%.
• BEAM-1M: 80.6%.
• BEAM-10M: 66.4%.

On BEAM-10M, the top published score is 68.0%. BEAM-10M has an error bar of ±5.8 points, so Observational Memory matches the frontier within noise.

We ran a default Mastra agent connected to Observational Memory with recall enabled to get these scores. We didn’t add any system prompt or change any other settings, so users can expect to get similar results running the Mastra agent with Observational Memory out of the box.

We used Gemini 3.8 Flash as the observer and reflector models, the answering Mastra agent used Claude Opus 5.5 and the LLM-as-a-judge was Gemini 3 Flash (the same as the next best system, Exabase).

For more detailed information about our philosophy of how BEAM should be run and how we ran it, check out the Methodology section.

Scores by ability

BEAM produces scores for 10 abilities, which it then averages for the final score. It is useful to look at how Observational Memory did per ability. For what each ability stands for, check out BEAM.

Score per ability at each size.

As we take a look at per ability scores, there's a clear trend emerging. Most abilities perform pretty well at 100K tokens. We see a slight drop at 1M. At 10M, the capabilities of Observational Memory are stretched to the maximum.

Between the easiest and most difficult tiers, instruction following, information extraction, and summarization score around the same at both the 100K token tier and the 10M token tier. Every other ability drops at 10M tokens of context, with multi-session reasoning dropping the most drastically, from 74% at 1M tokens to 25% at 10M tokens. Preference following also drops from 90% to 62%.

Overall, we lose 2.7 points from 100K to 1M and 14.2 further points from 1M to 10M. Later sections describe why such a steep drop happens.

The best and worst abilities

As we can see, there are particular abilities that observational memory shines at and saturates the benchmark. There are some others where observational memory, at its default settings, might not be well suited, and some configuration might be needed to make it good at those.

Observational memory is best at following instructions and tracking changes across a conversation.

Instruction following stays around 97%: When a user asks the assistant to do things a certain way, the observer is really good at writing it down, and the instructions stay in the notes through every rewrite. The agent sees it on every turn, so it rarely even has to search for it. Even at 10M tokens, it searches on only 23% of such questions

Knowledge update scores 81% on 100K and 1M, and 68% at 10M: These are questions asked about the current value of something that might have changed across the conversation. Maybe it's a number that the user updated or a date of a flight that might have been cancelled. Every note the observer takes is dated, and when something changes, the reflector keeps both the old and new value. The notes are easy to parse for the agent, and it can figure out which value is current. This is also why Observational Memory leads Exabase M-1, the only other system that publishes per-ability scores, by 22 points or more at every size.

Observational memory trails on connecting data from separate sessions and abstaining from answering

Why a count fails. Illustration: the archive of one 10M chat, with the note groups that mention the $30 sessions.

Multi-session reasoning drops to 25% at 10M: Many of these questions are about counting instances of something that happened, which would need every matching note. At a size of 10M tokens, most notes live in the archive, and search only returns the closest few, so the agent finds some and stops there. It doesn't know that there might be any further. It's hard for every system we've seen. The only other system we have public numbers for scored 9.6% here.

Abstention sits between 68% and 77%: These questions ask about something that never happened, and the right answer is to claim that it never happened. The problem is that searching for it usually finds something nearby, often advice the assistant itself gave on that topic. The agent then has to figure out who said this and sometimes passes its own advice as the user's. This misattribution issue causes loss in abstention scores.

Compared to other systems

Exabase M-1 is the only other system that publishes a score per ability, so that’s the one we can compare against:

Observational Memory’s score minus Exabase M-1's, in points. Blue means OM scores higher.

We can see how both systems differ in different abilities at each size. Observational memory seems well ahead in knowledge update, multi-session reasoning, and contradiction resolution, while falling behind in abstention and summarization.

This is comparative between these two systems and does not say much about each system's absolute capabilities.

When the agent fails, it is confident that it found the answer and states wrong facts

Share of answers that scored zero, by what the answer did. Mean of two runs.

Past the abilities to the answers themselves, one thing stands out. When the agent gets a question completely incorrect, it almost never says that it doesn't know.

Outside of abstention questions, 80 of the 93 answers where we got a zero: the agent stated a particular fact, and it's usually a real fact from the chat, but not relevant to the question being asked. Maybe it's the old value of something that changed, a total from another session, or the date of another related event.

The issue is not that the agent is not able to answer. It is that it finds something that vaguely looks like the answer and might trust it when it shouldn't.

BEAM-10M tests the limits of how much memory systems can handle today

Change in points per ability, going from 100K to 1M, and from 1M to 10M.

As you can see, most of the drop happens after we scale from 1M to 10M. Observational Memory holds its abilities from 100K to 1M pretty well. The only exception to this is temporal reasoning, where there is an early drop, but then it holds. The reason is that 10M is the first size where the notes can't reasonably hold the chat, and the reflection process starts mattering.

While we at Mastra want to share these results, we also want to go through how exactly these results happened: How did observational memory weather BEAM, and what could users learn from it to optimize observational memory for their own use cases.

How the 10M tokens fit into our observation notes

Tracing how we maintain speed while maintaining accuracy over such a large context

BEAM chats can get extremely long. There is no LLM context window in the world which can fit anywhere close to 10M tokens of context. Even for models which could take 1M tokens in their context window, a BEAM-1M chat would fill it up to the brim and the response quality of such an agent/LLM would be terrible. Consider on top of that the cost and speed of such chats, and it is obvious that just loading raw data into context is a bad idea.

This is why Observational Memory needs to be very careful about how to curate context and how to decide what goes into the model’s context window. This is why our note-taking exists.

Across BEAM tiers, the chat grows 80 times longer.

Median chat length against the size of the notes the agent reads. Log scale.

Even as the chat grows exponentially larger, Observational Memory keeps only a very small amount of tokens (about 40K tokens by default) which gets passed on to the agent. At 10M tokens, Observational Memory is keeping notes 339x smaller than the original context.

This keeps the amount of context the agent gets nimble on most turns, preserving cost and speed. These observation notes are also designed to take ample advantage of prompt caching, which makes every turn even cheaper and faster.

Compressing 10M tokens to 40K tokens per turn is not trivial, and does not come without tradeoffs. Here we take a look at how that works, and where things trip up.

The reflector runs a lot for longer chats

Average Observer calls and Reflector rewrites per chat. Log scale.

Since the observer runs once we have 30,000 tokens of unobserved messages (or whatever limit you configure, this is the default), as the chat context grows, the number of times the observer runs grows with it, almost linearly. This process feels invisible because it runs in the background every time 20% of the quota is reached.

Reflection runs when the observation notes themselves become too large to package in the 40K token limit (this can be configured too).

The number of times reflection runs grows exponentially: BEAM 100K and BEAM 1M are mostly just testing the observer. The reflector really only matters a lot on BEAM 10M.

Due to this distribution and the respective tier size, notes keep becoming more and more biased towards recent tokens/facts in the context.

Share of figures within the text that belong to each third of the chat.

Every time reflection rewrites the notes, it prefers keeping more details from the end of the notes or more recent notes than older notes. After about 75 rewrites, which is the average for BEAM-10M, 96% of the notes describe only the last third of the chat, with the first two-thirds of the entire chat getting only 4% of the attention.

In fact, even trying to attribute the summary to the last third of the conversation might be misleading.

Even recent events don’t benefit from recency bias at 10M

Even the last third of the conversation has mostly been through the reflector at 10M tokens. Only about 3% of the conversation is directly included by the end of the 10M context chat as notes, the rest might or might not be there, only as a summary.

Score on questions whose evidence sits in one part of the chat, by which third it's in. Mean of two runs.

At 100K and 1M, questions about the last third of the chat score significantly better than the first two-thirds, and the relationship is pretty linear. If the fact to be found is more recent, it is more likely that the answer would be correct.

This breaks down at 10M. At 10M, this trend remains flat. You are no more likely to find something from the beginning of the chat than you are by about the end of it. The reason behind this is simple: 10M tokens is just simply too big for recency to help at all.

This points towards another idea: at 10M, recall will decide answer quality rather than the observation notes. We will dive deeper into this in a later section.

The more often the notes are rewritten, the fewer exact numbers they keep

Share of figures from the chat still in the notes, by how many rewrites have happened since they were written down.

We did the math by trying to track every number when it appears in context and then seeing if it's still in the notes after one rewrite, two rewrites, three rewrites, and so on.

68% of numbers survive two rewrites, but only 21% survive three. From there, they slowly go down to 5% as you have the notes around them rewritten 40 times.

As an example from a 1M chat, the user once says that their growth rate is 0.035:

After 0 rewrites:  User stated they are practicing with k = 0.035 for the population growth model
After 1 rewrite:   -> with k = 0.03 ... ; with k = 0.035, P(10) ≈ 1419
After 4 rewrites:  (gone)

The user statement was turned into an example, and then it completely disappeared. Later, BEAM poses the question of what the growth rate was. The agent does a search, but only from after this number was presented, skipping this archived note. It found a different population model from later in the conversation and quoted the number there, which was incorrect.

The archive keeps more exact numbers than the notes, but not everything

Share of exact figures from the chat that each store still holds.

The recall tool allows the agent to check the archived notes out. These are all the observation notes that have been written before reflection. The recall tool allows the agent to access these and also the raw messages behind these.

We took a look at how many of the exact numbers are being kept per tier by the archived notes. Turns out, for smaller contexts, more of the numbers are preserved as expected. But as the context size grows, the archive holds stable in storing about 35% of all exact numbers quoted.

This means the archive is a pretty reliable store from which facts can be recalled even at higher tiers, and that any future optimizations might want to go towards the search function instead of trying to store more.

The thing to realize also is that the archive wouldn't include everything. It would actually include less than half of the exact numbers quoted in the chat, not only because it's lossy but also because a lot of them would just not be relevant. Take this with a pinch of salt.

Each note does have raw messages behind it, so if the raw data is needed, it is available.

How the agent searches

At 10M tokens, recall decides the score.

Whenever the user asks a question, the agent has two places to find the answer.

1. The notes: These are always presented to it directly in the context window and are easy to reference.
2. The archive: Every note the observer ever wrote is kept as it was originally written. It works sort of like a filing cabinet, preserving all the work the observer has done so that later on, if the agent requires revisiting that work, it can. While everything is in there, the agent explicitly has to go search for it using the recall tool.

For a short chat, the notes will almost hold everything. For longer chats, the agent would need to dip into the archive. By 10M tokens, only the last 3% of the chat is still in the notes, the way the observer first wrote it. Everything else is in the archive.

So at extremely long-context, the score is decided by how well the agent can search.

The agent searches on 21% of questions at 100K and 74% at 10M

For BEAM, we did not tell the agent when to search. Observational memory has built-in guidance that does that, and the agent picked that up. We saw that when chats get longer, it searches more.

Share of questions where the agent called the recall tool, and calls per question.

At 10M, it makes nearly three recall calls for every question. It also doesn't search for things randomly. It only does a search when it knows that a search would be the reasonable thing to do. For example, if it's looking for an exact date, a total, or proof that something was never said, it reaches for search.

At 10M, the agent also searches for every question that requires dates and durations to be exact, and on every question about counting instances of something across sessions. Even at 100K, it searches on 95% of abstention questions.

Where it lacks is that it almost never searches for preference questions. It only searches for 10% of them at 10M. As we'll see, this matters a lot.

The way it uses search at 100K tokens and 10M tokens is different

At 100K, the notes generally already have the facts, so the agent mostly is using recall to zoom into the raw messages. At 10M, that's not the case. The facts are not in the notes most of the time, so it uses recall to look for the facts themselves.

The two ways the agent uses recall.

Search only finds the evidence 45% of the time at 10M

We define finding the evidence as finding the exact messages that BEAM cites. The agent can still answer from the rewritten notes, but taking a look at exact matches is also important.

When this agent searches the archive, the evidence BEAM is looking for is found for 74% of the searches on 1M, but only 45% of the searches at 10M.

BEAM-10M questions, grouped by whether the agent's searches brought back the messages BEAM cites as the source of the answer.

This shows that search doesn't always find what it needs and that answering from rewritten notes is worse than directing the agent to the actual messages that BEAM thinks the answer is present in.

Search is the clearest place to make Observational Memory better at extremely long-context workloads, and we are investigating ways to improve the recall tool.

Now let's take a look at the two major failure modes of why search goes wrong: misattribution and the agent never even looking.

Sometimes the agent takes its own words as the user's

In a long chat, most of the text is written by the assistant, not the user in BEAM.

Share of characters in BEAM's chats written by each side.

Throughout BEAM, the assistant wrote 94% of the text. The user asks a concise question, and the assistant presents extremely long replies.

That matters a lot for abstention questions. Here's an example from a 1M chat:

How misattribution happens, from a BEAM-1M chat.

For the purposes of this example, Scott never recommended any drills at all. The user had asked a question about the drills before. The assistant had created its own advice on this, and then later, when it looks for “Scott left foot drills”, acts as a perfect match. It is now left to the agent to figure out whose words these are.

If the agent were to get the person who said this wrong, it could pass off its own advice as Scott's advice. This is what we mean by misattribution. It is also a major reason why observational memory sometimes fails on abstention questions.

Opus 5.5 happens to get this one correct in both runs:

"I don't have a record of Scott recommending specific drills. His April 7 video analysis pointed out that your left-foot shots were weak, but he didn't prescribe drills for it. When you asked me on April 10 what to do..."

In our appendix runs with Gemini 3.8 Flash answering, the agent gets this question wrong.

If a preference drops out of the note, the agent never searches for it again

In a 10M chat, the user mentions a preference of choosing Udemy over edX for an online course. It does survive four reflection calls. The chat continues for 79 more reflections. During this entire rest of the run, this preference only exists in archives.

The Udemy note is in the notes only for rewrites 6 to 10, out of 89. The archive keeps it the whole time.

Later, BEAM asks the agent to recommend one of the course platforms. The agent does read its note. It doesn't see anything about a platform preference and hence doesn't even know that there is a preference to begin with, and just answers right away. It never searches for anything and just says that the recommendation would be MIT Open Courseware, a reasonable pick for this user, but incorrect nonetheless.

This is the core trade-off you make for a system to be fast and cheap while maintaining accuracy. A pure retrieval system might be able to find details like this, but would waste an inordinate amount of compute in looking through mostly irrelevant data every turn. More curated context systems, by design, will miss some of these preference-style details sometimes, since when something falls out, especially something where there is no indication it was there in the first place, nothing tells the agent that it went missing.

Methodology

We ran Observational Memory the same way a Mastra agent would run it out-of-the-box.

Using memory as it runs in production

The philosophy behind our design is simple: the scores we get here should represent performance that is replicable in the wild by users with the least amount of friction possible. The current results use the absolute default configuration, with recall turned on.

• We use models that we see being used with Mastra frequently. We don’t use older models for the sake of absolute comparability.
• Other memory systems on the market often run BEAM as 2 separate processes: one for ingesting the conversations and generating memories, and another to search through memory and answer questions. We don’t do these separately, instead relying on a single Mastra agent thread where conversations are fed in just as they would be in production, and then a question is asked.

How we ran BEAM

We wanted to achieve 2 goals with our setup: it needs to be representative of real-world usage of Mastra, and it needs to run BEAM in a valid manner without advantaging or impeding our evaluations compared to other systems.

These were our design decisions for running BEAM:

• Fed the conversation content into a chat: We ran every conversation that BEAM provides into a separate chat. This is how Observational Memory encounters content in the real world.
• We clone chats per question: We create 20 clones of the chats per conversation. One question for that chat is dropped into each clone. We wanted to isolate each question to its own thread so the answers to previous questions don’t interfere with the next question.
• Plain Mastra agent: We used Mastra’s agent primitive as-is, without overriding defaults or reflection limits, etc. This is again to show how Mastra would perform out-of-the-box. There are a ton of optimizations possible, which we would cover in another post.
• Model set:
  • Memory: We used Gemini 3.8 Flash over OpenRouter as observer and reflector models (basically all the models involved in building memory for Observational Memory).
  • Judge: The LLM-as-a-judge model was Gemini 3 Flash. The judge was chosen to be the same as the one the current best performing system, Exabase M-1, uses.
  • Agent: We use Claude Opus 5.5 for answering, and have numbers for what it would look like to use Gemini 3.8 Flash for answering instead in the appendix.
• BEAM scores: They are calculated as the paper describes they should be.

We wanted to ensure that the scores we quote for Mastra are as stable as possible, and are not appearing due to chance events or cherry-picked runs.

We ran the questions for all 3 tiers on the same memory twice and averaged the score to get the final scores we quoted.

Variability between runs and judge models

We did some experiments around how different components of our setup could cause the score to change and by how much. We did this to make sure we’re not reporting biased numbers.

Asking the same questions again doesn’t move the score much

We saved the memories and ran the same questions over them twice and then graded them with the same model.

We got a 1.7 percentage point variation for the 10M tier, while 100K and 1M tiers had variations below a single point.

This is how our official scores are derived above.

The judge model can swing the score drastically

The 400 answers from an earlier experimental BEAM-100K run, scored by seven grader models.

In this experiment, we saved the memories, also saved the answers the agent created, and then gave the same answers to different agents to grade/judge.

The results here are stark, ranging from Gemini 3 Flash judging it to be 82.4% while Claude Sonnet 5 grades this at 71.5%. It seems that generally, stronger models generally give harsher scores on BEAM. Or conversely, using older, weaker models generally gives you a higher score.

This is why we chose Gemini 3 Flash as the judge model, for maintaining comparability against the systems that perform the best currently, since the current SotA, Exabase M-1, also uses this model as judge.

Minor suggestions towards improving BEAM

We believe BEAM is an extremely useful benchmark to measure how well agent memory performs, and we’d like to help make it better.

During our early, experimental runs, we found a few questions that might not be marked with the correct answers. Some examples include:

• BEAM-100K. Abstention question on Chat 18
  • Question: "What specific advice or leadership strategies did Patrick share during the July 1 phone call?"
  • BEAM’s expected answer: the chat has no information on this.
  • User said in a message: "Sure, Patrick mentioned the importance of building trust and being adaptable. He also stressed the value of regular check-ins and open communication. Let's focus on those points.". This makes the expected answer incorrect.
  • Marked 0: grader says “the response includes information that the rubric asserts does not exist in the source material.”
• BEAM-10M, information extraction question on Chat 5
  • Question: "Which version of the tokenization library am I using in my implementation?"
  • Expected answer: “4.35.0 of the Hugging Face Transformers library.”
  • Never shows up: The number 4.35 shows up in an unrelated context, no mention of the strings “token” or “Hugging Face”.
  • Marked 0: grader says "The rubric criterion explicitly requires the LLM to state the version number '4.35.0'. The response fails to provide this information."

While such mistakes might exist, they represent a very small subset of questions in the question set, and thus don't have major implications when it comes to scoring on BEAM.

Limitations

• Different answering models: we kept the memory and grading models in line with other providers, but our answering model differs. We answered with Claude Opus 5.5, while Exabase answered with Gemini 3 Flash. With Gemini 3.8 Flash answering, Observational Memory scores 78.4% at 100K (still the top published score), 72.8% at 1M (behind Exabase M-1 at 75.0% and Hindsight at 73.9%), and 57.8% at 10M. The appendix has the details. Observational Memory works best with frontier answering models, even as the note-takers can be much cheaper ones.
• Very few chats on the 10M tier: given that there are only 10 conversations in the 10M tier, the error margin is wide.
• We did not run BEAM’s 500K tier: Many other providers skipped it, and it seemed a bit redundant to the 1M tier as models have progressed.
• Self-reported scores: We have not independently reproduced or verified any scores for other providers. We take every provider’s first-party claims at their word for our purposes.

Appendix

Running a weaker answering model

We answered the same questions on the same memories with Gemini 3.8 Flash, one run per tier.

The same memories answered by two models. Opus 5.5 vs Gemini 3.8 Flash

With Gemini answering, Observational Memory still has the top published score at 100K. On the same memory, Gemini falls further behind Opus as chats get longer: 4.9 points behind at 100K, 7.8 at 1M and 8.6 at 10M.

Gemini 3.8 Flash minus Opus 5.5, in points, on the same questions.

Most abilities are not losing out too much. The core difference between the two models is for abstention and contradiction resolution.

Score on abstention questions.

Abstention is low because of Misattribution. Gemini actually ended up searching a lot deeper than Opus did on abstention questions. It even opened the original messages behind a note on 92% of abstention questions, compared to Opus's 58%, and pulled in about eight times more of the older text. But it never once said that it itself had suggested some content, and just takes everything it sees without trying to attribute the correct source.

How often the answer says the record contradicts itself, as marked by the grader.

Contradictions: Gemini always picks a winner. When the record says two things that conflict with each other, Opus points out the conflict 96% of the time on the 100K tier, while Gemini only does it 47% of the time. At 10M, Gemini never even sees a contradiction. Interestingly, Gemini was also slower. It took 26 seconds per answer against Opus's 13 at 100K, while also reading twice as many tokens.

Overall, a weaker model also preserves most of the performance, although it would probably serve well to tune the default Mastra agent if used with these models (in case you want to eke out more performance and accuracy). It seems frontier models do much better out of the box by default, but smaller models might need more guidance, especially for more specific use cases and workloads.

Citations

1. Tavakoli, M., et al. "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs." ICLR 2026.
2. Exabase. "Exabase achieves state of the art on BEAM benchmark."
3. Bartholomew, B. "Hindsight Is #1 on BEAM." April 2, 2026.
4. Plastic Labs. "AI memory benchmarks: Honcho on LongMem, LoCoMo, BEAM."
5. Barnes, T. "Observational Memory: 95% on LongMemEval." Mastra Research, 2026.
6. BEAM dataset and evaluation code. github.com/mohammadtavakoli78/BEAM