Decision Models Get Context Rot Too

We benchmarked Jev, Clef, and Luna Decisions against reasoning models. When we added up to 26k tokens of padding, we found that average accuracy declined from 68% to 62% for Jev and from 64% to 56% for Luna Decisions, while Clef and Gemini 3.8 Flash held around 70% and 95% respectively on 400 questions from public datasets and our own generated tasks.

Previous research has shown that context rot is an issue for LLMs. What our research shows is that decision models struggle with this too, especially when the context window is stuffed with things that look like the answer.

What is a decision model?

Decision models recently emerged as a much cheaper alternative to traditional LLMs for calibrated decisions. Instead of generating text from text, we give them a question with a set of answers, and they generate probabilities for how likely each answer is as per the context they're given. They're used for cheap decision-making for straightforward actions like workflow routing, security checks before tool use and browser automation.

While LLM context windows are quite long, decision models generally feature quite small context windows, like Jev's 32k window. Of the models we tested, Luna Decisions is the only one with a context window comparable to LLMs, at approximately 1M tokens. These windows can easily get filled with extraneous data like previous messages and tool outputs.

Thus, we wanted to see how well these models hold up as the context grows.

How we created our tasks

Our tasks aim to test a mix of skills, ranging from spotting small changes within a large corpus of data to connecting multiple entities to derive an answer.

We had a certain set of properties in mind when we made the tasks:

1. The tasks should be saturated by at least one reasoning LLM. This establishes that the tasks are solvable and sets an upper bound on capability, since we want to demonstrate how decision models would perform if they replace this more expensive model. This LLM ended up being Gemini 3.8 Flash for us, which averaged 96% with no padding (ticket counting, at 76%, was the exception).
2. The tasks should not be written directly by LLMs. We built scripts that generate the tasks deterministically and such that their provenance can be understood and they have the properties that we want.

Each set has 50 tasks. Banking77 and BoolQ are based off of public datasets (BoolQ from HuggingFace, Banking77 from PolyAI's GitHub). The other six sets are generated via a script along with a fixed seed, so the task generation process is deterministic.

Padding

We employed two types of padding:

1. Distraction padding: Padding content that looks like it could be relevant to the answer/content, but isn't. Banking77 customer messages are used as such for the stacked sets and the card-retention needle set. Banking77 and BoolQ use other messages and passages from their own datasets.
2. Filler padding: Ticket counting, counterfactual capitals and plain office lookup use either fictional logs or notes built from a set of 5 sentences which look like "The amber bowl rests underneath a linen cover."

The task is placed according to its shape.

We aimed for about the context limit of Jev for our maximum length.

How we ran our tasks

The models

• Decision Models: Jev, Clef, Clef-flash, GPT-6 Luna Decisions
• Reasoning LLMs: Gemini 3.8 Flash, GPT-6 Luna (thinking on)
• Non-reasoning LLMs: Qwen3.8-27B, GPT-6 Luna (thinking off)

Decision Models

Decision models were run through their providers' APIs (GPT-6 Luna Decisions via OpenRouter). We send the tasks as-is:

{
  "model": "jev-1.13.0",
  "state": "<context>",
  "questions": {
    "answer": { "type": "choice", "instructions": "<the question>", "criteria": { "<option>": "<description>", ... } }
  }
}

No separate prompt is needed for decision models.

LLM runner

The LLMs ran through a basic Mastra agent:

• One fixed prompt which asks the LLM to answer from the text, give a probability for yes/no questions and a set of probabilities that add up to 1 for multiple choice questions.
• The context and question as a message.
• A strict JSON schema: - For yes/no questions: {"noul": a number from 0 to 1} - For multiple choice questions: a choice which is one out of all possible options provided as enums + a probability from 0 to 1 for every option. The 77 probabilities an LLM returns often don't add up to 1 (a JSON schema can't enforce that they sum to 1), so we rescale them when needed.

What we found

At a glance, no trends seem very obvious beyond Gemini nearly saturating the tasks and Jev and Luna Decisions being the fastest models. Looking deeper provides more interesting clues.

Gemini was by far the most accurate model, and maintained accuracy even with padding.

Decision models are weaker to begin with, and mostly degrade as padding is added, with Clef being the best performer in accuracy across the breadth of tasks. Decision models perform quite similarly to single-pass (non-reasoning) LLMs.

Plain office lookup is completely saturated. Decision models completely lose out on stacked banking and stacked office, even with no padding.

Jev was the cheapest model. Gemini cost 33 times as much as Jev, with much higher accuracy (94% vs 62% at full padding).

Clef seems to be an outlier in the decision model class, as it cost more than every LLM except Gemini.

Looking deeper, there are a few key takeaways.

Decision models get context rot much more steeply than reasoning models

Since thinking models overperform and decision models get near-zero or coin-flip scores on the stacked sets and ticket counting (we explore this later), we exclude those sets so we compare context rot only on tasks the models could solve before the context grew.

Every decision model lost a significant amount of accuracy after padding was added.

Reasoning models held steady, with almost no loss in accuracy.

Interestingly, non-reasoning LLMs seem to perform on par with decision models on our tasks, suggesting that thinking is a major component of success here.

This effect can be seen in isolation in Banking77.

It seems that in decision models, context rot can begin early. Clef's window is 64k but it shows a 14-point loss on Banking77 with only about 40% of its window full.

Non-reasoning LLMs show context rot quite similar to decision models.

Decision models especially struggled when the padding looked similar to the answer

• For unrelated text: We look at Plain Office Lookup, since the padding is unrelated to the problem and questions.
• For padding that looked like the answer: We use Card-Retention Needle. One case is hidden within a lot of closed customer cases, which look similar to the target.

Decision models mostly maintain accuracy when we add padding that is unrelated to the question being asked. This points towards these models being able to ignore irrelevant context and focus on relevant context.

• With unrelated padding: 7 out of 8 models kept scores of 94% or higher at 22k tokens of padding. The models found these tasks easy.
• With padding that looked like the answer, Jev, Clef and Luna Decisions lost significant accuracy, and Clef-flash answered "yes" to every task.
• Banking77 shows a similar pattern to Card-Retention Needle, and it too has padding that looked like the answer.

Decision models do badly on multi-step reasoning

The stacked sets have chains of facts, some of which void each other. Models need to be able to interpret the final state correctly to score.

This is the starkest victory for thinking models. Gemini makes no mistakes at all. Jev and Luna Decisions score near zero on Stacked Banking, padded or not.

At a glance, it seems Clef did great on Stacked Banking, but those tasks list 77 options and only 2 of those are correct. Clef-flash picked the same option on 99 of 100 runs, and Clef leaned the same way, hence their scores near 50%.

Thinking doesn't necessarily saturate the benchmark either, as Luna (thinking)'s drop from 100% to 50% on stacked office shows.

Decision models are currently poor at multi-step reasoning tasks. They'd be unsuitable for tasks where they might have to put together multiple pieces of evidence to form an answer.

Decision models have pretty good AUROC scores

AUROC is a score for how good the predictive power of a classifier is. A better AUROC score means the classifier ranks correct answers above wrong ones across all its answers. A score of 0.5 is a coin flip.

Three of the four decision models, led by Clef, outrank every LLM on this metric. As an example, when Jev was 90% sure or more, it was right 89% of the time, the best of any decision model.

Jev also made the fewest confident mistakes of any decision model. It had 52 wrong answers at 0.9 or higher, compared to Clef-flash's 324.

We would caveat this with the fact that LLMs state their confidence as numbers in their JSON answer rather than producing probabilities natively, so the process is not as native to these models. The core idea behind generating calibrated decisions seems to work well.

Appendix

Reproducing the results

The task generators, model runner and scoring script are open source at mastra-ai/decision-model-context-rot. Running pnpm run generate rebuilds the exact 1,100 requests every model received, and score/score.py reproduces the accuracy, significance, AUROC, cost and latency figures in this post from saved runs.

Statistical significance

We call a change significant when its 95% bootstrap confidence interval excludes zero, using paired resampling of the tasks. With 50 tasks per set, changes under about 10 points usually aren't significant.

Token counting

Our tokens are tokenized in cl100k_base, so token counts might differ from how other models, such as Jev, tokenize. Parity was attempted, with approximately 26k cl100k_base reaching Jev's 32k limit.

Limitations

• Every input stayed under Jev's 32k-token limit, so this measures context rot in short-to-moderate contexts, not 100k-token prompts.
• Each set has 50 tasks, and each model answered each task once.
• Hosted models change over time, so a new run may not give identical answers.