You can now assert a whole conversation with multi-turn evals. Gates assert deterministic actions like tool calls, and scorers can use an LLM-as-judge to grade the conversational context for accuracy.
runEvals accepts both gates and scorers:
gates: can use quick checks likecalledTool,includes, orexcludesto return a0or1result.scorers: can use a multi-turn compatible built-in scorer like the multi-turn judge to grade the conversation and return ascoreandreason.
You can also use quick checks with scorers, to test if a message includes a certain word. runEvals can be used for per-turn, and multi-turn evaluation simultaneously,
Before multi-turn evals, grading a conversation meant setting up gates and scorers per-turn and stitching the results together to reason about the whole exchange. With multi-turn, the judge uses a criterion to score the whole conversation, and gates can be used to bulk-assert deterministic actions, e.g. checks.calledTool("get_weather", { times: 3 }).
To use multi-turn evals, the agent requires a valid memory configuration so each turn can observe prior runs.
Get started
Install the required Mastra core and evals packages:
npm install @mastra/core @mastra/evals@mastra/core@1.61.0 or later, added in PR #21930.Multi-turn assertions take an inputs array, and gates apply to the whole run. Give the judge scorer a criterion to grade the conversation and an optional threshold to be met:
import { runEvals } from "@mastra/core/evals";
import { checks } from "@mastra/evals/checks";
import { createMultiTurnJudgeScorer } from "@mastra/evals/scorers/prebuilt";
import { travelAgent } from "../agents/travel-agent";
const result = await runEvals({
target: travelAgent,
data: [
{
inputs: [
"I'm planning a 3-city trip next week — London, Paris, and Tokyo.",
"How's the weather looking in London?",
"And Paris?"
// ...
]
}
],
gates: [checks.calledTool("get_weather", { times: 3 })],
scorers: [
{
scorer: createMultiTurnJudgeScorer({
model: "anthropic/claude-haiku-4-5",
criterion: "The agent provided weather forecasts for London, Paris, and Tokyo, and gave weather-appropriate packing or clothing advice."
}),
threshold: 1
}
],
onItemComplete: ({ scorerResults }) => {
const judge = scorerResults?.["multi-turn-judge-scorer"];
if (judge?.reason) console.log(judge.reason);
}
});Per-turn assertions can be configured using a turns array where each turn declares its own gates and scorers, and an optional threshold to be met:
const result = await runEvals({
// ...
data: [
{
turns: [
{ input: "I'm planning a 3-city trip next week — London, Paris, and Tokyo." },
{
input: "How's the weather looking in London?",
gates: [checks.calledTool("get_weather", { times: 1 })],
scorers: [{ scorer: checks.includes("London"), threshold: 1 }]
},
{
input: "And Paris?",
gates: [checks.calledTool("get_weather", { times: 1 })],
scorers: [{ scorer: checks.includes("Paris"), threshold: 1 }]
}
// ...
]
}
]
});For more information and full configuration options, see:
