Multi-turn evals

Multi-turn evals test agent behavior across a conversation. Instead of a single input, you provide an inputs array. Each entry is sent sequentially to the agent on the same thread, and scorers see the accumulated output from all turns.

When to use multi-turn evals

• The agent uses memory and must recall context from earlier turns
• The agent handles follow-up questions that depend on prior responses
• You need to verify tool-call sequences across a conversation
• The agent runs a multi-step workflow (search, confirm, execute)

Quickstart

src/evals/conversation-eval.ts

import { runEvals } from '@mastra/core/evals'
import { checks } from '@mastra/evals/checks'
import { weatherAgent } from '../agents'

const result = await runEvals({
  data: [
    {
      inputs: [
        'What is the weather in Brooklyn?',
        'What about tomorrow?',
        'Compare the two forecasts.',
      ],
    },
  ],
  target: weatherAgent,
  scorers: [checks.calledTool('get_weather', { times: 2 }), checks.includes('Brooklyn')],
})

Each turn runs agent.generate() with the same thread ID, so the agent sees the full conversation history. Scorers receive the accumulated output messages from all turns.

Memory is required for cross-turn recall

Multi-turn recall depends on the agent having a memory store configured. The shared thread ID is what lets each turn see the earlier ones — but a thread only persists history when the agent has memory. If the agent has no memory configured, the turns still run sequentially and their outputs still accumulate for scoring, but the agent will not recall earlier turns (each input runs in isolation). runEvals logs a warning when you use inputs on an agent without memory.

runEvals manages the conversation identity for you: it generates the shared threadId and injects a resourceId (Mastra memory scopes messages by resource + thread, so both are required for recall to work). By default the resource is derived from the generated thread so each conversation is isolated. To pin a specific resource — for example to reuse an existing user's memory — pass targetOptions.memory.resource; runEvals still owns the thread, so you don't provide one:

await runEvals({
  target: weatherAgent,
  data: [{ inputs: ['What is the weather in Brooklyn?', 'What about tomorrow?'] }],
  scorers: [checks.similarity('weather forecast')],
  targetOptions: { memory: { resource: 'user-42' } },
})

See Memory for how to configure a memory store.

How it works

When a data item has an inputs array, runEvals:

1. Creates a fresh thread (unique threadId) and resource for the conversation (a caller-provided targetOptions.memory.resource is preserved)
2. Sends each input sequentially via agent.generate() on that thread
3. Accumulates all output messages across turns
4. Passes the full accumulated output to scorers for evaluation

Scorers see the complete conversation output, not just the last turn.

Scoring semantics

Two details matter when writing scorers for multi-turn items:

• run.output is the accumulated output from every turn. Output-based scorers — checks.includes, checks.calledTool, checks.similarity, and similar — evaluate the whole conversation. For example, checks.calledTool('get_weather', { times: 2 }) counts calls across all turns.
• run.input is only the first turn's input. Scorers that compare input against output (faithfulness, answer relevancy, and other input-relative LLM scorers) only see the first user message, not the full conversation. Prefer output-based checks for multi-turn, or build scorers that read the accumulated run.output directly.

Per-turn assertions with turns

The inputs form scores the accumulated output holistically — a single score over every turn's output. That can hide per-turn failures: an output-based check like checks.includes('Brooklyn') passes if any turn mentions Brooklyn, even when the follow-up turn is broken.

When you need to assert that a specific turn behaved correctly, use turns instead. Each turn is an object with its own input and optional gates/scorers that evaluate only that turn's input and output:

src/evals/per-turn-eval.ts

import { runEvals } from '@mastra/core/evals'
import { checks } from '@mastra/evals/checks'
import { weatherAgent } from '../agents'

const result = await runEvals({
  data: [
    {
      turns: [
        {
          input: 'What is the weather in Brooklyn?',
          gates: [checks.calledTool('get_weather')],
        },
        {
          // The follow-up must call the tool again — it can't be satisfied
          // by the first turn's tool call.
          input: 'What about tomorrow?',
          gates: [checks.calledTool('get_weather')],
          scorers: [{ scorer: checks.similarity('tomorrow forecast'), threshold: 0.5 }],
        },
      ],
    },
  ],
  target: weatherAgent,
})

result.verdict // 'passed' | 'scored' | 'failed'
result.turnResults // per-turn gate/threshold/scorer outcomes

Semantics:

• A per-turn gate or scorer sees only that turn's run.input and run.output — never the accumulated conversation. This fixes both blind spots of inputs: the wrong turn can't satisfy a check, and run.input is correct for each turn.
• Per-turn outcomes fold into the overall verdict: a failing turn gate makes the verdict failed; a missed turn threshold (with gates passing) makes it scored.
• result.turnResults[i] reports each turn's gateResults, thresholdResults, and scores, so a failure points at the exact turn. Across multiple conversations, turn results are averaged by turn index.
• A turn with no gates/scorers just advances the conversation.
• Top-level scorers/gates still run holistically over the accumulated output, so you can combine "this turn must call the tool" with "the overall answer mentions Brooklyn."
• When the agent has storage configured, each per-turn scorer/gate result is persisted like top-level scores, so per-turn outcomes appear in your scores store.

Use inputs when a single holistic score over the whole conversation is enough; use turns when correctness depends on individual turns. turns cannot be combined with input or inputs in the same data item.

Combining with gates and thresholds

Multi-turn data items work with gates and verdicts. Use output-based scorers so gating reflects the full conversation:

src/evals/memory-eval.ts

import { runEvals } from '@mastra/core/evals'
import { checks } from '@mastra/evals/checks'

const result = await runEvals({
  data: [
    {
      inputs: ['My favorite city is Brooklyn.', 'What is the weather in my favorite city?'],
    },
  ],
  target: weatherAgent,
  gates: [checks.calledTool('get_weather')],
  scorers: [{ scorer: checks.similarity('Brooklyn weather forecast'), threshold: 0.5 }],
})

result.verdict // 'passed' | 'scored' | 'failed'

Mixing single-turn and multi-turn

A single runEvals call can include both single-turn and multi-turn data items:

src/evals/mixed-eval.ts

const result = await runEvals({
  data: [
    { input: 'What is the weather in Brooklyn?' },
    {
      inputs: ['My favorite city is Brooklyn.', 'What is the weather in my favorite city?'],
    },
  ],
  target: weatherAgent,
  scorers: [checks.includes('Brooklyn')],
})

Single-turn items use input as usual. Multi-turn items use inputs — input can be omitted entirely.

Validation

runEvals throws a MastraError if inputs is present but empty:

// Throws: 'inputs' must be a non-empty array
await runEvals({
  data: [{ inputs: [] }],
  target: myAgent,
  scorers: [myScorer],
})

Related

• runEvals() reference: Full API for runEvals parameters and returns
• Gates and verdicts: Enforce hard requirements and quality thresholds
• Quick Checks: Zero-LLM composable micro-scorers