Multi-turn Judge scorer

Added in: @mastra/evals@1.9.0

The createMultiTurnJudgeScorer() function creates an LLM-as-judge scorer that grades a whole conversation against a single plain-English criterion. It returns a binary score: 1 when the criterion is satisfied, otherwise 0, and the reason echoes the criterion with the judge's explanation.

Unlike the other prebuilt LLM judges, which read a single assistant message, this scorer reads every assistant turn accumulated in run.output, so it works with the multi-turn inputs form of runEvals().

Parameters

model:

MastraModelConfig

The language model used to grade the conversation. A smaller, cheaper model is usually sufficient for grading.

criterion:

string

What the conversation must satisfy, in plain English, e.g. "The agent gave forecasts for London and Paris, and weather-appropriate packing advice".

options:

MultiTurnJudgeScorerOptions

Configuration options for the scorer

.run() returns

score:

number

1 when the judge considers the criterion satisfied, otherwise 0 (multiplied by scale).

reason:

string

The verdict, the criterion it graded, and the judge's explanation of why the criterion is or is not satisfied.

Usage with multi-turn evals

Pass the scorer to runEvals alongside an inputs array. Every assistant turn is included in the prompt sent to the judge:

src/evals/trip-planning-eval.ts

import { runEvals } from '@mastra/core/evals'
import { createMultiTurnJudgeScorer } from '@mastra/evals/scorers/prebuilt'
import { weatherAgent } from '../agents'

const result = await runEvals({
  data: [
    {
      inputs: [
        "I'm planning a trip to London, Paris, and Tokyo next week.",
        "How's the weather looking in London?",
        'And Paris?',
        'Tokyo too?',
        'Should I pack an umbrella for the London leg?',
      ],
    },
  ],
  target: weatherAgent,
  scorers: [
    {
      scorer: createMultiTurnJudgeScorer({
        model: 'anthropic/claude-haiku-4-5',
        criterion:
          'The agent provided weather forecasts for London, Paris, and Tokyo, and gave weather-appropriate packing or clothing advice.',
      }),
      threshold: 1,
    },
  ],
})

Use threshold: 1 to turn the verdict into a pass or fail: the score is binary, so any lower threshold always passes.

Persisting scores

Scores are only written to the scores store when a scorer with the same ID is registered on the Mastra instance, because persistence resolves scorer metadata through Mastra.getScorerById(). Only the ID is looked up, so the registered instance's criterion can be a placeholder:

src/mastra/index.ts

import { Mastra } from '@mastra/core'
import { LibSQLStore } from '@mastra/libsql'
import { createMultiTurnJudgeScorer } from '@mastra/evals/scorers/prebuilt'

export const mastra = new Mastra({
  agents: { weatherAgent },
  storage: new LibSQLStore({ url: 'file:./mastra.db' }),
  scorers: {
    'multi-turn-judge-scorer': createMultiTurnJudgeScorer({
      model: 'anthropic/claude-haiku-4-5',
      criterion: 'placeholder',
    }),
  },
})

See Score persistence for the full requirement and the warning you get when a scorer isn't registered.

Scoring details

The scorer runs in two phases:

1. Grade: Every assistant message in run.output is collected in order and rendered as a numbered transcript, then the judge decides whether the conversation as a whole satisfies the criterion. Assistant messages with no text (a turn that only carried tool calls, for example) are skipped.
2. Score: A satisfied verdict scores 1 and anything else scores 0, multiplied by scale.

The judge only sees what the assistant said. The user's turns and any tool results aren't included, so write criteria in terms of the agent's responses. This keeps the graded text limited to the agent's own output, but it also means a reply that only makes sense next to the question that prompted it ("Yes, bring one.") can't be judged on its own. For criteria that depend on the user's turns, grade each turn with turns[].scorers or write a custom scorer that renders both roles.

The transcript is passed to the judge as untrusted data, fenced with explicit delimiters and an instruction to ignore anything inside it that reads as an instruction, so an agent response can't talk its way into a passing verdict.

Related

• Multi-turn evals
• runEvals()
• Rubric scorer
• createScorer