> Discover all available pages from the documentation index: https://mastra.ai/llms.txt

# Multi-turn Judge scorer

**Added in:** `@mastra/evals@1.9.0`

The `createMultiTurnJudgeScorer()` function creates an LLM-as-judge scorer that grades a whole conversation against a single plain-English criterion. It returns a **binary** score: `1` when the criterion is satisfied, otherwise `0`, and the `reason` echoes the criterion with the judge's explanation.

Unlike the other prebuilt LLM judges, which read a single assistant message, this scorer reads every assistant turn accumulated in `run.output`, so it works with the [multi-turn `inputs`](https://mastra.ai/docs/evals/multi-turn) form of [`runEvals()`](https://mastra.ai/reference/evals/run-evals).

## Parameters

**model** (`MastraModelConfig`): The language model used to grade the conversation. A smaller, cheaper model is usually sufficient for grading.

**criterion** (`string`): What the conversation must satisfy, in plain English, e.g. "The agent gave forecasts for London and Paris, and weather-appropriate packing advice".

**options** (`MultiTurnJudgeScorerOptions`): Configuration options for the scorer

## `.run()` returns

**score** (`number`): 1 when the judge considers the criterion satisfied, otherwise 0 (multiplied by scale).

**reason** (`string`): The verdict, the criterion it graded, and the judge's explanation of why the criterion is or is not satisfied.

## Usage with multi-turn evals

Pass the scorer to `runEvals` alongside an `inputs` array. Every assistant turn is included in the prompt sent to the judge:

```typescript
import { runEvals } from '@mastra/core/evals'
import { createMultiTurnJudgeScorer } from '@mastra/evals/scorers/prebuilt'
import { weatherAgent } from '../agents'

const result = await runEvals({
  data: [
    {
      inputs: [
        "I'm planning a trip to London, Paris, and Tokyo next week.",
        "How's the weather looking in London?",
        'And Paris?',
        'Tokyo too?',
        'Should I pack an umbrella for the London leg?',
      ],
    },
  ],
  target: weatherAgent,
  scorers: [
    {
      scorer: createMultiTurnJudgeScorer({
        model: 'anthropic/claude-haiku-4-5',
        criterion:
          'The agent provided weather forecasts for London, Paris, and Tokyo, and gave weather-appropriate packing or clothing advice.',
      }),
      threshold: 1,
    },
  ],
})
```

Use `threshold: 1` to turn the verdict into a pass or fail: the score is binary, so any lower threshold always passes.

## Persisting scores

Scores are only written to the scores store when a scorer with the same ID is registered on the Mastra instance, because persistence resolves scorer metadata through `Mastra.getScorerById()`. Only the ID is looked up, so the registered instance's `criterion` can be a placeholder:

```typescript
import { Mastra } from '@mastra/core'
import { LibSQLStore } from '@mastra/libsql'
import { createMultiTurnJudgeScorer } from '@mastra/evals/scorers/prebuilt'

export const mastra = new Mastra({
  agents: { weatherAgent },
  storage: new LibSQLStore({ url: 'file:./mastra.db' }),
  scorers: {
    'multi-turn-judge-scorer': createMultiTurnJudgeScorer({
      model: 'anthropic/claude-haiku-4-5',
      criterion: 'placeholder',
    }),
  },
})
```

See [Score persistence](https://mastra.ai/docs/evals/overview) for the full requirement and the warning you get when a scorer isn't registered.

## Scoring details

The scorer runs in two phases:

1. **Grade**: Every assistant message in `run.output` is collected in order and rendered as a numbered transcript, then the judge decides whether the conversation as a whole satisfies the criterion. Assistant messages with no text (a turn that only carried tool calls, for example) are skipped.
2. **Score**: A `satisfied` verdict scores `1` and anything else scores `0`, multiplied by `scale`.

The judge only sees what the assistant said. The user's turns and any tool results aren't included, so write criteria in terms of the agent's responses. This keeps the graded text limited to the agent's own output, but it also means a reply that only makes sense next to the question that prompted it ("Yes, bring one.") can't be judged on its own. For criteria that depend on the user's turns, grade each turn with `turns[].scorers` or write a [custom scorer](https://mastra.ai/docs/evals/multi-turn) that renders both roles.

The transcript is passed to the judge as untrusted data, fenced with explicit delimiters and an instruction to ignore anything inside it that reads as an instruction, so an agent response can't talk its way into a passing verdict.

## Related

- [Multi-turn evals](https://mastra.ai/docs/evals/multi-turn)
- [`runEvals()`](https://mastra.ai/reference/evals/run-evals)
- [Rubric scorer](https://mastra.ai/reference/evals/rubric)
- [createScorer](https://mastra.ai/reference/evals/create-scorer)