Summarization scorer

The createSummarizationScorer() function creates a scorer that evaluates a summary on two axes: whether every claim it makes is supported by the source text, and whether it preserves the information the source states. The final score is the lower of the two, so a summary cannot pass by being faithful but empty, or thorough but wrong.

The summary is the agent's last message that carries text, and the source text defaults to the first user message of the run input. Pass source or sourceExtractor when the text being summarized lives somewhere else, such as a tool result.

Usage example

Score a summary against the document it condenses.

src/mastra/scorers/summarization.ts

import { createSummarizationScorer } from '@mastra/evals/scorers/prebuilt'

const scorer = createSummarizationScorer({
  model: 'openai/gpt-5.6-sol',
})

const result = await scorer.run({
  input: {
    inputMessages: [{ id: '1', role: 'user', content: sourceDocument }],
  },
  output: [{ id: '2', role: 'assistant', content: summary }],
})

console.log(result.score)
console.log(result.reason)

Summarization evaluation

Use this scorer when an agent condenses text:

• Document and transcript summarization
• Support thread and email digests
• Any step that compresses a long input into a short output

Parameters

model:

MastraModelConfig

The language model to use for judging claims and coverage questions

options?:

SummarizationMetricOptions

Configuration options for the scorer

SummarizationMetricOptions

source?:

string

Text the summary is judged against. Defaults to the user message of the run input

sourceExtractor?:

(input, output) => string

Function to derive the source text from the run input and output. Takes precedence over source

maxQuestions?:

number

Upper bound on the coverage questions drawn from the source (default: 10)

scale?:

number

Scale factor to multiply the final score (default: 1)

.run() returns

score:

number

Summarization score between 0 and scale (default 0-1), the lower of the alignment and coverage scores

reason:

string

Human-readable explanation naming the axis that produced the score and the claims or questions behind it. Both axis scores appear in the text

preprocessStepResult:

object

The alignment verdicts and the questions drawn from the source

object

alignment:

{ claim: string; supported: boolean; reason: string }[]

One verdict per claim the summary makes

questions:

string[]

The coverage questions drawn from the source text

analyzeStepResult:

object

The coverage verdicts

object

coverage:

{ question: string; answered: boolean; reason: string }[]

One verdict per question, answered from the summary alone

The axis scores are derived from these verdicts rather than stored: alignment is the share of alignment entries with supported: true, and coverage is the share of questions whose coverage entry has answered: true.

Scoring details

Two-axis evaluation

The scorer runs a three-step pipeline:

1. Source judgement: the claims the summary makes are extracted and checked against the source, and closed-ended questions are drawn from the source. Every question is written so the source answers it "yes".
2. Coverage: each question is answered using the summary alone.
3. Scoring: the two ratios are computed and the lower one becomes the score.

The coverage step runs as a separate model call that never receives the source text. A judge that could see the source would answer questions from it rather than from the summary, which would hide the omissions this axis exists to measure.

Scoring formula

Alignment    = supported_claims / total_claims
Coverage     = answered_questions / total_questions
Summarization = min(Alignment, Coverage) × scale

The score is 0 when the summary yields no claims or the source yields no questions.

Score interpretation

These ranges assume the default scale of 1. When using a custom scale, multiply accordingly.

• 0.9-1.0: Excellent summary, faithful to the source and covering its main points
• 0.7-0.8: Good summary with a small omission or an unsupported detail
• 0.4-0.6: Moderate summary, either missing significant information or drifting from the source
• 0.1-0.3: Poor summary, most of the source is lost or contradicted
• 0.0: The summary supports no claims, answers no questions, or produced nothing to judge

Reading the two axes

Both axes leave their verdicts on the run result: the alignment verdicts on the preprocess step, and the coverage verdicts on the analyze step. Each verdict carries the claim or question it belongs to and the reason behind it. The two failure modes look different:

• A low alignment score with high coverage means the summary invents or distorts detail
• A low coverage score with high alignment means the summary is accurate but leaves too much out

The reason field names whichever axis produced the score.

What the score leaves out

Length plays no part in the score. A summary that repeats the source word for word supports every claim and answers every question, so it scores 1. Add a length check of your own when compression is part of what you're testing.

Cost

Each evaluation makes three model calls. maxQuestions bounds the coverage half of the work, which otherwise grows with source length. Raise it for long documents where ten questions cannot represent the content.

Scorer configuration

Summarizing the run input

const scorer = createSummarizationScorer({
  model: 'openai/gpt-5.6-sol',
})

Summarizing a document from elsewhere

import { extractToolResults } from '@mastra/evals/scorers/utils'

const scorer = createSummarizationScorer({
  model: 'openai/gpt-5.6-sol',
  options: {
    sourceExtractor: (input, output) => {
      return extractToolResults(output)
        .filter(({ toolName }) => toolName === 'fetchDocument')
        .map(({ result }) => String(result))
        .join('\n\n')
    },
    maxQuestions: 20,
  },
})

Example

Evaluate a summarization agent against a set of documents:

src/example-summarization.ts

import { runEvals } from '@mastra/core/evals'
import { createSummarizationScorer } from '@mastra/evals/scorers/prebuilt'
import { summarizerAgent } from './agent'

const scorer = createSummarizationScorer({
  model: 'openai/gpt-5.6-sol',
  options: { maxQuestions: 10 },
})

const result = await runEvals({
  target: summarizerAgent,
  scorers: [scorer],
  data: [
    {
      input:
        'The company was founded in 1995 by John Smith. It started with 10 employees and grew to 500 by 2020. The company is based in Seattle.',
    },
  ],
  onItemComplete: ({ scorerResults }) => {
    console.log({
      score: scorerResults[scorer.id].score,
      reason: scorerResults[scorer.id].reason,
    })
  },
})

console.log(result.scores)

For more details on runEvals, see the runEvals reference.

To add this scorer to an agent, see the Scorers overview guide.

Comparison with faithfulness

Use case	Summarization	Faithfulness
What it measures	Support and coverage together	Support only
Judged against	The source text being condensed	Retrieved context or tool results
Catches omission	Yes	No
Needs the full source	Yes	No, context alone is enough

Use faithfulness when the question is whether an answer stays grounded in retrieved context. Use summarization when the output is meant to stand in for a longer text.

Related

• Faithfulness Scorer: Measures answer groundedness in context
• Completeness Scorer: Compares element coverage without a model
• Content Similarity Scorer: Compares text similarity without a model
• Custom Scorers: Creating your own evaluation metrics