Textual Difference Scorer

The createTextualDifferenceScorer() function uses sequence matching to measure the textual differences between two strings. It provides detailed information about changes, including the number of operations needed to transform one text into another.

For a usage example, see the Textual Difference Examples.

Parameters

The createTextualDifferenceScorer() function does not take any options.

This function returns an instance of the MastraScorer class. See the MastraScorer reference for details on the .run() method and its input/output.

.run() Returns

.run() returns a result in the following shape:

{
  runId: string,
  analyzeStepResult: {
    confidence: number,
    ratio: number,
    changes: number,
    lengthDiff: number
  },
  score: number
}

Scoring Details

The scorer calculates several measures:

• Similarity Ratio: Based on sequence matching between texts (0-1)
• Changes: Count of non-matching operations needed
• Length Difference: Normalized difference in text lengths
• Confidence: Inversely proportional to length difference

Scoring Process

1. Analyzes textual differences:
   • Performs sequence matching between input and output
   • Counts the number of change operations required
   • Measures length differences
2. Calculates metrics:
   • Computes similarity ratio
   • Determines confidence score
   • Combines into weighted score

Final score: (similarity_ratio * confidence) * scale

Score interpretation

A textual difference score between 0 and 1:

• 1.0: Identical texts – no differences detected.
• 0.7–0.9: Minor differences – few changes needed.
• 0.4–0.6: Moderate differences – noticeable changes required.
• 0.1–0.3: Major differences – extensive changes needed.
• 0.0: Completely different texts.

Examples

No differences example

In this example, the texts are exactly the same. The scorer identifies complete similarity with a perfect score and no detected changes.

import { createTextualDifferenceScorer } from "@mastra/evals/scorers/code";
 
const scorer = createTextualDifferenceScorer();
 
const input = 'The quick brown fox jumps over the lazy dog';
const output = 'The quick brown fox jumps over the lazy dog';
 
const result = await scorer.run({
  input: [{ role: 'user', content: input }],
  output: { role: 'assistant', text: output },
});
 
console.log('Score:', result.score);
console.log('AnalyzeStepResult:', result.analyzeStepResult);

No differences output

The scorer returns a high score, indicating the texts are identical. The detailed info confirms zero changes and no length difference.

{
  score: 1,
  analyzeStepResult: {
    confidence: 1,
    ratio: 1,
    changes: 0,
    lengthDiff: 0,
  },
}

Minor differences example

In this example, the texts have small variations. The scorer detects these minor differences and returns a moderate similarity score.

import { createTextualDifferenceScorer } from "@mastra/evals/scorers/code";
 
const scorer = createTextualDifferenceScorer();
 
const input = 'Hello world! How are you?';
const output = 'Hello there! How is it going?';
 
const result = await scorer.run({
  input: [{ role: 'user', content: input }],
  output: { role: 'assistant', text: output },
});
 
console.log('Score:', result.score);
console.log('AnalyzeStepResult:', result.analyzeStepResult);

Minor differences output

The scorer returns a moderate score reflecting the small variations between the texts. The detailed info includes the number of changes and length difference observed.

{
  score: 0.5925925925925926,
  analyzeStepResult: {
    confidence: 0.8620689655172413,
    ratio: 0.5925925925925926,
    changes: 5,
    lengthDiff: 0.13793103448275862
  }
}

Major differences example

In this example, the texts differ significantly. The scorer detects extensive changes and returns a low similarity score.

import { createTextualDifferenceScorer } from "@mastra/evals/scorers/code";
 
const scorer = createTextualDifferenceScorer();
 
const input = 'Python is a high-level programming language';
const output = 'JavaScript is used for web development';
 
const result = await scorer.run({
  input: [{ role: 'user', content: input }],
  output: { role: 'assistant', text: output },
});
 
console.log('Score:', result.score);
console.log('AnalyzeStepResult:', result.analyzeStepResult);

Major differences output

The scorer returns a low score due to significant differences between the texts. The detailed analyzeStepResult shows numerous changes and a notable length difference.

{
  score: 0.3170731707317073,
  analyzeStepResult: {
    confidence: 0.8636363636363636,
    ratio: 0.3170731707317073,
    changes: 8,
    lengthDiff: 0.13636363636363635
  }
}

Related

• Content Similarity Scorer
• Completeness Scorer
• Keyword Coverage Scorer