Skip to Content
ReferenceScorersTextualDifference

Textual Difference Scorer

The createTextualDifferenceScorer() function uses sequence matching to measure the textual differences between two strings. It provides detailed information about changes, including the number of operations needed to transform one text into another.

For a usage example, see the Textual Difference Examples.

Parameters

The createTextualDifferenceScorer() function does not take any options.

This function returns an instance of the MastraScorer class. See the MastraScorer reference for details on the .run() method and its input/output.

.run() Returns

runId:

string
The id of the run (optional).

analyzeStepResult:

object
Object with difference metrics: { confidence: number, changes: number, lengthDiff: number }

score:

number
Similarity ratio (0-1) where 1 indicates identical texts.

Scoring Details

The scorer calculates several measures:

  • Similarity Ratio: Based on sequence matching between texts (0-1)
  • Changes: Count of non-matching operations needed
  • Length Difference: Normalized difference in text lengths
  • Confidence: Inversely proportional to length difference

Scoring Process

  1. Analyzes textual differences:
    • Performs sequence matching between input and output
    • Counts the number of change operations required
    • Measures length differences
  2. Calculates metrics:
    • Computes similarity ratio
    • Determines confidence score
    • Combines into weighted score

Final score: (similarity_ratio * confidence) * scale

Score interpretation

(0 to scale, default 0-1)

  • 1.0: Identical texts - no differences
  • 0.7-0.9: Minor differences - few changes needed
  • 0.4-0.6: Moderate differences - significant changes
  • 0.1-0.3: Major differences - extensive changes
  • 0.0: Completely different texts
KeywordCoverageToneConsistency