DocsReferenceEvalsTextualDifference

TextualDifferenceMetric

The TextualDifferenceMetric class uses sequence matching to measure the textual differences between two strings. It provides detailed information about changes, including the number of operations needed to transform one text into another.

Basic Usage

import { TextualDifferenceMetric } from "@mastra/evals/nlp";
 
const metric = new TextualDifferenceMetric();
 
const result = await metric.measure(
  "The quick brown fox",
  "The fast brown fox"
);
 
console.log(result.score); // Similarity ratio from 0-1
console.log(result.info); // Detailed change metrics

measure() Parameters

input:

string
The original text to compare against

output:

string
The text to evaluate for differences

Returns

score:

number
Similarity ratio (0-1) where 1 indicates identical texts

info:

Detailed metrics about the differences
number

confidence:

number
Confidence score based on length difference between texts (0-1)
number

ratio:

number
Raw similarity ratio between the texts
number

changes:

number
Number of change operations (insertions, deletions, replacements)
number

lengthDiff:

number
Normalized difference in length between input and output (0-1)

Scoring Details

The metric calculates several measures:

  • Similarity Ratio: Based on sequence matching between texts (0-1)
  • Changes: Count of non-matching operations needed
  • Length Difference: Normalized difference in text lengths
  • Confidence: Inversely proportional to length difference

Scoring Process

  1. Analyzes textual differences:

    • Performs sequence matching between input and output
    • Counts the number of change operations required
    • Measures length differences

  2. Calculates metrics:

    • Computes similarity ratio
    • Determines confidence score
    • Combines into weighted score

Final score: (similarity_ratio * confidence) * scale

Score interpretation

(0 to scale, default 0-1)

  • 1.0: Identical texts - no differences
  • 0.7-0.9: Minor differences - few changes needed
  • 0.4-0.6: Moderate differences - significant changes
  • 0.1-0.3: Major differences - extensive changes
  • 0.0: Completely different texts

Example with Analysis

import { TextualDifferenceMetric } from "@mastra/evals/nlp";
 
const metric = new TextualDifferenceMetric();
 
const result = await metric.measure(
  "Hello world! How are you?",
  "Hello there! How is it going?"
);
 
// Example output:
// {
//   score: 0.65,
//   info: {
//     confidence: 0.95,
//     ratio: 0.65,
//     changes: 2,
//     lengthDiff: 0.05
//   }
// }

Related

SummarizationToneConsistency