Skip to main content

Content Similarity Scorer

The createContentSimilarityScorer() function measures the textual similarity between two strings, providing a score that indicates how closely they match. It supports configurable options for case sensitivity and whitespace handling.

Parameters​

The createContentSimilarityScorer() function accepts a single options object with the following properties:

ignoreCase:

boolean
= true
Whether to ignore case differences when comparing strings.

ignoreWhitespace:

boolean
= true
Whether to normalize whitespace when comparing strings.

This function returns an instance of the MastraScorer class. See the MastraScorer reference for details on the .run() method and its input/output.

.run() Returns​

runId:

string
The id of the run (optional).

preprocessStepResult:

object
Object with processed input and output: { processedInput: string, processedOutput: string }

analyzeStepResult:

object
Object with similarity: { similarity: number }

score:

number
Similarity score (0-1) where 1 indicates perfect similarity.

Scoring Details​

The scorer evaluates textual similarity through character-level matching and configurable text normalization.

Scoring Process​

  1. Normalizes text:
    • Case normalization (if ignoreCase: true)
    • Whitespace normalization (if ignoreWhitespace: true)
  2. Compares processed strings using string-similarity algorithm:
    • Analyzes character sequences
    • Aligns word boundaries
    • Considers relative positions
    • Accounts for length differences

Final score: similarity_value * scale

Example​

Evaluate textual similarity between expected and actual agent outputs:

src/example-content-similarity.ts
import { runExperiment } from "@mastra/core/scores";
import { createContentSimilarityScorer } from "@mastra/evals/scorers/code";
import { myAgent } from "./agent";

const scorer = createContentSimilarityScorer();

const result = await runExperiment({
  data: [
    {
      input: "Summarize the benefits of TypeScript",
      groundTruth:
        "TypeScript provides static typing, better tooling support, and improved code maintainability.",
    },
    {
      input: "What is machine learning?",
      groundTruth:
        "Machine learning is a subset of AI that enables systems to learn from data without explicit programming.",
    },
  ],
  scorers: [scorer],
  target: myAgent,
  onItemComplete: ({ scorerResults }) => {
    console.log({
      score: scorerResults[scorer.name].score,
      groundTruth: scorerResults[scorer.name].groundTruth,
    });
  },
});

console.log(result.scores);

For more details on runExperiment, see the runExperiment reference.

To add this scorer to an agent, see the Scorers overview guide.

On this page