Content Similarity Scorer

The createContentSimilarityScorer() function measures the textual similarity between two strings, providing a score that indicates how closely they match. It supports configurable options for case sensitivity and whitespace handling.

ParametersDirect link to Parameters

The createContentSimilarityScorer() function accepts a single options object with the following properties:

ignoreCase:

boolean

= true

Whether to ignore case differences when comparing strings.

ignoreWhitespace:

boolean

= true

Whether to normalize whitespace when comparing strings.

This function returns an instance of the MastraScorer class. See the MastraScorer reference for details on the .run() method and its input/output.

.run() ReturnsDirect link to .run() Returns

runId:

string

The id of the run (optional).

preprocessStepResult:

object

Object with processed input and output: { processedInput: string, processedOutput: string }

analyzeStepResult:

object

Object with similarity: { similarity: number }

score:

number

Similarity score (0-1) where 1 indicates perfect similarity.

Scoring DetailsDirect link to Scoring Details

The scorer evaluates textual similarity through character-level matching and configurable text normalization.

Scoring ProcessDirect link to Scoring Process

Normalizes text:
- Case normalization (if ignoreCase: true)
- Whitespace normalization (if ignoreWhitespace: true)
Compares processed strings using string-similarity algorithm:
- Analyzes character sequences
- Aligns word boundaries
- Considers relative positions
- Accounts for length differences

Final score: similarity_value * scale

ExampleDirect link to Example

Evaluate textual similarity between expected and actual agent outputs:

src/example-content-similarity.ts
import { runExperiment } from "@mastra/core/scores";
import { createContentSimilarityScorer } from "@mastra/evals/scorers/code";
import { myAgent } from "./agent";

const scorer = createContentSimilarityScorer();

const result = await runExperiment({
  data: [
    {
      input: "Summarize the benefits of TypeScript",
      groundTruth:
        "TypeScript provides static typing, better tooling support, and improved code maintainability.",
    },
    {
      input: "What is machine learning?",
      groundTruth:
        "Machine learning is a subset of AI that enables systems to learn from data without explicit programming.",
    },
  ],
  scorers: [scorer],
  target: myAgent,
  onItemComplete: ({ scorerResults }) => {
    console.log({
      score: scorerResults[scorer.name].score,
      groundTruth: scorerResults[scorer.name].groundTruth,
    });
  },
});

console.log(result.scores);

For more details on runExperiment, see the runExperiment reference.

To add this scorer to an agent, see the Scorers overview guide.