Content similarity scorer

The createContentSimilarityScorer() function measures the textual similarity between two strings, providing a score that indicates how closely they match. It supports configurable options for case sensitivity and whitespace handling.

Parameters

The createContentSimilarityScorer() function accepts a single options object with the following properties:

ignoreCase:

boolean

= true

Whether to ignore case differences when comparing strings. Defaults to true only when no options object is passed. If you pass an options object, an omitted value is treated as false.

ignoreWhitespace:

boolean

= true

Whether to normalize whitespace when comparing strings. Defaults to true only when no options object is passed. If you pass an options object, an omitted value is treated as false.

This function returns an instance of the MastraScorer class. See the MastraScorer reference for details on the .run() method and its input/output.

.run() returns

runId:

string

The id of the run.

preprocessStepResult:

object

Object with processed input and output: { processedInput: string, processedOutput: string }

score:

number

Similarity score (0-1) where 1 indicates perfect similarity.

Scoring details

The scorer evaluates textual similarity through character-level matching and configurable text normalization.

Scoring Process

1. Normalizes text:
   • Case normalization (if ignoreCase: true)
   • Whitespace normalization (if ignoreWhitespace: true)
2. Compares processed strings using string-similarity algorithm:
   • Analyzes character sequences
   • Aligns word boundaries
   • Considers relative positions
   • Accounts for length differences

Final score: similarity_value (0-1, returned directly from the string-similarity comparison)

Example

Evaluate textual similarity between expected and actual agent outputs:

src/example-content-similarity.ts

import { runEvals } from '@mastra/core/evals'
import { createContentSimilarityScorer } from '@mastra/evals/scorers/prebuilt'
import { myAgent } from './agent'

const scorer = createContentSimilarityScorer()

const result = await runEvals({
  data: [
    {
      input: 'Summarize the benefits of TypeScript',
      groundTruth:
        'TypeScript provides static typing, better tooling support, and improved code maintainability.',
    },
    {
      input: 'What is machine learning?',
      groundTruth:
        'Machine learning is a subset of AI that enables systems to learn from data without explicit programming.',
    },
  ],
  scorers: [scorer],
  target: myAgent,
  onItemComplete: ({ scorerResults }) => {
    console.log({
      score: scorerResults[scorer.id].score,
      groundTruth: scorerResults[scorer.id].groundTruth,
    })
  },
})

console.log(result.scores)

For more details on runEvals, see the runEvals reference.

To add this scorer to an agent, see the Scorers overview guide.

Score interpretation

A similarity score between 0 and 1:

• 1.0: Perfect match, content is nearly identical.
• 0.7 to 0.9: High similarity, minor differences in word choice or structure.
• 0.4 to 0.6: Moderate similarity, general overlap with noticeable variation.
• 0.1 to 0.3: Low similarity, few common elements or shared meaning.
• 0.0: No similarity, completely different content.

Related

• Completeness Scorer
• Textual Difference Scorer
• Answer Relevancy Scorer
• Keyword Coverage Scorer