Content Similarity Evaluation
Use ContentSimilarityMetric
to evaluate how similar the response is to a reference based on content overlap. The metric accepts a query
and a response
, and returns a score and an info
object containing a similarity value.
Installation
npm install @mastra/evals
High similarity example
In this example, the response closely resembles the query in both structure and meaning. Minor differences in tense and phrasing do not significantly affect the overall similarity.
import { ContentSimilarityMetric } from "@mastra/evals/nlp";
const metric = new ContentSimilarityMetric();
const query = "The quick brown fox jumps over the lazy dog.";
const response = "A quick brown fox jumped over a lazy dog.";
const result = await metric.measure(query, response);
console.log(result);
High similarity output
The output receives a high score because the response preserves the intent and content of the query with only subtle wording changes.
{
score: 0.7761194029850746,
info: {
similarity: 0.7761194029850746
}
}
Moderate similarity example
In this example, the response shares some conceptual overlap with the query but diverges in structure and wording. Key elements remain present, but the phrasing introduces moderate variation.
import { ContentSimilarityMetric } from "@mastra/evals/nlp";
const metric = new ContentSimilarityMetric();
const query = "A brown fox quickly leaps across a sleeping dog.";
const response = "The quick brown fox jumps over the lazy dog.";
const result = await metric.measure(query, response);
console.log(result);
Moderate similarity output
The output receives a mid-range score because the response captures the general idea of the query, though it differs enough in wording to reduce overall similarity.
{
score: 0.40540540540540543,
info: {
similarity: 0.40540540540540543
}
}
Low similarity example
In this example, the response and query are unrelated in meaning, despite having a similar grammatical structure. There is little to no shared content overlap.
import { ContentSimilarityMetric } from "@mastra/evals/nlp";
const metric = new ContentSimilarityMetric();
const query = "The cat sleeps on the windowsill.";
const response = "The quick brown fox jumps over the lazy dog.";
const result = await metric.measure(query, response);
console.log(result);
Low similarity output
The output receives a low score because the response does not align with the content or intent of the query.
{
score: 0.25806451612903225,
info: {
similarity: 0.25806451612903225
}
}
Metric configuration
You can create a ContentSimilarityMetric
instance with default settings. No additional configuration is required.
const metric = new ContentSimilarityMetric();
See ContentSimilarityMetric for a full list of configuration options.
Understanding the results
ContentSimilarityMetric
returns a result in the following shape:
{
score: number,
info: {
similarity: number
}
}
Similarity score
A similarity score between 0 and 1:
- 1.0: Perfect match – content is nearly identical.
- 0.7–0.9: High similarity – minor differences in word choice or structure.
- 0.4–0.6: Moderate similarity – general overlap with noticeable variation.
- 0.1–0.3: Low similarity – few common elements or shared meaning.
- 0.0: No similarity – completely different content.
Similarity info
An explanation for the score, with details including:
- Degree of overlap between query and response.
- Matching phrases or keywords.
- Semantic closeness based on text similarity.