Tone Consistency Evaluation
Use ToneConsistencyMetric
to evaluate emotional tone patterns and sentiment consistency in text. The metric accepts a query
and a response
, and returns a score and an info
object containing sentiment scores and their difference.
Installation
npm install @mastra/evals
Positive tone example
In this example, the texts exhibit a similar positive sentiment. The metric measures the consistency between the tones, resulting in a high score.
import { ToneConsistencyMetric } from "@mastra/evals/nlp";
const metric = new ToneConsistencyMetric();
const query = "This product is fantastic and amazing!";
const response = "The product is excellent and wonderful!";
const result = await metric.measure(query, response);
console.log(result);
Positive tone output
The metric returns a high score reflecting strong sentiment alignment. The info
field provides sentiment values and the difference between them.
{
score: 0.8333333333333335,
info: {
responseSentiment: 1.3333333333333333,
referenceSentiment: 1.1666666666666667,
difference: 0.16666666666666652
}
}
Stable tone example
In this example, the text’s internal tone consistency is analyzed by passing an empty response. This signals the metric to evaluate sentiment stability within the single input text, resulting in a score reflecting how uniform the tone is throughout.
import { ToneConsistencyMetric } from "@mastra/evals/nlp";
const metric = new ToneConsistencyMetric();
const query = "Great service! Friendly staff. Perfect atmosphere.";
const response = "";
const result = await metric.measure(query, response);
console.log(result);
Stable tone output
The metric returns a high score indicating consistent sentiment throughout the input text. The info
field includes the average sentiment and sentiment variance, reflecting tone stability.
{
score: 0.9444444444444444,
info: {
avgSentiment: 1.3333333333333333,
sentimentVariance: 0.05555555555555556
}
}
Mixed tone example
In this example, the input and response have different emotional tones. The metric picks up on these variations and gives a lower consistency score.
import { ToneConsistencyMetric } from "@mastra/evals/nlp";
const metric = new ToneConsistencyMetric();
const query = "The interface is frustrating and confusing, though it has potential.";
const response = "The design shows promise but needs significant improvements to be usable.";
const result = await metric.measure(query, response);
console.log(result);
Mixed tone output
The metric returns a low score due to the noticeable differences in emotional tone. The info
field highlights the sentiment values and the degree of variation between them.
{
score: 0.4181818181818182,
info: {
responseSentiment: -0.4,
referenceSentiment: 0.18181818181818182,
difference: 0.5818181818181818
}
}
Metric configuration
You can create a ToneConsistencyMetric
instance with default settings. No additional configuration is required.
const metric = new ToneConsistencyMetric();
See ToneConsistencyMetric for a full list of configuration options.
Understanding the results
ToneConsistencyMetric
returns a result in the following shape:
{
score: number,
info: {
responseSentiment?: number,
referenceSentiment?: number,
difference?: number,
avgSentiment?: number,
sentimentVariance?: number
}
}
Tone consistency score
A tone consistency score between 0 and 1:
- 0.8–1.0: Very consistent tone.
- 0.6–0.7: Generally consistent tone.
- 0.4–0.5: Mixed tone.
- 0.0–0.3: Conflicting tone.
Tone consistency info
An explanation for the score, with details including:
- Sentiment alignment between input and response.
- Tone stability within a single text.
- Degree of sentiment difference or variance.