Tone Consistency Scorer
Use createToneConsistencyScorer
to evaluate emotional tone patterns and sentiment consistency in text.
Installation
npm install @mastra/evals
For complete API documentation and configuration options, see
createToneScorer
.
Positive tone example
In this example, the texts exhibit a similar positive sentiment. The scorer measures the consistency between the tones, resulting in a high score.
import { createToneScorer } from "@mastra/evals/scorers/code";
const scorer = createToneScorer();
const input = 'This product is fantastic and amazing!';
const output = 'The product is excellent and wonderful!';
const result = await scorer.run({
input: [{ role: 'user', content: input }],
output: { role: 'assistant', text: output },
});
console.log('Score:', result.score);
console.log('AnalyzeStepResult:', result.analyzeStepResult);
Positive tone output
The scorer returns a high score reflecting strong sentiment alignment. The analyzeStepResult
field provides sentiment values and the difference between them.
{
score: 0.8333333333333335,
analyzeStepResult: {
responseSentiment: 1.3333333333333333,
referenceSentiment: 1.1666666666666667,
difference: 0.16666666666666652,
},
}
Stable tone example
In this example, the text’s internal tone consistency is analyzed by passing an empty response. This signals the scorer to evaluate sentiment stability within the single input text, resulting in a score reflecting how uniform the tone is throughout.
import { createToneScorer } from "@mastra/evals/scorers/code";
const scorer = createToneScorer();
const input = 'Great service! Friendly staff. Perfect atmosphere.';
const output = '';
const result = await scorer.run({
input: [{ role: 'user', content: input }],
output: { role: 'assistant', text: output },
});
console.log('Score:', result.score);
console.log('AnalyzeStepResult:', result.analyzeStepResult);
Stable tone output
The scorer returns a high score indicating consistent sentiment throughout the input text. The analyzeStepResult
field includes the average sentiment and sentiment variance, reflecting tone stability.
{
score: 0.9444444444444444,
analyzeStepResult: {
avgSentiment: 1.3333333333333333,
sentimentVariance: 0.05555555555555556,
},
}
Mixed tone example
In this example, the input and response have different emotional tones. The scorer picks up on these variations and gives a lower consistency score.
import { createToneScorer } from "@mastra/evals/scorers/code";
const scorer = createToneScorer();
const input = 'The interface is frustrating and confusing, though it has potential.';
const output = 'The design shows promise but needs significant improvements to be usable.';
const result = await scorer.run({
input: [{ role: 'user', content: input }],
output: { role: 'assistant', text: output },
});
console.log('Score:', result.score);
console.log('AnalyzeStepResult:', result.analyzeStepResult);
Mixed tone output
The scorer returns a low score due to the noticeable differences in emotional tone. The analyzeStepResult
field highlights the sentiment values and the degree of variation between them.
{
score: 0.4181818181818182,
analyzeStepResult: {
responseSentiment: -0.4,
referenceSentiment: 0.18181818181818182,
difference: 0.5818181818181818,
},
}
Scorer configuration
You can create a ToneConsistencyScorer
instance with default settings. No additional configuration is required.
const scorer = createToneScorer();
See ToneConsistencyScorer for a full list of configuration options.
Understanding the results
.run()
returns a result in the following shape:
{
runId: string,
analyzeStepResult: {
responseSentiment?: number,
referenceSentiment?: number,
difference?: number,
avgSentiment?: number,
sentimentVariance?: number,
},
score: number
}
score
A tone consistency score between 0 and 1:
- 0.8–1.0: Very consistent tone.
- 0.6–0.7: Generally consistent tone.
- 0.4–0.5: Mixed tone.
- 0.0–0.3: Conflicting tone.
runId
The unique identifier for this scorer run.
analyzeStepResult
Object with tone metrics:
- responseSentiment: Sentiment score for the response (comparison mode).
- referenceSentiment: Sentiment score for the input/reference (comparison mode).
- difference: Absolute difference between sentiment scores (comparison mode).
- avgSentiment: Average sentiment across sentences (stability mode).
- sentimentVariance: Variance of sentiment across sentences (stability mode).