Tone Consistency Scorer

Use createToneConsistencyScorer to evaluate emotional tone patterns and sentiment consistency in text.

Installation

npm install @mastra/evals

For complete API documentation and configuration options, see createToneScorer.

Positive tone example

In this example, the texts exhibit a similar positive sentiment. The scorer measures the consistency between the tones, resulting in a high score.

import { createToneScorer } from "@mastra/evals/scorers/code";
 
const scorer = createToneScorer();
 
const input = 'This product is fantastic and amazing!';
const output = 'The product is excellent and wonderful!';
 
const result = await scorer.run({
  input: [{ role: 'user', content: input }],
  output: { role: 'assistant', text: output },
});
 
console.log('Score:', result.score);
console.log('AnalyzeStepResult:', result.analyzeStepResult);

Positive tone output

The scorer returns a high score reflecting strong sentiment alignment. The analyzeStepResult field provides sentiment values and the difference between them.

{
  score: 0.8333333333333335,
  analyzeStepResult: {
    responseSentiment: 1.3333333333333333,
    referenceSentiment: 1.1666666666666667,
    difference: 0.16666666666666652,
  },
}

Stable tone example

In this example, the text’s internal tone consistency is analyzed by passing an empty response. This signals the scorer to evaluate sentiment stability within the single input text, resulting in a score reflecting how uniform the tone is throughout.

import { createToneScorer } from "@mastra/evals/scorers/code";
 
const scorer = createToneScorer();
 
const input = 'Great service! Friendly staff. Perfect atmosphere.';
const output = '';
 
const result = await scorer.run({
  input: [{ role: 'user', content: input }],
  output: { role: 'assistant', text: output },
});
 
console.log('Score:', result.score);
console.log('AnalyzeStepResult:', result.analyzeStepResult);

Stable tone output

The scorer returns a high score indicating consistent sentiment throughout the input text. The analyzeStepResult field includes the average sentiment and sentiment variance, reflecting tone stability.

{
  score: 0.9444444444444444,
  analyzeStepResult: {
    avgSentiment: 1.3333333333333333,
    sentimentVariance: 0.05555555555555556,
  },
}

Mixed tone example

In this example, the input and response have different emotional tones. The scorer picks up on these variations and gives a lower consistency score.

import { createToneScorer } from "@mastra/evals/scorers/code";
 
const scorer = createToneScorer();
 
const input = 'The interface is frustrating and confusing, though it has potential.';
const output = 'The design shows promise but needs significant improvements to be usable.';
 
const result = await scorer.run({
  input: [{ role: 'user', content: input }],
  output: { role: 'assistant', text: output },
});
 
console.log('Score:', result.score);
console.log('AnalyzeStepResult:', result.analyzeStepResult);

Mixed tone output

The scorer returns a low score due to the noticeable differences in emotional tone. The analyzeStepResult field highlights the sentiment values and the degree of variation between them.

{
  score: 0.4181818181818182,
  analyzeStepResult: {
    responseSentiment: -0.4,
    referenceSentiment: 0.18181818181818182,
    difference: 0.5818181818181818,
  },
}

Scorer configuration

You can create a ToneConsistencyScorer instance with default settings. No additional configuration is required.

const scorer = createToneScorer();

See ToneConsistencyScorer for a full list of configuration options.

Understanding the results

.run() returns a result in the following shape:

{
  runId: string,
  analyzeStepResult: {
    responseSentiment?: number,
    referenceSentiment?: number,
    difference?: number,
    avgSentiment?: number,
    sentimentVariance?: number,
  },
  score: number
}

score

A tone consistency score between 0 and 1:

• 0.8–1.0: Very consistent tone.
• 0.6–0.7: Generally consistent tone.
• 0.4–0.5: Mixed tone.
• 0.0–0.3: Conflicting tone.

runId

The unique identifier for this scorer run.

analyzeStepResult

Object with tone metrics:

• responseSentiment: Sentiment score for the response (comparison mode).
• referenceSentiment: Sentiment score for the input/reference (comparison mode).
• difference: Absolute difference between sentiment scores (comparison mode).
• avgSentiment: Average sentiment across sentences (stability mode).
• sentimentVariance: Variance of sentiment across sentences (stability mode).