Skip to main content

ToneConsistencyMetric

Scorers

This documentation refers to the legacy evals API. For the latest scorer features, see Scorers.

The ToneConsistencyMetric class evaluates the text's emotional tone and sentiment consistency. It can operate in two modes: comparing tone between input/output pairs or analyzing tone stability within a single text.

Basic Usage​

import { ToneConsistencyMetric } from "@mastra/evals/nlp";

const metric = new ToneConsistencyMetric();

// Compare tone between input and output
const result1 = await metric.measure(
  "I love this amazing product!",
  "This product is wonderful and fantastic!",
);

// Analyze tone stability in a single text
const result2 = await metric.measure(
  "The service is excellent. The staff is friendly. The atmosphere is perfect.",
  "", // Empty string for single-text analysis
);

console.log(result1.score); // Tone consistency score from 0-1
console.log(result2.score); // Tone stability score from 0-1

measure() Parameters​

input:

string
The text to analyze for tone

output:

string
Reference text for tone comparison (empty string for stability analysis)

Returns​

score:

number
Tone consistency/stability score (0-1)

info:

object
Detailed tone info

info Object (Tone Comparison)​

responseSentiment:

number
Sentiment score for the input text

referenceSentiment:

number
Sentiment score for the output text

difference:

number
Absolute difference between sentiment scores

info Object (Tone Stability)​

avgSentiment:

number
Average sentiment score across sentences

sentimentVariance:

number
Variance in sentiment between sentences

Scoring Details​

The metric evaluates sentiment consistency through tone pattern analysis and mode-specific scoring.

Scoring Process​

  1. Analyzes tone patterns:

    • Extracts sentiment features
    • Computes sentiment scores
    • Measures tone variations

  2. Calculates mode-specific score: Tone Consistency (input and output):

    • Compares sentiment between texts
    • Calculates sentiment difference
    • Score = 1 - (sentiment_difference / max_difference)

    Tone Stability (single input):

    • Analyzes sentiment across sentences
    • Calculates sentiment variance
    • Score = 1 - (sentiment_variance / max_variance)

Final score: mode_specific_score * scale

Score interpretation​

(0 to scale, default 0-1)

  • 1.0: Perfect tone consistency/stability
  • 0.7-0.9: Strong consistency with minor variations
  • 0.4-0.6: Moderate consistency with noticeable shifts
  • 0.1-0.3: Poor consistency with major tone changes
  • 0.0: No consistency - completely different tones

Example with Both Modes​

import { ToneConsistencyMetric } from "@mastra/evals/nlp";

const metric = new ToneConsistencyMetric();

// Tone Consistency Mode
const consistencyResult = await metric.measure(
  "This product is fantastic and amazing!",
  "The product is excellent and wonderful!",
);
// Example output:
// {
//   score: 0.95,
//   info: {
//     responseSentiment: 0.8,
//     referenceSentiment: 0.75,
//     difference: 0.05
//   }
// }

// Tone Stability Mode
const stabilityResult = await metric.measure(
  "Great service! Friendly staff. Perfect atmosphere.",
  "",
);
// Example output:
// {
//   score: 0.9,
//   info: {
//     avgSentiment: 0.6,
//     sentimentVariance: 0.1
//   }
// }