Skip to Content
ExamplesEvalsCompleteness

Completeness Evaluation

This example demonstrates how to use Mastra’s Completeness metric to evaluate how thoroughly responses cover key elements from the input.

Overview

The example shows how to:

  1. Configure the Completeness metric
  2. Evaluate responses for element coverage
  3. Analyze coverage scores
  4. Handle different coverage scenarios

Setup

Dependencies

Import the necessary dependencies:

src/index.ts
import { CompletenessMetric } from '@mastra/evals/nlp';

Metric Configuration

Set up the Completeness metric:

src/index.ts
const metric = new CompletenessMetric();

Example Usage

Complete Coverage Example

Evaluate a response that covers all elements:

src/index.ts
const text1 = 'The primary colors are red, blue, and yellow.';
const reference1 = 'The primary colors are red, blue, and yellow.';
 
console.log('Example 1 - Complete Coverage:');
console.log('Text:', text1);
console.log('Reference:', reference1);
 
const result1 = await metric.measure(reference1, text1);
console.log('Metric Result:', {
  score: result1.score,
  info: {
    missingElements: result1.info.missingElements,
    elementCounts: result1.info.elementCounts,
  },
});
// Example Output:
// Metric Result: { score: 1, info: { missingElements: [], elementCounts: { input: 8, output: 8 } } }

Partial Coverage Example

Evaluate a response that covers some elements:

src/index.ts
const text2 = 'The primary colors are red and blue.';
const reference2 = 'The primary colors are red, blue, and yellow.';
 
console.log('Example 2 - Partial Coverage:');
console.log('Text:', text2);
console.log('Reference:', reference2);
 
const result2 = await metric.measure(reference2, text2);
console.log('Metric Result:', {
  score: result2.score,
  info: {
    missingElements: result2.info.missingElements,
    elementCounts: result2.info.elementCounts,
  },
});
// Example Output:
// Metric Result: { score: 0.875, info: { missingElements: ['yellow'], elementCounts: { input: 8, output: 7 } } }

Minimal Coverage Example

Evaluate a response that covers very few elements:

src/index.ts
const text3 = 'The seasons include summer.';
const reference3 = 'The four seasons are spring, summer, fall, and winter.';
 
console.log('Example 3 - Minimal Coverage:');
console.log('Text:', text3);
console.log('Reference:', reference3);
 
const result3 = await metric.measure(reference3, text3);
console.log('Metric Result:', {
  score: result3.score,
  info: {
    missingElements: result3.info.missingElements,
    elementCounts: result3.info.elementCounts,
  },
});
// Example Output:
// Metric Result: {
//   score: 0.3333333333333333,
//   info: {
//     missingElements: [ 'four', 'spring', 'winter', 'be', 'fall', 'and' ],
//     elementCounts: { input: 9, output: 4 }
//   }
// }

Understanding the Results

The metric provides:

  1. A score between 0 and 1:

    • 1.0: Complete coverage - contains all input elements
    • 0.7-0.9: High coverage - includes most key elements
    • 0.4-0.6: Partial coverage - contains some key elements
    • 0.1-0.3: Low coverage - missing most key elements
    • 0.0: No coverage - output lacks all input elements

  2. Detailed analysis of:

    • List of input elements found
    • List of output elements matched
    • Missing elements from input
    • Element count comparison




View Example on GitHub
BiasContent Similarity