Using Evaluation Metrics in Mastra
Evaluation metrics help measure and validate AI model outputs across different dimensions. Here's a comprehensive guide to Mastra's evaluation suite. All of these metrics output a score between 0 and 1.
We are starting with some NLP-based metrics, and will add additional LLM-as-judgemetrics soon.
Core Evaluation Metrics
Model Configuration
First, set up the model configuration needed for several metrics:
const model = "openai/gpt-6.1-sol";Answer Relevancy
Answer relevancy evaluates if responses address queries appropriately:
import { createAnswerRelevancyScorer } from "@mastra/evals/scorers/prebuilt";
import { createAgentTestRun, createTestMessage } from "@mastra/evals/scorers/utils";
const scorer = createAnswerRelevancyScorer({
model,
options: {
uncertaintyWeight: 0.3,
scale: 10,
},
});
const result = await scorer.run(
createAgentTestRun({
inputMessages: [createTestMessage({ role: "user", content: "What is the capital of France?" })],
output: [createTestMessage({ role: "assistant", content: "Paris is the capital of France." })],
}),
);This metric uses an LLM to judge how well responses address queries, scoring yes/no/unsure verdicts with uncertainty weighting. Score is calculated as (relevancyCount / totalVerdicts) * scale, where "unsure" counts as 0.3.
Completeness
Completeness measures how thoroughly a response covers the key elements from the input:
import { createCompletenessScorer } from "@mastra/evals/scorers/prebuilt";
import { createAgentTestRun, createTestMessage } from "@mastra/evals/scorers/utils";
const scorer = createCompletenessScorer();
const result = await scorer.run(
createAgentTestRun({
inputMessages: [
createTestMessage({
role: "user",
content: "Explain the water cycle: evaporation, condensation, precipitation",
}),
],
output: [
createTestMessage({
role: "assistant",
content:
"Water evaporates from surfaces, forms clouds through condensation, and returns as precipitation",
}),
],
}),
);Specifically, this metric extracts and compares key elements (nouns, verbs, topics) between input and output using NLP. Score represents the ratio of matched elements to total elements, with intelligent partial matching for longer words.
Content Similarity
This metric performs direct string comparison using a similarity library, with configurable case and whitespace sensitivity. Perfect matches score 1.0, with decreasing scores based on string differences.
import { createContentSimilarityScorer } from "@mastra/evals/scorers/prebuilt";
import { createAgentTestRun, createTestMessage } from "@mastra/evals/scorers/utils";
const scorer = createContentSimilarityScorer({
ignoreCase: true,
ignoreWhitespace: true,
});
const result = await scorer.run(
createAgentTestRun({
inputMessages: [createTestMessage({ role: "user", content: "The quick brown fox" })],
output: [createTestMessage({ role: "assistant", content: "the Quick Brown fox" })],
}),
);Context Precision
Context precision measures accurate use of provided context by calculating precision at each relevant position in the response. Score is normalized sum of precision values at relevant positions divided by number of relevant items.
import { createContextPrecisionScorer } from "@mastra/evals/scorers/prebuilt";
import { createAgentTestRun, createTestMessage } from "@mastra/evals/scorers/utils";
const scorer = createContextPrecisionScorer({
model,
options: {
scale: 10,
context: [
"Research showed 45% improvement",
"Results were statistically significant",
],
},
});
const result = await scorer.run(
createAgentTestRun({
inputMessages: [createTestMessage({ role: "user", content: "What did the research find?" })],
output: [createTestMessage({ role: "assistant", content: "The study found significant improvements" })],
}),
);Difference
Difference metric calculates detailed text differences:
import { createTextualDifferenceScorer } from "@mastra/evals/scorers/prebuilt";
import { createAgentTestRun, createTestMessage } from "@mastra/evals/scorers/utils";
const scorer = createTextualDifferenceScorer();
const result = await scorer.run(
createAgentTestRun({
inputMessages: [createTestMessage({ role: "user", content: "Original text version" })],
output: [createTestMessage({ role: "assistant", content: "Modified text version" })],
}),
);
// Provides ratio, number of changes, and length differencesKeyword Coverage
Extracts and compares keywords between input and output using a keyword extraction library. Score is simply matched keywords divided by total keywords from input.
import { createKeywordCoverageScorer } from "@mastra/evals/scorers/prebuilt";
import { createAgentTestRun, createTestMessage } from "@mastra/evals/scorers/utils";
const scorer = createKeywordCoverageScorer();
const result = await scorer.run(
createAgentTestRun({
inputMessages: [createTestMessage({ role: "user", content: "Explain photosynthesis: chlorophyll, sunlight, glucose" })],
output: [createTestMessage({ role: "assistant", content: "Plants use chlorophyll to convert sunlight into glucose" })],
}),
);Prompt Alignment
Uses an LLM to check adherence to specific instructions with binary yes/no scoring. Final score is the ratio of followed instructions to total instructions, scaled to range.
import { createPromptAlignmentScorerLLM } from "@mastra/evals/scorers/prebuilt";
import { createAgentTestRun, createTestMessage } from "@mastra/evals/scorers/utils";
const scorer = createPromptAlignmentScorerLLM({
model,
options: {
scale: 10,
},
});
const result = await scorer.run(
createAgentTestRun({
systemMessages: [
{ role: "system", content: "Use formal language" },
{ role: "system", content: "Include specific examples" },
{ role: "system", content: "Stay under 100 words" },
],
inputMessages: [createTestMessage({ role: "user", content: "Describe quantum computing" })],
output: [createTestMessage({ role: "assistant", content: "Quantum computing uses quantum bits..." })],
}),
);Tone Consistency
Analyzes sentiment consistency either between input/output or within sentences of a single text. Score is based on sentiment difference or variance, with 1.0 indicating perfect consistency.
import { createToneScorer } from "@mastra/evals/scorers/prebuilt";
import { createAgentTestRun, createTestMessage } from "@mastra/evals/scorers/utils";
const scorer = createToneScorer();
const result = await scorer.run(
createAgentTestRun({
inputMessages: [createTestMessage({ role: "user", content: "Write a positive product review" })],
output: [createTestMessage({ role: "assistant", content: "This product exceeded my expectations!" })],
}),
);
// Measures sentiment stability and alignmentCombining Metrics
Combine metrics for more assessment:
import {
createAnswerRelevancyScorer,
createContextPrecisionScorer,
createPromptAlignmentScorerLLM,
createCompletenessScorer,
createContentSimilarityScorer,
createTextualDifferenceScorer,
createKeywordCoverageScorer,
createToneScorer,
} from "@mastra/evals/scorers/prebuilt";
import { createAgentTestRun, createTestMessage } from "@mastra/evals/scorers/utils";
async function evaluateResponse({
input,
output,
context,
instructions,
}: {
input: string;
output: string;
context?: string[];
instructions?: string[];
}) {
const model = "openai/gpt-6.1-sol";
const scorers = [
createAnswerRelevancyScorer({ model }),
createCompletenessScorer(),
createContentSimilarityScorer(),
createTextualDifferenceScorer(),
createKeywordCoverageScorer(),
createPromptAlignmentScorerLLM({ model }),
createToneScorer(),
];
// Context Precision requires a non-empty context array, so only add it when
// the caller actually retrieved something.
if (context?.length) {
scorers.push(createContextPrecisionScorer({ model, options: { context } }));
}
const run = createAgentTestRun({
systemMessages: (instructions || []).map((instruction) => ({
role: "system" as const,
content: instruction,
})),
inputMessages: [createTestMessage({ role: "user", content: input })],
output: [createTestMessage({ role: "assistant", content: output })],
});
const results = await Promise.all(scorers.map((scorer) => scorer.run(run)));
return results;
}