Supported evals in Mastra

Mastra provides several eval metrics for assessing Agent outputs. Mastra is not limited to these metrics, and you can also define your own evals.

Accuracy and Reliability

• hallucination: Detects fabricated or unsupported information
• faithfulness: Checks output alignment with source material
• content-similarity: Compares text similarity
• textual-difference: Measures text changes
• completeness: Measures if all required information is present
• answer-relevancy: Measures how well an answer addresses the input question

Understanding Context

• context-position: Evaluates the placement of context in responses
• context-precision: Assesses the accuracy of context usage
• context-relevancy: Measures the relevance of used context
• contextual-recall: Evaluates information recall from context

Output Quality

• tone: Analyzes writing style and tone
• toxicity: Detects harmful or inappropriate content
• bias: Detects potential biases in the output
• prompt-alignment: Measures adherence to prompt instructions
• summarization: Evaluates summary quality
• keyword-coverage: Checks for presence of key terms