Evals
Measuring agent quality with scorers, judges and test suites.
18 posts
LLM Evaluation Metrics: What to Measure and Why
Learn about LLM evaluation metrics and how to calculate scores, validate judges, and compare agent experiments.
Sep 28, 2026BlogIntroducing Classifiers with Jev
Evaluation model support for making fast, predictable decisions.
Sep 28, 2026ArticleWhat is Jev and when to use it (with examples)
Where Jev fits in AI agents, where it is less useful, and how to use evaluation models for workflows, guardrails, tool approvals, and evals.
Sep 24, 2026BlogIntroducing Vitest Integration for Testing
Catch regressions and errors before you deploy.
Sep 23, 2026BlogIntroducing Experiment Tool Mocks
Run experiments without hitting live APIs.
Sep 22, 2026BlogIntroducing Experiment Lifecycle Hooks
Configure each stage of an experiment run.
Sep 21, 2026BlogIntroducing Experiment Targets
Run experiments on workflows and scorers too, not just agents.
Aug 24, 2026BlogIntroducing Multi-turn Evals for Mastra Agents
Assert multi-turn conversations using deterministic gates and LLM-as-judge scorers.
Aug 10, 2026ArticleLLM as a judge: how to evaluate LLM and agent outputs at scale
Learn what LLM as a judge is, how to build a reliable LLM judge, avoid bias, and evaluate agent outputs at scale with prompting and calibration methods.
Jul 28, 2026BlogIntroducing Rubric Scorers for Mastra Agents
Evaluate agent output against a rubric of criteria.
Jul 24, 2026ArticleLLM evaluation: frameworks, metrics, methods, and best practices
Learn how to evaluate LLMs with proven metrics, benchmarks, LLM-as-a-judge methods, and agent evaluation strategies for development and production.
Jul 13, 2026BlogIntroducing Gates and Verdicts for Mastra Evals
Catch agent regressions and ship with more confidence.
Jun 22, 2026ArticleHow to build AI agent evaluation that ships reliable agents
Choose metrics, build graders, run evals in CI/CD, and monitor production to catch failures before your users do.
Mar 3, 2026BlogChange, Run, and Compare with Experiments in Mastra Studio
Run test cases against agents and workflows, score the results, and track quality over time.
Feb 19, 2026BlogIntroducing Datasets in Mastra Studio
Define test cases, run experiments, and track response quality for every change you make.
Sep 5, 2025BlogIntroducing Cloneable Scorers
Cloneable scorers let you copy, customize, and extend Mastra’s evals into your project.
Aug 6, 2025BlogIntroducing Scorers in Mastra
We're excited to announce the release of scorers in Mastra, a new way to evaluate and rank the quality of your agent's responses.
Jan 20, 2025BlogIntroducing Evals in Mastra
A guide to Mastra's evaluation metrics suite, demonstrating how to measure and validate AI model outputs across dimensions including answer relevancy, completeness, content similarity, context usage, and tone consistency.
