Skip to Content
DocsScorersOverview

Scorers overview

Scorers are evaluation tools that measure the quality, accuracy, or performance of AI-generated outputs. Scorers provide an automated way to assess whether your agents, workflows, or language models are producing the desired results by analyzing their responses against specific criteria.

Scores are numerical values (typically between 0 and 1) that quantify how well an output meets your evaluation criteria. These scores enable you to objectively track performance, compare different approaches, and identify areas for improvement in your AI systems.

Evaluation pipeline

Mastra scorers follow an optional three-step pipeline that allows for evaluation workflows:

  1. Extract (Optional): Identify and isolate relevant content for focused evaluation
  2. Analyze (Required): Perform the core evaluation and generate a score
  3. Reason (Optional): Provide explanations or justifications for the score

This modular structure enables both simple single-step evaluations and complex multi-stage analysis workflows, allowing you to build evaluations that match your specific needs.

When to use each step

Extract step - Use when your content is complex or needs preprocessing:

  • Separating facts from opinions in mixed responses
  • Focusing evaluation on specific sections of long outputs
  • Parsing multiple claims that need individual evaluation
  • Example: Bias detection that first identifies opinion statements

Analyze step - Always required for core evaluation:

  • Straightforward scenarios: Direct scoring of input/output pairs
  • Complex scenarios: Evaluate preprocessed content and generate detailed results
  • Applies your scoring criteria and calculates the numerical score

Reason step - Use when explanations are important:

  • Users need to understand why a score was assigned
  • Debugging and transparency are critical
  • Compliance or auditing requires explanations
  • Providing actionable feedback for improvement

To learn how to create your own Scorers, see Creating Custom Scorers.

Live evaluations

Live evaluations allow you to automatically score AI outputs in real-time as your agents and workflows operate. Instead of running evaluations manually or in batches, scorers run asynchronously alongside your AI systems, providing continuous quality monitoring.

Adding scorers to agents

You can add built-in scorers to your agents to automatically evaluate their outputs. See the full list of built-in scorers for all available options.

src/mastra/agents/evaluated-agent.ts
import { Agent } from "@mastra/core/agent";
import { openai } from "@ai-sdk/openai";
import { 
  createAnswerRelevancyScorer,
  createToxicityScorer 
} from "@mastra/evals/scorers/llm";
 
export const evaluatedAgent = new Agent({
  // ...
  scorers: {
    relevancy: {
      scorer: createAnswerRelevancyScorer({ model: openai("gpt-4o-mini") }),
      sampling: { type: "ratio", rate: 0.5 }
    },
    safety: {
      scorer: createToxicityScorer({ model: openai("gpt-4o-mini") }),
      sampling: { type: "ratio", rate: 1 }
    }
  }
});

Adding scorers to workflow steps

You can also add scorers to individual workflow steps to evaluate outputs at specific points in your process:

src/mastra/workflows/content-generation.ts
import { createWorkflow, createStep } from "@mastra/core/workflows";
import { z } from "zod";
import { customStepScorer } from "../scorers/custom-step-scorer";
 
const contentStep = createStep({
  // ...
  scorers: {
    customStepScorer: {
      scorer: customStepScorer(),
      sampling: {
        type: "ratio",
        rate: 1, // Score every step execution
      }
    }
  },
});
 
export const contentWorkflow = createWorkflow({ ... })
  .then(contentStep)
  .commit();

How live evaluations work

Asynchronous execution: Live evaluations run in the background without blocking your agent responses or workflow execution. This ensures your AI systems maintain their performance while still being monitored.

Sampling control: The sampling.rate parameter (0-1) controls what percentage of outputs get scored:

  • 1.0: Score every single response (100%)
  • 0.5: Score half of all responses (50%)
  • 0.1: Score 10% of responses
  • 0.0: Disable scoring

Automatic storage: All scoring results are automatically stored in the mastra_scorers table in your configured database, allowing you to analyze performance trends over time.

Testing scorers locally

Mastra provides a CLI command mastra dev to test your scorers. The playground includes a scorers section where you can run individual scorers against test inputs and view detailed results.

For more details, see the Local Dev Playground docs.

Next steps

Running in CIOff the Shelf Scorers