MastraScorer

The MastraScorer class is the base class for all scorers in Mastra. It provides a standard .run() method for evaluating input/output pairs and supports multi-step scoring workflows with preprocess → analyze → generateScore → generateReason execution flow.

Note: Most users should use createScorer to create scorer instances. Direct instantiation of MastraScorer isn't recommended.

How to get a MastraScorer instance

Use the createScorer factory function, which returns a MastraScorer instance:

import { createScorer } from '@mastra/core/evals'

const scorer = createScorer({
  name: 'My Custom Scorer',
  description: 'Evaluates responses based on custom criteria',
}).generateScore(({ run, results }) => {
  // scoring logic
  return 0.85
})

// scorer is now a MastraScorer instance

.run() method

The .run() method is the primary way to execute your scorer and evaluate input/output pairs. It processes the data through your defined steps (preprocess → analyze → generateScore → generateReason) and returns a comprehensive result object with the score, reasoning, and intermediate results.

const result = await scorer.run({
  input: 'What is machine learning?',
  output: 'Machine learning is a subset of artificial intelligence...',
  runId: 'optional-run-id',
  requestContext: {
    /* optional context */
  },
})

.run() input

input:

any

Input data to be evaluated. Can be any type depending on your scorer's requirements.

output:

any

Output data to be evaluated. Can be any type depending on your scorer's requirements.

runId:

string

Optional unique identifier for this scoring run.

requestContext:

any

Optional request context from the agent or workflow step being evaluated.

groundTruth:

any

Optional expected or reference output for comparison during scoring. Automatically passed when using runEvals.

.run() returns

runId:

string

The unique identifier for this scoring run.

score:

number

Numerical score computed by the generateScore step.

reason:

string

Explanation for the score, if generateReason step was defined (optional).

preprocessStepResult:

any

Result of the preprocess step, if defined (optional).

analyzeStepResult:

any

Result of the analyze step, if defined (optional).

preprocessPrompt:

string

Preprocess prompt, if defined (optional).

analyzePrompt:

string

Analyze prompt, if defined (optional).

generateScorePrompt:

string

Generate score prompt, if defined (optional).

generateReasonPrompt:

string

Generate reason prompt, if defined (optional).

Step execution flow

When you call .run(), the MastraScorer executes the defined steps in this order:

1. preprocess (optional) - Extracts or transforms data
2. analyze (optional) - Processes the input/output and preprocessed data
3. generateScore (required) - Computes the numerical score
4. generateReason (optional) - Provides explanation for the score

Each step receives the results from previous steps, allowing you to build complex evaluation pipelines.

Usage example

const scorer = createScorer({
  name: 'Quality Scorer',
  description: 'Evaluates response quality',
})
  .preprocess(({ run }) => {
    // Extract key information
    return { wordCount: run.output.split(' ').length }
  })
  .analyze(({ run, results }) => {
    // Analyze the response
    const hasSubstance = results.preprocessStepResult.wordCount > 10
    return { hasSubstance }
  })
  .generateScore(({ results }) => {
    // Calculate score
    return results.analyzeStepResult.hasSubstance ? 1.0 : 0.0
  })
  .generateReason(({ score, results }) => {
    // Explain the score
    const wordCount = results.preprocessStepResult.wordCount
    return `Score: ${score}. Response has ${wordCount} words.`
  })

// Use the scorer
const result = await scorer.run({
  input: 'What is machine learning?',
  output: 'Machine learning is a subset of artificial intelligence...',
})

console.log(result.score) // 1.0
console.log(result.reason) // "Score: 1.0. Response has 12 words."

Integration

MastraScorer instances can be used for agents and workflow steps

See the createScorer reference for detailed information on defining custom scoring logic.