createScorer

Mastra allows you to define your own custom code scorers for evaluating input/output pairs using any logic you choose. Custom scorers integrate seamlessly with the Mastra scoring framework and can be used anywhere built-in scorers are used.

For a usage example, see the Custom Code Scorer Examples.

How to Create a Custom Scorer

Use the createScorer factory to define your scorer. You must provide at least a name, description, and an analyze function. Optionally, you can provide extract and reason functions for multi-step or more advanced logic.

createScorer Options

name:

string

Name of the scorer.

description:

string

Description of what the scorer does.

analyze:

function

Main scoring logic

extract:

function

Optional pre-processing step.

reason:

function

Optional reason/explanation step.

metadata:

object

Optional metadata for the scorer.

This function returns an instance of the MastraScorer class. See the MastraScorer reference for details on the .run() method and its input/output.

Step Function Signatures

extract

input:

Record<string, any>[]

Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. `[{ role: 'user', content: 'hello world' }]`. If the scorer is used in a workflow, this will be the input of the workflow.

output:

Record<string, any>

Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.

Returns: { results: any }
The method must return an object with a results property. The value of results will be passed to the analyze function as extractStepResult.

analyze

input:

Record<string, any>[]

output:

Record<string, any>

Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.

extractStepResult:

object

Result of the extract step, if defined (optional).

Returns: { score: number, results?: any }
The method must return an object with a score property (required). Optionally, it may return a results property. The value of results will be passed to the reason function as analyzeStepResult.

reason

input:

Record<string, any>[]

output:

Record<string, any>

Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.

score:

number

Score computed by the analyze step.

analyzeStepResult:

object

Result of the analyze step.

extractStepResult:

object

Result of the extract step, if defined (optional).

Returns: { reason: string }
The method must return an object with a reason property, which should be a string explaining the score.

All step functions can be async.