Scorers

A file-based agent discovers scorers from its scorers/ directory. Use this page for the file-based convention; use the scorers guide for built-in scorers, custom scorers, and sampling.

Each file under scorers/ default-exports one scorer, and the filename becomes its key. The default export is either a MastraScorer or a { scorer, sampling } entry.

Quickstart

Add a scorer by placing a file under the agent's scorers/ directory. This example uses a custom scorer:

src/mastra/agents/weather/scorers/relevance.ts

import { createScorer } from '@mastra/core/evals'

export default createScorer({
  id: 'relevance',
  description: 'Scores how relevant the response is',
}).generateScore(() => 1)

The scorer is registered on the weather agent under the key relevance, the same as adding it to config.scorers.

Sampling

To control how often a scorer runs, default-export a { scorer, sampling } entry instead of a bare scorer:

src/mastra/agents/weather/scorers/relevance.ts

import { createScorer } from '@mastra/core/evals'

const relevance = createScorer({
  id: 'relevance',
  description: 'Scores how relevant the response is',
}).generateScore(() => 1)

export default {
  scorer: relevance,
  sampling: { type: 'ratio', rate: 0.5 },
}

Precedence with config

Discovered scorers are merged with the scorers defined in config.ts. On a key collision, config.scorers wins with a warning. If config.scorers is a function, discovered scorers are ignored with a warning because function-valued scorers can't be statically merged.

When keys collide, precedence is:

1. Function-valued config.scorers
2. Record-valued config.scorers
3. Discovered files under scorers/

Non-colliding discovered scorers are preserved.

Visit createScorer() for the full scorer interface.