Vitest integration

The @mastra/evals/vitest module integrates runEvals with Vitest so evaluations behave like regular tests. Fluent assertions fail the test when the eval doesn't pass, and a reporter prints a score table in the runner output.

To use it, install @mastra/evals and vitest (version 3 or 4):

npm install @mastra/evals vitest

Configuring Vitest

Register the reporter and matchers in vitest.config.ts:

vitest.config.ts

import { defineConfig } from 'vitest/config'
import { MastraEvalsReporter } from '@mastra/evals/vitest'

export default defineConfig({
  test: {
    reporters: ['default', new MastraEvalsReporter()],
    setupFiles: ['@mastra/evals/vitest/setup'],
  },
})

The setup file registers the custom matchers on expect. Alternatively, call registerEvalMatchers() from @mastra/evals/vitest in your own setup file.

Asserting on a dataset with expectEvals

expectEvals runs a runEvals evaluation inside a regular test() and asserts a minimum pass rate. It accepts the same configuration as runEvals: a target agent or workflow, data items, and scorers, gates, or thresholds. Gates score each item pass/fail, so toPass(0.8) requires at least 80% of items to pass every gate. Scorer thresholds still compare the average score across items and must pass regardless of the rate:

tests/weather-agent.eval.test.ts

import { test } from 'vitest'
import { expectEvals } from '@mastra/evals/vitest'
import { checks } from '@mastra/evals/checks'
import { mastra } from '../src/mastra'

test('weather agent calls the tool and answers relevantly', { timeout: 60_000 }, async () => {
  await expectEvals({
    target: mastra.getAgent('weatherAgent'),
    data: [{ input: "What's the weather in London?" }],
    gates: [checks.calledTool('weatherTool'), checks.noToolErrors(), checks.includes('London')],
    scorers: [{ scorer: mastra.getScorer('answerRelevancyScorer'), threshold: 0.7 }],
  }).toPass(0.8)
})

toPass() without an argument requires every item to pass every gate. Always await the assertion: it resolves with the full RunEvalsResult for further checks and attaches the run's scores to the current test so MastraEvalsReporter displays them.

LLM-backed evals are far slower than Vitest's default 5-second timeout, so pass a per-test timeout (or set testTimeout in the Vitest config).

Matrix testing with expectEval

expectEval is the single-item variant: data is one item instead of an array. Combine it with test.for (or test.each) to get one test (and one reporter entry) per data item instead of one aggregated result per dataset:

tests/weather-agent.matrix.eval.test.ts

import { test } from 'vitest'
import { expectEval } from '@mastra/evals/vitest'
import { checks } from '@mastra/evals/checks'
import { mastra } from '../src/mastra'

test.for([
  { input: "What's the weather in London?", groundTruth: 'London' },
  { input: "What's the weather in Tokyo?", groundTruth: 'Tokyo' },
  { input: "What's the weather in Sydney?", groundTruth: 'Sydney' },
])('weather agent: $input', { timeout: 60_000 }, async item => {
  await expectEval({
    target: mastra.getAgent('weatherAgent'),
    data: item,
    gates: [checks.calledTool('weatherTool'), checks.includes(item.groundTruth)],
  }).toPass()
})

Each item passes or fails independently. A single regression shows up as one failing test instead of a lowered aggregate pass rate.

Asserting on results with matchers

For finer-grained control, call runEvals directly inside a regular test() and use the custom matchers on the result:

tests/weather-agent.matchers.eval.test.ts

import { test, expect } from 'vitest'
import { runEvals } from '@mastra/core/evals'
import { checks } from '@mastra/evals/checks'
import { mastra } from '../src/mastra'

test('weather agent quality', { timeout: 60_000 }, async () => {
  const result = await runEvals({
    target: mastra.getAgent('weatherAgent'),
    data: [{ input: "What's the weather in London?" }],
    scorers: [mastra.getScorer('answerRelevancyScorer')],
    gates: [checks.noToolErrors()],
  })

  expect(result).toHaveVerdict('passed')
  expect(result).toPassGates()
  expect(result).toHaveScoreAbove('answer-relevancy-scorer', 0.7)
})

Available matchers:

• toHaveVerdict(verdict): asserts the run's verdict ("passed", "scored", or "failed").
• toHaveScoreAbove(scorerName, min) / toHaveScoreBelow(scorerName, max): asserts a scorer's average score. Categorized scorer configs use dot-paths, for example "agent.my-scorer" or "steps.step-1.my-scorer".
• toPassGates(): asserts all gates passed. Fails when no gates were configured.
• toPassThresholds(): asserts all scorer thresholds passed. Fails when no thresholds were configured.

Evaluating a workflow

Workflows are valid targets too. Use a WorkflowScorerConfig object for scorers to score the workflow output, individual steps, or the step execution trajectory, and reference nested scores with dot-paths in the matchers:

tests/weather-workflow.eval.test.ts

import { test, expect } from 'vitest'
import { runEvals } from '@mastra/core/evals'
import { mastra } from '../src/mastra'

test('weather workflow runs fetch-weather then plan-activities', { timeout: 60_000 }, async () => {
  const result = await runEvals({
    target: mastra.getWorkflow('weatherWorkflow'),
    data: [
      {
        input: { city: 'London' },
        expectedTrajectory: {
          steps: [
            { stepType: 'workflow_step', name: 'fetch-weather' },
            { stepType: 'workflow_step', name: 'plan-activities' },
          ],
        },
      },
      {
        input: { city: 'Tokyo' },
        expectedTrajectory: {
          steps: [
            { stepType: 'workflow_step', name: 'fetch-weather' },
            { stepType: 'workflow_step', name: 'plan-activities' },
          ],
        },
      },
    ],
    scorers: {
      trajectory: [mastra.getScorer('trajectoryAccuracyScorer')],
    },
  })

  expect(result).toHaveScoreAbove('trajectory.code-trajectory-accuracy-scorer', 0.9)
})

The plan-activities step in this workflow calls mastra.getAgent('weatherAgent') at runtime, which only works because the target was resolved with mastra.getWorkflow(). See the runEvals reference for the full WorkflowScorerConfig shape.

Reading the reporter output

MastraEvalsReporter prints a score table for every eval test after the run completes:

 Mastra Evals

✓ weather agent calls the tool and answers relevantly (1 item)
   check-called-tool (gate)                        1.0  ✓
   check-no-tool-errors (gate)                     1.0  ✓
   check-includes (gate)                           1.0  ✓
   answer-relevancy-scorer (threshold: min 0.7)    0.9  ✓

 Eval runs: 1 (1 passed)

Each entry shows the run's verdict, gates, thresholds, and the average score per scorer. The reporter reads the metadata that expectEval/expectEvals attach to task.meta.mastraEval, so it works with parallel test files and any test that populates that field.

Rate limits and concurrency

Vitest runs test files in parallel, and runEvals accepts its own concurrency option, so total LLM traffic is multiplied across both. If you hit provider rate limits, lower concurrency in your eval options or set fileParallelism: false in the Vitest config.