Introducing Vitest Integration for Testing

You can now run Mastra evals as part of your CI workflow using Vitest. Each eval behaves like a regular unit test. You can test your agents, workflows, and scorers.

The reporter prints gate and score results at the end of each run so you can see which tests passed or failed, and why.

Catching regressions before you deploy can prevent eg prompt tweaks from changing how your agent behaves in production. Running unit tests as part of your CI means every pull request is validated against the assertions defined in the tests, ensuring only fully functioning code gets deployed.

The alternate path is to use runEvals inside a test and assert using gates and verdicts. But it can be nice to use a familiar testing framework so we shipped this.

Testing LLM responses isn't like testing deterministic code. The same input will often produce a different output, causing exact assertions to fail. The Vitest integration lets you test using tolerances and thresholds, so the tests absorb the variance.

Get started

Install @mastra/evals and Vitest (v3 or v4):

Terminal

npm install @mastra/evals vitest

note

Requires @mastra/evals@1.10.0 or later, added in PR #22665.

vitest.config.ts

import { defineConfig } from "vitest/config";
import { MastraEvalsReporter } from "@mastra/evals/vitest";
 
export default defineConfig({
  test: {
    include: ["evals/**/*.test.ts"],
    reporters: ["verbose", new MastraEvalsReporter()],
    setupFiles: ["dotenv/config", "@mastra/evals/vitest/setup"],
    fileParallelism: false
  }
});

Agents

The tests below run a weather agent with gates for tool calls and tool errors, and a scorer for answer relevancy. runEvals returns the result directly; expectEvals wraps the same call as a Vitest assertion and prints a per-gate and per-scorer breakdown in the test output. Each test sets { timeout: 60_000 } to override Vitest's default 5s, which is too short for LLM-backed evals.

Agents with `runEvals`

Use runEvals when you want a simplified test output:

evals/weather-agent-run-evals.test.ts

import { test, expect } from "vitest";
import { runEvals } from "@mastra/core/evals";
import { checks } from "@mastra/evals/checks";
import { mastra } from "../src/mastra";
 
test("weather agent calls the tool and answers relevantly", { timeout: 60_000 }, async () => {
  const result = await runEvals({
    target: mastra.getAgent("weatherAgent"),
    data: [{ input: "What's the weather in London?" }],
    gates: [checks.calledTool("weatherTool"), checks.noToolErrors(), checks.includes("London")],
    scorers: [{ scorer: mastra.getScorer("answerRelevancyScorer"), threshold: 0.7 }]
  });
 
  expect(result).toHaveVerdict("passed");
  expect(result).toPassGates();
  expect(result).toPassThresholds();
  expect(result).toHaveScoreAbove("answer-relevancy-scorer", 0.7);
});

Output:

 ✓ evals/weather-agent-run-evals.test.ts > weather agent calls the tool and answers relevantly 32772ms
 
 Test Files  1 passed (1)
      Tests  1 passed (1)

Agents with `expectEvals`

Use expectEvals when you want a per-gate and per-scorer breakdown:

evals/weather-agent-expect-evals.test.ts

import { test } from "vitest";
import { expectEvals } from "@mastra/evals/vitest";
import { checks } from "@mastra/evals/checks";
import { mastra } from "../src/mastra";
 
test("weather agent calls the tool and answers relevantly", { timeout: 60_000 }, async () => {
  await expectEvals({
    target: mastra.getAgent("weatherAgent")
    // ...
  }).toPass(0.8);
});

Output:

 ✓ evals/weather-agent-expect-evals.test.ts > weather agent calls the tool and answers relevantly 22076ms
 
 Test Files  1 passed (1)
      Tests  1 passed (1)
 
 Mastra Evals
 
✓ weather agent calls the tool and answers relevantly (1 item)
   check-called-tool (gate)                        1.0  ✓
   check-no-tool-errors (gate)                     1.0  ✓
   check-includes (gate)                           1.0  ✓
   answer-relevancy-scorer (threshold: min 0.7)    1.0  ✓
 
 Eval runs: 1 (1 passed)

Workflows

Both APIs accept a workflow as target. The trajectory records workflow_step entries in execution order, with any tool invocations nested as tool_call children. createTrajectoryAccuracyScorerCode compares the actual trajectory against an expectedTrajectory and scores 1.0 when every step (and every named child tool call) matches in order. Used as a gate, it fails the test on any deviation.

Workflows with `runEvals`

Use runEvals when you want a simplified test output:

evals/weather-workflow-run-evals.test.ts

import { test, expect } from "vitest";
import { runEvals } from "@mastra/core/evals";
import { createTrajectoryAccuracyScorerCode } from "@mastra/evals/scorers/prebuilt";
import { mastra } from "../src/mastra";
 
test("weather workflow runs fetch-weather then plan-activities", { timeout: 60_000 }, async () => {
  const result = await runEvals({
    target: mastra.getWorkflow("weatherWorkflow"),
    data: [{ input: { city: "London" } }],
    gates: [
      createTrajectoryAccuracyScorerCode({
        expectedTrajectory: [{ stepType: "workflow_step", name: "fetch-weather", status: "success" }]
      }),
      createTrajectoryAccuracyScorerCode({
        expectedTrajectory: [{ stepType: "workflow_step", name: "plan-activities", status: "success" }]
      }),
      createTrajectoryAccuracyScorerCode({
        expectedTrajectory: [
          {
            stepType: "workflow_step",
            name: "plan-activities",
            children: { steps: [{ stepType: "tool_call", name: "weatherTool" }] }
          }
        ]
      })
    ]
  });
 
  expect(result).toHaveVerdict("passed");
  expect(result).toPassGates();
});

Output:

 ✓ evals/weather-workflow-run-evals.test.ts > weather workflow runs fetch-weather then plan-activities 6238ms
 
 Test Files  1 passed (1)
      Tests  1 passed (1)

Workflows with `expectEvals`

Use expectEvals when you want a per-gate breakdown:

evals/weather-workflow-expect-evals.test.ts

import { test } from "vitest";
import { expectEvals } from "@mastra/evals/vitest";
import { createTrajectoryAccuracyScorerCode } from "@mastra/evals/scorers/prebuilt";
import { mastra } from "../src/mastra";
 
test("weather workflow runs fetch-weather then plan-activities", { timeout: 60_000 }, async () => {
  await expectEvals({
    target: mastra.getWorkflow("weatherWorkflow")
    // ...
  }).toPass();
});

Output:

 ✓ evals/weather-workflow-expect-evals.test.ts > weather workflow runs fetch-weather then plan-activities 16001ms
 
 Test Files  1 passed (1)
      Tests  1 passed (1)
 
 Mastra Evals
 
✓ weather workflow runs fetch-weather then plan-activities (1 item)
   code-trajectory-accuracy-scorer (gate)    1.0  ✓
   code-trajectory-accuracy-scorer (gate)    1.0  ✓
   code-trajectory-accuracy-scorer (gate)    1.0  ✓
 
 Eval runs: 1 (1 passed)

Scorers

Scorers aren't a target, but a custom scorer can be tested on its own with a scorer's .run() method and an expected output. This scorer checks the answer mentions a temperature:

src/mastra/scorers/mentions-temperature.ts

import { createScorer } from "@mastra/core/evals";
import { getAssistantMessageFromRunOutput } from "@mastra/evals/scorers/utils";
 
const TEMPERATURE_PATTERN = /-?\d+(\.\d+)?\s?°\s?[CF]\b/;
 
export const mentionsTemperatureScorer = createScorer({
  id: "mentions-temperature",
  description: "Checks the answer states a temperature in °C or °F",
  type: "agent"
})
  .preprocess(({ run }) => {
    const response = getAssistantMessageFromRunOutput(run.output) ?? "";
    return { response, match: response.match(TEMPERATURE_PATTERN)?.[0] };
  })
  .generateScore(({ results }) => {
    return results.preprocessStepResult?.match ? 1 : 0;
  })
  .generateReason(({ results, score }) => {
    if (score === 1) {
      return `Answer states a temperature: ${results.preprocessStepResult?.match}`;
    }
    return "Answer does not state a temperature in °C or °F";
  });

Scorers with `.run()`

Use .run() with createAgentTestRun and createTestMessage to test the scorer against an expected output:

evals/weather-scorer-run.test.ts

import { test, expect } from "vitest";
import { createAgentTestRun, createTestMessage } from "@mastra/evals/scorers/utils";
import { mastra } from "../src/mastra";
 
test("scores 1 when the answer states a temperature", async () => {
  const testRun = createAgentTestRun({
    inputMessages: [createTestMessage({ content: "What's the weather in London?", role: "user" })],
    output: [createTestMessage({ content: "It's 15.7°C and clear in London.", role: "assistant" })]
  });
 
  const result = await mastra.getScorer("mentionsTemperatureScorer").run({
    input: testRun.input,
    output: testRun.output
  });
 
  expect(result.score).toBe(1);
});

Output:

 ✓ evals/weather-scorer-run.test.ts > scores 1 when the answer states a temperature 4ms
 
 Test Files  1 passed (1)
      Tests  1 passed (1)

Matrix testing

Combine Vitest's test.for with expectEval (or runEvals with a single-item data) for matrix testing against an agent or workflow, running one test per data item so a single regression shows up as one failing test. Both APIs also accept a data array for a single rolled-up pass rate across scenarios.

For more information and full configuration options, see: