Introducing Gates and Verdicts for Mastra Evals

You can now test your Mastra agents using gates and verdicts. Integrate with your existing test suite — vitest, jest, or mocha — and any CI pipeline to catch regressions before they're merged.

Testing agents requires covering both deterministic and non-deterministic behavior. Mastra's eval primitives can be used for both:

1. gates for deterministic checks — was a tool called, did it error, returns a score of 0 or 1
2. scorers for non-deterministic checks — how complete is the response, how appropriate is its tone, returns averageScore and threshold values

Evals return a verdict letting you more easily assert tests for both behavior types.

Before gates and verdicts, you had to write custom assertions for every scorer and stitch them together to determine whether tests passed or failed. Now, tests can pass or fail with a single assertion.

A verdict can also return a scored result. This is for tracking quality drift — missed thresholds can be logged and still allow a test to pass. Each result returns a gateResults or thresholdResults array so you can log what was missed.

Get started

This example tests the default Mastra weather agent using vitest, Mastra quick checks and prebuilt scorers.

Create test conditions by passing gates or scorers to runEvals(). Test assertions can be defined using the returned verdict:

import { describe, it, expect } from "vitest";
import { runEvals } from "@mastra/core/evals";
import { checks } from "@mastra/evals/checks";
import { createCompletenessScorer, createToneScorer } from "@mastra/evals/scorers/prebuilt";
import { weatherAgent } from "./weather-agent";
 
const data = [{ input: "What's the weather in Paris?" }, { input: "How's the weather in Tokyo today?" }, { input: "Is it raining in Seattle?" }];
 
describe("Weather agent", () => {
  it("calls the weatherTool with no errors", async () => {
    const result = await runEvals({
      data,
      target: weatherAgent,
      gates: [checks.calledTool("weatherTool"), checks.noToolErrors()]
    });
 
    expect(result.verdict).not.toBe("failed");
  });
 
  it("response meets quality thresholds", async () => {
    const result = await runEvals({
      data,
      target: weatherAgent,
      scorers: [
        { scorer: createCompletenessScorer(), threshold: 0.7 },
        { scorer: createToneScorer(), threshold: 0.5 }
      ]
    });
 
    expect(result.verdict).toBe("passed");
  });
});

Example output for gates:

{
  "scores": { "check-includes": 1 },
  "summary": { "totalItems": 3 },
  "gateResults": [
    { "id": "check-called-tool", "passed": true, "score": 1 },
    { "id": "check-no-tool-errors", "passed": true, "score": 1 }
  ],
  "verdict": "passed" // 'failed' | 'scored'
}

Example output for scorers:

{
  "scores": {
    "completeness-scorer": 0.8492063492063492,
    "tone-scorer": 0.9745706219973028
  },
  "summary": { "totalItems": 3 },
  "thresholdResults": [
    {
      "id": "completeness-scorer",
      "passed": true,
      "averageScore": 0.8492063492063492,
      "threshold": 0.7
    },
    {
      "id": "tone-scorer",
      "passed": true,
      "averageScore": 0.9745706219973028,
      "threshold": 0.5
    }
  ],
  "verdict": "passed" // 'failed' | 'scored'
}

For more information and full configuration options, see:

• Gates and verdicts
• Running scorers in CI
• Quick Checks
• Built-in scorers
• runEvals() reference