Introducing Experiment Tool Mocks

You can now run agent experiments without hitting live APIs using tool mocks. Each dataset item can declare mock inputs and outputs that experiments use in place of real tool call results.

Tools are often used to interact with APIs. Some functionality can be destructive or even permanent — "delete user", "refund payment", "send email". Using tool mocks, you can assert the expected results without invoking the tool's execute function and hitting the real API.

Before tool mocks, testing agents that touched payment or messaging APIs meant running experiments in test environments or managing env var swaps. With tool mocks, nothing in your project needs to change — the agent, env vars, and tool config stay exactly as they would when running in production.

Starting today, tool mocks are supported for agent targets, with workflow tool mocks to follow.

Get started

Running experiments requires a storage adapter to persist datasets and experiment results.

npm install @mastra/libsql

Add Turso environment variables to your LibSQLStore config. When you deploy to the Mastra platform, a Turso database will be automatically provisioned for you:

import { Mastra } from "@mastra/core/mastra";
import { LibSQLStore } from "@mastra/libsql";
 
export const mastra = new Mastra({
  // ...
  storage: new LibSQLStore({
    url: process.env.TURSO_DATABASE_URL,
    authToken: process.env.TURSO_AUTH_TOKEN
  })
});

Setup

The billingAgent is configured with two tools — getOrder and refundPayment:

import { Agent } from "@mastra/core/agent";
import { getOrder } from "../tools/get-order";
 
export const billingAgent = new Agent({
  // ...
  tools: { getOrder }
});

Each tool has a defined inputSchema and outputSchema. Tool mocks must use the same data shape (see below):

import { createTool } from "@mastra/core/tools";
import { z } from "zod";
 
export const getOrder = createTool({
  // ...
  inputSchema: z.object({
    orderId: z.number().int()
  }),
  outputSchema: z.object({
    id: z.number(),
    product: z.string(),
    amount: z.number(),
    status: z.string(),
    chargedAt: z.string().optional()
  }),
  execute: async ({ orderId }) => {
    // e.g. fetch the order from Shopify
  }
});

Create datasets and run experiments using the TS API or Studio.

Each dataset item declares a groundTruth for scoring and a toolMocks array. Each mock requires a toolName with args and output matching the defined tool's inputSchema and outputSchema:

const dataset = await mastra.datasets.create({
  name: "refund-decisions-dataset"
});
 
await dataset.addItem({
  input: "Get order status for 5510.",
  groundTruth: "Order 5510 is still pending and hasn't been charged.",
  toolMocks: [
    {
      toolName: "getOrder",
      args: { orderId: 5510 },
      output: { id: 5510, product: "Notebook set", amount: 1500, status: "pending" }
    }
  ]
});

Target a dataset by its id then attach .startExperiment(). Configure the experiment using targetType, targetId, and scorers.

Set unmockedToolPolicy: 'deny' to block undeclared tools. This prevents live tool calls when mocks aren't declared:

const dataset = await mastra.datasets.get({ id: "<dataset-id>" });
 
const summary = await dataset.startExperiment({
  name: "refund-decisions-experiment",
  targetType: "agent",
  targetId: "billingAgent",
  scorers: ["answer-similarity-scorer"],
  unmockedToolPolicy: "deny"
});
 
for (const item of summary.results) {
  const report = item.toolMockReport;
  console.log(`Served:  ${report?.served?.length ?? 0}`);
  console.log(`Live:    ${report?.liveCalls?.length ?? 0}`);
  console.log(`Failure: ${report?.failure?.code ?? "none"}`);
}

For more information and full configuration options, see:

• Tool mocks
• Block undeclared tools
• startExperiment
• addItem reference