Introducing Experiment Targets

You can now run experiments against workflows and scorers using experiment targets.

In February, we launched experiments that could run against predefined datasets, so you could catch regressions after prompt tweaks, model swaps, or code changes. But they were only available for agents.

Now, you can also evaluate your workflows – as well as the scorers themselves. Experiment targets let you test each primitive individually with defined inputs and expected outputs (ground truths), persisted in storage to inspect, compare, or send on to downstream services.

Experiments can be run from four Mastra surfaces, each suited to a different use case:

• Studio: Human review and comparison.
• TS API: Internal evaluation applications.
• CLI: Pipelines to block deploys.
• HTTP endpoints: Downstream monitoring services.

Agent and workflow targets accept dataset items with a defined input and groundTruth. Outputs are typically produced by an LLM or a tool call. Scorer targets require the input, groundTruth, and output to be defined, since scorers themselves don't produce outputs.

Get started

Running experiments requires a storage adapter to persist datasets and experiment results.

npm install @mastra/libsql

Add Turso environment variables to your LibSQLStore config. When you deploy to the Mastra platform, a Turso database will be automatically provisioned for you:

import { Mastra } from "@mastra/core/mastra";
import { LibSQLStore } from "@mastra/libsql";
 
export const mastra = new Mastra({
  agents: {
    /* ... */
  },
  workflows: {
    /* ... */
  },
  scorers: {
    /* ... */
  },
  storage: new LibSQLStore({
    url: process.env.TURSO_DATABASE_URL,
    authToken: process.env.TURSO_AUTH_TOKEN
  })
});

With your project deployed and a database provisioned, you can create and run experiments from the Studio UI.

Create a dataset in the Datasets tab in Studio, then add items with inputs and expected outputs (ground truths).

Run an experiment against the dataset with a target (agent, workflow, or scorer):

Or, select two runs in the Experiments tab to Compare results side by side:

Programmatic access

In addition to Studio, you can create datasets and run experiments programmatically:

Create datasets

Create experiments

Run experiments

For more information and full configuration options, see:

• Experiments overview
• Experiment targets
• Datasets
• libSQL storage