Skip to main content

dataset.startExperiment()

Added in: @mastra/core@1.4.0

Runs an experiment on the dataset and waits for completion. Executes all items against a target (agent, workflow, or scorer) with optional scoring.

Usage example
Direct link to Usage example

import { Mastra } from '@mastra/core'

const mastra = new Mastra({/* storage config */})

const dataset = await mastra.datasets.get({ id: 'dataset-id' })

// Run against a registered agent with a flat scorer list
const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'my-agent',
  scorers: ['accuracy', 'relevancy'],
  maxConcurrency: 10,
})

// Or pass the same categorised shape accepted by runEvals
const summary2 = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'my-agent',
  scorers: {
    agent: [accuracyScorer],
    trajectory: [toolOrderScorer],
  },
})

// For workflow targets, score individual steps with their own scorers
const summary3 = await dataset.startExperiment({
  targetType: 'workflow',
  targetId: 'my-workflow',
  scorers: {
    workflow: [overallScorer],
    steps: {
      'fetch-data': [fetchScorer],
      transform: [transformScorer],
    },
    trajectory: [executionPathScorer],
  },
})

console.log(`${summary.succeededCount}/${summary.totalItems} succeeded`)
console.log(`Status: ${summary.status}`)
console.log(`${summary2.succeededCount}/${summary2.totalItems} succeeded`)
console.log(`Status: ${summary2.status}`)

Parameters
Direct link to Parameters

targetType?:

'agent' | 'workflow' | 'scorer'
Type of registered target to run items against. Use with targetId.

targetId?:

string
ID of the registered target. Use with targetType.

scorers?:

(MastraScorer | string)[] | AgentScorerConfig | WorkflowScorerConfig
Scorers to evaluate each result. Accepts a flat array of MastraScorer instances or registered scorer IDs, or the same categorised config shape used by runEvals (AgentScorerConfig / WorkflowScorerConfig). When this option is provided, including as [] or an empty categorized config, it overrides item-level and dataset-level scorer IDs for every item. When omitted, each item uses its own scorerIds when present, then falls back to the dataset scorer IDs. Sources aren't merged. Trajectory scorers (type: "trajectory") automatically receive a pre-extracted Trajectory as their output regardless of which form is used. Workflow step scorers are supported only through this run-level categorized configuration. Their results carry the originating stepId and keep targetScope: "span" (matching runEvals).

name?:

string
Display name for the experiment.

description?:

string
Description of the experiment.

metadata?:

Record<string, unknown>
Arbitrary metadata for the experiment.

version?:

number
Pin to a specific dataset version. Defaults to the latest version.

maxConcurrency?:

number
Maximum concurrent item executions. Defaults to 5.

signal?:

AbortSignal
AbortSignal for cancelling the experiment.

itemTimeout?:

number
Per-item execution timeout in milliseconds.

maxRetries?:

number
Maximum retries per item on failure. Defaults to 0 (no retries). Abort errors are never retried.

unmockedToolPolicy?:

'allow' | 'deny'
= 'allow'
Controls undeclared agent tool calls. allow executes them live. deny fails the item with TOOL_MOCK_NOT_DECLARED before execution. An item-level value overrides this experiment default.

beforeAll?:

(args: ExperimentHookArgs) => void | Promise<void>
Runs once before any item executes. Receives experimentId, mastra, and signal. A failure fails the experiment and no items run.

beforeEach?:

(args: ExperimentItemHookArgs) => void | Promise<void>
Runs before each item executes. Also receives the item (id, input, groundTruth, metadata). A failure fails that item with EXPERIMENT_ITEM_BEFORE_EACH_FAILED and skips its target, scorers, and afterEach.

afterEach?:

(args: ExperimentItemResultHookArgs) => void | Promise<void>
Runs after each item completes. Also receives the item's result with its scores. Skipped when beforeEach failed. A failure is logged and doesn't change the item's outcome.

afterAll?:

(args: ExperimentRunResultHookArgs) => void | Promise<void>
Runs once after the experiment finishes, on every exit path including failure. Also receives the summary. A failure is logged and doesn't change the summary.

persistence?:

ExperimentPersistencePolicy
Controls whether this run writes experiment records and score records. Targets and scorers still execute, and results remain available in the returned summary.
ExperimentPersistencePolicy

experiments?:

'default' | 'none'
Set to none to skip experiment creation, item results, progress, and terminal status writes.

scores?:

'default' | 'none'
Set to none to skip score writes while still running scorers.

Returns
Direct link to Returns

result:

Promise<ExperimentSummary>
Summary of the completed experiment.
ExperimentSummary

experimentId:

string
Unique ID of the experiment.

status:

'pending' | 'running' | 'completed' | 'failed'
Final status of the experiment.

totalItems:

number
Total number of items in the dataset.

succeededCount:

number
Number of items that succeeded.

failedCount:

number
Number of items that failed.

skippedCount:

number
Number of items skipped (e.g., due to abort).

completedWithErrors:

boolean
true if the run completed but some items failed.

startedAt:

Date
When the experiment started.

completedAt:

Date
When the experiment completed.

results:

ItemWithScores[]
All item results with their scores.
ItemWithScores

itemId:

string
ID of the dataset item.

itemVersion:

number
Dataset version of the item when executed.

input:

unknown
Input data passed to the target.

output:

unknown | null
Output from the target, or null if failed.

groundTruth:

unknown | null
Expected output from the dataset item.

error:

{ message: string; stack?: string; code?: string } | null
Structured error if execution failed.

startedAt:

Date
When item execution started.

completedAt:

Date
When item execution completed.

retryCount:

number
Number of retry attempts.

scores:

ScorerResult[]
Results from all scorers for this item.
ScorerResult

scorerId:

string
ID of the scorer.

scorerName:

string
Display name of the scorer.

score:

number | null
Computed score, or null if the scorer failed.

reason:

string | null
Reason/explanation for the score.

error:

string | null
Error message if the scorer failed.
On this page