Running experiments

Added in: @mastra/core@1.4.0

An experiment runs every item in a dataset through a target (an agent, a workflow, or a scorer) and then optionally scores the outputs. Use a scorer as the target when you want to evaluate an LLM judge itself. By default, results are persisted to storage so you can compare runs across different prompts, models, or code changes.

For AI agents: Run npx mastra api experiment run dataset_123 '{"name":"translation-baseline"}' to start an experiment directly instead of opening Studio or writing a temporary script. Use a dataset ID returned by npx mastra api dataset list in place of the sample ID. The command requires a running Mastra server with dataset storage and registered experiment targets; start the local server with npx mastra dev, or pass the reachable server's base URL with --url. Run npx mastra api experiment run --schema before constructing different input, and get user approval before starting an experiment because it can make model calls. Install Mastra's skill with npx skills add mastra-ai/skills --skill mastra for complete API CLI discovery, targeting, schema, authentication, and error-handling guidance.

Basic experiment

Call startExperiment() with a target and scorers:

src/mastra/experiments/basic.ts

import { mastra } from '../index'

const dataset = await mastra.datasets.get({ id: 'translation-dataset-id' })

const summary = await dataset.startExperiment({
  name: 'gpt-5.1-baseline',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy', 'fluency'],
})

console.log(summary.status) // 'completed' | 'failed'
console.log(summary.succeededCount) // number of items that ran successfully
console.log(summary.failedCount) // number of items that failed

startExperiment() blocks until all items finish. For fire-and-forget execution, see async experiments.

Studio

You can also run experiments in Studio. After you've added a dataset item, open it and select Run Experiment and configure the target, scorers, and options.

To configure scorers for one item, edit the item and enable Override dataset scorers. Select one or more scorers, or leave the selection empty to run no scorers for that item.

After running an experiment, the Experiments tab shows all runs for that dataset (with status, counts, and timestamps). Select an experiment to see per-item results, scores, and execution traces.

In the Experiments tab, select Compare and choose two or more experiments to compare their scores and results side by side.

Experiment targets

You can point an experiment at a registered agent, workflow, or scorer.

Registered agent

Point to an agent registered on your Mastra instance:

src/mastra/experiments/agent-target.ts

const summary = await dataset.startExperiment({
  name: 'agent-v2-eval',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy'],
})

Each item's input is passed directly to agent.generate(), so it must be a string, string[], or CoreMessage[].

Memory-enabled agents

When the target agent has its own memory and the request context carries a resource id (MASTRA_RESOURCE_ID_KEY, set by auth middleware, the experiment or item requestContext, or the Studio Run Experiment form), the experiment runner injects a fresh memory thread for each item. A resource id in the request context means "run as this resource": each item's conversation persists as a thread under that resource, and retried items get a new thread per attempt so earlier failed attempts can't leak into the retry's context.

Injected threads are tagged so you can map them back to the run: thread metadata carries the experimentId and the dataset item's id as experimentItemId. No thread title is generated for them.

Because the threads belong to the caller's resource, resource-scoped memory features both read and write that resource's state during the run:

• Resource-scoped working memory updates persist to the resource, and later items in the run see updates made by earlier items.
• Resource-scoped semantic recall can surface the resource's prior conversations to the experiment, and experiment transcripts become recallable in that resource's later conversations.

Use this approach to evaluate an agent against a real user's accumulated context. To keep experiment runs from touching real user state, use a dedicated evaluation resource id instead.

Thread injection is skipped in the following cases:

• If the request context also sets MASTRA_THREAD_ID_KEY, the runner uses that thread as-is, so every item (and retry) shares the same conversation.
• If the agent has no memory, or the request context has no resource id, the run is memoryless and nothing is persisted.

Registered workflow

Point to a workflow registered on your Mastra instance:

src/mastra/experiments/workflow-target.ts

const summary = await dataset.startExperiment({
  name: 'workflow-eval',
  targetType: 'workflow',
  targetId: 'translation-workflow',
  scorers: ['accuracy'],
})

The workflow receives each item's input as its trigger data.

Registered scorer

Point to a scorer to evaluate an LLM judge against ground truth:

src/mastra/experiments/scorer-target.ts

const summary = await dataset.startExperiment({
  name: 'judge-accuracy-eval',
  targetType: 'scorer',
  targetId: 'accuracy',
})

The scorer receives each item's input and groundTruth. LLM-based judges can drift over time as underlying models change, so it's important to periodically realign them against known-good labels. A dataset gives you a stable benchmark to detect that drift.

Scoring results

Scorers automatically run after each item's target execution. Pass scorer instances or registered scorer IDs:

Scorer IDs

// Reference scorers registered on the Mastra instance
const summary = await dataset.startExperiment({
  name: 'with-registered-scorers',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy', 'fluency'],
})

Scorer instances

import { createAnswerRelevancyScorer } from '@mastra/evals/scorers/prebuilt'

const relevancy = createAnswerRelevancyScorer({ model: 'openai/gpt-5-mini' })

const summary = await dataset.startExperiment({
  name: 'with-scorer-instances',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: [relevancy],
})

Each item's results include per-scorer scores:

for (const item of summary.results) {
  console.log(item.itemId, item.output)
  for (const score of item.scores) {
    console.log(`  ${score.scorerName}: ${score.score} — ${score.reason}`)
  }
}

Visit the Scorers overview for details on available and custom scorers.

Select scorers per item

Add registered scorer IDs to a dataset item to override the scorers attached to its dataset:

src/mastra/datasets/items.ts

await dataset.addItem({
  input: 'Translate "hello" to French.',
  scorerIds: ['accuracy', 'fluency'],
})

An experiment uses exactly one scorer source for each item, in this order:

1. The experiment's scorers option, when provided
2. The item's scorerIds field, when provided
3. The dataset's scorerIds field
4. No scorers

Mastra doesn't merge these sources. An explicit empty array at the run or item level selects no scorers and prevents fallback to the next source. An empty categorized run-level configuration has the same effect. Duplicate item or dataset IDs run once, in the order of their first occurrence.

The following updates switch between an explicit empty override and dataset inheritance:

src/mastra/datasets/items.ts

// Run no scorers for this item
await dataset.updateItem({
  itemId: 'translation-item-id',
  scorerIds: [],
})

// Remove the item override and inherit the dataset's scorers
await dataset.updateItem({
  itemId: 'translation-item-id',
  scorerIds: null,
})

Omitting scorerIds from an update preserves its current value. Each item version stores its scorer ID list, including an empty list. Scorer definitions aren't copied into the item version and continue to use their existing registry or Editor behavior.

Mastra resolves each item's scorer IDs before running its target. It checks registered scorers first, then asks the Editor to hydrate a stored scorer. A stale item-level ID produces the EXPERIMENT_ITEM_SCORER_NOT_FOUND code for that item. Mastra skips target execution without retrying that item, while the rest of the experiment continues.

A missing ID in the selected run-level or dataset-level source fails experiment setup. Mastra doesn't resolve IDs from lower-priority sources that the precedence rules ignore.

Workflow step scorers are available only through a run-level categorized scorers configuration. Item-level IDs apply to the flat agent, workflow, and trajectory scorer dispatch.

Control persistence per run

Use persistence to skip storage writes for a specific run. Experiment records and score records can be disabled independently:

src/mastra/experiments/no-persistence.ts

const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy'],
  persistence: {
    experiments: 'none',
    scores: 'none',
  },
})

The target and scorers still run, and startExperiment() still returns the item results and scores in summary. The settings are independent. For example, set only scores: 'none' to persist the experiment and its item results without creating score records.

Omitted settings default to 'default', which preserves the standard storage behavior. This policy only controls experiment and score records created by the run. It doesn't disable storage used by the target, such as agent memory, vectors, observability, or custom tool storage.

When startExperimentAsync() runs with experiments: 'none', it doesn't persist an experiment record, progress updates, or item results. Score persistence remains controlled separately by persistence.scores. Without an experiment event observer, the run is fire-and-forget, and the experiment API can't report whether it completed or failed.

Use synchronous startExperiment() when the caller needs the returned summary. An experiment event observer can receive lifecycle events and the terminal summary.

Observe experiment events

Use onEvent to receive versioned, JSON-safe lifecycle events while an experiment runs. This works with startExperiment(), startExperimentAsync(), and runExperiment().

src/mastra/experiments/observe.ts

import type { ExperimentEvent } from '@mastra/core/datasets'

const events: ExperimentEvent[] = []

await dataset.startExperimentAsync({
  task: async ({ input }) => processItem(input),
  persistence: { experiments: 'none' },
  onEvent: async event => {
    events.push(event)
    await publishEvent(event)
  },
})

The observer receives these event types:

• experiment.run.started: Identifies the run, target, resolved dataset version, and item count.
• experiment.item.completed: Reports a committed item result after scoring, including scores, errors, retry count, tool mock details, and stable item identity.
• experiment.run.finished: Reports the terminal outcome and summary counters.

Mastra awaits each observer call before delivering the next event. This serialized delivery applies backpressure and ensures event sequence values match delivery order, while item execution can remain concurrent.

If the observer throws or rejects, Mastra aborts the remaining run and rejects runExperiment() with a MastraError whose id is EXPERIMENT_EVENT_OBSERVER_FAILED. It doesn't send a terminal event through the failed observer. For startExperimentAsync(), the method has already returned when a detached observer fails, so handle delivery failures inside the observer when the caller needs direct error reporting.

The experiment.run.finished event is awaited before Mastra persists the final experiment status. Treat the event as the authoritative terminal signal when experiment persistence is disabled, but don't use it as a read-after-write signal for storage.

The exported event types are ExperimentEvent, ExperimentRunStartedEvent, ExperimentItemCompletedEvent, and ExperimentRunFinishedEvent. Use the discriminated type field to narrow an event before reading event-specific properties.

Lifecycle hooks

Use lifecycle hooks to prepare state before a target runs and clean it up afterward. This is useful when an item can't be evaluated against an empty environment. A run might need a fixture file copied into the agent's workspace, or a sandbox provisioned before the agent can touch it.

Hooks run at two levels. beforeAll and afterAll run once per experiment, and beforeEach and afterEach run once per item:

src/mastra/experiments/hooks.ts

const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'document-agent',
  scorers: ['accuracy'],
  beforeAll: async ({ experimentId }) => {
    await createWorkspace(experimentId)
  },
  beforeEach: async ({ item }) => {
    await copyFixture(item.metadata?.fixture)
  },
  afterEach: async ({ item, result }) => {
    await clearWorkspaceFiles(item.id)
  },
  afterAll: async ({ summary }) => {
    await deleteWorkspace(summary.experimentId)
  },
})

Every hook can be async. Each one receives the experimentId, the mastra instance, and the run-level signal, so long-running setup can be cancelled along with the experiment. The per-item hooks also receive item. The teardown hooks receive the result they follow: afterEach receives the item's result including scores, and afterAll receives the summary that's about to be returned.

The item passed to hooks exposes id, input, groundTruth, and metadata. Fields that control execution, such as tool mocks and scorer selection, aren't exposed, so a hook can't change how the item runs.

Hook failures

Each hook has a different consequence when it throws, based on how much of the run depends on it:

Hook	On failure
beforeAll	Fails the experiment. No items run.
beforeEach	Fails that item with EXPERIMENT_ITEM_BEFORE_EACH_FAILED. Other items continue.
afterEach	Logged. The item's recorded outcome doesn't change.
afterAll	Logged. The returned summary doesn't change.

When beforeAll fails, the experiment is marked failed and the experiment.run.finished event is still emitted before the error propagates.

When beforeEach fails, the target and its scorers are skipped for that item, since the item's preconditions were never met. afterEach is also skipped for that item, on the basis that setup which didn't finish owns its own cleanup.

Teardown failures are logged rather than propagated. By the time afterEach runs, the target has already produced a real result, and discarding it because cleanup was untidy would lose the data the experiment was run to collect.

afterAll runs on every exit path, including when the experiment fails, when beforeAll fails, and when an event observer fails, so teardown isn't skipped when something goes wrong. It runs at most once per experiment.

Tool mocks

When an experiment runs an agent that calls side-effecting tools, attach static tool mocks to individual dataset items to make the run deterministic. During the experiment, a mocked tool returns its declared output instead of executing. Tools without a mock on the item run live by default.

Mocks live on the dataset item, so they version with the row and travel with the test case. Each mock declares a tool name, the arguments it expects, and the output to return:

src/mastra/experiments/tool-mocks.ts

await dataset.addItem({
  input: 'What is the weather in Seattle?',
  toolMocks: [
    {
      toolName: 'getWeather',
      args: { city: 'Seattle' },
      output: { temperature: 60, conditions: 'rainy' },
    },
  ],
})

Tool mocks are supported for agent targets only.

Block undeclared tools

Set unmockedToolPolicy: 'deny' on an experiment to block every tool call that doesn't have a mock. This is useful when a live call could cause side effects:

const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'weather-agent',
  unmockedToolPolicy: 'deny',
})

The default policy is 'allow'. You can override the experiment policy on an individual stored or inline item:

await dataset.addItem({
  input: 'What is the weather in Seattle?',
  unmockedToolPolicy: 'allow',
})

The item value takes precedence over the experiment value. A denied call fails with TOOL_MOCK_NOT_DECLARED before the tool executes. The failure isn't retried or added to liveCalls.

Matching and consumption

Arguments are matched strictly: object key order is ignored and array order is substantial, plus there is no type coercion. A mock is served only when the agent calls the tool with arguments that deep-equal the mock's args.

When an item declares several mocks for the same tool and arguments, they're consumed in order, the first call gets the first mock, the next call gets the second, and so on. Ordering is tracked per (toolName, args) group and is independent across different arguments.

Matching mode

By default each mock matches strictly on its args. Set matchArgs: 'ignore' to match on the tool name only, the mock's args aren't compared and the next unconsumed mock for that tool is served regardless of how the agent called it:

const subAgentMock = {
  toolName: 'agent-balanceAgent',
  args: { prompt: 'look up the balance for YJ' },
  output: { text: "YJ's balance is $100." },
  matchArgs: 'ignore',
}

This is useful when a tool's arguments are noisy or generated by the model. The most common case is mocking a sub-agent's response: a delegated sub-agent is exposed to the parent as an agent-<name> tool, and its arguments include an LLM-authored prompt plus runtime-injected fields. Mocking agent-<name> returns the canned response in place of running the sub-agent and its inner tools. When you create a mock from a trace, sub-agent delegation calls are derived with matchArgs: 'ignore' automatically. You can change it to 'strict' to pin the exact arguments.

Failures

A tool call fails the item when it violates the mock configuration:

• TOOL_MOCK_MISMATCH: the tool was called with arguments that no mock matches.
• TOOL_MOCK_EXHAUSTED: every matching mock has already been consumed.
• TOOL_MOCK_NOT_DECLARED: the tool has no mock and the effective unmockedToolPolicy is 'deny'.

On any of these failures, the agent run is aborted immediately, so the model can't go on to call any further tools, including unmocked, side-effecting tools that would otherwise run live. These failures are deterministic, so they're not retried. Mocks that are declared but never used don't fail the item, they're reported as unconsumed.

While mock interception is active, the agent's tools execute sequentially so repeated (toolName, args) mocks are consumed in the provider's call order. Interception is active when the item declares mocks or its effective unmockedToolPolicy is 'deny'.

Diagnostics

Each item result carries a toolMockReport describing what the run did with the item's mocks:

for (const item of summary.results) {
  const report = item.toolMockReport
  if (!report) continue

  console.log(report.served) // mocks matched and returned
  console.log(report.unconsumed) // mocks declared but never used
  console.log(report.liveCalls) // undeclared tools allowed to run live
  console.log(report.failure) // the first deterministic mock failure, if any
}

In Studio, edit a dataset item to author tool mocks as a JSON array, and open an experiment result to see the same report.

Limitations

• No tool span for mocked calls. A mocked call returns its output before the tool executes, so it doesn't create a tool span. Trajectory scorers backed by stored traces may therefore not see mocked tool calls. Trajectory extraction that falls back to the agent's message output still sees them, so trajectory scoring can differ depending on your observability configuration.
• Storage support. The LibSQL, PostgreSQL, MongoDB, and Spanner adapters persist tool mocks and tool mock reports, while the MySQL adapter rejects writes that carry either one. All dataset storage adapters persist unmockedToolPolicy.

Async experiments

startExperiment() blocks until every item completes. For long-running datasets, use startExperimentAsync() to start the experiment in the background:

src/mastra/experiments/async.ts

const { experimentId, status } = await dataset.startExperimentAsync({
  name: 'large-dataset-run',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy'],
})

console.log(experimentId) // UUID
console.log(status) // 'pending'

Poll for completion using getExperiment():

let experiment = await dataset.getExperiment({ experimentId })

while (experiment.status === 'pending' || experiment.status === 'running') {
  await new Promise(resolve => setTimeout(resolve, 5000))
  experiment = await dataset.getExperiment({ experimentId })
}

console.log(experiment.status) // 'completed' | 'failed'

Caller-driven experiments

startExperiment() puts Mastra in charge of the whole run. If a durable orchestrator (for example Temporal, Airflow, or a custom worker fleet) owns the loop instead, create the experiment first and drive it yourself in one of these shapes:

• Caller drives the loop, Mastra runs each item. Create the experiment with a target, then call runExperimentItem() once per item. Mastra executes the target and runs the scorers, then persists the result.
• Caller runs everything. Create the experiment without a target, execute and score items on your own infrastructure, and ingest each result with submitExperimentResult().

Both shapes finish with finalizeExperiment(), and both write to the same tables as native runs, so Studio views, comparisons, and review summaries work unchanged.

Run items server-side

Create the experiment with createExperiment(), passing the target and scorers. Pass your own id (for example a workflow run id) to make creation idempotent: retrying the call with the same id returns the existing experiment instead of failing.

src/mastra/experiments/caller-driven.ts

const dataset = await mastra.datasets.get({ id: 'translation-dataset-id' })

const { experimentId, totalItems, datasetVersion } = await dataset.createExperiment({
  id: 'temporal-wf-run-42', // optional: reuse your workflow run id for idempotent creates
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy'],
})

Then run each item from your orchestrator. Mastra resolves the item at the pinned dataset version and executes the target with the resolved scorers. The result upserts keyed by (experimentId, itemId, attempt), so a retried activity converges on a single row:

const { result, scores } = await dataset.runExperimentItem({
  experimentId,
  itemId: 'item-1',
})

Retries and timeouts belong to your orchestrator: each runExperimentItem call executes the item exactly once. Scorers resolve with the same precedence as native runs: experiment scorers win over item scorerIds, which win over dataset scorerIds.

Ingest external results

If your workers execute and score items themselves, create the experiment without a target:

src/mastra/experiments/ingestion.ts

const { experimentId } = await dataset.createExperiment({
  id: 'temporal-wf-run-42',
  name: 'external-eval',
})

Submit one result per item with submitExperimentResult(). Submissions are upserts keyed by (experimentId, itemId, attempt), so a retried worker converges on a single row instead of duplicating results:

await dataset.submitExperimentResult({
  experimentId,
  itemId: 'item-1',
  output: { translation: 'Hola' },
  scores: [{ scorerId: 'accuracy', score: 0.92, reason: 'Faithful translation' }],
})

input and groundTruth default to the dataset item's values at the experiment's pinned dataset version. Inline scores are persisted to the scores store under the experiment, so they show up in comparisons alongside native scorer runs. Experiments created with a target reject submitExperimentResult so two writers can't race on the same rows.

Finalize

When all items are done, call finalizeExperiment(). Mastra computes succeededCount, failedCount, and skippedCount from the persisted rows, so your workers never track completion bookkeeping. Finalization is idempotent. Calling it again returns the stored record:

const experiment = await dataset.finalizeExperiment({ experimentId })

console.log(experiment.status) // 'completed'
console.log(experiment.succeededCount) // items with at least one attempt that didn't error
console.log(experiment.failedCount) // items where every attempt errored
console.log(experiment.skippedCount) // items never submitted

The same lifecycle is available over HTTP (POST /api/datasets/:datasetId/experiments with start: false, POST .../experiments/:experimentId/items/:itemId/run, POST .../experiments/:experimentId/results, POST .../experiments/:experimentId/finalize, see Server routes) and through @mastra/client-js (see the Datasets API).

Retry and idempotency contract

Caller-driven calls are designed to be retried aggressively (for example by a Temporal retry policy). The exact guarantees:

• Create is idempotent on id. Calling createExperiment again with the same caller-supplied id returns the existing experiment. Reusing an id that belongs to another dataset or an experiment with a different target fails with EXPERIMENT_ID_CONFLICT (HTTP 409).
• Results upsert on (experimentId, itemId, attempt). Re-submitting the same key updates the existing row, and the last write wins for all mutable fields. A retried worker never creates duplicates.
• attempt separates deliberate trials from retries. Retries of the same run should reuse the same attempt (default 0) so they converge. To record repeated trials of an item as separate rows, submit with attempt: 0, attempt: 1, and so on.
• The dataset version is pinned at creation. Submissions are validated against the items visible at that version, so editing or deleting items afterward doesn't affect an in-flight experiment. Submitting an itemId that isn't visible at the pinned version fails with 404.
• Finalize is idempotent and terminal. Finalizing an already-completed experiment returns the stored record without recomputing. After finalization, further submissions are rejected with EXPERIMENT_ALREADY_FINALIZED (HTTP 409).
• Counts are server-computed and per-item. At finalize time Mastra rolls attempts up per item: succeededCount (at least one attempt without an error), failedCount (every attempt errored), skippedCount (never submitted). succeededCount + failedCount + skippedCount === totalItems always holds, and callers keep no completion bookkeeping. Attempt-level rows remain available via listExperimentResults.

Configuration options

Concurrency

Control how many items run in parallel (default: 5):

const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  maxConcurrency: 10,
})

Timeouts and retries

Set a per-item timeout (in milliseconds) and retry count:

const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  itemTimeout: 30_000, // 30 seconds per item
  maxRetries: 2, // retry failed items up to 2 times
})

Retries use exponential backoff. Abort errors are never retried.

Aborting an experiment

Pass an AbortSignal to cancel a running experiment:

const controller = new AbortController()

// Cancel after 60 seconds
setTimeout(() => controller.abort(), 60_000)

const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  signal: controller.signal,
})

Remaining items are marked as skipped in the summary.

Pinning a dataset version

Run against a specific snapshot of the dataset:

const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  version: 3, // use items from dataset version 3
})

Viewing results

Listing experiments

const { experiments, pagination } = await dataset.listExperiments({
  page: 0,
  perPage: 10,
})

for (const exp of experiments) {
  console.log(`${exp.name} — ${exp.status} (${exp.succeededCount}/${exp.totalItems})`)
}

Experiment details

const experiment = await dataset.getExperiment({
  experimentId: 'exp-abc-123',
})

console.log(experiment.status)
console.log(experiment.startedAt)
console.log(experiment.completedAt)

📹 Watch

Watch Mastra datasets and experiments workflow to see how datasets and experiments help improve reliability.

Item-level results

const { results, pagination } = await dataset.listExperimentResults({
  experimentId: 'exp-abc-123',
  page: 0,
  perPage: 50,
})

for (const result of results) {
  console.log(result.itemId, result.output, result.error)
}

Understanding the summary

startExperiment() returns an ExperimentSummary with counts and per-item results:

• completedWithErrors is true when the experiment finished but some items failed.
• Items cancelled via signal appear in skippedCount.

Visit the startExperiment reference for the full parameter and return type documentation.

Related

• Datasets overview
• Scorers overview
• startExperiment reference
• listExperimentResults reference