> Discover all available pages from the documentation index: https://mastra.ai/llms.txt

# Running experiments

**Added in:** `@mastra/core@1.4.0`

An experiment runs every item in a dataset through a target (an agent, a workflow, or a scorer) and then optionally scores the outputs. Use a scorer as the target when you want to evaluate an LLM judge itself. By default, results are persisted to storage so you can compare runs across different prompts, models, or code changes.

**For AI agents:** Run `npx mastra api experiment run dataset_123 '{"name":"translation-baseline"}'` to start an experiment directly instead of opening Studio or writing a temporary script. Use a dataset ID returned by `npx mastra api dataset list` in place of the sample ID. The command requires a running Mastra server with dataset storage and registered experiment targets; start the local server with `npx mastra dev`, or pass the reachable server's base URL with `--url`. Run `npx mastra api experiment run --schema` before constructing different input, and get user approval before starting an experiment because it can make model calls. Install Mastra's skill with `npx skills add mastra-ai/skills --skill mastra` for complete API CLI discovery, targeting, schema, authentication, and error-handling guidance.

## Basic experiment

Call [`startExperiment()`](https://mastra.ai/reference/datasets/startExperiment) with a target and scorers:

```typescript
import { mastra } from '../index'

const dataset = await mastra.datasets.get({ id: 'translation-dataset-id' })

const summary = await dataset.startExperiment({
  name: 'gpt-5.1-baseline',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy', 'fluency'],
})

console.log(summary.status) // 'completed' | 'failed'
console.log(summary.succeededCount) // number of items that ran successfully
console.log(summary.failedCount) // number of items that failed
```

`startExperiment()` blocks until all items finish. For fire-and-forget execution, see [async experiments](#async-experiments).

## Studio

You can also run experiments in [Studio](https://mastra.ai/docs/studio/overview). After you've added a dataset item, open it and select **Run Experiment** and configure the target, scorers, and options.

To configure scorers for one item, edit the item and enable **Override dataset scorers**. Select one or more scorers, or leave the selection empty to run no scorers for that item.

After running an experiment, the **Experiments** tab shows all runs for that dataset (with status, counts, and timestamps). Select an experiment to see per-item results, scores, and execution traces.

In the **Experiments** tab, select **Compare** and choose two or more experiments to compare their scores and results side by side.

## Experiment targets

You can point an experiment at a registered agent, workflow, or scorer.

### Registered agent

Point to an agent registered on your Mastra instance:

```typescript
const summary = await dataset.startExperiment({
  name: 'agent-v2-eval',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy'],
})
```

Each item's `input` is passed directly to `agent.generate()`, so it must be a `string`, `string[]`, or `CoreMessage[]`.

#### Memory-enabled agents

When the target agent has its own memory and the request context carries a resource id (`MASTRA_RESOURCE_ID_KEY`, set by auth middleware, the experiment or item `requestContext`, or the Studio **Run Experiment** form), the experiment runner injects a fresh memory thread for each item. A resource id in the request context means "run as this resource": each item's conversation persists as a thread under that resource, and retried items get a new thread per attempt so earlier failed attempts can't leak into the retry's context.

Injected threads are tagged so you can map them back to the run: thread metadata carries the `experimentId` and the dataset item's id as `experimentItemId`. No thread title is generated for them.

Because the threads belong to the caller's resource, resource-scoped memory features both read and write that resource's state during the run:

- Resource-scoped working memory updates persist to the resource, and later items in the run see updates made by earlier items.
- Resource-scoped semantic recall can surface the resource's prior conversations to the experiment, and experiment transcripts become recallable in that resource's later conversations.

Use this approach to evaluate an agent against a real user's accumulated context. To keep experiment runs from touching real user state, use a dedicated evaluation resource id instead.

Thread injection is skipped in the following cases:

- If the request context also sets `MASTRA_THREAD_ID_KEY`, the runner uses that thread as-is, so every item (and retry) shares the same conversation.
- If the agent has no memory, or the request context has no resource id, the run is memoryless and nothing is persisted.

### Registered workflow

Point to a workflow registered on your Mastra instance:

```typescript
const summary = await dataset.startExperiment({
  name: 'workflow-eval',
  targetType: 'workflow',
  targetId: 'translation-workflow',
  scorers: ['accuracy'],
})
```

The workflow receives each item's `input` as its trigger data.

### Registered scorer

Point to a scorer to evaluate an LLM judge against ground truth:

```typescript
const summary = await dataset.startExperiment({
  name: 'judge-accuracy-eval',
  targetType: 'scorer',
  targetId: 'accuracy',
})
```

The scorer receives each item's `input` and `groundTruth`. LLM-based judges can drift over time as underlying models change, so it's important to periodically realign them against known-good labels. A dataset gives you a stable benchmark to detect that drift.

## Scoring results

Scorers automatically run after each item's target execution. Pass scorer instances or registered scorer IDs:

**Scorer IDs**:

```typescript
// Reference scorers registered on the Mastra instance
const summary = await dataset.startExperiment({
  name: 'with-registered-scorers',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy', 'fluency'],
})
```

**Scorer instances**:

```typescript
import { createAnswerRelevancyScorer } from '@mastra/evals/scorers/prebuilt'

const relevancy = createAnswerRelevancyScorer({ model: 'openai/gpt-5-mini' })

const summary = await dataset.startExperiment({
  name: 'with-scorer-instances',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: [relevancy],
})
```

Each item's results include per-scorer scores:

```typescript
for (const item of summary.results) {
  console.log(item.itemId, item.output)
  for (const score of item.scores) {
    console.log(`  ${score.scorerName}: ${score.score} — ${score.reason}`)
  }
}
```

Visit the [Scorers overview](https://mastra.ai/docs/evals/overview) for details on available and custom scorers.

## Select scorers per item

Add registered scorer IDs to a dataset item to override the scorers attached to its dataset:

```typescript
await dataset.addItem({
  input: 'Translate "hello" to French.',
  scorerIds: ['accuracy', 'fluency'],
})
```

An experiment uses exactly one scorer source for each item, in this order:

1. The experiment's `scorers` option, when provided
2. The item's `scorerIds` field, when provided
3. The dataset's `scorerIds` field
4. No scorers

Mastra doesn't merge these sources. An explicit empty array at the run or item level selects no scorers and prevents fallback to the next source. An empty categorized run-level configuration has the same effect. Duplicate item or dataset IDs run once, in the order of their first occurrence.

The following updates switch between an explicit empty override and dataset inheritance:

```typescript
// Run no scorers for this item
await dataset.updateItem({
  itemId: 'translation-item-id',
  scorerIds: [],
})

// Remove the item override and inherit the dataset's scorers
await dataset.updateItem({
  itemId: 'translation-item-id',
  scorerIds: null,
})
```

Omitting `scorerIds` from an update preserves its current value. Each item version stores its scorer ID list, including an empty list. Scorer definitions aren't copied into the item version and continue to use their existing registry or Editor behavior.

Mastra resolves each item's scorer IDs before running its target. It checks registered scorers first, then asks the Editor to hydrate a stored scorer. A stale item-level ID produces the `EXPERIMENT_ITEM_SCORER_NOT_FOUND` code for that item. Mastra skips target execution without retrying that item, while the rest of the experiment continues.

A missing ID in the selected run-level or dataset-level source fails experiment setup. Mastra doesn't resolve IDs from lower-priority sources that the precedence rules ignore.

Workflow step scorers are available only through a run-level categorized `scorers` configuration. Item-level IDs apply to the flat agent, workflow, and trajectory scorer dispatch.

## Control persistence per run

Use `persistence` to skip storage writes for a specific run. Experiment records and score records can be disabled independently:

```typescript
const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy'],
  persistence: {
    experiments: 'none',
    scores: 'none',
  },
})
```

The target and scorers still run, and `startExperiment()` still returns the item results and scores in `summary`. The settings are independent. For example, set only `scores: 'none'` to persist the experiment and its item results without creating score records.

Omitted settings default to `'default'`, which preserves the standard storage behavior. This policy only controls experiment and score records created by the run. It doesn't disable storage used by the target, such as agent memory, vectors, observability, or custom tool storage.

When `startExperimentAsync()` runs with `experiments: 'none'`, it doesn't persist an experiment record, progress updates, or item results. Score persistence remains controlled separately by `persistence.scores`. Without an experiment event observer, the run is fire-and-forget, and the experiment API can't report whether it completed or failed.

Use synchronous `startExperiment()` when the caller needs the returned summary. An experiment event observer can receive lifecycle events and the terminal summary.

## Observe experiment events

Use `onEvent` to receive versioned, JSON-safe lifecycle events while an experiment runs. This works with `startExperiment()`, `startExperimentAsync()`, and `runExperiment()`.

```typescript
import type { ExperimentEvent } from '@mastra/core/datasets'

const events: ExperimentEvent[] = []

await dataset.startExperimentAsync({
  task: async ({ input }) => processItem(input),
  persistence: { experiments: 'none' },
  onEvent: async event => {
    events.push(event)
    await publishEvent(event)
  },
})
```

The observer receives these event types:

- `experiment.run.started`: Identifies the run, target, resolved dataset version, and item count.
- `experiment.item.completed`: Reports a committed item result after scoring, including scores, errors, retry count, tool mock details, and stable item identity.
- `experiment.run.finished`: Reports the terminal outcome and summary counters.

Mastra awaits each observer call before delivering the next event. This serialized delivery applies backpressure and ensures event `sequence` values match delivery order, while item execution can remain concurrent.

If the observer throws or rejects, Mastra aborts the remaining run and rejects `runExperiment()` with a `MastraError` whose `id` is `EXPERIMENT_EVENT_OBSERVER_FAILED`. It doesn't send a terminal event through the failed observer. For `startExperimentAsync()`, the method has already returned when a detached observer fails, so handle delivery failures inside the observer when the caller needs direct error reporting.

The `experiment.run.finished` event is awaited before Mastra persists the final experiment status. Treat the event as the authoritative terminal signal when experiment persistence is disabled, but don't use it as a read-after-write signal for storage.

The exported event types are `ExperimentEvent`, `ExperimentRunStartedEvent`, `ExperimentItemCompletedEvent`, and `ExperimentRunFinishedEvent`. Use the discriminated `type` field to narrow an event before reading event-specific properties.

## Lifecycle hooks

Use lifecycle hooks to prepare state before a target runs and clean it up afterward. This is useful when an item can't be evaluated against an empty environment. A run might need a fixture file copied into the agent's workspace, or a sandbox provisioned before the agent can touch it.

Hooks run at two levels. `beforeAll` and `afterAll` run once per experiment, and `beforeEach` and `afterEach` run once per item:

```typescript
const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'document-agent',
  scorers: ['accuracy'],
  beforeAll: async ({ experimentId }) => {
    await createWorkspace(experimentId)
  },
  beforeEach: async ({ item }) => {
    await copyFixture(item.metadata?.fixture)
  },
  afterEach: async ({ item, result }) => {
    await clearWorkspaceFiles(item.id)
  },
  afterAll: async ({ summary }) => {
    await deleteWorkspace(summary.experimentId)
  },
})
```

Every hook can be async. Each one receives the `experimentId`, the `mastra` instance, and the run-level `signal`, so long-running setup can be cancelled along with the experiment. The per-item hooks also receive `item`. The teardown hooks receive the result they follow: `afterEach` receives the item's `result` including scores, and `afterAll` receives the `summary` that's about to be returned.

The item passed to hooks exposes `id`, `input`, `groundTruth`, and `metadata`. Fields that control execution, such as tool mocks and scorer selection, aren't exposed, so a hook can't change how the item runs.

### Hook failures

Each hook has a different consequence when it throws, based on how much of the run depends on it:

| Hook         | On failure                                                                       |
| ------------ | -------------------------------------------------------------------------------- |
| `beforeAll`  | Fails the experiment. No items run.                                              |
| `beforeEach` | Fails that item with `EXPERIMENT_ITEM_BEFORE_EACH_FAILED`. Other items continue. |
| `afterEach`  | Logged. The item's recorded outcome doesn't change.                              |
| `afterAll`   | Logged. The returned summary doesn't change.                                     |

When `beforeAll` fails, the experiment is marked failed and the `experiment.run.finished` event is still emitted before the error propagates.

When `beforeEach` fails, the target and its scorers are skipped for that item, since the item's preconditions were never met. `afterEach` is also skipped for that item, on the basis that setup which didn't finish owns its own cleanup.

Teardown failures are logged rather than propagated. By the time `afterEach` runs, the target has already produced a real result, and discarding it because cleanup was untidy would lose the data the experiment was run to collect.

`afterAll` runs on every exit path, including when the experiment fails, when `beforeAll` fails, and when an [event observer](#observe-experiment-events) fails, so teardown isn't skipped when something goes wrong. It runs at most once per experiment.

## Tool mocks

When an experiment runs an agent that calls side-effecting tools, attach static tool mocks to individual dataset items to make the run deterministic. During the experiment, a mocked tool returns its declared output instead of executing. Tools without a mock on the item run live by default.

Mocks live on the dataset item, so they version with the row and travel with the test case. Each mock declares a tool name, the arguments it expects, and the output to return:

```typescript
await dataset.addItem({
  input: 'What is the weather in Seattle?',
  toolMocks: [
    {
      toolName: 'getWeather',
      args: { city: 'Seattle' },
      output: { temperature: 60, conditions: 'rainy' },
    },
  ],
})
```

Tool mocks are supported for `agent` targets only.

### Block undeclared tools

Set `unmockedToolPolicy: 'deny'` on an experiment to block every tool call that doesn't have a mock. This is useful when a live call could cause side effects:

```typescript
const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'weather-agent',
  unmockedToolPolicy: 'deny',
})
```

The default policy is `'allow'`. You can override the experiment policy on an individual stored or inline item:

```typescript
await dataset.addItem({
  input: 'What is the weather in Seattle?',
  unmockedToolPolicy: 'allow',
})
```

The item value takes precedence over the experiment value. A denied call fails with `TOOL_MOCK_NOT_DECLARED` before the tool executes. The failure isn't retried or added to `liveCalls`.

### Matching and consumption

Arguments are matched strictly: object key order is ignored and array order is substantial, plus there is no type coercion. A mock is served only when the agent calls the tool with arguments that deep-equal the mock's `args`.

When an item declares several mocks for the same tool and arguments, they're consumed in order, the first call gets the first mock, the next call gets the second, and so on. Ordering is tracked per `(toolName, args)` group and is independent across different arguments.

### Matching mode

By default each mock matches strictly on its `args`. Set `matchArgs: 'ignore'` to match on the tool name only, the mock's `args` aren't compared and the next unconsumed mock for that tool is served regardless of how the agent called it:

```typescript
const subAgentMock = {
  toolName: 'agent-balanceAgent',
  args: { prompt: 'look up the balance for YJ' },
  output: { text: "YJ's balance is $100." },
  matchArgs: 'ignore',
}
```

This is useful when a tool's arguments are noisy or generated by the model. The most common case is mocking a **sub-agent's response**: a delegated sub-agent is exposed to the parent as an `agent-<name>` tool, and its arguments include an LLM-authored `prompt` plus runtime-injected fields. Mocking `agent-<name>` returns the canned response in place of running the sub-agent and its inner tools. When you create a mock from a trace, sub-agent delegation calls are derived with `matchArgs: 'ignore'` automatically. You can change it to `'strict'` to pin the exact arguments.

### Failures

A tool call fails the item when it violates the mock configuration:

- `TOOL_MOCK_MISMATCH`: the tool was called with arguments that no mock matches.
- `TOOL_MOCK_EXHAUSTED`: every matching mock has already been consumed.
- `TOOL_MOCK_NOT_DECLARED`: the tool has no mock and the effective `unmockedToolPolicy` is `'deny'`.

On any of these failures, the agent run is aborted immediately, so the model can't go on to call any further tools, including unmocked, side-effecting tools that would otherwise run live. These failures are deterministic, so they're not retried. Mocks that are declared but never used don't fail the item, they're reported as unconsumed.

While mock interception is active, the agent's tools execute sequentially so repeated `(toolName, args)` mocks are consumed in the provider's call order. Interception is active when the item declares mocks or its effective `unmockedToolPolicy` is `'deny'`.

### Diagnostics

Each item result carries a `toolMockReport` describing what the run did with the item's mocks:

```typescript
for (const item of summary.results) {
  const report = item.toolMockReport
  if (!report) continue

  console.log(report.served) // mocks matched and returned
  console.log(report.unconsumed) // mocks declared but never used
  console.log(report.liveCalls) // undeclared tools allowed to run live
  console.log(report.failure) // the first deterministic mock failure, if any
}
```

In [Studio](https://mastra.ai/docs/studio/overview), edit a dataset item to author tool mocks as a JSON array, and open an experiment result to see the same report.

### Limitations

- **No tool span for mocked calls.** A mocked call returns its output before the tool executes, so it doesn't create a tool span. Trajectory scorers backed by stored traces may therefore not see mocked tool calls. Trajectory extraction that falls back to the agent's message output still sees them, so trajectory scoring can differ depending on your observability configuration.
- **Storage support.** The LibSQL, PostgreSQL, MongoDB, and Spanner adapters persist tool mocks and tool mock reports, while the MySQL adapter rejects writes that carry either one. All dataset storage adapters persist `unmockedToolPolicy`.

## Async experiments

`startExperiment()` blocks until every item completes. For long-running datasets, use [`startExperimentAsync()`](https://mastra.ai/reference/datasets/startExperimentAsync) to start the experiment in the background:

```typescript
const { experimentId, status } = await dataset.startExperimentAsync({
  name: 'large-dataset-run',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy'],
})

console.log(experimentId) // UUID
console.log(status) // 'pending'
```

Poll for completion using [`getExperiment()`](https://mastra.ai/reference/datasets/getExperiment):

```typescript
let experiment = await dataset.getExperiment({ experimentId })

while (experiment.status === 'pending' || experiment.status === 'running') {
  await new Promise(resolve => setTimeout(resolve, 5000))
  experiment = await dataset.getExperiment({ experimentId })
}

console.log(experiment.status) // 'completed' | 'failed'
```

## Caller-driven experiments

`startExperiment()` puts Mastra in charge of the whole run. If a durable orchestrator (for example Temporal, Airflow, or a custom worker fleet) owns the loop instead, create the experiment first and drive it yourself in one of these shapes:

- **Caller drives the loop, Mastra runs each item.** Create the experiment with a target, then call [`runExperimentItem()`](https://mastra.ai/reference/datasets/runExperimentItem) once per item. Mastra executes the target and runs the scorers, then persists the result.
- **Caller runs everything.** Create the experiment without a target, execute and score items on your own infrastructure, and ingest each result with [`submitExperimentResult()`](https://mastra.ai/reference/datasets/submitExperimentResult).

Both shapes finish with [`finalizeExperiment()`](https://mastra.ai/reference/datasets/finalizeExperiment), and both write to the same tables as native runs, so Studio views, comparisons, and review summaries work unchanged.

### Run items server-side

Create the experiment with [`createExperiment()`](https://mastra.ai/reference/datasets/createExperiment), passing the target and scorers. Pass your own `id` (for example a workflow run id) to make creation idempotent: retrying the call with the same id returns the existing experiment instead of failing.

```typescript
const dataset = await mastra.datasets.get({ id: 'translation-dataset-id' })

const { experimentId, totalItems, datasetVersion } = await dataset.createExperiment({
  id: 'temporal-wf-run-42', // optional: reuse your workflow run id for idempotent creates
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy'],
})
```

Then run each item from your orchestrator. Mastra resolves the item at the pinned dataset version and executes the target with the resolved scorers. The result upserts keyed by `(experimentId, itemId, attempt)`, so a retried activity converges on a single row:

```typescript
const { result, scores } = await dataset.runExperimentItem({
  experimentId,
  itemId: 'item-1',
})
```

Retries and timeouts belong to your orchestrator: each `runExperimentItem` call executes the item exactly once. Scorers resolve with the same precedence as native runs: experiment `scorers` win over item `scorerIds`, which win over dataset `scorerIds`.

### Ingest external results

If your workers execute and score items themselves, create the experiment without a target:

```typescript
const { experimentId } = await dataset.createExperiment({
  id: 'temporal-wf-run-42',
  name: 'external-eval',
})
```

Submit one result per item with [`submitExperimentResult()`](https://mastra.ai/reference/datasets/submitExperimentResult). Submissions are upserts keyed by `(experimentId, itemId, attempt)`, so a retried worker converges on a single row instead of duplicating results:

```typescript
await dataset.submitExperimentResult({
  experimentId,
  itemId: 'item-1',
  output: { translation: 'Hola' },
  scores: [{ scorerId: 'accuracy', score: 0.92, reason: 'Faithful translation' }],
})
```

`input` and `groundTruth` default to the dataset item's values at the experiment's pinned dataset version. Inline `scores` are persisted to the scores store under the experiment, so they show up in comparisons alongside native scorer runs. Experiments created with a target reject `submitExperimentResult` so two writers can't race on the same rows.

### Finalize

When all items are done, call [`finalizeExperiment()`](https://mastra.ai/reference/datasets/finalizeExperiment). Mastra computes `succeededCount`, `failedCount`, and `skippedCount` from the persisted rows, so your workers never track completion bookkeeping. Finalization is idempotent. Calling it again returns the stored record:

```typescript
const experiment = await dataset.finalizeExperiment({ experimentId })

console.log(experiment.status) // 'completed'
console.log(experiment.succeededCount) // items with at least one attempt that didn't error
console.log(experiment.failedCount) // items where every attempt errored
console.log(experiment.skippedCount) // items never submitted
```

The same lifecycle is available over HTTP (`POST /api/datasets/:datasetId/experiments` with `start: false`, `POST .../experiments/:experimentId/items/:itemId/run`, `POST .../experiments/:experimentId/results`, `POST .../experiments/:experimentId/finalize`, see [Server routes](https://mastra.ai/reference/server/routes)) and through `@mastra/client-js` (see the [Datasets API](https://mastra.ai/reference/client-js/datasets)).

### Retry and idempotency contract

Caller-driven calls are designed to be retried aggressively (for example by a Temporal retry policy). The exact guarantees:

- **Create is idempotent on `id`.** Calling `createExperiment` again with the same caller-supplied `id` returns the existing experiment. Reusing an `id` that belongs to another dataset or an experiment with a different target fails with `EXPERIMENT_ID_CONFLICT` (HTTP `409`).
- **Results upsert on `(experimentId, itemId, attempt)`.** Re-submitting the same key updates the existing row, and the last write wins for all mutable fields. A retried worker never creates duplicates.
- **`attempt` separates deliberate trials from retries.** Retries of the same run should reuse the same `attempt` (default `0`) so they converge. To record repeated trials of an item as separate rows, submit with `attempt: 0`, `attempt: 1`, and so on.
- **The dataset version is pinned at creation.** Submissions are validated against the items visible at that version, so editing or deleting items afterward doesn't affect an in-flight experiment. Submitting an `itemId` that isn't visible at the pinned version fails with `404`.
- **Finalize is idempotent and terminal.** Finalizing an already-completed experiment returns the stored record without recomputing. After finalization, further submissions are rejected with `EXPERIMENT_ALREADY_FINALIZED` (HTTP `409`).
- **Counts are server-computed and per-item.** At finalize time Mastra rolls attempts up per item: `succeededCount` (at least one attempt without an error), `failedCount` (every attempt errored), `skippedCount` (never submitted). `succeededCount + failedCount + skippedCount === totalItems` always holds, and callers keep no completion bookkeeping. Attempt-level rows remain available via `listExperimentResults`.

## Configuration options

### Concurrency

Control how many items run in parallel (default: 5):

```typescript
const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  maxConcurrency: 10,
})
```

### Timeouts and retries

Set a per-item timeout (in milliseconds) and retry count:

```typescript
const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  itemTimeout: 30_000, // 30 seconds per item
  maxRetries: 2, // retry failed items up to 2 times
})
```

Retries use exponential backoff. Abort errors are never retried.

### Aborting an experiment

Pass an `AbortSignal` to cancel a running experiment:

```typescript
const controller = new AbortController()

// Cancel after 60 seconds
setTimeout(() => controller.abort(), 60_000)

const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  signal: controller.signal,
})
```

Remaining items are marked as skipped in the summary.

### Pinning a dataset version

Run against a specific snapshot of the dataset:

```typescript
const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  version: 3, // use items from dataset version 3
})
```

## Viewing results

### Listing experiments

```typescript
const { experiments, pagination } = await dataset.listExperiments({
  page: 0,
  perPage: 10,
})

for (const exp of experiments) {
  console.log(`${exp.name} — ${exp.status} (${exp.succeededCount}/${exp.totalItems})`)
}
```

### Experiment details

```typescript
const experiment = await dataset.getExperiment({
  experimentId: 'exp-abc-123',
})

console.log(experiment.status)
console.log(experiment.startedAt)
console.log(experiment.completedAt)
```

> **📹 Watch:** Watch [Mastra datasets and experiments workflow](https://www.youtube.com/watch?v=R6pjAdGhxhQ) to see how datasets and experiments help improve reliability.

### Item-level results

```typescript
const { results, pagination } = await dataset.listExperimentResults({
  experimentId: 'exp-abc-123',
  page: 0,
  perPage: 50,
})

for (const result of results) {
  console.log(result.itemId, result.output, result.error)
}
```

## Understanding the summary

`startExperiment()` returns an `ExperimentSummary` with counts and per-item results:

- `completedWithErrors` is `true` when the experiment finished but some items failed.
- Items cancelled via `signal` appear in `skippedCount`.

Visit the [`startExperiment` reference](https://mastra.ai/reference/datasets/startExperiment) for the full parameter and return type documentation.

## Related

- [Datasets overview](https://mastra.ai/docs/datasets/overview)
- [Scorers overview](https://mastra.ai/docs/evals/overview)
- [`startExperiment` reference](https://mastra.ai/reference/datasets/startExperiment)
- [`listExperimentResults` reference](https://mastra.ai/reference/datasets/listExperimentResults)