Skip to main content

dataset.createExperiment()

Added in: @mastra/core@1.61.0

Creates an experiment without starting a run. Your own orchestrator (for example a Temporal workflow) drives the loop in one of two shapes:

  • With a target: pass targetType and targetId, then call runExperimentItem() per item. Mastra executes the target and runs the scorers server-side.
  • Without a target: omit both and run everything on your own infrastructure, ingesting each result with submitExperimentResult().

Both shapes finish with finalizeExperiment().

The experiment pins the dataset version at creation time, so submissions are validated against a stable set of items even if the dataset changes afterwards.

Usage example
Direct link to Usage example

import { Mastra } from '@mastra/core'

const mastra = new Mastra({/* storage config */})

const dataset = await mastra.datasets.get({ id: 'dataset-id' })

const { experimentId, totalItems, datasetVersion } = await dataset.createExperiment({
  id: 'temporal-wf-run-42', // optional: idempotent create on retry
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy'],
})

Passing your own id makes creation idempotent: calling it again with the same id returns the existing experiment instead of failing, so a retried workflow activity is safe. If the id belongs to an experiment on another dataset or an experiment with a different target, the call throws an EXPERIMENT_ID_CONFLICT error.

targetType and targetId must be provided together, and the target must exist in the Mastra registry at create time. scorers requires a target because Mastra never scores target-less experiments; submit flat scores through submitExperimentResult instead.

Parameters
Direct link to Parameters

id?:

string
Caller-supplied experiment ID (for example a workflow run ID). Makes creation idempotent on retry.

targetType?:

'agent' | 'workflow' | 'scorer'
Type of target that runExperimentItem() executes. Provide together with targetId, or omit both for pure ingestion.

targetId?:

string
ID of the registered target. Provide together with targetType.

scorers?:

string[]
Run-level scorer IDs resolved server-side by runExperimentItem(). Requires a target.

name?:

string
Human-readable experiment name.

description?:

string
Experiment description.

metadata?:

Record<string, unknown>
Arbitrary metadata stored on the experiment.

provenance?:

ExperimentProvenance
Where the experiment came from (source system, ID, and version).

grouping?:

ExperimentGrouping
Grouping fields (experimentSetId, comparisonId, variantId, trialIndex) for organizing related runs.

version?:

number
Dataset version to pin. Defaults to the current dataset version.

Returns
Direct link to Returns

result:

Promise<object>
Immediate response with experiment ID.
object

experimentId:

string
ID of the created (or existing) experiment.

status:

ExperimentStatus
'running' for a new experiment; the stored status when an existing experiment is returned.

totalItems:

number
Number of dataset items visible at the pinned version.

datasetVersion:

number
The pinned dataset version.
On this page