Datasets API
The Datasets API exposes Mastra's dataset and experiment routes from MastraClient. This page covers the caller-driven experiment methods, which let an orchestrator you own (for example a Temporal workflow) drive the experiment loop while Mastra acts as the system of record. Create the experiment, then either have Mastra execute each item server-side with runExperimentItem or ingest results you computed yourself with submitExperimentResult, and call finalize when the run is done.
Item runs, result submission, and finalization are safe to retry. Creation is safe to retry only when the request includes a caller-supplied id; without one, each retry creates a new experiment.
createDatasetExperimentis idempotent on a caller-suppliedidrunExperimentItemandsubmitExperimentResultupsert on(experimentId, itemId, attempt)finalizeExperimentreturns the stored record if the experiment is already completed
Usage exampleDirect link to Usage example
import { MastraClient } from '@mastra/client-js'
const client = new MastraClient({
baseUrl: 'http://localhost:4111',
})
// 1. Create the experiment. Pass your orchestrator's run ID so a
// retried workflow converges on the same experiment.
const { experimentId, totalItems, datasetVersion } = await client.createDatasetExperiment({
datasetId: 'clinical-triage-evals',
id: 'temporal-wf-run-42',
targetType: 'agent',
targetId: 'triage-agent',
scorerIds: ['clinical-judge'],
})
// 2. Run one item per activity. Mastra executes the agent, runs the
// scorers, and upserts the result. Retried activities converge on
// a single row per (experimentId, itemId, attempt).
const { result, scores } = await client.runExperimentItem({
datasetId: 'clinical-triage-evals',
experimentId,
itemId: 'item-1',
})
// 3. Finalize. The server computes succeeded/failed/skipped counts
// from the persisted rows.
const experiment = await client.finalizeExperiment({
datasetId: 'clinical-triage-evals',
experimentId,
})
For pure ingestion, create the experiment without targetType/targetId and replace step 2 with submitExperimentResult.
createDatasetExperiment()Direct link to createDatasetExperiment()
Creates an experiment without starting a run. No runner is spawned, and the dataset version is pinned at creation time. Include a target when Mastra should execute items via runExperimentItem. Omit the target for pure ingestion via submitExperimentResult.
datasetId:
id?:
targetType?:
targetId?:
scorerIds?:
name?:
description?:
metadata?:
version?:
provenance?:
source, sourceId, sourceVersion, metadata).grouping?:
experimentSetId, comparisonId, variantId, trialIndex) for organizing related runs.Returns Promise<{ experimentId, status, totalItems, datasetVersion }>.
runExperimentItem()Direct link to runExperimentItem()
Executes one experiment item server-side: Mastra runs the experiment's target against the item, runs the resolved scorers, and upserts the result keyed by (experimentId, itemId, attempt). Requires an experiment created with a target. Scorers resolve with the same precedence as native runs: experiment scorerIds win over item scorerIds, which win over dataset scorerIds.
datasetId:
experimentId:
itemId:
attempt?:
requestContext?:
Returns Promise<{ result, scores }> with the persisted result row and the scores produced for the item.
Calling it on a target-less experiment returns a 400. Calling it after finalization returns a 409.
submitExperimentResult()Direct link to submitExperimentResult()
Submits (or re-submits) one externally computed item result for a target-less experiment. Submitting the same (experimentId, itemId, attempt) key again updates the existing row instead of creating a duplicate. Use a different attempt value to record deliberate repeated trials as separate rows.
datasetId:
experimentId:
itemId:
attempt?:
input?:
output?:
groundTruth?:
error?:
startedAt?:
completedAt?:
traceId?:
scores?:
runId = experimentId, so they appear in comparisons like scorer-produced scores.Returns Promise<DatasetExperimentResult>, the persisted result row.
Submitting to an experiment that has a target returns a 400. Submitting after finalization returns a 409.
finalizeExperiment()Direct link to finalizeExperiment()
Marks a caller-driven experiment completed. The server computes per-item counts from the persisted result rows, so the caller keeps no bookkeeping: succeededCount (at least one attempt without an error), failedCount (every attempt errored), and skippedCount (never submitted). Idempotent: finalizing an already-completed experiment returns the stored record.
datasetId:
experimentId:
Returns Promise<DatasetExperiment>, the updated experiment record.