Datasets API
The Datasets API exposes Mastra's dataset and experiment routes from MastraClient. It includes caller-driven experiment methods and experiment deletion methods. The caller-driven methods let an orchestrator you own (for example a Temporal workflow) drive the experiment loop while Mastra acts as the system of record. Create the experiment, then either have Mastra execute each item server-side with runExperimentItem or ingest results you computed yourself with submitExperimentResult, and call finalize when the run is done.
Item runs, result submission, and finalization are safe to retry. Creation is safe to retry only when the request includes a caller-supplied id; without one, each retry creates a new experiment.
createDatasetExperimentis idempotent on a caller-suppliedidrunExperimentItemandsubmitExperimentResultupsert on(experimentId, itemId, attempt)finalizeExperimentreturns the stored record if the experiment is already completed
Usage exampleDirect link to Usage example
import { MastraClient } from '@mastra/client-js'
const client = new MastraClient({
baseUrl: 'http://localhost:4111',
})
// 1. Create the experiment. Pass your orchestrator's run ID so a
// retried workflow converges on the same experiment.
const { experimentId, totalItems, datasetVersion } = await client.createDatasetExperiment({
datasetId: 'clinical-triage-evals',
id: 'temporal-wf-run-42',
targetType: 'agent',
targetId: 'triage-agent',
scorerIds: ['clinical-judge'],
})
// 2. Run one item per activity. Mastra executes the agent, runs the
// scorers, and upserts the result. Retried activities converge on
// a single row per (experimentId, itemId, attempt).
const { result, scores } = await client.runExperimentItem({
datasetId: 'clinical-triage-evals',
experimentId,
itemId: 'item-1',
})
// 3. Finalize. The server computes succeeded/failed/skipped counts
// from the persisted rows.
const experiment = await client.finalizeExperiment({
datasetId: 'clinical-triage-evals',
experimentId,
})
For pure ingestion, create the experiment without targetType/targetId and replace step 2 with submitExperimentResult.
createDatasetExperiment()Direct link to createDatasetExperiment()
Creates an experiment without starting a run. No runner is spawned, and the dataset version is pinned at creation time. Include a target when Mastra should execute items via runExperimentItem. Omit the target for pure ingestion via submitExperimentResult.
datasetId:
id?:
targetType?:
targetId?:
scorerIds?:
name?:
description?:
metadata?:
version?:
provenance?:
source, sourceId, sourceVersion, metadata).grouping?:
experimentSetId, comparisonId, variantId, trialIndex) for organizing related runs.Returns Promise<{ experimentId, status, totalItems, datasetVersion }>.
runExperimentItem()Direct link to runExperimentItem()
Executes one experiment item server-side: Mastra runs the experiment's target against the item, runs the resolved scorers, and upserts the result keyed by (experimentId, itemId, attempt). Requires an experiment created with a target. Scorers resolve with the same precedence as native runs: experiment scorerIds win over item scorerIds, which win over dataset scorerIds.
datasetId:
experimentId:
itemId:
attempt?:
requestContext?:
Returns Promise<{ result, scores }> with the persisted result row and the scores produced for the item.
Calling it on a target-less experiment returns a 400. Calling it after finalization returns a 409.
submitExperimentResult()Direct link to submitExperimentResult()
Submits (or re-submits) one externally computed item result for a target-less experiment. Submitting the same (experimentId, itemId, attempt) key again updates the existing row instead of creating a duplicate. Use a different attempt value to record deliberate repeated trials as separate rows.
datasetId:
experimentId:
itemId:
attempt?:
input?:
output?:
groundTruth?:
error?:
startedAt?:
completedAt?:
traceId?:
scores?:
runId = experimentId, so they appear in comparisons like scorer-produced scores.Returns Promise<DatasetExperimentResult>, the persisted result row.
Submitting to an experiment that has a target returns a 400. Submitting after finalization returns a 409.
finalizeExperiment()Direct link to finalizeExperiment()
Marks a caller-driven experiment completed. The server computes per-item counts from the persisted result rows, so the caller keeps no bookkeeping: succeededCount (at least one attempt without an error), failedCount (every attempt errored), and skippedCount (never submitted). Idempotent: finalizing an already-completed experiment returns the stored record.
datasetId:
experimentId:
Returns Promise<DatasetExperiment>, the updated experiment record.
deleteDatasetExperiment()Direct link to deleteDatasetExperiment()
Deletes an experiment through its dataset. The server deletes the experiment's result records and attempts to delete its observability traces, including their spans and trace-linked signals, but unsupported storage leaves the traces in place and causes the server to log a warning. If trace cleanup fails after an earlier batch succeeds, the promise rejects and preserves the experiment and result records even though some traces may already have been removed.
await client.deleteDatasetExperiment('dataset-id', 'experiment-id', {
organizationId: 'organization-id',
projectId: 'project-id',
})
datasetId:
experimentId:
tenancy.organizationId?:
tenancy.projectId?:
Returns Promise<{ success: boolean }>. A missing experiment, an experiment associated with another dataset, or a dataset outside the supplied tenancy returns a 404 response.
deleteExperiment()Direct link to deleteExperiment()
Deletes an experiment by ID without requiring a dataset reference. Use this method for experiments orphaned by dataset deletion. The server deletes the experiment's result records and attempts to delete its observability traces, but unsupported storage leaves the traces in place and causes the server to log a warning. If trace cleanup fails after an earlier batch succeeds, the promise rejects and preserves the experiment and result records even though some traces may already have been removed.
await client.deleteExperiment('experiment-id', {
organizationId: 'organization-id',
projectId: 'project-id',
})
experimentId:
options.organizationId?:
options.projectId?:
Returns Promise<{ success: boolean }>. An unscoped request returns a 404 response when the experiment doesn't exist. A tenancy-scoped request that doesn't match the experiment returns success without deleting it.
purgeDatasetItem()Direct link to purgeDatasetItem()
Scrubs an item's content from existing dataset history and linked experiment results, including result tags and comments, while preserving version history, experiment counters, and review status. Later result submissions for the item are stored with redacted content, and later dataset item updates are rejected.
await client.purgeDatasetItem('dataset-id', 'item-id', {
organizationId: 'organization-id',
projectId: 'project-id',
})
The optional third argument scopes the purge to a tenant organization and project. The server returns 404 when the dataset doesn't belong to that scope.
Returns Promise<{ success: boolean }>. The operation is idempotent and can't be undone. Purge serializes or conflicts with concurrent dataset item writers without guaranteeing which operation completes first. If a mutating item update loses the race, storage re-reads the purge marker and rejects it with DATASET_ITEM_PURGED. Deletes remain idempotent, and any deletion tombstone created during the race stays redacted. MongoDB storage requires a replica set or sharded deployment with transaction support. See dataset.purgeItem() for the complete purge behavior.