Introducing Experiment Lifecycle Hooks

You can now configure each stage of an experiment run using lifecycle hooks: beforeAll, beforeEach, afterEach, afterAll. Seed a database, spin up a sandbox, copy a fixture — then tear it all down when the run completes.

Running experiments with mocked data is an effective way to catch regressions, but there are occasions where you'll want a production-like environment to gauge real-world behavior. Lifecycle hooks let you create and configure short-lived test environments for each experiment run.

Before lifecycle hooks, preparing a test environment for every experiment meant rolling setup scripts or pointing agents at production data. With lifecycle hooks, experiments can provision their own environment and clean them up when they're finished. An experiment runs the target against each dataset item.

Get started

Running experiments requires a storage adapter to persist datasets and experiment results.

npm install @mastra/libsql

Add Turso environment variables to your LibSQLStore config. When you deploy to the Mastra platform, a Turso database will be automatically provisioned for you:

import { Mastra } from "@mastra/core/mastra";
import { LibSQLStore } from "@mastra/libsql";
 
export const mastra = new Mastra({
  //...
  storage: new LibSQLStore({
    url: process.env.TURSO_DATABASE_URL,
    authToken: process.env.TURSO_AUTH_TOKEN
  })
});

Use hooks to configure the environment at each stage of the experiment lifecycle. This example creates a Neon branch when the experiment starts, seeds it before each test case, resets it after, and deletes the branch when the experiment ends.

await dataset.startExperiment({
  name: "order-lookups",
  targetType: "agent",
  targetId: "supportAgent",
  scorers: ["answer-similarity-scorer"],
  maxConcurrency: 1,
 
  beforeAll: async ({ experimentId }) => {
    await neon.branches.createAndConnect({...});
  },
 
  beforeEach: async ({ item }) => {
    console.log(`[beforeEach] Seeding Neon test branch...\n`);
    for (const s of item.metadata.seed) {
      await sql`INSERT INTO ...`;
    }
  },
  afterEach: async ({ item, result }) => {
    console.log(`[afterEach] Reset database tables.\n`);
    await sql`TRUNCATE ...`;
 
    console.log(item)
    console.log(result)
  },
  afterAll: async () => {
      await neon.branches.delete({...});
      console.log(`[afterAll] Neon branch deleted\n`);
  }
});

For more information and full configuration options, see:

• Lifecycle hooks
• Hook failures
• startExperiment reference