Introducing Classifiers with Jev

You can now use evaluation models in Mastra with a classifier. Define questions as choice, score, or boolean, then set the criteria to evaluate, and use the answer to make fast, predictable decisions.

With an evaluation model, you provide instructions describing what to decide and criteria outlining what can be answered. A choice returns one of the criteria keys, a score returns a position from an ordered list, and a boolean returns the probability the answer is true.

We shipped this after the dramatic rise of Jev a couple weeks ago, and ran a workshop showing how it works: Building a Classifier in Mastra with Jev.

Before classifiers, picking from a fixed set of options meant prompting an LLM, describing the options, defining the output shape, then parsing and validating whatever came back. With a classifier, the options are the criteria, and the answer shows why the decision was made, with a probability for each option, so your code can decide what happens next.

A classifier can hold several questions of mixed types, all evaluated against the same input, with each answer keyed by the question name you gave it. instructions and criteria aren't limited to strings either — both accept any JSON-compatible value, so a criteria description can be an object or an array.

Get started

Install the latest Mastra core package and an evaluation model provider:

Terminal

npm install @mastra/core @ai-sdk/typesafe-ai

note

Requires @mastra/core@1.69.0 or later, added in PR #24458.

Define a classifier with an id, a model, and one or more questions. This example uses a route question with the "choice" type to pick one key from the defined criteria.

src/mastra/classifiers/marketing-classifier.ts

import { Classifier } from "@mastra/core/classifier";
import { createTypeSafeAi } from "@ai-sdk/typesafe-ai";
 
export const marketingClassifier = new Classifier({
  id: "marketing-classifier",
  model: createTypeSafeAi({ apiKey: process.env.TYPESAFE_AI_API_KEY }).evaluationModel("jev-latest"),
  questions: {
    route: {
      type: "choice",
      instructions: "Choose which marketing agent should handle this request.",
      criteria: {
        pmmLaunchAgent: "What has and hasn't launched",
        releaseNotesAgent: "Writing release notes",
        newsletterAgent: "Drafting the weekly newsletter"
      }
    }
  }
});

Classifiers are registered on the main Mastra instance, the same as agents, tools and workflows:

src/mastra/index.ts

import { Mastra } from "@mastra/core";
import { marketingClassifier } from "./classifiers/marketing-classifier";
 
export const mastra = new Mastra({
  agents: {
    /* ... */
  },
  tools: {
    /* ... */
  },
  workflows: {
    /* ... */
  },
  classifiers: { marketingClassifier }
});

A classifier will return an answers object similar to this:

answers: {
  "route": {
    "type": "choice",
    "choice": "pmmLaunchAgent",
    "probabilities": {
      "newsletterAgent": 0,
      "releaseNotesAgent": 0,
      "pmmLaunchAgent": 1
    }
  }
}
agent: pmm-launch-agent // choice

Each criteria key gets a probability, and the highest one becomes the choice. A workflow can use the choice to drive the branch logic and pick an appropriate agent to answer the user's query:

src/mastra/workflows/marketing-router-workflow.ts

import { createStep, createWorkflow } from "@mastra/core/workflows";
import { z } from "zod";
import { newsletterAgent } from "../agents/newsletter-agent";
import { pmmLaunchAgent } from "../agents/pmm-launch-agent";
import { releaseNotesAgent } from "../agents/release-notes-agent";
import { marketingClassifier } from "../classifiers/marketing-classifier";
 
export const marketingRouterWorkflow = createWorkflow({
  id: "marketing-router-workflow",
  description: "Routes a marketing request to the right agent",
  inputSchema: z.object({ message: z.string() }),
  outputSchema: z.any()
})
  .classifier(marketingClassifier)
  .map(async ({ inputData, getInitData }) => {
    return {
      choice: inputData.answers.route.choice,
      prompt: getInitData<{ message: string }>().message
    };
  })
  .branch([
    [
      async ({ inputData }) => {
        return inputData.choice === "pmmLaunchAgent";
      },
      createStep(pmmLaunchAgent)
    ],
    [
      async ({ inputData }) => {
        return inputData.choice === "releaseNotesAgent";
      },
      createStep(releaseNotesAgent)
    ],
    [
      async ({ inputData }) => {
        return inputData.choice === "newsletterAgent";
      },
      createStep(newsletterAgent)
    ]
  ])
  .commit();

For more information and full configuration options, see: