Build an Eval Loop for More Reliable Agents
Every team building agents runs into this sooner or later. The demo works, the first users are happy, and then someone changes a prompt or upgrades the model. Something quietly breaks three tool calls into a workflow, and nobody catches it until a customer does.
Gut checks are fine while you’re prototyping. They fall apart in production. So does reading traces one by one once you have real traffic. At that point, you need an eval loop: a consistent way to score what your agent does, catch regressions before you deploy, and learn from the traces you’re already collecting.
We’ll start with a short, practical introduction to evals. You’ll learn what scorers do, when to use an LLM judge, when a deterministic check is enough, and how scoring fits into the day-to-day process of building and maintaining an agent.
Then we’ll get into the code. Most of the workshop is live demos, including building an eval loop in Mastra and reusing the same scorers for offline tests and production monitoring. We’ll show how to catch regressions in CI, run experiments over datasets, and find useful patterns in production traces without inspecting them one at a time.
We’ll cover:
-
Mastra’s built-in scorers for accuracy, hallucination, and tool-call correctness
-
Fast, deterministic checks that do not require another LLM call
-
Custom scorers for behavior specific to your agent
-
Regression tests with runEvals and your existing test framework
-
Dataset experiments for comparing prompts, models, and code changes
-
Mocking tools with side effects so experiments stay repeatable
-
Sampling, tags, and metadata for filtering large volumes of production traces
-
Scoring live traffic and turning useful traces into new dataset items
We’ll work through real code throughout the session. Bring the eval problems you’re dealing with, too. The engineers who built Mastra’s evals, datasets, and tracing will be there to dig into them with you.
Hosted by
-
Alex Booker, Developer Experience, Mastra
-
Yujohn Nattrass, Engineering, Mastra
Recording and code examples will be available to everyone who registers.
Workshop Hosts
