The hardest part of building an agent is ensuring it’s accurate enough to ship into production. Teams can spend weeks or months reviewing user traces and turning them into agent fixes.
That’s why we’re excited to announce Trace Intelligence in beta today. Trace Intelligence is grouping traces into clusters so you can identify common goals, behaviors, sentiments, and outcomes across many runs. It’s built into observability on the Mastra platform.
While building our own agents, Trace Intelligence has helped us prioritize fixes, select traces for evals, and verify that changes improve user experience.
Embedding and clustering traces
Trace Intelligence takes completed traces, extract a compact trace representation along with metadata, generates signals per trace including goal, sentiment, behavior, and outcome, embed each signal and cluster similar signals together.
We use UMAP for dimensionality reduction of signal embeddings and HDBSCAN for density-based clustering in the reduced space. Then, we serve time-windowed views showing theme volume, trends, cross-signal flows, history, and representative traces.
Completing the agent learning loop
Trace intelligence is our second step towards agent learning.
The full agent accuracy loop is something like:
- review traces
- create datasets
- experiment with potential fixes;
- ship fixes to prod
Back in March, we shipped Datasets and Experiments. Now, Trace Intelligence helps with the trace review and selection.
After this, we’ll be working on automating the experiment flow by generating proposed fixes. Soon, you’ll be able to automate the entire agent learning loop within Mastra.
Get Started
Sign up for the beta here: mastra.ai/trace-intelligence.
