A lot of teams building on Mastra also use Braintrust. Factorial used it for tracing and evals on their Mastra agents, and our AI engineering guide calls it a good pick for tracing and evals. That works because the two products mostly do different jobs. Mastra provides a TypeScript runtime for agents, workflows, tools, and memory, with evaluation and observability alongside them. Braintrust provides evaluation and observability across applications, including applications built with Mastra.
If your runtime is already settled, Braintrust can add a shared quality layer. If you still need to implement the agent application, Mastra supplies the execution primitives too. You can use both when those responsibilities belong to different teams.
TL;DR
- Choose Braintrust when your runtime is already settled and the missing layer is quality across it. It traces applications and runs evals across frameworks and languages.
- Choose Mastra when you are still building the application layer. Agents and workflows run in one TypeScript runtime, and memory and retrieval are application infrastructure, not only things to measure after the fact.
- Mastra provides workflow execution and recovery options. Suspend/resume uses snapshots in configured storage; crash recovery depends on the engine and recovery settings.
- Braintrust centralizes evals across runtimes. Mastra keeps datasets and experiments attached to the agents, workflows, and scorers in your application.
- The two can run together. Braintrust can sit above a Mastra application when an organization wants one cross-framework evaluation layer.
Mastra vs Braintrust: At a glance
Braintrust positions itself as an observability platform for agents, helping developers understand and improve how their AI applications behave. Mastra focuses on building and running those applications, providing the framework for developing AI agents.
| Area | Mastra | Braintrust |
|---|---|---|
| Primary role | Build and run agentic applications | Evaluate and observe AI applications across runtimes |
| Agents | First-class agent runtime with tools, memory, workflows, scorers, and server APIs | Can trace existing agents and host prompts, tools, scorers, and preview workflows |
| Orchestration | Independent workflows with typed steps, control flow, suspension, and persisted execution state | Can compose hosted Functions while letting another runtime own broader orchestration |
| Evaluations | Datasets, scorers, experiments, and targets tied directly to Mastra agents and workflows | Dedicated cross-framework evaluation system |
| Observability | Framework-aware traces, logs, metrics, feedback, and scores | Framework-neutral tracing across multiple runtimes |
| Memory and retrieval | Memory and RAG are runtime primitives | Traces and evaluates memory or retrieval systems that may live elsewhere |
| Development surface | Studio operates on agents, tools, workflows, traces, datasets, and experiments | UI centers on logs, datasets, prompts, scorers, experiments, and production analysis |
| Best fit | TypeScript teams choosing how the agent application itself should run | Teams that already have one or more runtimes and need a common quality layer |
Both Mastra and Braintrust support evaluations. The difference is that Mastra also lets you build and run the agents and workflows you're evaluating. Braintrust focuses on evaluating and monitoring AI applications, even when they're built with other frameworks.
1. Agent runtime
Mastra treats the agent as an application primitive. An agent can call typed tools, use memory, participate in workflows, and expose capabilities through MCP. The same project can then expose those primitives through Mastra Server.
Braintrust has moved closer to execution than a traditional eval platform. Its Functions system can host prompts, tools, and scorers for invocation through the API, SDK, or UI. It also has workflows in public preview, currently limited to playgrounds rather than SDK agent deployment. The rest of the application can continue running in another framework.
| Runtime question | Mastra | Braintrust |
|---|---|---|
| Where is the agent defined? | In the Mastra application | Can remain in another framework |
| Who owns tool execution? | Mastra tools and agent runtime | Braintrust Functions can host tools, or the existing runtime can |
| Who owns application APIs? | Mastra Server or a Mastra server adapter | Usually the existing application runtime |
| Can it work with another framework? | Mastra is itself the framework | Yes, this is a core use case |
Verdict: Choose Mastra when you need to implement the runtime. Choose Braintrust when execution already lives elsewhere and you need evaluation, tracing, or prompt iteration across those applications.
2. Workflows and durability
Long-running AI work may need to finish several steps, stop for an approval or external event, then continue later. Mastra saves workflow snapshots to configured storage; use persistent storage to retain them across process restarts. Recovery behavior depends on the execution engine and persistence settings. For example, Temporal-backed workflows delegate execution, retries, and state persistence to Temporal.
On resume, the suspended step's execute function runs again with resumeData. It can satisfy the approval condition, finish that step, and then advance to the next step. Earlier completed steps retain their saved outputs. Code before the suspension can run again, so external side effects need idempotency or deduplication.
Mastra's Durable Agents (beta) run the agent loop inside a workflow, so a client can reconnect to a live stream and a crashed run can pick up from its last checkpoint. Crash recovery is opt-in and needs persistent storage. Recovery can replay model and tool calls, so make external actions idempotent.
Braintrust can trace and score workflows implemented in another system, and its Functions can compose multi-step AI logic. But if your application needs persisted suspend-and-resume semantics, another runtime still has to own that lifecycle.
Verdict: Mastra provides the workflow APIs and configurable recovery behavior. Braintrust can measure that execution while the application framework or workflow engine manages its lifecycle.
3. Evaluations
Braintrust organizes evaluation around datasets, experiments, scorers, and production traces. Teams can compare prompt, model, and code changes while keeping the eval system independent of any one application framework.
Mastra puts that loop directly inside the framework. Datasets provide versioned test cases. Scorers measure application-specific quality. Experiments run those cases against a target and preserve results for comparison. Experiment targets include agents, workflows, and scorers. A scorer target lets you test the judge itself against human labels, as in this LLM-as-a-judge calibration guide.
The difference is not whether both products can run evals. It is where evals attach.
| Evaluation model | Mastra | Braintrust |
|---|---|---|
| Primary target | Mastra runtime primitives | Any supported application or model workflow |
| Cross-framework use | Possible through integrations and exported telemetry | Core design goal |
| Workflow evaluation | First-class workflow targets | Evaluate traces or outputs from workflows in other runtimes |
| Best reason to choose it | Keep tests attached to the objects you deploy | Standardize evals across different stacks |
Verdict: Choose Braintrust to standardize evaluation across different stacks. Use Mastra's datasets and experiments when you want to test the registered agents, workflows, and scorers in your Mastra application.
4. Observability
Braintrust is designed to ingest telemetry from systems it does not control. That is valuable when a company runs several AI frameworks and wants one place to inspect traces, attach scores, search production behavior, and compare experiments.
Mastra's tracing system represents agent generations, tool calls, workflow steps, and memory operations as spans. Mastra Observability stores logs, traces, metrics, scores, and feedback. Studio puts those traces alongside the agents, tools, workflows, and experiments in the same project.
Braintrust can receive Mastra's telemetry too. Its Mastra integration captures agent and workflow runs, workflow steps, tools, and model calls, including inputs, outputs, token metrics, and entity metadata. Mastra also provides an exporter; its implementation preserves the original span type and metadata.
| Observability concern | Mastra | Braintrust |
|---|---|---|
| Trace sources | Instruments operations in the Mastra runtime | Receives Mastra spans and telemetry from other supported runtimes |
| Development surface | Studio combines traces with the project's runtime objects | Braintrust combines logs, datasets, and experiments across integrations |
| Production scoring | Integrated with Mastra scorers and experiments | Core Braintrust workflow |
| Portability | Can export telemetry outward | Designed to accept telemetry from many runtimes |
The decision is where your team wants to inspect and evaluate telemetry. Mastra Studio keeps that work beside the application configuration. Braintrust gives you a shared destination for applications using different frameworks; connecting spans across services still requires appropriate instrumentation and trace propagation.
Verdict: Use Studio for iteration within a Mastra project, Braintrust for a shared evaluation and observability layer, or both through the integration.
5. Memory and retrieval
Memory and retrieval make the ownership split especially clear. Mastra provides both as application infrastructure, not only as things to measure after the fact.
Mastra Memory can maintain conversation history, working memory, semantic recall, multi-user threads, and Observational Memory. Retrieval uses the same framework approach. The RAG pipeline covers document chunking and embedding, while the vector database interface gives applications a consistent abstraction over supported stores.
Braintrust does not need to replace those systems to add value. It can trace retrieval calls, score retrieved context, evaluate generated answers, and compare changes to the retrieval pipeline over time.
| Capability | Mastra | Braintrust |
|---|---|---|
| Conversation memory | Runtime primitive | Can evaluate or trace an external memory system |
| Long-context strategy | Observational Memory plus other memory modes | Measures the behavior of whichever memory layer you use |
| RAG implementation | First-party chunking, embeddings, and vector-store abstractions | Can observe and score RAG implemented elsewhere |
| Main ownership | Build the subsystem | Measure the subsystem |
Verdict: Mastra supplies memory and retrieval components for the application. Braintrust can evaluate those components, including systems implemented outside Mastra.
6. Agent editing and iteration
Shipping an agent once is easy compared with changing it safely over time. Instructions, tools, models, and policies move independently, so teams need a way to test edits before they affect production behavior.
Mastra keeps that work close to the runtime. The Agent Editor supports draft and published versions so instructions and tools can change without immediately replacing the live configuration. The Mastra Platform connects that editing surface with Studio, datasets, traces, metrics, and Server.
A typical Mastra iteration loop looks like this:
- Edit agent instructions or tools.
- Run the draft in Studio.
- Execute a dataset against the new version.
- Compare experiment scores and traces.
- Publish only after the change clears the team's threshold.
Braintrust approaches iteration from the quality layer. Prompts, datasets, scorers, traces, and experiments can remain centralized even when the underlying applications use different frameworks. That makes Braintrust attractive to platform teams that want one evaluation standard without mandating one runtime.
Verdict: Mastra keeps editing, execution, and testing inside one application model. Braintrust keeps prompt and evaluation workflows independent of the framework underneath.
7. Deployment
Braintrust can be introduced without moving the application. Its self-hosting architecture lets teams operate the data plane in their own cloud account. The UI, authentication, and management control plane remain Braintrust-managed SaaS.
Mastra can own the application server itself. The Mastra Platform overview separates Studio, Server, and managed services. Our deployment models cover managed, mixed, and self-hosted setups, and server adapters let Mastra run inside existing Express, Hono, Fastify, or Koa applications.
| Deployment question | Mastra | Braintrust |
|---|---|---|
| Does it need to become the application runtime? | It can, or it can integrate into an existing server | No |
| Can it run in your infrastructure? | Yes | Yes, including self-hosted data plane options |
| What gets deployed? | Agents, workflows, tools, APIs, Studio, observability components | Evaluation and observability infrastructure plus optional hosted Functions |
| Migration cost | Higher if replacing an existing runtime, lower for a new app | Usually lower because the runtime can stay in place |
For a new TypeScript application, Mastra can remove several separate infrastructure decisions. For an established platform with several reliable runtimes, Braintrust can add a common quality layer without forcing those services to converge.
Verdict: Braintrust is easier to add around an existing architecture. Mastra is more compelling when the runtime and its operational surface are still being designed.
How to pick between Mastra and Braintrust
Choose Braintrust when your runtime is already settled and the missing capability is quality across that runtime. This is especially strong for companies with multiple frameworks or languages that want one system for traces, datasets, scorers, experiments, and prompt iteration.
Choose Mastra when you are still building the application layer itself. Agents, tools, workflows, memory, retrieval, experiments, tracing, and deployment share one runtime model. The Mastra guides reflect that architecture: the framework is organized around building and operating the application, not only evaluating its outputs.
| Your situation | Better fit |
|---|---|
| Existing AI applications across several frameworks | Braintrust |
| Central evaluation and observability standard | Braintrust |
| Add quality tooling without changing the runtime | Braintrust |
| New TypeScript agent application | Mastra |
| Durable workflows and suspend/resume | Mastra |
| First-class memory and retrieval | Mastra |
| Evaluate agents and workflows as runtime objects | Mastra |
| One framework for runtime, Studio, observability, and deployment | Mastra |
Braintrust and Mastra are not mutually exclusive. Braintrust can sit above a Mastra application if an organization wants one cross-framework evaluation layer. But if the team is starting from an empty repository and still needs to decide how agents, workflows, memory, tools, and APIs should run, I would start with Mastra because it solves the larger part of the application architecture.
Getting started with Mastra
This path assumes you have not installed Mastra and do not already have a project.
Prerequisites
Before creating the project, install or obtain:
- Node.js 22.13.0 or later. Mastra requires Node 22.13.0+.
- npm. It is included with Node.js and is used below to scaffold and run the project.
- A model-provider API key. The example uses OpenAI, so you need an
OPENAI_API_KEY. Mastra also supports Anthropic, Google, xAI, and many other providers through the Mastra model router. - A terminal and code editor. Git is useful but not required for the local quickstart.
You can confirm Node and npm are available with:
node --version
npm --version1. Create a new Mastra project
Create a project and choose OpenAI as the model provider:
npm create mastra@latest my-mastra-app -- --llm openaiThe scaffold installs the Mastra dependencies and creates the project structure for you. Then enter the project directory:
cd my-mastra-app2. Add your model-provider credential
Create or open the project's .env file and add your OpenAI key:
OPENAI_API_KEY=your-openai-api-keyMastra models use provider/model identifiers and automatically read the corresponding provider environment variable. If you choose another provider during setup, use that provider's required environment variable instead.
3. Start Mastra and open Studio
Run the development server:
npm run devThen open:
http://localhost:4111
Mastra Studio lets you run the generated agent, inspect tool calls and traces, and work with workflows without first building a separate frontend.
4. Run the first agent before adding infrastructure
Open the generated agent in Studio and send a simple prompt. Confirm that the model responds and that the run appears in the local trace view.
Once that basic path works, add one capability at a time:
- Add a tool if the agent needs to call application code or an external API.
- Add memory if the state must survive across turns.
- Add a workflow when execution needs deterministic steps, branching, or suspend/resume.
- Add a dataset and scorer before making repeated prompt or model changes.
That order gives you a working baseline before storage, retrieval, workflows, and eval infrastructure make failures harder to isolate.
