Query traces and spans

Query completed traces or spans with structured filters, or aggregate them into counts, error rates, latency percentiles, tokens, and cost. Each operation is available through the client SDK, HTTP API, and CLI:

Operation	Use it to	Client SDK	HTTP endpoint	CLI command
Query traces	List traces matching root-span fields or conditions on related spans.	queryTraces()	POST /api/observability/traces/query	mastra api trace query
Query spans	List individual spans across traces, including nested agent runs and tool calls.	querySpans()	POST /api/observability/spans/query	mastra api span query
Aggregate traces	Compute measures over matching traces, optionally grouped or bucketed by time.	aggregateTraces()	POST /api/observability/traces/aggregate	mastra api trace aggregate
Aggregate spans	Compute measures over individual matching spans at any depth.	aggregateSpans()	POST /api/observability/spans/aggregate	mastra api span aggregate

The trace operations share a trace selection: root-span predicates and spans.some / spans.none clauses. The span operations share a direct span predicate, without nested spans clauses. Reuse a selection between the query and aggregate operations for the same record type.

Option	Query traces	Query spans	Aggregate traces or spans
Maximum time-range duration	No maximum	No maximum	365 days
Pagination	Keyset, numbered pages, or delta polling	Keyset only	No pagination; limit selects groups
orderBy shape	One-item array	One-item array	One object

CLI commands take filters and request options together as one JSON argument and wrap the response in data. To inspect a query result, use its traceId to retrieve the full trace, or call getSpan(traceId, spanId) to retrieve one span.

The endpoints use the same authentication as other observability routes and require the observability:read permission. Every request requires a time range.

These API time-range limits are separate from data retention. For hosted traces, your Platform plan's retention period applies. See Query observability data on Mastra Platform for retention, project credentials, and regional endpoints.

Check capabilities

The configured observability store must support each API. Call getObservabilityCapabilities() or GET /api/observability/capabilities and check these flags before showing a feature:

Capability	Store feature	Enables
traceQuery	trace-query	queryTraces()
spanQuery	span-query	querySpans()
spanQueryContextFields	span-query-context-fields	Additional identity, context, and parent/root entity predicates in span queries and aggregates
spanQueryModelFields	span-query-model-fields	model and provider predicates in span queries and aggregates
traceAggregate	trace-aggregate	aggregateTraces()
spanAggregate	span-aggregate	aggregateSpans()
traceQueryRootDuration	trace-query-root-duration	Trace-scope durationMs predicates
traceQueryTimeToFirstChunk	trace-query-time-to-first-chunk	timeToFirstChunkMs predicates and results
traceQueryTimeToFirstChunkSort	trace-query-time-to-first-chunk-sort	orderBy on timeToFirstChunkMs
traceQueryTimeToFirstToken	trace-query-time-to-first-token	timeToFirstTokenMs predicates and results
traceQueryTimeToFirstTokenSort	trace-query-time-to-first-token-sort	orderBy on timeToFirstTokenMs
aggregateTimeToFirstMeasures	aggregate-time-to-first-measures	timeToFirstToken.* and timeToFirstChunk.* measures in aggregateTraces() and aggregateSpans()
traceQueryContextIds	trace-query-context-ids	runId, sessionId, userId, and organizationId predicates
traceQueryContextFields	trace-query-context-fields	requestId, serviceName, and experimentId predicates
traceQueryRootSpanFields	trace-query-root-span-fields	Trace-scope name and spanType predicates, dimensions, and countDistinct measures
traceQueryRootEntityId	trace-query-root-entity-id	Trace-scope entityId predicates, dimensions, and countDistinct measures
traceQueryOutcomeFields	trace-query-outcome-fields	outcome and stopReason in trace and spans.some / spans.none predicates, trace dimensions, and value discovery, plus the root span's outcome fields on trace results
spanQueryOutcomeFields	span-query-outcome-fields	outcome and stopReason in querySpans() and aggregateSpans() predicates, dimensions, and countDistinct measures, plus the outcome fields on span results
traceQueryDiscovery	trace-query-discovery	Field and value discovery
traceQueryTenantScope	trace-query-tenant-scope	Trusted tenant scope

A missing traceAggregate value means the server doesn't support aggregation. Servers without the aggregate route return 404, and stores without the feature return 501 TRACE_AGGREGATE_UNSUPPORTED. A missing spanAggregate value means the same for span aggregation, and stores without the feature return 501 SPAN_AGGREGATE_UNSUPPORTED. Against a store without a sub-feature, a request that uses a gated field returns 501 before storage runs. Unsupported root span name, spanType, and entityId, and span predicates requiring spanQueryContextFields, return 422 with a field_not_allowed issue instead. A store without spanQuery returns 501 SPAN_QUERY_UNSUPPORTED.

Select traces

Every request selects traces with two fields:

Field	Required	Description
timeRange	Yes	Trace start-time boundary. from is inclusive and to is exclusive. Both values must be ISO timestamps, and from must be earlier than to.
where	No	Recursive trace predicate. Supports scalar conditions and spans.some or spans.none clauses.

This selection matches traces from the first week of August that contain a tool call with an error:

const selection = {
  timeRange: {
    from: '2026-08-01T00:00:00.000Z',
    to: '2026-08-08T00:00:00.000Z',
  },
  where: {
    spans: {
      some: {
        op: 'and',
        args: [
          { op: 'eq', left: { path: 'spanType' }, right: { literal: 'tool_call' } },
          { op: 'exists', path: 'error' },
        ],
      },
    },
  },
}

The same selection returns the same traces from either API. Aggregation counts exactly the traces that a query would list.

A spans.some clause matches when one span satisfies all its conditions. In this example, the same span must be a tool call and have an error. Use spans.none to find traces with no spans matching the conditions.

Span clauses examine the current root span and current child spans. The root timeRange applies only to the selected current root's startedAt. Related spans can participate even when their own timestamps are outside that range. Related records correlate only through a matching non-null traceId.

Fields and operators

Predicate context	Fields	Operators
Trace	traceId, threadId, resourceId, runId, sessionId, userId, organizationId, requestId, serviceName, experimentId, entityId, entityName, entityType, name, spanType, environment, status, outcome, stopReason	eq, ne, in, notIn, exists, notExists
Trace metadata	metadata.<key>	eq, ne, in, notIn, exists, notExists
Trace	startedAt, endedAt, durationMs, timeToFirstChunkMs, timeToFirstTokenMs	eq, ne, in, notIn, lt, lte, gt, gte, exists, notExists
Trace	tags	includes, notIncludes, exists, notExists
Span	name	eq, ne, in, notIn, exists, notExists, matches, notMatches
Span	spanType, model, provider, status, outcome, stopReason, entityType, entityId, entityName, entityVersionId, parentEntityVersionId, rootEntityVersionId, runId, sessionId, userId, organizationId, requestId, serviceName, experimentId	eq, ne, in, notIn, exists, notExists
Span	startedAt, endedAt, durationMs, timeToFirstChunkMs, timeToFirstTokenMs	eq, ne, in, notIn, lt, lte, gt, gte, exists, notExists
Span	error	exists, notExists

Trace-scope name, spanType, and entityId read the current root span, so { op: 'eq', left: { path: 'spanType' }, right: { literal: 'workflow_run' } } selects traces started by a workflow run. Inside spans.some or spans.none, the same fields match any span in the trace.

Compose predicates with { op: 'and', args: [...] }, { op: 'or', args: [...] }, and { op: 'not', arg: ... }. Comparison predicates place a field reference on the left and a literal on the right. Membership predicates use a field reference in value and a homogeneous literal array in set. Tag predicates name the stored tag collection in path. includes and notIncludes also take one tag in value, while exists and notExists take only op and path. See Filter by tags. Text predicates use the comparison shape with a string literal. See Filter by text.

String comparisons are case-sensitive, and literals are never coerced. matches and notMatches are the exception: they compare words case-insensitively. Canonical string fields compare exact stored values. Metadata string values are trimmed before comparison, as described below. Duration, time-to-first-chunk, and time-to-first-token predicates require numbers, while timestamp predicates require ISO timestamp strings. A missing value satisfies neither positive nor ordered predicates, although it does satisfy the negative operators ne and notIn. Combine a negative predicate with exists when the field must also be present.

Filter by root duration

At trace scope, durationMs is the current completed root span's elapsed time in milliseconds. The value is derived from endedAt - startedAt when the query runs:

const slowRootTraces = {
  op: 'gt',
  left: { path: 'durationMs' },
  right: { literal: 5000 },
}

This top-level predicate doesn't inspect child spans. In contrast, spans.some with a durationMs predicate examines the current root span and current child spans, so a long child can satisfy that clause even when its root is shorter.

Trace queries exclude incomplete roots before evaluating predicates. As a result, { op: 'notExists', path: 'durationMs' } doesn't find running traces or malformed roots without a usable endedAt.

Filter by time to first chunk

timeToFirstChunkMs is the time in milliseconds from a span's start to the first chunk of its model response. Agent run and model spans record it when the model streams a response. At trace scope, the field reads the current root span's value. Inside spans.some or spans.none, it reads each related span.

This predicate finds traces with a model call that took more than 2 seconds to stream its first chunk:

const slowFirstChunk = {
  spans: {
    some: {
      op: 'and',
      args: [
        { op: 'eq', left: { path: 'spanType' }, right: { literal: 'model_generation' } },
        { op: 'gt', left: { path: 'timeToFirstChunkMs' }, right: { literal: 2000 } },
      ],
    },
  },
}

The value is null when the span didn't stream a chunk, for example a tool call or a workflow run. A null value counts as missing, so it matches notExists and never matches ordered predicates. A value of 0 isn't missing. It means the first chunk arrived in the same millisecond the span started.

Trace query results include the root span's timeToFirstChunkMs, and querySpans() results include each span's value. Stores advertise support through the trace-query-time-to-first-chunk feature, and the server reports it as the traceQueryTimeToFirstChunk capability. Against an older store, results omit the field, a predicate on it returns 501 before storage runs, and field discovery omits it.

Sort by time to first chunk

Pass orderBy: [{ field: 'timeToFirstChunkMs', direction: 'desc' }] to list the slowest first chunks first. Traces sort by the root span's value. Missing values sort last in both directions, and 0 sorts as a value. Ties break by traceId for traces and by span identity for querySpans().

Stores advertise the sort through the trace-query-time-to-first-chunk-sort feature, and the server reports it as the traceQueryTimeToFirstChunkSort capability. Against an older store, the sort returns 501 before storage runs.

Filter and sort by time to first token

timeToFirstTokenMs is the time in milliseconds from the start of a model call, the provider request itself, to the first chunk that call streams. It leaves out the work Mastra does before calling the model, such as input processors. The value is measured once, on the model inference span that streamed the chunk. Model step, model generation, and agent run spans carry the value of the nearest such inference below them: each step takes its own, and the generation and agent run take the first step's. A step can run more than one inference, for example a fallback model or a retry that opens a new inference span. The value then comes from the first inference that streamed a chunk. A fallback attempt that failed ends its inference span with its error. That span has a value only if it streamed a chunk first. Mastra storage keeps the value in its own column and leaves it out of the stored attributes. Read it from the span's timeToFirstTokenMs field. When an in-process agent retries a failed model call, the AI SDK retries inside a single inference span. The value then includes the failed attempt and the retry wait. Durable agents open one inference span per attempt and don't. At trace scope, the field reads the current root span's value. Inside spans.some or spans.none, it reads each related span.

This predicate finds traces whose root agent waited more than 2 seconds for the model's first token:

const slowFirstToken = { op: 'gt', left: { path: 'timeToFirstTokenMs' }, right: { literal: 2000 } }

null, 0, and missing values follow the same rules as timeToFirstChunkMs. Pass orderBy: [{ field: 'timeToFirstTokenMs', direction: 'desc' }] to list the slowest first tokens first, with the same ordering and tie-breaks.

Stores advertise support through the trace-query-time-to-first-token feature and the sort through the trace-query-time-to-first-token-sort feature. The server reports them as the traceQueryTimeToFirstToken and traceQueryTimeToFirstTokenSort capabilities. Against an older store, results omit the field, and a predicate or sort on it returns 501 before storage runs.

Filter by outcome

outcome records how a span ended, for example success, error, abort, limit, or unknown. stopReason records why, for example max_steps. Both are free strings, so custom spans can record their own values. They're separate from status, which still reads only whether the span recorded an error.

At trace scope, both fields read the current root span, the span with no parent that started the trace. Inside spans.some or spans.none, they read each related span. A trace whose root succeeded but whose tool call hit a step limit doesn't match a trace-scope outcome of limit. Find it through spans.some:

const hitALimit = {
  spans: {
    some: { op: 'eq', left: { path: 'outcome' }, right: { literal: 'limit' } },
  },
}

Spans written by an SDK that predates these fields, or a root span that's still running, have no outcome. A missing outcome means "not recorded", never success. Select those spans with { op: 'notExists', path: 'outcome' }. As with other optional string fields, ne and notIn match spans with no value.

Both fields support eq, ne, in, notIn, exists, and notExists, group trace and span aggregates, and return values from value discovery. They can't sort individual trace or span query results. In aggregates, either field can be used in orderBy when it's included in groupBy. Results also return stopReasonContext, an object with details about the stop reason such as { maxSteps: 5 }. stopReasonContext is returned only and can't be filtered, grouped, or sorted.

Stores advertise support through the trace-query-outcome-fields feature for trace queries and aggregates, and the span-query-outcome-fields feature for span queries and aggregates. The server reports them as the traceQueryOutcomeFields and spanQueryOutcomeFields capabilities. Against an older store, results omit the fields, and a request that uses them returns 501 before storage runs.

Filter by context identifiers

runId, sessionId, userId, and organizationId are available at trace scope and inside spans.some / spans.none. Trace-scope predicates read the current root span. Span-scope predicates bind to one related current span, so a run recorded only on a child span is found through spans.some:

const sessionTracesWithRun = {
  op: 'and',
  args: [
    { op: 'eq', left: { path: 'organizationId' }, right: { literal: 'org-123' } },
    { op: 'eq', left: { path: 'sessionId' }, right: { literal: 'session-123' } },
    {
      spans: {
        some: { op: 'eq', left: { path: 'runId' }, right: { literal: 'run-42' } },
      },
    },
  ],
}

These identifiers support eq, ne, in, notIn, exists, and notExists. Field discovery lists them, but value discovery doesn't return their recorded values. Stores advertise support through the trace-query-context-ids feature. A predicate on one of these fields against an older store returns 501 before storage runs, and field discovery omits them for that store.

requestId, serviceName, and experimentId work the same way, at trace scope and inside spans.some / spans.none. This predicate finds checkout requests that reached the payments service on a child span. The child span records metadata: { serviceName: 'payments' }; the root keeps checkout:

const checkoutRequestsThroughPayments = {
  op: 'and',
  args: [
    { op: 'eq', left: { path: 'serviceName' }, right: { literal: 'checkout' } },
    { op: 'exists', path: 'requestId' },
    {
      spans: {
        some: { op: 'eq', left: { path: 'serviceName' }, right: { literal: 'payments' } },
      },
    },
  ],
}

Stores advertise support for these three fields through the trace-query-context-fields feature, reported as the traceQueryContextFields capability. Against an older store, a predicate on one of them returns 501 before storage runs, and field discovery omits them.

Filter by span properties

Every condition inside one spans.some or spans.none clause applies to the same current span. For example, this predicate finds a failed tool span whose name is medication_lookup; a matching name on one span and an error on another don't satisfy it:

const failedMedicationLookup = {
  spans: {
    some: {
      op: 'and',
      args: [
        { op: 'eq', left: { path: 'name' }, right: { literal: 'medication_lookup' } },
        { op: 'eq', left: { path: 'spanType' }, right: { literal: 'tool_call' } },
        { op: 'exists', path: 'error' },
      ],
    },
  },
}

Use status for the derived success or error outcome. Use error when you only need to test whether error details are present.

durationMs is the span's elapsed time in milliseconds. Span startedAt and endedAt conditions filter related spans independently of the top-level root timeRange:

const slowModelCalls = {
  spans: {
    some: {
      op: 'and',
      args: [
        { op: 'eq', left: { path: 'spanType' }, right: { literal: 'model_generation' } },
        { op: 'gt', left: { path: 'durationMs' }, right: { literal: 5000 } },
        {
          op: 'gte',
          left: { path: 'startedAt' },
          right: { literal: '2026-07-15T00:00:00.000Z' },
        },
      ],
    },
  },
}

model and provider read the span's model and provider columns. When a column is empty, they fall back to the string attributes.model and attributes.provider values that older @mastra/observability versions exported, so existing filters keep matching older spans. Trace results omit model and provider, because a trace can match through one span while its other spans used a different model.

const selectedModel = {
  spans: {
    some: {
      op: 'and',
      args: [
        {
          op: 'eq',
          left: { path: 'model' },
          right: { literal: 'claude-sonnet-5-5' },
        },
        { op: 'eq', left: { path: 'provider' }, right: { literal: 'anthropic' } },
      ],
    },
  },
}

Use generic entity fields for tool and retrieval identity. The version fields let one clause qualify the same span by its entity lineage:

const versionedRetrieval = {
  spans: {
    some: {
      op: 'and',
      args: [
        { op: 'eq', left: { path: 'entityType' }, right: { literal: 'rag_ingestion' } },
        { op: 'eq', left: { path: 'entityId' }, right: { literal: 'medication-index' } },
        { op: 'eq', left: { path: 'entityVersionId' }, right: { literal: 'index-v3' } },
        { op: 'eq', left: { path: 'rootEntityVersionId' }, right: { literal: 'agent-v2' } },
      ],
    },
  },
}

Filter by metadata

Use metadata.<key> paths for custom dimensions stored on the current root span. Metadata predicates compare trimmed, non-empty string values exactly and case-sensitively. Leading and trailing whitespace in a stored string value is ignored. Values empty after trimming count as missing. Null, numeric, boolean, object, and array values also count as missing.

const messageTrace = {
  op: 'and',
  args: [
    { op: 'eq', left: { path: 'metadata.messageId' }, right: { literal: 'message-123' } },
    { op: 'in', value: { path: 'metadata.actorRole' }, set: ['assistant', 'tool'] },
    { op: 'exists', path: 'metadata.protocolVersion' },
    {
      op: 'or',
      args: [
        { op: 'eq', left: { path: 'metadata.temporalRunId' }, right: { literal: 'run-456' } },
        { op: 'eq', left: { path: 'metadata.externalTraceId' }, right: { literal: 'trace-789' } },
      ],
    },
    { op: 'not', arg: { op: 'exists', path: 'metadata.parentMessageId' } },
  ],
}

The key must name one top-level property. Empty keys and nested paths are rejected. Metadata keys aren't trimmed, so leading and trailing whitespace remains part of the exact key identity. When metadata duplicates a canonical trace field, such as resourceId, threadId, or environment, prefer the canonical field because it uses the dedicated storage column. The metadata.<key> form remains available when you specifically need the value from the metadata object. Metadata and canonical values may differ.

Trace-query discovery returns executable top-level string metadata fields observed in the selected time range.

Filter by tags

Tags are a list of strings stored on the current root span, so they use collection operators instead of the scalar in and notIn operators. This query finds production traces tagged for manual review that don't carry the archived tag:

const manualReview = {
  op: 'and',
  args: [
    { op: 'eq', left: { path: 'environment' }, right: { literal: 'production' } },
    { op: 'includes', path: 'tags', value: 'manual-review' },
    { op: 'notIncludes', path: 'tags', value: 'archived' },
  ],
}

Operator	Matches when
includes	The trace has the requested tag
notIncludes	The trace has at least one tag, but not the requested tag
exists	The trace has at least one tag
notExists	The trace has no tags

Tags compare exactly and case-sensitively as whole strings. value must be one string with at least one non-whitespace character. Combine several includes predicates with and or or to require or allow multiple tags. A trace with no recorded tags and a trace with an empty tag list behave the same: both satisfy notExists, neither satisfies includes, notIncludes, or exists. Use { op: 'not', arg: { op: 'includes', ... } } when untagged traces should also match.

tags is available in trace predicates only. in, notIn, and the comparison operators are rejected for tags. Value discovery for tags returns each observed tag with the number of qualifying traces that carry it.

Filter by text

Span names support word matching in addition to exact comparison. matches succeeds when the literal's words appear in the field, in order, as whole words. notMatches succeeds when the field exists but doesn't contain those words. This query finds traces with a span whose name contains the words "document search":

const documentSearch = {
  spans: {
    some: {
      op: 'matches',
      left: { path: 'name' },
      right: { literal: 'document search' },
    },
  },
}

Both the literal and the stored value are lowercased and split into words. A word is a run of letters, digits, and combining marks, so any other character, such as punctuation, spaces, or _, separates words. incorrect matches This answer is incorrect. but not This was incorrectly formatted., and gpt 5 matches llm: 'gpt-5'. A literal with several words must appear as a contiguous phrase in that order. Letters keep their accents: café doesn't match cafe. Text is composed to Unicode NFC first, so café matches whether the accent is stored as part of the letter or as a separate mark, and marks stay inside their word, so नमस doesn't match नमस्ते. The Turkish İ folds to i, and a Greek final ς equals σ. PostgreSQL lowercases letters according to the database locale, so a database created with the C locale doesn't fold the case of letters outside ASCII. Use a UTF-8 locale such as en_US.UTF-8 for consistent results.

Operator	Matches when
matches	The field contains the literal's words as a contiguous whole-word phrase
notMatches	The field is present and doesn't contain that phrase

The literal must contain at least one letter or digit; otherwise the request fails with 422 TRACE_QUERY_INVALID. A missing field satisfies neither operator. Use { op: 'not', arg: { op: 'matches', ... } } when records without the field should also match. Text operators are available on span name. Other string fields, numbers, timestamps, tags, and metadata reject them. Substring, wildcard, regular-expression, and fuzzy matching aren't supported, and there is no search across several fields.

Query limits

The planner rejects a query before storage execution when it exceeds any of these limits:

Input	Maximum
Predicate nesting depth	12 levels
Predicate nodes	100
Related spans clauses	8 total
Values in one in or notIn set	100
Comparison literals and membership values	1,000 total
String literal	4,096 UTF-8 bytes
Raw predicate path	128 UTF-8 bytes
page.limit	1,000
HTTP request body	256 KiB

Each comparison literal counts as one literal. Each member of an in or notIn set also counts as one literal, even when the set is within its per-set limit.

Hono and Fastify enforce the request-body limit before JSON parsing. Express and Elysia reject an oversized body before the route handler or storage runs, but the host framework may already have parsed a request without a reliable Content-Length header. Configure the host application's JSON parser or body limit when you need a hard pre-parse memory ceiling.

Query traces

Use queryTraces(), mastra api trace query, or POST /api/observability/traces/query to list the selected traces. All three accept the same request, which adds ordering and pagination to the trace selection.

Query traces with the client SDK

This query finds traces containing a tool call with an error:

import { MastraClient } from '@mastra/client-js'

const mastraClient = new MastraClient({ baseUrl: 'http://localhost:4111' })

const result = await mastraClient.queryTraces({
  timeRange: {
    from: '2026-08-01T00:00:00.000Z',
    to: '2026-08-08T00:00:00.000Z',
  },
  where: {
    spans: {
      some: {
        op: 'and',
        args: [
          { op: 'eq', left: { path: 'spanType' }, right: { literal: 'tool_call' } },
          { op: 'exists', path: 'error' },
        ],
      },
    },
  },
  orderBy: [{ field: 'startedAt', direction: 'desc' }],
  page: { limit: 25 },
})

Query traces with the CLI

Pass the query as one JSON argument to mastra api trace query. Filters and pagination options belong in the JSON input. This command runs the same failed-tool query as the SDK example:

mastra api --url http://localhost:4111 trace query '{
  "timeRange": {
    "from": "2026-08-01T00:00:00.000Z",
    "to": "2026-08-08T00:00:00.000Z"
  },
  "where": {
    "spans": {
      "some": {
        "op": "and",
        "args": [
          { "op": "eq", "left": { "path": "spanType" }, "right": { "literal": "tool_call" } },
          { "op": "exists", "path": "error" }
        ]
      }
    }
  },
  "orderBy": [{ "field": "startedAt", "direction": "desc" }],
  "page": { "limit": 25 }
}'

The explicit --url targets a local Mastra server. Without it, observability commands default to https://observability.mastra.ai, even when a local server is running. See CLI target resolution for authentication and regional endpoints. The target server and configured store must support the requested query capabilities.

Inspect the command with:

mastra api trace query --help

For a local Mastra server, inspect its request and response schemas with:

mastra api --url http://localhost:4111 trace query --schema

Mastra Platform doesn't currently expose the schema endpoint used by --schema. Use Request fields and Fields and operators to construct queries for either target. To find fields and values recorded in your project, use field and value discovery.

Send an HTTP request

Send the same query as a JSON request body:

curl --request POST \
  --url http://localhost:4111/api/observability/traces/query \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{
    "timeRange": {
      "from": "2026-08-01T00:00:00.000Z",
      "to": "2026-08-08T00:00:00.000Z"
    },
    "where": {
      "spans": {
        "some": {
          "op": "and",
          "args": [
            { "op": "eq", "left": { "path": "spanType" }, "right": { "literal": "tool_call" } },
            { "op": "exists", "path": "error" }
          ]
        }
      }
    },
    "orderBy": [{ "field": "startedAt", "direction": "desc" }],
    "page": { "limit": 25 }
  }'

Request fields

Field	Required	Description
timeRange	Yes	Trace start-time boundary. from is inclusive and to is exclusive. Both values must be ISO timestamps, and from must be earlier than to.
where	No	Recursive trace predicate. Supports scalar conditions and spans.some or spans.none clauses.
orderBy	No	One item ordering results by startedAt, endedAt, timeToFirstChunkMs, or timeToFirstTokenMs, in asc or desc order. Defaults to startedAt desc. Not accepted in delta mode.
page	No	{ limit, after }. limit defaults to 100 and has a maximum of 1000. Pass the opaque page.next value as after.
pagination	No	Numbered pages: { page, perPage }. Defaults to page 0 and 10 results. perPage has a maximum of 100.
mode	No	Set to 'delta' to poll using a delta cursor instead of page or pagination.
after	No	Delta mode only. Opaque deltaCursor from a numbered page or previous poll. Omit to establish the current watermark.
limit	No	Delta mode only. Maximum results per batch. Defaults to 10 and has a maximum of 100.

Unknown fields are rejected. Requests can't select a projection, declare joins, request counts or measures, or control authorization. Use aggregateTraces() for counts and measures. Request bodies are limited to 256 KiB.

Responses

A query using keyset pagination returns lightweight completed traces and a cursor for the next page:

{
  "traces": [
    {
      "traceId": "trace-123",
      "rootSpanId": "span-123",
      "name": "agent run: support-agent",
      "entityId": "support-agent",
      "parentSpanId": null,
      "createdAt": "2026-08-03T10:00:00.000Z",
      "metadata": null,
      "inputPreview": "Where is my order?",
      "outputPreview": "Your order arrives tomorrow.",
      "errorPreview": null,
      "threadId": "thread-123",
      "resourceId": "resource-123",
      "startedAt": "2026-08-03T10:00:00.000Z",
      "endedAt": "2026-08-03T10:00:01.000Z",
      "entityName": "support-agent",
      "entityType": "agent",
      "environment": "production",
      "status": "success",
      "timeToFirstChunkMs": 412,
      "timeToFirstTokenMs": 380,
      "outcome": "success",
      "stopReason": null,
      "stopReasonContext": null
    }
  ],
  "page": { "next": null }
}

The response includes name, createdAt, identity fields, metadata, and input/output/error previews. Supported nullable fields remain present with null when no value was recorded. Fields that require an optional store feature are omitted when the store lacks that feature. Previews summarize payloads; retrieve the full trace for complete input, output, and error details.

timeToFirstChunkMs and timeToFirstTokenMs are present only when the configured store supports them. See Filter by time to first chunk and Filter and sort by time to first token. outcome, stopReason, and stopReasonContext read the root span and are present only when the store supports the traceQueryOutcomeFields capability. They're null when the root span didn't record them. See Filter by outcome.

The response doesn't include the related spans that matched the predicate. Use getTrace() to load the full trace after selecting a result.

CLI responses

The CLI wraps the API response in data:

{
  "data": {
    "traces": [],
    "page": { "next": null }
  }
}

Read results from data.traces. Pagination metadata is available at data.page.next for keyset queries, data.pagination for numbered pages, and data.deltaCursor and data.delta.hasMore for delta polling.

Each CLI invocation makes one query. To fetch another keyset page, pass data.page.next as page.after in the next request. For delta polling, repeat the command with data.deltaCursor as the top-level after value. Keep the same time range and predicate. See Pagination and errors for mode-specific behavior and polling limitations.

To inspect a selected trace, use mastra api --url http://localhost:4111 trace get <traceId> --verbose. Use the same target and authentication as the query.

Pagination and errors

Choose a pagination mode

Trace queries support three mutually exclusive modes:

Mode	Request fields	Response metadata
Keyset, the default	page: { limit, after }	page: { next }
Numbered pages	pagination: { page, perPage }	pagination: { total, page, perPage, hasMore } and, when delta polling is supported, deltaCursor
Delta polling	mode: 'delta', optional top-level after and limit	delta: { limit, hasMore } and deltaCursor

Numbered pages are zero-based. Their total and rows are read consistently within each request. Don't combine page, pagination, or delta mode. Top-level after and limit are accepted only in delta mode.

Poll after loading a numbered page

Use the numbered response's deltaCursor to replace the numbered-page-to-delta-polling workflow of listTracesLight(). Results use traces instead of spans and contain completed traces only. The configured store must support delta polling.

const timeRange = {
  from: '2026-08-01T00:00:00.000Z',
  to: '2026-08-08T00:00:00.000Z',
}
const initial = await mastraClient.queryTraces({
  timeRange,
  pagination: { page: 0, perPage: 100 },
})
if (!initial.deltaCursor) throw new Error('Delta polling is unavailable')

const traces = new Map(initial.traces.map(trace => [trace.traceId, trace]))
let after = initial.deltaCursor

async function poll() {
  let hasMore: boolean
  do {
    const result = await mastraClient.queryTraces({
      timeRange,
      mode: 'delta',
      after,
      limit: 100,
    })
    for (const trace of result.traces) traces.set(trace.traceId, trace)
    after = result.deltaCursor
    hasMore = result.delta.hasMore
  } while (hasMore)
}

await poll()

Retain after and call poll() again at your application's polling interval. A trace can appear in the initial page and a later batch, or in multiple batches, so merge results by traceId. To load historical traces omitted from the initial page, request the remaining numbered pages.

A delta request without after returns an empty array and establishes the current cursor. It doesn't return historical matches. Always retain the returned cursor, including for empty batches. hasMore means another matching trace exists beyond the batch limit. Continue immediately while it's true.

Delta ordering uses the storage ingestion watermark and adapter-specific tie-breakers, so matching timestamps don't make the cursor ambiguous. Don't pass orderBy in delta mode. Keyset and delta cursors aren't interchangeable, and delta cursors can't move between storage adapters.

Keep the same normalized where predicate and exact timeRange bounds for every poll. The range continues to select root startedAt, even when a trace completes later. Changing timeRange.to to the current time invalidates the cursor. Reload a numbered page and use its cursor whenever the selection or authorization scope changes. You can change the batch limit without restarting.

New completed roots and roots completing after the cursor can be returned. Related spans are evaluated when a root is selected, but later related-record writes don't guarantee that the trace is emitted again. Deleted traces and traces that stop matching aren't returned as removals or tombstones. Refresh numbered pages when you need to reconcile those changes.

ClickHouse polling is best effort. Its delta index and trace records use separate materialized-view tables. Concurrent queries can observe an insert in one table before another, as described in ClickHouse's materialized-view visibility rules. A poll can advance past an index entry before the matching trace becomes visible and miss that trace in later polls. Reload numbered pages periodically to reconcile these gaps.

ClickHouse's delta index retains two days of events and doesn't backfill historical rows. Reload numbered pages after a polling interruption longer than this retention window. Delta polling isn't a durable change feed and doesn't guarantee delivery of every matching trace.

Keyset ordering and errors

Ordering is deterministic. Trace ordering appends traceId ascending as a tie-breaker.

Keyset cursors are bound to the operation, accepted normalized query, and ordering. Reusing a keyset cursor after changing the trace selection, predicates, or ordering returns 409.

Numbered-page handoff cursors and delta cursors also bind the authorization state. Changes to the caller's roles or permissions invalidate these cursors and return 409. If a delta poll returns 409, reload the numbered pages and resume polling with the new deltaCursor. A malformed cursor returns 400.

Cursor pagination is deterministic, but it isn't a database snapshot. Traces or replacement signals written between page requests can change later pages.

PostgreSQL and ClickHouse stop trace queries after 15 seconds by default and return 504 when the database timeout is exceeded. Set observability.traceQueryTimeoutMs on PostgresStoreVNext, or observability.traceQuery.timeoutMs on ClickhouseStoreVNext, to an integer from 1 through 300,000 milliseconds to change the timeout. DuckDB doesn't currently provide query-scoped timeout or cancellation through its driver wrapper, so this 504 guarantee doesn't apply to DuckDB.

Status	Meaning
400	Malformed JSON or malformed cursor.
409	The cursor doesn't match the query.
413	The request body exceeds 256 KiB.
422	The JSON is well formed, but the request is structurally or semantically invalid. The response includes stable issue codes and paths. Discovery validation uses TRACE_QUERY_INVALID.
501	The installed Core or configured observability store doesn't support the requested operation. Discovery returns TRACE_QUERY_DISCOVERY_UNSUPPORTED unless Core provides the discovery contract and the store implements both field and value discovery.
503	Discovery exceeded a backend memory or resource budget and returned TRACE_QUERY_RESOURCE_LIMIT.
504	A PostgreSQL or ClickHouse query exceeded its configured database execution timeout. Discovery returns TRACE_QUERY_EXECUTION_TIMEOUT.

Query spans

Use querySpans(), mastra api span query, or POST /api/observability/spans/query to return completed span summaries across traces. It requires a positive ISO timestamp range over span start time and supports cursor pagination. This example finds tool calls that took at least one second:

const result = await mastraClient.querySpans({
  timeRange: {
    from: '2026-10-01T00:00:00.000Z',
    to: '2026-10-02T00:00:00.000Z',
  },
  where: {
    op: 'and',
    args: [
      { op: 'eq', left: { path: 'spanType' }, right: { literal: 'tool_call' } },
      { op: 'gte', left: { path: 'durationMs' }, right: { literal: 1000 } },
    ],
  },
  orderBy: [{ field: 'startedAt', direction: 'desc' }],
  page: { limit: 100 },
})

// Pass result.page.next as page.after to continue with the same filters and ordering.
console.log(result.spans)

Pages default to 100 rows and allow at most 1,000. The response is { spans, page: { next } }, without a total count. Ordering accepts exactly one of startedAt, endedAt, timeToFirstChunkMs, or timeToFirstTokenMs, ascending or descending, and defaults to descending start time. Spans without a value for the timeToFirstChunkMs or timeToFirstTokenMs sort field sort last in both directions.

Fields	Operators
traceId, spanId, parentSpanId, threadId	eq, ne, in, notIn, exists, notExists
organizationId, resourceId, runId, sessionId, userId, requestId, serviceName, experimentId, environment, executionSource	Same string operators
Entity, parent entity and root entity Type, Id, Name, VersionId fields, such as parentEntityId	Same string operators
name, spanType, status	Same string operators; name also accepts matches / notMatches for normalized whole words and phrases
model, provider, outcome, stopReason	Same string operators
startedAt, endedAt, durationMs, timeToFirstChunkMs, timeToFirstTokenMs	Comparisons, membership and presence
error	exists, notExists

Combine predicates with and, or and not. Membership uses { op: 'in', value: { path: 'spanType' }, set: ['agent_run', 'tool_call'] }. A span's status is error when it has an error and success otherwise.

Each summary includes identity, context, entity lineage, status, timestamps, durationMs, model/provider and input/output/error previews.

Check capabilities.spanQueryContextFields from getObservabilityCapabilities() before filtering on traceId, spanId, parentSpanId, threadId, resourceId, requestId, serviceName, experimentId, environment, executionSource, or the parent/root entity Type, Id, and Name fields. If this capability is missing or false, requests using these filters return a structured 422 error. It doesn't gate runId, sessionId, userId, organizationId, or entity VersionId filters.

Check capabilities.spanQueryModelFields before filtering on model or provider. These filters read only the span's model and provider columns. @mastra/observability fills the columns only when a span has both a model and a provider, and versions before it filled them left them empty. Filters treat empty columns as missing, so eq, in and exists don't match those spans while ne, notIn and notExists do. Results still return model and provider for them from the span's attributes. Trace query spans.some filters fall back to the attributes and do match those spans. With an older adapter, requests using these filters return 501.

Check capabilities.spanQueryOutcomeFields before filtering or grouping spans by outcome and stopReason. Rows then also include outcome, stopReason and stopReasonContext. These record how the span ended, separately from status, and are null when the span didn't record them. A missing outcome means "not recorded", not success. stopReasonContext can't be used in filters and is omitted when the store lacks span-query-outcome-fields. For traces, capabilities.traceQueryOutcomeFields enables the same fields on queryTraces() and aggregateTraces(), where they read the root span. With an older adapter, requests using these fields return 501:

const { capabilities } = await mastraClient.getObservabilityCapabilities()

if (capabilities.spanQueryOutcomeFields) {
  const limited = await mastraClient.querySpans({
    timeRange: { from: '2026-10-01T00:00:00.000Z', to: '2026-10-02T00:00:00.000Z' },
    where: { op: 'eq', left: { path: 'outcome' }, right: { literal: 'limit' } },
  })
  console.log(limited.spans.map(span => [span.stopReason, span.stopReasonContext]))
}

See Filter by outcome for trace-scope and spans.some semantics.

DuckDB returns null for parent/root entity type, ID and name on spans written before @mastra/duckdb stored them. Equality filters on these fields don't match those spans.

The CLI returns the same response under data: read data.spans and pass data.page.next as page.after. Span queries support only keyset pagination, so reuse the same time range, predicate, and ordering when following the cursor. A malformed cursor returns 400, while a cursor that doesn't match the query or scope returns 409.

Use mastra api trace span <traceId> <spanId> or getSpan(traceId, spanId) to retrieve a selected span's full details.

Aggregate traces

Use aggregateTraces() or POST /api/observability/traces/aggregate to count the selected traces and compute measures over them. Add groupBy to split the result by up to two dimensions, and interval to split each group into time buckets. The response contains one row per group, or one row per group and bucket.

Aggregate traces with the client SDK

This request compares agents by volume, error rate, and latency over one week:

import { MastraClient } from '@mastra/client-js'

const mastraClient = new MastraClient({ baseUrl: 'http://localhost:4111' })

const { capabilities } = await mastraClient.getObservabilityCapabilities()

if (capabilities.traceAggregate) {
  const { rows, truncated } = await mastraClient.aggregateTraces({
    timeRange: {
      from: '2026-08-01T00:00:00.000Z',
      to: '2026-08-08T00:00:00.000Z',
    },
    where: { op: 'eq', left: { path: 'entityType' }, right: { literal: 'agent' } },
    groupBy: ['entityName'],
    measures: ['count', 'errorRate', 'duration.p95'],
    orderBy: { field: 'errorRate', direction: 'desc' },
    limit: 10,
  })
}

The client retries server errors like queryTraces(), but doesn't retry 422 validation errors.

Send an aggregate HTTP request

Send the same request as a JSON body:

curl --request POST \
  --url http://localhost:4111/api/observability/traces/aggregate \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{
    "timeRange": {
      "from": "2026-08-01T00:00:00.000Z",
      "to": "2026-08-08T00:00:00.000Z"
    },
    "where": { "op": "eq", "left": { "path": "entityType" }, "right": { "literal": "agent" } },
    "groupBy": ["entityName"],
    "measures": ["count", "errorRate", "duration.p95"],
    "orderBy": { "field": "errorRate", "direction": "desc" },
    "limit": 10
  }'

Aggregate request fields

Field	Required	Description
timeRange	Yes	Trace start-time boundary, as in Select traces. The range can span up to 365 days.
where	No	Trace predicate, as in Select traces.
measures	Yes	One or more distinct measure names.
groupBy	No	Up to two distinct dimensions. Defaults to [], which treats all selected traces as one group. With interval, that group has one row per non-empty bucket. No matching traces returns no rows.
interval	No	Bucket width: '1m', '5m', '15m', '1h', or '1d'. See Time buckets.
having	No	Scalar predicate over measures that removes groups after grouping. See Filter and order groups.
orderBy	No	{ field, direction }. field is a requested measure, count, or a groupBy dimension, and direction is asc or desc. Defaults to count desc.
limit	No	Maximum number of groups. Defaults to 100 and has a maximum of 1,000.

Unknown fields are rejected. Aggregation isn't paginated. Narrow the selection, or rank by a different measure, when truncated is true.

Dimensions

Dimensions read the current root span of each selected trace:

Dimension	Notes
entityType, entityName, environment, status, serviceName, executionSource	Root span attributes.
threadId, resourceId, userId, sessionId, organizationId, experimentId	Context identifiers recorded on the root span.
name, spanType	Root span name and type. Requires the traceQueryRootSpanFields capability.
entityId	Root span entity ID. Groups traces that have no entityName, such as scorer runs. Requires the traceQueryRootEntityId capability.
outcome, stopReason	How the root span ended and why. Traces whose root span didn't record a value group as null. Requires the traceQueryOutcomeFields capability.
metadata.<key>	One non-empty top-level metadata key. Values are trimmed. Missing, empty, and non-string values group as null.

Every dimension except executionSource is also a trace-scope where field, so you can drill into a group by adding its values to the predicate. traceId, spanId, runId, and requestId aren't groupable. Traces with no value for a dimension form their own group with a null value. To select that group in a query, use notExists for the dimension instead of eq with literal: null.

Measures

Measure	Value
count	Number of selected traces.
countDistinct.<field>	Number of distinct non-null values of a dimension or traceId, such as countDistinct.userId.
duration.avg, duration.min, duration.max	Root span duration in milliseconds.
duration.p50, duration.p90, duration.p95, duration.p99	Root span duration percentile in milliseconds. Percentiles can be approximate, depending on the store.
errorCount	Number of traces whose root span recorded an error.
errorRate	errorCount divided by count, from 0 to 1.
tokens.input.sum, tokens.input.avg	Input tokens.
tokens.output.sum, tokens.output.avg	Output tokens.
tokens.total.sum, tokens.total.avg	Input plus output tokens.
tokens.reasoning.sum, tokens.reasoning.avg	Reasoning output tokens.
tokens.cached.sum, tokens.cached.avg	Input tokens read from the provider's cache.
cost.sum, cost.avg	Estimated model cost.
timeToFirstToken.avg, timeToFirstToken.min, timeToFirstToken.max	Root span time to first token in milliseconds. Requires the aggregateTimeToFirstMeasures capability.
timeToFirstToken.p50, timeToFirstToken.p90, timeToFirstToken.p95, timeToFirstToken.p99	Root span time to first token percentile in milliseconds. Requires the aggregateTimeToFirstMeasures capability.
timeToFirstToken.count	Number of traces whose root span recorded a time to first token. Requires the aggregateTimeToFirstMeasures capability.
timeToFirstChunk.avg, timeToFirstChunk.min, timeToFirstChunk.max	Root span time to first chunk in milliseconds. Requires the aggregateTimeToFirstMeasures capability.
timeToFirstChunk.p50, timeToFirstChunk.p90, timeToFirstChunk.p95, timeToFirstChunk.p99	Root span time to first chunk percentile in milliseconds. Requires the aggregateTimeToFirstMeasures capability.
timeToFirstChunk.count	Number of traces whose root span recorded a time to first chunk. Requires the aggregateTimeToFirstMeasures capability.

Numeric measures over metadata aren't supported.

Token and cost measures

Token and cost measures read the model usage metrics recorded for each trace, so they're only available when the store retains metrics for the selected time range. A metric belongs to a trace when it shares the trace's traceId and was recorded at or after timeRange.from. Every such metric counts, including usage recorded after timeRange.to and usage recorded before a suspend and resume. Retried metric exports are counted once.

A trace that recorded at least one usage metric is usage-bearing. tokens.<kind>.sum adds the tokens of a group's usage-bearing traces, and tokens.<kind>.avg divides that sum by the number of usage-bearing traces, not by count. Both are null when the group has no usage-bearing traces.

Cost comes from estimated cost recorded on total input and total output token metrics. cost.sum adds the cost of traces with at least one priced metric, and cost.avg divides by the number of those traces. A partly priced call, where Mastra could price some token types but not others, adds the cost of the priced token types. A metric with any other pricing failure adds no cost.

When you request any cost.* measure, every row also includes a cost object:

Field	Description
coverage	Fraction of usage-bearing traces whose usage was fully priced, from 0 to 1. A trace with a partly priced call isn't fully priced, so it lowers coverage even though its known cost is in cost.sum. null when the group has no usage-bearing traces.
unit	Cost unit of the priced metrics, such as USD. 'mixed' when they use more than one unit, and null when none are priced.

cost.sum and cost.avg are null when the group has no priced traces or unit is 'mixed'. A coverage below 1 means the cost understates the group's spend. The cost object isn't a measure, so having and orderBy can't reference it.

Time to first token and chunk measures

timeToFirstToken.* and timeToFirstChunk.* read the root span's timeToFirstTokenMs and timeToFirstChunkMs, the same values queryTraces() returns. An agent run copies them from the first model generation of its own loop:

• timeToFirstChunk.* is measured from the agent run's start, so it includes work done before the first model call, such as input processors and memory recall.
• timeToFirstToken.* is the time to first token of the model call that streamed that first chunk.

Traces without a value are skipped, never counted as 0. This includes traces whose agent run didn't stream, and traces rooted in a workflow, since a workflow run records neither value. avg, min, max, and the percentiles are null for a group where no trace has a value, and .count is the number of traces that do have one. Percentiles can be approximate, depending on the store.

Stores advertise support through the aggregate-time-to-first-measures feature, and the server reports it as the aggregateTimeToFirstMeasures capability. Against an older store, a request for any of these measures returns 501 before storage runs, and field discovery omits them.

const { capabilities } = await mastraClient.getObservabilityCapabilities()

if (capabilities.traceAggregate && capabilities.aggregateTimeToFirstMeasures) {
  const { rows } = await mastraClient.aggregateTraces({
    timeRange: {
      from: '2026-08-01T00:00:00.000Z',
      to: '2026-08-02T00:00:00.000Z',
    },
    where: { op: 'eq', left: { path: 'entityType' }, right: { literal: 'agent' } },
    groupBy: ['entityName'],
    measures: ['timeToFirstChunk.p50', 'timeToFirstChunk.p95', 'timeToFirstChunk.count'],
    orderBy: { field: 'timeToFirstChunk.p95', direction: 'desc' },
  })
}

Filter and order groups

A group is one distinct combination of groupBy values. having, orderBy, and limit operate on groups and use each group's measures over the whole time range, even when interval is set. The server applies having first, then orders the remaining groups and keeps the first limit. truncated is true when more groups remained than limit.

having uses the scalar predicate shape with a measure name as the path, such as { op: 'gt', left: { path: 'duration.p95' }, right: { literal: 5000 } }. It can reference requested measures and count, even when count isn't requested. It supports comparison operators, in, and notIn with number literals, combined with and, or, and not. Presence, tag, and text operators aren't supported. A null measure never satisfies a having comparison, so the group is removed.

This request keeps workflows with a 95th-percentile duration above five seconds:

const slowWorkflows = await mastraClient.aggregateTraces({
  timeRange: {
    from: '2026-08-01T00:00:00.000Z',
    to: '2026-08-08T00:00:00.000Z',
  },
  where: { op: 'eq', left: { path: 'entityType' }, right: { literal: 'workflow_run' } },
  groupBy: ['entityName'],
  measures: ['count', 'duration.p95', 'duration.p99'],
  having: { op: 'gt', left: { path: 'duration.p95' }, right: { literal: 5000 } },
  orderBy: { field: 'duration.p95', direction: 'desc' },
})

Ordering is deterministic. Ties are broken by dimension values in ascending order. null measure and dimension values sort last in both directions.

Time buckets

With interval, each group that survives having and limit expands into one row per non-empty bucket. Buckets are aligned to UTC, so 1d buckets start at midnight UTC, and each row's bucket is the ISO start time of its bucket. A trace belongs to the bucket containing its root startedAt. Rows are ordered by group, then by bucket ascending.

Buckets without traces are omitted rather than returned as zero rows. Fill gaps on the client when you draw a chart. Because groups are ranked over the whole time range first, every returned group includes its full series of non-empty buckets.

This request returns daily token usage and cost for each agent:

const dailyCost = await mastraClient.aggregateTraces({
  timeRange: {
    from: '2026-08-01T00:00:00.000Z',
    to: '2026-08-08T00:00:00.000Z',
  },
  where: { op: 'eq', left: { path: 'entityType' }, right: { literal: 'agent' } },
  groupBy: ['entityName'],
  interval: '1d',
  measures: ['count', 'tokens.total.sum', 'cost.sum', 'cost.avg'],
  orderBy: { field: 'cost.sum', direction: 'desc' },
  limit: 5,
})

Aggregate responses

{
  "rows": [
    {
      "dimensions": { "entityName": "supportAgent" },
      "bucket": "2026-08-01T00:00:00.000Z",
      "measures": {
        "count": 412,
        "tokens.total.sum": 1830442,
        "cost.sum": 4.21,
        "cost.avg": 0.0103
      },
      "cost": { "coverage": 0.99, "unit": "USD" }
    }
  ],
  "truncated": false
}

dimensions is keyed by groupBy field and is present when groupBy isn't empty. bucket is present when interval is set. measures is keyed by the requested measure names. When no traces match, or having removes every group, rows is empty.

Aggregate limits and errors

Limit	Maximum
Time range	365 days
groupBy dimensions	2
limit	1,000 groups
Buckets in the time range	1,000
limit × buckets	10,000 rows

Predicate limits from Query limits apply to both where and having, and the request body is limited to 256 KiB. A time range that doesn't start on a bucket boundary touches one more bucket than its length divided by the interval. If the range has too many buckets, choose a wider interval. If limit × buckets exceeds 10,000, reduce limit or widen the interval.

PostgreSQL and ClickHouse use the same execution timeout for aggregations as for trace queries, defaulting to 15 seconds. The timeout doesn't apply to DuckDB.

Status	Meaning
400	Malformed JSON.
413	The request body exceeds 256 KiB.
422	The request is invalid and returns TRACE_QUERY_INVALID with issues. A time range exceeding 365 days produces an invalid_request issue. Other issue codes include too_many_buckets, too_many_rows, field_not_allowed, invalid_metadata_key, and operator_not_allowed.
501	The installed Core or configured observability store doesn't support aggregation or a requested field. Returns TRACE_AGGREGATE_UNSUPPORTED.
504	A PostgreSQL or ClickHouse aggregation exceeded its execution timeout. Returns TRACE_QUERY_EXECUTION_TIMEOUT without partial results.

Aggregate spans

Use aggregateSpans() or POST /api/observability/spans/aggregate to compute measures over individual spans instead of traces. Trace aggregation only sees each trace's root span, so an agent run nested inside a workflow isn't counted. Span aggregation counts every span that matches, at any depth.

The request has the same shape as aggregateTraces(), with the same fixed measures, having, orderBy, interval, and shared limits. Its where, dimensions, and countDistinct fields follow the span contract below. Unsupported span aggregation returns 501 SPAN_AGGREGATE_UNSUPPORTED.

Aggregate spans with the client SDK

This request counts agent runs by status per day, including nested runs:

const { capabilities } = await mastraClient.getObservabilityCapabilities()

if (capabilities.spanAggregate) {
  const { rows } = await mastraClient.aggregateSpans({
    timeRange: {
      from: '2026-08-01T00:00:00.000Z',
      to: '2026-08-08T00:00:00.000Z',
    },
    where: { op: 'eq', left: { path: 'spanType' }, right: { literal: 'agent_run' } },
    groupBy: ['status'],
    interval: '1d',
    measures: ['count', 'duration.p95'],
  })
}

Send the same body to POST /api/observability/spans/aggregate to use HTTP directly.

Span selection

Aggregation uses the same span predicates as querySpans(), with the ClickHouse selection differences described below:

• timeRange bounds each span's startedAt, and can span up to 365 days.
• where accepts the querySpans() fields and operators, such as spanType, name, status, durationMs, model, provider, entity and parent/root entity fields, and context identifiers. Like querySpans(), model and provider filters read only the span's columns, not its attributes.
• Only completed spans count. When a span was exported more than once, only its latest copy counts.

count counts selected spans in each group or time bucket. Buckets use each span's startedAt. duration.* measures read the span's own duration, and errorCount and errorRate read whether the span itself recorded an error.

timeToFirstToken.* and timeToFirstChunk.* read the span's own timeToFirstTokenMs and timeToFirstChunkMs, and skip spans without a value. Several span types record them:

• model_inference: each provider call. Filter to this type for per-call time to first token.
• model_step: each step of a generation, with the values of the inference that streamed the step's first chunk.
• model_generation: the whole generation, with the values of its first streamed inference only. A multi-step generation, such as a tool loop, reports only its first call.
• agent_run: the agent run, with time to first chunk measured from the run's start. Filter to this type for latency as the caller of each agent sees it.

Without a spanType filter, one agent call contributes its agent run, its model generation, and every model step and model inference.

On ClickHouse, a direct spanType eq or in predicate, or one immediately inside an and, can select the smaller branch table. All selected types must be branch types: agent_run, workflow_run, processor_run, scorer_run, rag_ingestion, tool_call, or mcp_tool_call. Every requested field must be available in that table. An equivalent or predicate doesn't select this optimization. The branch path can differ from querySpans():

• Spans written before an upgrade from @mastra/clickhouse 1.4, 1.5, or 1.6 aren't in the branch table, so older windows can count fewer spans.
• After ClickHouse merges the copies of a span exported more than once, the branch table keeps the copy written last, not the one that ended last. The retained version's status and duration can differ. This can change filtered counts and group membership even though the span identity is unchanged.

Span dimensions

Dimension	Notes
name, spanType, status	The span's own name, type, and status. status is error when the span recorded an error and success otherwise.
outcome, stopReason	How the span ended and why. Spans without a recorded value group as null. Requires the spanQueryOutcomeFields capability.
entityType, entityId, entityName	The entity that produced the span.
parentEntityType, parentEntityName, rootEntityType, rootEntityName	The calling entity and the trace's root entity, such as the agent that made a model call.
environment, serviceName, executionSource	Deployment attributes recorded on the span.
threadId, resourceId, userId, sessionId, organizationId, experimentId	Context identifiers recorded on the span.
model, provider	The model and provider recorded on the span's model and provider columns, such as claude-sonnet-5-5 and anthropic. Requires the spanAggregateModelDimensions capability.

countDistinct.<field> accepts these dimensions and traceId, so countDistinct.traceId counts the traces the selected spans belong to. Span aggregation can't group by metadata.<key> or identity fields such as traceId, spanId, or runId.

Like the model and provider filters, the model and provider dimensions read only the span's columns, not its attributes. Existing spans aren't backfilled, so spans exported before @mastra/observability filled the columns group under null, and countDistinct.model doesn't count them.

Stores advertise support through the span-aggregate-model-dimensions feature, and the server reports it as the spanAggregateModelDimensions capability. Against an older store, a request that groups by model or provider or counts their distinct values returns 501 before storage runs.

Span tokens and cost

Token and cost measures use the usage metrics recorded by each span, matched by traceId and spanId. The trace token rules apply per span: metrics from timeRange.from on count, retried exports count once, and averages divide by usage-bearing spans.

Usage isn't added up to parent spans. Model calls record usage on model_generation spans, so an agent_run span has no tokens of its own unless a model call under it wasn't exported. When a model_generation span isn't exported, its usage is recorded on the nearest exported ancestor instead. For token totals per agent, select model_generation spans and group by the calling agent:

const tokensByAgent = await mastraClient.aggregateSpans({
  timeRange: {
    from: '2026-08-01T00:00:00.000Z',
    to: '2026-08-08T00:00:00.000Z',
  },
  where: { op: 'eq', left: { path: 'spanType' }, right: { literal: 'model_generation' } },
  groupBy: ['parentEntityName'],
  measures: ['tokens.total.sum', 'cost.sum'],
  orderBy: { field: 'cost.sum', direction: 'desc' },
})

To compare models instead, group the same spans by model:

const { capabilities } = await mastraClient.getObservabilityCapabilities()

if (capabilities.spanAggregateModelDimensions) {
  const { rows } = await mastraClient.aggregateSpans({
    timeRange: {
      from: '2026-08-01T00:00:00.000Z',
      to: '2026-08-08T00:00:00.000Z',
    },
    where: { op: 'eq', left: { path: 'spanType' }, right: { literal: 'model_generation' } },
    groupBy: ['provider', 'model'],
    measures: ['count', 'tokens.total.sum', 'cost.sum'],
  })
  // rows[0]: { dimensions: { provider: 'anthropic', model: 'claude-sonnet-5-5' }, measures: { ... } }
}

List the spans behind a group

Aggregation never returns span records. To inspect a group, send the same timeRange and where to querySpans(), and add non-null dimension values as eq conditions. For a null dimension, use notExists, such as { op: 'notExists', path: 'entityName' }. eq with literal: null is rejected. For example, add { op: 'eq', left: { path: 'status' }, right: { literal: 'error' } } to list the failed agent runs from the first example.

Trusted tenant scope

Hosts that serve more than one tenant supply a trusted scope outside the query document. The server reads the reserved organizationId request-context key, which only server-side authentication can set, and passes { organizationId } to the planner. The scope is carried on the trusted plan and ANDed into every scan: current roots, related spans, the usage metrics read by token and cost measures, span aggregation, and both discovery routes. A related record from another tenant that shares a traceId never qualifies a trace.

Hosts that call storage directly can pass a fuller scope to planTraceQuery(), planTraceAggregate(), planSpanAggregate(), planTraceQueryObservedFields(), and planTraceQueryValues():

const plan = planTraceQuery(parseTraceQueryRequest(request), {
  scope: { organizationId: 'org_123', resourceId: 'project_456' },
})

resourceId is optional and narrows within the organization. Keyset and delta cursors are bound to the scope. A cursor created under one scope and reused under another returns 409 before storage runs. A scoped request against an observability store or @mastra/core that predates tenant scope returns 501 instead of running unscoped. Stores advertise support through the trace-query-tenant-scope feature. Callers can't name projectId in predicates. An organizationId predicate compares recorded data and is ANDed with the scope, so it can only narrow the tenant's own traces and never widens it. Without a scope the routes behave as before, so self-hosted installations keep their own tenant model.

Discover fields and values

Use trace-query discovery to build field and value autocomplete without scanning trace payloads or copying the query grammar into your client. The configured observability store must support trace-query-discovery.

The CLI doesn't expose dedicated discovery commands, so use the SDK methods below or the HTTP endpoints POST /api/observability/traces/query/fields and POST /api/observability/traces/query/values.

Call getTraceQueryFields() for one predicate scope:

const timeRange = {
  from: '2026-08-01T00:00:00.000Z',
  to: '2026-08-08T00:00:00.000Z',
}

const fields = await mastraClient.getTraceQueryFields({
  timeRange,
  predicateScope: 'trace',
  search: 'region',
  limit: 25,
})

The response separates canonical fields from observed metadata fields:

{
  "canonicalFields": [],
  "observedFields": [
    {
      "path": "metadata.region",
      "valueKind": "string",
      "operators": ["eq", "ne", "in", "notIn", "exists", "notExists"],
      "valueSuggestions": true,
      "occurrences": 125
    }
  ],
  "observedFieldsTruncated": false
}

Canonical fields come from the same registry used to validate trace queries. They don't consume limit. The canonical order follows that registry. observedFields contains top-level string metadata.<key> paths found on qualifying current root spans. These fields are ordered by occurrences descending, then by path. Nested metadata, keys containing ., empty keys, non-string values, and oversized paths are omitted.

predicateScope selects the local predicate grammar. It isn't an authorization scope:

predicateScope	Example returned path	Query usage
trace	environment or metadata.region	Root where predicate
spans	model	Inside spans.some or spans.none

A field picker can request the trace and spans scopes in parallel and label each field with its scope. Every returned path is directly accepted by queryTraces() in its matching scope. metadata by itself isn't a field. A user interface can offer manual metadata-key entry separately.

For the trace scope, when the server and store support aggregation, the fields response also includes an aggregate object that describes what aggregateTraces() accepts:

{
  "aggregate": {
    "dimensions": [{ "path": "entityName", "valueKind": "string" }],
    "measures": [
      {
        "name": "duration.p95",
        "kind": "duration",
        "statistic": "p95",
        "unit": "milliseconds",
        "approximate": true
      }
    ]
  }
}

dimensions lists the canonical dimensions that the store supports. Every observed metadata.<key> field is also a dimension, and countDistinct.<field> accepts any dimension or traceId. measures lists each fixed measure with its kind, unit, and whether it's approximate.

After a user selects one field with valueSuggestions: true, call getTraceQueryValues():

const abortController = new AbortController()
const suggestions = await mastraClient.getTraceQueryValues(
  {
    timeRange,
    predicateScope: 'trace',
    path: 'metadata.region',
    search: 'west',
    limit: 25,
  },
  { signal: abortController.signal },
)

{
  "values": [
    { "value": "eu-west-1", "count": 82 },
    { "value": "us-west-2", "count": 43 }
  ],
  "valuesTruncated": false
}

Discovery returns string values only. Missing values, empty strings, and strings larger than 4,096 UTF-8 bytes are omitted. Values are ordered by count descending, then by value. Suggestions are advisory: queryTraces() continues to accept valid manual literals that discovery doesn't return.

Value suggestions are available for these canonical fields:

Scope	Fields
Trace	entityName, entityType, name, spanType, environment, status, outcome, stopReason, tags
Spans	name, spanType, model, provider, status, outcome, stopReason, entityType, entityName

outcome and stopReason suggestions require the traceQueryOutcomeFields capability. Each executable top-level string metadata.<key> field also supports value suggestions in the trace scope. Identifiers, timestamps, durations, time to first chunk, time to first token, errors, and version IDs don't. The fields response still includes canonical fields with valueSuggestions: false. The values endpoint rejects those paths with 422 TRACE_QUERY_INVALID.

Both discovery requests require a half-open time range. They use the same current-record semantics as queryTraces(): qualifying completed roots satisfy from <= startedAt < to, and related values come only from current records joined to those roots by traceId. Discovery doesn't accept or apply a draft where predicate.

search defaults to the empty string, is trimmed, and matches case-insensitive literal substrings. Characters such as % and _ have no wildcard meaning. An empty search returns the most frequent results. limit defaults to 25 and has a maximum of 100. Both endpoints fetch one extra result to set their truncation flag, but don't provide a cursor. When a flag is true, refine search. It doesn't imply that a next page is available. No matches return an empty array and false truncation.

If discovery exceeds its execution timeout, the route returns 504 TRACE_QUERY_EXECUTION_TIMEOUT. If the backend exceeds a memory or resource budget, it returns 503 TRACE_QUERY_RESOURCE_LIMIT. Neither failure returns partial suggestions or sets a truncation flag. Truncation only means a complete ranking was cut to the requested response limit.

ClickHouse discovery requests default to a 5-second timeout and a 256 MiB per-query memory limit. Configure these with observability.traceQuery.discovery.timeoutMs and observability.traceQuery.discovery.memoryLimitBytes on ClickhouseStoreVNext. Discovery falls back to observability.traceQuery.timeoutMs when its timeout isn't configured, without changing ordinary queryTraces() execution limits.

Both routes require observability:read and use the configured observability storage boundary. Don't send organization or project authorization fields in the request. Treat discovered paths, values, and counts as observability data.

Limitations

The query and aggregate APIs consider completed traces and spans only, not running ones. Trace queries can't return custom projections, embedded evidence, or summaries, and numbered trace pages include a total count. Trace aggregation can't group by span fields other than the root span's name, spanType, and entityId, or by traceId or attributes.*. Aggregation can't compute numeric measures over metadata, paginate groups, or zero-fill empty buckets. Span aggregation can't group by model or provider, and it attributes tokens only to the span that recorded them. The CLI's mastra api trace aggregate covers trace aggregation, and mastra api span query and mastra api span aggregate cover span queries and span aggregation.

Related

• Client SDK observability reference
• Tracing overview
• Span interfaces