Advanced trace queries

Use POST /api/observability/traces/query to find completed logical traces that match trace fields and related span or score records. The endpoint returns a fixed lightweight trace projection, or distinct thread IDs when you group by threadId.

The endpoint uses the same authentication as other observability routes and requires the observability:read permission. The configured observability store must support advanced trace queries.

Query traces with the client SDK

Pass the query to queryTraces():

const result = await mastraClient.queryTraces({
  timeRange: {
    from: '2026-08-01T00:00:00.000Z',
    to: '2026-08-08T00:00:00.000Z',
  },
  where: {
    scores: {
      some: {
        op: 'and',
        args: [
          { op: 'eq', left: { path: 'scorerId' }, right: { literal: 'factuality' } },
          { op: 'lt', left: { path: 'score' }, right: { literal: 0.6 } },
        ],
      },
    },
  },
  orderBy: [{ field: 'startedAt', direction: 'desc' }],
  page: { limit: 25 },
})

A some clause matches when one related record satisfies its complete nested predicate. In this example, scorerId and score must match on the same score record. A none clause matches when no related record satisfies its complete nested predicate.

Span clauses examine the current root span and current child spans. The root timeRange applies only to the selected current root's startedAt. Related spans and scores can participate even when their own timestamps are outside that range. Related records correlate only through a matching non-null traceId.

Send an HTTP request

curl --request POST \
  --url http://localhost:4111/api/observability/traces/query \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{
    "timeRange": {
      "from": "2026-08-01T00:00:00.000Z",
      "to": "2026-08-08T00:00:00.000Z"
    },
    "where": {
      "spans": {
        "some": {
          "op": "and",
          "args": [
            { "op": "eq", "left": { "path": "spanType" }, "right": { "literal": "tool_call" } },
            { "op": "exists", "path": "error" }
          ]
        }
      }
    }
  }'

Request fields

Field	Required	Description
timeRange	Yes	Trace start-time boundary. from is inclusive, to is exclusive, and the range can span at most 31 days. Both values must be ISO timestamps.
where	No	Recursive trace predicate. Supports scalar conditions and spans.some, spans.none, scores.some, and scores.none.
group	No	Set to { by: ['threadId'] } to return distinct non-null thread IDs.
orderBy	No	One item ordering ungrouped results by startedAt or endedAt, in asc or desc order. Defaults to startedAt desc. Not accepted with group.
page	No	{ limit, after }. limit defaults to 100 and has a maximum of 1000. Pass the opaque page.next value as after.

Unknown fields are rejected. The request can't select a projection, declare joins, or control authorization. Request bodies are limited to 256 KiB.

Query limits

The planner rejects a query before storage execution when it exceeds any of these limits:

Input	Maximum
Root timeRange	31 days
Predicate nesting depth	12 levels
Predicate nodes	100
Related spans and scores clauses	8 total
Values in one in or notIn set	100
Comparison literals and membership values	1,000 total
String literal	4,096 UTF-8 bytes
Raw predicate path	128 UTF-8 bytes
page.limit	1,000
HTTP request body	256 KiB

Each comparison literal counts as one literal. Each member of an in or notIn set also counts as one literal, even when the set is within its per-set limit.

Hono and Fastify enforce the request-body limit before JSON parsing. Express and Elysia reject an oversized body before the route handler or storage runs, but the host framework may already have parsed a request without a reliable Content-Length header. Configure the host application's JSON parser or body limit when you need a hard pre-parse memory ceiling.

Fields and operators

Predicate context	Fields	Operators
Trace	traceId, threadId, resourceId, entityName, entityType, environment, status	eq, ne, in, notIn, exists, notExists
Trace	startedAt, endedAt	eq, ne, in, notIn, lt, lte, gt, gte, exists, notExists
Span	spanType	eq, ne, in, notIn, exists, notExists
Span	error	exists, notExists
Score	scorerId	eq, ne, in, notIn, exists, notExists
Score	score	eq, ne, in, notIn, lt, lte, gt, gte, exists, notExists

Compose predicates with { op: 'and', args: [...] }, { op: 'or', args: [...] }, and { op: 'not', arg: ... }. Comparison predicates place a field reference on the left and a literal on the right. Membership predicates use a field reference in value and a homogeneous literal array in set.

Responses

An ungrouped query returns only lightweight completed traces:

{
  "traces": [
    {
      "traceId": "trace-123",
      "rootSpanId": "span-123",
      "threadId": "thread-123",
      "resourceId": "resource-123",
      "startedAt": "2026-08-03T10:00:00.000Z",
      "endedAt": "2026-08-03T10:00:01.000Z",
      "entityName": "support-agent",
      "entityType": "agent",
      "environment": "production",
      "status": "success"
    }
  ],
  "page": { "next": null }
}

A grouped query returns distinct non-null thread IDs in ascending order:

const result = await mastraClient.queryTraces({
  timeRange: {
    from: '2026-08-01T00:00:00.000Z',
    to: '2026-08-08T00:00:00.000Z',
  },
  group: { by: ['threadId'] },
})
// { groups: [{ threadId: 'thread-123' }], page: { next: null } }

Related evidence isn't embedded in either response. Use the trace-detail and branch APIs to load spans after selecting a result.

Pagination and errors

Ordering is deterministic. Ungrouped ordering appends traceId ascending as a tie-breaker. Grouped queries always order by threadId ascending.

Cursors are bound to the accepted normalized query shape and ordering. Reusing a cursor after changing the time range, predicates, grouping, or ordering returns 409. A malformed cursor returns 400.

Cursor pagination is deterministic, but it isn't a database snapshot. Traces or replacement signals written between page requests can change later pages.

PostgreSQL and ClickHouse stop an advanced trace query after 15 seconds by default and return 504 when the database timeout is exceeded. Set traceQueryTimeoutMs in the store's vNext observability configuration to an integer from 1 through 300,000 milliseconds to change the timeout. DuckDB doesn't currently provide query-scoped timeout or cancellation through its driver wrapper, so this 504 guarantee doesn't apply to DuckDB.

Status	Meaning
400	Malformed JSON or malformed cursor.
409	The cursor doesn't match the query.
413	The request body exceeds 256 KiB.
422	The JSON is well formed, but the request is structurally or semantically invalid. The response includes stable issue codes and paths.
501	The configured observability store doesn't support advanced trace queries.
504	A PostgreSQL or ClickHouse query exceeded its configured database execution timeout.

Limitations

The endpoint returns completed traces only. It doesn't support running traces, custom projections, embedded evidence, summaries, aggregations, grouping by fields other than threadId, or conditions over an entire group.

Related

• Client SDK observability reference
• Tracing overview
• Span interfaces