Advanced trace queries

Use queryTraces() or POST /api/observability/traces/query to find completed logical traces that match trace fields and related span, score, or feedback records. Use queryTraceThreads() or POST /api/observability/threads/query to find thread identities that qualify across multiple eligible traces.

Both endpoints use the same authentication as other observability routes and require the observability:read permission. The configured observability store must support the requested trace-query or thread-query operation.

Thread grouping is deprecated

The group option remains supported until the next major release. Use queryTraceThreads() to retrieve matching thread identities in new code.

Query traces with the client SDK

Pass the query to queryTraces():

const result = await mastraClient.queryTraces({
  timeRange: {
    from: '2026-08-01T00:00:00.000Z',
    to: '2026-08-08T00:00:00.000Z',
  },
  where: {
    scores: {
      some: {
        op: 'and',
        args: [
          { op: 'eq', left: { path: 'scorerId' }, right: { literal: 'factuality' } },
          { op: 'eq', left: { path: 'scorerVersion' }, right: { literal: '2.1.0' } },
          { op: 'lt', left: { path: 'score' }, right: { literal: 0.6 } },
        ],
      },
    },
  },
  orderBy: [{ field: 'startedAt', direction: 'desc' }],
  page: { limit: 25 },
})

A some clause matches when one related record satisfies its complete nested predicate. In this example, scorerId, scorerVersion, and score must match on the same score record. A none clause uses anti-existence semantics: it matches when no related record satisfies its complete nested predicate. A trace with no related scores therefore matches scores.none.

Span clauses examine the current root span and current child spans. The root timeRange applies only to the selected current root's startedAt. Related spans, scores, and feedback can participate even when their own timestamps are outside that range. Related records correlate only through a matching non-null traceId.

Query threads across traces

Pass an eligible trace selection and an optional thread predicate to queryTraceThreads():

const result = await mastraClient.queryTraceThreads({
  traces: {
    timeRange: {
      from: '2026-08-01T00:00:00.000Z',
      to: '2026-08-08T00:00:00.000Z',
    },
    where: {
      op: 'eq',
      left: { path: 'environment' },
      right: { literal: 'production' },
    },
  },
  where: {
    op: 'and',
    args: [
      {
        traces: {
          some: {
            scores: {
              some: {
                op: 'lt',
                left: { path: 'score' },
                right: { literal: 0.6 },
              },
            },
          },
        },
      },
      {
        traces: {
          some: {
            feedback: {
              some: {
                op: 'eq',
                left: { path: 'feedbackType' },
                right: { literal: 'clinician-correction' },
              },
            },
          },
        },
      },
    ],
  },
  page: { limit: 25 },
})

The server applies traces.timeRange and traces.where first. This produces the complete eligible trace population. It then derives distinct non-null threadId values and evaluates the top-level where predicate against eligible traces in each thread.

One traces.some clause requires one eligible trace to satisfy its complete nested trace predicate. Separate traces.some clauses are independent, so different traces in the same thread can satisfy them. Nested spans.some, scores.some, and feedback.some clauses still require one related record to satisfy every condition inside that clause.

traces.none: P uses anti-existence semantics: it matches only when no eligible trace in the thread satisfies P. This differs from traces.some: { feedback: { none: P } }, which requires at least one eligible trace with no matching feedback record.

The response contains identities only:

{
  "threads": [{ "threadId": "thread-123" }],
  "page": { "next": null }
}

Send an HTTP request

curl --request POST \
  --url http://localhost:4111/api/observability/traces/query \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{
    "timeRange": {
      "from": "2026-08-01T00:00:00.000Z",
      "to": "2026-08-08T00:00:00.000Z"
    },
    "where": {
      "spans": {
        "some": {
          "op": "and",
          "args": [
            { "op": "eq", "left": { "path": "spanType" }, "right": { "literal": "tool_call" } },
            { "op": "exists", "path": "error" }
          ]
        }
      }
    }
  }'

Discover fields and values

Use trace-query discovery to build field and value autocomplete without scanning trace payloads or copying the query grammar into your client. The configured observability store must support trace-query-discovery.

Call getTraceQueryFields() for one predicate scope:

const timeRange = {
  from: '2026-08-01T00:00:00.000Z',
  to: '2026-08-08T00:00:00.000Z',
}

const fields = await mastraClient.getTraceQueryFields({
  timeRange,
  predicateScope: 'trace',
  search: 'region',
  limit: 25,
})

The response separates canonical fields from observed metadata fields:

{
  "canonicalFields": [],
  "observedFields": [
    {
      "path": "metadata.region",
      "valueKind": "string",
      "operators": ["eq", "ne", "in", "notIn", "exists", "notExists"],
      "valueSuggestions": true,
      "occurrences": 125
    }
  ],
  "observedFieldsTruncated": false
}

Canonical fields come from the same registry used to validate trace queries. They don't consume limit. The canonical order follows that registry. observedFields contains top-level string metadata.<key> paths found on qualifying current root spans. These fields are ordered by occurrences descending, then by path. Nested metadata, keys containing ., empty keys, non-string values, and oversized paths are omitted.

predicateScope selects the local predicate grammar. It isn't an authorization scope:

predicateScope	Example returned path	Query usage
trace	environment or metadata.region	Root where predicate
spans	model	Inside spans.some or spans.none
scores	scorerId	Inside scores.some or scores.none
feedback	feedbackType	Inside feedback.some or feedback.none

A global field picker can request all four scopes in parallel and qualify display labels locally. Every returned path is directly accepted by queryTraces() in its matching scope. metadata by itself isn't a field. A user interface can offer manual metadata-key entry separately.

After a user selects one field with valueSuggestions: true, call getTraceQueryValues():

const abortController = new AbortController()
const suggestions = await mastraClient.getTraceQueryValues(
  {
    timeRange,
    predicateScope: 'trace',
    path: 'metadata.region',
    search: 'west',
    limit: 25,
  },
  { signal: abortController.signal },
)

{
  "values": [
    { "value": "eu-west-1", "count": 82 },
    { "value": "us-west-2", "count": 43 }
  ],
  "valuesTruncated": false
}

Discovery returns string values only. Missing values, empty strings, and strings larger than 4,096 UTF-8 bytes are omitted. Values are ordered by count descending, then by value. Suggestions are advisory: queryTraces() continues to accept valid manual literals that discovery doesn't return.

Value suggestions are available for these canonical fields:

Scope	Fields
Trace	entityName, entityType, environment, status, tags
Spans	name, spanType, model, provider, status, entityType, entityName
Scores	scorerId, scorerVersion, scoreSource
Feedback	feedbackType, feedbackSource

Each executable top-level string metadata.<key> field also supports value suggestions in the trace scope. Identifiers, timestamps, durations, numeric score or feedback values, comments, errors, and version IDs don't. The fields response still includes canonical fields with valueSuggestions: false. The values endpoint rejects those paths with 422 TRACE_QUERY_INVALID.

Both discovery requests require a half-open time range of at most 31 days. They use the same current-record semantics as queryTraces(): qualifying completed roots satisfy from <= startedAt < to, and related values come only from current records joined to those roots by traceId. Discovery doesn't accept or apply a draft where predicate.

search defaults to the empty string, is trimmed, and matches case-insensitive literal substrings. Characters such as % and _ have no wildcard meaning. An empty search returns the most frequent results. limit defaults to 25 and has a maximum of 100. Both endpoints fetch one extra result to set their truncation flag, but don't provide a cursor. When a flag is true, refine search. It doesn't imply that a next page is available. No matches return an empty array and false truncation.

If discovery exceeds its execution timeout, the route returns 504 TRACE_QUERY_EXECUTION_TIMEOUT. If the backend exceeds a memory or resource budget, it returns 503 TRACE_QUERY_RESOURCE_LIMIT. Neither failure returns partial suggestions or sets a truncation flag. Truncation only means a complete ranking was cut to the requested response limit.

ClickHouse discovery requests default to a 5-second timeout and a 256 MiB per-query memory limit. Configure these with observability.traceQuery.discovery.timeoutMs and observability.traceQuery.discovery.memoryLimitBytes on ClickhouseStoreVNext. Discovery falls back to observability.traceQuery.timeoutMs when its timeout isn't configured, without changing ordinary queryTraces() execution limits.

Both routes require observability:read and use the configured observability storage boundary. Don't send organization or project authorization fields in the request. Treat discovered paths, values, and counts as observability data.

Request fields

Trace queries

Field	Required	Description
timeRange	Yes	Trace start-time boundary. from is inclusive and to is exclusive. Both values must be ISO timestamps, from must be earlier than to, and the range can't exceed 31 days.
where	No	Recursive trace predicate. Supports scalar conditions and spans, scores, and feedback some or none clauses.
group	No	Deprecated. Set to { by: ['threadId'] } to return distinct non-null thread IDs. Use queryTraceThreads() for new code.
orderBy	No	One item ordering ungrouped results by startedAt or endedAt, in asc or desc order. Defaults to startedAt desc. Not accepted with group or in delta mode.
page	No	{ limit, after }. limit defaults to 100 and has a maximum of 1000. Pass the opaque page.next value as after.
pagination	No	Numbered pages: { page, perPage }. Defaults to page 0 and 10 results. perPage has a maximum of 100.
mode	No	Set to 'delta' to poll using a delta cursor instead of page or pagination.
after	No	Delta mode only. Opaque deltaCursor from a numbered page or previous poll. Omit to establish the current watermark.
limit	No	Delta mode only. Maximum results per batch. Defaults to 10 and has a maximum of 100.

Thread queries

Field	Required	Description
traces	Yes	Eligible trace selection containing timeRange and an optional trace where predicate.
where	No	Recursive thread predicate composed with boolean operators and traces.some or traces.none. Each quantifier contains a complete trace predicate.
page	No	{ limit, after }. Thread identities always use fixed ordinal threadId ascending order. Pass the opaque page.next value as after.

Unknown fields are rejected. Requests can't select a projection, declare joins, request counts or measures, or control authorization. Request bodies are limited to 256 KiB.

Query limits

The planner rejects a query before storage execution when it exceeds any of these limits:

Input	Maximum
Trace selection timeRange	31 days
Predicate nesting depth	12 levels
Predicate nodes	100
Related traces, spans, scores, and feedback clauses	8 total
Values in one in or notIn set	100
Comparison literals and membership values	1,000 total
String literal	4,096 UTF-8 bytes
Raw predicate path	128 UTF-8 bytes
page.limit	1,000
HTTP request body	256 KiB

Each comparison literal counts as one literal. Each member of an in or notIn set also counts as one literal, even when the set is within its per-set limit.

Hono and Fastify enforce the request-body limit before JSON parsing. Express and Elysia reject an oversized body before the route handler or storage runs, but the host framework may already have parsed a request without a reliable Content-Length header. Configure the host application's JSON parser or body limit when you need a hard pre-parse memory ceiling.

Fields and operators

Predicate context	Fields	Operators
Trace	traceId, threadId, resourceId, entityName, entityType, environment, status	eq, ne, in, notIn, exists, notExists
Trace metadata	metadata.<key>	eq, ne, in, notIn, exists, notExists
Trace	startedAt, endedAt, durationMs	eq, ne, in, notIn, lt, lte, gt, gte, exists, notExists
Trace	tags	includes, notIncludes, exists, notExists
Span	name, spanType, model, provider, status, entityType, entityId, entityName, entityVersionId, parentEntityVersionId, rootEntityVersionId	eq, ne, in, notIn, exists, notExists
Span	startedAt, endedAt, durationMs	eq, ne, in, notIn, lt, lte, gt, gte, exists, notExists
Span	error	exists, notExists
Score	scorerId, scorerVersion, scoreSource, entityVersionId, parentEntityVersionId, rootEntityVersionId	eq, ne, in, notIn, exists, notExists
Score	score, timestamp	eq, ne, in, notIn, lt, lte, gt, gte, exists, notExists
Score	spanId	exists, notExists
Feedback	feedbackType, feedbackSource, feedbackUserId, sourceId, entityVersionId, parentEntityVersionId, rootEntityVersionId	eq, ne, in, notIn, exists, notExists
Feedback	value, timestamp	eq, ne, in, notIn, lt, lte, gt, gte, exists, notExists
Feedback	comment	exists, notExists

Compose predicates with { op: 'and', args: [...] }, { op: 'or', args: [...] }, and { op: 'not', arg: ... }. Comparison predicates place a field reference on the left and a literal on the right. Membership predicates use a field reference in value and a homogeneous literal array in set. Tag predicates name the stored tag collection in path. includes and notIncludes also take one tag in value, while exists and notExists take only op and path. See Filter by tags.

String comparisons are case-sensitive, and literals are never coerced. Canonical string fields compare exact stored values. Metadata string values are trimmed before comparison, as described below. Numeric predicates, including score and durationMs, require numbers, while timestamp predicates require ISO timestamp strings. A missing value satisfies neither positive nor ordered predicates, although it does satisfy the negative operators ne and notIn. Combine a negative predicate with exists when the field must also be present.

Filter by root duration

At trace scope, durationMs is the current completed root span's elapsed time in milliseconds. The value is derived from endedAt - startedAt when the query runs:

const slowRootTraces = {
  op: 'gt',
  left: { path: 'durationMs' },
  right: { literal: 5000 },
}

This top-level predicate doesn't inspect child spans. In contrast, spans.some with a durationMs predicate examines the current root span and current child spans, so a long child can satisfy that clause even when its root is shorter.

Trace queries exclude incomplete roots before evaluating predicates. As a result, { op: 'notExists', path: 'durationMs' } doesn't find running traces or malformed roots without a usable endedAt.

Filter by span properties

Every condition inside one spans.some or spans.none clause applies to the same current span. For example, this predicate finds a failed tool span whose name is medication_lookup; a matching name on one span and an error on another don't satisfy it:

const failedMedicationLookup = {
  spans: {
    some: {
      op: 'and',
      args: [
        { op: 'eq', left: { path: 'name' }, right: { literal: 'medication_lookup' } },
        { op: 'eq', left: { path: 'spanType' }, right: { literal: 'tool_call' } },
        { op: 'exists', path: 'error' },
      ],
    },
  },
}

Use status for the derived success or error outcome. Use error when you only need to test whether error details are present.

durationMs is the span's elapsed time in milliseconds. Span startedAt and endedAt conditions filter related spans independently of the top-level root timeRange:

const slowModelCalls = {
  spans: {
    some: {
      op: 'and',
      args: [
        { op: 'eq', left: { path: 'spanType' }, right: { literal: 'model_generation' } },
        { op: 'gt', left: { path: 'durationMs' }, right: { literal: 5000 } },
        {
          op: 'gte',
          left: { path: 'startedAt' },
          right: { literal: '2026-07-15T00:00:00.000Z' },
        },
      ],
    },
  },
}

model and provider read the canonical attributes.model and attributes.provider span attributes. Only stored string values are queryable; missing, null, and non-string values count as missing.

const selectedModel = {
  spans: {
    some: {
      op: 'and',
      args: [
        {
          op: 'eq',
          left: { path: 'model' },
          right: { literal: 'claude-sonnet-4-6' },
        },
        { op: 'eq', left: { path: 'provider' }, right: { literal: 'anthropic' } },
      ],
    },
  },
}

Use generic entity fields for tool and retrieval identity. The version fields let one clause qualify the same span by its entity lineage:

const versionedRetrieval = {
  spans: {
    some: {
      op: 'and',
      args: [
        { op: 'eq', left: { path: 'entityType' }, right: { literal: 'rag_ingestion' } },
        { op: 'eq', left: { path: 'entityId' }, right: { literal: 'medication-index' } },
        { op: 'eq', left: { path: 'entityVersionId' }, right: { literal: 'index-v3' } },
        { op: 'eq', left: { path: 'rootEntityVersionId' }, right: { literal: 'agent-v2' } },
      ],
    },
  },
}

Filter by score time

Score timestamps are independent of the required top-level trace timeRange. The top-level range selects trace candidates by trace start time. A nested timestamp predicate limits related score records:

const query = {
  timeRange: {
    from: '2026-08-01T00:00:00.000Z',
    to: '2026-08-08T00:00:00.000Z',
  },
  where: {
    scores: {
      some: {
        op: 'and',
        args: [
          { op: 'eq', left: { path: 'scoreSource' }, right: { literal: 'automated' } },
          {
            op: 'gte',
            left: { path: 'timestamp' },
            right: { literal: '2026-07-15T00:00:00.000Z' },
          },
          {
            op: 'lt',
            left: { path: 'timestamp' },
            right: { literal: '2026-08-01T00:00:00.000Z' },
          },
        ],
      },
    },
  },
}

Filter by span anchoring

Use spanId only to test whether a score targets a span. The predicate doesn't join the score to a matching span predicate:

// At least one score targets a span.
const anchoredToSpanWhere = { scores: { some: { op: 'exists', path: 'spanId' } } }

// At least one score applies to the trace rather than a span.
const traceLevelScoreWhere = { scores: { some: { op: 'notExists', path: 'spanId' } } }

The deprecated source field and scorerName aren't available as score predicate fields. Use the canonical scoreSource and scorerId fields.

Filter by metadata

Use metadata.<key> paths for custom dimensions stored on the current root span. Metadata predicates compare trimmed, non-empty string values exactly and case-sensitively. Leading and trailing whitespace in a stored string value is ignored. Values empty after trimming count as missing. Null, numeric, boolean, object, and array values also count as missing.

const messageTrace = {
  op: 'and',
  args: [
    { op: 'eq', left: { path: 'metadata.messageId' }, right: { literal: 'message-123' } },
    { op: 'in', value: { path: 'metadata.actorRole' }, set: ['assistant', 'tool'] },
    { op: 'exists', path: 'metadata.protocolVersion' },
    {
      op: 'or',
      args: [
        { op: 'eq', left: { path: 'metadata.temporalRunId' }, right: { literal: 'run-456' } },
        { op: 'eq', left: { path: 'metadata.externalTraceId' }, right: { literal: 'trace-789' } },
      ],
    },
    { op: 'not', arg: { op: 'exists', path: 'metadata.parentMessageId' } },
  ],
}

The key must name one top-level property. Empty keys and nested paths are rejected. Metadata keys aren't trimmed, so leading and trailing whitespace remains part of the exact key identity. When metadata duplicates a canonical trace field, such as resourceId, threadId, or environment, prefer the canonical field because it uses the dedicated storage column. The metadata.<key> form remains available when you specifically need the value from the metadata object. Metadata and canonical values may differ.

Metadata fields aren't available for grouping. Trace-query discovery returns executable top-level string metadata fields observed in the selected time range.

Filter by tags

Tags are a list of strings stored on the current root span, so they use collection operators instead of the scalar in and notIn operators. This query finds production traces tagged for manual review that don't carry the archived tag:

const manualReview = {
  op: 'and',
  args: [
    { op: 'eq', left: { path: 'environment' }, right: { literal: 'production' } },
    { op: 'includes', path: 'tags', value: 'manual-review' },
    { op: 'notIncludes', path: 'tags', value: 'archived' },
  ],
}

Operator	Matches when
includes	The trace has the requested tag
notIncludes	The trace has at least one tag, but not the requested tag
exists	The trace has at least one tag
notExists	The trace has no tags

Tags compare exactly and case-sensitively as whole strings. value must be one string with at least one non-whitespace character. Combine several includes predicates with and or or to require or allow multiple tags. A trace with no recorded tags and a trace with an empty tag list behave the same: both satisfy notExists, neither satisfies includes, notIncludes, or exists. Use { op: 'not', arg: { op: 'includes', ... } } when untagged traces should also match.

tags is available in trace predicates only, including those inside traces.some or traces.none. in, notIn, and the comparison operators are rejected for tags. Value discovery for tags returns each observed tag with the number of qualifying traces that carry it.

Filter by feedback

Every condition inside one feedback.some or feedback.none clause applies to the same current feedback record. feedbackType and feedbackSource are exact application-defined strings rather than built-in enums. This query finds traces with a numeric patient rating below zero:

const negativePatientRating = {
  feedback: {
    some: {
      op: 'and',
      args: [
        { op: 'eq', left: { path: 'feedbackType' }, right: { literal: 'rating' } },
        { op: 'eq', left: { path: 'feedbackSource' }, right: { literal: 'patient' } },
        { op: 'lt', left: { path: 'value' }, right: { literal: 0 } },
      ],
    },
  },
}

Strict stored-value types are the portable contract for feedback predicates. PostgreSQL and ClickHouse distinguish numeric 3 from textual '3' for equality and ordered comparisons. DuckDB currently persists feedback values as VARCHAR, so numeric-looking strings may be coerced for equality and ordered numeric predicates. OBS-306 will remove this DuckDB exception through typed persistence. Ordered operators require a finite numeric literal. eq and ne accept one string or number, while in and notIn require a non-empty set containing only strings or only numbers. exists and notExists test for either value type.

Use none to select traces without a matching record. Traces with no feedback also match:

const missingClinicianReview = {
  feedback: {
    none: {
      op: 'and',
      args: [
        { op: 'eq', left: { path: 'feedbackType' }, right: { literal: 'clinical-review' } },
        { op: 'eq', left: { path: 'feedbackSource' }, right: { literal: 'clinician' } },
      ],
    },
  },
}

Feedback timestamp predicates are independent of the root timeRange. Use comment only with exists or notExists. Comment contents aren't searchable. The deprecated feedback fields source and userId aren't available. Use feedbackSource and feedbackUserId.

Responses

An ungrouped query returns only lightweight completed traces:

{
  "traces": [
    {
      "traceId": "trace-123",
      "rootSpanId": "span-123",
      "threadId": "thread-123",
      "resourceId": "resource-123",
      "startedAt": "2026-08-03T10:00:00.000Z",
      "endedAt": "2026-08-03T10:00:01.000Z",
      "entityName": "support-agent",
      "entityType": "agent",
      "environment": "production",
      "status": "success"
    }
  ],
  "page": { "next": null }
}

A grouped trace query still returns distinct non-null thread IDs in ascending order. Grouping remains supported until the next major release:

const timeRange = {
  from: '2026-08-01T00:00:00.000Z',
  to: '2026-08-08T00:00:00.000Z',
}

// Deprecated
const legacyResult = await mastraClient.queryTraces({
  timeRange,
  group: { by: ['threadId'] },
})
// { groups: [{ threadId: 'thread-123' }], page: { next: null } }

// Replacement
const result = await mastraClient.queryTraceThreads({
  traces: { timeRange },
})
// { threads: [{ threadId: 'thread-123' }], page: { next: null } }

A thread query returns distinct non-null thread IDs in ordinal ascending order:

{
  "threads": [{ "threadId": "thread-123" }],
  "page": { "next": null }
}

Related evidence isn't embedded in either response. Use the trace-detail and branch APIs to load spans after selecting a result.

Pagination and errors

Choose a pagination mode

Trace queries support three mutually exclusive modes:

Mode	Request fields	Response metadata
Keyset, the default	page: { limit, after }	page: { next }
Numbered pages	pagination: { page, perPage }	pagination: { total, page, perPage, hasMore } and, when delta polling is supported, deltaCursor
Delta polling	mode: 'delta', optional top-level after and limit	delta: { limit, hasMore } and deltaCursor

Numbered pages are zero-based. Their total and rows are read consistently within each request. Don't combine page, pagination, or delta mode. Top-level after and limit are accepted only in delta mode. Thread queries and grouped compatibility queries remain keyset-only.

Poll after loading a numbered page

Use the numbered response's deltaCursor to replace the numbered-page-to-delta-polling workflow of listTracesLight(). Results use traces instead of spans and contain completed traces only. The configured store must support delta polling.

const timeRange = {
  from: '2026-08-01T00:00:00.000Z',
  to: '2026-08-08T00:00:00.000Z',
}
const initial = await mastraClient.queryTraces({
  timeRange,
  pagination: { page: 0, perPage: 100 },
})
if (!initial.deltaCursor) throw new Error('Delta polling is unavailable')

const traces = new Map(initial.traces.map(trace => [trace.traceId, trace]))
let after = initial.deltaCursor

async function poll() {
  let hasMore: boolean
  do {
    const result = await mastraClient.queryTraces({
      timeRange,
      mode: 'delta',
      after,
      limit: 100,
    })
    for (const trace of result.traces) traces.set(trace.traceId, trace)
    after = result.deltaCursor
    hasMore = result.delta.hasMore
  } while (hasMore)
}

await poll()

Retain after and call poll() again at your application's polling interval. A trace can appear in the initial page and a later batch, or in multiple batches, so merge results by traceId. To load historical traces omitted from the initial page, request the remaining numbered pages.

A delta request without after returns an empty array and establishes the current cursor. It doesn't return historical matches. Always retain the returned cursor, including for empty batches. hasMore means another matching trace exists beyond the batch limit. Continue immediately while it's true.

Delta ordering uses the storage ingestion watermark and adapter-specific tie-breakers, so matching timestamps don't make the cursor ambiguous. Don't pass orderBy in delta mode. Keyset and delta cursors aren't interchangeable, and delta cursors can't move between storage adapters.

Keep the same normalized where predicate and exact timeRange bounds for every poll. The range continues to select root startedAt, even when a trace completes later. Changing timeRange.to to the current time invalidates the cursor. Reload a numbered page and use its cursor whenever the selection or authorization scope changes. You can change the batch limit without restarting.

New completed roots and roots completing after the cursor can be returned. Related spans, scores, and feedback are evaluated when a root is selected, but later related-record writes don't guarantee that the trace is emitted again. Deleted traces and traces that stop matching aren't returned as removals or tombstones. Refresh numbered pages when you need to reconcile those changes.

ClickHouse polling is best effort. Its delta index and trace records use separate materialized-view tables. Concurrent queries can observe an insert in one table before another, as described in ClickHouse's materialized-view visibility rules. A poll can advance past an index entry before the matching trace becomes visible and miss that trace in later polls. Reload numbered pages periodically to reconcile these gaps.

ClickHouse's delta index retains two days of events and doesn't backfill historical rows. Reload numbered pages after a polling interruption longer than this retention window. Delta polling isn't a durable change feed and doesn't guarantee delivery of every matching trace.

Keyset ordering and errors

Ordering is deterministic. Trace ordering appends traceId ascending as a tie-breaker. Thread queries always use raw ordinal threadId ascending order. Callers can't override it.

Keyset cursors are bound to the operation, accepted normalized query, and ordering. Reusing a keyset cursor after changing the trace selection, predicates, or ordering returns 409. Trace and thread cursors aren't interchangeable.

Numbered-page handoff cursors and delta cursors also bind the authorization state. Changes to the caller's roles or permissions invalidate these cursors and return 409. If a delta poll returns 409, reload the numbered pages and resume polling with the new deltaCursor. A malformed cursor returns 400.

Trusted tenant scope

Hosts that serve more than one tenant supply a trusted scope outside the query document. The server reads the reserved organizationId request-context key, which only server-side authentication can set, and passes { organizationId } to the planner. The scope is carried on the trusted plan and ANDed into every scan: current roots, related spans, scores, feedback, and both discovery routes. A related record from another tenant that shares a traceId never qualifies a trace.

Hosts that call storage directly can pass a fuller scope to planTraceQuery(), planThreadQuery(), planTraceQueryObservedFields(), and planTraceQueryValues():

const plan = planTraceQuery(parseTraceQueryRequest(request), {
  scope: { organizationId: 'org_123', resourceId: 'project_456' },
})

resourceId is optional and narrows within the organization. Keyset and delta cursors are bound to the scope, so a cursor minted under one scope and reused under another returns 409 before storage runs. A scoped request against an observability store or @mastra/core that predates tenant scope returns 501 instead of running unscoped. Stores advertise support through the trace-query-tenant-scope feature. Callers can't name organizationId or projectId in predicates. Without a scope the routes behave as before, so self-hosted installations keep their own tenant model.

Cursor pagination is deterministic, but it isn't a database snapshot. Traces or replacement signals written between page requests can change later pages.

PostgreSQL and ClickHouse stop advanced trace and thread queries after 15 seconds by default and return 504 when the database timeout is exceeded. Set traceQueryTimeoutMs in the store's vNext observability configuration to an integer from 1 through 300,000 milliseconds to change the timeout. DuckDB doesn't currently provide query-scoped timeout or cancellation through its driver wrapper, so this 504 guarantee doesn't apply to DuckDB.

Status	Meaning
400	Malformed JSON or malformed cursor.
409	The cursor doesn't match the query.
413	The request body exceeds 256 KiB.
422	The JSON is well formed, but the request is structurally or semantically invalid. The response includes stable issue codes and paths. Discovery validation uses TRACE_QUERY_INVALID.
501	The installed Core or configured observability store doesn't support the requested operation. Discovery returns TRACE_QUERY_DISCOVERY_UNSUPPORTED unless Core provides the discovery contract and the store implements both field and value discovery.
503	Discovery exceeded a backend memory or resource budget and returned TRACE_QUERY_RESOURCE_LIMIT.
504	A PostgreSQL or ClickHouse query exceeded its configured database execution timeout. Discovery returns TRACE_QUERY_EXECUTION_TIMEOUT.

Limitations

Both operations consider completed traces only. They don't support running traces, custom projections, embedded evidence, summaries, aggregations, measures, or custom grouping. Numbered trace pages include a total count.

Related

• Client SDK observability reference
• Tracing overview
• Span interfaces