What is LLM observability? A span-by-span breakdown

There are two ways to know whether your agent is working.

The naive way is to look at the response. If the agent responded and the response reads well, we assume the agent is working. This check covers crashes and empty responses, but that's the extent of it. The agent could have hallucinated, answered a different question than the one asked, or skipped its tools and made something up, and the response would read just fine.

A better way is with LLM observability, where you look at the question asked by the user, the agent's response, and the steps the agent took to reach that solution. Those steps include which tools ran and with what arguments, what they returned, whether the model looked the answer up or invented it, how long each step took, and what it cost. That tracking is what I'll talk about in this article.

By the end of this piece you'll know:

• what LLM observability means technically, and why the monitoring you already run isn't enough
• what the trace of an agent run contains, read span by span from a live run's storage
• how frameworks create spans, and what a fabricated answer or a failed tool call looks like in a trace
• where the OpenTelemetry GenAI conventions stand as of mid-2026
• how evaluation scores attach to traces, and the four dials that control tracing cost in production

What is LLM observability?

LLM observability is the practice of recording what an LLM did, at the level of individual model calls, tool executions, and the data that flowed between them. It helps explain any single output and measure quality across thousands of outputs you otherwise would never read.

It extends classic observability (logs, metrics, traces) by adding signals that only exist once a model is in the loop: token usage, cost, tool decisions, and evaluation scores that stand in for the error codes models don't produce.

Why not use the existing monitoring solutions for LLMs and AI agents?

Existing monitoring was built for deterministic systems, where the same input produces the same output every time. When one of those systems broke, you had a crash or some error the system could alert on.

An agent behaves differently on both counts, and that's where you face some new challenges these traditional telemetry systems were never built to handle.

Wrong answers do not produce errors

A wrong answer from an LLM still comes back with a 200, because the body looks normal and the response arrives within the usual time. Your telemetry has no way to tell a hallucinated answer apart from one that was grounded in your own documentation.

That is because monitoring was never looking at the answer itself. It only checks whether the response was delivered and how long it took, and neither of those changes when the content turns out to be wrong.

You cannot reproduce a failure

The second problem is that you cannot reproduce a bad answer, because the same prompt can return something different the next time you send it. The debugging habit most of us rely on with traditional software depends on the failure being repeatable, and an LLM failure usually is not.

Charity Majors made this point back in 2023 when she wrote that LLMs "cannot be debugged or tested using traditional software engineering techniques".

That leaves you one option, to record what the model did in production and study that recording after it happened.

One request is many operations

The third problem is that a single request to your agent is not a single operation but a whole loop. The model reads the conversation, decides to call a tool, gets the result back, and then runs again on the enlarged context, and it keeps going like that until it stops asking for tools.

Because the work is spread across that loop, one number measured over the whole request cannot tell you much. A total latency figure or a final status code says nothing about which pass through the loop was slow, or which one produced the wrong tool call.

Traditional signals do not map cleanly

The signals you already watch have equivalents in an agent, but I'd not call the equivalents the same as the traditional signals:

Because an agent still has CPUs and requires memory and can throw 500s when it breaks, you need the existing monitoring. LLM observability is an added layer on top of that existing stack.

Most teams have already made that move. When LangChain surveyed 1,340 teams in late 2025, 89% were running some form of agent observability and 71% had detailed tracing, and the same survey named quality as the top barrier to shipping AI agents.

Source: LangChain, State of Agent Engineering 2025

What does the trace of an agent run look like?

Tracing is a tree of spans, where each span records one operation, and the structure itself is the same one distributed tracing has used for years.

What is new for LLMs is what the spans hold, because the operations being recorded are model calls and tool calls instead of HTTP handlers and database queries.

The shape of a trace

Every agent trace has roughly the same shape, whatever framework produced it.

The root span sits at the top, covering the whole request.

Inside it is one span for each time the model was called, and one span for each tool the model ran. A vector search gets its own span, a memory read or write gets one too, and when a tool calls another agent, that agent's spans nest inside the tool's span. The tree always mirrors what called what.

Each span also records what happened during its own operation. A model-call span keeps the model name, the settings it ran with, the messages going in and out, the input and output token counts, why the model stopped, and how long it took.

A tool-call span keeps the arguments the model passed in, what the tool returned, and whether it worked.

We want this recorded per operation because the failures and the cost are spread across the loop. So just looking at one number for the entire operation cannot explain if it was the tool call or the inference or memory that broke.

Let me show you what it looks like in practice, with a small Mastra agent I ran against a local model through Ollama.

Building a simple demo setup

To have something concrete to point at, I built a small support agent with a single tool called lookupOrder, which stands in for the kind of internal orders API a support agent would really call. It runs on a local model, qwen3.5:2b through Ollama, so the timings you are about to see come from my laptop's CPU rather than a hosted GPU, and I turned on tracing in Mastra pointed at a local SQLite file.

Note: qwen3.5:2b is a reasoning model, so a few of the spans below are the model thinking before it acts.

Here's the tiny tracing config:

export const mastra = new Mastra({
  agents: { supportAgent },
  storage: new LibSQLStore({ id: 'demo-storage', url: 'file:./mastra.db' }),
  observability: new Observability({
    configs: {
      default: {
        serviceName: 'support-app',
        exporters: [new MastraStorageExporter()],
        spanOutputProcessors: [new SensitiveDataFilter()],
      },
    },
  }),
});

The span tree of one request

Here is the tree, queried straight out of the mastra_ai_spans table (the trace ID supplied here is a unique value Mastra assigns to the run).

You can now connect this tree with the diagram I showed earlier:

• The root agent_run span covers the entire run.
• The model_generation beneath it covers the whole conversation with the model, which happened in the two steps indented under it.
• In step 0 the model read the question and asked for lookupOrder, the tool ran and returned its result.
• In step 1 the model used that result to write the answer.
• The model_chunk entries under each inference are the pieces emitted while the response streams, where a chunk: 'reasoning' is the model thinking before it acts.

Now there are two details in this tree you need to look at.

The first is that the tool call sits beside model_inference under the step rather than inside it, and that placement is because it lets the inference span measure model time on its own without the tool mixed in.

You can see why that helps here, since my agent spent 20,504ms of its 20,571ms inside the two inference spans and only 2ms inside the tool. So a slow run like this one points straight at the model (or maybe the fact that the model was generating tokens on CPU).

The second detail is that every span has a type drawn from a fixed list of about two dozen span types, and the type decides which attributes the span is allowed to carry, which TypeScript then enforces at compile time.

The model generation span

We also have the model_generation span which carries the following data:

{
  "model": "qwen3.5:2b",
  "provider": "ollama.chat",
  "streaming": true,
  "parameters": {},
  "finishReason": "stop",
  "responseId": "chatcmpl-472",
  "responseModel": "qwen3.5:2b",
  "completionStartTime": "2026-07-08T17:11:15.886Z",
  "usage": {
    "inputTokens": 769,
    "outputTokens": 129,
    "inputDetails": { "cacheRead": 0, "text": 769 },
    "outputDetails": { "reasoning": 0, "text": 129 }
  }
}

In this one, I usually look at finishReason and completionStartTime.

finishReason records why the model stopped and it is the first field I check when an answer comes back cut off, because a length limit and a content filter leave different values here and the field tells you which one you hit.

completionStartTime is there so you can compute time-to-first-token against the span's start, and that number matters because it is the latency a user feels while the response streams in, separate from how long the full generation took.

In this case, the reasoning bucket is shown as 0 even though you saw the model reasoning in the chunks above. That's because Ollama does not report reasoning tokens separately, but a hosted reasoning model would fill that field in and show you how much of the spend went to thinking rather than answering.

The tool call span

The tool span records every tool call that was made for every response:

{
  "toolDescription": "Look up the status of a customer order by its order id",
  "toolType": "tool",
  "success": true
}

The input holds the arguments the model chose and the output holds what the tool returned, which means the model's decision and the tool's behavior are now two separate facts you can inspect on their own, instead of being blended together in a single log line.

Where do spans come from?

The framework running your agent creates spans at execution boundaries, and these boundaries decide what you can and cannot see in your telemetry data.

Instrumentation points

Mastra has three creation points.

• The agent's execution path opens the agent_run span when a run begins.
• The model loop opens a model_generation span for each generation, and a tracker it spins up opens the model_step, model_inference, and model_chunk spans beneath it as the response streams.
• The tool builder opens a tool_call span for each tool the model invokes.

The tool builder opens its span before input validation runs, so a call that fails validation still appears in the trace. That ordering looks minor until the day a model starts passing malformed arguments, and the trace shows it happening instead of swallowing it.

Context propagation

Then there is context propagation, the plumbing that makes the span tree a tree.

When a tool runs, we need a way to remember that the tool span is the parent for whatever happens inside it, and that gets tricky when the tool calls another agent.

Mastra handles this by giving the tool a context that carries its span, and by wrapping the mastra instance the tool receives in a proxy. Errors follow the same path as context, which means the failure is recorded exactly where it happened.

Ways to capture telemetry

Everything above is framework-native instrumentation, where the framework running the loop opens the spans itself because it already knows where every boundary is, and it is one of three common approaches.

Auto-instrumentation libraries like OpenLLMetry and OpenLIT patch the provider SDKs instead, so they catch every model call in the process regardless of framework, but they cannot see your own logic between those calls, which means the agent's structure has to be inferred or annotated by hand. Gateway tools like Helicone go further out and put a proxy in front of the provider API by swapping the base URL, which needs no code changes and works from any language, but only sees the HTTP traffic and misses everything on your side of the wire.

The tradeoff runs the same direction every time. Instrumentation that sits closer to your application logic can tell more of the story, and in return it ties you more tightly to whatever is doing the instrumenting.

What does a failure look like in a trace?

The clearest failure is a span that ran but came back marked as failed, with the error attached to it. The other is a run where no span fails but time and tokens get pooled somewhere they should not.

The failed tool call below walks through the first, and the section after it covers the second.

A failed tool call

The clearest failure to read in a trace is a tool that threw an error. I ran the same order-status request again while deliberately making the orders API unreachable:

{
  "toolDescription": "Look up the status of a customer order by its order id",
  "toolType": "tool",
  "success": false
}

This time the tool span came back with success: false, and its error info carried the cause, "orders-api: connection refused", with the stack included.

The trace makes the cause of failure unambiguous, letting us know the infrastructure underneath the model broke. That distinction is what LLM observability gets you.

Loops, latency, and cost spikes

The remaining failure shapes are similar to the ones above. An agent stuck in a loop shows up as the same tool call repeating across steps, and a latency regression shows you which inference span grew or whether the delay moved into a tool.

A cost spike is usually the per-step inputTokens climbing as history piles up, which even in my toy run I could observe as the input grew from 335 to 434 in a single iteration and would keep growing on every step of a longer run.

Is there a standard for LLM telemetry?

The space is still young but there is some agreement on what we need to track.

OpenTelemetry publishes a set of GenAI semantic conventions that standardize the names of things, so any compliant instrumentation and any compliant backend can meet on the same vocabulary.

Under those conventions a model call becomes a span whose gen_ai.operation.name is chat, carrying gen_ai.provider.name, gen_ai.request.model, and the token counts, while agent and tool operations get their own names, invoke_agent and execute_tool, alongside conventions for embeddings, memory, and MCP calls.

I am pro standardized naming convention because a shared vocabulary is what lets you change backends later without re-instrumenting your code.

Mapping framework spans to the standard

Mastra keeps its spans as the typed records you saw earlier, and its OTel exporter maps them to the conventions.

function getOperationName(span: AnyExportedSpan): string {
  switch (span.type) {
    case SpanType.MODEL_GENERATION:
      return 'chat';
    case SpanType.RAG_EMBEDDING:
      return 'embeddings';
    case SpanType.TOOL_CALL:
    case SpanType.MCP_TOOL_CALL:
      return 'execute_tool';
    case SpanType.AGENT_RUN:
      return 'invoke_agent';
    case SpanType.WORKFLOW_RUN:
      return 'invoke_workflow';
    default:
      return span.type.toLowerCase();
  }
}

That single function is all it takes to stay out of lock-in, because your internal span model stays stable while only the export layer tracks the standard. So when the standard renames something again, the change lands in one exporter instead of across your whole application.

It is also why the choice of backend has stopped being a big commitment, since the same spans flow into Langfuse, LangSmith, Datadog, or a plain OTel collector, and moving between them is a config change rather than a re-instrumentation.

What does LLM observability cost in production?

LLM observability through tracing costs you mostly in data. You pay to capture the payloads, to store them, and for whatever your backend charges to ingest them.

Capturing full inputs and outputs is the right default while you build, but in production it becomes a call you have to make, because those payloads carry whatever a user typed into the prompt.

The OTel conventions clearly mention that message content attributes need to be opt-in and instrumentations "SHOULD NOT" capture them by default.

From here, you have a few dials that can control your cost.

1. Redaction

Prompts and tool outputs are generally rich with secrets because they carry whatever the user typed and whatever an internal API sent back. So a production setup has to redact them before anything is stored.

In my demo tool, I deliberately returned an apiKey field next to the order status, the way internal APIs often leak session tokens into their responses. Here is the tool span's output as it was stored:

{"orderId":"88231","status":"shipped","carrier":"DHL","eta":"2026-07-09","apiKey":"[REDACTED]"}

The SensitiveDataFilter in my config caught that field before it left the process. Its default list covers password, token, secret, apikey, jwt, credential, ssn, and a few variants, and it normalizes keys before matching so api-key and apiKey both hit. It also reaches into nested objects and into JSON stored as a string.

With Mastra, the filter runs as the last step, after any processors you add yourself. So a step that maybe copies header data into a span cannot leak a secret past the filter.

2. Truncation

The same payloads are large as well as sensitive, so tracers bound the values before they store them.

In Mastra the defaults cap each string at 128KB, arrays at 50 items, objects at 50 keys, and nesting at depth 8, all of them configurable.

Without them a single run with a large context can write megabytes of span data, most of it the same conversation history serialized once per step, which is the 335-to-434 jump from earlier, turning up in your storage bill as well as your token count.

3. Sampling

You almost never need every trace, so you pick a sampling strategy. The options are the same across tracers: always, never, a fixed ratio like one request in ten, or a custom function that decides per request. The custom option is how you trace all of a new feature while keeping only a fraction of the steady traffic.

The decision happens once, at the very start of the request. If a request is not sampled, its spans are built as empty placeholders that record and export nothing, so skipping a trace costs almost nothing rather than building every span in full and discarding it at the end.

4. Span volume

Some backends charge per span, so you want to decide what to keep.

Mastra lets you exclude whole span types through excludeSpanTypes, with one useful detail, which is that an excluded model span still rolls its token usage up onto the nearest span you kept, so your cost accounting survives the cut.

Cost itself is derived, not measured, since a token count only becomes dollars against a price list. Mastra estimates it from a pricing table bundled with the package so no network call is involved, and wherever your cost numbers come from, that source is what decides whether the cost column on a dashboard can be trusted.

5. Storage backends

One more thing showed up when my run exited. The console warned that my SQLite backend "does not support batch creating metrics".

That happens because traces, metrics, and logs each want a different kind of store. Spans fit a plain relational table, while metrics like latency percentiles need an analytical store built for aggregation. That is why one database rarely serves all three well, and why production setups usually pair a relational store for traces with an analytical one for metrics.

Wrapping up

Underneath the vendors, LLM observability comes down to a simple idea, that an agent's work has to leave a record more detailed than its answer. The answer on its own cannot tell you whether the work was done right. That record is a tree of typed spans with quality scores, and a young standard is settling the names.

The best part is, there's not much configuration required to set up observability now. It's usually just a couple of lines of code at the top of your agent with the shape defined so you have standardized logs. That's it.

P.S. Everything here ran on Mastra's built-in observability, and the observability docs cover the pieces this article used: tracing, exporters, scorers, and the OTel integration.