Prompt caching: how it works and how to cut LLM token costs

You send a request with a 2,000-token system prompt, and the model reprocesses every token. Send the next turn, and it reprocesses them again. Twenty turns in, you have paid to compute the same static instructions twenty times over.

Prompt caching fixes this by storing the model’s computed state for the parts of your prompt that do not change, so those tokens are reused across requests instead of recomputed. The result, according to ngrok’s technical breakdown, is input tokens that can be roughly 10x cheaper and responses that return faster.

This guide explains what prompt caching is, how it works under the hood, how major providers implement it, and how you can verify your cache is actually being hit.

What is prompt caching?

Prompt caching is a technique that stores the model’s processed representation of repeated prompt content so it does not have to reprocess those tokens on every request. You mark the stable parts of your system prompt, and the provider reuses their computed state on subsequent calls that share the same prefix.

The two problems it solves are cost and latency. Every token you send costs money to process, and long system prompts take time to work through before the first output token appears. When your instructions, function schemas, or document context stay constant across calls, reprocessing them is pure waste. Caching removes that waste.

It is worth separating this from semantic caching, which stores full prompt-response pairs and returns a saved answer for similar queries. Prompt caching does not reuse outputs. You still get a fresh, potentially different response each time, because only the intermediate computation is cached, not the final result.

Semantic caching targets identical or near-identical queries. Prompt caching targets the shared prompt prefix regardless of the new question you append.

How prompt caching differs from conventional caching

You already know conventional caching from web and database work, where a stored HTML page, query result, or image is returned verbatim when the same key is requested. That model is deterministic. The same key always returns the exact same stored value, and the point is to skip fetching or recomputing an identical result.

Prompt caching behaves differently in one crucial way. It does not return a stored answer. It reuses the internal computation for a shared prompt prefix, then the model still generates a new response token by token. Send the same cached prompt a dozen times and you can get a dozen different completions, even while the usage data confirms cached tokens.

The scope and invalidation logic also differ. Conventional caches expire through explicit TTL, versioning, or manual invalidation tied to your data changing. Prompt cache entries live on the provider side for a short window and expire automatically.

A single character change anywhere in the cached prefix invalidates the match, so exact prefix matches matter more here than in most caching you have worked with.

How prompt caching works

You get the clearest picture of prompt caching once you know what actually gets stored, and it is not your text or your response. To understand it, you need a quick look at how a large language model processes a prompt through its attention layers, because that is exactly where caching happens.

Inside the model, your tokens are converted into embeddings, then passed through transformer layers where the attention mechanism computes how much each token should influence the next. That computation produces two matrices per token, conventionally called the key and value matrices.

These are the numbers that represent your prompt’s processed state, and they are what providers store. This is why prompt caching is often called KV cache or KV caching.

Prefix matching and cache breakpoints

You benefit from caching only when a new request starts with the exact same sequence of tokens as a cached one. The cache works on prompt prefixes, so the model reuses stored key and value matrices for the leading tokens that match, then computes only the new tokens that follow.

A cache breakpoint marks where the stable prefix ends and the variable content begins. Everything before the breakpoint is a candidate for reuse. Everything after it is computed fresh. Placing explicit cache breakpoints correctly is the single biggest lever you have over your cache hit rates, which is why prompt structure gets its own section below.

What gets cached and what doesn’t

Your cacheable content is whatever sits at the front of the context and stays identical across calls. Static content is the target: long system prompts, tool definitions, few-shot examples, and large reference documents. Conversation history can be cached up to the most recent exchange.

Variable content is never cached. The new user message, injected timestamps, session IDs, and any per-request data change every call and must live after the breakpoint. If dynamic values leak into the cached prefix, the prefix stops matching and no cache hit is possible.

Cache lifetime, TTL, and the rolling window

Your cache entries have a short lifespan. Providers hold the stored key and value matrices for a short TTL window, commonly around five to ten minutes since the last matching request, and then evict them. The time-to-live refreshes each time you hit the entry, so an active conversation keeps its cache warm.

Once the window lapses, the next request recomputes and rewrites the cache from scratch. This rolling behavior is why sporadic traffic gets fewer cache hits than steady traffic. If your requests arrive far apart, you pay the cache write cost repeatedly and rarely collect the read savings.

Implementing prompt caching across LLM providers

Your implementation depends heavily on which provider you use, because they split into two camps. Some cache automatically with no code changes, while others require you to mark cache points explicitly and give you tighter control in return. The table below compares how the main providers expose caching before we walk through each one.

Anthropic Claude cache control

You control caching directly with Anthropic Claude. The API uses cache_control annotations, specifically "cache_control": {"type": "ephemeral"}, placed on the message blocks you want stored. Without those markers, even a perfectly static system prompt is reprocessed every call because the client never asked for it to be cached.

Here is a minimal example of how you annotate a message block for Anthropic Claude:

{
  "model": "claude-sonnet-4-5-20250929",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "You are a coding assistant. Follow the style guide below...",
      "cache_control": {"type": "ephemeral"}
    }
  ],
  "messages": [
    {"role": "user", "content": "Refactor this function to use async/await."}
  ]
}

Anthropic lets you set up to four breakpoints per request, so you can cache your system prompt, tool definitions, and a large document context independently. This spans the Claude family, including Claude Haiku 4.5, Claude Sonnet 4.5, and Claude Opus 4.5.

The tradeoff is that explicit control costs you a small write premium, but in practice you get reliable, near-total cache hits when you request them. Anthropic's own documentation covers the full set of TTL options and breakpoint rules in more depth.

OpenAI automatic caching

You get caching for free with OpenAI. It applies automatically to prompts above a token threshold, commonly 1,024 tokens, routing repeated prompt prefixes to cached entries without any code changes. There is nothing to annotate, which makes adoption effortless.

The tradeoff is control. Because routing is automatic and best-effort, hit rates can be inconsistent, particularly for long context windows where time-to-first-token varies. You can pass an optional prompt_cache_key to steer requests toward the same cache pool, which improves consistency when many users share a common system prompt. OpenAI's prompt caching guide details the exact breakpoint behavior and how routing keys affect hit rates.

Google Gemini context caching

You have two options with Google Gemini. Implicit caching works automatically for repeated prefixes, similar to OpenAI’s approach. Explicit context caching lets you create a named cached content object from a large document or instruction set, then reference it across many requests.

The explicit path suits workloads where you query the same large context repeatedly, such as a long PDF or a codebase. You pay to create the cached content, then reference it cheaply. This makes Gemini’s context caching well suited to document-heavy applications with predictable reuse. Google's context caching documentation lays out the minimum token counts and TTL defaults per model.

Amazon Bedrock

You can also access prompt caching through Amazon Bedrock, which relays caching capabilities from supported foundation models. If you run Anthropic Claude models via Bedrock, the same caching annotations apply. Amazon Bedrock adds its own layer of access management and data residency controls, which matters if your organization restricts where cached prompt data is stored geographically.

Structuring prompts for maximum cache hits

You get the most from any provider by ordering your system prompt from most stable to most variable. Put the unchanging content at the front and the dynamic content at the end, so the longest possible prefix stays identical across calls.

The table below summarizes what belongs in the cached prefix and what must stay outside it.

Follow these ordering rules to keep your prefix matching intact:

• Place static content first: your system prompt, tool definitions, and few-shot examples belong at the very top of the context.

• Keep instructions byte-for-byte identical: a single changed character breaks the cache hit for everything after it.

• Move dynamic values out of the prefix: timestamps, user names, and session IDs go into the user turn, never the cached instruction block.

• Set explicit cache breakpoints at block boundaries: mark the end of a static block, not the middle of a sentence in variable content, so the provider has a clean dividing line for reuse.

• Preserve conversation structure: reformatting or compressing history between turns changes the token sequence and misses the cache.

How prompt caching affects cost and rate limits

You care about caching mostly because of what it does to your bill and your throughput. The economics are straightforward once you separate the one-time write cost from the repeated read savings, and the effect on rate limits is larger than most teams expect.

Cached vs uncached token pricing

Your cached tokens are dramatically cheaper to read than standard input tokens. Across most Claude models, cache read tokens are priced at roughly 10% of the normal input token cost, according to Anthropic’s pricing.

Writing to the cache costs slightly more than a standard input token, but you pay that once. The net effect is a sharp reduction in input token costs for any prefix you reuse more than twice.

Providers surface this split in the response. You will see cache write tokens billed at a small premium on the first call and cache read tokens billed at the deep discount on every hit after. The break-even point arrives fast, usually within the second or third reuse of a prefix.

A practical cost example

Say you are running a specialized assistant with a 2,000-token system prompt and a 20-turn multi-turn conversation. Without caching, every turn reprocesses all 2,000 tokens plus a growing history, so by turn 20 you have paid full input rates for that system prompt twenty times.

With caching working, turn one writes the instructions to cache, and turns 2 through 20 read them at roughly a tenth of the price. You pay near-full cost for one turn’s worth of processing, then lightweight incremental costs after. The same conversation, same output, at a fraction of the compute.

Impact on subscription and usage limits

Your rate limits are often tied to compute, not raw message count. As the MindStudio breakdown of Claude limits explains, a session with a large system prompt and long history consumes far more of your budget per turn than a short exchange.

Because caching cuts the per-turn compute cost, efficient caching directly stretches how many turns you get before hitting a wall. When caching silently breaks, every message is processed as if it were the first in the conversation, and your limits drain far faster than your actual usage would suggest. Mastra's guide to AI agent architecture goes deeper into the design choices, like context compaction and model routing, that keep costs predictable at scale.

Use cases for prompt caching

You will see the biggest gains from caching wherever the same large block of tokens rides along on request after request. Three patterns dominate, and they cover most production agent and chatbot workloads.

Long instructions and static context

Your detailed instructions are the classic caching win. A coding assistant with an extensive style guide, or a support bot with a large policy document baked into its system prompt, sends that same block on every single turn. Cache it once and every following request reads it cheaply. The longer and more static the instruction set, the more you save.

Multi-turn conversations and agents

Your multi-turn conversations accumulate history that gets replayed in full on each call. Caching the stable prefix means each new turn only pays full rate for the incremental message. This matters most for ai agents, where a single user request can trigger several model calls that all share the same instructions and prior context. Mastra's guide to agent memory covers how to structure that shared context so it stays both cache-friendly and useful across turns.

RAG and large document context

Your RAG and document workflows attach big chunks of retrieved context to the prompt. When you ask multiple questions against the same document or knowledge base within the cache window, caching that context avoids reprocessing thousands of tokens per query. If your retrieval pipeline keeps retrieved context stable across calls, it remains cacheable within the TTL window.

Prompt caching in Mastra

You build agents in TypeScript with Mastra, an open-source framework that gives you agents, workflows, memory, and observability in one place. Its model router reaches 90+ providers through a single interface, so you configure Claude, OpenAI, or Gemini without rewriting your caching logic per vendor.

Where caching matters most is in observability. Because Mastra records every model call as a span with inputs, outputs, latency, and token usage, you can confirm cache write tokens and read tokens at each step instead of reading a flat total. That makes it straightforward to catch a broken prefix the moment your cached input tokens drop to zero.

Build your first traced TypeScript agent with Mastra.

Verifying and monitoring cache performance

You should never assume caching is working, because it fails silently. A misplaced timestamp or a reformatted history block produces no error, just a quiet return to full pricing. The only way to know is to read the token metadata and watch your cache hit rates over time.

Reading cache read and write tokens in API responses

You confirm a cache hit by inspecting the usage block in the API response. Anthropic reports cache_creation_input_tokens for writes and cache_read_input_tokens for reads. OpenAI exposes a cached_tokens field in its usage details.

Here is what a healthy Anthropic response usage block looks like when your prompt prefix is cached:

{
  "usage": {
    "input_tokens": 47,
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 2048,
    "output_tokens": 312
  }
}

The signal is simple. If cache_read_input_tokens stays at zero across a multi-turn conversation, caching is not happening even if you expected it to. A healthy session shows a write on the first call and reads on every call after that shares the prefix.

Common reasons caching silently breaks

Your cache breaks in a handful of predictable ways, and none of them produce cache misses with an error message. The table below maps each cause to its effect so you can diagnose issues quickly.

Tracing cache behavior in agent frameworks

You lose the clean single-response usage block once an agent makes multiple model calls per request, which is where structured tracing earns its keep. A tracing framework captures each model call as a span with its own token counts, so you can see cache write tokens and read tokens step by step across a full agent run.

Prompt caching trade-offs and points to consider

You should weigh a few real limitations before leaning on caching everywhere. It is not free, and it does not fit every workload cleanly.

The write premium means short-lived or rarely-repeated prompt prefixes can cost more than they save. If your requests arrive further apart than the TTL window, you pay to write the cache repeatedly and almost never collect the read discount. Caching pays off with steady traffic against stable content, not sporadic one-off API calls.

There is also operational complexity. Getting maximum cache hits requires careful prompt ordering, correct breakpoint placement, and discipline about keeping cached content byte-for-byte identical. Partial-reuse edge cases add design effort, and a subtle change upstream can quietly cut your hit rate without any visible failure. Monitoring is not optional if caching is load-bearing in your cost model.

You should also consider data residency. Some providers cache prompt prefixes on shared infrastructure, which may conflict with compliance requirements that restrict where your data is stored geographically. Check your provider’s documentation for region and tenancy guarantees before enabling caching on sensitive workloads.

Wrapping up

Prompt caching comes down to two habits: order your prompt from stable to variable, and verify hits by reading token metadata rather than trusting that it works. Start with your longest static instruction block, confirm the read tokens, then expand from there. If your agents run on TypeScript, Mastra surfaces cache read and write tokens per span so you can verify hits across every model call.