// cost_engineering
Prompt caching, explained
Most production prompts resend the same few thousand tokens on every call — instructions, tool schemas, reference documents. Prompt caching bills that repeated part at a steep discount. Done right it is the cheapest optimization you will ever ship; done carelessly it quietly costs more than not caching at all.
The short version
A provider can keep the processed form of a prompt's opening tokens and reuse it when the next request starts identically. Reusing it — a cache read — is billed at a fraction of the input price. Storing it — a cache write — is billed at the input price or above.
So caching pays when a large, identical prefix is reused often. It never touches output tokens or the part of the prompt that changes, and on models that charge a write premium it loses money below a break-even hit rate.
What is actually being cached
To generate a response, a model first processes every input token and builds attention state — the key/value (KV) tensors for each token in the context. For a long prompt this prefill step is a large share of the compute, and it is identical every time the same tokens appear in the same order.
Prompt caching keeps that state for a recent prefix. When a new request begins with the same tokens, the provider loads the stored state instead of recomputing it, then processes only what comes after. Because attention runs left to right, the match is strictly a prefix: one changed token and everything after it is recomputed at the full price.
Cached state is held briefly — typically minutes of inactivity, sometimes longer for a fee — then evicted. That is why hit rate depends on traffic shape as much as on prompt design.
Three ways providers expose it
The mechanics differ by provider and change over time, so treat this as the shape of the options and check the linked docs for current details.
| Approach | How you opt in | What you pay | Watch for |
|---|---|---|---|
| Explicit cache breakpoints | Mark where the cacheable prefix ends in the request (Anthropic works this way). | A write premium above normal input on a miss; a deep discount on a hit. | Low hit rates lose money. Choose the expiry that matches your traffic. |
| Automatic prefix caching | Nothing — long enough repeated prefixes are cached for you (OpenAI and DeepSeek work this way). | No write premium; cached tokens billed at a discount. | You control hits only through prompt order. There is a minimum prefix length. |
| Explicit cache objects | Create a named cache of content, then reference it (Google Gemini offers this alongside implicit caching). | A discounted rate per use, plus storage for as long as the cache is held. | Storage accrues while the cache sits idle — delete what you stop using. |
What caching costs on today’s strongest models
Live from the catalogue, for the 8 highest-scoring models that publish a cached-input price. Savings use an assistant workload: 4,000-token prefix, 600 changing tokens, 400 output tokens, 85% hits.
| Model | Input / 1M | Cached read | Cache write | Break-even | Saving |
|---|---|---|---|---|---|
| Claude Fable 5.1 | $10.00 | $0.250 (3% of input) | $12.50 (+25%) | 20% hits | 48% |
| Claude Opus 5.5 | $4.00 | $0.200 (5% of input) | $5.00 (+25%) | 21% hits | 47% |
| Claude Fable 5 | $10.00 | $1.00 (10% of input) | $12.50 (+25%) | 22% hits | 44% |
| GPT-6 Astra | $10.00 | $1.00 (10% of input) | $12.50 (+25%) | 22% hits | 44% |
| Muse Spark 1.3 | $1.25 | $0.150 (12% of input) | input rate | any hit rate | 50% |
| DeepSeek V4.1 Flash | $0.100 | $0.010 (10% of input) | input rate | any hit rate | 46% |
| GPT-5.6 Sol | $2.00 | $0.200 (10% of input) | $2.50 (+25%) | 22% hits | 44% |
| GPT-5.5 | $5.00 | $0.500 (10% of input) | input rate | any hit rate | 44% |
Put in your own prompt sizes and hit rate in the prompt caching calculator, which covers every model with published cache pricing.
Structuring a prompt for cache hits
- 01
Order from most stable to least stable
System instructions first, then tool definitions, then shared reference material, then conversation history, and the new user message last. Anything that changes per request belongs after everything that doesn’t. - 02
Make the prefix byte-for-byte identical
Serialize tool schemas and documents deterministically — stable key order, stable whitespace. A JSON object that reorders its keys between requests is a cache miss that looks like a cache hit in code review. - 03
Move variables out of the system prompt
Today’s date, the user’s name, a request ID or a feature flag near the top invalidates everything after it. Pass them in the final message instead. - 04
Append to conversation history, never rewrite it
Multi-turn chats and agent loops cache well because each turn extends the previous prompt. Summarizing or trimming old turns rewrites the prefix and forces a full re-write. - 05
Measure hits, don’t assume them
Most APIs report cached input tokens in the usage block of each response. Log them next to total input tokens and track hit rate as a metric — it is the number that decides whether caching is paying.
The silent regression
When caching is the wrong lever
Output-heavy workloads
Short or unique prompts
Sparse, bursty traffic
Retrieval that changes every time
Prompt caching FAQ
What is prompt caching?
Prompt caching lets a provider reuse the work it already did on the start of a prompt. When a new request begins with exactly the same tokens as a recent one, the provider skips recomputing that prefix and bills it at a discounted cached-input rate. Only an identical prefix qualifies — the first changed token ends the cached region.
How much does prompt caching save?
It depends on how much of each request is a stable prefix and how often it is reused. Cached reads commonly cost a small fraction of the normal input price, so a workload dominated by a long, repeated prefix — a big system prompt, tool schemas, a document questioned many times — can cut its bill by half or more. Output tokens and the changing part of the input are billed in full, which caps the saving.
Can prompt caching increase costs?
Yes. Some providers charge a premium to write a prefix into the cache. If too few requests reuse it before it expires, you pay that premium repeatedly without collecting the discount. Each model has a break-even hit rate below which caching loses money; the table on this page and the prompt caching calculator show it.
Does prompt caching change the model’s output?
No. Caching reuses the model’s intermediate computation for identical input; the model sees the same tokens either way. It affects cost and latency, not what the model generates.
Why is my cache hit rate low?
The usual causes are something variable near the top of the prompt (a timestamp, a user name, a request ID), tool definitions or documents serialized in a different order each time, traffic too sparse to keep the cache warm between requests, or a prefix shorter than the provider’s minimum cacheable length.