// architecture_decision

RAG vs fine-tuning vs long context

Three ways to make a model useful on your data, routinely argued about as if they were rivals. They aren't — they change different things. Picking the right one starts with asking whether your problem is knowledge, behavior, or just fitting the material in.

Checked against the sources below · September 22, 2026Production RAG systems →

The short version

RAG gives a model knowledge it doesn't have — large, changing or citable data, fetched per question. Fine-tuning changes behavior — format, tone, a narrow task done reliably and cheaply — and is a poor way to add facts. Long context skips retrieval by putting the material in the prompt — ideal for one big document or a small corpus, expensive at volume.

Most mature systems use two of the three together.

Side by side

RAG, fine-tuning and long context compared
	RAG	Fine-tuning	Long context
What it changes	What the model knows, per request	How the model behaves, permanently	What the model can see, per request
Fresh data	Re-index and it is live	Retrain to update	Live — whatever you send
Citations	Natural — you know which chunks were used	Not available	Possible, less precise across huge inputs
Cost profile	Index + small prompts per query	Dataset + training + evals, then cheap inference	Large prompt on every query
Latency	Retrieval hop + short prefill	Fastest — short prompts	Long prefill on every query
Data scale	Effectively unbounded	Hundreds to thousands of examples	Bounded by the context window
Typical failure	Retrieves the wrong chunks	Confidently wrong “memorized” facts	Misses details buried mid-context

What “just put it all in the prompt” costs

103 models listed today accept a million tokens or more, which makes stuffing the context tempting. Here is the per-query arithmetic on Claude Fable 5.1, the strongest benchmarked model that fits a 400K-token corpus, at live list price.

Per-query input cost: long context vs RAG
Approach	Input per query	Cost per query	Per 10K queries
Whole corpus in context	400,000 tokens	$4.00	$40,000
Whole corpus, cached	400,000 tokens at cached rate	$0.100	$1,000
RAG: retrieved chunks only	8,000 tokens	$0.080	$800.00

Input only, at $10.00 per 1M tokens ($0.250 cached) — see Claude Fable 5.1. Caching narrows the gap sharply when the same corpus is queried repeatedly, which is why long context plus prompt caching is a legitimate design for a fixed document set.

Bigger windows, not equally good attention

A context window is a ceiling on what a model can accept, not a guarantee of how well it uses it. Research on long-context behavior has repeatedly found that accuracy depends on where information sits and how much surrounds it: models do best with material near the start or end of the input and can miss details buried in the middle, and performance on simple retrieval tasks degrades as irrelevant context grows.

That is the strongest argument for retrieval even when everything would fit: a prompt holding the ten most relevant passages gives the model less to get lost in than one holding ten thousand. Measure it on your own data — the effect varies by model and task.

Deciding, in order

01
Is the problem knowledge or behavior?
If the model gives wrong or outdated facts, you need knowledge — RAG or long context. If it has the facts but gets the format, tone or task wrong, you need behavior — better prompts first, fine-tuning if they plateau.
02
Does the material fit, and how often is it queried?
One contract, a codebase module, a single report: put it in context, and cache it if it is queried repeatedly. A knowledge base that grows, changes or runs to millions of tokens: retrieve.
03
Do answers need to cite a source?
Support, legal, medical and internal-policy answers usually do. Retrieval gives you the exact passages used, which makes citation and auditing straightforward.
04
Is cost or latency at volume the constraint?
A fine-tuned smaller model with short prompts can replace a large model with long instructions for a narrow, high-volume task — classification, extraction, routing — once you have the examples to train it.
05
Then evaluate before and after
Build the eval set first. Each approach fails differently, and only a fixed set of real questions tells you whether a change helped.

The combinations that work

most common

RAG + a long window

Retrieve generously, rerank, and pass more context than a small window would allow. Retrieval keeps prompts focused; the window gives headroom for whole sections rather than fragments.

fixed documents

Long context + prompt caching

Load a stable document set once and question it many times. The cached read rate makes repeated queries over the same corpus affordable.

mature systems

RAG + fine-tuning

Retrieval supplies facts; a tuned model uses retrieved context in your exact format, cites consistently and follows domain conventions.

high volume

Fine-tuned small model + router

A tuned small model handles the common, narrow requests cheaply; a router escalates anything unusual to a stronger model with retrieval.

Start cheap, add machinery only when measured

A strong model, a good prompt and long context is the fastest prototype. Add retrieval when cost, latency or accuracy on your eval set says so, and fine-tune last — it is the hardest of the three to change your mind about.

RAG vs fine-tuning FAQ

Should I use RAG or fine-tuning?

Use RAG when the model needs knowledge — facts, documents, data that changes or that you must cite. Use fine-tuning when you need to change behavior — a consistent format, tone, classification scheme or a narrow task done cheaply at high volume. Fine-tuning is a poor way to add facts: they are hard to update, hard to attribute and prone to being recalled inaccurately.

Do million-token context windows make RAG obsolete?

Not for most production systems. Putting a whole corpus in every prompt costs far more per query than retrieving a few relevant chunks, adds latency, and accuracy tends to degrade as context grows — models miss details buried in long inputs. Long context is excellent for a single large document or a small corpus, and for prototyping before you build retrieval.

Can I combine RAG and fine-tuning?

Yes, and it is common. Retrieval supplies the facts; a fine-tuned model is better at using retrieved context in your format, citing it consistently or following domain conventions. Fine-tune only once a RAG system with good prompts has plateaued on behavior, not on knowledge.

Is fine-tuning expensive?

The training run is usually the small cost. The larger costs are building and maintaining a labeled dataset, evaluating each new version, and retraining when the base model is deprecated or your requirements change. Parameter-efficient methods such as LoRA reduce compute but not that ongoing work.

How big a document set can I just put in the prompt?

Mechanically, up to the model’s context window. Practically, it depends on query volume and accuracy needs: a few long documents queried occasionally fit well in context — especially with prompt caching — while a large or growing corpus queried at volume is cheaper and usually more accurate with retrieval.

Sources

Production RAG systems Vector database guide Prompt caching explained Longest context LLMs