RAG vs fine-tuning: a decision guide

Retrieval-augmented generation (RAG) gives a model access to knowledge it was not trained on, at query time, with sources it can cite; fine-tuning changes the model's behaviour, style, format and skill at a task by adjusting its weights. Choose RAG when the problem is what the model knows, choose fine-tuning when the problem is how the model responds, and expect many production systems to need a measure of both.

TL;DR

• RAG changes knowledge: fresh, private or frequently updated information, with citations and per-user access control.
• Fine-tuning changes behaviour: tone, output format, domain vocabulary and task skill, often letting a smaller, cheaper model do the job.
• Freshness, citations and access control favour RAG; latency, consistent style, structured output and unit cost favour fine-tuning.
• Build an evaluation set before choosing either, and combine them when you need both grounded facts and reliable behaviour.

What does each technique actually change?

The confusion between RAG and fine-tuning comes from treating them as two routes to the same destination. They are not.

RAG leaves the model untouched. At query time, the system searches an index of your documents, selects the most relevant passages and places them in the prompt alongside the question. The model answers from that context. What changes is the information available to the model on each call. Swap a document in the index and the next answer reflects it.

Fine-tuning continues training the model on examples you provide, typically pairs of inputs and ideal outputs, so its weights shift toward producing responses like those. What changes is the model's default behaviour: how it phrases things, what format it returns, which domain conventions it follows, how well it performs a narrow task. It is a poor way to teach facts, because the model does not reliably recall specific training examples, and it cannot know anything that happened after training.

A useful shorthand: RAG is an open-book exam; fine-tuning is coaching the student. Our RAG systems and AI fine-tuning teams start every engagement by asking which of the two the client's problem actually is.

When does RAG win?

RAG is the default answer for anything grounded in a body of information, and it wins decisively when:

• Content changes. Policies, prices, product catalogues, tickets and contracts update daily or weekly. Re-indexing is minutes; re-training is a project.
• Users need citations. Support, legal, compliance and research applications need to show where an answer came from. RAG can return the passage; a fine-tuned model can only assert.
• Access differs by user. If a manager may see documents an intern may not, retrieval can filter by permission before the model sees anything. Knowledge baked into weights cannot be partitioned per user.
• The corpus is large. Millions of documents will not fit in a training run in any practical sense; they fit comfortably in a search index.
• You need to explain and correct errors. When RAG answers wrongly you can inspect what was retrieved and fix the chunking, the ranking or the source. When a fine-tuned model answers wrongly, you re-train and hope.

The cost of RAG is more moving parts: an ingestion pipeline, an index, a retriever whose quality bounds the whole system, and longer prompts that add tokens and latency per call.

When does fine-tuning win?

Fine-tuning earns its place when the problem is behavioural rather than informational:

• Consistent tone and voice. A brand style that prompting keeps drifting from can be trained in and stays put.
• Strict output structure. Domain-specific JSON, tagged extraction, classification labels or a house report format, produced reliably without long instructions in every prompt.
• Domain language. Clinical, legal or engineering shorthand that general models paraphrase awkwardly.
• Latency and cost at scale. A small model fine-tuned on a narrow task can match a much larger general model on that task, with a fraction of the per-call cost and response time. At high volume this is often the deciding factor.
• Shorter prompts. Behaviour that once needed several hundred tokens of instructions and examples moves into the weights, cutting cost per request.

The costs are real too. You need hundreds to thousands of high-quality examples, a training and evaluation loop, versioning for model artefacts and a way to redeploy when the base model changes. Fine-tuning also does not solve knowledge freshness at all, and it can quietly degrade general capability if the data is narrow.

RAG or fine-tuning: a side-by-side view

Should you combine them?

Often, yes. The two techniques address different failure modes, so a system that needs both grounded facts and dependable behaviour benefits from both. Common combinations:

• Fine-tune for the format, retrieve for the facts. A model tuned to answer in your house style and always cite in a particular structure, fed passages by a RAG pipeline.
• Fine-tune the retriever. Embedding models trained on your query-and-document pairs can lift retrieval quality more than any change to the generator.
• Distil a RAG system. Once a RAG pipeline produces good answers, use its outputs to fine-tune a smaller model for the stable, high-volume subset of queries, keeping RAG for the long tail.

Sequence matters. Start with prompting and RAG because they are cheaper to change; fine-tune once you have a stable task definition and enough real examples to train on. Retro-fitting RAG onto a fine-tuned model works; retro-fitting a fine-tune onto a system whose task keeps shifting does not. Our generative AI and LLM engagements almost always follow that order.

Why evaluation has to come first

Neither choice can be made responsibly without an evaluation set: real inputs paired with agreed-good outputs, scored automatically where possible and by people where not. Build it before choosing an approach, for three reasons.

First, it tells you what the problem is. If a strong general model with the right documents in the prompt already clears the bar, you do not need fine-tuning. If it has all the facts and still answers in the wrong shape, you do.

Second, it makes the comparison honest. Prompt-only, RAG, fine-tuned and combined variants can be run against the same cases and the same metrics.

Third, it protects you afterwards. Base models change, documents change and prompts drift. A standing evaluation run before every release is the only way to know a change helped.

Key takeaways

• RAG changes what a model knows at query time; fine-tuning changes how it behaves by default.
• Freshness, citations, access control and large corpora point to RAG.
• Consistent style, strict formats, domain language and low unit cost at volume point to fine-tuning.
• Combining them is common: retrieve for facts, fine-tune for behaviour, and consider fine-tuning the retriever itself.
• Build the evaluation set first; it decides the approach and guards every change after.

If you are weighing the two for a specific system, contact us and we will help you set up the evaluation that settles it.

FAQ

Can fine-tuning teach a model our company's knowledge?

Only unreliably. Fine-tuning shifts a model's tendencies rather than storing facts it can retrieve on demand, so it may paraphrase, blend or invent details, and it cannot cite where an answer came from. It also freezes knowledge at training time. For company knowledge, RAG is the right tool; fine-tuning can then improve how the model uses what retrieval provides.

Which is cheaper, RAG or fine-tuning?

It depends on volume and volatility. RAG is cheaper to set up and change but adds tokens and a retrieval step to every call. Fine-tuning costs more upfront in data preparation and training, then can lower per-call cost by allowing a smaller model and shorter prompts. At low volume with changing content, RAG usually wins on cost; at high volume with a stable task, fine-tuning often does.

How much data do we need to fine-tune?

For style, format and narrow classification tasks, a few hundred carefully curated examples can produce a clear improvement; for broader behavioural changes, a few thousand is more typical. Quality matters far more than quantity: inconsistent or noisy examples teach inconsistency. Many teams find that assembling and reviewing this data is most of the effort, which is another reason to build the evaluation set first.