RAG vs Fine-Tuning: Which One Fixes Your Problem?

RAG puts facts into the prompt at inference time. Fine-tuning puts behaviour into the weights at training time. They aren’t competing solutions to one problem. They solve different ones, and most production systems that answer questions over private documents end up using both.

How do they differ mechanically?

They differ in what changes, when it changes, and what you have to redo when your data does.

	RAG	Fine-tuning
What changes	The prompt sent to the model	The model weights
When it changes	Every request	Once per training run
Adding a new document	Re-index (minutes)	Retrain (hours)
Changing the output format	Edit the prompt, hope	Add examples, retrain
Prompt length	Grows with retrieved context	Stays short
Inference latency	Retrieval hop plus generation	Generation only
Attribution	Cited chunks available	None by default
Fails by	Retrieving the wrong passage	Learning the wrong behaviour

Retrieval is a memory system bolted to the outside of a model. Lewis and colleagues described this as combining parametric memory (what the weights know) with non-parametric memory you can edit without touching the weights.

What is RAG genuinely better at?

RAG is better at anything where the facts change independently of the task. A support assistant over a product wiki, a policy lookup over a compliance library, a codebase question-answering tool: all of these have a stable job and a moving corpus.

It’s also the only one of the two that gives you attribution for free. If a regulator or a customer asks where an answer came from, retrieval hands you the passage. A fine-tuned model that has memorised the fact can’t show its work.

RAG is better at new facts specifically. Ovadia and colleagues compared knowledge injection methods head to head and found RAG consistently outperformed unsupervised fine-tuning, both for knowledge the model had seen in pretraining and for knowledge it hadn’t, with the added observation that models struggle to learn new factual information through unsupervised fine-tuning at all.

What is fine-tuning genuinely better at?

Fine-tuning is better at everything the model does with the facts. Format compliance, tool selection, refusal behaviour, answer length, ignoring an irrelevant retrieved chunk: these are all behaviours, and behaviours are what gradient updates are good at teaching.

It’s also what makes small models viable at all. In a distil labs test on a Siemens S7-1200 controller manual, a Llama 1B base model scored 45.1% on 144 held-out questions; after fine-tuning on the same documentation it reached 61.1%, effectively matching a Llama 3B base model at 60.4%. The full write-up has the breakdown. The retrieval setup was identical in all three cases, so what changed was the model’s ability to use what retrieval handed it.

That gap isn’t cosmetic. Small base models handed three candidate chunks amplify retrieval noise, hallucinate parameter codes, and drift off the requested output format. A 1B model that can do RAG properly is a fine-tuned one.

What does the combination look like?

The combination is a specific, named thing rather than a vague “use both”. You retrieve as normal, and you fine-tune the generator on examples that pair a question with the passage it should answer from, plus deliberately irrelevant passages it should ignore.

RAFT formalised this: train the model to cite the right sequence from the relevant document while distractor documents sit alongside it. On the distil labs platform the same idea is one config line, since synthgen.num_distractor_context_blocks above zero enables distractor training, documented in the config reference.

base:
  task: question-answering-open-book
  student_model_name: Qwen3-1.7B
  teacher_model_name: zai.glm-5
synthgen:
  num_distractor_context_blocks: 2
  teacher_temperature: 0.6

The teacher_temperature value matters: reasoning teachers such as GLM-5 require it between 0.5 and 0.7, per the supported models catalog.

Which should you pick first?

Pick RAG first if you’ve never shipped either. It’s faster to stand up, it doesn’t require labelled examples, and it tells you quickly whether your corpus actually contains the answers, which is information you need before any training run.

Add fine-tuning when one of these is true:

• Retrieval finds the right passage and the model still answers badly
• You need a model small enough to run on-device or on-prem, where base quality is the binding constraint
• Output format compliance matters and prompt instructions keep getting ignored
• Per-request cost matters and you want to stop shipping long prompts

Skip RAG entirely only when the knowledge is genuinely fixed and small: a stable taxonomy, a closed tool schema, a fixed set of policies. That’s what the question-answering-closed-book task in task selection is for, and what is a small language model covers what that size band can hold.

For the broader three-way decision including prompting, see fine-tune, RAG, or prompt. For the mechanics of running both together, can you combine RAG and fine-tuning.