← All learn articles

Can You Combine RAG and Fine-Tuning?

Can You Combine RAG and Fine-Tuning?

Yes. The retrieval layer stays exactly as it is; you fine-tune the model that consumes its output, teaching it to answer strictly from the retrieved passage and to ignore passages that were retrieved but don’t help. The task type for this is open-book question answering.

How does the combination work?

The two layers stay separate and only meet at the prompt. Retrieval is unchanged: same embeddings, same vector store, same top-k. What changes is the generator, which has been trained on examples of exactly the shape it will see at inference.

That training signal is what a base model doesn’t have. Lewis and colleagues established the pattern of pairing a fixed parametric memory with an editable non-parametric index; what the original formulation left open is how well the generator handles an imperfect index. RAFT closed that gap by training the model with distractor documents in context, so it learns to cite the right sequence from the relevant document instead of averaging over everything it was handed.

On the distil labs platform it’s the question-answering-open-book task with one synthgen setting:

base:
  task: question-answering-open-book
  student_model_name: Qwen3-1.7B
  teacher_model_name: zai.glm-5
synthgen:
  num_distractor_context_blocks: 2
  teacher_temperature: 0.6

num_distractor_context_blocks above zero is what enables distractor training, per the config reference. The temperature value isn’t arbitrary. Reasoning teachers such as zai.glm-5 require 0.5 to 0.7, listed under task compatibility in the supported models catalog.

What does the training data look like?

Each training example is a question, the passage that answers it, and the answer, not a question-answer pair alone. The context field is what distinguishes open-book data from ordinary QA data.

Field Contents Why it’s there
messages One user turn with the question, one assistant turn with the answer The behaviour to learn
context The passage the answer is grounded in Teaches grounding, not recall
unstructured.jsonl Chunks from the same corpus, context field only Steers synthetic generation toward your domain

The open-book QA data preparation guide has the exact JSONL shapes. In practice the unstructured file is just your existing RAG chunks. If you’ve already chunked a corpus for retrieval, that file is a re-export, not new work.

When is the combination worth the training run?

It’s worth it when retrieval is already finding the right passage and the answer is still wrong. To check, run your retriever, read the top chunk yourself, and see whether a careful reader could answer from it. If they could, the generator is your bottleneck.

Two situations make it clearly worth it:

  • The model has to be small. In a distil labs test on a Siemens PLC manual, a Llama 1B base model scored 45.1% across 144 questions and 61.1% after fine-tuning, reaching parity with a Llama 3B base model at 60.4%. The write-up covers the setup. At 1B, fine-tuning is the difference between viable and not.
  • The corpus has hostile formatting. Signal paths, parameter codes, nested tables. Base models below a few billion parameters misattribute values across table rows and invent plausible-looking codes.

It isn’t worth it when a large hosted model already answers correctly from your chunks and volume is low, since the training run would buy you nothing you don’t already have.

What breaks when you combine them?

The most common breakage is a mismatch between training-time and inference-time context. If you train on clean, hand-picked passages and serve on noisy top-3 retrieval, the model has never seen the distribution it now faces, which is the exact problem distractor blocks exist to prevent.

Three others worth knowing:

  1. Chunk size drift. Retrain if you change your chunking strategy. A model trained on 1000-character chunks handles 4000-character ones worse.
  2. Corpus churn outrunning the model. The facts live in the index and update freely, but if the shape of your documents changes (new manual format, new ticket schema), the learned behaviour goes stale even though the facts are fresh.
  3. Evaluating grounding with the wrong metric. Exact match punishes correct paraphrase. The metrics guide covers why LLM-as-a-judge is usually the right primary score for open-book QA.

Sources

Related

All Alternatives articles →