What Is Overfitting in Fine-Tuning?

Overfitting is when a model learns your specific training examples rather than the pattern they illustrate. The symptom is a gap: training loss keeps falling while held-out scores plateau or decline, and the model handles inputs that look like the seed set but fails on paraphrases of the same request.

How do you know you have overfitted?

You know from the gap between what the model saw and what it did not. A single accuracy number cannot tell you — you need two, measured on disjoint data.

The distil labs pipeline gives you both by default. train_eval_split in the config file reserves a fraction of the training data for evaluation, defaulting to 0.2, and the minimal dataset layout requires a separate test.jsonl held-out evaluation set alongside train.jsonl.

Read them together:

What you observe	Reading
Training split high, held-out split high	Learning. Nothing to fix.
Training split high, held-out split much lower	Overfitting. The model memorised rather than generalised.
Both low	Not overfitting — the task or the data is the problem
Held-out high but production quality poor	Your test set does not resemble production traffic

There is a qualitative check too. Send the model a training example with the wording changed but the meaning identical. A model that answers the original correctly and the rewrite incorrectly has memorised strings.

Why is it so easy to hit when fine-tuning a small model?

Because the datasets are small and the runs are short, which is exactly the regime where memorisation is cheap. Fine-tuning a small language model often starts from a few dozen high-quality examples, and a model with billions of pretrained parameters can fit a few dozen examples very comfortably.

Kaplan et al. established the relationships governing overfitting and training speed relative to model and dataset size: for a given dataset, overfitting becomes more likely as model size grows, and larger models are more sample-efficient in ways that cut both ways. Applied here, it means the same dataset that generalises on a 0.6B student can be memorised by a 8B one.

Two things about the distillation workflow push against this, and both are worth understanding before you start turning knobs:

• Training does not happen on your handful of seed examples directly. The teacher expands them into a much larger synthetic set — the default generation_target is 10,000 examples.
• Generated examples too close to the seed data are discarded. validation_similarity_threshold defaults to 0.95, and generated data above that similarity to seed data is removed.

That is deduplication acting as a generalisation guard. It is also why the fix for overfitting here is usually about generation breadth rather than about epochs.

Which knobs actually reduce it?

In the order worth trying, cheapest and most effective first:

Signal	First change to make
Held-out plateaus while training loss falls	Reduce num_train_epochs from its default of 4
Model fails on paraphrases	Widen basic_mutators_to_use beyond the default ["complexity"] to include length and specificity
Outputs cluster around a few seed phrasings	Set mutation_topics so generation samples across the topics you actually serve
Model reproduces seed examples verbatim	Lower validation_similarity_threshold so near-duplicates are filtered more aggressively
Adapter has more capacity than the task needs	Reduce lora_r from its default of 64

The rank lever is the most direct capacity control you have. LoRA freezes the pretrained weights and trains rank-decomposition matrices whose size scales linearly in r, so halving the rank halves what the adapter can encode. What is LoRA rank covers the arithmetic.

Reach for epochs and rank after data breadth, not before. A model overfitting a narrow synthetic set will still overfit it in fewer epochs; it will just do so less thoroughly. Generating synthetic training data is the lever that changes what there is to overfit to.

What if the test set is the problem instead?

This is the case people miss, and it produces the opposite error — believing you overfitted when you did not, or the reverse.

• Leakage. If the same examples seeded both generation and the test set, a high held-out score is not evidence of anything. The platform sidesteps this when you provide your own test file: num_traces_as_testing_base is documented as ignored if a test set is provided.
• A tiny test set. With a few dozen held-out items, run-to-run variance can look exactly like a regression. Before concluding anything, check whether the gap exceeds the noise.
• A mismatched metric. The metrics guide notes that low Exact-Match with high LLM-as-a-Judge usually means correct answers phrased differently. That is a scoring artefact, not memorisation.
• A test set from the wrong distribution. A held-out split carved from curated seed examples measures generalisation across that curation, not across production traffic.

A checklist before you retrain

Run through this before you change a single hyperparameter:

1. Confirm the training and held-out sets are genuinely disjoint.
2. Confirm the held-out set is large enough that the gap you are chasing is bigger than the noise.
3. Read twenty individual wrong predictions rather than the aggregate.
4. Check whether the failures are paraphrases of training items — that is the memorisation signature.
5. Only then change one parameter, and re-run with everything else fixed.

If the model is worse than where it started rather than merely narrow, that is a different diagnosis: see why did my fine-tuned model get worse.