Why Did My Fine-Tuned Model Get Worse?
A fine-tuned model that scores below its own starting point on its own task has usually hit one of four things: a metric that does not match the task, a task the teacher never solved, training data that contradicts itself, or a comparison against the wrong baseline. Check them in that order.
Worse than what — the base model, the teacher, or your expectations?
Name the baseline before diagnosing anything, because the three failures have nothing in common. A distil labs scorecard reports the teacher’s accuracy, the untuned student’s accuracy, and the tuned student’s accuracy side by side, as the vibe-tuning walkthrough shows.
- Below the untuned student. Genuinely alarming. Training actively damaged the model; something in the data or labels is wrong.
- Below the teacher. Expected to a degree. The model training guide sets the bar at performance reasonably close to the teacher, typically within one standard deviation — not at parity.
- Below what you hoped. Not a regression at all. The teacher’s score is the first approximation of what a student can reach, so an unmet hope is usually an unrealistic one.
Only the first case means the fine-tune made things worse. Read the rest of this page with that distinction held.
Is the drop real, or a metric artefact?
Often it is an artefact, and the fastest check costs nothing. The metrics guide spells out the tell: if Exact-Match is low but LLM-as-a-Judge is high, the answers are probably correct but paraphrased.
That is the single most common false alarm. Fine-tuning frequently changes phrasing before it changes correctness, and Exact-Match returns 0 for any wording that is not identical to the reference. ROUGE-L has its own bias — it rewards longer answers that reuse reference phrasing, so a model trained to be terser can score lower while answering better.
Judge-based metrics have failure modes of their own, which is why they should be read rather than trusted blindly. The survey of LLM-as-a-judge methods catalogues the known biases, and Zheng et al. documented position and verbosity effects when introducing MT-Bench. If your judge model changed between the teacher evaluation and the training evaluation, you are comparing two different rulers.
Did the teacher ever solve the task?
If the teacher scored poorly, the student was never going to score well, and the regression you are looking at started before training. Teacher evaluation exists precisely as this gate: it validates whether a large model can solve your task at all, before you spend a training run finding out.
The docs are explicit that low teacher performance points at the task definition rather than the model — revise the task description to be more specific, improve example quality, check the dataset for inconsistencies, and confirm the task is well defined and solvable. A student trained against a confused task learns the confusion faithfully.
Which cause matches your scorecard?
| What the scorecard shows | Likely cause | Fix |
|---|---|---|
| Exact-Match down, LLM-as-a-Judge flat or up | Correct answers, different wording | Score with LLM-as-a-Judge; add the paraphrases to your reference set |
| Every metric low, teacher was also low | Task under-specified | Rewrite the job description, re-run teacher evaluation before retraining |
| Every metric low, teacher was high | Too few or inconsistent training examples | Add examples, tighten labels, then consider a larger student |
| Strong on seed-like inputs, weak on held-out | Memorisation, not learning | See what is overfitting in fine-tuning |
staged_tool_call around 0.5 |
Right function, wrong parameters | Fix parameter schemas and defaults in the seed data |
staged_tool_call around 0.25 |
Valid JSON, wrong function | Disambiguate the tool descriptions |
The staged scores are documented behaviour, not estimates: the metrics guide awards 0.25 for valid JSON, 0.50 for the correct function name, 0.75 for correct parameter keys, and 1.0 for an exact match. That makes tool-calling regressions unusually easy to localise.
How do you re-run the comparison properly?
Download the per-example predictions before changing anything. Reading twenty wrong answers tells you more than any aggregate:
distil model download-training-predictions <model-id>Then re-run with one variable changed, not four. The remediation list in the training docs is ordered for a reason — more training examples first, a more specific task description second, configuration changes such as more epochs third, and a larger student model last. Jumping straight to a bigger student hides a data problem behind compute.
Two habits keep this cheap. Freeze the judge model and the test set across runs so scores stay comparable. And keep a held-out set that never fed generation, so an improvement is an improvement rather than a leak. If the honest answer after all this is that the task does not need a fine-tuned model, is fine-tuning worth it is the better page to read next.