Is Your Fine-Tuned Model Good Enough?
A single accuracy number can’t answer this. You need four comparisons: against the base model, against the teacher, against whatever is in production today, and against run-to-run noise. A model that beats all four is shippable. A model that beats only the first is a training run that worked, not a product.
What are you comparing against?
Four baselines, and each answers a different question. Reporting only one is the most common way an evaluation misleads.
| Baseline | Question it answers | Where the number comes from |
|---|---|---|
| Base model, prompted | Did fine-tuning buy anything? | distil seed-dataset metrics <seed-dataset-id>, or a prompted run on the same test set |
| Teacher model | Did the student reach the ceiling it was aiming at? | distil teacher-evaluation metrics <teacher-evaluation-id> |
| Incumbent system | Is this better than what you already ship? | Score the current model on the same test set |
| The same model, re-run | Is the gap bigger than the noise? | Repeat runs, or reported standard deviations |
The distil labs platform benchmark runs a fourth reference point worth borrowing: the seed student, trained on the initial labelled set alone with no synthetic generation. On Banking77 the seed student scored 0.76 against 0.895 for the fully trained student. The synthetic generation stage, not the fine-tuning mechanics, was doing the work.
Which criteria decide it?
Read them in this order, and stop at the first one that fails.
| Criterion | Threshold that means “ship it” | If it fails |
|---|---|---|
| Beats the prompted base model | Any gap larger than run-to-run noise | Training didn’t take. See why did my fine-tuned model get worse |
| Within reach of the teacher | The training docs suggest “reasonably close, typically within one standard deviation” | More training data or a better teacher; check teacher evaluation first |
| Meets the absolute bar the application needs | Task-specific; set it before you look at the score | The task may need a larger student, not a better recipe |
| Gap survives repeated runs | The difference exceeds the reported standard deviation | You measured noise |
| Holds on inputs the test set doesn’t contain | Qualitative, from reading predictions | See building a test set that catches real failures |
Beating the teacher and being good enough are different claims. In our 12-model benchmark, the fine-tuned Qwen3-4B beat its teacher on SQuAD 2.0 by 19 points, and still only reached 0.71. Set the absolute bar before you look at the score.
What does each verdict mean in practice?
Student ≈ teacher. This is the expected outcome and the signal to ship. On our platform benchmark the trained student reached or exceeded the teacher on eight datasets; the one shortfall, Banking77, sat within 1 percentage point and inside the reported confidence interval.
Student > teacher. Common, not suspicious. Validation removes the teacher’s own failures from the training set and the student spends all of its capacity on one task. Confirm the test set is clean before celebrating, then read can a small model beat its teacher.
Student well below teacher. The gap is diagnostic. If the base model was also far below, the task is hard and the student may be too small, so check what size model do you need. If the base model was close, something in the training configuration is wrong.
Student ≈ base model. Fine-tuning bought nothing measurable. Either the base model already solved the task, or the synthetic data didn’t cover the inputs the test set contains.
What makes a model look better than it is?
Five failure modes, all of which produce a number that’s technically correct and practically useless.
- The test set came from the training distribution. If both were carved from the same curated seed set, you measured internal consistency. This is the single most common cause of a model that passes eval and fails in production.
- The test set was filtered by the same pipeline that built the training set. When trace processing drops low-coherence traces, the survivors form an easier test set than production traffic.
- The metric doesn’t match the task. Exact-Match on free-text generation understates a correct model; accuracy on an imbalanced classification set overstates a useless one. Accuracy, F1 or exact match covers the mapping.
- Per-example scores read as per-conversation reliability. For a multi-turn agent, 95% per turn is roughly 0.95^20, about 35% of 20-turn interactions fully correct, as our traces benchmark spells out.
- The gap is inside the noise. In the deferral cascade benchmark, LLM-as-a-judge scores carried roughly ±0.03 run-to-run variance, and a 0.03 gap was reported as not statistically significant under a paired McNemar test.
Test set size sets a hard floor on what you can detect. On 50 examples one item is 2 percentage points, and the width of a binomial confidence interval around a proportion narrows only with the square root of the sample. How big should your test set be has the sizes used in published work.
Should you ever ship a model that scores below the teacher?
Often, yes, because accuracy isn’t the only axis and the teacher isn’t a deployment candidate. The teacher is a data-generation tool you never serve.
Ship the lower-scoring student when the gap is inside the confidence interval, when the deployment constraint (on-premises, air-gapped, latency budget, cost per request) rules the teacher out entirely, or when a cascade recovers the difference. The cascade case is measured: routing about 4% of hard turns to a frontier model left quality statistically indistinguishable from running the frontier model on everything.
Don’t ship when the shortfall concentrates in one class or one input type. An aggregate that hides a systematic failure on 5% of traffic is worse than a lower aggregate that fails evenly, because the failure will be correlated with a user segment you care about. Read the per-example predictions before you decide.