Can a Small Model Beat Its Teacher?
Yes, and on narrow tasks it happens routinely. A fine-tuned Qwen3-0.6B beat its 120B teacher by 29 points on smart-home tool calling. A 350M LFM2.5 beat the same teacher on two of three tool-calling benchmarks. The student is a specialist; the teacher is a generalist.
How much can a student beat its teacher by?
The largest margin in our published work is 29 percentage points, from a student 200x smaller than its teacher.
| Task | Student | Teacher score | Base student | Tuned student | Margin |
|---|---|---|---|---|---|
| Smart home routing (MASSIVE IoT) | Qwen3-0.6B | 50.0% | 10.26% | 79.49% | +29.5 |
| Shell commands (Gorilla) | LFM2.5-350M | 97.03% | 61.4% | 98.0% | +1.0 |
| Smart home control | LFM2.5-350M | 92.11% | 63.2% | 96.7% | +4.6 |
| Banking voice assistant | LFM2.5-350M | 96.95% | 34.5% | 95.9% | −1.1 |
| Smart home control | FunctionGemma 270M | 92.11% | 38.82% | 96.71% | +4.6 |
| Shell commands (Gorilla) | FunctionGemma 270M | 97.03% | 9.90% | 96.04% | −1.0 |
All figures are tool call equivalence. The teacher throughout is GPT-OSS-120B. Sources: the dlt trace case study for row one, the LFM2.5 benchmark for rows two to four, and the FunctionGemma benchmark for the last two.
The pattern is not confined to tool calling. Across the ten datasets in our platform benchmark, a Llama3 3B student distilled from a Llama3 70B teacher reached or exceeded the teacher on eight, with the sole shortfall being Banking77 at 0.895 against 0.91 — inside the reported confidence interval.
It also holds with a different teacher and a different metric. In our traces-vs-synthetic benchmark a Qwen3-1.7B student distilled from GLM-5 scored 0.844–0.866 LLM-as-a-judge across five trace-corruption scenarios against 0.835 for GLM-5 itself, at a 437x size difference. The Claude skill walkthrough reports the same pairing at 0.846 for the tuned student against 0.808 for the 744B teacher, with the untuned base student at 0.513.
Why does a 0.6B model outscore a 120B one?
Because the two models are being asked different questions. The teacher is scored zero- or few-shot on a schema it has never seen; the student was trained on nothing else.
The dlt case study puts it plainly: the teacher “scores lower because it is a general-purpose model: it has never seen your specific function schemas or the phrasing patterns of your users.” That task was built from Amazon MASSIVE, 16,000+ utterances across 60 intents, narrowed to the IoT scenario and 9 functions. A generalist has to infer the right function catalogue from the prompt on every call. A specialist has it in the weights.
Three mechanisms stack up:
- Task-shaped priors. The student only ever emits calls from your catalogue, so whole classes of teacher error — wrong function, invented parameter — become impossible.
- Curated training data. Teacher outputs are validated and filtered before training, so the student learns from the teacher’s correct answers, not its average behaviour.
- Format lock-in. Structured-output tasks reward consistency, and a fine-tuned model is far more consistent than a prompted one.
How were these numbers measured?
Tool call equivalence on a held-out test split, with the same split and evaluation harness for teacher and student.
Tool call equivalence is a binary exact-match score comparing predicted and reference tool calls, with unset parameters treated as taking their defaults — the full definition is in metrics. The Gorilla task is the file-system task from the Berkeley Function Calling Leaderboard.
The pipeline behind each row is the same: task description plus seed examples, teacher generates roughly 10,000 in-domain synthetic examples, rule-based validators filter them, the student is fine-tuned on seeds plus survivors, then evaluated. The dlt run used ~10,000 generated examples grounded in 1,107 real traces and completed training in under 12 hours. Platform benchmark values are averaged over three runs.
Two caveats worth stating. Teacher scores are measured with a fixed prompt and k-shot examples drawn from the training split; a heavily engineered prompt would raise them. And these are held-out test splits from the same distribution as training, which is the honest setting for a specialist but not a test of generalisation.
When does the teacher stay ahead?
On open-ended generation and on the hardest structured tasks.
| Where the teacher leads | Evidence |
|---|---|
| Large function catalogues with noisy input | Banking voice assistant: 14 functions plus ASR artifacts; LFM2.5 reached 95.9% vs 96.95% |
| Open-ended QA | In our mid-size MoE benchmark the 550B teacher led on resume-roast and roman-empire by 0.04–0.07 |
| Anything outside the trained distribution | The student has no general capability to fall back on |
None of these gaps are large, but they are real, and a 95.9% score means roughly 1 in 24 calls needs correction. Production systems in that band pair the student with an orchestrator or a confidence threshold that escalates to a larger model.
What changes the size of the gap?
Four things, in rough order of leverage.
- How weak the teacher is on your task. The widest margins come from tasks where the teacher itself struggles — 0.50 to 0.70 on pizza tool calling, 50.0% to 79.49% on MASSIVE IoT. Run a teacher evaluation first and you will know which case you are in.
- How narrow the task is. Constrained catalogues and fixed output schemas favour the student; free-form text favours the teacher.
- Training epochs. LFM2.5 on the voice assistant task went 34.5% → 86.8% → 92.4% → 95.9% → 95.4% across epochs 0–4, peaking at epoch 3 and dipping after. More training is not monotonically better.
- Base student strength. Weakly correlated at best. Base FunctionGemma scored 2–6x lower than base LFM2.5 and still finished within a couple of points on two of three tasks.
If the student lands well below the teacher, the cause is usually upstream of model choice — see when does distillation fail. For the mechanism itself, teacher-student distillation and knowledge distillation for LLMs cover the pipeline end to end.