What Is a Student Model in Knowledge Distillation?
The student model is the small model you fine-tune on teacher-generated data and then deploy. It is the only model that reaches production. On the distil labs catalog students run from 135M to 9B parameters, drawn from the Llama, Qwen, Gemma, SmolLM2 and Liquid LFM families.
What makes a model a student rather than a teacher?
Deployability. A student is picked for latency, cost and hardware fit; a teacher model is picked purely for accuracy during data generation.
That difference shows up in the catalog itself. The supported models page keeps two separate lists, and no model appears in both. Students top out at 9B; teachers start well above that. You set the student once in the base block of your config as student_model_name, and that value determines what you download, quantise and serve.
How small can a student model be?
Smaller than most people expect for a narrow task. The catalog spans roughly a 65x range in parameter count.
| Size band | Catalog models | Typical fit |
|---|---|---|
| Under 500M | SmolLM2-135M-Instruct, gemma-3-270m-it, functiongemma-270m-it, Qwen3-0.6B, LFM2-350M, LFM2.5-350M |
On-device, CPU, NPU; needs more training data to hit a quality bar |
| 0.6B–2B | Llama-3.2-1B-Instruct, gemma-3-1b-it, Qwen3-1.7B, Qwen3.5-0.8B, LFM2.5-1.2B-Instruct, SmolLM2-1.7B-Instruct |
The usual starting point; single consumer GPU |
| 3B–9B | Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, Qwen3-4B-Instruct-2507, Qwen3-8B, Qwen3.5-9B, gemma-3-4b-it |
Higher out-of-the-box quality, slower and costlier inference |
The docs’ own advice is to start with something like Qwen3-1.7B, check the evaluation metrics, and only move up in size if results fall short. Note that tool-calling tasks accept a subset of this list — the Qwen3, Qwen3.5, Llama 3, LFM2/LFM2.5, FunctionGemma and Gemma 4 models.
Model cards are worth reading before you commit. LiquidAI’s LFM2.5-350M is explicit that it should be fine-tuned for a specific use case, and SmolLM2-135M-Instruct is positioned for on-device work rather than general assistance.
How does the student learn from the teacher?
Through supervised fine-tuning on filtered synthetic examples, not by copying weights.
The loop is: the teacher generates in-domain examples from your task description and seeds, rule-based validators discard the malformed and off-distribution ones, and the student is fine-tuned on the seed set plus what survived. Our platform benchmark also reports a “seed student” baseline trained on the seed examples alone — on TREC it reached 0.81 against 0.92 for the fully distilled student, which is a direct measure of what the synthetic generation step buys you.
This is a different mechanism from classic response-based distillation such as DistilBERT, where the student was trained against the teacher’s output distribution and shared its tokenizer. Text-level distillation has no such constraint, which is why student and teacher families are unrelated.
Where do students beat the model they learned from?
On narrow, well-specified tasks — regularly.
In our LFM2.5 tool-calling benchmark, a fine-tuned 350M student scored 98.0% tool call equivalence on the Gorilla shell task against 97.03% for its 120B teacher, and 96.7% against 92.11% on smart home control. On the harder banking voice assistant task it came in at 95.9% against the teacher’s 96.95% — close, but behind.
The pattern holds because the student is a specialist and the teacher is a generalist. Can a small model beat its teacher goes through the numbers in detail.
What does a student give up?
Breadth. A distilled student is trained on one task distribution and will not transfer outside it. Ask a fine-tuned smart-home router to summarise a contract and you get nothing useful.
| Trade | Student gains | Student loses |
|---|---|---|
| Specialisation | Higher accuracy on the trained task | General-purpose capability |
| Size | Cheap, fast, on-device inference | Headroom on long or compositional inputs |
| Determinism | Consistent structured output | Open-ended reasoning and creativity |
If your workload is genuinely open-ended, a small student is the wrong tool — see when does distillation fail.
Related terms
- Base student — the same small model prompted for the task with no fine-tuning; the lower bound.
- Seed student — trained only on your hand-written examples, without synthetic data.
- Teacher model — the large model that generates the training data.
- Tool call equivalence — the exact-match metric used for tool-calling students.