Knowledge Distillation Explained: Teacher-Student Training for LLMs
Knowledge distillation is the process of transferring the capabilities of a large, powerful model (the teacher) into a smaller, efficient model (the student). The student learns to replicate the teacher’s behaviour on a specific task — producing a compact model that’s faster, cheaper, and often just as accurate.
It’s the reason you can replace a 70-billion-parameter API call with a 1-billion-parameter model running on a single GPU.
How does knowledge distillation work?
Large language models are general-purpose. They know a lot about everything, but you only need them to do one thing well. Knowledge distillation exploits this gap: instead of deploying the full model, you train a small model to mimic the large one on your specific task.
The process works in three stages:
- Teacher generates — The large model produces outputs (predictions, labels, reasoning) on a set of inputs relevant to your task
- Student learns — The small model is fine-tuned on the teacher’s outputs, learning to replicate its behaviour
- Student deploys — The trained student model goes into production, running at a fraction of the cost
The key insight is that the teacher’s outputs contain more information than raw labels alone. A teacher model doesn’t just say “positive” — it demonstrates how to reason about the input, what formatting to use, and how to handle edge cases. The student absorbs all of this.
Why Not Just Use a Smaller Model Directly?
You can fine-tune a small model on labelled data without a teacher. But there are practical reasons distillation works better:
- You rarely have enough labelled data. Most teams have a handful of examples, not thousands. A teacher model can generate the training data you’re missing.
- Teacher outputs are richer. A human label might say “urgent.” A teacher’s output demonstrates the reasoning, formatting, and confidence you want the student to replicate.
- It’s faster to iterate. Changing your task description and regenerating synthetic data is faster than relabelling a dataset by hand.
What does distillation look like in practice?
Say you want a model that classifies customer support tickets into categories: billing, technical, account, and other. The two routes look like this:
| Without distillation | With distillation | |
|---|---|---|
| Starting data | 2,000+ manually labelled tickets | 10–20 example tickets |
| Who produces the training set | Your team, by hand | A teacher model, synthetically |
| Handling ambiguous cases | Label more data, retrain, repeat | Teacher demonstrates the reasoning |
| Time to first model | Weeks | Hours |
| What the student learns | Surface patterns in the labels | The teacher’s decision-making |
The second approach gets you to production faster, with less manual effort, and often with better accuracy on edge cases. See how to fine-tune a small language model for the mechanics of the training step itself.
What makes distillation work well?
Not all distillation is equal. The quality of the result depends on:
Teacher quality
The teacher needs to be genuinely good at your task. A model that’s only 80% accurate will pass its mistakes to the student. Always evaluate the teacher on your test set before generating training data.
Data diversity
If the teacher generates 1,000 examples that all look the same, the student learns a narrow pattern. Effective distillation pipelines use mutation strategies — varying input complexity, length, topic, and phrasing — to ensure the training data covers the full distribution of real-world inputs.
Validation and filtering
Not every example the teacher generates is good. Automated validation catches formatting errors, duplicates, off-topic outputs, and low-quality responses before they pollute the training set.
Task specificity
Distillation works best on well-defined tasks. Classification, information extraction, question answering, and tool calling are ideal — see task selection for the full set. Open-ended creative writing is harder because “correct” is subjective.
If you are still deciding between distillation and plain fine-tuning, the comparison is here.
Distillation in the LLM Era
The concept of knowledge distillation predates large language models — Hinton and colleagues introduced it in 2015 for image classifiers, and DistilBERT applied it to language models in 2019, retaining most of BERT’s performance at 40% of its size. But LLMs have made it dramatically more practical:
- Teachers are available off the shelf. You don’t need to train a teacher — frontier models like GPT-4, Llama 3.3 70B, or Qwen3 235B are ready to use.
- Synthetic data generation scales. A teacher can produce thousands of labelled examples in minutes, solving the data scarcity problem.
- Small models are surprisingly capable. Modern SLMs in the 1B–8B parameter range have enough capacity to absorb task-specific knowledge from much larger teachers.
When should you use knowledge distillation?
Distillation makes sense when:
- You’re calling a large model API and want to reduce cost or latency
- You need to run inference on-premises or at the edge
- You have a well-defined task but limited labelled data
- You want deterministic, consistent outputs instead of prompt-dependent behaviour
It’s less suited for:
- Tasks where you need the full breadth of a general-purpose model
- Rapidly changing requirements where retraining is impractical
- Domains where no existing teacher model performs well
The Bottom Line
Knowledge distillation is how you get from “this works in a demo with GPT-4” to “this runs in production at scale.” It’s the bridge between powerful-but-expensive large models and fast-but-specialised small ones.
The teacher does the thinking. The student does the work.