Knowledge distillation is how a large model’s capability on one task gets transferred into a model small enough to run on your own hardware. A big model — the teacher — produces outputs on inputs relevant to your task. A small model — the student — is trained to reproduce that behaviour. What you deploy is the student.
The idea predates large language models. Hinton and colleagues introduced it in 2015 for image classifiers, and DistilBERT applied it to language models in 2019. What changed with LLMs is that the teacher no longer has to be trained. Frontier models are available off the shelf, they can generate thousands of labelled examples in minutes, and small models in the 0.3B to 8B range turn out to have enough capacity to absorb task-specific knowledge from a model a hundred times their size.
Why distillation rather than plain fine-tuning
The two are often confused, and they are genuinely related — distillation usually ends in a fine-tuning step. The difference is where the training data comes from.
Fine-tuning assumes you already have a labelled dataset. Distillation assumes you do not, and uses a teacher model to produce one. That matters because the dataset is where most projects stall: teams typically have a handful of examples, not thousands.
Teacher outputs also carry more than a label. A human annotator writes “urgent”. A teacher model demonstrates the reasoning, the output format, and how to handle the awkward cases at the boundary. The student absorbs all of it.
What determines whether it works
Four things, roughly in order of how often they are the problem:
Teacher quality on your specific task. A teacher that is 80% accurate hands its mistakes to the student. Evaluating the teacher before generating any training data is the single highest-value check in the pipeline, and it is cheap.
Data diversity. A teacher asked for a thousand examples will happily produce a thousand near-identical ones. The student then learns a narrow pattern that collapses on real inputs. Effective pipelines vary input length, complexity, phrasing, and topic deliberately.
Validation and filtering. Not everything a teacher generates is usable. Format errors, duplicates, and off-topic outputs need to be caught before they reach the training set.
Task definition. Distillation works best where “correct” is well defined — classification, extraction, question answering, tool calling. Open-ended generation is harder, because there is no clean signal for what the student should be matching.
Where this cluster goes
The articles below cover the mechanics: what teacher and student models actually are, how to choose between them, whether a student can exceed the teacher it learned from (it can, and there is benchmark data on when), and the failure modes worth recognising early.
If you are trying to decide between distillation and something else entirely — retrieval, prompting, or simply a bigger model — the alternatives cluster covers that comparison directly. If you have already decided and want the operational path, start with the training data cluster, which covers the three ways to get a dataset onto the platform.