When Does Distillation Fail?
Distillation fails when the task is under-specified, when the labels in your seed data contradict each other, or when the desired output is open-ended rather than narrow. Teacher evaluation catches the first two before you spend a training run. The third is a task-selection problem, not a training one.
What does a failed distillation look like?
Three distinct symptoms, and they point at different causes.
| Symptom | Most likely cause | Where to look first |
|---|---|---|
| Teacher evaluation score is low | Task is under-specified or seed data is inconsistent | Task description, then seed examples |
| Teacher scores well, student lands far below | Task needs capacity the student does not have, or too few seeds | Student size, seed count |
| Both score well, production accuracy is poor | Test split does not match real traffic | Data distribution |
| Exact-match low, LLM-as-a-Judge high | Not a failure — answers are correct but paraphrased | Metric choice |
The last row matters because it is regularly misread as a failure. The metrics page is explicit: if exact-match is low while LLM-as-a-Judge is high, the answers are probably right and phrased differently — add the paraphrases to your reference set rather than retraining.
Which causes are most common?
Under-specification, by a wide margin. It is also the cheapest to detect.
1. The teacher cannot solve the task. This is the hard stop. Docs put it directly: “if the teacher model can solve the task, the student model will be able to learn it effectively. If the teacher model cannot solve the task, you have an opportunity to refine your inputs before investing time in full SLM training.” Run teacher evaluation before anything else.
2. The task description is vague. The teacher generates from your description. Ambiguity there becomes noise in ten thousand synthetic examples, and the student learns the noise. This is the same failure mode described in the Self-Instruct line of work — generated data inherits the ambiguity of its instructions.
3. Seed labels contradict each other. Two near-identical inputs with different labels teach the student that the boundary is arbitrary. Classification tasks are the usual victims.
4. The task is genuinely open-ended. Distillation compresses knowledge into a narrow specialism. If you need creative or general-purpose generation, a small student is the wrong tool and no amount of data fixes it. Task selection lists the six task types the pipeline supports; if yours does not map onto one of them cleanly, that is a signal.
5. The wrong task type was chosen. A tool-calling problem framed as question answering will train, evaluate, and disappoint. Structured output needs a structured task type.
How do you fix each cause?
Work top-down; the fixes get more expensive as you go.
| Cause | Fix | Cost |
|---|---|---|
| Teacher cannot solve the task | Revise the task description to be more specific | Minutes |
| Vague description | Add explicit output format, edge cases and negative examples | Minutes |
| Contradictory seed labels | Download teacher predictions, read the errors, resolve the conflicts | An hour |
| Wrong task type | Re-frame and re-upload | A rerun |
| Under-sized student | Move up one size band and retrain | A training run |
| Genuinely open-ended task | Do not distil; keep the LLM | — |
The prediction dump is the single most useful debugging artefact. distil model download-teacher-evaluation-predictions <model-id> writes per-example predictions as JSON Lines; twenty rows of it usually distinguishes “the teacher is weak” from “my labels disagree with each other.”
Only after those are exhausted is it worth changing models. The config reference covers teacher and student parameters, and how to choose a teacher model covers the selection itself.
Can the student exceed a teacher that scores badly?
Often, yes — which is why a mediocre teacher score is not automatically a stop signal.
In our platform benchmark, the teacher scored 0.50 on the pizza tool-calling task and the distilled student reached 0.70. On SQuAD 2.0 the teacher scored 0.57 and the student 0.64. The teacher was producing useful in-domain coverage even where its own answers were unreliable, and the filtering step kept the good ones.
What this does not rescue is a teacher that fails because the task is ill-posed. A teacher scoring badly on an ambiguous label set is generating ambiguous data, and the student inherits it. The distinction is whether the teacher’s errors are random or systematic — the prediction dump tells you which. See can a small model beat its teacher for the cases where the gap goes the student’s way.
How do you prevent it?
Front-load the cheap checks.
- Pick the task type first. Match your output shape to one of the six supported types before writing any data.
- Run teacher evaluation on every new task. It is the cheapest step in the pipeline and the only one that can abort a bad run early.
- Keep a held-out set that looks like production. A test split drawn from clean examples will overstate accuracy on messy real traffic.
- Read predictions, not just scores. Aggregate metrics hide systematic errors; black-box distillation gives you the generated data to inspect, so inspect it.
- Decide the quality bar before training. If you cannot say what score would make this a success, you cannot tell a failure from a work in progress — is fine-tuning worth it is the question to settle first.