Does the Student Need the Same Architecture as the Teacher?
No. Knowledge is transferred through generated text, not through weights or logits, so the student can come from a completely unrelated model family. On the distil labs supported models catalog the student and teacher lists share no family at all — the pairing is cross-architecture by design.
Why does the architecture not need to match?
Because the teacher’s only output is data. It writes input-output pairs; the student is fine-tuned on them like any other supervised dataset. Nothing in that loop touches the teacher’s internals.
The survey on knowledge distillation of LLMs splits the field into white-box methods, which read the teacher’s parameters or output distributions, and black-box methods, which see only generated text. Text-level distillation is squarely black-box. That is also why a hosted teacher works at all — you never need weight access.
The original formulation in Hinton et al. (2015) sat on the other side of that line: the student matched softened class probabilities from the teacher, which requires the two models to share an output space. Modern LLM distillation dropped that requirement, and dropped the architectural constraint with it.
When would architecture matter?
In three cases, none of which apply to a text-level pipeline.
| Distillation style | Needs matching architecture? | Why |
|---|---|---|
| Logit / response-based | Shared vocabulary and tokenizer | The student regresses onto the teacher’s token distribution |
| Feature or hidden-state matching | Comparable layer widths and depth | Losses are computed between intermediate activations |
| Speculative decoding pairs | Same tokenizer | Draft and target models must agree token-for-token |
| Text-level (synthetic data) | No | The interface is plain text |
If you are hand-rolling a logit-matching pipeline, tokenizer mismatch is a real blocker and people usually solve it by distilling within a family. If you are generating synthetic data, it is a non-issue.
What pairings do people actually use?
Cross-family ones, in every published run we have.
| Student | Teacher | Task | Source |
|---|---|---|---|
| Qwen3-1.7B | zai.glm-5 |
Restaurant booking from noisy traces | traces-vs-synthetic benchmark |
| Qwen3-8B | deepseek.v3.1 |
pytest generation | pytest generator |
| LFM2.5-350M | GPT-oss-120B | Multi-turn tool calling, three benchmarks | LFM2.5 benchmark |
| FunctionGemma 270M | 120B teacher | Multi-turn tool calling, three benchmarks | FunctionGemma benchmark |
| Qwen3-0.6B | openai.gpt-oss-120b |
Smart home tool calling | dlt trace case study |
| Llama3 3B | Llama3 70B | Ten classification, QA and tool-calling datasets | platform benchmark |
Six runs, five teacher families, and only the Llama3 pair shares a family — that one was a benchmarking control rather than a requirement. Qwen students appear against three different teachers, which is the point: the student is chosen for the deployment target and the teacher for accuracy, independently.
The LFM2.5 case is the sharpest illustration. Liquid’s hybrid architecture uses multiplicative gates and short convolutions in place of most attention layers, so it is not even a standard transformer, and it still absorbed behaviour from a transformer teacher — reaching 98.0% tool call equivalence against the teacher’s 97.03% on the Gorilla shell task.
Does mixing families cost you accuracy?
Not in a way the numbers show. What predicts the outcome is the student’s tunability, not its resemblance to the teacher.
Compare two students trained from the same 120B teacher on the same three tasks. Base FunctionGemma 270M scored 9.9–38.8% tool call equivalence and finished at 90.9–96.7%; base LFM2.5-350M scored 34.5–63.2% and finished at 95.9–98.0%. Two unrelated architectures, one teacher, both landing in production range.
Swapping the teacher family does not change the picture either. A Qwen3-1.7B student distilled from GLM-5 scored 0.844–0.866 LLM-as-a-judge across five trace-corruption scenarios, against 0.835 for the GLM-5 teacher itself — a Qwen student passing a GLM teacher.
Our mid-size benchmark pushes this further: base-model rank did not predict fine-tuned rank, and the model with the worst zero-shot scores ended up tied for best after fine-tuning. Architecture family is a weaker signal than most people assume — best small language model for fine-tuning covers how to pick on that basis.
What should you match instead?
Task compatibility and deployment target.
- Task type. Tool-calling tasks accept only a subset of students — Qwen3, Qwen3.5, Llama 3, LFM2/LFM2.5, FunctionGemma and Gemma 4 — and only teachers marked tool-calling capable.
- Runtime. Pick a student with the export format your serving stack wants, then check local deployment.
- Size budget. Latency and memory are yours to choose; the teacher has no say.
Everything else about the pairing is free. For the roles each side plays, see what is a teacher model and what is a student model.