We fine-tuned four comparable mid-size mixture-of-experts (MoE) models on seven tasks and ranked them before and after fine-tuning. The short version:
- Base rank does not predict fine-tuned rank. The model with the worst zero-shot scores ends up tied for the best after fine-tuning.
- Best after fine-tuning: Nemotron-3-Nano-30B and Qwen3.6-35B tie for first (average rank 1.00 of 4), and the Nemotron is the smaller of the two, with both running only 3B active. Neither is clearly beaten on any of the seven tasks.
- Most improved by fine-tuning: Nemotron-3-Nano-30B. It has the weakest base scores of the four but the largest gain.
- A fine-tuned Nemotron-3-Nano-30B matches a model ~18x its size. With only 3B active parameters, the tuned student matches or beats its own 550B teacher on classification, tool-calling, and text2sql.
The takeaway: fine-tuning matters more than base-model results, and Nemotron-3-Nano-30B is a great basis for fine-tuning.
Which mid-size model should you fine-tune?
If you are building a product feature on a language model, sooner or later you pick a base model to fine-tune. The obvious move is to take whichever mid-size model tops the zero-shot leaderboard and tune that one. We ran this benchmark to check whether that instinct holds, and it does not.
We fine-tuned four comparable mid-size MoE models, gemma-4-26b, gpt-oss-20b, Nemotron-3-Nano-30B, and Qwen3.6-35B, on seven tasks spanning classification, function calling, and question answering. For each model we measured the base (zero-shot) score and the score after fine-tuning, ranked the four on every task, and averaged the ranks across the seven tasks; lower is better. The full setup is in the methodology section below.
We came in with four questions:
- Which model produces the best results after fine-tuning?
- Which model improves the most from fine-tuning?
- Which model is best before fine-tuning?
- Can the best student match the much larger teacher it learned from?
What is distil labs
distil labs is a platform that fine-tunes task-specific small language models automatically. Most teams use it to swap the general-purpose LLM in their agentic system for a smaller, purpose-built one: same quality on the narrow task, but around 80% lower cost and latency. You give it the production traces your agent already collects, it generates synthetic training data from them, trains a small model that matches frontier-model quality on the task, and deploys it to an OpenAI-compatible endpoint. Setup takes under 30 minutes and the model is usually ready within one to three days. This benchmark is the same machinery pointed at one question: when you fine-tune, how much does the base model you start from actually matter?
Q1: Which model produces the best fine-tuned results?
After fine-tuning, two models tie for first: Nemotron-3-Nano-30B and Qwen3.6-35B, both at an average rank of 1.00, and neither is clearly beaten on any of the seven tasks. Nemotron-3-Nano-30B reaches that tie with fewer total parameters than Qwen3.6-35B, both running only 3B active. gemma-4-26b and gpt-oss-20b follow at 1.86, each falling behind on two tasks. The fine-tuned field is tight at the top.
Q2: Which model is the most tunable?
Tunability is how much a model improves from base to fine-tuned, and Nemotron-3-Nano-30B gains the most. It is the clear leader (average rank 1.29), ahead of Qwen3.6-35B (2.00) and gemma-4-26b and gpt-oss-20b (both 2.57), with the largest gains on the tasks where it starts furthest behind.
Q3: How do the base (zero-shot) models rank?
Before any fine-tuning, the order is almost reversed. gemma-4-26b has the best base scores (average rank 1.43), then gpt-oss-20b (1.71) and Qwen3.6-35B (2.14), with Nemotron-3-Nano-30B last (3.43). Put the three rankings together and the pattern is hard to miss: the model with the worst zero-shot scores is the one that improves the most and ends up tied for the best after fine-tuning. Base-model rank does not predict fine-tuned rank.
Q4: Can the best student match its teacher?
The students learned from a much larger teacher, Nemotron-3-Ultra-550B (55B active). Nemotron-3-Nano-30B is roughly 18x smaller and far cheaper to serve, since only 3B parameters are active per token. Once fine-tuned, it matches or beats this teacher on classification, on all three function-calling tasks, and on text2sql, with the widest margins on tool-calling where the teacher is weak (0.93 vs 0.56 on pizza, 0.94 vs 0.76 on massive-iot). The teacher still leads on the two open-ended QA tasks, resume-roast and roman-empire, but by small margins of 0.04 to 0.07.
Methodology
Models. We compared four mid-size MoE models: gemma-4-26b, gpt-oss-20b, Nemotron-3-Nano-30B, and Qwen3.6-35B. Each was evaluated base (zero-shot) and after fine-tuning with the distil labs pipeline.
Tasks and metrics. Seven tasks across three types, each scored with the metric that fits it:
- Classification (ai-slop): accuracy.
- Function calling (massive-iot, pizza, voice-assist): staged tool call.
- Question answering (resume-roast, roman-empire, text2sql): a reference-free llm-as-a-judge, with GLM5.1 as the judge.
Teacher and training. The teacher, used to generate the synthetic training data, was Nemotron-3-Ultra-550B. The pipeline is the standard distil labs loop: start from a short task description and a handful of seed examples, confirm the base model is not already good enough, have the teacher generate about 10,000 in-domain synthetic examples, filter them with rule-based validators, and LoRA fine-tune the student on the seed plus curated synthetic data. Each model is then evaluated on a held-out test set. Full hyperparameters are in Training details.
Ranking. We rank the four models on each task, then average the ranks across the seven tasks; lower is better. Because scores are noisy, we treat two models as tied on a task when their scores are within one standard deviation of each other, so near-ties are not scored as wins or losses. We report the average rank with its standard error.
How to choose
| If you want | Pick | Why |
|---|---|---|
| The best result after fine-tuning | Nemotron-3-Nano-30B or Qwen3.6-35B | Tied for first, never clearly beaten |
| The biggest gain from fine-tuning | Nemotron-3-Nano-30B | Most tunable; weakest base to tied-best |
| The best model without fine-tuning | gemma-4-26b | Best zero-shot scores |
Next steps and limitations
We aimed for three runs per model-task combination, but some runs failed, so not every combination has three. Where a standard deviation was missing, we imputed it as the average standard deviation of the other models on the same dataset. Small differences should be read as within noise; the tie rule already folds those into shared ranks.
This is also a small benchmark: four models and seven tasks. Next we plan to complete the runs, add more models and tasks, and report measured cost and latency alongside accuracy. If there is a model or task you would like us to add, let us know.
Training details
All models were fine-tuned with the same configuration: 4 epochs, learning rate 5e-5, linear scheduler, LoRA rank 64, on roughly 10,000 synthetic examples per task generated by the teacher, and evaluated on held-out test sets never seen during training.
All seven datasets, both the seed data and the synthetic data the students were fine-tuned on, are available at distil-labs/distil-moe-benchmark-datasets for reproduction.
Conclusion
Across four comparable mid-size MoE models, the base leaderboard told us little about what happens after fine-tuning. The model with the weakest zero-shot scores, Nemotron-3-Nano-30B, gained the most and tied for the best result, and once tuned it matched a teacher roughly 18x its size on most tasks. Fine-tuning matters more than base-model results, and Nemotron-3-Nano-30B is a great basis for fine-tuning.
If you have a narrow, high-volume task in your product, you do not need the biggest model. Start with a short task description and a handful of examples on the distil labs platform, and we will take it from there.
Related reading: which small language model is best for fine-tuning and we benchmarked 12 small language models across 8 tasks.
distil labs trains task-specific small language models that are cheaper and faster per request than general-purpose LLMs, with equal-or-better accuracy on bounded tasks. Docs · GitHub · Slack
Appendix: full results
Per-task fine-tuned score with rank in parentheses (lower rank is better; bold = rank 1, not clearly beaten on that task).
| Dataset | Nemotron-3-Nano-30B | Qwen3.6-35B | gemma-4-26b | gpt-oss-20b |
|---|---|---|---|---|
| ai-slop | 1.000 (1) | 1.000 (1) | 1.000 (1) | 1.000 (1) |
| massive-iot | 0.936 (1) | 0.926 (1) | 0.926 (1) | 0.894 (4) |
| pizza | 0.934 (1) | 0.940 (1) | 0.901 (4) | 0.925 (1) |
| voice-assist | 0.971 (1) | 0.971 (1) | 0.970 (2) | 0.985 (1) |
| resume-roast | 0.933 (1) | 1.000 (1) | 1.000 (1) | 0.867 (1) |
| roman-empire | 0.957 (1) | 0.978 (1) | 0.957 (1) | 0.978 (1) |
| text2sql | 0.987 (1) | 0.980 (1) | 0.960 (3) | 0.920 (4) |
| Avg rank | 1.00 | 1.00 | 1.86 | 1.86 |
Per-dataset scores for all four models, base and after fine-tuning:
