We added NVIDIA’s Nemotron 3.5 Lightning to our mid-size mixture-of-experts (MoE) fine-tuning benchmark, alongside the four models from our previous round, and ran all five across six tasks. The short version:
- Best after fine-tuning: Nemotron 3.5 Lightning, average rank 1.17 of 5. It is clearly beaten on only one of the six tasks.
- Its base scores did not predict that. Untuned it ranks third of five.
- It beats its own 550B teacher on four of six tasks at roughly 18x fewer parameters, and trails it slightly on the other two.
The takeaway is the one we keep arriving at: fine-tuning matters more than base-model choice, and you cannot read the answer off a zero-shot leaderboard.
Which mid-size model should you fine-tune?
If you are putting a language model behind a product feature, at some point you pick a base model and tune it. The tempting shortcut is to take whichever mid-size model looks best zero-shot and start there.
We have now run that comparison twice, a month ago with Nemotron 3 Nano and now repeating it
with Nemotron 3.5 Lightning. In both cases we fine-tuned five comparable mid-size MoE models
(incl gemma-4-26b, gpt-oss-20b, Qwen3.6-35B) on six tasks spanning classification,
function calling, and question answering. For each we measured the base (zero-shot) score and
the score after fine-tuning, ranked the five on every task, and averaged the ranks; lower is
better.
What is distil labs
distil labs is a platform that fine-tunes task-specific small language models automatically. Most teams use it to swap the general-purpose LLM in their agentic system for a smaller, purpose-built one: same quality on the narrow task, but around 80% lower cost and latency. You give it the production traces your agent already collects, it generates synthetic training data from them, trains a small model that matches frontier-model quality on the task, and deploys it to an OpenAI-compatible endpoint. Setup takes under 30 minutes and the model is usually ready within one to three days. This benchmark is that same machinery pointed at one question: when you fine-tune, how much does the model you start from actually matter?
Q1: which model is best after fine-tuning?
Nemotron 3.5 Lightning finishes first at an average rank of 1.17. Qwen3.6-35B follows at 1.33, then Nemotron-3-Nano-30B and gpt-oss-20b tied at 1.50, and gemma-4-26b at 1.67.
The field is tight, and first place here means “least often behind” rather than “ahead everywhere”. No model is alone in first place on any single task. Two of the six tasks, ai-slop and massive-iot, end in a five-way tie once the noise floor is applied. Nemotron 3.5 Lightning shares first place on four more, and the only task where it is clearly beaten is text2sql, where gemma-4-26b reaches 0.900 against its 0.850. That one loss is the entire difference between its 1.17 and a perfect 1.00.
Q2: how do the base models rank?
Before any fine-tuning, gemma-4-26b and gpt-oss-20b share first at 1.83, Nemotron 3.5 Lightning sits third at 2.17, Qwen3.6-35B fourth at 2.67, and Nemotron-3-Nano-30B is last at 3.83.
Put the two charts side by side and the ordering barely survives. The model that finishes first after fine-tuning was third before it, and the model that was last before it finishes in the middle. Base-model rank does not predict fine-tuned rank.
The clearest case is ai-slop binary classification. Nemotron 3.5 Lightning scores 0.500 there untuned, no better than a coin flip, and it is not even the worst: Qwen3.6-35B scores 0.458 and Nemotron-3-Nano-30B 0.361, against 1.000 for gemma-4-26b. After fine-tuning all five models sit at 1.000. A spread of 0.64 between the best and worst base model collapsed to nothing.
Q3: which model gains the most?
Here we will measure which model gained the most with fine-tuning, measured as the delta between the base and tuned performances; we rank the gains on all datasets and average the ranks to get the final result. Note that the gains are driven as much by where a model starts as by where it ends up.
Nemotron-3-Nano-30B gained the most with average rank 1.33, with Nemotron 3.5 Lightning second at 1.83, then Qwen3.6-35B at 2.17, gpt-oss-20b at 3.00 and gemma-4-26b at 3.50. The pattern is the same for the top two: the largest gains land on the tasks where the base model starts worst.
Q4: can the student match its teacher?
The students learned from Nemotron-3-Ultra-550B, roughly 18x larger than the 30B student. Both are mixture-of-experts models, so the gap in what you actually serve is the same 18x: 55B active parameters per token for the teacher against 3B for the student. After fine-tuning, Nemotron 3.5 Lightning beats that teacher on four of the six tasks and trails it slightly on the other two.
| Task | Teacher 550B | Nemotron 3.5 Lightning (tuned) | Difference |
|---|---|---|---|
| pizza | 0.555 | 0.924 | +0.369 |
| massive-iot | 0.756 | 0.934 | +0.178 |
| ai-slop | 0.917 | 1.000 | +0.083 |
| voice-assist | 0.976 | 0.994 | +0.018 |
| text2sql | 0.860 | 0.850 | −0.010 |
| resume-roast | 1.000 | 0.967 | −0.033 |
The margins are widest on function calling, where the teacher is weakest, and the two losses are both small enough to sit inside the noise on their test sets. A student can beat its teacher because the synthetic data is filtered before training, which strips the teacher’s mistakes, and because the student spends all of its capacity on one task instead of everything.
Methodology
Models. Five mid-size MoE models: gemma-4-26b,
gpt-oss-20b,
Nemotron-3-Nano-30B,
Qwen3.6-35B, and Nemotron-3.5-Lightning. Each was
evaluated base (zero-shot) and after fine-tuning with the distil labs pipeline.
Tasks and metrics. Six tasks, each scored with the metric that fits it:
- Classification (ai-slop): accuracy.
- Function calling (massive-iot, pizza, voice-assist): staged tool call.
- Question answering (resume-roast, text2sql): llm-as-a-judge with the reference answer
provided, judged by
gpt-oss-120b.
Teacher and training. The teacher was Nemotron-3-Ultra-550B. The pipeline is the standard distil labs loop: start from a short task description and a handful of seed examples, confirm the base model is not already good enough, have the teacher generate about 10,000 in-domain synthetic examples, filter them with rule-based validators, and LoRA fine-tune the student on the seed plus curated synthetic data. Each model is then evaluated on a held-out test set.
Ranking. We rank the five models on each task, then average the ranks; lower is better. Two models tie on a task when their scores are within half the sum of their standard deviations, so near-ties are not scored as wins or losses. We report the average rank with its standard error.
Three changes that make this round more reliable
All three came out of looking hard at what the previous round could and could not support. All three make it harder for us to claim a difference, not easier.
1. We ran every model more than once
Last time most cells rested on a single training and evaluation run, and where a standard deviation was missing we borrowed one from another dataset. This round every model was re-run twice across all tests. Ideally we would run many more evals, but having two instead of one is an improvement, since single-run rankings are difficult to read into: some of the difference can be just due to noise.
2. We dropped a task that fine-tuning could not move
The previous round included roman-empire, an open-ended QA task. Averaged across all five students, its base score is 0.912 and its fine-tuned score is 0.925. A benchmark task that cannot separate a tuned model from an untuned one is not measuring the thing the benchmark exists to measure.
3. We stopped reporting differences the test sets cannot resolve
A score measured on n examples cannot be known more precisely than sampling allows. Several of the standard deviations we carried into the last round sat well below their own limit, some because they were imputed from other datasets rather than measured. We now enforce that limit explicitly.
For every cell, meaning one model on one task, base or tuned, we take whichever is larger: the standard deviation measured across runs, or a sampling floor derived from the size of the test set.
sigma_used = max(sigma_measured, sigma_floor)
p * n + 1 p_adj * (1 - p_adj)
p_adj = ---------------- , sigma_floor = sqrt( ------------------- )
n + 2 n + 2Here p is the observed score and n the number of scored predictions in the test set. The
floor is the standard error of a binomial proportion with the Agresti-Coull adjustment: the
plain form sqrt(p(1-p)/n) returns exactly 0 at a perfect score, which would assert that 36
out of 36 is measured without error. Every model scores 1.000 on ai-slop after fine-tuning, so
without the adjustment those cells would separate on nothing at all. With it, the floor there
is 0.026. Across the six tasks the floor binds in 51 of the 60 cells, which is another way of
saying that most of the spread we reported last time was smaller than the test sets could
resolve.
The ranking rule is otherwise unchanged from the previous round: two models tie on a task when
their scores fall within (σ_a + σ_b) / 2 of each other. For the tunability question the base
and tuned standard deviations are floored first and then combined as sqrt(σ_base² + σ_tuned²), so the floor propagates into the gain.
How to choose
| If you want | Pick | Why |
|---|---|---|
| The best result after fine-tuning | Nemotron 3.5 Lightning | First at 1.17, clearly beaten on one task of six |
| The biggest gain from fine-tuning | Nemotron-3-Nano-30B | Most tunable at 1.33, from the weakest base |
| The best model without fine-tuning | gemma-4-26b or gpt-oss-20b | Joint best zero-shot at 1.83 |
Limitations
The tuned field is tight, and most of it is tied. First place at 1.17 means least often clearly behind, not clearly ahead. No model is alone in first place on any single task, and two of the six tasks end in a five-way tie.
text2sql is the weakest tuned result, at 0.850. That is roughly one query in seven wrong, so it is not a task you would ship unsupervised on these numbers.
Small test sets. Three of the six tasks have fewer than 60 scored examples, and resume-roast has 15. That is why the sampling floor does so much work, and why a handful of examples can move a score by more than the gap between two models.
Training details
All models were fine-tuned with the same configuration: 1 epoch, bf16 LoRA with rank 64 on attention projections, on roughly 10,000 synthetic examples per task generated by the teacher, and evaluated on held-out test sets never seen during training. Nemotron 3.5 Lightning carries the same parameter counts as Nemotron-3-Nano: 30B total, 3B active.
All datasets, both the seed data and the synthetic data the students trained on, are available at distil-labs/distil-moe-benchmark-datasets for reproduction.
Conclusion
Across five comparable mid-size MoE models, the zero-shot leaderboard again told us little about what happens after fine-tuning. The model that ends up first was third before tuning and could not beat a coin flip on binary classification. Once tuned it is clearly beaten on one task out of six, and it matches or beats a teacher roughly 18x its size on four of them.
If you have a narrow, high-volume task in your product, you do not need the biggest model, and you should not spend long agonising over which mid-size model to start from. Start with a short task description and a handful of examples on the distil labs platform, and we will take it from there.
Related reading: does base-model accuracy predict fine-tuned performance and we benchmarked 12 small language models across 8 tasks.
distil labs trains task-specific small language models that are cheaper and faster per request than general-purpose LLMs, with equal-or-better accuracy on bounded tasks. Docs · GitHub · Slack
Appendix: full results
Per-task fine-tuned score with rank in parentheses, as the mean of two to four runs. Bold marks rank 1, meaning not clearly beaten on that task.
| Task | Nemotron 3.5 Lightning | Qwen3.6-35B | Nemotron-3-Nano-30B | gemma-4-26b | gpt-oss-20b |
|---|---|---|---|---|---|
| ai-slop | 1.000 (1) | 1.000 (1) | 1.000 (1) | 1.000 (1) | 1.000 (1) |
| massive-iot | 0.934 (1) | 0.909 (1) | 0.909 (1) | 0.925 (1) | 0.928 (1) |
| pizza | 0.924 (1) | 0.933 (1) | 0.934 (1) | 0.879 (5) | 0.941 (1) |
| voice-assist | 0.994 (1) | 0.982 (2) | 0.979 (3) | 0.985 (1) | 0.991 (1) |
| resume-roast | 0.967 (1) | 0.967 (1) | 0.950 (1) | 1.000 (1) | 0.911 (2) |
| text2sql | 0.850 (2) | 0.820 (2) | 0.825 (2) | 0.900 (1) | 0.780 (3) |
| Avg rank | 1.17 | 1.33 | 1.50 | 1.67 | 1.50 |
Per-dataset scores for all five models, base and after fine-tuning:
