Does Base Model Accuracy Predict Fine-Tuned Performance?
No. In our four-model, seven-task benchmark the ordering nearly inverted: the model with the worst zero-shot ranks (3.43 of 4) gained the most from training and tied for first afterwards. Base leaderboards are a poor guide to which model to fine-tune.
What did the ranking actually show?
Three rankings over the same four models and the same seven tasks, from the mid-size MoE benchmark. Lower average rank is better.
| Model | Base rank | Tunability rank | Fine-tuned rank |
|---|---|---|---|
| Nemotron-3-Nano-30B | 3.43 (worst) | 1.29 (best) | 1.00 (tied best) |
| Qwen3.6-35B | 2.14 | 2.00 | 1.00 (tied best) |
| gemma-4-26b | 1.43 (best) | 2.57 | 1.86 |
| gpt-oss-20b | 1.71 | 2.57 | 1.86 |
Read the first row across. The model that looked worst before training gained the most and finished tied for first. Read the third row across for the mirror image: the best zero-shot model finished last-equal after fine-tuning.
The magnitude is worth stating too. Once tuned, Nemotron-3-Nano-30B matched or beat its own teacher — Nemotron-3-Ultra-550B, roughly 18x larger — on classification, all three function-calling tasks, and text2sql, with margins of 0.93 against 0.56 on pizza and 0.94 against 0.76 on massive-iot. The teacher kept a lead only on the two open-ended QA tasks, by 0.04 to 0.07.
How were the ranks measured?
By ranking within each task, then averaging. Every model was fine-tuned with the same pipeline and the same hyperparameters: 4 epochs, learning rate 5e-5, LoRA rank 64, roughly 10,000 teacher-generated synthetic examples per task, evaluated on held-out test sets.
Two details keep the numbers honest. Scores are noisy, so two models are treated as tied on a task when they fall within one standard deviation of each other — near-ties are not scored as wins. And the reported figure is an average rank with a standard error, not a raw accuracy, which is why you cannot read a percentage off the table above.
The tasks span three types: classification scored with accuracy, three function-calling tasks scored with staged tool call, and three question-answering tasks scored with a reference-free LLM judge. The teacher for synthetic data generation was held constant across all four students.
Does the same pattern hold for small models?
Yes, in the same direction and with a weaker inversion. Our 12-model benchmark across eight tasks found:
| Ranking | Winner | Average rank |
|---|---|---|
| Best base (zero-shot) | Qwen3-8B | 1.75 |
| Best after fine-tuning | Qwen3-4B-Instruct-2507 | 2.25 |
| Most tunable | Llama-3.2-1B-Instruct | 3.44 |
Three different winners for three different questions, from one experiment. The base-performance leader did not win after fine-tuning, and the most tunable model — a 1B — appears in neither of the other two top spots. The study’s own summary is that tunability “inverts the size hierarchy”: smaller models start weaker and gain most.
When does base accuracy still matter?
In two situations, and they are both real.
When you are not going to fine-tune. If you are shipping a prompted model, base performance is the only number that describes it. That is why the 12-model study recommends Qwen3-8B for the “no fine-tuning possible” row.
As a feasibility floor for the teacher. Teacher accuracy is a different quantity from student base accuracy, and it does predict outcomes. The teacher evaluation docs put it directly: if the teacher can solve your task the student can learn it, and the teacher’s accuracy is the first approximation of what to expect.
Which teacher that is varies by study: the four-model benchmark above used Nemotron-3-Ultra-550B, and the 12-model benchmark used openai/gpt-oss-120b — 117B total, 5.1B active — which is still the documented default in the config reference. Recent distil labs runs more often set teacher_model_name: zai.glm-5, and which teacher model should you pick has the measured comparison.
What should you do instead of reading base leaderboards?
Measure two candidates on your own data. Run teacher evaluation once, then fine-tune the two students you would actually be willing to deploy — usually adjacent size tiers — at platform defaults so the results are comparable to the published benchmarks, and read the metrics.
The corollary is liberating rather than annoying: if base rank does not predict fine-tuned rank, you are free to choose your base model on deployment constraints — memory, latency, task eligibility — and let training close the accuracy gap. That is the argument behind what size model do you need, and it is why fine-tuning is usually worth it even when the base model looks unpromising. Mechanism is in knowledge distillation for LLMs.