What Size Model Do You Need?
Size is decided by your hardest deployment constraint, not by your task. Once you have fine-tuned, models from 270M to 8B land closer together than their parameter counts suggest — so pick the smallest tier that satisfies the constraint, train it, and only move up if the metrics say to.
Which constraint should drive the decision?
The one you cannot negotiate. Work down this table until a row matches something in your requirements document, and take that tier.
| Binding constraint | Tier | Candidate students |
|---|---|---|
| Must run in a browser, on a microcontroller, or inside a mobile app bundle | 135M-350M | SmolLM2 135M, Gemma 3 270M, FunctionGemma 270M, LFM2.5 350M |
| Must run on a CPU or a small NPU with no accelerator budget | 350M-1B | LFM2.5 350M, Qwen3 0.6B, Gemma 3 1B |
| Must fit a 4 GB GPU at BF16 | ≤1.5B | Llama 3.2 1B, LFM2.5 1.2B |
| Must fit a 4 GB GPU quantized to 4-bit | ≤6B | Qwen3 4B, Gemma 3 4B |
| No hardware ceiling, want the best measured fine-tuned accuracy | 4B | Qwen3 4B Instruct 2507 |
| No fine-tuning possible at all | 8B | Qwen3 8B |
The 4 GB rows are derived arithmetic, not measurements — the full working is in which SLM fits in 4GB of VRAM. The bottom two rows come from our 12-model benchmark, where Qwen3-4B-Instruct-2507 took first place after fine-tuning (average rank 2.25) and Qwen3-8B took first before fine-tuning (1.75).
What does each size tier actually give you?
Roughly: capability at the low end is bounded by task shape, and at the high end by what you are willing to serve.
Under 500M buys you deployment options nothing else offers. It costs you generality — LFM2.5-350M’s card says plainly the model “is not recommended for knowledge-intensive tasks and programming.” Tasks with a fixed output space work; open-ended ones do not.
500M to 2B is the common production band. Enough capacity for large label sets and multi-turn tool schemas, small enough to serve cheaply. This is where the docs tell you to start.
3B to 4B is where fine-tuned accuracy peaks in our data. Qwen3-4B-Instruct-2507 reports 4.0B total and 3.6B non-embedding parameters with a 262,144-token native context.
8B and up is mainly for the case where you will not fine-tune. Its advantage is base performance, and base performance is the thing fine-tuning erases.
Does a bigger student always win?
No, and the counter-examples are not marginal. In the same 12-model study, Qwen3-4B (rank 2.25) finished ahead of the larger Qwen3-8B (2.75) after fine-tuning. In our mid-size MoE benchmark, the model with the worst zero-shot scores of four ended up tied for best after training.
The most extreme case is at the bottom of the range: a fine-tuned FunctionGemma 270M reached 96.04% on shell-command tool calling against a 120B teacher’s 97.03%. On a bounded task, 445x fewer parameters cost about one point.
What size does still buy reliably is tolerance for a bad dataset. Larger students need less data to reach the same place. If your seed set is thin, size is one way to compensate — training data quality is the better one.
What mistakes do people make when sizing?
Sizing from base-model leaderboards. Zero-shot rankings do not survive fine-tuning; this is measured, not theoretical, and the evidence is in does base-model accuracy predict fine-tuned performance.
Sizing for a task you do not have. “We might need long documents later” turns a 600M deployment into a 4B one for a requirement nobody has written down.
Forgetting task eligibility. Tool-calling tasks only accept Qwen3, Qwen3.5, Llama 3, LFM2/LFM2.5, FunctionGemma, and Gemma 4 students, per the supported models catalog. Sizing correctly into an ineligible family means starting over.
Budgeting weights and forgetting the KV cache. A model that fits at load will not fit at a long context.
Treating one training run as the answer. Confidence intervals in our rankings routinely overlap. One run tells you less than you think.
How do you check whether you sized it right?
Run teacher evaluation before you train anything. The teacher’s score on your test set is the first approximation of what any student can reach — if the teacher fails, the task description or the data is the problem, and no parameter count fixes it.
Then train the smallest candidate at platform defaults, read the evaluation metrics, and move up exactly one tier if you are short. The docs give the same rule: start with a Qwen, check the metrics, and only move up if results fall short. Two runs at adjacent sizes cost far less than a wrong deployment, and few-shot fine-tuning from 10 examples explains why the first run can start from very little data.