What Is a Small Language Model?
A small language model (SLM) is a language model with few enough parameters to fine-tune and serve on commodity hardware — in practice roughly 100M to 10B parameters. The name describes deployment economics, not architecture: an SLM is the same transformer stack as a frontier model, trained smaller.
What counts as a small language model?
There is no official threshold, so the useful definition is operational: a model is “small” if a single GPU, or often a single CPU, can serve it at production latency. That puts the boundary somewhere below 10B parameters today.
The distil labs model catalog makes the band concrete. Its student models run from 135M to 9B parameters, and every one of them is a model you download, quantize, and run yourself.
Which small language models are actually available?
Five families cover most of the field. The table below lists one representative size from each, with the figures its own model card states.
| Family | Example | Parameters | Context | Vendor |
|---|---|---|---|---|
| SmolLM2 | SmolLM2-135M-Instruct | 135M | 8K | Hugging Face |
| Gemma 3 | gemma-3-270m-it | 270M | 32K | |
| LFM2.5 | LFM2.5-350M | 350M | 32K | Liquid AI |
| Llama 3.2 | Llama-3.2-1B-Instruct | 1.23B | 128K | Meta |
| Qwen3 | Qwen3-1.7B | 1.7B (1.4B non-embedding) | 32K | Alibaba |
Note how much of a tiny model is embedding table rather than transformer: Qwen3-1.7B reports 1.4B non-embedding parameters, meaning roughly 300M of its weights are vocabulary lookup. At the 135M end, that ratio gets worse, which is one reason very small models plateau early on open-ended generation.
How is an SLM different from a large language model?
The difference that matters is generality, not quality. A large model is trained to be adequate at everything; a small model has to spend its capacity on something. Untuned, that shows up as weaker instruction following and thinner world knowledge — Liquid AI says outright on the LFM2.5-350M card that the model “is not recommended for knowledge-intensive tasks and programming.”
Once you fine-tune on a bounded task, the comparison inverts. In our 12-model benchmark, a fine-tuned Qwen3-4B matched or exceeded the 120B-parameter teacher that generated its training data on 7 of 8 benchmarks, and beat it by 19 points on SQuAD 2.0.
When does a small model matter?
An SLM is the right choice when your task is narrow, high-volume, or has to run somewhere a frontier API cannot go.
- Narrow tasks. Classification, extraction, and tool calling have a fixed output space. Generality buys you nothing there.
- High volume. Per-request cost scales with parameters actually computed, so a 1B model at scale is a different budget line from an 8B one.
- Data residency and offline use. On-device and on-premise deployment is only possible if the weights are small enough to ship.
The trade-off is real: an SLM will not answer questions outside the task you trained it for, and it will not degrade gracefully when you ask it to. If your workload is genuinely open-ended, keep the large model. The honest version of this argument is in is fine-tuning worth it.
What terms should you know alongside this one?
Four terms come up constantly once you start working with small models.
| Term | What it means here |
|---|---|
| Student | The small model you fine-tune and deploy |
| Teacher | The large model that generates and validates training data |
| Distillation | Transferring a teacher’s task behaviour into a student |
| Quantization | Storing the same parameters at lower precision to shrink the model |
Distillation and quantization are often confused because both produce a smaller artefact. They compose rather than compete — see distillation vs quantization for where each one applies, and knowledge distillation explained for the mechanism.
Where should you start?
Start at the size you can afford to serve, not the size you think you need. Sizing guidance is in what size model do you need, and the practical training path is in how to fine-tune a small language model.