When Not to Use a Small Language Model
Don’t train one if the task isn’t defined, if the requirements change weekly, if the volume is too low to notice on a bill, if you need broad general capability, or if no large model can solve the task either. Any one of these is enough to make the project a bad use of your time.
What are the five disqualifying conditions?
Check all five before you write a job description. Each has a cheap test.
| Condition | Test | If it fails |
|---|---|---|
| Task is undefined | Two reviewers label 20 examples independently | Fix the definition first |
| Requirements unstable | Has the spec changed in the last month? | Stay on prompts |
| Volume too low | Does inference show up on your bill? | Use an API |
| Needs broad capability | Is the input distribution bounded? | Use a large model |
| No teacher can solve it | Run teacher evaluation | Rethink the task |
The rest of this article is what each one looks like in practice, because the tests are easy to wave through.
Is your task actually defined?
If two competent reviewers disagree on how to label the same example, you don’t have a task yet, you have an opinion, and training on it produces a model that has learned the disagreement.
The symptom shows up as a metrics ceiling nobody can explain. Accuracy plateaus in the seventies, every hyperparameter change moves it by noise, and the errors look defensible when you read them. That’s label inconsistency, and no amount of data or compute fixes it. When does distillation fail ranks this first among causes.
Test it before anything else: take 20 examples, have two people label them separately, and count disagreements. Above roughly one in ten, stop and write a better spec. That work isn’t wasted, since it’s the same work you’d need for a prompt, an eval set, or a hand-written rule engine.
Are your requirements stable enough?
A trained model is a snapshot. If the label set, the tool schema, or the output format changes weekly, you’re shipping a retraining pipeline rather than a model, and the pipeline will lag your requirements permanently.
Concrete cases where the answer is “not yet”:
- A product still in discovery, where the categories are being invented
- A tool schema under active development
- A compliance rule set awaiting a regulator’s decision
- Anything where a stakeholder is still negotiating the output format
Prompts are the right tool here precisely because they’re cheap to change. Revisit once the definition has held still for a month or two. Is fine-tuning worth it covers what “held still” looks like.
Is your volume high enough to matter?
If inference cost doesn’t appear on your bill, the economic argument for a small model doesn’t exist, and the remaining arguments have to stand on their own.
Those arguments do sometimes stand alone. Data residency, offline operation, and hard latency budgets are real requirements that survive at any volume. The industrial RAG case is a low-volume deployment where a segmented network, not cost, is the driver.
But if none of those apply and you’re handling a few thousand requests a month, the honest answer is to call an API and spend the engineering time elsewhere; when self-hosting beats an API is where that line sits. The distil labs inference benchmark says as much in its own conclusions: route to a frontier API when volume is low enough that it doesn’t show up on your balance sheet.
Do you need broad general capability?
Small models trade breadth for depth. That’s the mechanism rather than a temporary limitation. Kaplan and colleagues established that capability scales predictably with parameters, and specialising a small model doesn’t repeal that for tasks outside its specialisation.
Don’t use a small model when:
- The input distribution is genuinely unbounded (general assistance, arbitrary code help)
- Free-form prose quality is the product, not a byproduct
- The task needs world knowledge you can’t supply as context
- Users will ask it anything and expect a reasonable answer
Even the strongest published argument for small models is narrower than its title suggests. Belcak and colleagues argue small models suit agentic workloads because agent nodes are narrow and repeatable, which is a claim about task shape rather than about small models being generally sufficient. The distil labs 8-dataset benchmark shows the boundary empirically: fine-tuned models ranked first on 4 of 8 tasks but placed 5th of 11 on docstring generation, where part of the output is a free-form plain-language description.
Can a large model solve it at all?
If a frontier-scale model can’t solve your task from a prompt with your examples, a small one distilled from it won’t either. It’s the cheapest disqualifying test and the one most often skipped.
This is why teacher evaluation runs before training rather than after. The RAG tutorial states the logic plainly: if the teacher can’t solve your task, small models won’t either, so stop there instead of two hours later.
A failed teacher evaluation isn’t usually a capability verdict. It usually means the task description is ambiguous, the test set has wrong reference answers, or the metric is punishing correct paraphrases. All three are fixable, and all three are cheaper to fix before a training run than after one.
What should you do instead?
Match the fix to the condition rather than reaching for the same one every time.
| Blocked by | Do this instead |
|---|---|
| Undefined task | Write the spec, label 50 examples, measure agreement |
| Unstable requirements | Prompt, and keep the good input-output pairs |
| Low volume | Use a hosted API; revisit at scale |
| Needs breadth | Use a large model, or split the task into narrow parts |
| Teacher fails | Fix the task description, test set, or metric |
The split option in row four is the one worth trying before giving up. “Answer any question about this domain” rarely fine-tunes well; the same work broken into extraction, classification, and targeted generation often does. Do you actually need a bigger model works through that decomposition, and task selection lists the shapes a task has to fit into.