Model Routing vs a Single Specialised Model
A router is a separate component that inspects each request and decides which model answers it. A single specialised model handles everything itself. There’s a third option: one specialised model that handles most requests and escalates the rest, with the escalation decision trained into the model rather than bolted on.
What are the three architectures?
They differ in where the decision lives and how many components you have to keep in sync.
| Router in front | Single specialised model | Trained deferral | |
|---|---|---|---|
| Components to operate | Router plus 2+ models | One model | One model plus a fallback |
| Where the decision lives | A separate classifier or threshold | Nowhere, no decision | Inside the model’s output |
| Decision input | Usually the request text alone | None | The model’s full view of the request |
| Tuning surface | Thresholds, router training data | None | Retrain the model |
| Failure mode | Router misroutes silently | Hard requests answered wrongly | Over- or under-deferring |
| Cost profile | Blended, tunable | Flat and low | Blended, set by deferral rate |
What is the honest case for a router?
It’s a stronger case than small-model vendors usually admit, and the research behind it is serious.
RouteLLM trains router models on human preference data to choose between a stronger and a weaker LLM per request, reporting “cost reduction by over 2 times in certain cases without compromising the quality of responses” along with transfer across different model pairs. FrugalGPT generalises the idea into cascades and reports matching GPT-4 performance with “up to 98% cost reduction”.
Routers also have genuine structural advantages:
- Model-agnostic. You can swap either endpoint without retraining anything.
- No training data required. A heuristic router ships today; a fine-tuned model doesn’t.
- Legible. The routing rule is a thing you can read, version, and argue about in review.
- Right for heterogeneous traffic. If requests genuinely span unrelated task types, something has to dispatch them, and that something is a router.
If your traffic is a mix of unrelated jobs, stop here. You want a router, and the rest of this article is about a different situation.
Where do routers get expensive to own?
They get expensive when the router’s decision needs the same understanding as the task itself. A difficulty classifier looking only at request text is making a judgement it doesn’t have the context to make.
The distil labs deferral post is direct about the ongoing cost: a separate classifier or a confidence threshold off logprobs is “extra infrastructure to tune and keep in sync with your policy”, and confidence heuristics are poorly calibrated to start with. Every policy change becomes a change in two places.
The objection is maintenance rather than accuracy. Routers work, they’re just another component with its own training data, its own drift, and its own on-call story.
What does trained deferral do differently?
It moves the decision inside the model, expressed as an ordinary tool call. The small model handles the request or emits defer_to_larger_model(reason), and the orchestrator honours whichever it gets. There’s no second model and no threshold.
The distil labs airline-support demo measured it against an all-frontier baseline on held-out turns, matched per-turn and scored by an independent GLM-5 judge:
| System | Quality (strict judge) | Quality (relaxed judge) | Frontier-model calls |
|---|---|---|---|
| Frontier model only (GLM-5) | 0.79 ± 0.03 | 0.88 ± 0.03 | 100% |
| Cascade (Qwen3-1.7B + escalation) | 0.76 ± 0.03 | 0.85 ± 0.03 | ~4% |
The gap is +0.03 ± 0.03 strict and +0.03 ± 0.02 relaxed; the confidence interval of the difference includes zero and a paired McNemar test finds it not significant under either rubric. Roughly 96% of turns are handled by the 1.7B model.
The control run is the informative part: the same untrained Qwen3-1.7B scored 0.42 on the set and deferred on 0% of turns. Knowing when to escalate is learned, not free. The training recipe adds one tool plus a guideline defining “hard” by problem structure (non-obvious eligibility, several interacting rules, multi-step arithmetic, ambiguous judgement), and the teacher marks those turns during distillation. The task type is multi-turn-tool-calling-closed-book, listed in task selection.
Which should you build?
Match the architecture to the shape of your traffic, not to the cost story; how much you can save replacing an LLM API puts numbers on that story.
- Heterogeneous traffic across unrelated tasks. Build a router. Something has to dispatch, and a router is the honest name for it.
- One task, uniform difficulty. A single specialised model, since there’s no decision to make and no reason to build machinery to make it.
- One task with a hard tail. Trained deferral, because the decision needs the model’s own read of the request.
- No training data at all yet. A heuristic router today, revisited once you have traces. Upload traces covers turning production logs into training data.
One warning that applies to all three: the traces benchmark found training directly on raw production traces scored 14–28 percentage points below curated data, and multi-turn agents compound per-turn errors, so 95% per-turn accuracy is roughly 35% of 20-turn conversations fully correct. Whichever architecture you pick, the data pipeline behind it decides whether it works, and traces vs synthetic data is where that comparison is worked out.
For choosing the model that sits in the cheap tier, see what size model do you need and qwen vs llama vs gemma for tool calling.