Your product calls a frontier model and it works, and the bill grows with traffic. When you look at where those calls actually go, most of them turn out to be the same handful of narrow jobs done over and over: classifying an incoming ticket, pulling six fields out of an invoice, deciding which tool to call next.
A general-purpose model brings capability you aren’t using for those jobs, and you pay for all of it on every call. Fine-tune a small model on one job and it only has to be good at that one thing, which is why something orders of magnitude smaller can match the frontier model on it. One classification stage we moved this way came out 68% cheaper, and accuracy went from 81% to 93% at the same time. Picking the model is the easy part.
Getting one into a system that’s already running is much harder. A frontier model in a working product is rarely a component you can unplug, because it’s the thing the architecture grew around: easy traffic and hard traffic go through the same call, the prompt holds business logic that exists nowhere else, and everything downstream is written against its output shape. Taking one job away from it changes how requests move through your product: what still calls the frontier model, what calls the small one, and which of them is in charge.
Those choices come out in three patterns, and almost every system we work on lands in one of them: the small model answers first and hands off what it can’t handle, the frontier model stays in charge and calls the small one as a tool, or a single stage of a fixed pipeline swaps over from LLM to SLM.
What moves, in all three cases
Whichever pattern you pick, most of your product doesn’t change: it keeps calling the frontier model exactly as it does today, and only one narrow job moves off it. The three differ only in what sits around that job:
Whether a small model can hold the job at all depends less on the model than on what it’s trained on, and building that training set is what distil labs automates. You describe the job in plain English and hand over 20 to 100 examples, or just the request and response logs you already keep. It’s the same process for all three patterns.
Your current frontier model then becomes the teacher: it writes the training data and it sets the bar the small model has to match. It generates around 10,000 in-domain examples, and rule-based validators throw out whatever comes back malformed, off-schema or near-duplicate. The small model is fine-tuned on what survives, then scored on data it has never seen against both your frontier model and its own untrained version, so you see the comparison before you switch anything. Training on that validated data rather than on the raw logs is what makes this hold up on messy production traffic, and it’s worth up to 26 accuracy points.
The trained model is served behind an OpenAI-compatible endpoint in every case, so the integration itself is a base URL change. What differs is who calls that URL, and when.
Pattern 1: handoff, where the small model answers first
This is the pattern people find first, because it slots in front of a call that already exists. The small model owns the front door and answers what it can, and when something is out of its depth it hands over with one explicit tool call so the frontier model can pick up the conversation. Your frontier path itself doesn’t change, it just becomes the fallback, which makes this easy to reason about and easy to roll back.
We shipped a working version of this in June: an airline customer-support bot where a fine-tuned Qwen3-1.7B handles the turns it can and escalates the rest. Held-out turns, scored by an independent GLM-5 judge, matched per turn:
| System | Quality (strict judge) | Quality (relaxed judge) | Frontier-model calls |
|---|---|---|---|
| Frontier model only (GLM-5) | 0.79 ± 0.03 | 0.88 ± 0.03 | 100% |
| Cascade (Qwen3-1.7B + escalation) | 0.76 ± 0.03 | 0.85 ± 0.03 | ~4% |
The difference is +0.03 ± 0.03 strict and +0.03 ± 0.02 relaxed, the confidence interval includes zero, and a paired McNemar test doesn’t find it significant under either rubric, so the two systems are statistically indistinguishable. The small model handles around 96% of turns, which is 25x fewer frontier calls, and at 800 input and 100 output tokens per turn the frontier model costs about USD 3,000 per million turns against USD 700 for the cascade.
The escalation is something you train rather than something you get for free: there’s no router, no confidence threshold and no difficulty classifier. During fine-tuning the teacher marks the genuinely hard turns, the small model learns to recognise them, and at runtime it emits a single defer_to_larger_model call like any other tool. The same Qwen3-1.7B without fine-tuning scores 0.42 on that set and defers 0% of the time.
Pattern 2: skill, where the big model delegates
In this pattern the frontier model never leaves the loop. It plans and it decides, and whenever there’s narrow work to do it calls a small model as a tool behind a strict output contract, the same way it would call any other function.
The argument for this one is token accounting: in the incident-response harness we published, orchestration is about 50 tokens out of roughly 1,300, so the frontier model’s thinking is cheap and nearly everything you pay for is the domain work in between. That work is also what repeats often enough to be worth training, and moving it took the cost from roughly USD 0.07 to USD 0.001 per incident, which at 10,000 events a day is USD 700 down to under USD 10.
Our PII redactor is the worked example: it ships as an OpenClaw skill whose inner model call runs on a fine-tuned 1B, so the skill file holds the procedure and the small model does the redaction. On our held-out set that 1B scores 0.81 ± 0.02 against 0.84 ± 0.03 for a 600B-class frontier model, so the gap is real and small, and whether you take it depends on the task.
Pattern 3: stage swap, where the pipeline keeps its shape
Plenty of production AI isn’t agentic at all. It’s a fixed workflow with a model call at one or two of the steps, extract then classify then generate, and moving one of those steps to a small model needs no new architecture.
It’s the least interesting pattern of the three, and it’s where our clearest results come from. The classification stage from the opening is Knowunity’s: they run it at hundreds of millions of requests a month, and moving it off Gemini 2.5 Flash Lite to a fine-tuned small model cut inference cost 68% while accuracy went from 81% to 93%. Linkup moved their structured-output traffic the same way after it cleared their own quality-parity bar, and cut cost 40%.
Both are the same move: one stage, one input shape, one output shape, everything around it untouched. That’s also why it’s the easiest to evaluate, because the stage boundary already exists in your code, so the eval is a comparison on one contract rather than a judgement about a whole system’s behaviour.
How to pick
| Pattern | Fits when your system | You have to build | What it buys | Where it hurts |
|---|---|---|---|---|
| Stage swap | is a fixed pipeline with a model call at a step | almost nothing, the boundary exists | the biggest cut for the smallest diff | needs a genuinely stable stage |
| Handoff | sends easy and hard traffic through one call | an orchestrator that honours the deferral | frontier quality on the tail, small-model cost on the bulk | 0.03 on the strict rubric, and deferral has to be trained |
| Skill | already has a frontier model orchestrating narrow work | a strict output contract per skill | the token-heavy majority of the work moves | the orchestrator’s tokens stay on your bill |
If a stage boundary already exists in your code, start there. It’s the smallest change, the easiest thing to evaluate, and it’s where our clearest customer results come from. Reach for the handoff when your blocker is quality on a hard tail rather than the shape of your pipeline, and for the skill pattern when a frontier orchestrator is already looping over narrow work. In all three cases the pattern follows the architecture you already have.
Start with one task
Bring us one task and the request logs you already have for it. Within days you get a trained model, a held-out evaluation against your current model, and our read on which of these three patterns fits your system. The training is free, so the output is a working model and its eval report rather than a proposal.
distil labs trains task-specific small language models that are cheaper and faster per request than general-purpose LLMs, with equal-or-better accuracy on bounded tasks. Docs · GitHub · Slack
distil labs trains task-specific small language models that are cheaper and faster per request than general-purpose LLMs, with equal-or-better accuracy on bounded tasks. Docs · GitHub · Slack