What Latency Can You Expect from an SLM?
Between tens of milliseconds and a few hundred, and which end you land on depends more on load and hardware than on the model. Our published figures span 10ms on a CPU core to a 390ms p50 on an H100 at saturation — both for fine-tuned small models.
What latency figures have actually been published?
These, each with the setup that produced it. The setup column is not decoration; without it the number means nothing.
| Setup | Measured |
|---|---|
| FunctionGemma 270M fine-tuned, single CPU core or in-browser | 10–50ms range |
| Qwen3-0.6B fine-tuned, local inference, tool-calling task | Under 50ms |
| Qwen3-0.6B fine-tuned, self-hosted local, voice brain stage (TTFT + ~100 output tokens), single stream | 40–100ms without network, ~200ms with |
| distil labs classification model, production hosting, benchmark with bursts to 10 RPS | p50 0.27s / p95 0.59s / p99 0.66s |
| Qwen3-4B fine-tuned, vLLM, single H100, BF16, at 222 RPS sustained ceiling | p50 390ms / p95 640ms / p99 870ms |
Sources in order: the FunctionGemma write-up, the dlt trace-to-model case study, the voice assistant post, the Knowunity case study and the inference-cost benchmark.
For comparison, the same posts report cloud API latencies measured alongside: GPT-oss-120B on managed Cerebras/Groq infrastructure at 400–700ms for the voice brain stage, Claude Haiku 4.5 at 700ms–1.2s, and Gemini 2.5 Flash Lite at p50 0.49s / p95 0.81s / p99 1.28s in the Knowunity benchmark. The dlt post quotes the same 400–700ms band for a cloud API call against its under-50ms local figure — that comparison is scoped to that one 0.6B tool-calling model and its local deployment, not a general claim about cloud APIs.
Why is there no per-model latency table?
Because the two rows that look most contradictory in the table above are both correct.
A fine-tuned Qwen3-0.6B answering in 40ms and a fine-tuned Qwen3-4B showing a 390ms p50 are not in tension. The first is a single request on a quiet local deployment. The second is the median at the saturation point of a GPU pushing 222 requests per second — a number chosen deliberately to characterise the throughput ceiling, not the response time a lightly loaded server would give. Divide the same GPU’s work across ten times fewer requests and the percentile moves.
So a table of “model → latency” would be a table of one team’s load assumptions dressed up as a model property. Anyone publishing per-model millisecond figures without stating hardware, batch size, prompt length, output length and offered load is telling you about their benchmark, not about the model.
What you can take from the numbers above is a band. A small fine-tuned model, served locally, on a bounded task with short outputs, answers in tens of milliseconds. The same class of model under production load through a managed endpoint lands in the low hundreds. Both are well inside a single cloud API round trip.
What drives latency?
Five factors, and only one of them is the model.
| Factor | Direction | Mechanism |
|---|---|---|
| Parameter count | More parameters, slower per token | More bytes to stream per token; memory bandwidth binds generation |
| Quantization | Fewer bits, faster | Fewer bytes per weight to move; largest effect on CPU |
| Offered load vs capacity | Near saturation, much slower | Queueing — percentiles climb steeply as utilisation approaches 100% |
| Prompt length | Longer prompt, slower first token | Prefill is compute-bound and scales with input tokens |
| Output length | Longer answer, longer total | Decode is sequential; total ≈ TTFT + tokens × per-token time |
| Network hop | Adds a fixed floor | The voice-assistant figure moves from 40ms to ~200ms on this alone |
The third row is where serving software earns its keep. vLLM exists to push the saturation point higher through continuous batching, so the same hardware holds acceptable percentiles at higher offered load; a single-stream runtime such as llama.cpp is excellent at row one and two and has no answer for row three.
The last row is the one that decides architecture. In the voice study the same model measured 40–100ms locally and about 200ms once a network call was in the path — the model got no slower, the deployment did. This is why on-device and in-VPC deployment shows up in latency conversations at all, and it is covered in on-device LLM inference in 2026.
Output length deserves a second mention because it is under your control. Fine-tuned task models emit structured, short outputs — a tool call, a label, a redacted string — and that is a large part of why they are fast. A model that has been taught to answer in ten tokens beats a model that reasons for four hundred, whatever their sizes.
How should you measure your own?
Measure at your load, on your hardware, with your prompts, and report percentiles rather than an average.
Four rules that make the number mean something:
- Separate TTFT from total. Time to first token tells you what your prompt length costs; total tells you what your output length costs. An interactive product cares about the first, a batch job about the second.
- Sweep offered load. A single-concurrency number is not a production number. Our Knowunity benchmark ramps from 1 RPS through a sustained 7 RPS and back down specifically to surface autoscaling behaviour, and treats anything over 8s as a failure.
- Use real prompt lengths. The voice-assistant economics assume roughly 800 input tokens and 100 output tokens per turn. Your payload shape will differ and it changes prefill directly.
- Report p95 and p99, not the mean. The mean hides the tail, and the tail is what users experience as “sometimes it hangs”.
What target should you aim for?
Set it from the product, not from the model, and check the whole pipeline rather than the model stage.
Voice is the clearest example because the constraint is external and well established: conversational research places the threshold for natural-feeling interaction at 500–800ms total, across speech recognition, the model, and speech synthesis. A cloud brain at 400–700ms consumes most of that budget on its own, which is what makes replacing it with a locally served SLM a product decision rather than a cost decision.
For a classification or extraction step inside a larger workflow, the useful target is usually “small enough not to be the bottleneck”, and the Knowunity figures — p50 0.27s against a cloud small model’s 0.49s at the same time as higher accuracy and lower cost — are the shape of what that looks like in production.
If latency is your binding constraint, the levers in order are: shorten the output, shrink the model, quantize, remove the network hop. What size model do you need covers the second, Q4 vs Q8 vs FP16 the third, and self-hosted vs managed inference the fourth.