Latency as a Cost
Treating latency as a cost means putting response time on the same ledger as spend, because in interactive systems it’s the constraint that decides the architecture. In a distil labs voice assistant, replacing a cloud LLM with a fine-tuned 0.6B student cut the reasoning stage from roughly 700ms to about 40ms, and that mattered more than the money.
What does it mean to treat latency as a cost?
It means latency has a budget you can overspend, not a score you try to improve. Once you frame it that way, three things follow.
First, it’s spent in stages, and stages add up. Second, it’s bounded by a threshold rather than optimised without limit: past the threshold the product feels broken, under it further gains stop mattering. Third, it trades against other costs, so it belongs in the same conversation as GPU hours and token rates.
The voice assistant write-up makes the budget concrete: conversational UX research places the threshold for natural-feeling interaction at 500–800ms total, and the reasoning stage alone consumed over 70% of processing time when a cloud LLM was doing it.
How does latency turn into money?
Through three channels, only one of which is on an invoice.
| Channel | Mechanism |
|---|---|
| Abandonment | Users leave slow interactions; the request was paid for and produced nothing |
| Throughput ceiling | Slower per-request service means more concurrent capacity for the same load |
| Design workarounds | Streaming, caching, and optimistic UI are engineering hours spent hiding delay |
The second is the one that shows up directly as infrastructure spend. If each request occupies a worker for twice as long, you need roughly twice the concurrency for the same traffic. Latency has quietly become a capacity bill.
The first is larger and harder to attribute. In voice specifically, accuracy compounds across turns and so does patience: the source write-up notes handle times and completion rates move with response time, which is a revenue effect rather than a cost line.
What does the measured difference look like?
Large enough that it isn’t an optimisation. These are brain-stage latencies from the same benchmark, covering time-to-first-token plus roughly 100 output tokens.
| Model | Hosting | Brain latency |
|---|---|---|
| Claude Haiku 4.5 | Cloud API | 700ms–1.2s |
| GPT-4o-mini | Cloud API | 500–900ms |
| Gemini 2.5 Flash Lite | Cloud API | 500–800ms |
| GPT-oss-120B | Managed | 400–700ms |
| Qwen3-0.6B fine-tuned | Self-hosted | 40–100ms (no network) |
Two things drive the gap, and only one is the model. A Qwen3-0.6B student generates far fewer tokens per second of wall-clock than a large model needs, and it answers without a network round trip. With the network included, the same student sits around 200ms. The full pipeline in that system runs ASR at ~200ms, the student at ~40ms, and TTS at ~75ms, and the source code is public.
Latency improvements aren’t confined to voice. Knowunity’s production classification model moved p50 from 0.49s to 0.27s and p99 from 1.28s to 0.66s while also costing less. The p99 improvement is the one users actually notice.
When does latency dominate the decision?
When the interaction is synchronous and a human is waiting. Voice, autocomplete, in-editor assistance, live routing, and anything inside a request path that has its own timeout.
It stops mattering when work is batched or asynchronous. A nightly classification job over a million documents cares about throughput and price, not about p99. There, an API’s latency profile is irrelevant.
Be careful about the reverse error too: shaving 40ms off a stage that already sits well inside budget buys nothing. The voice case is compelling because the cloud brain pushed the total past the threshold, not because 700ms is intrinsically bad.
What related terms should you know?
- Time to first token (TTFT). The part of latency a user perceives as responsiveness, distinct from total generation time.
- The 10x inference tax. The spend-side counterpart, usually paid for the same reason: a model larger than the task requires.
- Hidden costs of running your own models. What self-hosting adds back once you’ve taken the latency win.
- What size model do you need. The lever, since latency falls with parameter count on the same hardware.
For the throughput side of the same trade-off, GPU requirements and cost for SLM inference has measured RPS and percentile figures on a single card.