Cost Per Million Tokens: Local vs API
A self-hosted 4B student runs at roughly $0.04 per million tokens on distil labs hosted inference, against $0.18 to $2.00 for the small cloud models listed beside it. The gap is real, but the two numbers are built differently: one is a rented GPU divided by throughput, the other is a metered price per token.
What does each option cost per million tokens?
Here are the rates the pricing page publishes, blended 3:1 (three input tokens for every output token) at sustained load on one H100.
| Option | Blended $/1M tokens | Billing model |
|---|---|---|
| Distilled Qwen3 4B, self-hosted | $0.04 | GPU hour |
| Distilled Qwen3.5 9B, self-hosted | $0.08 | GPU hour |
| Gemini 2.5 Flash-Lite | $0.18 | Per token |
| GPT-5.4 Nano | $0.46 | Per token |
| Claude Haiku 4.5 | $2.00 | Per token |
That puts the 4B student roughly 5x under Flash-Lite and 12x under GPT-5.4 Nano. Vendor list prices move often, so treat the right-hand rows as a snapshot and confirm against OpenAI, Anthropic and Google before you build a forecast on them.
Why aren’t these the same kind of number?
Because only one of them is a price. The API figure is what you’re charged per token, full stop. The local figure is a rented GPU divided by how many tokens you actually pushed through it.
The denominator is what varies. Rent a card and use half of it, and your effective rate doubles: same hardware bill, half the tokens. The API rate is flat no matter how little you use it.
So the comparison above is only fair at sustained load. At 10% utilisation the local column moves against you by roughly an order of magnitude while the API column stays put, which is why the pricing page puts the break-even for a dedicated GPU at about 30% utilisation.
What is an API better at?
Bursty, low-volume, and open-ended work. Three cases where per-token billing wins outright:
- Spiky traffic. You pay for the spike and nothing between spikes. A dedicated GPU bills through the quiet hours.
- Unproven tasks. Before you know a task is stable, paying per call to find out is cheaper than provisioning for it.
- Broad reasoning. The inference tax benchmark found fine-tuned students ranked 5th of 11 on docstring generation, where part of the output is free-form prose. Structure is what small models buy you; breadth isn’t.
What is self-hosting better at?
Steady high volume on a narrow task, plus anything with a data-residency requirement. In that benchmark, fine-tuned students placed first on 4 of 8 datasets while costing $3 per million requests against $6,241 for the top-ranked frontier model.
Two advantages don’t show up in the token rate at all. Latency: a locally served student answers without a network hop, which the voice assistant write-up measured as a brain stage of roughly 40ms against a ~700ms cloud median. And control: weights you host are weights no third party sees.
Which should you pick?
Pick by utilisation first, task shape second.
| Your situation | Pick |
|---|---|
| Under ~30% GPU utilisation | API |
| Steady high volume, narrow task | Self-host |
| Open-ended generation | API |
| Data can’t leave your infrastructure | Self-host |
| Still defining the task | API, then revisit |
Most production systems end up running both: specialists on the structured high-volume paths, an API behind everything else. If you land on self-hosting, vLLM is the throughput-oriented server and llama.cpp the lightweight local one, both covered in the local deployment guide.
For the decision in more depth see when does self-hosting beat an API, and for the costs that don’t appear in either column of the table above, hidden costs of running your own models.