When Does Self-Hosting Beat an API?

Above roughly 30% GPU utilisation on a narrow, well-defined task. That’s the threshold the distil labs pricing page gives, and it’s the first thing to check. Below it, a dedicated GPU is idle silicon you’re paying for and an API is cheaper regardless of anything else.

Which criteria actually decide it?

Five, in this order. If the first one fails, the rest don’t matter.

Criterion	Self-host when	Stay on an API when
GPU utilisation	Above ~30% sustained	Bursty or low volume
Task shape	Structured: classification, tool calling, extraction, SQL	Open-ended generation, broad knowledge
Latency budget	Sub-100ms brain stage matters	Hundreds of ms is fine
Data residency	Data can’t leave your infrastructure	No constraint
Team capacity	Someone owns serving and monitoring	Nobody does

The rows aren’t weighted equally. Utilisation is a hard gate. Data residency is a hard gate in the other direction: if the data legally can’t leave, cost stops being the deciding variable.

Does your volume justify a dedicated GPU?

Check utilisation, not request count. A model that clears 222 requests per second at its ceiling handles over 19 million requests a day on one card, so “high volume” is a higher bar than most teams assume.

The arithmetic is unforgiving in one direction only. A rented GPU bills the same whether you send it one request an hour or saturate it, so your effective per-request cost is the hardware bill divided by what you actually pushed through. The inference tax benchmark reports its $3-per-million-requests figure at full utilisation and notes the conclusions still hold at a pessimistic 10%. They do, but the effective rate moves by 10x between those two assumptions.

If your traffic is genuinely spiky, serverless GPU platforms that scale to zero sit between the two options. We run our own hosted inference this way, in partnership with Cerebrium.

Is your task narrow enough to fine-tune?

Self-hosting only pays if a small model can do the job, and that depends on structure rather than difficulty. Function calling, classification, entity extraction and SQL generation are the categories where fine-tuned students beat every frontier model tested.

On docstring generation, where the output includes free-form prose, the student placed 5th of 11. If your task needs broad world knowledge, self-hosting a small model isn’t a cost decision, it’s a quality regression. What size model do you need and when does distillation fail are the two pages to read before committing.

Does latency or data residency force your hand?

Either can override the cost calculation entirely.

On latency, a self-hosted Qwen3-0.6B student answered in roughly 40ms with no network hop, against a 400ms–1.2s range for cloud models in the same voice assistant benchmark. If you’re building something conversational, no amount of API discounting closes that gap.

On residency, self-hosting means no patient records, financial data or PII reaches a third party. The benchmark’s PII redaction model scored 94.0% running entirely on-premise. When compliance is the driver, the question isn’t whether self-hosting is cheaper but whether the model is good enough. That’s what teacher evaluation exists to tell you before you spend anything.

What mistakes do teams make here?

Four, repeatedly:

• Comparing list prices instead of effective rates. A per-token API price and a GPU-hour bill divided by throughput aren’t the same kind of number. Cost per million tokens, local vs API works through why.
• Forgetting the ops line. Serving with vLLM isn’t free labour. See hidden costs of running your own models.
• Treating it as all-or-nothing. The best setups run both: specialists on structured high-volume paths, an API behind the rest.
• Provisioning before proving. Train on the free tier, evaluate, then decide. Two full training runs cost nothing, and the weights are downloadable.

If you want to put numbers on your own case rather than borrow these, calculating the ROI of a task-specific model walks through it step by step.