Self-Hosted vs Managed Inference
Managed inference wins on time-to-first-request and on not owning an on-call rotation. Self-hosting wins when data cannot leave your network, when latency cannot include a network hop, or when volume is high enough that a GPU you rent by the hour beats tokens you buy by the million.
How do the two options compare?
Across the dimensions that actually decide it:
| Dimension | Managed endpoint | Self-hosted |
|---|---|---|
| Time to first request | Minutes — one command | Hours to days, plus provisioning |
| Who owns uptime | The provider | You |
| Data residency | Leaves your network | Stays inside it |
| Network hop | Always | Optional |
| Cost model | Per request or per token | Per GPU-hour, amortised over utilisation |
| Cost at high volume | Scales linearly with requests | Falls per request as utilisation rises |
| Cost at low volume | Cheap | Poor — you pay for idle |
| Control over runtime | None | Total: precision, batching, context, version pinning |
| Works air-gapped | No | Yes |
Note that “managed” covers two different things on distil labs. distil model deploy remote <model-id> provisions a playground endpoint for testing, and the inference docs are explicit that these are not intended for production use. Production managed hosting is a separate arrangement. Self-hosting starts from distil model download <model-id> and the runtime of your choice — see local deployment.
What is managed inference good for?
Everything up to the point where one of the self-hosting constraints binds.
The strongest case is the early one. You have just trained a model and you do not yet know whether it is good. Standing up a GPU to answer that is out of proportion to the question, and one command gets you a URL, an API key and a client script:
distil model deploy remote <model-id>
distil model invoke <model-id>Deactivating when you are done matters, because playground deployments consume credits while active:
distil model deploy remote --deactivate <model-id>The second strong case is production traffic that is genuinely variable. Knowunity’s models serve up to 4 million requests per day with traffic ranging to 150 requests per second, and their write-up is explicit that rapid autoscaling to meet changing demand is the crucial property. Building that yourself is a real engineering project, and on a managed platform it is someone else’s.
The third is organisational: managed inference means the model swap is an endpoint change. Knowunity moved by updating the API endpoint in their existing workflows.
What is self-hosting good for?
Four situations, and in each of them managed inference is not a worse option but an unavailable one.
Data cannot leave your perimeter. This is the most common and the least negotiable. Our PII redaction model scored 94.0% with everything running on-premise, and that arrangement exists precisely because patient records cannot hit a third-party API. The extreme version is covered in air-gapped and on-premise LLM deployment.
The network hop is the latency budget. In our voice-assistant work the same fine-tuned Qwen3-0.6B measured 40–100ms locally and about 200ms with network latency in the path. When the whole conversational budget is 500–800ms, that difference is architectural. See what latency can you expect from an SLM.
The model runs on a device. No managed option exists for a model embedded in a controller or shipped inside an app. That is llama.cpp territory — see the recipe and CPU deployment.
You need control of the runtime. Precision, batching policy, context length, tool-call parser, version pinning — vLLM exposes all of it and a managed endpoint exposes none of it. The vLLM recipe covers the flags.
What does the cost comparison actually look like?
Self-hosted cost per request falls as utilisation rises, which is the whole difference between the two models.
| Workload | Self-hosted or dedicated SLM | Cloud API comparator |
|---|---|---|
| Mixed benchmark, 8 datasets, fine-tuned 0.6B–8B models | $3 per 1M requests | $45 (GPT-5 nano) to $6,241 (Claude Opus 4.6) |
| Banking voice assistant, ~800 in / ~100 out tokens per turn | ~$5 per 1M turns | $120 (Gemini 2.5 Flash Lite) to $1,300 (Claude Haiku 4.5) |
| Knowunity classification, production traffic over several days | $35 per 1M requests | $109 (Gemini 2.5 Flash Lite) |
Read the assumptions before you use these. The $3 figure comes from sustained vLLM throughput on a single H100 node at roughly $2.40/hr, computed at full utilisation — the benchmark states this openly and notes the conclusions still hold at a pessimistic 10% utilisation. The Knowunity $35 is from real production usage rather than a synthetic benchmark, which is why it is an order of magnitude above the idealised figure and closer to what you should plan against.
The pattern is consistent: a dedicated small model is cheaper per request than a cloud API at volume, and the gap widens with utilisation. Below some volume it inverts, because an idle GPU still bills.
Which should you pick?
Start managed, move to self-hosted when a constraint forces it, and let volume decide the rest.
The sequence that wastes the least time: deploy to a managed endpoint to find out whether the model is good enough; if it is, check whether data residency, device deployment or the latency budget rules managed hosting out; if none of them does, compute your cost both ways at your actual request volume and utilisation.
Most teams end up with both. High-volume structured tasks run on a dedicated small model, and open-ended work routes to a frontier API — a split argued in is fine-tuning worth it and reflected in the decision table in how to deploy a fine-tuned small language model.