Hidden Costs of Running Your Own Models
Hidden costs are the parts of self-hosting that don’t appear in a per-token comparison: idle GPU time, the engineer who owns the serving stack, monitoring, retraining as data drifts, and the security controls a managed API was quietly providing. They rarely reverse the decision, but they routinely halve the saving.
What are the hidden costs of self-hosting?
Five categories, and only the first is measurable in advance.
| Cost | Why it’s invisible in a per-token comparison |
|---|---|
| Idle GPU time | You’re billed for uptime, not for requests served |
| Serving expertise | Someone has to run and tune the inference server |
| Monitoring and on-call | An API’s uptime was somebody else’s problem |
| Retraining | A model that fit last quarter’s data drifts |
| Security and access control | Moves from the provider’s responsibility to yours |
The production considerations section of the deployment docs names three of these directly: resource requirements, access controls for sensitive data, and container packaging for consistency across environments. None of them shows up as a number, and all of them are work.
Where do they actually come from?
From the difference between renting capacity and renting outcomes. An API price bundles hardware, operations, availability and security into one metered rate. Self-hosting unbundles it, and you re-buy each piece separately.
Idle time is the biggest and the most predictable. Because a GPU bills by uptime, your effective per-request cost is the hardware bill divided by requests actually served. A card at 10% utilisation costs roughly 10x per request what the same card costs saturated. This is why the pricing page puts the threshold for a dedicated GPU at roughly 30% utilisation rather than at a request count.
The operational costs come from a subtler shift. Serving with vLLM or llama.cpp is well-documented and not especially hard, but “not hard” isn’t “free”. It’s an ongoing claim on someone’s attention, including during incidents.
Retraining belongs in the build cost too. A task-specific model is fitted to a distribution, and distributions move. The right unit for a build cost is runs per year, not runs.
When do they outweigh the savings?
At low volume, on unstable tasks, and in teams with nobody to own the stack.
Low volume is arithmetic: below the utilisation floor, idle time alone erases the gap. Instability is subtler: if your task definition changes every few weeks, you’re paying the retraining cost repeatedly and never amortising it.
If no engineer owns inference, these costs don’t disappear. They go unpaid until something breaks. Managed inference exists for exactly this case: we run our hosted endpoints on autoscaling serverless GPU infrastructure that scales to zero, which absorbs the idle-time and on-call problems while keeping the model yours.
What don’t they outweigh?
Compliance requirements, and gaps of two orders of magnitude.
When data can’t legally leave your infrastructure, hidden costs are the price of operating rather than a factor in a comparison. And when the raw gap is large enough (the inference tax benchmark measured $3 per million requests against $6,241 for the top-ranked frontier model), a doubling of the true cost of self-hosting doesn’t change the conclusion. It’s the 68% savings cases, like Knowunity’s, where the hidden line items decide whether the project was worth doing.
What related ideas should you know?
- Utilisation. The denominator behind idle-time cost, and the single number that most changes a self-hosting decision.
- The 10x inference tax. The cost on the other side of the ledger, which these hidden costs are netted against.
- Latency as a cost. A benefit of self-hosting that a spend comparison omits, in the same way these costs are omitted.
- What is catastrophic forgetting. One reason retraining isn’t a free repeat of the original run.
To put your own numbers against these, calculating the ROI of a task-specific model has the break-even formula, and when does self-hosting beat an API has the criteria table.