Calculating the ROI of a Task-Specific Model

ROI here is one number: the request volume at which a fine-tuned model’s one-off build cost is repaid by its lower per-request rate. Everything below produces that number. Steps 1 to 3 cost nothing, because the free tier covers two full training runs.

What you need before you start

Four inputs, and you probably already have three of them.

Input	Where it comes from
Current monthly request volume for one task	Your API provider’s usage dashboard
Current monthly spend on that task	Same, filtered to the task
~50 example inputs and correct outputs	Production logs or traces
A held-out test set	Split from the same data

Isolate a single task first. Blended spend across everything you send to an API produces a meaningless average, because the tasks that distil well and the tasks that don’t are mixed together. The inference tax benchmark found students ranking 1st on structured tasks and 5th on free-form ones in the same study.

Step 1: Measure what the task costs today

Pull the real figure rather than a modelled one. Token counts per request multiplied by list price is almost always wrong, because retries, system prompts and few-shot examples inflate it.

Record three things: requests per month, spend per month, and observed p50/p95 latency. Divide spend by requests to get your current cost per million requests, which is the number every later step compares against. For reference points published at that unit, how much can you save replacing an LLM API collects them.

Step 2: Prove a small model can do the job

Run teacher evaluation before spending anything on training. This is the gate, and skipping it is the most expensive mistake in the whole procedure.

distil seed-dataset create --data ./processed
# Output: Upload successful. Seed dataset ID: <seed-dataset-id>

distil teacher-evaluation create-from-seed-dataset <seed-dataset-id>
# Output: Teacher evaluation started. Teacher Evaluation ID: <teacher-evaluation-id>

distil teacher-evaluation status <teacher-evaluation-id>

If the teacher can’t solve your task on your test set, the student won’t either. Teacher evaluation exists precisely so you find that out in minutes rather than after a full run. A failure here is a genuine result: it usually means the task is too broad and needs splitting into narrower sub-tasks.

Step 3: Train and measure the replacement

Train on the free tier and measure the student rather than assuming it. Training is two jobs: the teacher writes the training set, then the student learns from it.

distil training-dataset create-from-seed-dataset <seed-dataset-id>
# Output: Synthetic data generation started. Training Dataset ID: <training-dataset-id>

distil slm create-from-training-dataset <training-dataset-id>
# Output: Training started. SLM ID: <slm-id>

distil slm download --destination ./model <slm-id>

slm download writes the weights, the config and a generated model_client.py. Serve them and benchmark on your own traffic shape. You need two outputs: accuracy against your held-out test set, and sustained throughput in requests per second. The second one is what converts a GPU-hour bill into a per-request cost: divide the hourly rate by requests served per hour. Serve with vLLM if you’re measuring throughput seriously. A few calls through the generated client check correctness, not capacity.

Don’t skip the accuracy column. A cheaper model that’s worse is a cost increase disguised as a saving, and the point of distillation is that it doesn’t have to be.

Step 4: Find the break-even volume

Now assemble the three numbers into a threshold.

break-even requests = build cost ÷ (current cost per request − new cost per request)

Build cost is training plus integration effort plus any annual retraining you expect. The distil labs pricing page currently lists credit packs at $1,000 for 10 runs, and the free tier covers the first two; what does it cost to fine-tune a small language model breaks down what a run consumes.

Compare the result to your monthly volume from step 1. If break-even lands inside a few months, the case is straightforward. If it lands past a year, the honest answer is usually no. The utilisation floor matters more than the arithmetic anyway: below roughly 30% GPU utilisation a dedicated endpoint costs more than the API regardless of what this formula says.

Step 5: Stress-test the assumptions

Re-run the calculation with each assumption moved against you. Four are worth checking:

• Utilisation at 10% instead of full. Your per-request cost moves by roughly an order of magnitude; the API’s doesn’t move at all.
• Retraining twice a year instead of once. Build cost isn’t a one-off for a model in production.
• API prices falling. They have been. Your baseline is a moving target, so check OpenAI and Anthropic rather than a figure from last quarter.
• Ops time. Someone owns serving, monitoring and upgrades. Hidden costs of running your own models lists what usually gets left out.

Two benefits deliberately sit outside this model because they’re hard to price and easy to undervalue: latency, which is itself a cost, and data residency, which for regulated workloads isn’t a trade-off at all. If either one is binding, the break-even volume isn’t the deciding number.