How to Deploy a Fine-Tuned Small Language Model
Choose from your hardest constraint, not your preference. A managed endpoint if the model only has to be reachable, llama.cpp if it has to run on a device, vLLM if it has to serve concurrent traffic, and a plain weights download if the model can never leave your network.
Which deployment path fits your constraint?
Work down this table until a row describes something you cannot negotiate, and take that row.
| Binding constraint | Path | How you start it |
|---|---|---|
| You just want to try the model against real inputs | distil labs managed endpoint | distil model deploy remote <model-id> |
| It must run on a laptop, a handheld, or an embedded board | llama.cpp | distil model deploy local <model-id> |
| It must serve concurrent requests behind a service | vLLM on your own GPU | vllm serve model --api-key EMPTY |
| Weights must stay inside your network | Download and self-host | distil model download <model-id> |
| You want it in a local development loop with model management | Ollama | ollama create from the GGUF |
The commands come from the inference and local deployment docs. Each path has its own recipe: llama.cpp, vLLM, Ollama, and CPU-only.
What does the managed endpoint actually give you?
A hosted, OpenAI-compatible URL and an API key, provisioned by one command, with no infrastructure on your side.
distil model deploy remote <model-id> returns the endpoint URL, an API key, and a client script. distil model invoke <model-id> prints a ready-to-run command that uses that script. When you are done, distil model deploy remote --deactivate <model-id> releases it, because playground deployments consume credits while active.
The important caveat is in the docs and it is not a footnote: these deployments are for testing, not production. Production hosting is a separate conversation with distil labs. Treat the playground as the fastest way to find out whether the model is good, not as the thing you point traffic at.
What does self-hosting require?
A GPU or a CPU with enough memory, a runtime, and someone who owns the process when it falls over.
distil model download <model-id> gives you a tarball containing model/, model-adapters/, model_client.py, and a README. The model/ directory is what you hand to vLLM. Models can also be pushed to a private Hugging Face repository through the API, which produces two repos — one in GGUF and one in safetensors — so you can pull whichever format your runtime wants.
The choice of runtime is the choice of workload shape. vLLM exists for throughput and concurrency; llama.cpp exists to run quantized models with minimal setup on whatever hardware is in front of you. Picking between them is covered in self-hosted vs managed inference and, at the runtime level, in Q4 vs Q8 vs FP16.
What goes wrong most often?
Changing the prompt format. This is the one that costs teams the most time. Both deployment docs warn about it explicitly: a fine-tuned SLM expects exactly the message formatting and system prompt it saw during training. Swap in your own system prompt and accuracy falls off a cliff, with no error to tell you why. Use the client script the platform hands you, and if you must reimplement it, reimplement it byte for byte. Open-book question-answering models are the sharpest case — the context has to be wrapped in a <context> tag followed by a newline inside the first user message.
Sizing for weights and forgetting the KV cache. A model that loads happily at a 2K context will not necessarily survive at 32K. The working is in how much VRAM does a 1B, 3B or 8B model need.
Benchmarking latency in the wrong regime. A single-stream local number and a p50 under saturation are different measurements that people compare as if they were the same. What latency can you expect from an SLM separates them.
Quantizing before you have a baseline. Quantize after you know the FP16 score on your own test set, never before, or you will not know which of the two changes moved the number.
Deploying a size you never validated. Deployment does not fix a model that was too small for the task. What size model do you need is the earlier decision.
How do you check the deployment is correct?
Send the same test set through the endpoint that you evaluated on the platform, and compare the score.
This sounds obvious and is skipped constantly. The failure modes above — wrong chat template, wrong system prompt, wrong quantization — all produce a server that responds fine and answers worse. A response with a 200 status is not evidence of a working deployment; a matching accuracy number is.
Three checks, in order. Confirm the server is up: llama.cpp exposes GET /health, and both llama.cpp and vLLM expose GET /v1/models. Confirm the format round-trips by running one known example through the provided client. Then run the full test set and compare against the evaluation metrics the platform reported. If the numbers diverge, the deployment is wrong, not the model.
For environments with no network at all, the constraints change again — see air-gapped and on-premise LLM deployment.