Local deployment

Your trained model is yours to run. Download the weights and serve them on your own infrastructure - no dependency on distil labs at inference time.

If you just want to try the model quickly, hosted inference is fewer steps.

Downloading your model

distil slm download --destination ./model <slm-id>

This writes model.tar and config.yaml. You need both to serve the model, since the config is not inside the tarball.

The tarball runs to several gigabytes, about 1.2 GB for a Qwen3-0.6B run, and the command checks free disk space before it starts writing. It also answers only once training has reached JOB_SUCCESS. Before then it names the job’s state and exits 1 rather than writing an empty file.

Extract it, and you get:

Path	What it is
model/	Merged fine-tuned weights (safetensors)
model-adapter/	LoRA adapter, when use_lora: true
model_client.py	Generated inference client - the canonical way to query the model
README.md	Serving instructions
LICENSE, TEACHER_LICENSE, STUDENT_LICENSE	Licences

Just the client

If you only need the inference client - to point at a hosted deployment, say - fetch it without the weights. A few kilobytes instead of gigabytes:

distil slm download-metadata --destination ./model <slm-id>
# ./model/config.yaml, ./model/job_description.json, ./model/model_client.py

Serving with vLLM

vLLM is the documented serving backend, and it’s OpenAI-compatible.

python -m venv serve
source serve/bin/activate
pip install vllm openai

Start the server. model here is the directory holding your weights:

vllm serve model --api-key EMPTY          # serves on port 8000

For tool-calling models:

vllm serve model --enable-auto-tool-choice --tool-call-parser hermes --api-key EMPTY

The server runs in the foreground, so start it in its own window or as a background process.

Querying your model

Use the client that shipped with the model:

python model_client.py --conversation '[{"role": "user", "content": "Your question here"}]'

It defaults to http://127.0.0.1:8000/v1. From Python:

from model_client import DistilLabsLLM

client = DistilLabsLLM(model_name="model", base_url="http://127.0.0.1:8000/v1")
print(client.invoke([{"role": "user", "content": "..."}]))

For question-answering models that need context, wrap it in a <context> tag followed by a newline, inside the first user message:

python model_client.py \
  --conversation '[{"role": "user", "content": "<context>Your context here</context>\nYour question here"}]'

Run it without --conversation and it replays a baked-in example from your test data - a quick way to confirm serving works.

Don’t hand-build the chat request

A small model degrades sharply outside its training-time setup, and the client reproduces that setup exactly: the trained system prompt baked in as SYSTEM_PROMPT, temperature=0, thinking disabled, tool-calling models called with tool_choice="required", and QA context inlined as <context>…</context>.

A raw chat-completions request matches none of that, and it fails quietly: the endpoint answers 200, reasoning text leaks into the answer, and the score falls well below what your evaluation metrics promised.

Each model carries its own client, generated for it at training time. Use the one that came with the model you’re querying.

Production considerations

• Hardware. Even small models benefit from GPU acceleration under throughput.
• Security. Apply your own access controls, especially if the model handles sensitive information. Nothing is enforced for you here.
• Packaging. Consider a container image so the weights, the config and the client version travel together.
• Ollama isn’t supported. vLLM is the documented backend.

Next

If the model came in short of the bar, Improving your model covers the second iteration. To try a model without serving it yourself, use hosted inference instead.