Local deployment

Your trained model is yours to run. Download it and serve it on your own GPU with vLLM, with no dependency on distil labs at inference time.

To try the model without serving it yourself, hosted inference is fewer steps.

Downloading your model

Download the model into ./model:

distil slm download --destination ./model <slm-id>

This writes model.tar and config.yaml. The command answers only once training has reached JOB_SUCCESS, and checks free disk space before it starts writing. Extract the tarball:

cd ./model && tar -xf model.tar

The tarball holds:

Path	What it is
model-adapter/	The LoRA adapter (adapter_config.json and the adapter weights). This is the trained model. Also holds LICENSE, TEACHER_LICENSE and STUDENT_LICENSE
model_client.py	Generated inference client, the way to query the model
README.md	Serving instructions

The adapter is served on the student’s base model, which vLLM downloads from HuggingFace, so the tarball holds no base weights. Keep config.yaml next to it: it isn’t inside the tarball, and it names the base model and the LoRA rank.

To fetch only the client, a few kilobytes, use distil slm download-metadata --destination ./model <slm-id>.

Serving with vLLM

vLLM is the supported server (Ollama is not supported). It loads the base model and applies the adapter on top of it, the same way the hosted deployment serves it. Start it from ./model, where the tarball was extracted:

vllm serve <huggingface-base-model> \
  --served-model-name base --enable-lora --lora-modules model=./model-adapter --max-lora-rank <tuning.lora_r> --api-key EMPTY <family-flags>

The server listens on port 8000. <huggingface-base-model> and <family-flags> come from the tables below.

• --lora-modules model=./model-adapter serves the adapter under the name model, which is the name model_client.py sends. The base model answers under base, so a request for model reaches your trained model and a request for base the untrained one.
• --max-lora-rank must be at least tuning.lora_r from config.yaml (default 64).
• The server runs in the foreground, so start it in its own terminal.

The base model

<huggingface-base-model> follows from base.student_model_name in config.yaml:

student_model_name	HuggingFace base model
Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct	meta-llama/<name>
Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B-Instruct-2507, Qwen3-8B	Qwen/<name>
Qwen3.5-0.8B, Qwen3.5-2B, Qwen3.5-4B, Qwen3.5-9B, Qwen3.6-35B-A3B, Qwen3.8-27B	Qwen/<name>
LFM2-350M, LFM2-1.2B, LFM2-2.6B, LFM2.5-350M, LFM2.5-1.2B-Instruct	LiquidAI/<name>
gemma-3-270m-it, gemma-3-1b-it, gemma-3-4b-it, gemma-4-E2B-it, gemma-4-E4B-it, functiongemma-270m-it	google/<name>
SmolLM2-135M-Instruct, SmolLM2-1.7B-Instruct	HuggingFaceTB/<name>

The Llama and Gemma base models are gated on HuggingFace: accept the license on the model page and export HF_TOKEN before vllm serve.

Family flags

<family-flags> are the tool-call and reasoning parser flags of the student’s family. The hosted deployment passes the same flags. Every family with a tool-call parser also takes --enable-auto-tool-choice:

Student family	--tool-call-parser	--reasoning-parser
Qwen3	hermes	qwen3
Qwen3.5, Qwen3.6, Qwen3.8	qwen3_xml	qwen3
Llama 3	llama3_json	none
LFM2, LFM2.5	lfm2	none
Gemma 4	gemma4	gemma4
FunctionGemma	functiongemma	none
Gemma 3, SmolLM2	none	none

For a Qwen3.5 student, <family-flags> is --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3. Gemma 3 and SmolLM2 take no family flags.

Querying your model

Use the client that came with the model, from ./model:

python model_client.py --conversation '[{"role": "system", "content": "<system prompt>"}, {"role": "user", "content": "Your question here"}]'

It defaults to http://127.0.0.1:8000/v1 and the model name model. Without --conversation it sends an example from your test data, a quick way to confirm serving works. From Python, in the same directory:

from model_client import DistilLabsLLM

client = DistilLabsLLM(model_name="model", base_url="http://127.0.0.1:8000/v1")
print(client.invoke([{"role": "system", "content": "<system prompt>"}, {"role": "user", "content": "..."}]))

The conversation starts with the system prompt the model was trained with; DEFAULT_CONVERSATION in the client shows it. Query through the client rather than a hand-built request: it sends the sampling settings the model was evaluated with and the thinking setting of training, and a request without them answers 200 with a much worse result. distil labs inference explains what the client does.

When quality is worse than expected, check the prompt format first, then that the request names the adapter (model) and not the base model (base).

Next

If the model came in short of the bar, Model iterations covers the next iteration. To put the model in front of the traffic it was trained on, see Serving behind the endpoint.