distil labs inference

distil labs inference serves a trained model for you and puts an OpenAI-compatible inference endpoint in front of it. It’s the quickest way to try a model on real requests without setting up serving yourself.

Creating a deployment

Deploy the trained model. The command prints the new deployment’s id:

distil deployment create-from-slm --output json <slm-id> | jq -r .id

Check its status with that id until it reads JOB_SUCCESS:

distil deployment status --output json <deployment-id> | jq -r .deployment_status

The deployment serves the model’s LoRA adapter with vLLM on the student’s HuggingFace base model. The job doesn’t return until vLLM answers, so JOB_SUCCESS means serving rather than merely scheduled. It takes about 40 minutes. Deployments report deployment_status and endpoint_status instead of status.

A model can be deployed when it was trained with a tuning.lora_r of 8, 16, 32, 64, 128, 256, 320 or 512. Any other SLM is refused.

Putting an endpoint in front of it

A deployment isn’t called directly. You reach it through an inference endpoint that calls the deployment first and falls back to a model you name when the deployment can’t answer:

distil inference-endpoint create-from-deployment --name my-slm \
  --fallback-model "openai/gpt-4.1-mini" <deployment-id>

Create it once the deployment reports JOB_SUCCESS. Before that the command refuses with Deployment is not ready. --name is a prefix: the platform appends a suffix, and the command prints the unique endpoint name on its Endpoint Name line, here my-slm-Qk3bZ1. Every request names it as its model. --fallback-model is an OpenRouter slug in owner/model form.

An endpoint can’t be edited or deleted. A different fallback or a new deployment means a new endpoint with a new unique name.

Callers authenticate with an inference API key, linked to the endpoint. Create a key named my-slm-test and link it:

distil api-keys create my-slm-test
distil inference-endpoint link-api-key my-slm-Qk3bZ1 my-slm-test

distil api-keys create prints the secret once and writes it to my-slm-test.json; the examples below call it <api-key>. A key is a separate credential from your CLI session. One key can be linked to several endpoints, and an account holds at most 5. A newly linked key takes up to a minute to start working.

Querying your model

Every trained model has its own inference client, generated at training time for the prompt shape the model was trained with. Fetch it without downloading the model:

distil slm download-metadata --destination ./model <slm-id>

This writes config.yaml, job_description.json and model_client.py into ./model. Point it at the endpoint, with the unique endpoint name as the model:

uv run ./model/model_client.py --base-url https://inference.distillabs.ai/v1 \
  --api-key <api-key> --model my-slm-Qk3bZ1 \
  --conversation '[{"role": "user", "content": "Your question here"}]'

From Python, run from ./model so that model_client imports:

from model_client import DistilLabsLLM

client = DistilLabsLLM(
    model_name="my-slm-Qk3bZ1",
    base_url="https://inference.distillabs.ai/v1",
    api_key="<api-key>",
)
print(client.invoke([{"role": "user", "content": "..."}]))

A request the deployment can’t serve is answered by the fallback, so an answer alone doesn’t tell you which model gave it. The endpoint’s records do: metadata.source is primary for your model and fallback for the other one. Download the records and count them by source:

distil inference-endpoint download-traces my-slm-Qk3bZ1
jq -r .metadata.source my-slm-Qk3bZ1-traces.jsonl | sort | uniq -c

The output is one count per source, such as primary and fallback.

Why the client, and not a raw request

A small model degrades sharply outside its training-time setup, and the client reproduces that setup:

• The trained system prompt, which the client contains as SYSTEM_PROMPT.
• temperature=0, and the thinking setting of training sent as chat_template_kwargs (see Reasoning models).
• Chat completion models with tools call with tools=TOOLS, tool_choice="auto", so the model decides whether to call. invoke returns the whole assistant message, content and tool calls included.

A hand-built chat completions request matches none of that, and it doesn’t fail: the endpoint answers 200, reasoning text appears in the answer of a model trained without thinking, and the score falls well below the evaluation metrics. So every new caller and integration queries through the client, and every deployment is tested with it on a few test-set rows.

The model must run with the system prompt it was trained with: the task_description of its job description. The client sends it. If your application calls the endpoint with its own requests, it must send that same system prompt, with temperature=0 and thinking as in training. When your production prompt differs from task_description, change your application’s system prompt to it, or call through the client.

The client sends one request per call and doesn’t run the agentic loop. For a chat-completion-agentic model, your code executes the calls, appends the assistant message and one {"role": "tool", "tool_call_id": ..., "content": ...} per call to the history, and calls invoke again.

Shutting it down

Delete the deployment when you are done with it:

distil deployment delete <deployment-id>

After the delete, deployment_status stays JOB_SUCCESS; endpoint_status going to stopped confirms it’s down.

A deployment is a session

A hosted deployment is for trying the model. It stops on its own after six hours, or after one hour with no traffic, and it bills until it stops. It can’t be restarted. From then on the endpoint in front of it answers every call from the fallback, and serving the model again needs a new deployment and a new endpoint. To serve a model permanently, email contact@distillabs.ai.

Doing this over the REST API

See Using the REST API.

Credits

A deployment spends one deployments_from_slms_post credit, and an endpoint one inference_endpoints_post credit. A new account starts with 2 and 10. Replacing a stopped deployment spends another deployment credit. Check your balances with:

distil credits-balance

Next

• Run the model on your own GPU instead: Local deployment.
• Put the model in front of your production traffic: Serving behind the endpoint.