distil labs inference

distil labs inference gives you a hosted, OpenAI-compatible endpoint for a trained model. It’s the quickest way to try a model without setting up serving yourself.

These deployments are meant for testing rather than production. When you’re ready for production, email contact@distillabs.ai and we’ll set you up.

Creating a deployment

distil deployment create-from-slm --output json <slm-id> | jq -r .id
# <deployment-id>

distil deployment status --output json <deployment-id> | jq -r .deployment_status

The deployment serves the model with vLLM, and the job doesn’t return until vLLM answers, so JOB_SUCCESS means serving rather than merely scheduled. It can take up to 40 minutes.

Deployments report deployment_status and endpoint_status, and have no status field at all, unlike every other entity. Before the job succeeds, endpoint answers {"url": null, "api_key": null} and exits 0 either way, so poll the status rather than probing the endpoint.

Getting the endpoint

distil deployment endpoint --output json <deployment-id>
# {"url": "https://…", "api_key": "…"}

The API key is load-bearing: the tunnel has no authentication of its own, and the URL is otherwise open to anyone who has it.

Querying your model

Every trained model ships with its own inference client, generated at training time to match the prompt shape that model was trained with. Fetch it without downloading the several gigabytes of weights:

distil slm download-metadata --destination ./model <slm-id>
# ./model/config.yaml, ./model/job_description.json, ./model/model_client.py

Point it at the deployment. Take the URL and the API key from distil deployment endpoint <deployment-id> above. The client wants an OpenAI-style base URL, so drop the trailing slash from that URL and append /v1.

uv run ./model/model_client.py --base-url https://<endpoint>/v1 --api-key <api-key> \
  --conversation '[{"role": "user", "content": "Your question here"}]'

From Python:

from model_client import DistilLabsLLM

client = DistilLabsLLM(
    model_name="model",
    base_url="https://<endpoint>/v1",
    api_key="<api-key>",
)
print(client.invoke([{"role": "user", "content": "..."}]))

For question-answering models that need context, wrap it in a <context> tag followed by a newline, inside the first user message:

uv run ./model/model_client.py --base-url https://<endpoint>/v1 --api-key <api-key> \
  --conversation '[{"role": "user", "content": "<context>Your context here</context>\nYour question here"}]'

Why the client, and not a raw request

A small model degrades sharply outside its training-time setup, and a hand-built chat-completions request matches none of it. It fails quietly: the endpoint answers 200, reasoning text leaks into the answer, and the score falls well below what your evaluation metrics promised.

The client reproduces the setup exactly:

• The trained system prompt, baked in as SYSTEM_PROMPT.
• temperature=0, with thinking disabled (chat_template_kwargs: {"enable_thinking": false}).
• Tool-calling models call with tools=TOOLS, tool_choice="required" and return the first tool call.
• QA context inlined into the first user message as <context>…</context>, matching training and evaluation.

Use the client that came with the model you’re querying, and smoke-test any deployment with it on a few test-set rows before integrating.

Shutting it down

distil deployment delete <deployment-id>     # alias: shutdown

A deployment is a session, not a permanent endpoint

It stops on its own after six hours, or after one hour with no traffic, and it bills until that idle timeout. Delete it when you’re done.

It can’t be restarted, and a new deployment carries a new URL and a new key. Replacing a stopped one spends another deployments_from_slms_post credit, so have your test inputs ready before you create it. After the delete, deployment_status stays JOB_SUCCESS, and endpoint_status going to stopped is what confirms it’s down.

Doing this over the REST API

See Using the REST API.

Credits

Deployments are metered on deployments_from_slms_post, and a new account starts with two.

distil credits-balance

Reach out at contact@distillabs.ai when you need more.

Serving it yourself instead

Local deployment.