Inference endpoint

An inference endpoint is an OpenAI-compatible gateway at https://inference.distillabs.ai/v1. Your application calls it with an ordinary chat completions request, exactly as it calls your LLM provider today, and the model behind it answers. The endpoint keeps a record of every call.

That is the whole point of starting here: you change three strings in your client and nothing else, your users see the same answers, and the records become the traces your first model is built from. Later, the trained model goes behind an endpoint of its own and takes the traffic (Serving behind the endpoint).

An endpoint has one model behind it to start with, the fallback: the model your application calls in production today, named by its OpenRouter slug in owner/model form. If your production model is not on OpenRouter, export its logs and start from Dataset instead.

Create the endpoint

Create the endpoint with a name prefix and your production model as the fallback:

distil inference-endpoint create --name support --fallback-model "openai/gpt-4.1-mini"

--name is a prefix. The platform appends a suffix, and the command prints the unique endpoint name on its Endpoint Name line, here support-yeOdAS. Every other command takes that name, and so does the model field of every request, so record it. There is no rename, and distil inference-endpoint list is how you find it again.

Name the same model your application calls today, so the endpoint answers exactly as production does and the traces describe the system you want to replace. An endpoint cannot be edited or deleted; changing its fallback means a new endpoint with a new unique name.

Create and link a key

A caller authenticates with an inference API key. It is a separate credential from your CLI session, and it reaches the endpoint and nothing else. Create a key named support-prod and link it to the endpoint:

distil api-keys create support-prod
distil inference-endpoint link-api-key support-yeOdAS support-prod

distil api-keys create prints the secret and writes it to support-prod.json. The secret is shown at creation and never again. Use --no-file when it should go straight into a secret store. One key can be linked to several endpoints, so the same key works on the endpoint that serves your trained model later. An account holds at most 5 keys. A newly linked key takes up to a minute to start working.

Point your application at it

Change three strings in your existing OpenAI client: the base URL, the key, and the model name. In the examples below, <api-key> is the secret of support-prod.

from openai import OpenAI

client = OpenAI(
    base_url="https://inference.distillabs.ai/v1",
    api_key="<api-key>",
)
client.chat.completions.create(
    model="support-yeOdAS",
    messages=[
        {"role": "system", "content": "<your production system prompt>"},
        {"role": "user", "content": "…"},
    ],
)

Keep everything else, including the system prompt your application sends. The endpoint records the whole request, and the system prompt’s content goes into the job description’s task_description later (Dataset overview).

Send one request by hand before you move the application:

curl https://inference.distillabs.ai/v1/chat/completions \
  -H "Authorization: Bearer <api-key>" -H "Content-Type: application/json" -d '{"model": "support-yeOdAS", "messages": [{"role": "user", "content": "Say hi in three words."}]}'

A chat completions response means the key, the link and the fallback work. The endpoint’s page in the dashboard prints this call with your own name filled in.

Next

Collect traces with the endpoint.