Collect traces with an endpoint

An inference endpoint is an OpenAI-compatible gateway at https://inference.distillabs.ai/v1. Your application calls it with an ordinary chat completions request, and the endpoint keeps a record of every call. A collecting endpoint has one model behind it, the fallback: the model your application calls in production today. The fallback answers every call exactly as it does now, and the records become the traces your first model is built from.

The fallback is named by its OpenRouter slug in owner/model form. If your production model is not on OpenRouter, export its logs and start from Trace inputs instead.

Create the endpoint

Create the endpoint with a name prefix and your production model as the fallback:

distil inference-endpoint create --name support --fallback-model "openai/gpt-4.1-mini"

--name is a prefix. The platform appends a suffix, and the command prints the unique endpoint name on its Endpoint Name line, here support-yeOdAS. Every other command takes that name, and so does the model field of every request, so record it. There is no rename, and distil inference-endpoint list is how you find it again.

--fallback-model is the model your application calls today. Name the same model, so the endpoint answers exactly as production does and the traces describe the system you want to replace.

By default the endpoint records every call. --trace-sample-rate 0.2 records one call in five, which is worth setting only when traffic is very high. The rate is fixed at creation, and distil inference-endpoint show reports it as trace_sampling_rate.

An endpoint cannot be edited or deleted. Changing its fallback means a new endpoint with a new unique name.

Create and link a key

A caller authenticates with an inference API key. It is a separate credential from your CLI session, and it reaches the endpoint and nothing else. Create a key named support-prod and link it to the endpoint:

distil api-keys create support-prod
distil inference-endpoint link-api-key support-yeOdAS support-prod

distil api-keys create prints the secret and writes it to support-prod.json. The secret is shown at creation and never again. Use --no-file when it should go straight into a secret store. One key can be linked to several endpoints, so the same key works on the endpoint that serves your trained model later. An account holds at most 5 keys. A newly linked key takes up to a minute to start working.

Point your application at it

Change three strings in your existing OpenAI client: the base URL, the key, and the model name. In the examples below, <api-key> is the secret of support-prod.

from openai import OpenAI

client = OpenAI(
    base_url="https://inference.distillabs.ai/v1",
    api_key="<api-key>",
)
client.chat.completions.create(
    model="support-yeOdAS",
    messages=[
        {"role": "system", "content": "<your production system prompt>"},
        {"role": "user", "content": "…"},
    ],
)

Keep everything else, including the system prompt your application sends. The endpoint records the whole request, and the system prompt’s content goes into the job description’s task_description (see Trace inputs).

Send one request by hand before you move the application:

curl https://inference.distillabs.ai/v1/chat/completions \
  -H "Authorization: Bearer <api-key>" \
  -H "Content-Type: application/json" \
  -d '{"model": "support-yeOdAS", "messages": [{"role": "user", "content": "Say hi in three words."}]}'

A chat completions response means the key, the link and the fallback work. The endpoint’s page in the dashboard prints this call with your own name filled in.

Let it collect

Traces accumulate at the rate of your traffic, so this step takes days to weeks. The test set and the seed dataset are both built from these traces: 400 at the default settings (How many traces you need). Check how many the endpoint has recorded by downloading every record since a start date:

distil inference-endpoint download-traces --from-start-time 2026-09-01 --all support-yeOdAS

The command prints the count and the file it wrote, for example Downloaded 312 traces to support-yeOdAS-traces.jsonl.

Next

Turn the records into a traces object.