Collect traces with an endpoint

An inference endpoint is an OpenAI-compatible gateway that sits in front of the model you already run in production. Your application calls the endpoint instead of the provider, the model you name as the fallback answers exactly as it does today, and the endpoint keeps a copy of every call. Those copies are traces, and trace processing turns them into the seed dataset for your student model.

This is the default way to start: send us your production traffic and the platform collects the data for you. If you can’t route traffic through us but hold a file of production logs, go to Trace inputs. If you have no LLM in production at all, start from a seed dataset.

Create the endpoint

distil inference-endpoint create --name support --fallback-model "openai/gpt-4.1-mini"
# Endpoint Name:   support-yeOdAS
# Endpoint:        https://inference.distillabs.ai/v1/chat/completions

--name is a prefix. The platform appends a suffix and returns the unique endpoint name, here support-yeOdAS. Every other command takes that name, and so does the model field of every request, so record it. There is no rename, and distil inference-endpoint list is how you find it again.

--fallback-model is the model your application calls today, as an OpenRouter slug in owner/model form. Name the same model, so the endpoint answers exactly as production does and the traces describe the system you want to replace.

By default the endpoint records every call. --trace-sample-rate 0.2 records one call in five, which is worth setting only when traffic is very high. The rate is fixed at creation, and distil inference-endpoint show reports it as trace_sampling_rate. Endpoints created with a CLI older than 0.27.0 record one call in a hundred, so create a new one if you need every call.

Create and link a key

A caller authenticates with an inference API key. It’s a separate credential from your CLI session, and it reaches the endpoint and nothing else.

distil api-keys create support-prod
# prints the secret once, and writes support-prod.json

distil inference-endpoint link-api-key support-yeOdAS support-prod

The secret is shown at creation and never again. Use --no-file when it should go straight into a secret store. One key can be linked to several endpoints, which matters later when a second endpoint fronts your trained model. A newly linked key takes up to a minute to start working.

Point your application at it

Change three strings in your existing OpenAI client: the base URL, the key, and the model name.

from openai import OpenAI

client = OpenAI(
    base_url="https://inference.distillabs.ai/v1",
    api_key="<support-prod secret>",
)
client.chat.completions.create(
    model="support-yeOdAS",
    messages=[
        {"role": "system", "content": "<your production system prompt>"},
        {"role": "user", "content": "…"},
    ],
)

Nothing else changes. Keep the system prompt your application sends today. The endpoint records the whole request, and trace processing later moves the system prompt’s content into the job description, so what the model is trained on is what production asked for.

Send one request by hand before you move the application:

curl https://inference.distillabs.ai/v1/chat/completions \
  -H "Authorization: Bearer <support-prod secret>" \
  -H "Content-Type: application/json" \
  -d '{"model": "support-yeOdAS", "messages": [{"role": "user", "content": "Say hi in three words."}]}'

The endpoint’s page in the dashboard prints this call with your own name filled in.

Let it collect

Traces accumulate at the rate of your traffic, and trace processing wants a few hundred of them, so this step takes hours to weeks depending on your volume. Records become downloadable about an hour after the call. Check how many the endpoint has recorded with a download:

distil inference-endpoint download-traces --all support-yeOdAS
# Downloaded 312 traces to support-yeOdAS-traces.jsonl

Next

Turn the records into trace inputs.