Serving behind the endpoint

Your student was trained on traffic that an inference endpoint collected. This page puts it in front of that traffic: a hosted deployment serves the model, a new endpoint calls it first and falls back to the production model when it can’t answer, and your application moves over by changing one string.

A hosted deployment is a session for trying the student on live traffic: it stops after six hours, or after an hour with no traffic. To serve the student permanently, email contact@distillabs.ai.

1. Deploy the model

Deploy the trained model. The command prints the new deployment’s id:

distil deployment create-from-slm --output json <slm-id> | jq -r .id

Check its status with that id. It reads JOB_SUCCESS after about 40 minutes:

distil deployment status --output json <deployment-id> | jq -r .deployment_status

2. Create an endpoint with the deployment as primary

Create the endpoint once the deployment reports JOB_SUCCESS:

distil inference-endpoint create-from-deployment --name support-slm \
  --fallback-model "openai/gpt-4.1-mini" <deployment-id>

The command prints the unique endpoint name on its Endpoint Name line, here support-slm-Qk3bZ1. Link it to support-prod, the key you created in Collect traces with an endpoint:

distil inference-endpoint link-api-key support-slm-Qk3bZ1 support-prod

--fallback-model is the model production runs today, so a request the student can’t serve is answered the way it was answered before. The endpoint calls the deployment first and falls back whenever it fails, a timeout included.

An endpoint can’t be edited, so this is a new endpoint, never the one that collected the traces. The old one keeps recording until your application moves, and the key linked to it can be linked to this one too.

Try a few test-set rows before the application moves, with the model’s own client, as on distil labs inference. Download the client into ./model and send a row through it, where <api-key> is the secret of support-prod:

distil slm download-metadata --destination ./model <slm-id>
uv run ./model/model_client.py --base-url https://inference.distillabs.ai/v1 \
  --api-key <api-key> --model support-slm-Qk3bZ1 \
  --conversation '[{"role": "user", "content": "Your question here"}]'

Then check which model answered. Every record the endpoint writes names it: metadata.source is primary for the student and fallback for the production model. A fallback answer to these calls usually means the deployment failed or has stopped. Download the records and count them by source:

distil inference-endpoint download-traces support-slm-Qk3bZ1
jq -r .metadata.source support-slm-Qk3bZ1-traces.jsonl | sort | uniq -c

3. Move the application

Change the model string to the new unique endpoint name. The system prompt must be the one the student was trained with: the task_description of its job description. If your production prompt differs from it, change the prompt to it, or call through model_client.py (Why the client).

client.chat.completions.create(
    model="support-slm-Qk3bZ1",
    messages=[
        {"role": "system", "content": "<the task_description from the job description>"},
        {"role": "user", "content": "…"},
    ],
)

Any new caller or integration uses the model’s model_client.py instead, which sends exactly the request the model was trained and evaluated with.

The deployment is a session

Once the deployment stops it can’t be restarted. From then on every call goes to the fallback, the endpoint keeps answering, and source reads fallback on every record. Delete the deployment when the student should stop answering: distil deployment delete <deployment-id>.

4. Start the next iteration

The new endpoint records its traffic like the first one did. With a permanent deployment, those records are the student’s traffic; after a session ends they are the fallback’s answers. Either way they are the traces for the next iteration: turn them into a traces object as in Endpoint records to traces, filtering on source when you want only one model’s answers, and continue with trace processing. Model iterations covers what to change from one iteration to the next.