Serving behind the endpoint

Your student was trained on traffic that an inference endpoint collected. This page puts it in front of that traffic: a hosted deployment serves the model, a new endpoint calls it first and falls back to the production model when it can’t, and your application moves over by changing one string.

1. Deploy the model and check it answers

distil deployment create-from-slm --output json <slm-id> | jq -r .id
# <deployment-id>

distil deployment status --output json <deployment-id> | jq -r .deployment_status
# JOB_SUCCESS, after up to 40 minutes

distil deployment endpoint --output json <deployment-id>
# {"url": "https://…/", "api_key": "…"}

Query the deployment directly before you put anything in front of it, with the model’s own client, as on distil labs inference:

distil slm download-metadata --destination ./model <slm-id>
uv run ./model/model_client.py --base-url https://<deployment-host>/v1 --api-key <deployment-api-key> \
  --conversation '[{"role": "user", "content": "Your question here"}]'

Try a few test-set rows. Behind the endpoint, a request the model can’t serve is answered by the fallback, so check the model on its own first.

2. Create an endpoint with the deployment as primary

distil inference-endpoint create --name support-slm \
  --fallback-model "openai/gpt-4.1-mini" \
  --primary-url "https://<deployment-host>" \
  --primary-api-key "<deployment-api-key>"
# Endpoint Name:   support-slm-Qk3bZ1

distil inference-endpoint link-api-key support-slm-Qk3bZ1 support-prod

--primary-url is the deployment’s URL as deployment endpoint prints it, without /v1: the endpoint appends /v1/chat/completions itself. --primary-api-key is the key from the same output. The two flags come together or not at all. --fallback-model is the model production ran before, so a request the student can’t serve is answered the way it was answered yesterday.

The endpoint calls the primary first, and falls back whenever the primary fails, a timeout included. It forwards the request as received, with the model field rewritten to the name the deployment serves, so there’s nothing to configure on the deployment side.

An endpoint can’t be edited after it’s created, which is why this is a new endpoint rather than a change to the one that collected the traces. The old one keeps recording until your application moves, and the key you linked to it works on this one too.

3. Move the application

Change the model string to the new name. Keep everything else, including the system prompt: the student was trained against a job description that mirrors it, and it expects the same request shape it saw during collection.

client.chat.completions.create(
    model="support-slm-Qk3bZ1",
    messages=[
        {"role": "system", "content": "<the same production system prompt>"},
        {"role": "user", "content": "…"},
    ],
)

Where your code can use it, the model’s own model_client.py is the safer client, since it sends exactly the request the model was evaluated with. Point it at the endpoint:

uv run ./model/model_client.py --base-url https://inference.distillabs.ai/v1 \
  --api-key <support-prod secret> --model support-slm-Qk3bZ1 \
  --conversation '[{"role": "user", "content": "Your question here"}]'

Every record the new endpoint writes names which model answered: metadata.source is primary for the student and fallback for the production model.

distil inference-endpoint download-traces support-slm-Qk3bZ1
jq -r .metadata.source support-slm-Qk3bZ1-traces.jsonl | sort | uniq -c

The deployment behind the primary is a session

A hosted deployment stops after six hours, or after an hour with no traffic, and it can’t be restarted. From that moment every call goes to the fallback, the endpoint keeps answering, and source reads fallback on every record. This setup is for trying the student on live traffic. When the student holds up, email contact@distillabs.ai for an endpoint whose primary stays up.

4. Start the next iteration

The new endpoint records its traffic like the first one did. Those records are the traces for the next round: download them, keep the answers you want the next student to learn from by filtering on source, and go back to Endpoint records to trace inputs. Everything from there on is the same pipeline, with the student’s own score on the new test set as the baseline to beat.

Serving it yourself instead

A model served on your own infrastructure can be the primary too. Serve it with vLLM as on Local deployment, reachable from the internet behind an API key, and pass its base URL and key as --primary-url and --primary-api-key.