Overview

Building a model on distil labs is one loop. An inference endpoint records the traffic of the model you run in production. Those records become a traces object, the traces become a test set and a seed dataset, the teacher writes the training data from the seed dataset, and the student is trained on it. The trained student then goes behind a new inference endpoint, which records the traffic it serves, and those records start the next iteration.

 A  LLM in production ──►  1  Collect traces
                              │
 B  trace file ─────────►  2  Create the traces object ◄──┐
                              │                           │
                           3  Test set from traces        │
                              │                           │
 C  labeled dataset ────►  4  Seed dataset                │
                              │                           │ next
                           5  Teacher evaluation          │ iteration
                              │                           │
                           6  Synthetic data              │
                              │                           │
                           7  Model training              │
                              │                           │
                           8  Serve behind an endpoint ───┘

Where you start

You have	Start at
An LLM in production that OpenRouter serves, and you can point your application at an inference endpoint	Collect traces with an endpoint
A file of production traces, including exported logs of a production model that OpenRouter does not serve	Trace inputs
A labeled dataset with a test set, and no production traffic to collect	Seed dataset

An inference endpoint’s fallback is an OpenRouter model, and the last step serves your model behind an endpoint. So when your production model is on OpenRouter, starting with the endpoint uses the same endpoint setup at both ends of the loop.

A labeled dataset brings its own test set, which every score is measured on. Without traces there is no original model to score, so its results have no original-model score to compare with.

The stages

Durations are for a full-size run; a small dataset finishes in minutes.

Stage	What it does	Typical duration
Task selection	Picks the task type, which sets the shape of your data	-
Collect traces with an endpoint	Records your production traffic	as long as your traffic takes
Endpoint records to traces	Turns the records into a traces object	-
Test set from traces	Builds the test set and scores the original model	scales with the test set size
Trace processing	Turns the traces into a seed dataset	45 min
Teacher evaluation	Scores the teacher on your test set	30 min
Synthetic data generation	The teacher writes the training data	90 min
Model training	Fine-tunes and evaluates the student	90 min
Serving behind the endpoint	Puts the student in front of your traffic	40 min
Model iterations	Finds what to change for the next iteration	-

How the jobs, entities and credits behind these stages work is in How the platform works. The agent skill runs this loop for you and decides what to run next.

The two files every entry needs

Every entry point needs two files, written before you create the traces object or the seed dataset:

• config.yaml sets the task type and the models.
• job_description.json describes the task the model learns.

An example config. Every field not shown has a default; Config file lists them all:

base:
  task: question-answering
  student_model_name: Qwen3.5-4B
  teacher_model_name: zai.glm-5.3-low-thinking
evaluation:
  llm_as_a_judge_model_name: zai.glm-5.3-low-thinking

An example job description for question answering. The fields for the other task types are in Job description:

{
  "task_description": "Answer the customer's question about their order in one or two sentences, using only the order details in the message."
}