Model iterations

Improving a model means running the build loop again with changed settings. An iteration starts at any entry point of the build loop and ends with a trained model and its evaluation results. The last model’s predictions show what to change, and each iteration reuses everything upstream of the stage you change. New traces recorded by the serving endpoint can also start an iteration: create a traces object from them with your current test.jsonl, and process it with trace processing.

One iteration has five steps:

 1 Read the scores
   ▼
 2 Inspect the predictions
   ▼
 3 Find the stage the errors come from
   ▼
 4 Change one setting in that stage
   ▼
 5 Run from that stage through training ─────► back to 1

1. Read the scores

A trained model’s score is read against four others, all measured on the same test set:

• The teacher’s score from teacher evaluation, the upper limit of what the student can learn from it.
• The base student’s, the untrained student, which training starts from.
• The original production model’s, when your test set came from traces (Test set from traces).
• The previous iteration’s.

Scores are on the task’s primary metric: llm-as-a-judge-reference-free, or accuracy for classification. Two scores are comparable only when they come from the same test set and the same judge, and a difference is meaningful only when it is larger than the run-to-run variation of the evaluation. See Metrics.

Re-testing a model on a new test set

No command scores an existing model on new rows. To compare an earlier model on a new test set, deploy it, send each test row through its model_client.py pointed at the endpoint, and compare the answers with the references. Then delete the deployment. See Serving behind the endpoint.

2. Inspect the predictions

Download the trained student’s predictions and the teacher’s predictions on the same test set. Each command writes a JSONL file to the current directory:

distil slm download-predictions <slm-id>
distil teacher-evaluation download-predictions <teacher-evaluation-id>

Match rows between the two files on the last user message, since the prompts differ in their few-shot examples and one conversation can produce several evaluation rows, one per assistant turn. For each row the student got wrong, note what it answered, what the reference says, and whether the teacher got it right.

Group the errors into named failure modes: specific scenarios such as “refunds for annual plans”, “dates in European format” or “calls search when lookup was needed”, not “low score”. Count each mode. A mode with one example says little; a mode with many examples is a pattern.

3. Find the stage the errors come from

Each failure mode points at the earliest stage that can change it:

What the predictions show	Likely cause	Stage to change
The reference answer itself looks wrong, or the teacher and the student are wrong in the same way	wrong test labels	Test set from traces (relabeling), or correct the rows
The teacher is wrong where the student is wrong	the teacher cannot do the task	Teacher evaluation (another teacher), then synthetic data
The judge marks correct answers bad, or wrong answers good	the measure is wrong	the judge instructions; earlier scores are then not comparable
The teacher is right; the student is wrong on a slice the training data barely covers	coverage	Synthetic data generation
The teacher is right; the student is wrong the same way everywhere (format, a misread rule)	the training data teaches it wrong	Synthetic data generation, or Trace processing when the seed rows carry the mistake
The teacher is right; the errors have no pattern and the gap to the teacher is small	capacity or too little training	Model training
The test set has no rows for a failure you see in production	the test set cannot measure it	Test set from traces

distil training-dataset sample shows rows the student learned from, so a failure mode can also be looked for in the training data.

4. What can be changed

Changing one setting per iteration shows what that setting did. Several training runs in parallel (different students or epochs) still count as one change to training.

• Model training: the student (a larger one has more capacity, at a higher serving cost), the epochs and learning rate, the LoRA rank, the evaluation length. Nothing regenerates.
• Synthetic data generation: synthetic_data_generation_instructions, which moves every generated row the same way; mutators, which change the mix of the data; and the synthgen config, which changes how much is generated and how much survives validation.
• Teacher evaluation: the teacher. It writes the training data, so its ceiling is the student’s ceiling. A new teacher means synthetic data and training again.
• Trace processing: the relabeling teacher or committee, trace_processing_instructions, relevance filtering, and how many traces become seed rows.
• Test set from traces: the test set size and the relabeling settings. Scores measured on the earlier test set are not comparable with scores on the new one.

From least to most expensive: training; synthetic data or a new teacher; trace processing; the test set. Each stage page lists its settings in full.

task_description stays matched to the prompt your production system runs (Job description), so it is not one of the settings an iteration changes.

Reusing the last training data

Unless the last training data was bad, it can be the seed of the next one: download it, pair its train.jsonl with your test set and the same job description, create a new seed dataset from the directory, and set generation_target to the number of rows to add. Generation deduplicates against the seed, so the result is the old rows plus the new ones. The download spends a training_datasets_download_get credit, which starts at zero.

5. Run

Run the stage you changed and every stage after it, through training. The change is an override on the same parent entity (How the platform works), except when the iteration creates a new entity: a seed dataset built from reused data, or a new test set.

The agent skill runs this loop for you and decides what to change in each iteration.