Improving your model

Your first pass shipped a model, or stalled short of the bar, and you know where it falls down. Improving it isn’t starting over. It’s a second iteration of the same pipeline that reuses the first iteration’s artifacts.

The job description stays constant throughout, since changing it means you’re no longer comparing like with like.

1. Diagnose the gaps ─── name the failure modes; they become mutation topics
        │
        ▼
2. Refresh teacher evaluation ─── only if the teacher or test set changed
        │
        ▼
3. Generate iteration-2 data ─┬─ iteration-1 data good ► blend and top up
        │                     └─ iteration-1 data bad ─► start fresh, full target
        ▼
4. Retrain ─── usually the fast path with iteration 1's winning student
        │
        ▼
5. Compare ─── iteration 2 against iteration 1, on the same test set

1. Diagnose the gaps

From the training analysis, the prediction file, and what you know from production, name the failure modes concretely. Specific scenarios, not “the score is too low”.

These names become your mutation topics in step 3, which is why the specificity matters. “Billing disputes where the customer has already been refunded” is a topic. “Accuracy” isn’t.

The per-example predictions file is where these live:

distil slm download-predictions <slm-id>

Group the rows the model got wrong by whatever your task’s failure modes are: a class, a format, a rule. Reading the aggregate and guessing doesn’t work.

2. Refresh teacher evaluation, only if needed

Skip it when the teacher, the job description and the test set are all unchanged.

Run it again when you’re switching the teacher, so the ceiling is refreshed before anything gets judged against it. See Teacher evaluation.

3. Generate iteration-2 data

Run synthetic data generation again, with two changes.

New mutation topics

Encode the gaps you named in step 1 as synthgen.mutation_topics. This is the whole reason for naming them concretely.

The seed depends on iteration-1 data quality

If iteration 1’s data was good, blend and top up. Use the previous iteration’s generated dataset as the new seed. De-duplication runs against the seed, so the new run avoids repeating iteration 1, and the output is old plus new, merged.

# Download iteration 1's merged train.jsonl
distil training-dataset download -d ./iter1 <training-dataset-id>

# Pair it with your test set and the unchanged job description,
# then stage the directory as a fresh seed dataset
distil seed-dataset create --data ./iter2-input

Set generation_target to the top-up amount, not the total. It counts new examples only, so a fresh 10,000 against a blended seed doubles your dataset rather than replacing it.

Two things to check before choosing this branch. It spends a seed_datasets_post credit on top of the generation one, and it needs training_datasets_download_get, which starts at zero, because holding the merged file is the whole reason for the branch and the free sample returns at most 128 rows. At zero on the download route, take the fresh-seed branch below whatever the data quality.

Keep the test set the one you’re judging against, not iteration 1’s training rows.

If iteration 1’s data was bad, start fresh. Seed from your original data with a full generation_target, after fixing whatever made iteration 1’s data bad, usually the generation config or the mutators. Blending bad data in just carries the problem forward.

4. Retrain

Run model training on the new dataset. The student is usually already chosen, iteration 1’s winner, so this is a single run rather than a sweep. Re-sweep only if the gaps suggest a capacity problem rather than a data problem.

5. Compare

Read the results as you did the first time (see model training), plus one more comparison: iteration 2 against iteration 1 on the same metrics.

Compare only scores measured on the same test set. If you changed the test set between iterations, iteration 1’s recorded score was measured on the old one, so it isn’t a baseline. Either re-score iteration 1 against the new test set first, or say plainly that iteration 2 has no prior to beat and judge it against the teacher and base student alone. Two numbers from two different test sets don’t belong in the same column.

Improved and good enough, deploy it. Still gapped, back to step 1 with what the new analysis shows.

When the problem is upstream

Not every disappointing result is a data problem. Before running another iteration, check whether the answer is further back:

• The teacher score was never good enough. The student can’t exceed the teacher, so go back to teacher evaluation and try a better-suited teacher.
• The task type is wrong. See Task selection.
• The judge is mismeasuring. Sample predictions marked bad, and if they look correct, fix llm_as_a_judge_instructions before anything else. See Job description.
• The test set doesn’t cover the failure. You can’t fix what you can’t measure, so collect or curate test examples for the gap first.