Teacher evaluation

Teacher evaluation runs the teacher model on your full test set and scores its answers. It shows whether the teacher can solve the task at all.

The student learns from data the teacher writes, so the teacher’s score is the ceiling for the student. This stage costs one credit, which makes it the least expensive place to find out that a task isn’t going to work.

Running it

Start the evaluation on your seed dataset. The command prints the new teacher evaluation’s id:

distil teacher-evaluation create-from-seed-dataset --output json <seed-dataset-id> | jq -r .id

Check its status with that id until it reads JOB_SUCCESS:

distil teacher-evaluation status --output json <teacher-evaluation-id> | jq -r .status

The evaluation runs on the full test set every time; there is no smoke run. It usually finishes within 30 minutes, and each run spends one teacher_evaluations_post credit.

The test set must be populated: an empty test.jsonl stops the job. The evaluation takes its few-shot examples from the seed dataset’s training set. Only when you have no training set, set evaluation.num_few_shot_examples: 0.

Configuring the evaluation

• base.teacher_model_name. The model under evaluation, and the one synthetic data generation will use. Choose it from Supported models.
• llm_as_a_judge_instructions (in job_description.json). What the judge accepts on free-text tasks. Vague instructions make every score less reliable. Not valid for classification. See Job description.
• evaluation.num_few_shot_examples (default 1). Worked examples from your training set, at most the number of training rows.
• evaluation.llm_as_a_judge_model_name. The judge model. Scores from two different judges are not comparable, so the same judge is used in every run, including when the teacher changes.

Reading the results

The aggregated scores and the per-example predictions come back separately. Read the scores:

distil teacher-evaluation metrics --output json <teacher-evaluation-id> | jq .teacher_performance

The command prints one score per metric:

{"rouge": 1, "binary": 0.82, "llm-as-a-judge": 1, "llm-as-a-judge-reference-free": 1}

Download the per-example predictions:

distil teacher-evaluation download-predictions <teacher-evaluation-id>

This writes <teacher-evaluation-id>-teacher-evaluation-predictions.jsonl to the current directory, one test example per line, with prompt, completion, prediction and that example’s score under each metric name.

The primary metric is llm-as-a-judge-reference-free for question answering and chat completion, and accuracy for classification (Metrics). There’s no absolute good score: the teacher’s score is the reference point the student is read against later, so 0.72 from the student is close to a teacher at 0.75 and far from a teacher at 1.00.

The predictions show why the score is what it is:

• Predictions the judge marked bad that look correct mean the judge is mismeasuring, and the judge instructions are what to change.
• Errors that share a pattern, such as one class always wrong or one format always missed, point at that class or format rather than at the teacher as a whole.

Scores vary between runs of the same evaluation (Metrics).

What can be changed

• The teacher (base.teacher_model_name). It writes the training data, so a stronger or better-suited teacher raises what the student can reach. The judge stays the same.
• The judge instructions (llm_as_a_judge_instructions). What counts as a correct answer on the free-text metrics, including format rules. Change them, or use a stronger judge model, when the judge mismeasures. Every score measured with different instructions or a different judge is not comparable.
• The test rows. Wrong or ambiguous reference answers lower every score. Fixing them changes the data, so it needs a new seed dataset.

task_description stays as it is: it mirrors the system prompt you run in production and is the same in every iteration.

The teacher and the judge instructions are overrides on the same seed dataset. Download the seed dataset’s config.yaml and job_description.json into te/iter-2:

distil seed-dataset download-metadata -d te/iter-2 <seed-dataset-id>

Change base.teacher_model_name in te/iter-2/config.yaml, or llm_as_a_judge_instructions in te/iter-2/job_description.json, then submit both files with the evaluation:

distil teacher-evaluation create-from-seed-dataset --output json \
  --config te/iter-2/config.yaml \
  --job-description te/iter-2/job_description.json <seed-dataset-id> | jq -r .id

Each file replaces the parent’s whole, and anything you leave out takes its default, so read the parent’s file back, change one field, and send all of it (How the platform works).

Next

Synthetic data generation uses the teacher you evaluated here.