Teacher evaluation

Teacher evaluation runs a model on your full test set and scores its answers. On the test-set dataset it serves two purposes, both before the train set is built:

• Pick the teacher. One evaluation per candidate on the same test set; the best one by the primary metric writes the train set. The student learns from data the teacher writes, so the teacher’s score is the ceiling for the student. If no teacher can solve the task, nothing downstream will.
• Score the production model. One evaluation with the model you run in production as the teacher. Its score is the baseline your student has to beat.

The evaluation runs on the full test set every time; there is no smoke run. Each run spends one teacher_evaluations_post credit (new accounts start with 5), so plan the shortlist against the balance.

Running it

Start the evaluation on the test-set dataset. The command prints the new teacher evaluation’s id:

distil teacher-evaluation create-from-dataset <dataset-id>

Check its status with that id until it reads Success:

distil teacher-evaluation status <teacher-evaluation-id>

The test set must be populated: an empty test.jsonl stops the job. The evaluation is zero-shot: the teacher answers each test row with its system prompt and nothing added, so it doesn’t read the train split, which is usually still empty at this point.

Each candidate is a config override on the same dataset. Download the dataset’s files into te/<teacher>, set base.teacher_model_name, and submit:

distil dataset download-metadata -d te/glm <dataset-id>
# in te/glm/config.yaml: base.teacher_model_name: zai.glm-5.3-low-thinking
distil teacher-evaluation create-from-dataset --config te/glm/config.yaml <dataset-id>

Each file replaces the parent’s whole, and anything you leave out takes its default, so read the parent’s file back, change one field, and send all of it (How the platform works). The runs are independent; submit the whole shortlist at once.

The shortlist

Start with the default teacher (Supported models) and the production model, plus any teacher you want to compare. Two runs is the usual shortlist; add more when the task is unusual.

The production model as the baseline

The production model is named by its entry in the teacher catalog (provider.model, Supported models), not by the OpenRouter slug your endpoint uses. Find the matching entry and run an evaluation with it as base.teacher_model_name. Its teacher_performance is the production model’s answers scored with the same metrics and the same references as your student will be, which makes it the floor the student has to clear (Metrics).

When the production model has no catalog entry (most proprietary models), there is no baseline: judge the student on its absolute score and on the gap to the teacher.

Configuring the evaluation

• base.teacher_model_name. The model under evaluation.
• llm_as_a_judge_instructions (in job_description.json). What the judge accepts on free-text tasks. Vague instructions make every score less reliable. Not valid for classification. See Job description.
• evaluation.llm_as_a_judge_model_name. The judge model. Scores from two different judges are not comparable, so the same judge is used in every run, including when the teacher changes.

Reading the results

The aggregated scores and the per-example predictions come back separately. Read the scores:

distil teacher-evaluation metrics <teacher-evaluation-id>

The command prints one score per metric: rouge, binary, llm-as-a-judge and llm-as-a-judge-reference-free for question answering and chat completion, the tool-call metrics when the task has tools, and accuracy with the per-class scores for classification.

Download the per-example predictions:

distil teacher-evaluation download-predictions <teacher-evaluation-id>

This writes <teacher-evaluation-id>-teacher-evaluation-predictions.jsonl to the current directory, one test example per line, with prompt, completion, prediction and that example’s score under each metric name.

Put the shortlist’s scores on the primary metric in one table: llm-as-a-judge-reference-free for question answering and chat completion, accuracy for classification (Metrics). There’s no absolute good score: the best teacher’s score is the reference point the student is read against later, and the production model’s score is the floor.

The predictions show why a score is what it is:

• Predictions the judge marked bad that look correct mean the judge is mismeasuring, and the judge instructions are what to change.
• Errors that share a pattern, such as one class always wrong or one format always missed, point at that class or format rather than at the teacher as a whole.
• A teacher below the production model can’t write a train set that beats it.
• A candidate that also relabeled the test set is scored against its own answers on the reference-based metrics, which flatters it. The reference-free judge metric doesn’t have that bias.

Scores vary between runs of the same evaluation (Metrics).

What can be changed

• The teacher (base.teacher_model_name). It writes the training data, so a stronger or better-suited teacher raises what the student can reach. The judge stays the same.
• The judge instructions (llm_as_a_judge_instructions). What counts as a correct answer on the free-text metrics, including format rules. Change them, or use a stronger judge model, when the judge mismeasures. Every score measured with different instructions or a different judge is not comparable, so every evaluation runs again.
• The test rows. Wrong or ambiguous reference answers lower every score. Fixing them changes the data (Building the test set).

task_description stays as it is: it mirrors the system prompt you run in production and is the same in every iteration. With base.variable_system_prompt on, it describes the task, and a second evaluation with the flag off checks that the answers depend on each row’s system prompt (System prompts).

Next

Record the chosen teacher and the production model’s score with the test-set dataset ID, and build the train set with that teacher.