Teacher evaluation

Teacher evaluation runs the teacher model on your full test set. It answers one question: can the teacher solve this task at all?

If it can’t, nothing downstream will. The student learns from the teacher, so the teacher’s score is the ceiling everything else distils from. This stage takes minutes and costs one credit, which makes it the cheapest place by far to find out that a task isn’t going to work.

Running it

distil teacher-evaluation create-from-seed-dataset --output json <seed-dataset-id> | jq -r .id
# <teacher-evaluation-id>

distil teacher-evaluation status --output json <teacher-evaluation-id> | jq -r .status

It runs on the full test set every time, there’s no partial mode, and it usually finishes within 30 minutes. Each run spends one teacher_evaluations_post credit.

Configuring the evaluation

• base.teacher_model_name. The model under evaluation, and the one you’re auditioning for synthetic data generation. Choose it deliberately from Supported models, since tool-calling tasks restrict the choice. Trying a different teacher is the main reason to run this stage again.
• llm_as_a_judge_instructions (in job_description.json). Matters most for free-text tasks, since it defines what the judge accepts and a vague one makes every downstream number noisy. Not valid for classification. See Job description.
• evaluation.num_few_shot_examples (default 1). How many worked examples the teacher is shown at evaluation time. This is unrelated to the per-class minimum on the training data you supply.
• evaluation.llm_as_a_judge_model_name. Picks the judge model. It defaults to base.teacher_model_name, which is usually what you want.

Change the judge whenever you change the teacher. The judge default resolves when the config is first expanded, so a config read back for an override already names a specific judge. Change the teacher alone and your new teacher gets judged by the old one. Nothing errors, the numbers just aren’t comparable.

Reading the results

Two things come back by different routes: the aggregated scores inline, and the per-example predictions as a separate file.

# Aggregated scores
distil teacher-evaluation metrics --output json <teacher-evaluation-id> | jq .teacher_performance
# {"rouge": 1, "binary": 0.82, "llm-as-a-judge": 1, "llm-as-a-judge-reference-free": 1}

# Per-example detail, one row per test example
distil teacher-evaluation download-predictions <teacher-evaluation-id>

The predictions file is JSONL with prompt, completion, prediction and that example’s own score under each metric name. Note that prompt is the full prompt as a JSON-encoded message list, not the user text alone, so parse these rather than comparing them as raw strings.

Which metric to read for your task: Metrics.

Analyse in this order

1. Sanity-check the judge first. Sample some predictions the judge marked bad. If they look correct to you, the score is lying and the judge instructions need fixing before you look at anything else.
2. Then look for failure patterns. Group the incorrect predictions. One class always wrong, or one format always missed, points at a targeted fix rather than a fundamental problem.

The failure patterns are in the per-example file, not in the aggregate score.

Deciding whether to continue

Review the score and the failure patterns, and ask whether this quality is acceptable in production. There’s no absolute threshold, because 0.72 is a good result against a teacher ceiling of 0.75 and a poor one against a ceiling of 1.00.

If it’s good enough, continue to synthetic data generation.

If it isn’t, the primary lever is the teacher model. This stage is where you pick the right teacher for the problem, so try a better-suited one from Supported models. Fixing mislabelled or ambiguous test examples is fair game too.

If a better teacher doesn’t help either, step back before touching more levers. Is the task type right? Is the task well defined? Is the judge measuring the right thing? See Improving your model.

One lever to leave alone: task_description. It has to stay compliant with the system prompt you run in production, and constant across iterations, or you’re no longer comparing like with like. Adjust the judge instructions only when step 1 above showed the judge is mismeasuring.

Iterating

A different teacher is a config change, and sharper judge instructions are a job-description change. Both are overrides on the same seed dataset, and neither needs a new one:

distil seed-dataset download-metadata -d ./iter-2 <seed-dataset-id>
# ./iter-2/config.yaml          → base.teacher_model_name (and the judge model)
# ./iter-2/job_description.json → llm_as_a_judge_instructions

distil teacher-evaluation create-from-seed-dataset --output json \
  --config ./iter-2/config.yaml \
  --job-description ./iter-2/job_description.json <seed-dataset-id> | jq -r .id

Each file is sent whole and anything omitted reverts to the library default, so read the parent’s file back, change one field, and send all of it. See How the platform works.

Every score here is a sample. The judge runs at non-zero temperature, so the same model on the same test set scores differently run to run, a measured ±0.03 on a 50-row set. Any difference smaller than that band is noise rather than an improvement.