Metrics

Aggregated scores come back inline as the *_performance object in an entity’s metrics response. Per-example predictions are a separate download alongside it.

Metrics by task

Suite	Tasks	Metric keys
QA	question-answering, -open-book, -closed-book	rouge, binary, llm-as-a-judge, llm-as-a-judge-reference-free
Classification	classification	accuracy, plus one key per class label
Tool calling	tool-calling-closed-book, multi-turn-tool-calling-closed-book	rouge, tool_call_equivalence, binary_tool_call, staged_tool_call, llm-as-a-judge, llm-as-a-judge-reference-free

All metrics are on a 0-1 scale.

• llm-as-a-judge. An LLM verdict against the reference answer: good = 1, bad = 0. The judge sees the full prefix, including the question and any context. The -reference-free variant omits the reference. Unparseable verdicts become NaN and drop out of the mean.
• accuracy. Exact label match.
• binary. Exact string equality on the raw strings. Useful as a cross-check, not as something to rely on.
• rouge. Text overlap. A secondary signal.
• tool_call_equivalence. Exact match, except that an argument set to its schema default counts as equal to omitting it.
• binary_tool_call. Strict name and argument equality.
• staged_tool_call. 0.25 per stage: one call on each side, then the name matches, then the argument keys match, then the arguments match. A score of 0.5 means the right tool with the wrong arguments.

The classification object isn’t flat

Alongside accuracy it carries one key per class label, each holding a per-class breakdown, which gives you precision and recall for free:

{
  "accuracy": 1.0,
  "lane_cold":    {"precision": 1.0, "recall": 1.0, "f1-score": 1.0, "support": 9.0},
  "lane_general": {"precision": 1.0, "recall": 1.0, "f1-score": 1.0, "support": 7.0}
}

If you’re parsing this, iterate by key rather than assuming every value is a number: accuracy is a float and every other entry is a dict. There’s no confusion_matrix and no classification_report.

Primary metric per task

Read one metric for the decision and treat the rest as diagnostics.

Task	Primary metric
Classification	accuracy
Question answering, all variants	llm-as-a-judge
Tool calling, all variants	llm-as-a-judge

A metric a run didn’t compute comes back as null rather than being absent, so filter those out before averaging.

Reading a score

A score on its own says very little. Read it against the reference points you have, because 0.72 is a good result against a teacher at 0.75 and a poor one against a teacher at 1.00.

Depending on the stage, you have up to four:

Reference point	Where it comes from	What it tells you
Teacher	Teacher evaluation	The ceiling. The student learns from the teacher, so it can’t reliably exceed it.
Base student	base_model_performance on the SLM	The floor. How much of the task the student can already do untrained.
Tuned student	tuned_model_performance on the SLM	What you got.
Production model	base_model_performance on a trace-derived seed dataset	The model you’re replacing, so the bar that matters in production.

The useful question after training is how far the tuned student moved from the base toward the teacher, rather than what its raw score was.

Scores are samples, not constants

Evaluation runs the judge at non-zero temperature, so the same model on the same test set scores differently run to run, a measured ±0.03 on a 50-row set. One untrained model scored 0.64, 0.60 and 0.58 across three runs of the same evaluation. base.random_seed doesn’t pin this.

Any difference smaller than that band is noise. Quote the run a number came from, and never resolve a decision on a difference inside the band.

Larger test sets narrow the band. If you’re trying to distinguish two close configurations, a bigger test set is a more reliable investment than another training run.

The predictions file

Every metrics response has a matching download that writes the per-example detail. This is where failure patterns live, and reading the aggregate and guessing doesn’t work.

distil teacher-evaluation download-predictions <teacher-evaluation-id>
distil slm download-predictions <slm-id>

Three things about the file:

• It’s JSONL, one test example per line, carrying prompt, completion, prediction and that example’s own score under each metric name.
• prompt is the full prompt as a JSON-encoded message list, not the user text alone, and completion and prediction are JSON-encoded assistant messages. Parse them rather than comparing them as raw strings.
• Group the rows the model got wrong by whatever your task’s failure modes are: a class, a format, a rule.

Sanity-check the judge before you trust a low score. Sample some predictions the judge marked bad. If they look correct to you, the score is lying, and llm_as_a_judge_instructions needs fixing before you change anything else. See Job description.