What Is LLM-as-a-Judge Evaluation?

LLM-as-a-judge is evaluation where a large model grades another model’s output. On distil labs it’s the recommended metric for question answering, classification and RAG, because it scores semantic correctness when many phrasings are valid. The cost is that you’re trusting one model’s opinion of another, with its own variance and biases.

How does an LLM judge produce a score?

The judge is shown the prediction, the reference answer and the task description, and asked whether the prediction is a good answer. The metrics guide phrases the underlying question as: “If we let a large language model act as a human grader, does it say this answer is good?”

Two variants appear in distil labs work and they behave differently:

Variant	What the judge sees	Used where
Reference-based, binary	Prediction, reference, task description; returns 0 or 1	Information extraction, open-book QA and closed-book QA in the platform benchmark
Reference-free, rubric	Prediction and a rubric, no reference answer	The deferral cascade benchmark, scored under both a strict and a relaxed rubric

Reference-based scoring is stricter and more reproducible. Reference-free scoring is what you fall back on when there’s no single right answer, and it’s far more sensitive to how the rubric is written. In that cascade benchmark the same system scored 0.79 under the strict rubric and 0.88 under the relaxed one, which is nine points of difference from the same outputs and the same judge.

Which model does the judging?

A configurable one, and it’s a separate setting from the teacher. In the config file, evaluation.llm_as_a_judge_model_name controls the judge used in teacher evaluation and defaults to openai.gpt-oss-120b. If you enable the optional RLVR stage, tuning.rlvr_llm_as_a_judge_model_name picks the model that supplies reward signals during reinforcement learning, with the same default.

Published distil labs work has used several judges, and the choice is recorded each time because it’s part of the measurement:

Study	Judge
Traces versus synthetic benchmark	GPT-OSS-120B
The 10x inference tax	Claude Sonnet 4.6, default effort
Deferral cascade	GLM-5, independent of the models under test

Two practical rules follow. Keep the judge fixed across everything you intend to compare, since switching judges rescales the numbers and silently invalidates a before-and-after. And prefer a judge that isn’t one of the systems being scored, which is why that last study describes its GLM-5 judge as independent: the frontier model under test was also GLM-5, and using it to grade itself would have been indefensible.

How stable are judge scores between runs?

Less stable than exact match, and the variance is large enough to swallow small differences. Published distil labs numbers put it in a usable range.

In the traces benchmark, five teacher models scored on the same held-out set carried reported standard deviations between 0.006 and 0.020. In the deferral cascade work, reference-free judge ratings were described as carrying roughly ±0.03 run-to-run noise, with the explicit note that “differences this small are not meaningful.” A 0.03 gap there failed a paired McNemar test.

Take that as the working floor. A three-point difference in a judge score isn’t a result unless you’ve run it repeatedly and shown otherwise. The 10x inference tax benchmark handles this by running frontier models three times and reporting means with standard deviations, while fine-tuned models (which default to temperature 0) get a single run.

When should you distrust the judge?

Four situations, in rough order of how often they bite.

• The test set was filtered before the judge saw it. In the trace-training walkthrough, an original-model score of 0.948 came with an explicit warning: the test set only contained traces that survived relevance filtering, so the number reads as “how consistent the clean examples are,” not as a general quality score.
• The rubric is doing the work. If moving from strict to relaxed wording moves the score by nine points, the rubric is a bigger lever than the model. Write it once, freeze it, and publish it alongside the number.
• The judge shares a lineage with the model under test. The research literature documents self-preference and position effects in pairwise judging; Wang and colleagues show that simply swapping the order of two candidate responses can flip an LLM evaluator’s verdict.
• The task has one correct string. For class labels, extracted fields and tool calls, a judge that tolerates near-misses hides real errors. Use Exact-Match or tool_call_equivalence instead, as accuracy, F1 or exact match sets out.

What a judge score can’t tell you

It can’t tell you where the model failed, and it can’t tell you how a per-example score translates into a per-session experience.

For the first, download the predictions (distil teacher-evaluation download-predictions <teacher-evaluation-id> for the teacher, distil seed-dataset download-traces-predictions <seed-dataset-id> for the base model) and read the failures. For the second, remember that judge scores are per example. A multi-turn agent at 95% per turn is roughly 0.95^20 over a twenty-turn conversation, which the traces benchmark works out to about 35% of conversations fully correct.

The wider research picture is captured in the survey on LLM-as-a-judge and in the MT-Bench and Chatbot Arena work, which established that strong judges can agree with human preferences at rates comparable to human-human agreement while carrying identifiable biases. Both hold at once, which is why the metric is recommended rather than trusted blindly.

See also: what is teacher evaluation, why METEOR and BLEU are legacy metrics, and which teacher model should you pick.