Why METEOR and BLEU Are Legacy Metrics
METEOR is a legacy metric on distil labs: removed from the docs and the CLI, not returned for new evaluations, and shown on older ones only because the value is already stored. BLEU was never on the scorecard at all. Both belong to a class of word-overlap metrics built for machine translation.
What does the platform score today?
Three metrics for text generation, and LLM-as-a-Judge is the recommended one. The metrics guide is explicit that it “best captures semantic correctness regardless of exact phrasing.”
| Metric | Status | What it asks |
|---|---|---|
| LLM-as-a-Judge | Current, recommended | Would a large model grading this call it a good answer? |
| Exact-Match (Binary) | Current | Is the answer character-identical to the reference? |
| ROUGE-L | Current | How long is the longest common subsequence with the reference? |
| METEOR | Legacy | Retired, not returned for new evaluations |
| BLEU | Never used | n/a |
The change was mechanical, and it’s worth knowing exactly what it touched, because “legacy” here doesn’t mean “deprecated with a warning.” METEOR was dropped from the metrics how-to, from the metric row in the question-answering and RAG tutorials, and from the distil labs CLI skill, which moved to version 4.4.0 in the process. The evaluation view models in the CLI and client app now render the “Meteor” row only when the score is present, and the type guard that identifies a QA evaluation result was re-keyed on the binary field alone so that results arriving without a METEOR score are still recognised. The scorecard guidance was updated from “if all four numbers are low” to “if all three numbers are low.”
Why do word-overlap metrics mislead on generation tasks?
Because they score string similarity and you care about meaning, and those two come apart in both directions. A correct answer phrased differently scores low; a fluent answer that’s factually wrong but reuses reference vocabulary scores high.
BLEU was introduced in 2002 to compare machine-translation systems by modified n-gram precision against reference translations. METEOR followed in 2005 and improved on it by adding stemming and synonym matching and by balancing precision against recall, specifically to correlate better with human judgement at the segment level. Both were designed for a setting with one input, a small set of reference translations, and a research goal of ranking systems.
That setting isn’t the one a fine-tuned small language model operates in. Two lines of criticism are directly relevant. Callison-Burch and colleagues showed that BLEU improvements don’t necessarily correspond to improvements in translation quality, and that systems can be ranked incorrectly by it. Later work on natural language generation, including Novikova and colleagues, found that word-overlap metrics correlate only weakly with human judgements of generated output.
The concrete consequence on a distil labs task: a model answering “the total is $540” against a reference of “$540” is correct and scores badly on overlap. A model answering “$450” against the same reference shares most of its characters and scores far better than it deserves. This is why the metrics guide tells you to read low Exact-Match with high LLM-as-a-Judge as a paraphrase signal rather than a quality problem.
Was BLEU ever on the distil labs scorecard?
No. The platform has never returned a BLEU score, and no task type has ever listed it. If you’ve seen BLEU discussed alongside distil labs work, it was as background on evaluation practice, not as a metric you can request.
This is worth stating plainly because BLEU and METEOR are usually named together, and the retirement of one is sometimes read as the retirement of both. The accurate statement is narrower: METEOR was a supported metric and is now legacy; BLEU was never supported. ROUGE-L, which shares the same word-overlap ancestry, remains current. It’s genuinely useful as a secondary signal for summarisation-shaped tasks, and the docs are candid that it “favours longer answers that reuse reference phrases.”
Where will you still see METEOR?
Three places, and none of them mean the metric is coming back.
- Older evaluation results. Evaluations run before the change carry the score, and the CLI and client app still render the row for them. Nothing was deleted retroactively.
- Older blog posts. distil labs demo write-ups published before the change quote METEOR alongside ROUGE and binary scores. Those posts were deliberately left alone. Rewriting a published result to drop a metric would falsify the record.
- Other tooling. METEOR is still widely implemented elsewhere. Its absence here is a product decision about what’s worth reporting, not a claim that the metric is broken in every context.
If you’re comparing a new evaluation against an old one, the old scorecard has four rows and the new one has three, and the row that vanished isn’t the row you should have been reading anyway. Compare on LLM-as-a-Judge, which is present in both.
What to use instead
LLM-as-a-Judge as the primary number, Exact-Match as a strictness check, ROUGE-L as a hint about phrasing. For tool calling the question doesn’t arise. tool_call_equivalence compares structured calls, and text-overlap metrics were never appropriate there.
The honest caveat is that swapping a word-overlap metric for a judge trades one problem for another: you’re now trusting one model’s opinion of another’s output, with its own variance and its own biases. What is LLM-as-judge evaluation covers what that costs and how much the scores move between runs. For choosing among the current set, see accuracy, F1 or exact match; for reading a scorecard as a ship-or-not decision, is your fine-tuned model good enough.