Accuracy, F1 or Exact Match: Which Metric Should You Use?

Use accuracy for balanced classification, F1 when classes are imbalanced or false positives cost differently from false negatives, exact match only when there’s exactly one correct string, and LLM-as-a-judge for anything free-text. The distil labs scorecard reports the last three directly; F1 you compute yourself from downloaded predictions.

Which metric fits which task?

Match the metric to the shape of the output, not to the domain.

Metric	Reported by the platform	Fits	Fails on
Accuracy	Yes, as the classification benchmark metric	Balanced multi-class classification	Imbalanced classes: 95% on a 95/5 split means nothing
F1	No, derive it from predictions	Imbalanced classes; asymmetric error costs	Multi-class without deciding on micro/macro/weighted averaging
Exact-Match (Binary)	Yes	Single-correct-string answers: IDs, dates, amounts, class labels	Any output where a synonym is also right
ROUGE-L	Yes	Comparing generated text against a reference for overlap	Rewards long answers that reuse reference phrasing
LLM-as-a-Judge	Yes, and recommended for generation	Free text, multiple valid answers	Costs a model call; carries run-to-run variance
tool_call_equivalence	Yes, and recommended for tool calling	Function calls with default-valued parameters	Not a partial-credit metric
staged_tool_call	Yes	Debugging where a tool call broke	Reporting a headline number

The metrics guide is the authoritative list of what the platform returns for each task type.

What does each metric actually measure?

Accuracy is the fraction of examples the model got right. Nothing else. It’s the right default when every class matters equally and appears at roughly equal rates, which is why the classification datasets in our platform benchmark, TREC and Banking77 and the e-commerce set, are all scored on accuracy.

F1 is the harmonic mean of precision and recall, so it collapses when either is low. scikit-learn’s implementation exposes the averaging choice that matters most in practice: macro-F1 weights every class equally and will expose a class the model has quietly given up on, while micro-F1 won’t.

Exact-Match returns 1 for a character-identical answer and 0 for everything else. The docs describe it as “great for facts that have one correct phrasing, but harsh on synonyms,” and that’s the whole trade-off.

ROUGE-L measures the longest common subsequence between prediction and reference. It’s a word-overlap score, which means it has no notion of whether the overlapping words carry the meaning.

LLM-as-a-Judge asks a large model whether the answer is good given the reference and the task description. It’s the only one of these that tracks semantic correctness, and it’s the recommended metric for generation tasks on the platform.

When does accuracy mislead you?

Whenever the class distribution is skewed, which is most real classification problems. A model that predicts the majority class for every input scores exactly the majority-class frequency, and that number can look like success.

The related failure is aggregate accuracy hiding a class-shaped hole. A model at 90% overall can be at 100% on four classes and 20% on the fifth, and if that fifth class is the one your product exists to catch, the headline number is actively wrong. Macro-F1 catches this; accuracy doesn’t.

In our edtech case study, a fine-tuned sub-1B model on a 374-document held-out set was reported as detecting scams at 82% against a production model’s 71–83% range (read as parity), while precision rose to 91% from 74–85% and the false positive rate fell to 3.5%. Precision and false-positive rate were the deltas that held up; a single accuracy figure would have shown none of it.

Why is exact match too harsh for generation?

Because generation tasks admit many correct strings and exact match recognises one. “$540”, “540 dollars” and “The total is $540” are the same answer to a human and three different answers to exact match.

This produces a specific, readable signature on the scorecard, and the metrics guide names it: low Exact-Match with high LLM-as-a-Judge means the answers are probably right but paraphrased. The recommended fix is to add those paraphrases to your reference set rather than to change the metric.

The trap runs the other way too. Exact match is not too harsh when the output is a class label, a JSON field, or a normalised identifier. There a synonym is genuinely wrong, and a judge that accepts near-misses will hide real errors. Exact match on structured output is a feature.

Tool calling is where this is handled explicitly. binary_tool_call compares prediction and reference as dictionaries for strict equivalence; tool_call_equivalence, the recommended metric, treats parameters that were never set as carrying their default values, because that’s what actually happens at runtime. And staged_tool_call awards 0.25 at each of four stages (valid JSON, correct function name, correct parameter keys, exact match), so a score of 0.5 tells you the model picked the right function and got the arguments wrong.

Which should you pick?

One primary metric, chosen before you run anything, plus one or two secondary metrics you read for diagnosis rather than for the headline.

Task	Primary	Read alongside
Balanced classification	Accuracy	Per-class breakdown from the predictions file
Imbalanced classification	Macro-F1	Accuracy, precision, recall
Extraction of a specific field	Exact-Match	LLM-as-a-Judge, to detect paraphrase
Question answering, summarisation, free text	LLM-as-a-Judge	Exact-Match and ROUGE-L as sanity checks
Single-turn tool calling	tool_call_equivalence	staged_tool_call for diagnosis
Multi-turn tool calling	tool_call_equivalence	Per-conversation success, not per-turn

Two rules that outrank the table. First, fix the metric before training. Comparing a run scored one way against a run scored another way isn’t a comparison, and the research on judge-based evaluation makes clear how much the judge configuration moves the scale. Second, whatever the primary number says, download the per-example predictions and read twenty of the failures.

Related reading: what is LLM-as-judge evaluation, why METEOR and BLEU are legacy metrics, and is your fine-tuned model good enough.