How Big Should Your Test Set Be?
Between 50 and 250 examples for most tasks. Published distil labs evaluations span 10 to 374 held-out examples, and when the platform builds a test set from traces it seeds it from 200 of them by default. Below about 50 you can’t separate a real improvement from run-to-run noise.
What sizes does published distil labs work actually use?
Here’s every held-out test set size we’ve published, with the metric it was scored on.
| Study | Task | Test set size | Metric |
|---|---|---|---|
| Commit bot | Commit message generation | 10 | LLM-as-a-judge |
| Expense categorisation | Classification | 25 | LLM-as-a-judge |
| Traces vs synthetic | Multi-turn tool calling | 34 conversations → ~360 turn pairs | LLM-as-a-judge |
| Text2SQL | Query generation | 50 | LLM-as-a-judge, Exact Match, ROUGE |
| Smart Home | Tool calling | 50 | tool_call_equivalence |
| On-device RAG | Open-book QA | ~50 pairs | LLM-as-a-judge |
| Git Assistant | Tool calling | 116 | tool_call_equivalence |
| PII Redaction Healthcare | Extraction | 133 | LLM-as-a-judge |
| Banking77, E-commerce, TREC | Classification | 200 each | Accuracy |
| Docstring generation | Generation | 250 | LLM-as-a-judge |
| Docstring generation, separate run | Generation | 253 | LLM-as-a-judge |
| Edtech scam detection | Classification | 374 documents | Precision, recall, false-positive rate |
The clustering isn’t accidental. Fifty is the smallest size at which a single wrong answer moves the score by only two points. Two hundred is where classification benchmarks sit, and where the platform’s own default lands.
What does the platform default to?
Two hundred seed examples, with a hard floor at twenty. Three settings in the config file govern this:
| Setting | Default | Effect |
|---|---|---|
trace_processing.num_traces_as_testing_base |
200 | Traces used as the seed for generating test examples. Ignored entirely if you supply your own test.jsonl |
trace_processing.min_generated_examples |
20 | Trace processing errors out below this, rather than training on too little |
tuning.train_eval_split |
0.2 | Fraction of the training data reserved for in-training evaluation. Not your held-out test set |
The minimal dataset layout asks for 20+ labelled examples in train.jsonl and a separate test.jsonl, and our small expert agents write-up puts the working range at 20–100 labelled examples for both. Note that train_eval_split and the held-out test set are different things doing different jobs. Mistaking one for the other is a common source of confusion, covered in what is a held-out test set.
What resolution does each size buy you?
The arithmetic is unforgiving. One example on a test set of n is worth 100/n percentage points:
| Test set size | One example is worth | Practical reading |
|---|---|---|
| 10 | 10.0 pp | Smoke test. Detects catastrophic failure only |
| 25 | 4.0 pp | Detects large gaps; can’t rank close models |
| 50 | 2.0 pp | Minimum for a credible before-and-after |
| 100 | 1.0 pp | Comfortable for a single-model verdict |
| 200 | 0.5 pp | Standard for classification benchmarks |
| 374 | 0.27 pp | Enough to separate precision changes from noise |
Resolution isn’t the same as confidence. The width of a binomial confidence interval around an accuracy figure narrows with the square root of the sample size, so quadrupling the test set only halves the interval. Going from 50 to 200 examples is a real improvement; going from 200 to 250 isn’t. The same square-root relationship governs statistical power, the ability to detect a difference that genuinely exists.
Judge noise stacks on top. LLM-as-a-judge scores in our benchmarks carry standard deviations from 0.006 to 0.020 across repeated runs, and reference-free judge ratings were reported at roughly ±0.03 in the deferral cascade study. A test set fine enough to resolve one point is useless if the metric itself moves by three.
What pushes the number up?
Four things, and they compound.
Class count. A 77-class problem like Banking77 needs enough examples per class to say anything per class. Two hundred examples across 77 classes is about two and a half each, fine for an aggregate and meaningless for a per-class breakdown.
Rare-but-critical categories. If the thing you care about appears in 3% of traffic, a 50-example test set contains roughly one of them. The edtech evaluation is explicit about this trade-off: on a 374-document set, the precision and false-positive-rate gains held up statistically while the detection-rate difference was read as parity rather than a win.
Multi-turn structure. Conversations aren’t examples. The traces benchmark reserved 34 held-out conversations, which expanded into roughly 360 turn pairs; evaluation.expand_tool_calling_turns defaults to true and performs that expansion. Size the conversations, then check what they expand to.
The size of the difference you’re chasing. Detecting a 20-point gap needs far fewer examples than detecting a 2-point one. If you’re choosing between two base models that finished within a point of each other, no realistic test set will settle it. Does base model accuracy predict fine-tuned performance is the more useful question.
When is a small test set defensible?
When the gap is enormous, when the test set is a gate rather than a measurement, or when you’re honest about what it doesn’t prove.
A ten-example test set caught a real result in the commit bot demo because the gap between base and fine-tuned wasn’t subtle. Small sets are also fine as regression gates: you’re asking “did this break,” not “which of these is better.” What they can’t do is rank two similar models, support a per-class claim, or justify a decision that a two-point difference would reverse. If the test set is small, say the size out loud next to the number, as every study in the table above does. Then read building a test set that catches real failures, because composition matters more than count.