Building a Test Set That Catches Real Failures
A test set that catches real failures is reserved from real traffic before any pipeline touches it, contains the inputs you already know break things, and has a score attached for whatever you’re replacing. If you let the platform build one for you from your traces, you get a clean, filtered, easier-than-production test set.
What you need
Before you start, have these to hand:
| Input | Why |
|---|---|
| A sample of real production traffic | Synthetic test data measures internal consistency, not performance |
| A written task description | The same job_description.json your training will use |
| A list of known failure cases | Bug reports, escalations, the inputs your current system gets wrong |
| The system you’re replacing | You need a baseline number on the same data |
The data preparation overview lists test.jsonl as a required file for a minimal dataset, alongside train.jsonl and config.yaml. For trace-based training it’s optional, which is exactly the decision this recipe is about.
Step 1: Reserve real traffic before the pipeline touches it
Pull your held-out examples out of the raw log, before deduplication, filtering or relabelling. Then never put them back.
This matters because of what the trace pipeline does. Traces are split into train and test seed sets, scored for relevance and coherence against your job description, and low-scoring traces are dropped. min_relevance_score defaults to 4 and min_coherence_score to 3 in the config reference. A test set built downstream of that filter contains only traces the filter liked.
Our trace-training walkthrough states the consequence plainly: an original-model score of 0.948 on such a test set reads as “how consistent the clean examples in the original data are,” not as a general quality score. That’s a useful number, but it isn’t the number you want to ship against.
Step 2: Write it in the format the task expects
Test examples use the same shape as training examples, a messages array following the OpenAI chat completion format, with the assistant turn as the reference answer.
{"messages": [{"role": "user", "content": "Invoice #1234 from Acme Corp dated 2024-01-15. Total: $540. What is the total amount?"}, {"role": "assistant", "content": "$540"}]}Open-book QA adds a context field; tool calling puts tool_calls on the assistant turn. The per-task pages under data preparation give the exact shape for each of the six task types. Get this right before you scale up. A malformed test file fails late and wastes a processing run.
Step 3: Add the failures you already know about
Take your list of known-bad inputs and put every one of them in the test set, with the correct answer written by hand. This is the step that separates a test set from a sample.
Aim for four categories:
- Rare-but-critical classes. If the category your product exists to catch appears in 3% of traffic, a random sample will barely contain it. Over-represent it deliberately and note that you did.
- Near-miss pairs. Two inputs that look similar and have different correct answers. These catch a model that has learned surface features.
- Inputs your current system gets wrong. If the new model fixes them, that’s your headline. If it doesn’t, you’ve found out cheaply.
- Messy real inputs. Typos, truncation, mixed languages, the half-finished requests users actually send.
The corruption scenarios in our traces benchmark are a good template for what “messy” means in practice: noisy labels, schema drift across API versions, and traces from an adjacent service mixed into the log. Each was constructed from the Schema-Guided Dialogue corpus and each broke direct training by 12 to 26 points.
Step 4: Score the system you’re replacing on it
Attach a baseline before you train anything. Without one, a good-looking score at the end means nothing.
# Directory mode picks up test.jsonl from ./traces automatically
distil traces upload --data ./traces
# Or name each file explicitly
distil traces upload \
--traces ./traces.jsonl \
--job-description ./job_description.json \
--config ./config.yaml \
--test ./test.jsonl
distil seed-dataset create-from-traces <traces-id>
# Output: Processing started. Seed dataset ID: <seed-dataset-id>
distil seed-dataset status <seed-dataset-id>
distil seed-dataset metrics <seed-dataset-id>Supplying the test file is what keeps your curated set intact: when a test set is provided, num_traces_as_testing_base is ignored and the platform doesn’t generate one. Note that --data and the individual file flags are mutually exclusive. In directory mode a test.jsonl sitting in the directory is picked up on its own. distil seed-dataset metrics then reports how the model that produced the traces scored on your file. Download the per-example results with distil seed-dataset download-traces-predictions <seed-dataset-id> and read the failures. That’s the list you’re trying to shorten.
For the full base-versus-tuned procedure, see base model vs fine-tuned model comparison.
Step 5: Gate on teacher evaluation
Run the teacher against the same file before committing to a training run. If you came through step 4 you already have a seed dataset id, so skip the first command.
distil seed-dataset create --data ./data
# Output: Upload successful. Seed dataset ID: <seed-dataset-id>
distil teacher-evaluation create-from-seed-dataset <seed-dataset-id>
# Output: Teacher evaluation started. Teacher Evaluation ID: <teacher-evaluation-id>
distil teacher-evaluation status <teacher-evaluation-id>
distil teacher-evaluation download-predictions <teacher-evaluation-id>A teacher that scores badly here is usually telling you something about your test set rather than about the teacher. Ambiguous references, inconsistent labelling between examples, questions that can’t be answered from the input: all of them show up as teacher failures. Read those predictions before you conclude the task is hard; what is teacher evaluation covers how to act on the result.
Verifying it worked
Four checks, all cheap:
| Check | How | What a failure looks like |
|---|---|---|
| Disjoint from training | Compare the raw examples by hash | Any overlap invalidates the score |
| Large enough for your claim | Count it against how big should your test set be | One example moves the score more than the gap you’re chasing |
| Covers the classes you care about | Per-class counts from the predictions file | A critical class with two examples |
| Contains failures the baseline gets wrong | Baseline score well below 1.0 | A baseline at 100% means the test set is too easy |
The last row matters most. If the system you’re replacing already passes everything, the test set can’t demonstrate an improvement and can’t detect a regression. Rebuild it with harder inputs before you rely on the number.
Related: what is a held-out test set, why your model passes eval but fails in production, and what is overfitting in fine-tuning.