Why Your Model Passes Eval But Fails in Production
Almost always because the test set was easier than production traffic, or because a per-example score was read as a per-session guarantee. The model is usually fine. The measurement was answering a different question from the one you cared about.
What does the symptom look like?
A held-out score in the high eighties or above, and user-visible failures within hours of deployment. Three shapes recur:
- The model handles the demo inputs perfectly and fails on the first genuinely messy request.
- Aggregate accuracy is good but complaints cluster in one category or one customer segment.
- Single responses look correct in isolation while whole conversations go wrong.
Each shape points at a different cause, so identify yours before changing anything. If the model is worse than where it started rather than merely narrow, that’s a different diagnosis. See why did my fine-tuned model get worse.
Which cause is it most likely to be?
Ranked by how often it turns out to be the answer.
| Rank | Cause | Tell |
|---|---|---|
| 1 | Test set drawn from the same distribution as the training data | Both were built from the same curated seed set |
| 2 | Test set filtered by the pipeline that also built the training set | You didn’t supply your own test.jsonl |
| 3 | Per-example score read as per-conversation reliability | The product is multi-turn; the metric is per turn |
| 4 | Metric doesn’t measure what users experience | Exact-Match on generation, or accuracy on imbalanced classes |
| 5 | Genuine distribution shift since the traces were collected | Nothing changed in evaluation; traffic changed |
The first two are variations on one mistake, and it’s the mistake our evaluation hub leads with: if your training data is synthetic and your test set is also synthetic, your evaluation is measuring internal consistency rather than performance.
Did your test set survive its own filtering?
If the platform built it from your traces, yes, and that’s the problem. Trace processing deduplicates and splits traces, then scores each seed trace for relevance and coherence and drops the low scorers. min_relevance_score defaults to 4 and min_coherence_score to 3 in the config reference.
The examples that survive are, by construction, the ones your task description already describes well. Our trace-training walkthrough flags this explicitly: an original-model score of 0.948 on such a test set means “how consistent the clean examples in the original data are,” not a general quality score, because “the obviously bad ones were removed.”
Fix. Supply your own held-out file. Put test.jsonl in the directory you pass to distil traces upload --data ./traces, or name it explicitly with --test alongside --traces, --job-description and --config. When a test set is provided, num_traces_as_testing_base is ignored and no test set is generated. Build that file from raw traffic sampled before any filtering. Building a test set that catches real failures is the procedure.
Are you scoring turns when production runs conversations?
Per-turn accuracy compounds multiplicatively across a session, and the arithmetic is easy to miss.
Our traces benchmark works the example: 95% accuracy on a single turn is 0.95^20 over a twenty-turn interaction, roughly 35% of conversations fully correct. The SHELLper write-up measured the same effect on a real model, where a base Qwen3 0.6B at 84.16% single-call accuracy translated to 42.22% average accuracy over five-turn conversations.
| Per-turn accuracy | 5 turns | 20 turns |
|---|---|---|
| 84% | ~42% | ~3% |
| 95% | ~77% | ~36% |
| 99% | ~95% | ~82% |
Those are compounded rates, not measurements, but they’re the numbers your users experience. The traces benchmark puts it directly: a classification model wrong 5% of the time is annoying; a multi-turn agent making 5% more errors per turn is a disaster.
Fix. Report conversation-level success alongside turn-level. evaluation.expand_tool_calling_turns defaults to true and expands each multi-turn test line into one evaluation line per tool call, which is right for diagnosis and wrong for a headline reliability claim. Score both.
Is the metric measuring what your users care about?
Possibly not, and the mismatch is systematic rather than random. A model scoring well on Exact-Match can be producing correct-but-unhelpfully-terse answers; a model scoring well on a judge can be producing fluent answers with wrong values in them.
Tool-timing errors are the worst case here, because they’re invisible to string comparison. The traces benchmark describes them as calling an API when the agent should ask a clarifying question, or chatting when it should act, errors that “compound across turns and are among the hardest to debug because the model’s output looks syntactically correct.”
Fix. Choose the metric before training and keep a secondary metric for diagnosis; accuracy, F1 or exact match has the mapping. For classification, replace aggregate accuracy with a per-class breakdown. A 90% aggregate can be 100% on four classes and 20% on the one that matters.
How do you prevent it next time?
Five habits, in the order they pay off.
- Reserve held-out data from raw traffic before the pipeline runs. Everything else is downstream of this.
- Keep a permanent regression set of known failures. Every production incident becomes a test case, forever.
- Baseline the incumbent on the same file. A test set the current system already passes at 100% can’t detect anything. See base model vs fine-tuned model comparison.
- Size the test set to the gap you want to detect. How big should your test set be covers the arithmetic; judge noise alone runs to roughly ±0.03.
- Re-sample traffic periodically. Distribution shift is real and a test set frozen a year ago describes a product you no longer have.
Our traces benchmark built deliberately corrupted scenarios (noisy labels, schema drift, an 80% mix of traces from the wrong domain) on top of the clean Schema-Guided Dialogue corpus rather than testing on the corpus alone. Production data is never clean, and an evaluation built only from the clean parts of it will always overstate what you’re about to ship.