Why Every Path to Training Data Ends in Synthetic Data

Convergence means that none of the three data routes trains a model directly. Each one produces seed data. A teacher model then generates and validates a much larger synthetic dataset from those seeds, and the student trains on that. The route in changes the seeds, nothing after them.

What does convergence actually mean here?

It means the pipeline downstream of your data is identical regardless of how the data arrived. Seeds go in; synthetic examples come out; the student trains on the synthetic set.

Path	What becomes seed data	What becomes unstructured context
Live traffic collected for you	Filtered, relabelled conversations	Traffic beyond the seed budget
Traces you upload	Traces that pass relevance and coherence filtering	Traces left over after the train/test split
Minimal dataset you write	Your train.jsonl and test.jsonl rows	Optional unstructured.jsonl

Everything to the right of that table is shared: synthetic generation, validation, teacher evaluation, then model training.

How does the generation step work?

A teacher LLM is prompted with your task description, a rotating sample of seed examples, and optionally a randomly chosen unstructured document to keep it on-domain. It writes new input/output pairs. A validator then drops examples that are malformed, too long or short, off-schema, or too similar to something already in the set.

The loop repeats until coverage is good. The platform’s default generation_target is 10,000 examples, and its default deduplication threshold drops anything with similarity above 0.95 to the seed data — both documented in the config reference. This is the same bootstrapping idea Self-Instruct introduced, wrapped in task-specific validators.

The knowledge transfer from the large teacher to the small student is knowledge distillation in the sense of Hinton et al. — the student learns to reproduce the teacher’s behaviour on your domain rather than the teacher’s weights.

Why not just train on the seed data directly?

Because seeds are either too few or too dirty, and usually both.

Too few is the obvious case: 20 hand-written examples will not fine-tune a model to production quality on their own. Our own PII redaction run makes the point — a student trained only on the seed split scored 0.73, while the same student trained on seed plus generated data scored 0.87, above the teacher’s 0.85. Full numbers in small expert agents from 10 examples.

Too dirty is the less obvious case, and it is the one that catches teams with lots of traces. In our traces benchmark, training a Qwen3-1.7B directly on noisy traces dropped 14 to 28 percentage points below the curated ceiling. The synthetic pipeline, fed the same corrupted traces, stayed within 2 points of it. Under schema drift the gap was 25.9pp.

The mechanism is worth stating precisely. Traces carry distributional signal — what topics come up, how conversations flow, when tools get called. Your task description and schema carry normative signal — what correct looks like. Training directly on traces uses only the first and inherits its errors. Generation uses the traces for the first and the task description for the second, so the result is both domain-realistic and technically correct.

When does the convergence matter to you?

Mostly when you are choosing what to spend effort on. Two practical consequences follow.

First, seed quality dominates seed quantity. Because generation amplifies whatever is in the seeds, a thousand near-identical traces contribute roughly as much as ten of them. Diversity of seeds is the lever, not volume.

Second, the job description is not a formality. It is the normative half of the signal, and it is the only place where “correct” is defined independently of your existing system. A vague description produces vague synthetic data whichever path you came in on. See writing a job description for synthetic data.

Where can the convergence break down?

In one place: if the teacher cannot solve your task, no amount of generation will save it. The student learns from teacher-produced data, so the teacher’s ceiling is roughly the student’s starting expectation — although a tuned student can and often does exceed it, because validators inject signal the teacher alone does not have. Can a small model beat its teacher covers why that happens.

This is exactly what teacher evaluation is for. It runs before training and tells you whether the teacher is up to the job while it is still cheap to change the task definition, the seeds, or the teacher itself. How to choose a teacher model and which teacher model should you pick go into the selection.

What related terms should you know?

Term	Meaning in this pipeline
Seed data	The small, high-quality set that anchors generation
Unstructured context	Domain text sampled to keep generation on-domain; never used as labels
Relabelling	A committee of teacher models rewriting trace assistant turns into cleaner references
Validation	Schema, length, and near-duplicate filters applied to generated examples
Teacher evaluation	A feasibility check on the teacher, run before training

If you want the practical version rather than the conceptual one, fine-tune with synthetic data is the shorter walkthrough, and three ways to get training data covers the routes in.