Synthetic data generation

Your seed dataset has a few dozen examples. Training needs thousands. This stage is where the teacher writes them: it generates examples from your seed data, job description and mutators, then validates and de-duplicates them into a training dataset.

The result is a merged dataset. The root train.jsonl is your seed examples plus the surviving synthetic ones, so the synthetic count is the total minus your seed count.

Running it

distil training-dataset create-from-seed-dataset --output json <seed-dataset-id> | jq -r .id
# <training-dataset-id>

distil training-dataset status --output json <training-dataset-id> | jq -r .status

A full run can take 90 minutes and spends one training_datasets_from_seed_datasets_post credit. New accounts start with five.

The three levers on what gets generated

Every generation call is shaped by three inputs, and knowing which one to reach for is most of iterating well.

Lever	Where it lives	How it applies
`task_description`	`job_description.json`	Constant. Every generation call, and every evaluation and judge prompt
`synthetic_data_generation_instructions`	`job_description.json`	Constant. Every generation call, generation only
`mutation_topics` / `basic_mutators_to_use`	`config.yaml`, `synthgen`	Sampled. One value per active mutator, per call

task_description says how to solve the task. It defines what a correct answer is, so it also feeds evaluation and the judge, and changing it changes what “correct” means everywhere. Keep it matched to your production prompt and constant across iterations.
synthetic_data_generation_instructions says how to generate the data: what the inputs should look like, their formats, domains, register and noise. It touches generation only, which makes it the safe place to steer inputs without redefining the task.
Mutators shape the distribution. Each call samples one value per active mutator, so the composition of the list sets the proportions.

The two constants shift every example the same way, while the mutators decide how examples are distributed. So when the whole dataset is wrong in the same way, whether that’s a format, a misread rule or the wrong register throughout, fix a constant. When the mix is wrong, with a slice missing or over-represented against what production sees, change the mutator values and their proportions.

Mutators

Mutators append directives to the generation prompt, and each call samples one value per active mutator.

Topic mutators, the main lever

synthgen.mutation_topics steers generation toward named subjects. Use it when your seed data misses scenarios production will see, or the model underperforms on a specific slice.

# Single pool: each call samples one topic
synthgen:
  basic_mutators_to_use: []   # turn the built-ins off when adding custom mutators
  mutation_topics: ["billing disputes", "account cancellation", "technical support"]

Nested lists create independent dimensions, with one sample from each list per call:

synthgen:
  basic_mutators_to_use: []
  mutation_topics:
    - ["billing disputes", "account cancellation", "technical support"]
    - ["enterprise customers", "small business", "individual users"]

That covers a grid of nine scenario combinations without enumerating them. A flat list is treated as one pool.

Use 1-2 lists of 3-10 topics each. More dilutes the signal per topic, fewer limits diversity.

Proportions follow composition directly, so three values asking for English and one asking for French gives roughly a 75/25 split.

A single-item list is a useful special case. It applies the same directive to every call, turning the mutator into a constant extra generation instruction without touching the job description.

When you introduce custom mutators, set basic_mutators_to_use: [] explicitly. The default ["complexity"] stays active otherwise, and the stacked directives start tripping over one another.

Built-in mutators

synthgen.basic_mutators_to_use defaults to ["complexity"]. Use at most one, since combined built-ins give conflicting instructions. [] disables them.

Mutator	Samples uniformly from
`complexity`	trivial, simple, medium, complex, highly complex
`length`	short and concise (1-2 sentences), medium length (3-5 sentences), detailed (multiple paragraphs)
`specificity`	generic and vague, somewhat specific, specific, very specific

Active mutators render one line each under a shared prelude:

Important! Make sure to follow those guidelines:
Generated examples should be complex: Multiple factors, nuance, or ambiguity involved.
Generated examples should focus on: billing disputes.

Sampling is seeded from base.random_seed, so a config reproduces its own mutation sequence.

Configuring generation

The full table is in Config file. The ones to set deliberately:

validation_max_total_length. Caps the characters in question + answer (+ context) per example. Set it to the maximum combined length you expect in real data, since the default of 30,000 rejects longer uploaded examples and silently filters longer generated ones.
generation_in_single_call, num_positive_exemplars_per_generation and num_unlabelled_exemplars_per_generation. Bring these down when examples are long. Each multiplies the prompt and output size per teacher call.
output_is_json: true. Whenever answers have to be valid JSON. QA tasks only.
base.llm_num_parallel_requests. Above the default of 4 it can help, but don’t expect linear gains, since per-call latency and between-batch validation usually dominate.

Try a small run first

Problems visible in 64 examples will be everywhere in 10,000, and a regenerated dataset costs one credit while training on a bad one costs a training credit plus hours. Generate a small batch first:

distil seed-dataset download-metadata -d ./smoke <seed-dataset-id>
# in ./smoke/config.yaml, under synthgen:
#   generation_target: 64
#   generation_iteration_size: 16

distil training-dataset create-from-seed-dataset --output json \
  --config ./smoke/config.yaml <seed-dataset-id> | jq -r .id

Checking what came back

Reading a sample is free, and downloading the whole dataset costs a credit.

# Free: up to 128 train rows, drawn deterministically from the first 384
distil training-dataset sample --output json <training-dataset-id> > sample.json

# Size in bytes, not rows
distil training-dataset metrics --output json <training-dataset-id> | jq .train_data_size_bytes

Compare the count against your target first. Falling well short means validation filtered heavily, usually length caps or format problems.

Then check three things:

Form. Does each example parse, carry the required output format, and respect the stated constraints? A malformed row is a defect whatever the rate. Read a handful of answers to confirm the teacher understood the task, but don’t gate on a label error rate, since the student absorbs a low rate of teacher noise.
Distribution. Compare generated against seed data along the dimensions that matter for your task: length, topic coverage, style and register, class balance.
Targeted slices. If your mutators or job description asked for a particular slice, count it in the output. Mutation topics are suggestions and can silently yield nothing.

Axis 3 needs a big enough sample to be readable. A run makes generation_target / generation_in_single_call mutator draws, so a trial target of 64 with generation_in_single_call: 4 gives just 16 draws. A grid of more than 16 cells leaves cells empty by pigeonhole whatever the config says. Size a trial run to at least twice your grid before treating empty cells as a signal, or check this on the full run instead.

generation_target is a floor rounded up to the next generation_iteration_size batch, so landing above it is normal.

If the dataset isn’t good

Don’t carry it into training. The training metrics won’t tell you the data was bad. It shows up as a plateau you spend a full training run discovering.

Go back to the three levers: mutators for coverage gaps, the generation instructions for how inputs look, the relevant config field for correctness or format problems. Then generate again.

Model training.