Synthetic data generation

Your seed dataset has a few dozen examples. Training needs thousands. This stage is where the teacher writes them: it generates examples from your seed data, job description and mutators, then validates and de-duplicates them into a training dataset.

The result is a merged dataset. The root train.jsonl is your seed examples plus the surviving synthetic ones, so the synthetic count is the total minus your seed count.

This stage also runs from an empty seed set. The exemplar blocks drop to zero-shot and the teacher works from your task description, your tool or class definitions, and unstructured.jsonl. What it generates becomes the training set, which is why it has to run before training when you have no training examples. Nothing anchors format or style in that case, so read the output closely.

Running it

distil training-dataset create-from-seed-dataset --output json <seed-dataset-id> | jq -r .id
# <training-dataset-id>

distil training-dataset status --output json <training-dataset-id> | jq -r .status

A full run can take 90 minutes and spends one training_datasets_from_seed_datasets_post credit. New accounts start with five.

The three levers on what gets generated

Every generation call is shaped by three inputs, and knowing which one to reach for is most of iterating well.

Lever	Where it lives	How it applies
`task_description`	`job_description.json`	Constant. Every generation call, and every evaluation and judge prompt
`synthetic_data_generation_instructions`	`job_description.json`	Constant. Every generation call, generation only
`mutators`	`config.yaml`, `synthgen`	Sampled. One value per configured mutator, per call

task_description says how to solve the task. It defines what a correct answer is, so it also feeds evaluation and the judge, and changing it changes what “correct” means everywhere. Keep it matched to your production prompt and constant across iterations.
synthetic_data_generation_instructions says how to generate the data: what the inputs should look like, their formats, domains, register and noise. It touches generation only, which makes it the safe place to steer inputs without redefining the task.
Mutators shape the distribution. Each call samples one value per configured mutator, so the composition of the list sets the proportions.

The two constants shift every example the same way, while the mutators decide how examples are distributed. So when the whole dataset is wrong in the same way, whether that’s a format, a misread rule or the wrong register throughout, fix a constant. When the mix is wrong, with a slice missing or over-represented against what production sees, change the mutator values and their proportions.

Mutators

Mutators append directives to the generation prompt, and each call samples one value per configured mutator. synthgen.mutators holds one entry per dimension you want to vary:

synthgen:
  mutators:
    - name: complexity
      values: [simple, medium, complex]
    - name: topic
      values:
        - "about billing disputes: refunds, double charges, invoice errors"
        - "about account cancellation: downgrades and win-back attempts"
        - "about technical support: setup problems and integration errors"

Nothing is applied by default and there are no built-in mutators, so a config that lists none generates every example from the same prompt. The only variation left is the teacher temperature and the sampled in-context exemplars.

The fields

Field	Required	Notes
`name`	Yes	The dimension this mutator varies. Keep it unique across mutators: it labels the sampled value and seeds this mutator’s own random stream
`values`	No	The values to sample from, each a plain string. Leave the field out and the teacher detects them from your job description and seed data, which then needs a `description`. See Letting the teacher pick the values. An empty list is a config error
`method`	No	`fixed` (the default) samples from the target distribution and never looks at what was generated. `adaptive` measures the generated data and steers sampling towards values that are falling behind. See Adaptive mutators
`target_distribution`	No	The proportions the mutator values should have in the dataset: `uniform` (the default), `match_seed` (the proportions of values the teacher classifies in your seed data), or one weight per value such as `[5, 3, 2]`. Weights are normalised, so proportions and relative counts both work. A weight list needs nonempty `values` (i.e. no automatic detection)
`description`	No	What the dimension means, for example `how many reasoning steps the answer needs`. Required when `values` is left out, since it’s the teacher’s only instruction for detecting them. Otherwise read only by the classifier, so it matters with `adaptive` and `match_seed` and is ignored with a fixed, uniform mutator
`auto_detect_sample_size`	No	Only used when `values` is left out. How many seed examples the teacher sees when detecting the values, a random sample when the seed data is larger. Default `50`
`beta`	No	Advanced, `adaptive` only. How far the sampler may drift from the target distribution, between 0 and 1 (larger means more drift). Default `0.9`. See Adaptive mutators

Unknown fields on a mutator are rejected. An explicit weight list needs exactly one weight per value.

A value is one string, nothing more. The description field describes the whole dimension, not one value, so write the detail of a value into the same string, conventionally after a colon, and quote the whole thing so YAML reads it as a string rather than a mapping:

values:
  - "simple: minimal reasoning required"      # a value
  - simple: minimal reasoning required        # a mapping, and a validation error

What it renders

Each mutator renders one line under a shared prelude:

Important! Make sure to follow those guidelines:
Generated examples should be simple
Generated examples should be about billing disputes: refunds, double charges, invoice errors

Every line is Generated examples should be followed by the value verbatim, so phrase your values to finish that sentence. That’s why the topics above read about billing disputes rather than a bare billing.

Each mutator draws from its own stream, derived from base.random_seed and the mutator name. A config reproduces its own mutation sequence, two mutators with the same number of values don’t move in lockstep, and renaming a mutator changes its draws.

How many, and in what proportions

One mutator per independent dimension. Each call samples once from every mutator, so the two above cover a 3x3 grid of scenarios without enumerating nine combinations.

Use 1-2 mutators of 3-10 values each. More dilutes the signal per value, fewer limits diversity.

Sampling follows target_distribution, by default uniform (equal weight per value). You can also provide custom sampling weights: values: [English, French] with target_distribution: [3, 1] asks for English in roughly three calls out of four.

Letting the teacher pick the values

If you know the dimension you want to vary but not the values it should take, leave values out and describe the dimension instead:

synthgen:
  mutators:
    - name: topic
      description: which part of the product the customer is asking about

Once per run, before generation starts, the teacher reads your job description (the task description, plus the tools or classes when the job has them) and a random sample of your seed training data, and proposes the values. It aims for values that cover what the task will meet in production, not only what the seed examples happen to contain, so expect a few values the seed data doesn’t show. It is instructed to propose between 3 and 8 values (but more if more are needed), each a few words, sometimes with a short explanation after a dash. The detected values are printed in the run logs, so read them there and copy them into values once you want to pin them down or edit them.

A few things to know:

description is required. It is the only instruction the teacher gets about the dimension, so a bare name: topic without one is a config error. Be as specific as you would be for an adaptive mutator.
auto_detect_sample_size (default 50) caps how many seed examples the teacher sees. A smaller seed set is passed whole. The sample is drawn from this mutator’s own random stream, so the same base.random_seed picks the same examples.
It works from an empty seed set too. With no seed data the teacher proposes values from the job description alone.
Combine it freely with method and match_seed. An adaptive mutator classifies against the detected values, and target_distribution: match_seed measures the seed data against them. An explicit weight list is rejected, since there are no listed values to match the weights to.
It costs one teacher call per mutator, on top of any classification the mutator does later.

Matching the seed distribution (advanced)

target_distribution: match_seed has the teacher classify your seed training data along the dimension once, at the start of the run, and uses the proportions it finds as the target. It works with either method (fixed and adaptive), and it costs one teacher call per seed example.

Be aware that values that never appear in the seed data get zero weight and are never asked for (with a warning in the logs). If none of the seed examples matches mutator values, the mutator falls back to uniform. If you want a value in the dataset that the seed data lacks, give explicit weights instead.

Adaptive mutators (advanced)

A fixed mutator asks for each value in the target proportions and never checks what came back. This happens if the teacher ignored mutation instructions, or when validation dropped examples of one mutator value more. In that case, the dataset can end up short on some values, skewing the data distribution. An adaptive mutator closes that loop:

synthgen:
  mutator_update_frequency: 5
  mutators:
    - name: topic
      method: adaptive
      description: which part of the product the customer is asking about
      target_distribution: [0.5, 0.3, 0.2]
      values:
        - "about billing disputes: refunds, double charges, invoice errors"
        - "about account cancellation: downgrades and win-back attempts"
        - "about technical support: setup problems and integration errors"

Every synthgen.mutator_update_frequency batches, where a batch is one generation_iteration_size round of generation and validation, the teacher labels each surviving example with the value it actually landed on. The mutator then recomputes the sampling weights so that the underrepresented values are sampled more often. Once every value has reached its share, sampling returns to the target distribution.

**Note: ** When classifying a batch, each example means one call to the teacher => slowing synthetic data generation.

Recipes for the former built-in mutators

complexity, length and specificity used to be built in, with complexity on by default. They’re ordinary config now. Copy the one you want, and use one at a time, since their directives conflict when stacked.

synthgen:
  mutators:
    - name: complexity
      values:
        - "trivial: obvious answer, no reasoning needed"
        - "simple: straightforward, minimal reasoning required"
        - "medium: some reasoning, a few factors to consider"
        - "complex: multiple factors, nuance, or ambiguity involved"
        - "highly complex: expert-level, many interacting factors, edge cases"

    # or
    - name: length
      values:
        - "short and concise: 1-2 sentences, essentials only"
        - "medium length: 3-5 sentences, covers main points"
        - "detailed: multiple paragraphs, includes context and nuance"

    # or
    - name: specificity
      values:
        - "generic and vague: avoid specific references and concrete terms"
        - "somewhat specific: use descriptive but general references and terms"
        - "specific: use realistic terms and references"
        - "very specific: use precise, domain-heavy terms and references"

Migrating from `mutation_topics`

synthgen.mutation_topics still works and is translated into mutators: a flat list becomes one mutator named topics_1, and nested lists become topics_1, topics_2, one per list. The translation also clears mutation_topics, so a config you read back after a run shows the mutators and an empty topic list. Setting mutators and mutation_topics together is a config error, so migrate a config in one move rather than half way.

synthgen.basic_mutators_to_use is accepted and has no effect at all. If you were relying on its ["complexity"] default, add the complexity recipe above.

Two things to re-read as you migrate. Topics used to render as Generated examples should focus on: billing disputes and now render as Generated examples should be billing disputes, so old phrasings may need rewording. And a value is a string only: the {key: description} mapping form and {key: ..., description: ...} entries are rejected rather than converted, so fold any such detail into one string.

Configuring generation

The full table is in Config file. The ones to set deliberately:

validation_max_total_length. Caps the characters in question + answer (+ context) per example. Set it to the maximum combined length you expect in real data, since the default of 30,000 rejects longer uploaded examples and silently filters longer generated ones.
generation_in_single_call, num_positive_exemplars_per_generation and num_unlabelled_exemplars_per_generation. Bring these down when examples are long. Each multiplies the prompt and output size per teacher call.
num_positive_exemplars_per_generation and num_negative_exemplars_per_generation (both default 1). In-context examples drawn from your training set: the positive count for the class or tool being generated, the negative count for the ones that aren’t. Neither may exceed the number of training rows. See In-context exemplar counts.
output_is_json: true. Whenever answers have to be valid JSON. QA tasks only.
base.llm_num_parallel_requests. Above the default of 4 it can help, but don’t expect linear gains, since per-call latency and between-batch validation usually dominate.

Try a small run first

Problems visible in 64 examples will be everywhere in 10,000, and a regenerated dataset costs one credit while training on a bad one costs a training credit plus hours. Generate a small batch first:

distil seed-dataset download-metadata -d ./smoke <seed-dataset-id>
# in ./smoke/config.yaml, under synthgen:
#   generation_target: 64
#   generation_iteration_size: 16

distil training-dataset create-from-seed-dataset --output json \
  --config ./smoke/config.yaml <seed-dataset-id> | jq -r .id

Checking what came back

Reading a sample is free, and downloading the whole dataset costs a credit.

# Free: up to 128 train rows, drawn deterministically from the first 384
distil training-dataset sample --output json <training-dataset-id> > sample.json

# Size in bytes, not rows
distil training-dataset metrics --output json <training-dataset-id> | jq .train_data_size_bytes

Compare the count against your target first. Falling well short means validation filtered heavily, usually length caps or format problems.

Then check three things:

Form. Does each example parse, carry the required output format, and respect the stated constraints? A malformed row is a defect whatever the rate. Read a handful of answers to confirm the teacher understood the task, but don’t gate on a label error rate, since the student absorbs a low rate of teacher noise.
Distribution. Compare generated against seed data along the dimensions that matter for your task: length, topic coverage, style and register, class balance.
Targeted slices. If your mutators or job description asked for a particular slice, count it in the output.

Axis 3 needs a big enough sample to be readable. A run makes generation_target / generation_in_single_call mutator draws, so a trial target of 64 with generation_in_single_call: 4 gives just 16 draws. A grid of more than 16 cells leaves cells empty by pigeonhole whatever the config says. Size a trial run to at least twice your grid before treating empty cells as a signal, or check this on the full run instead.

generation_target is a floor rounded up to the next generation_iteration_size batch, so landing above it is normal.

If the dataset isn’t good

Don’t carry it into training. The training metrics won’t tell you the data was bad. It shows up as a plateau you spend a full training run discovering.

Go back to the three levers: mutators for coverage gaps, the generation instructions for how inputs look, the relevant config field for correctness or format problems. Then generate again.

Model training.