Synthetic data generation

Your seed dataset has a few dozen examples. Training needs thousands. This stage is where the teacher writes them: it generates examples from your seed data, job description and mutators, then validates and de-duplicates them into a training dataset.

The result is a merged dataset. The root train.jsonl is your seed examples plus the surviving synthetic ones, so the synthetic count is the total minus your seed count.

This stage also runs from an empty seed set. The exemplar blocks drop to zero-shot and the teacher works from your task description, your tool or class definitions, and unstructured.jsonl. What it generates becomes the training set, which is why it has to run before training when you have no training examples. Nothing anchors format or style in that case, so read the output closely.

Running it

distil training-dataset create-from-seed-dataset --output json <seed-dataset-id> | jq -r .id
# <training-dataset-id>

distil training-dataset status --output json <training-dataset-id> | jq -r .status

A full run can take 90 minutes and spends one training_datasets_from_seed_datasets_post credit. New accounts start with five.

Check your training credits before generating, not after

A generation run with no training credit left spends a credit on data you then can’t train on. Run distil credits-balance and read it against your whole plan.

The three levers on what gets generated

Every generation call is shaped by three inputs, and knowing which one to reach for is most of iterating well.

Lever	Where it lives	How it applies
task_description	job_description.json	Constant. Every generation call, and every evaluation and judge prompt
synthetic_data_generation_instructions	job_description.json	Constant. Every generation call, generation only
mutators	config.yaml, synthgen	Sampled. One value per configured mutator, per call

• task_description says how to solve the task. It defines what a correct answer is, so it also feeds evaluation and the judge, and changing it changes what “correct” means everywhere. Keep it matched to your production prompt and constant across iterations.
• synthetic_data_generation_instructions says how to generate the data: what the inputs should look like, their formats, domains, register and noise. It touches generation only, which makes it the safe place to steer inputs without redefining the task.
• Mutators shape the distribution. Each call samples one value per configured mutator, so the composition of the list sets the proportions.

The two constants shift every example the same way, while the mutators decide how examples are distributed. So when the whole dataset is wrong in the same way, whether that’s a format, a misread rule or the wrong register throughout, fix a constant. When the mix is wrong, with a slice missing or over-represented against what production sees, change the mutator values and their proportions.

Mutators

Mutators append directives to the generation prompt, and each call samples one value per configured mutator. synthgen.mutators holds one entry per dimension you want to vary:

synthgen:
  mutators:
    - name: complexity
      values: [simple, medium, complex]
    - name: topic
      values:
        - "about billing disputes: refunds, double charges, invoice errors"
        - "about account cancellation: downgrades and win-back attempts"
        - "about technical support: setup problems and integration errors"

Nothing is applied by default and there are no built-in mutators, so a config that lists none generates every example from the same prompt. The only variation left is the teacher temperature and the sampled in-context exemplars.

The fields

Field	Required	Notes
name	Yes	The dimension this mutator varies. Keep it unique across mutators: it labels the sampled value and seeds this mutator’s own random stream
values	Yes	At least one value, each a plain string. An empty list is a config error
method	No	uniform, the default and the only implemented option. Anything else is a config error rather than a silent fallback

Unknown fields on a mutator are rejected.

A value is one string, nothing more. There’s no separate description field, so write the detail into the same string, conventionally after a colon, and quote the whole thing so YAML reads it as a string rather than a mapping:

values:
  - "simple: minimal reasoning required"      # a value
  - simple: minimal reasoning required        # a mapping, and a validation error

What it renders

Each mutator renders one line under a shared prelude:

Important! Make sure to follow those guidelines:
Generated examples should be simple
Generated examples should be about billing disputes: refunds, double charges, invoice errors

Every line is Generated examples should be followed by the value verbatim, so phrase your values to finish that sentence. That’s why the topics above read about billing disputes rather than a bare billing.

Each mutator draws from its own stream, derived from base.random_seed and the mutator name. A config reproduces its own mutation sequence, two mutators with the same number of values don’t move in lockstep, and renaming a mutator changes its draws.

How many, and in what proportions

One mutator per independent dimension. Each call samples once from every mutator, so the two above cover a 3x3 grid of scenarios without enumerating nine combinations.

Use 1-2 mutators of 3-10 values each. More dilutes the signal per value, fewer limits diversity.

Sampling is uniform over the listed values, so composition sets the proportions. Repeat a value to weight it: three entries asking for English and one for French gives roughly a 75/25 split.

A single-value mutator is a useful special case. It applies the same directive to every call, turning the mutator into a constant extra generation instruction without touching the job description.

Recipes for the former built-in mutators

complexity, length and specificity used to be built in, with complexity on by default. They’re ordinary config now. Copy the one you want, and use one at a time, since their directives conflict when stacked.

synthgen:
  mutators:
    - name: complexity
      values:
        - "trivial: obvious answer, no reasoning needed"
        - "simple: straightforward, minimal reasoning required"
        - "medium: some reasoning, a few factors to consider"
        - "complex: multiple factors, nuance, or ambiguity involved"
        - "highly complex: expert-level, many interacting factors, edge cases"

    # or
    - name: length
      values:
        - "short and concise: 1-2 sentences, essentials only"
        - "medium length: 3-5 sentences, covers main points"
        - "detailed: multiple paragraphs, includes context and nuance"

    # or
    - name: specificity
      values:
        - "generic and vague: avoid specific references and concrete terms"
        - "somewhat specific: use descriptive but general references and terms"
        - "specific: use realistic terms and references"
        - "very specific: use precise, domain-heavy terms and references"

Migrating from mutation_topics

synthgen.mutation_topics still works and is translated into mutators: a flat list becomes one mutator named topics_1, and nested lists become topics_1, topics_2, one per list. The translation also clears mutation_topics, so a config you read back after a run shows the mutators and an empty topic list. Setting mutators and mutation_topics together is a config error, so migrate a config in one move rather than half way.

synthgen.basic_mutators_to_use is accepted and has no effect at all. If you were relying on its ["complexity"] default, add the complexity recipe above.

Two things to re-read as you migrate. Topics used to render as Generated examples should focus on: billing disputes and now render as Generated examples should be billing disputes, so old phrasings may need rewording. And a value is a string only: the {key: description} mapping form and {key: ..., description: ...} entries are rejected rather than converted, so fold any such detail into one string.

Configuring generation

The full table is in Config file. The ones to set deliberately:

• validation_max_total_length. Caps the characters in question + answer (+ context) per example. Set it to the maximum combined length you expect in real data, since the default of 30,000 rejects longer uploaded examples and silently filters longer generated ones.
• generation_in_single_call, num_positive_exemplars_per_generation and num_unlabelled_exemplars_per_generation. Bring these down when examples are long. Each multiplies the prompt and output size per teacher call.
• num_positive_exemplars_per_generation and num_negative_exemplars_per_generation (both default 1). In-context examples drawn from your training set: the positive count for the class or tool being generated, the negative count for the ones that aren’t. Neither may exceed the number of training rows. See In-context exemplar counts.
• output_is_json: true. Whenever answers have to be valid JSON. QA tasks only.
• base.llm_num_parallel_requests. Above the default of 4 it can help, but don’t expect linear gains, since per-call latency and between-batch validation usually dominate.

Try a small run first

Problems visible in 64 examples will be everywhere in 10,000, and a regenerated dataset costs one credit while training on a bad one costs a training credit plus hours. Generate a small batch first:

distil seed-dataset download-metadata -d ./smoke <seed-dataset-id>
# in ./smoke/config.yaml, under synthgen:
#   generation_target: 64
#   generation_iteration_size: 16

distil training-dataset create-from-seed-dataset --output json \
  --config ./smoke/config.yaml <seed-dataset-id> | jq -r .id

Checking what came back

Reading a sample is free, and downloading the whole dataset costs a credit.

# Free: up to 128 train rows, drawn deterministically from the first 384
distil training-dataset sample --output json <training-dataset-id> > sample.json

# Size in bytes, not rows
distil training-dataset metrics --output json <training-dataset-id> | jq .train_data_size_bytes

Compare the count against your target first. Falling well short means validation filtered heavily, usually length caps or format problems.

Then check three things:

1. Form. Does each example parse, carry the required output format, and respect the stated constraints? A malformed row is a defect whatever the rate. Read a handful of answers to confirm the teacher understood the task, but don’t gate on a label error rate, since the student absorbs a low rate of teacher noise.
2. Distribution. Compare generated against seed data along the dimensions that matter for your task: length, topic coverage, style and register, class balance.
3. Targeted slices. If your mutators or job description asked for a particular slice, count it in the output. Mutator values are suggestions and can silently yield nothing.

Axis 3 needs a big enough sample to be readable. A run makes generation_target / generation_in_single_call mutator draws, so a trial target of 64 with generation_in_single_call: 4 gives just 16 draws. A grid of more than 16 cells leaves cells empty by pigeonhole whatever the config says. Size a trial run to at least twice your grid before treating empty cells as a signal, or check this on the full run instead.

generation_target is a floor rounded up to the next generation_iteration_size batch, so landing above it is normal.

If the dataset isn’t good

Don’t carry it into training. The training metrics won’t tell you the data was bad. It shows up as a plateau you spend a full training run discovering.

Go back to the three levers: mutators for coverage gaps, the generation instructions for how inputs look, the relevant config field for correctness or format problems. Then generate again.

Next

Model training.