Mutators: set the mix of your synthetic dataset

Most of your production traces are regular traffic: the request your agent handles a thousand times a day, the password reset, the “can I speak to a human”. But sometimes, there are situations that go off script - the customer interrupts the agent and wants something completely different, starts speaking in a different language, or has a specific problem that appears only once a year. They are rare in the traces because they are rare in production. But that does not mean they are unimportant - quite the opposite.

A training set generated from those traces inherits that mix. The student sees the common case thousands of times and the edge cases only a handful of times. Without enough practice, it fails on the hard examples, and your customers lose trust in your platform.

With our new feature, you can now solve this. We are introducing prompt mutators: additional instructions to the teacher describing what the generated data should look like. You specify the situations your data should cover, the happy path, the edge cases, the rare request that costs money when it goes wrong, and how often each should appear, and we generate the data based on that. The model learns the important cases that you care about and not only the simple majority cases.

What is distil labs

distil labs fine-tunes task-specific small language models automatically. You bring the production traces your agent already produces, or a few dozen hand-written examples, and a teacher LLM generates a synthetic training set from them. Rule-based validators filter it, a small student model is fine-tuned on seed plus surviving synthetic data, and the result deploys behind an OpenAI-compatible endpoint. The student can beat its teacher on the narrow task because validation strips the teacher’s mistakes and the student spends all of its capacity on one job. What the teacher generates is decided by three inputs: the task description, the generation instructions, and the mutators. The first two are constant across every call. The mutators are the sampled one, and they are what this post is about.

The problem: the cases that are not in the data

Take a customer support agent for a subscription product. You export 400 traces and sort them by what the customer wanted:

What the customer wanted	Share of traces
Password reset, login problems	41%
To speak to a human	27%
Plan and pricing questions	20%
A billing dispute: refund, double charge, invoice error	9%
Customer is rude	3%

The first three rows are the easy part. The last two are where the agent has to be good: a billing dispute handled badly costs a refund, and a rude customer handled badly becomes an escalation. Together they make up 12% of the seed data.

If you generate 5,000 synthetic examples from this seed set with no further instructions, the generated data follows the same distribution. Every call uses the same prompt, and the only variation comes from the teacher’s temperature and which seed examples were sampled as exemplars. So about one example in eight covers the cases you care about, and the specific situations you have watched fail in production may not appear at all.

A mutator changes the mix where it is made. You name the property that is skewed, in this case topic, list the values you want covered, and tell the teacher how often to ask for each. The teacher then writes billing disputes and rude customers in the share you set, not the share the traces happened to have.

Quickstart

Add one mutator to config.yaml, under synthgen:

synthgen:
  mutators:
    - name: topic
      target_distribution: uniform
      values:
        - "about password resets and login problems"
        - "about reaching a human agent"
        - "about plans and pricing"
        - "about billing disputes: refunds, double charges, invoice errors"
        - "from a rude customer: insults, threats to leave, sarcasm"

Every generation call samples one value and appends it to the generation prompt. What the teacher sees on a call that drew the fourth value:

Important! Make sure to follow those guidelines:
Generated examples should be about billing disputes: refunds, double charges, invoice errors

With uniform, each topic is selected equally often. You can also give the target distribution as a list of probabilities, one per value: [0.225, 0.225, 0.225, 0.225, 0.1] keeps the first four topics even and asks for rude customers in one call out of ten, still three times their share in the traces. Then run a smoke generation and count the topics in the output.

A trial target of 64 examples with 16 per call is only four mutator draws, so read the counts against the target you set rather than expecting them to match exactly. Once they look right, run the full generation and train.

Setting the proportions

The quickstart used the default, uniform, and showed probabilities in passing. There are three ways to set the target distribution of mutator values, and choosing one depends on whether your seed data already has the mix you want.

Uniform is the default. Each value has the same sampling probability. If your seed data is skewed and you simply want the student to see every case equally often, add the mutator and stop.

Explicit weights override the mix. One weight per value, normalised for you, so proportions and relative counts both work:

    - name: language
      values: [English, French]
      target_distribution: [3, 1]

This asks for English in roughly three calls out of four. Choose this if you know what your topic distribution should be. You can also specify probabilities instead: [0.75, 0.25].

match_seed infers the distribution from seed data. At the start of the run, the teacher classifies each seed example along the mutator dimension and uses the proportions it finds as the target:

    - name: topic
      description: which part of the product the customer is asking about
      target_distribution: match_seed
      values:
        - "about billing disputes: refunds, double charges, invoice errors"
        - "about account cancellation: downgrades and win-back attempts"
        - "about technical support: setup problems and integration errors"

Use it when the seed data is already the distribution you want and you need the synthetic data to hold that shape rather than drift towards whatever the teacher finds easiest to write. It costs one teacher call per seed example, once, so 200 seed examples is 200 calls before generation starts. Two caveats - a value that never appears in the seed data gets zero weight and is never asked for; if you want something the seed lacks, that is a job for explicit weights. And if no seed example lands on any of your values, the mutator falls back to uniform, with a warning.

The description field helps the classifier decide the topic of each seed example.

By the way: when the data is hard to generate

A fixed mutator asks for each value in the target proportions and never checks what came back. Usually that is fine. But for hard-to-design mutation topics, the teacher sometimes struggles to follow instructions, or validation rejects the examples due to format errors more often. Because of that, the dataset ends up unbalanced.

For that case there is method: adaptive. Every five batches by default (synthgen.mutator_update_frequency), the teacher labels each example that survived validation with the value it actually landed on. The running counts are compared against the target for the whole run, sampling shifts towards the values furthest behind, and once every value has reached its share it returns to the target distribution. Batches in between are not labelled; their counts are extrapolated from the last labelled one. Each label is a teacher call, so the default labels one batch in five, and raising the frequency is cheaper and coarser.

synthgen:
  mutator_update_frequency: 5
  mutators:
    - name: topic
      method: adaptive
      description: which part of the product the customer is asking about
      target_distribution: [0.5, 0.3, 0.2]
      values:
        - "about billing disputes: refunds, double charges, invoice errors"
        - "about account cancellation: downgrades and win-back attempts"
        - "about technical support: setup problems and integration errors"

Reach for it only when a smoke run shows the dataset is imbalanced, and read the counts before you do. The classifier sees validated examples only, so a value that validation rejects looks under-generated and gets asked for more. If it is being rejected because it is genuinely hard to produce correctly, adapting burns teacher calls on it; the fix is the value’s wording, or the validator. Write the description for any adaptive mutator, for the same reason it matters with match_seed.

Try it on your own traces

You already know which situations your model has to handle and which ones your traces barely cover. Write them down as a list of values, add a topic mutator to your config, and generate a training set that has them in the proportions you want. The free tier includes two full training runs, so the first attempt costs you a config file and an afternoon.

The field reference, match_seed, the adaptive method and the recipes for complexity, length and specificity are in the synthetic data docs. If your dataset does not come out the way you expected, tell us in Slack and we will look at the counts with you.

distil labs trains task-specific small language models that are cheaper and faster per request than general-purpose LLMs, with equal-or-better accuracy on bounded tasks. Docs · GitHub · Slack