Writing a Job Description for Synthetic Data Generation
job_description.json is where you define what correct looks like. It is a JSON file with a task_description plus task-specific fields, and it does more work than any other input: it steers generation, it is the yardstick for relevance filtering, and it tells the judge how to score. Vague description, vague dataset.
What a job description is and is not
It is a specification, not a prompt. Both routes into the platform need one — the minimal dataset path and the trace upload path use the same file in the same format.
| It carries | It does not carry |
|---|---|
| What the model should do | Few-shot examples (those go in train.jsonl) |
| The exact output format | Domain background (that goes in unstructured.jsonl) |
| Edge-case rules and tie-breakers | Model or hyperparameter choices (those go in config.yaml) |
| Task-specific schemas, classes, tools | Anything about your infrastructure |
The reason this file matters disproportionately is the split between signal types. Your examples and traces carry distributional signal — what inputs look like. The job description carries normative signal — what a right answer is. Generation combines the two, and it is the only place the second one lives. Why every path ends in synthetic data sets out the mechanism.
Step 1: Start from your existing prompt
If you are already running an LLM feature, its system prompt is the fastest first draft. It already encodes the vocabulary, the constraints, and the edge cases you learned the hard way.
{
"task_description": "You are a helpful banking assistant that answers customer questions about their accounts, transactions, and banking products."
}Then edit it in one specific direction: a system prompt is written to steer a model at inference time, while a job description is written to define a task. Remove the politeness scaffolding and the “you are an expert” framing. Keep the rules.
Step 2: State the output format exactly
This is the highest-leverage sentence in the file, because format violations are what validators reject and what makes a model unusable downstream.
{
"task_description": "Extract the requested information from the provided invoice text. Return only the specific value asked for, without additional explanation. If the information is not found, respond with 'Not found'."
}Note the three things that short paragraph does: it says what to extract, it forbids commentary, and it specifies the behaviour for the failure case. Unspecified failure cases are where synthetic datasets go inconsistent — half the generated examples will apologise and half will return an empty string, and the student learns both.
Step 3: Add the task-specific fields
Each task type expects different additional fields, documented per task in the data preparation guides.
| Task | Additional fields |
|---|---|
question-answering |
input_description |
classification |
classes_description |
tool-calling-closed-book |
tools |
multi-turn-tool-calling-closed-book |
tools |
Tool schemas use the OpenAI function-calling format. Get them exactly right, because the schema is what lets generation ignore wrong function names that appear in noisy traces — that mechanism is worth 25.9 percentage points under schema drift in our traces benchmark.
For classification, write classes_description so the boundaries between classes are explicit. “Complaint” and “Query” overlap in ways that only show up once a teacher has generated four thousand examples that straddle them.
Step 4: Describe the input, not only the output
input_description is easy to skip and expensive to skip. It tells the teacher what to invent.
{
"task_description": "Extract the requested information from the provided invoice text. Return only the value asked for.",
"input_description": "Invoice text containing details such as invoice number, date, vendor name, line items, subtotal, tax, and total amount. The question will ask for a specific piece of information from the invoice.",
"llm_as_a_judge_instructions": "Compare the predicted answer to the reference answer for the given question. Output 'good' if the prediction matches the reference value or is semantically equivalent, otherwise output 'bad'."
}Without it, generated inputs drift toward whatever the teacher considers a prototypical example of the domain. With it, they look like your data. This is the same lever WizardLM exploits when it evolves instructions along controlled axes rather than sampling freely, and the reason Stanford Alpaca needed a seed pool at all.
Step 5: Write the judge instructions
llm_as_a_judge_instructions is optional and usually worth adding. It defines what counts as a correct answer when correctness is not literal string equality — which is most of the time for question answering.
Be specific about tolerance. Is $540 equivalent to 540.00? Is a longer but correct answer good or bad? Whatever you decide becomes the metric you optimise against, so decide deliberately rather than letting a default judge decide for you. See metrics for how the scores are computed.
Step 6: Keep it short enough to filter with
On the trace path there is a constraint that does not exist elsewhere: the job description is used to score every seed trace for relevance. A very long description can overwhelm the filtering model.
The config has a documented escape hatch — trace_processing.compress_job_description, off by default, which has the teacher compress the description before relevance filtering. Turning it on is a workaround. Writing a tighter description is the fix.
Checking whether it is good enough
Run teacher evaluation and read the result before training anything. A teacher that scores badly on your task with your description is telling you the description is ambiguous far more often than it is telling you the task is hard.
Then read twenty generated examples. If they are all subtly the same, your description is over-constrained. If a third of them are off-format, it is under-specified. Either way the file is cheaper to change than a training run — reprocess with distil upload create-from-traces <traces-id> --job-description ./job_description.json and compare. Turn production traces into training data and fine-tune with 20 examples cover the surrounding steps.