Overview

A dataset is the one entity that holds your data: the task definition, your production traces, and your labeled train and test rows. The platform grows it from there. Read this page before any task-specific one; it covers what every dataset needs, whatever the task type.

The dataset directory

Put the files in one directory. config.yaml sets the task type and the models, and job_description.json describes the task in words; both are required. The three data files are optional, and a missing one is an empty split. All data files are JSONL.

input-dir/
├── config.yaml
├── job_description.json
├── traces.jsonl          # production conversations, one per line
├── train.jsonl           # labeled training rows
└── test.jsonl            # held-out test rows

The same five files are what every dataset on the platform holds and what distil dataset download writes, so a downloaded directory is accepted unchanged by distil dataset create.

The three shapes

Most datasets start in one of three shapes, and the platform fills what is missing:

You have	Upload	What the platform fills
Traces only	`traces.jsonl`; train and test empty	The test set and the train set, both by relabeling traces and generating synthetic rows
Traces and a test set	`traces.jsonl` and `test.jsonl`; train empty	The train set. Your test set is used as it is, or extended
Train and test rows, few or no traces	`train.jsonl` and `test.jsonl`, and `traces.jsonl` if you have any	The train set, by synthetic rows generated from your rows (with the traces as context when there are any)

A test set with no train set works too: teacher evaluation runs zero-shot, generation fills the train split, and every score is available. A dataset with no test set and no traces can be trained on but never scored, so add test rows before anything else.

Create the dataset from the directory:

distil dataset create --data input-dir

The command prints Upload successful. Dataset ID: <dataset-id>. Every later job on this data takes that ID. You can also name the files one by one; --config and --job-description are required and the other three optional:

distil dataset create --config config.yaml --job-description job_description.json \
  --traces traces.jsonl --test test.jsonl

If your traces come from an inference endpoint, the dataset is created from the traces object instead, with the same two files (Endpoint records to traces).

Training and test data

Train and test rows are chat-format conversations.

Task	Row shape
`classification`, `question-answering`	`{"messages": [<user>, <assistant>]}`
`chat-completion`	`{"messages": [<user>, <assistant>, ...]}`, assistant turns carry text, `tool_calls` or both
`chat-completion-agentic`	`{"messages": [<user>, <assistant with tool_calls>, <tool>, ..., <assistant>]}`

<user> and <assistant> are {"role": ..., "content": "<non-empty>"}. The final assistant message carries the answer or label as its content.
A row may start with a system message. It is kept only with base.variable_system_prompt on, and replaced by the job description’s prompt otherwise (System prompts).
Chat completion rows, tool calls and the role rules are on the Chat completion and Agentic chat completion pages.
An assistant message may carry reasoning_content for a reasoning student; it is dropped on load unless base.enable_thinking is on.
Flat {"question": ..., "answer": ...} rows are refused; write every row as a messages conversation.

Trace rows follow trace_processing.observation_format; see Traces.

There’s no hard minimum row count. Aim for 20+ diverse training examples and a test set that covers your production distribution, a few hundred rows (Building the test set).

Job description

What your task is, in words. The teacher reads it when it relabels your traces, when it’s evaluated and when it generates your training data.

{
  "task_description": "...",
  "llm_as_a_judge_instructions": "..."
}

task_description describes the task: what the model should do, what a correct answer looks like, how the output is formatted. Derive it from the system prompt of the system you’re replacing, and keep it the same in every iteration. With per-request system prompts, see System prompts.
llm_as_a_judge_instructions tells the judge what to accept when it scores an answer. Optional, and not valid for classification.
trace_processing_instructions is added to the relabeling instructions only, for something unusual about your traces. Optional.

Classification also needs classes_description. Chat completion takes tools (required for the agentic variant). The full field reference is Job description.

Config

The task type, the two models and the judge:

base:
  task: question-answering
  student_model_name: Qwen3.5-4B
  teacher_model_name: zai.glm-5.3-low-thinking
evaluation:
  llm_as_a_judge_model_name: zai.glm-5.3-low-thinking
trace_processing:
  observation_format: openai_messages

base.teacher_model_name is also the model that relabels your traces, so it matters from the first job; the large GLM 5.3 teacher is the recommended choice (Supported models). Scores read by two different judges are not comparable, so the judge is set once for a project. Classification is scored on label accuracy and has no judge, so its config leaves out the evaluation section. Everything else has a default; see Config file.

Validation rules

These are enforced when the dataset is created and before every expand, so check them before submitting.

Every message content is non-empty, except a chat completion assistant turn that carries only tool_calls.
Per row, the total length is at most synthgen.validation_max_total_length.
Train and test share no identical rows. Exact duplicates fail.
For classification, the label sets in train.jsonl, test.jsonl and classes_description are identical, and a populated split holds every class.
Chat completion rows follow the conversation rules on their task page; rows with parallel tool calls need synthgen.max_tool_calls_per_turn of 2 or more.
visual_task matches whether the rows carry images.
traces.jsonl parses in the configured observation_format.

Creating the dataset is the validator. distil dataset create parses the directory exactly as the jobs will and rejects an invalid bundle with the reason, before any job is created. Every expand validates the dataset it writes with the same rules. The messages and their fixes are in Troubleshooting.

Empty splits

A missing or empty data file says you have no data of that kind. A populated split is held to every rule above. What each empty file means for the stages that follow:

No traces. Relabeling can’t run: it turns traces into rows and has nothing to work from. Synthetic data generation still runs, without the traces as context. This is the normal shape when you upload labeled train and test rows.
No test rows. Teacher evaluation fails, because there is nothing to score the teacher against, and training produces a model with no scores: distil slm metrics returns null, which is not a failed job. Build the test set from the traces first.
No train rows. Training fails, because there is nothing to train on. Teacher evaluation still runs, because it is always zero-shot, and synthetic data generation fills the split from the task description, the class or tool definitions and the traces, with no rows to imitate. Build the train set before training.

Pick your task’s page for the exact row format and a worked example:

The trace file format is on Traces. Then build the test set, or, with a complete test set uploaded, run teacher evaluation.