Traces
If you run an LLM in production, its logs are training data. A dataset holds them as
traces.jsonl, one conversation per line, next to your config and job description
(Dataset overview). Relabeling turns traces into test and train
rows, and synthetic data generation uses the rest as context
(The expand operations). If you have no logs,
an inference endpoint can collect them for you.
Pick your task type before you convert anything, since it decides the shape of everything else. See Task selection.
A trace file reaches the platform in one of three ways:
distil dataset create --traces traces.jsonlnext to--configand--job-description, and anytrain.jsonlortest.jsonlyou have;distil traces upload --traces traces.jsonl, thendistil dataset create-from-traceswith the config and the job description;distil traces create-from-inference-endpoint, where the platform pulls an endpoint’s records and you write no file (Endpoint records to traces).
Your traces
Section titled “Your traces”Your logs, one trace per line. Set trace_processing.observation_format in config.yaml to
match their shape.
openai_messages is the default. Each line carries a messages array with roles system,
user, assistant and tool. Assistant turns can carry tool_calls. A line can also carry the
tools the request offered (Tools in traces).
{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
A line may also end on the user’s turn, with no answer: relabeling then writes the answer instead of correcting one.
openai_messages_with_images is the same, except user content can be a list of parts:
[
{"type": "text", "text": "What is in this image?"},
{"type": "image_url", "image_url": {"url": "..."}}
]
It requires base.visual_task: true, and every image is a data:image/...;base64 URI on a user
message (Vision models).
unstructured_with_openai_messages gives each line a context field: a string that
contains the messages array as text. Set only by
Migrating old entities; you never write it by hand.
langfuse is the format of an inference endpoint’s records when the traces object is created
directly from the endpoint (Endpoint records to traces). You never
write it by hand; set it in the config of the dataset you create from that traces object. It
rejects image content.
Converting your logs
Section titled “Converting your logs”- The system prompt goes into the job description. Leading system messages are replaced by
the job description’s prompt, so write the system prompt’s content into
task_description. Withbase.variable_system_prompton, each trace keeps its own system message instead. Variable input becomes the user message and model output becomes the assistant message. - Every trace is a conversation, rewritten whole. A one-shot exchange is a two-turn conversation.
- Watch the length limit.
synthgen.validation_max_total_length(default 30,000 characters) applies to relabeled rows, so raise it if your inputs embed documents or schemas. - Write clean JSONL. Strip control characters and the unicode line separators U+2028 and U+2029, which are a common cause of parse failures in exported logs.
Tools in traces
Section titled “Tools in traces”For chat-completion, a trace can carry the tools its request offered, as a tools list in the
OpenAI function-calling format next to messages. Traces from an inference endpoint carry them
already. The trace’s own list works like a row’s
(Tools per row):
- A trace without a
toolskey gets the job description’s tools. An empty list means the request offered no tools. - Relabeling shows the teacher the trace’s own tools, checks its tool calls against them, and
the relabeled row keeps them. With
trace_processing.relabel: false, the row keeps them too. - A trace whose
toolsdon’t parse is dropped. So is a trace with a call to a tool the job description doesn’t declare, or with an argument name that tool doesn’t declare. - A log records only the tools of its last request, so a trace with its own list is also offered the job description’s definition of every tool its earlier messages call that the list lacks.
question-answering has no tools: its relabeled rows are offered none, whatever the trace
carries.
Job description
Section titled “Job description”What your task is, in words. The teacher reads it when it relabels your traces, and again when it generates training data. For a build from traces you usually want three fields:
{
"task_description": "...",
"llm_as_a_judge_instructions": "...",
"trace_processing_instructions": "..."
}
-
task_descriptiondescribes the task. Write it from the system prompt your production system runs, with the same care: the output format with an example, the include and exclude rules, the edge cases. It stays the same through every iteration. -
llm_as_a_judge_instructionstells the judge what counts as a correct answer, including format rules such as no code fences. Optional. -
trace_processing_instructionsis added to the relabeling instructions only. Use it when the relabeling has to respect something unusual about your traces:This is a live phone call; preserve the caller’s interruptions and any cut-off utterances verbatim.
Leave it out when no special handling is needed.
chat-completion also takes optional tools. The full field reference is
Job description.
Config
Section titled “Config”The task type, the models, and the judge:
base:
task: question-answering
student_model_name: Qwen3.5-4B
teacher_model_name: zai.glm-5.3-low-thinking
evaluation:
llm_as_a_judge_model_name: zai.glm-5.3-low-thinking
trace_processing:
observation_format: openai_messages
base.teacher_model_name is also the model that relabels your traces, unless
trace_processing.teacher_model_name names a different one, so it matters from the first job.
The large GLM 5.3 teacher, zai.glm-5.3-low-thinking, is the recommended choice
(Supported models). Scores read by two different judges are
not comparable, so the judge is usually set once for a project. Every other field has a default:
Relabeling and synthetic data covers the ones worth setting, and
Config file has the full table.
Create the dataset (Dataset overview), then build the test set.