Skip to content

Traces

If you run an LLM in production, its logs are training data. A dataset holds them as traces.jsonl, one conversation per line, next to your config and job description (Dataset overview). Relabeling turns traces into test and train rows, and synthetic data generation uses the rest as context (The expand operations). If you have no logs, an inference endpoint can collect them for you.

Pick your task type before you convert anything, since it decides the shape of everything else. See Task selection.

A trace file reaches the platform in one of three ways:

  • distil dataset create --traces traces.jsonl next to --config and --job-description, and any train.jsonl or test.jsonl you have;
  • distil traces upload --traces traces.jsonl, then distil dataset create-from-traces with the config and the job description;
  • distil traces create-from-inference-endpoint, where the platform pulls an endpoint’s records and you write no file (Endpoint records to traces).

Your logs, one trace per line. Set trace_processing.observation_format in config.yaml to match their shape.

openai_messages is the default. Each line carries a messages array with roles system, user, assistant and tool. Assistant turns can carry tool_calls. A line can also carry the tools the request offered (Tools in traces).

{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

A line may also end on the user’s turn, with no answer: relabeling then writes the answer instead of correcting one.

openai_messages_with_images is the same, except user content can be a list of parts:

[
  {"type": "text", "text": "What is in this image?"},
  {"type": "image_url", "image_url": {"url": "..."}}
]

It requires base.visual_task: true, and every image is a data:image/...;base64 URI on a user message (Vision models).

unstructured_with_openai_messages gives each line a context field: a string that contains the messages array as text. Set only by Migrating old entities; you never write it by hand.

langfuse is the format of an inference endpoint’s records when the traces object is created directly from the endpoint (Endpoint records to traces). You never write it by hand; set it in the config of the dataset you create from that traces object. It rejects image content.

  • The system prompt goes into the job description. Leading system messages are replaced by the job description’s prompt, so write the system prompt’s content into task_description. With base.variable_system_prompt on, each trace keeps its own system message instead. Variable input becomes the user message and model output becomes the assistant message.
  • Every trace is a conversation, rewritten whole. A one-shot exchange is a two-turn conversation.
  • Watch the length limit. synthgen.validation_max_total_length (default 30,000 characters) applies to relabeled rows, so raise it if your inputs embed documents or schemas.
  • Write clean JSONL. Strip control characters and the unicode line separators U+2028 and U+2029, which are a common cause of parse failures in exported logs.

For chat-completion, a trace can carry the tools its request offered, as a tools list in the OpenAI function-calling format next to messages. Traces from an inference endpoint carry them already. The trace’s own list works like a row’s (Tools per row):

  • A trace without a tools key gets the job description’s tools. An empty list means the request offered no tools.
  • Relabeling shows the teacher the trace’s own tools, checks its tool calls against them, and the relabeled row keeps them. With trace_processing.relabel: false, the row keeps them too.
  • A trace whose tools don’t parse is dropped. So is a trace with a call to a tool the job description doesn’t declare, or with an argument name that tool doesn’t declare.
  • A log records only the tools of its last request, so a trace with its own list is also offered the job description’s definition of every tool its earlier messages call that the list lacks.

question-answering has no tools: its relabeled rows are offered none, whatever the trace carries.

What your task is, in words. The teacher reads it when it relabels your traces, and again when it generates training data. For a build from traces you usually want three fields:

{
  "task_description": "...",
  "llm_as_a_judge_instructions": "...",
  "trace_processing_instructions": "..."
}
  • task_description describes the task. Write it from the system prompt your production system runs, with the same care: the output format with an example, the include and exclude rules, the edge cases. It stays the same through every iteration.

  • llm_as_a_judge_instructions tells the judge what counts as a correct answer, including format rules such as no code fences. Optional.

  • trace_processing_instructions is added to the relabeling instructions only. Use it when the relabeling has to respect something unusual about your traces:

    This is a live phone call; preserve the caller’s interruptions and any cut-off utterances verbatim.

    Leave it out when no special handling is needed.

chat-completion also takes optional tools. The full field reference is Job description.

The task type, the models, and the judge:

base:
  task: question-answering
  student_model_name: Qwen3.5-4B
  teacher_model_name: zai.glm-5.3-low-thinking
evaluation:
  llm_as_a_judge_model_name: zai.glm-5.3-low-thinking

trace_processing:
  observation_format: openai_messages

base.teacher_model_name is also the model that relabels your traces, unless trace_processing.teacher_model_name names a different one, so it matters from the first job. The large GLM 5.3 teacher, zai.glm-5.3-low-thinking, is the recommended choice (Supported models). Scores read by two different judges are not comparable, so the judge is usually set once for a project. Every other field has a default: Relabeling and synthetic data covers the ones worth setting, and Config file has the full table.

Create the dataset (Dataset overview), then build the test set.