Turn Production Traces Into Training Data

Export your logs to traces.jsonl, add a job description and a config, then run distil traces upload followed by distil upload create-from-traces. The platform deduplicates, filters, relabels and validates them into a train/test split you can fine-tune on. Training from traces is an experimental feature.

What you need

Three files, in one directory, plus an optional fourth.

File	Format	Required	Purpose
traces.jsonl	JSONL	Yes	One production trace per line
job_description.json	JSON	Yes	What the model should do
config.yaml	YAML	Yes	Task, student model, trace_processing parameters
test.jsonl	JSONL	No	Your own curated test set

If you omit test.jsonl, the platform builds one from your traces by running a slice of them through the same filtering and relabelling pipeline. Supplying your own is worth the effort — it is the only evaluation set you can fully vouch for.

Step 1: Export traces in a supported format

Each line of traces.jsonl is one conversation. The default observation_format is openai_messages: an object with a messages array following the OpenAI chat completion format, optionally with tools and response_format.

{"messages": [{"role": "system", "content": "You are a helpful banking assistant."}, {"role": "user", "content": "What is my account balance?"}, {"role": "assistant", "content": "Your current account balance is $1,234.56."}]}

If your observability stack is Langfuse, set observation_format: langfuse and export observation objects with id, input, and optionally output and metadata instead. Trace formats compared covers the trade-off between the two.

Multi-turn conversations are preserved in full. A single exchange is simply a two-turn conversation, so you do not need to flatten anything.

Step 2: Write the job description

This is the normative half of the signal — it defines correct independently of whatever your current system happens to do. Minimum viable version is a single field.

{
  "task_description": "You are a helpful banking assistant that answers customer questions about their accounts, transactions, and banking products."
}

Task types that need more get more: classes_description for classification, tools for tool calling. The per-task fields are in the data preparation guides, and writing a job description for synthetic data covers how to write one that actually produces good generations.

Step 3: Set the trace processing config

Pick the task, the student, and a teacher.

base:
  task: classification
  student_model_name: Qwen3-1.7B
  teacher_model_name: zai.glm-5

synthgen:
  teacher_temperature: 0.6

trace_processing:
  observation_format: openai_messages
  relabel: true

Two things to know. Reasoning teachers — the GLM, Kimi, MiniMax, DeepSeek and GPT OSS families — require synthgen.teacher_temperature between 0.5 and 0.7; anything outside that range is a validation error. And trace_processing has its own teacher_model_name, used for relevance filtering and for picking the best relabelled answer from the committee. Full parameter list in the config reference.

Step 4: Upload the trace files

distil traces upload --data ./traces
# Output: Prepared traces created. ID: <traces-id>

This only stores the files as a PreparedTraces resource. Nothing is processed yet, which is deliberate — it lets you reprocess later without re-uploading. You can also pass --traces, --job-description, --config and --test individually instead of using directory mode.

Step 5: Process them into an upload

distil upload create-from-traces <traces-id>
# Output: Processing started. Upload ID: <upload-id>

This is where the pipeline runs: deduplication and train/test seed splitting, relevance and coherence scoring against your job description, committee relabelling of the seed conversations, and validation of the rewritten output. Leftover traces beyond the seed budget become unstructured context for generation.

Processing typically takes several minutes.

Step 6: Check the baseline before you train

distil upload status <upload-id>
distil upload metrics <upload-id>

Because the platform evaluates the model that produced your traces against the new test set, upload metrics gives you the number your fine-tuned model has to beat. That baseline is the most useful artefact of this whole step, and it exists before you spend anything on training.

For per-example detail, distil upload download-traces-predictions <upload-id> writes out the base model’s predictions as JSONL.

Step 7: Train on the processed data

distil upload download <upload-id> --data-destination ./processed
distil model upload-data <model-id> --data ./processed
distil model run-teacher-evaluation <model-id>
distil model run-training <model-id>

upload download writes files under exactly the names directory mode expects, so the two commands chain. Run teacher evaluation before training — it tells you whether the teacher can solve the task while changing your mind is still cheap.

Verifying it worked

Three checks, in order.

First, processing succeeded at all: distil upload status reports failure explicitly, and the pipeline errors rather than proceeding if it produced fewer than 20 examples. Second, the splits look sane: download them and read a few rows, checking that the relabelled assistant turns say what you would have said. Third, the trained student beats the baseline from step 6 on the same test set.

If the student does not beat the baseline, the problem is usually upstream of training. Why did my fine-tuned model get worse and when does distillation fail are the two places to look.

Reprocessing without re-uploading

Because upload and processing are separate, you can retry with different parameters against the same stored traces:

distil traces list
distil upload create-from-traces <traces-id> --config ./config.yaml

Each run produces a new upload, so earlier attempts stay available for comparison. This is the cheapest way to test whether a looser min_relevance_score or a different relabelling committee helps — see how much traffic before traces are useful.