Trace processing

Trace processing turns raw production logs into a training-ready seed dataset. Traces are filtered for relevance, relabelled by a teacher, and split into train, test and unstructured context. The original production model is also evaluated on the generated test set, which gives you the baseline your trained student has to beat.

This stage is optional. It exists to produce a seed dataset, so if you already have labelled data you can skip it and go to Starting from a dataset.

Running it

Two commands. The first stages your files, the second processes them.

# 1. Upload the trace directory
distil traces upload --data ./traces-input
# Traces uploaded with ID: <traces-id>

# 2. Process them into a seed dataset
distil seed-dataset create-from-traces <traces-id>
# Seed dataset created with ID: <seed-dataset-id>

# 3. Poll until it finishes, up to 45 minutes
distil seed-dataset status --output json <seed-dataset-id> | jq -r .status

A run spends one prepared_traces_post credit and one seed_datasets_from_prepared_traces_post credit. See How the platform works.

Configuring trace processing

The full parameter table is in Config file. These are the ones to set deliberately.

• observation_format. Has to match the shape of your traces.jsonl. See Trace formats.
• relabel (default true). Has the teacher rewrite the assistant answers. Leave it on. relabel: false reduces this stage to filtering and splitting, so the original production answers pass through unreviewed and your student learns to imitate the model it’s meant to beat. If relabelled answers come back worse than the originals, fix the relabelling teacher or set relabelling_committee_models to use a committee.
• relevance_filtering (default false). Every trace flows straight through unless you turn this on. Set it true to have an LLM score traces and drop the low relevance or coherence ones. That costs an LLM pass over every trace, and min_relevance_score / min_coherence_score only apply once it’s on.
• num_traces_as_training_base / num_traces_as_testing_base (defaults 200/200). Seed the two splits; the testing base has to be at least 1. Equal counts are a reasonable default. Weight the testing base higher when you want a larger test set than training set. Leftover traces become unstructured context.
• min_generated_examples (default 1). A floor checked per split, separately for train and test, after filtering and relabelling have already dropped a fraction of the traces. Its ceiling is therefore the smaller of the two base counts, and well below that in practice. A supplied test.jsonl is rejected up front if it has fewer rows than this.
• evaluate_original_model (default true). Produces the baseline your student has to beat. It’s also this stage’s judge cost.
• compress_job_description. Set true if your task description is long enough to overwhelm the filtering model.

Check the results before moving on

Pull the processed output and look at it, because everything downstream is built on this.

# The processed train/test data, config and job description
distil seed-dataset download --destination ./processed <seed-dataset-id>

# The original model's score on the generated test set
distil seed-dataset metrics --output json <seed-dataset-id> | jq .base_model_performance

Check the survival rate first: how many traces made it through filtering and relabelling. Near-total loss usually means the relevance filter and the job description disagree about what the task is.

Then check two things:

1. Correctness. Are the rows valid examples of your task, in the right format? Are the relabelled answers actually correct, and better than the originals where they differ?
2. Distribution. Compare the processed examples against the incoming traces along dimensions that matter for your task: length, topic coverage, style. Watch for filtering that silently dropped a whole category.

Review the generated test set closely while you’re here. It’s what every downstream number is measured against, from teacher evaluation to the training results, so check its size, its label and length distribution, and whether it covers your edge cases. If it isn’t trustworthy, supply a curated test.jsonl and process again rather than discovering the problem after a training run.

Iterating

Changing a setting is an override on the same uploaded traces, so nothing re-uploads:

# Read back the config, edit one field, submit again
distil traces download-metadata -d ./iter-2 <traces-id>
# in ./iter-2/config.yaml, under trace_processing:
#   relevance_filtering: true

distil seed-dataset create-from-traces --config ./iter-2/config.yaml <traces-id>

Changing the traces themselves means distil traces upload again, because an uploaded trace set owns the file it was staged with.

An override replaces config.yaml whole, and anything you omit reverts to the library default rather than the value the parent used. So always read the parent’s config back, edit one field, and send the whole file. See How the platform works. Uploaded traces are the one exception to how these read back: they echo the config you staged them with, comments intact and nothing expanded.

Next

The processed output is a valid seed dataset, so continue to Teacher evaluation.