Trace processing

Trace processing turns a traces object into a seed dataset. Traces are filtered for relevance, relabeled by a teacher, and become the train split; the leftover traces become the rows of unstructured.jsonl. The test split is not built here: the job copies the test.jsonl of the traces object unchanged. Run Test set from traces first, or upload a curated test.jsonl with your traces. Without a test set the seed dataset has an empty test split, and teacher evaluation cannot run on it.

Configuring trace processing

The trace_processing section controls this job, and Test set from traces reads the same section for its relabeling, so a change here changes both. The full table is in Config file. These are the settings to choose deliberately:

• observation_format. Has to match the shape of your traces.jsonl. See Trace inputs.
• relabel (default true). Has the teacher rewrite the assistant answers. With relabel: false the production answers pass through unreviewed, and your student learns to imitate the model it is meant to beat. If relabeled answers come back worse than the originals, change the relabeling teacher or set relabelling_committee_models to relabel with a committee.
• relevance_filtering (default false). Set it true to have an LLM score each trace and drop the ones low on relevance or coherence. That costs an LLM pass over every trace, and min_relevance_score and min_coherence_score apply only when it is on.
• num_traces_as_training_base (default 200). How many traces are relabeled into the train split. The job refuses a trace set smaller than this.
• compress_job_description. Set it true when your task description is long enough to overwhelm the filtering model.

Trace jobs reject base.enable_thinking: true. For a reasoning student, see Reasoning models.

Two behaviors to expect:

• There is no floor on the train split. Filtering, relabeling and schema checks each drop traces, and whatever survives is written, an empty split included. A train split far below num_traces_as_training_base usually means the traces do not match the task type, for example multi-turn traces for a single-turn task. The job log counts each drop.
• The in-context exemplar counts can come out lower. No exemplar count may exceed the number of train rows (Cross-field validation), so this job lowers them to the size of the train split it produced and logs a warning. Read the counts from the processed config before relying on them.

Run a smoke first

A smoke run is a small run that checks the conversion and the settings before the full run. For trace processing, it runs on a small copy of your traces:

1. Get a local traces.jsonl: your trace file, or, for traces created directly from an endpoint, the endpoint’s records downloaded and converted (Endpoint records to traces).
2. Trim it to 50-100 lines, and set trace_processing.num_traces_as_training_base: 10 in a copy of config.yaml.
3. Upload the trimmed directory, here ./smoke-1, without test.jsonl. The upload prints the traces ID, which the processing job takes:

distil traces upload --data ./smoke-1
distil seed-dataset create-from-traces <traces-id>

The smoke spends one prepared_traces_post and one seed_datasets_from_prepared_traces_post credit.

Running it

Process the updated traces object from Test set from traces:

distil seed-dataset create-from-traces <updated-traces-id>

The command prints Processing started. Seed dataset ID: <seed-dataset-id>. Check the seed dataset’s status with that ID until it reads JOB_SUCCESS:

distil seed-dataset status --output json <seed-dataset-id> | jq -r .status

A run takes about 45 minutes and spends one seed_datasets_from_prepared_traces_post credit. See How the platform works.

Check the results

Download the seed dataset into ./processed:

distil seed-dataset download --destination ./processed <seed-dataset-id>

Check first how many traces reached train.jsonl. Near-total loss usually means the relevance filter and the job description disagree about what the task is. Then check:

1. Correctness. The rows are valid examples of your task in the right format, and the relabeled answers are correct and better than the originals where they differ.
2. Distribution. The processed rows match your traces on the dimensions that matter for your task: length, topic coverage, style. Watch for filtering that dropped a whole category.
3. The test set. test.jsonl is the test set you reviewed in Test set from traces.

Iterating

Changing a setting is an override on the same traces object, so nothing is uploaded again. Download the traces object’s config.yaml and job_description.json into ./iter-2:

distil traces download-metadata -d ./iter-2 <updated-traces-id>

Change one field in ./iter-2/config.yaml, for example trace_processing.relevance_filtering: true, and process the same traces object with the edited config:

distil seed-dataset create-from-traces --config ./iter-2/config.yaml <updated-traces-id>

An override replaces config.yaml whole, and anything you leave out takes the default rather than the value the parent used. Read the parent’s config back, edit one field, and send the whole file. See How the platform works. A different set of traces is a new upload.

Next

Teacher evaluation on the seed dataset.