Building the test set

The test set is the measure of everything that follows: the teacher evaluation, the production model’s baseline and every trained student are read on it. So it is built once, reviewed, and then carried unchanged through every dataset after it. The expands that build the train set copy test.jsonl through, so any dataset in the chain from here on has the same test set, and scores along the chain are comparable.

Two operations fill it, in this order:

1. Relabel traces into the test split. Real production inputs with reference answers written by the teacher. The stronger signal; use it for as many rows as your traces allow.
2. Generate synthetic test rows. Rows the teacher writes from the relabeled ones, the job description and the traces as context. They measure agreement with the teacher rather than with production, so they top the set up when the traces don’t reach the target.

Both are the expand operations described on Relabeling and synthetic data; this page is about the outcome.

If you uploaded a test set you consider complete, skip to Review the test set.

How many rows

Aim for a few hundred rows, between 100 and 1,000. More rows narrow the run-to-run noise of the scores (Variation between runs) and cover more of production; past a thousand the gain is small and the relabeling cost isn’t.

Then split the target between the two operations:

• trace_processing.num_test_relabelled is the number of traces relabeled into the test split. Default 200. It can’t exceed the distinct traces in the dataset, and every trace it uses is gone for the train set, so leave enough: the train set wants num_train_relabelled traces plus some context for generation (Building the train set).
• synthgen.test_generation_target is the number of synthetic test rows. Default 0; set it to the rest of your target when the relabeled rows and the uploaded ones don’t reach it. A generation run of T rows also takes T + min(T, 1000) traces as context, or all that are left (The expand operations); set synthgen.use_traces_as_context: false for this run when the traces are scarce.

For classification, the relabeled rows must cover every class or the expand fails validation; plan synthetic rows for the rare classes.

Relabel traces into the test split

Download the dataset’s config, set num_test_relabelled, and run the expand. The command prints the new dataset’s ID:

distil dataset download-metadata -d test-set <dataset-id>
# in test-set/config.yaml: trace_processing.num_test_relabelled: 300
distil dataset relabel-traces-test --config test-set/config.yaml <dataset-id>

Poll the new dataset until it reads Success:

distil dataset status <test-dataset-id>

The relabeling teacher writes the reference answers, which define what every score measures. It is base.teacher_model_name unless trace_processing.teacher_model_name names another; the large GLM 5.3 teacher is the default choice. --smoke relabels 128 traces as a side branch to check the settings first (Smoke runs); with a target near 128, run the full job directly. A run spends a datasets_from_datasets_relabel_traces_post credit, a smoke a datasets_from_datasets_smoke_post credit.

Generate synthetic test rows

When the plan has synthetic rows, run generation on the dataset relabeling produced, with synthgen.test_generation_target set:

distil dataset download-metadata -d test-set <test-dataset-id>
# in test-set/config.yaml: synthgen.test_generation_target: 200
distil dataset generate-synthetic-data-test --config test-set/config.yaml <test-dataset-id>

Each run is a new dataset; this one’s parent is the relabeled one. A run spends a datasets_from_datasets_generate_synthetic_data_post credit.

Review the test set

Download the last dataset and read its test.jsonl: the uploaded rows, then the relabeled traces, then the synthetic rows. Download the parent too to tell them apart. Check:

distil dataset download -d test-set/output <test-dataset-id>

• Size: the row count against your target. A large loss on the relabeling run means filtering or schema checks dropped traces; distil dataset logs has the cause.
• Correctness: the reference answers are correct against your job description, and better than the original answers where they differ. Read traces and rows side by side.
• Coverage: the rows cover the topics, lengths and classes your production traffic has. Read the synthetic rows with more care than the relabeled ones.

To change the settings, run the expand again on its parent; the earlier result is a side branch you leave behind. To edit rows by hand, download the dataset, edit test.jsonl, and upload the whole directory as a new dataset with distil dataset create --data. The traces come along, so the work continues from the upload, but the platform records no parent for an upload: note the jump yourself.

Lock it

The approved dataset’s ID is the test-set dataset: the input of teacher evaluation, and the dataset the train set is built on. Its test.jsonl is the project’s test set from here on. Scores measured on different test sets are not comparable, so when the test rows change later, every earlier score is stale: re-run the teacher evaluations on the new test set before reading any verdict (Model iterations).

What can be changed

• More relabeled rows (num_test_relabelled, with more traces). A larger test set makes smaller differences between iterations readable.
• Synthetic rows (test_generation_target, mutators on the test run). Rows beyond what the traces hold, measuring agreement with the teacher rather than with production.
• The reference answers (the relabeling teacher or committee, trace_processing_instructions). Who writes them and with what guidance. They define what every score measures.
• New traces. A test set can only measure what the traces contain. Traces that show a failure you care about, relabeled into the test split, add it to the test set.

Next

Teacher evaluation on the test-set dataset.