Skip to content

Migrating old entities

A seed dataset (train.jsonl, test.jsonl, optional unstructured.jsonl) and a training dataset (train.jsonl, test.jsonl) are readable with distil seed-dataset and distil training-dataset list, show, download and download-metadata; their other commands, and distil traces expand-test-set, exit 1 naming the command to use, and the REST API answers 410 on those routes. To build on one, move its files into a dataset:

  1. Download it. distil seed-dataset download -d old <id> or distil training-dataset download -d old <id>; both are free. For an old traces object, distil traces download -d old <id> gives its traces.jsonl only; the config, job description and test set it was uploaded with aren’t downloadable, so take them from your own files.
  2. Fix what a dataset rejects.
    • Flat rows ({"question": ..., "answer": ...}): rewrite every row as a messages conversation (Dataset overview).
    • unstructured.jsonl: if trace processing wrote it (each row is one serialized trace), rename it to traces.jsonl and set trace_processing.observation_format: unstructured_with_openai_messages in the config. If it holds uploaded documents, leave it out: a dataset has no place for context documents.
    • final-synthetic-dataset/: move its train.jsonl and test.jsonl to the top level.
    • Task types a dataset does not accept (question-answering-open-book, question-answering-closed-book, information-extraction, question-answering-open-book-synthetic-context, question-answering-legacy): set base.task: question-answering and drop context from the rows (Task selection). A multi-turn tool-calling task maps to chat-completion or chat-completion-agentic.
    • A question-answering job description with text in input_description or context_description: remove the two keys.
    • Deprecated config keys: replace or delete them (Config file).
  3. Upload the directory: distil dataset create --data old (Dataset overview). The create validates it and names the first row that fails.

The new dataset has no parent: it is the root of a new chain. A test set from an old entity is a locked test set like any other (Building the test set), and a train set from an old training dataset is a train set to top up (Building the train set).