Training Data for Small Language Models

Three ways to get training data for a task-specific small language model — from your live traffic, from production traces you already have, or from a minimal hand-written dataset.

Every trained model starts with data, and the most common reason a small language model project stalls is a belief that there is not enough of it. In practice there are three routes, and they differ mainly in how much work falls on you.

We collect traces from your live traffic. The lowest-effort path. A small share of production traffic is logged, and those real interactions become the raw material for training. You write nothing and label nothing. This suits teams already running an LLM-powered feature who would rather not build an export pipeline to get at their own logs.

You upload traces you already have. If you are running an LLM feature today, its logs are a training set. Upload them and the platform deduplicates them, splits them into train and test seeds, scores each one for relevance and coherence against your task description, drops the weak ones, has a committee of teacher models rewrite the assistant turns into cleaner reference answers, and validates the result. Anything left over is kept as unstructured context.

You upload a minimal dataset. With no production system yet, 20 to 100 hand-written examples plus a task description are enough to start. This is the path most new projects take, and it is the reason the platform exists — the alternative was labelling thousands of rows by hand.

All three converge

Whichever route you take, the next steps are identical: a teacher model generates and validates synthetic training data from your seeds, teacher evaluation confirms the teacher can actually solve the task, and then the student is trained on the result.

This is worth stating plainly because it changes how you should think about the choice. The path determines where the seed examples come from. It does not determine the quality ceiling of the final model. A well-written 30-example dataset can produce a better model than ten thousand noisy traces, because the synthetic generation step amplifies whatever signal — or noise — the seeds contain.

What “enough data” means

Less than most teams assume. The platform is built around the case where you have tens of examples rather than thousands, and the synthetic generation step exists precisely to close that gap. Twenty good examples that span the real variety of your inputs beat two hundred that all look alike.

The failure mode is not too little data. It is unrepresentative data — seeds that cover the easy centre of your distribution and none of the edges where the model will actually be tested.

Traces or hand-written?

If you have production traces, they carry information a hand-written set usually does not: the actual distribution of what users ask, including the awkward inputs nobody would think to invent. Our benchmarking on this compares the two directly, and the result is not a blanket win for either — it depends on how clean the traces are.

Where this cluster goes

The articles below cover all three paths in detail, what makes a trace worth training on, the supported trace formats, how to write a job description that produces good synthetic data, and how many examples you actually need.

How Many Examples Do You Actually Need to Fine-Tune a Model?

Twenty labelled examples is the documented floor and 20 to 100 the usual range — here are the published results behind those figures and what moves the number in either direction.

How Much Traffic Do You Need Before Traces Are Useful?

There is no published traffic threshold for trace-based training. This works out what the documented processing pipeline — dedup, seed splitting, relevance filtering, relabelling — implies about volume instead.

Trace Formats: OpenAI Messages vs Langfuse

A field-by-field comparison of the two trace formats accepted by distil labs, when to pick each, and the two additional observation formats for images and unstructured context.

Traces vs Synthetic Data: Which Trains a Better Model?

Benchmark results comparing a student fine-tuned directly on production traces against one trained on synthetic data generated from the same traces, across five corruption scenarios.

Fine-Tune a Model With 20 Examples

The minimal-dataset route end to end: four files, twenty labelled examples, and the distil CLI commands that turn them into a trained small language model.

Three Ways to Get Training Data for a Small Language Model

A decision guide to the three routes for getting training data onto the distil labs platform — live traffic collection, uploading existing traces, or a minimal hand-written dataset.

Turn Production Traces Into Training Data

A step-by-step recipe for converting logs from a live LLM feature into a training and test set, using distil traces upload and distil upload create-from-traces.

Writing a Job Description for Synthetic Data Generation

How to write the job_description.json that defines correct behaviour for your task — the field that carries the normative signal for every generated training example.

Fine-Tune with Synthetic Data: Generate Training Data from a Prompt

Learn how to use synthetic data generation to create high-quality training datasets for fine-tuning small language models — even when you have little or no labeled data.

Generate Synthetic Training Data for LLM Fine-Tuning

Learn how to generate high-quality synthetic training data using a teacher LLM to fine-tune smaller, faster models — even when you have little or no labeled data to start with.

Train a Model from 1% of Production Traffic

The lowest-effort route to a task-specific small model: let a small share of your live traffic be collected as training material, with nothing for you to write or label.

What Makes a Good Production Trace?

What the trace processing pipeline scores a conversation on — relevance, coherence, and repairability — and why a trace can be a poor training label yet still be excellent seed data.

Why Every Path to Training Data Ends in Synthetic Data

However you supply seed data — live traffic, uploaded traces, or a hand-written set — the platform converts it into synthetic training data before any fine-tuning happens. Here is why.