Three Ways to Get Training Data for a Small Language Model
There are three. distil labs collects traces from your live traffic, you upload traces you already have, or you upload 20 to 100 hand-written examples. They differ only in where the seed data comes from. All three then run the same pipeline: synthetic generation, teacher evaluation, training.
Which path should you pick?
Pick on effort and on what you already have, not on expected model quality — the quality ceiling is set downstream, not by the route in.
| Path 1: we collect from live traffic | Path 2: you upload traces | Path 3: you upload a minimal dataset | |
|---|---|---|---|
| Effort on you | Lowest | Medium | Highest |
| You write | Nothing | A job description | A job description and 20+ examples |
| You label | Nothing | Nothing | Every example |
| Prerequisite | An LLM feature in production | Exported logs from one | Only a clear task definition |
| Documented at | Contact us | Upload traces | Data preparation |
| Best when | You do not want to build an export pipeline | You already have logs on disk | There is no production system yet |
When does live-traffic collection make sense?
When you are already running an LLM-powered feature and would rather not build an export pipeline to get at your own logs. A small share of production traffic is collected and becomes the raw material for training. You write nothing and you label nothing, which makes it the lowest-effort route by a wide margin.
The mechanics of that collection are not documented publicly, and this corpus does not describe them. If the option interests you, get in touch — that is the honest answer rather than a guessed integration guide.
When should you upload traces you already have?
When your logs are already exported, or exporting them is a morning’s work. This is the fully documented version of path 1: you assemble a traces.jsonl file, a job description, and a config, and the platform runs its trace-processing pipeline over them.
Traces carry something a hand-written set usually does not — the real distribution of what your users ask, including the awkward inputs nobody would think to invent. Two formats are supported, plain OpenAI chat completion messages and Langfuse observation objects; see trace formats compared for which to use.
Start at turn production traces into training data.
When is a minimal hand-written dataset the right call?
When there is no production system yet, or when your traces are so noisy that curating 30 clean examples is faster than cleaning ten thousand dirty ones. The documented floor is 20 or more labelled examples in train.jsonl plus a held-out test.jsonl.
This is the path most new projects take. It is also the path with the best control: you decide exactly what correct looks like, rather than inferring it from what a previous model happened to do. See fine-tune with 20 examples.
What do all three have in common?
Everything after the seed data. Whichever route you take, a teacher model generates and validates synthetic training data from your seeds, teacher evaluation confirms the teacher can actually solve the task, and then the student is fine-tuned on the result.
This is the part worth internalising. The path determines where seeds come from; it does not determine the quality ceiling. The synthetic generation step — the same technique Self-Instruct popularised for bootstrapping instruction data — amplifies whatever signal, or noise, the seeds contain. A well-written 30-example set can beat ten thousand noisy traces. Why every path ends in synthetic data covers the convergence in detail.
What do teams get wrong here?
Four mistakes, in rough order of how often they show up.
| Mistake | Why it hurts | Fix |
|---|---|---|
| Training directly on raw traces | Bakes the old system’s errors into the weights | Use traces as seeds and context, not labels |
| Waiting to collect “enough” data | Delays a project that needed tens of examples, not thousands | See how many examples you actually need |
| Seeds that all look alike | Model handles the easy centre and fails at the edges | Deliberately include awkward cases |
| A vague job description | Vague seeds produce vague synthetic data | See writing a job description |
The failure mode is almost never too little data. It is unrepresentative data — seeds that cover the comfortable middle of your distribution and none of the edges where the model will actually be tested.
Can you mix paths?
Yes, and it is often the best move. Upload traces to get the real input distribution, then hand-write a small curated test.jsonl so your evaluation set is trustworthy even when the training seeds are not. The trace-processing pipeline will build a test set for you if you do not supply one, but a set you wrote yourself is the only one you can fully vouch for.
If you are still deciding whether a small model is the right target at all, is fine-tuning worth it and what size model do you need are the prior questions.