Train a Model from 1% of Production Traffic
You do not need to export your logs. distil labs can collect traces from your live traffic and turn them into training material, with nothing for you to write or label. It is the lowest-effort of the three data routes. Arranging it starts with a conversation: contact us.
Why is a small share of traffic enough?
Because traffic is never used as training data directly — it is used as seed and context for synthetic generation, and generation amplifies. A modest number of real interactions is enough to teach a teacher model what your domain looks like; the teacher then writes the volume.
Our own benchmarking on trace-derived training makes the amplification concrete: roughly 2,000 multi-turn conversations were generated from as few as 5 to 40 seed traces, and in the deliberately data-starved scenario a student seeded with just five traces scored 0.852 against 0.649 for direct training on the same five. That test used Google Research’s Schema-Guided Dialogue corpus as a controlled stand-in for production logs; details in why training on production traces fails.
Note that “1%” here is shorthand for “a small share”. distil labs does not publish a sampling rate, and this page does not invent one.
What problem does this solve?
The export problem. Most teams running an LLM feature already have the data they need, sitting in an observability tool or a log bucket, and the thing standing between them and a fine-tuned model is a week of pipeline work to get it out in a usable shape.
| Cost | Path 1 (collected for you) | Path 2 (you upload traces) | Path 3 (you write examples) |
|---|---|---|---|
| Export pipeline to build | None | One-off | None |
| Examples to label | None | None | 20+ |
| Job description to write | Discussed with us | Yes | Yes |
| Requires a live LLM feature | Yes | Yes | No |
Path 2 is the same idea with the export work on your side, and it is fully documented. Path 3 suits teams with no production system yet — see fine-tune with 20 examples.
Why does a small model fit this shape of work?
Because traffic collection implies a bounded, repetitive task, and bounded repetitive tasks are exactly where specialisation beats scale. If a feature handles enough requests for sampling to be meaningful, it is doing the same kind of thing over and over — which is the regime where a fine-tuned small model matches or beats a much larger general one.
The published example is an IoT smart-home function-calling agent trained from 1,107 production traces with no manual labelling. A 0.6B student reached 79.49% exact match against 50.0% for the 120B general-purpose teacher and 10.26% for the untuned 0.6B base. Full write-up: from production traces to a faster, cheaper, accurate model.
The economics follow from the size difference rather than from anything clever — the 10x inference tax you don’t have to pay works through the cost side across nine datasets.
What does a working setup look like downstream?
Once traces exist as a dataset, the pipeline is the ordinary documented one, and it is identical to every other route in:
- Traces are deduplicated and split into train and test seeds; the remainder is kept as unstructured context.
- Seed traces are scored for relevance and coherence against your task description, and weak ones are dropped.
- A committee of teacher models rewrites each surviving conversation into cleaner reference answers.
- A teacher generates and validates synthetic training data, in the manner Self-Instruct established.
- Teacher evaluation confirms the teacher can solve the task.
- The student is fine-tuned and evaluated against the held-out test set.
Because step 1 produces a test set from your own traffic, you get a baseline for the model that generated the traces — so the comparison against your current system is available before you commit to anything. Why every path ends in synthetic data covers steps 4 to 6 in more depth.
What is deliberately not described here?
The collection mechanism. There is no public documentation of how traffic is sampled, what the integration looks like, what is retained or for how long, or what the sampling rate is — so this page says none of those things rather than guessing at them. Anything you read elsewhere claiming to describe those mechanics is not sourced from distil labs documentation.
If you want the version you can implement today without talking to anyone, that is path 2: export your own logs and upload them.
How do you start?
Get in touch and describe the task. The useful things to bring to that conversation are the same ones every route needs: what the model should do, what a correct output looks like, and roughly what volume the feature handles.
If you would rather evaluate the idea before talking to anyone, the two prior questions are is fine-tuning worth it and what size model do you need. Both are answerable from the corpus without any data at all.