Train an SLM for Support Ticket Triage
Triage is a classification task, and the model is only ever as good as the taxonomy it is trained on. Spend your first day on the class definitions and your second on the CLI — a taxonomy two humans cannot apply consistently will not produce a model that can.
The taxonomy is the model
Most triage projects fail before any training runs, because the queue names that grew organically in a ticketing system are not a taxonomy. They overlap, some are workflow states rather than categories, and several exist because one team lead created them.
Three properties make a taxonomy trainable:
- Mutually exclusive. Every ticket has exactly one correct queue. “Billing” and “Refunds” overlap; “Billing — refund request” and “Billing — invoice query” do not.
- Decidable from the ticket text alone. If assigning a queue requires opening the customer’s account, the model cannot learn it from text and neither can the teacher.
- Balanced enough to seed. A class you cannot find twenty examples of is a class you should fold into its neighbour.
Run the cheap test first: take fifty resolved tickets, have two people label them independently, and measure how often they agree. Below roughly 90% agreement, fix the definitions rather than proceed. That disagreement is the ceiling on any model you train.
Step 1: Write the taxonomy as class descriptions
job_description.json carries the whole taxonomy. Each entry in classes_description is a definition the teacher reads when fabricating synthetic tickets, so write it for a new hire, not for a slide.
{
"task_description": "Assign each inbound support ticket to exactly one triage queue.",
"classes_description": {
"billing_refund_request": "The customer asks for money back for a completed charge — refunds, chargebacks, duplicate payments. Not questions about what a charge was for.",
"billing_invoice_query": "The customer asks what a charge was for, or wants a copy or correction of an invoice. No money is being requested back.",
"account_access": "The customer cannot sign in: password resets, MFA lockouts, SSO failures, disabled accounts."
}
}Notice the negative clauses. “Not questions about what a charge was for” does more work than three extra examples, because it tells the teacher where the boundary runs. The classification data preparation guide has the full file spec.
Step 2: Seed the confusable pairs, not the obvious ones
Twenty examples per class is enough to start, but which twenty decides the outcome. Obvious tickets teach the model nothing it will not infer from the class description. The tickets sitting on a boundary are the entire value of your seed set.
For each pair of classes that a human hesitated over during your agreement test, add two or three examples on each side. Keep the label strings byte-identical to the classes_description keys.
{"messages": [{"role": "user", "content": "I was charged twice for the same order in March, please sort this out"}, {"role": "assistant", "content": "billing_refund_request"}]}
{"messages": [{"role": "user", "content": "There's a £14 line on my invoice I don't recognise, what is it?"}, {"role": "assistant", "content": "billing_invoice_query"}]}Add an unstructured.jsonl of unlabelled real tickets if you have them. It costs nothing and it teaches the teacher how your customers actually write — abbreviations, product nicknames, and all.
Step 3: Upload and check the teacher can apply your taxonomy
Teacher evaluation is the taxonomy audit you cannot skip. It scores a large model on your held-out tickets using your class descriptions, before you spend a training run.
distil model create support-ticket-triage
# Output: Model created with ID: <model-id>
distil model upload-data <model-id> --data ./data
distil model upload-status <model-id>
distil model run-teacher-evaluation <model-id>
distil model teacher-evaluation <model-id>Your config.yaml:
base:
task: classification
student_model_name: Qwen3-1.7B
teacher_model_name: zai.glm-5
synthgen:
teacher_temperature: 0.6
match_generated_distribution_to_seed: truezai.glm-5 is a reasoning teacher, so the temperature must sit between 0.5 and 0.7. match_generated_distribution_to_seed keeps the synthetic set from over-generating your easy classes. Qwen3-1.7B is a reasonable default student for a taxonomy of a few dozen classes; drop to 0.6B if the queue count is small and latency matters.
If the teacher scores badly, download its mistakes and read them: distil model download-teacher-evaluation-predictions <model-id>. A teacher that confuses two classes is telling you those two classes are one class.
Step 4: Train, then read the per-class errors
distil model run-training <model-id>
distil model training <model-id>
distil model download-training-predictions <model-id>The aggregate accuracy is the least useful number in the file. Group the predictions by true label and look at what the errors have in common:
| Error pattern | What it means | Fix |
|---|---|---|
| Two classes swapped symmetrically | The boundary is undefined | Add a negative clause to both descriptions |
| One class absorbs its neighbours | Its description is too broad | Narrow it, or merge the neighbours into it |
| One rare class is never predicted | Too few seeds, skewed generation | Add seeds; set match_generated_distribution_to_seed |
| Errors spread evenly, no pattern | Genuinely hard task | Try a larger student before touching the taxonomy |
Fix the description, add four seed tickets on the boundary, retrain.
For scale: our email labelling model trained Qwen3-0.6B on 10 classes from 154 seed examples expanded to 10,000 synthetic ones, and moved from 38% for the base model to 93%, matching its teacher. That run used openai.gpt-oss-120b, the teacher it actually set.
Step 5: Deploy and route
distil model deploy local <model-id>
distil model invoke <model-id>deploy local needs llama-cpp installed and serves an OpenAI-compatible API on port 8000 (--port to change it). invoke prints a ready-to-run client command. For production options see local deployment.
Wire the label straight into your routing rules. Because the output is a bare label string, there is nothing to parse — which is the practical reason to use classification rather than prompting a general model for a sentence.
When triage should not be one classifier
If your queue assignment depends on the customer’s plan tier or open order state, that is a business rule, not a classification. Predict the topic from text and apply the rule in code — mixing them produces a model that is wrong whenever the account data changes.
If triage has to extract a value as well as pick a queue, you want tool calling instead. And if you are choosing a student, Qwen3 1.7B vs Gemma 3 1B for classification covers the trade-off; what is text classification covers the task type itself.