Fine-Tune a Model With 20 Examples

Write a job description, twenty labelled examples in train.jsonl, a held-out test.jsonl, and a config.yaml. Upload the directory, run teacher evaluation, run training. The twenty examples are seeds — a teacher generates thousands more from them before the student ever sees a gradient.

What you need

Four files in one directory, plus one optional fifth.

File	Format	Required	Description
job_description.json	JSON	Yes	What the model should do
train.jsonl	JSONL	Yes	20+ labelled examples, each a messages conversation
test.jsonl	JSONL	Yes	Held-out evaluation set
config.yaml	YAML	Yes	Task and training parameters
unstructured.jsonl	JSONL	No	Domain text to keep generation on-domain

No GPU, no labelling platform, no ML pipeline. The reason twenty is a workable floor is that the examples are not the training set — they anchor synthetic generation, in the manner Self-Instruct established for instruction data.

Step 1: Pick the task type

The task determines the file formats, the validators, and which models are eligible. Six are supported: question-answering, classification, tool-calling-closed-book, multi-turn-tool-calling-closed-book, question-answering-open-book for RAG, and question-answering-closed-book.

Get this right before writing anything, because the example format differs per task. Task selection walks through the choice.

Step 2: Write the job description

Plain English, in a JSON file. Your existing LLM prompt is a good starting point.

{
  "task_description": "Extract the requested information from the provided invoice text. Return only the specific value asked for, without additional explanation. If the information is not found, respond with 'Not found'.",
  "input_description": "Invoice text containing details such as invoice number, date, vendor name, line items, subtotal, tax, and total amount."
}

Task-specific fields go here too: classes_description for classification, tools for tool calling, and an optional llm_as_a_judge_instructions for how answers should be scored. Writing a job description for synthetic data covers what separates a good one from a vague one.

Step 3: Write twenty training examples

Each line of train.jsonl is a messages conversation: a user turn holding the input, an assistant turn holding the expected output.

{"messages": [{"role": "user", "content": "Invoice #1234 from Acme Corp dated 2024-01-15. Total: $540. What is the total amount?"}, {"role": "assistant", "content": "$540"}]}
{"messages": [{"role": "user", "content": "Invoice #1234 from Acme Corp dated 2024-01-15. Total: $540. Who is the vendor?"}, {"role": "assistant", "content": "Acme Corp"}]}

Spend your effort on coverage, not volume. Twenty examples that span the real variety of your inputs — including the awkward cases, the ambiguous ones, and the ones where the right answer is “not found” — beat two hundred that all look alike. Because generation amplifies whatever is in the seeds, a narrow seed set produces a narrow synthetic set.

Every seed must be correct. Errors here are amplified too. This is the same finding LIMA reported at a larger scale, where a carefully curated set of 1,000 prompts outperformed far larger but noisier alternatives.

Step 4: Hold out a test set

test.jsonl uses the identical format and must contain examples that are not in train.jsonl. This is the set every downstream number is reported against: the teacher’s score, the untuned student’s score, and your trained model’s score.

If you only have twenty examples total, split them rather than reusing any. A test set contaminated with training data will tell you the model is excellent right up until it reaches production.

Step 5: Add unstructured context if you have it

Optional, and cheap when available. unstructured.jsonl is a single context field per line — documentation, unlabelled examples, support tickets, industry literature.

{"context": "Invoice #9012 from Tech Solutions Inc dated 2024-03-10. Items: Software License x1 at $299. Subtotal: $299. Tax: $24. Total: $323."}

The teacher samples from these while generating, which keeps synthetic examples in your domain’s vocabulary rather than in generic English. If you have unlabelled data lying around, this is the place to spend it — no labelling required.

Step 6: Write the config

base:
  task: question-answering
  student_model_name: Qwen3-1.7B
  teacher_model_name: zai.glm-5

synthgen:
  teacher_temperature: 0.6
  generation_target: 5000

Qwen3-1.7B is a reasonable default student; move up only if the metrics fall short. Reasoning teachers — GLM, Kimi, MiniMax, DeepSeek, GPT OSS — require teacher_temperature between 0.5 and 0.7, and a value outside that range is a validation error. Everything else has a documented default in the config reference; which teacher model should you pick covers the teacher choice.

Step 7: Upload, evaluate the teacher, train

distil model create my-first-model
# Output: Model created with ID: <model-id>

distil model upload-data <model-id> --data ./data
distil model run-teacher-evaluation <model-id>
distil model teacher-evaluation <model-id>
distil model run-training <model-id>

Do not skip the teacher evaluation. It answers one question — can a large model solve this task at all, given your description and examples — while fixing the answer is still a text edit rather than a retrain. If the teacher scores poorly, the student will too. Teacher evaluation explains how to read the result.

How to tell it worked

Compare three numbers on the same held-out test set: the teacher’s score, the untuned student’s score, and your trained student’s. A successful run puts the trained student near or above the teacher, and well above the untuned baseline.

For reference, a published run on policy-aware PII redaction started from 50 seed examples split between train and test and finished with a trained Llama-3.2-3B at 0.87 LLM-as-a-judge, against 0.85 for the Llama-3.3-70B teacher, 0.73 for a student trained on the seed data alone, and 0.54 for the untuned base — small expert agents from 10 examples has the detail.

What to change if it did not

If the teacher scored badly, the job description is the first suspect, not the model. If the teacher scored well but the student did not, look at seed diversity and at why did my fine-tuned model get worse. If you are unsure twenty is enough for your task, how many examples do you actually need and few-shot fine-tuning both go deeper.