Question answering data preparation

The example on this page extracts information from invoices: the input carries the invoice text and the question, and the model returns the value asked for.

The shared rules for the input directory, row shapes and validation are in Overview. This page covers what’s specific to question answering.

Job description

What you expect the model to do, in the words you’d use to prompt an LLM.

• task_description. What the model should do and how it should format its output.
• synthetic_data_generation_instructions. Optional. What the generated inputs should look like: their format, the fields they carry, how much they vary. It’s read during generation and nowhere else, so it changes the inputs without redefining the task.
• llm_as_a_judge_instructions. Optional, but it does real work here: this task is scored by an LLM judge, so these instructions decide what counts as correct. The primary metric’s judge sees the input and the answer but not the reference answer, so state what a correct answer looks like, not how it compares to a reference.

Any document or retrieved text the model needs goes inside the user message, as the invoice text does below.

{
    "task_description": "Extract the requested information from the provided invoice text. Return only the specific value asked for, without additional explanation. If the information is not found, respond with 'Not found'.",
    "synthetic_data_generation_instructions": "Each input is one invoice text followed by a single question about that invoice. The invoice text carries an invoice number, a date, a vendor name, line items, a subtotal, tax and a total amount. Vary the vendors, the number of line items and the field the question asks for, and include invoices where the requested value is absent.",
    "llm_as_a_judge_instructions": "Output 'good' if the answer is exactly the value the question asks for, as written in the invoice text, with no extra explanation, or 'Not found' when the invoice does not contain that value. Otherwise output 'bad'."
}

Training data

Each row is a messages conversation: a user turn holding the input text and the question, and an assistant turn holding the expected answer.

Aim for 20+ diverse rows. The teacher generates thousands more from these.

{"messages": [{"role": "user", "content": "Invoice #1234 from Acme Corp dated 2024-01-15. Items: Widget x10 at $50 each. Subtotal: $500. Tax: $40. Total: $540. What is the total amount?"}, {"role": "assistant", "content": "$540"}]}
{"messages": [{"role": "user", "content": "Invoice #1234 from Acme Corp dated 2024-01-15. Items: Widget x10 at $50 each. Subtotal: $500. Tax: $40. Total: $540. What is the invoice number?"}, {"role": "assistant", "content": "1234"}]}
{"messages": [{"role": "user", "content": "Invoice #1234 from Acme Corp dated 2024-01-15. Items: Widget x10 at $50 each. Subtotal: $500. Tax: $40. Total: $540. Who is the vendor?"}, {"role": "assistant", "content": "Acme Corp"}]}
{"messages": [{"role": "user", "content": "Invoice #5678 from Global Services dated 2024-02-20. Items: Consulting 8hrs at $150/hr. Subtotal: $1200. Tax: $0. Total: $1200. What is the invoice date?"}, {"role": "assistant", "content": "2024-02-20"}]}
{"messages": [{"role": "user", "content": "Invoice #5678 from Global Services dated 2024-02-20. Items: Consulting 8hrs at $150/hr. Subtotal: $1200. Tax: $0. Total: $1200. What is the tax amount?"}, {"role": "assistant", "content": "$0"}]}

Test data

Same format as train.jsonl, held out for evaluation. This is what every score you’ll see is measured against: the teacher evaluation, the base student and the tuned student.

No row may be identical to a training row, or validation fails. Use different invoices rather than different questions about the same ones, so the test set measures generalization instead of recall.

Cover the awkward cases too. The example below includes a value that isn’t present, which is what tells you whether the model follows the “respond with ‘Not found’” rule.

{"messages": [{"role": "user", "content": "Invoice #4471 from Northwind Traders dated 2024-05-02. Items: Freight x1 at $220. Subtotal: $220. Tax: $17.60. Total: $237.60. What is the total amount?"}, {"role": "assistant", "content": "$237.60"}]}
{"messages": [{"role": "user", "content": "Invoice #4471 from Northwind Traders dated 2024-05-02. Items: Freight x1 at $220. Subtotal: $220. Tax: $17.60. Total: $237.60. Who is the vendor?"}, {"role": "assistant", "content": "Northwind Traders"}]}
{"messages": [{"role": "user", "content": "Invoice #8802 from Bright Labs dated 2024-06-19. Items: Assay kit x3 at $410 each. Subtotal: $1230. Tax: $98.40. Total: $1328.40. What is the invoice date?"}, {"role": "assistant", "content": "2024-06-19"}]}
{"messages": [{"role": "user", "content": "Invoice #8802 from Bright Labs dated 2024-06-19. Items: Assay kit x3 at $410 each. Subtotal: $1230. Tax: $98.40. Total: $1328.40. What is the purchase order number?"}, {"role": "assistant", "content": "Not found"}]}
{"messages": [{"role": "user", "content": "Invoice #1590 from Harbour Print dated 2024-07-08. Items: Poster run x500 at $1.20 each. Subtotal: $600. Tax: $0. Total: $600. What is the tax amount?"}, {"role": "assistant", "content": "$0"}]}

Traces as context (optional)

Production traces in the dataset give the teacher in-domain text to write realistic inputs from. Put them in traces.jsonl next to the labeled rows (Traces): the relabeling runs turn some into test and train rows, and generation uses the rest as context (The expand operations).

Config

The task type, the two models and the judge:

base:
  task: question-answering
  student_model_name: Qwen3.5-4B
  teacher_model_name: zai.glm-5.3-low-thinking
evaluation:
  llm_as_a_judge_model_name: zai.glm-5.3-low-thinking

When every answer must be valid JSON, add synthgen.output_is_json: true; every assistant answer in your data must then parse as JSON. Every other field has a default. See Config file for the full table and Supported models for the values you can use.

Next

Put the files in one directory, here input-dir, and create the dataset from it. This validates your files at the same time:

distil dataset create --data input-dir

The command prints Upload successful. Dataset ID: <dataset-id>. Run teacher evaluation on that ID next.