Closed book QA data preparation

The example on this page answers general-knowledge questions from the model’s own weights, with nothing supplied at inference time.

The shared rules for the input directory, row shapes and validation are in Overview. This page covers what’s specific to closed book QA.

Job description

What you expect the model to do, in the words you’d use to prompt an LLM.

task_description. The task itself. For closed book QA it can be very short, with extra context added only where you find it helps.
llm_as_a_judge_instructions. Optional, but it does real work here: this task is scored by an LLM judge, which sees the question, the reference answer and the prediction, and returns a binary good or bad. These instructions decide what counts as correct.

{
  "task_description": "The task is to answer the question using your internal knowledge",
  "llm_as_a_judge_instructions": "Evaluate whether the predicted answer correctly answers the question based on the reference answer. Output 'good' if the predicted answer is semantically equivalent to the reference answer or conveys the same key information, otherwise output 'bad'"
}

Training data

Each row is a messages conversation: a user turn holding the question, and an assistant turn holding the expected answer. These set the style of question and the length of answer the teacher generates from your corpus, so make them look like what you actually want.

Aim for 20+ diverse rows.

{"messages": [{"role": "user", "content": "Where did Sands and Chopin take shelter after the locals in Majorca became inhospitable upon discovering they were unmarried?"}, {"role": "assistant", "content": "a former Carthusian monastery"}]}
{"messages": [{"role": "user", "content": "When did the Computer Emergency Readiness Team, a division of the Department of Homeland Security, investigate 79 hacking incidents at energy companies?"}, {"role": "assistant", "content": "2014"}]}
{"messages": [{"role": "user", "content": "How is the final quarter of the Premier League's television rights revenue distributed?"}, {"role": "assistant", "content": "the final quarter is paid out as facilities fees for games that are shown on television, with the top clubs generally receiving the largest shares of this."}]}

Test data

Same format as train.jsonl, held out for evaluation. This is what every score you’ll see is measured against.

No row may be identical to a training row, or validation fails. Every answer here has to be supported by something in unstructured.jsonl, since that corpus is the only knowledge the model gets. A question whose answer isn’t in the corpus measures nothing except hallucination.

{"messages": [{"role": "user", "content": "Which piano manufacturer accompanied Chopin on his 1837 visit to London?"}, {"role": "assistant", "content": "Camille Pleyel"}]}
{"messages": [{"role": "user", "content": "How much of the Premier League's television money is divided equally between the clubs?"}, {"role": "assistant", "content": "half"}]}
{"messages": [{"role": "user", "content": "Which worm showed that equipment not connected to the Internet could still be damaged by malicious commands?"}, {"role": "assistant", "content": "Stuxnet"}]}
{"messages": [{"role": "user", "content": "How is the income from the Premier League's overseas television rights divided?"}, {"role": "assistant", "content": "equally between the twenty clubs"}]}

Unstructured data (required)

Unstructured data is the whole point of closed book QA. It’s how the knowledge gets into the model: the teacher generates question-answer pairs in the style of your training set, drawn from these contexts.

What it covers bounds what your model can answer, so coverage here matters more than anything else on this page. Rows carry a single context field.

{"context": "In June 1837 Chopin visited London incognito in the company of the piano manufacturer Camille Pleyel where he played at a musical soir\u00e9e at the house of English piano maker James Broadwood. On his return to Paris, his association with Sand began in earnest, and by the end of June 1838 they had become lovers. Sand, who was six years older than the composer, and who had had a series of lovers, wrote at this time: \"I must say I was confused and amazed at the effect this little creature had on me ... I have still not recovered from my astonishment, and if I were a proud person I should be feeling humiliated at having been carried away ...\" The two spent a miserable winter on Majorca (8 November 1838 to 13 February 1839), where, together with Sand's two children, they had journeyed in the hope of improving the health of Chopin and that of Sand's 15-year-old son Maurice, and also to escape the threats of Sand's former lover F\u00e9licien Mallefille. After discovering that the couple were not married, the deeply traditional Catholic people of Majorca became inhospitable, making accommodation difficult to find. This compelled the group to take lodgings in a former Carthusian monastery in Valldemossa, which gave little shelter from the cold winter weather."}
{"context": "The Premier League sells its television rights on a collective basis. This is in contrast to some other European Leagues, including La Liga, in which each club sells its rights individually, leading to a much higher share of the total income going to the top few clubs. The money is divided into three parts: half is divided equally between the clubs; one quarter is awarded on a merit basis based on final league position, the top club getting twenty times as much as the bottom club, and equal steps all the way down the table; the final quarter is paid out as facilities fees for games that are shown on television, with the top clubs generally receiving the largest shares of this. The income from overseas rights is divided equally between the twenty clubs."}
{"context": "Computers control functions at many utilities, including coordination of telecommunications, the power grid, nuclear power plants, and valve opening and closing in water and gas networks. The Internet is a potential attack vector for such machines if connected, but the Stuxnet worm demonstrated that even equipment controlled by computers not connected to the Internet can be vulnerable to physical damage caused by malicious commands sent to industrial equipment (in that case uranium enrichment centrifuges) which are infected via removable media. In 2014, the Computer Emergency Readiness Team, a division of the Department of Homeland Security, investigated 79 hacking incidents at energy companies."}

Config

The task type, plus the two models:

base:
  task: question-answering-closed-book
  student_model_name: Qwen3-0.6B
  teacher_model_name: openai.gpt-oss-120b

Every other field has a default. See Config file for the full table and Supported models for the values you can use.

Create the seed dataset, which validates your files at the same time:

distil seed-dataset create --data ./your-data-dir

Then run teacher evaluation.