Open book QA for RAG data preparation

The example on this page answers questions about Wikipedia articles, with the passage holding the answer supplied alongside each question. That’s the shape a retrieval pipeline needs: your retriever provides the chunk, the model answers from it.

The shared rules for the input directory, row shapes and validation are in Overview. This page covers what’s specific to open book QA.

Job description

What you expect the model to do, in the words you’d use to prompt an LLM.

task_description. The task itself.
llm_as_a_judge_instructions. Optional, but it does real work here: this task is scored by an LLM judge, which sees the question, the context, the reference answer and the prediction, and returns a binary good or bad.

{
  "task_description": "Answer the question using information in the context",
  "llm_as_a_judge_instructions": "Evaluate whether the predicted answer correctly answers the question based on the reference answer. Output 'good' if the predicted answer is semantically equivalent to the reference answer or conveys the same key information, otherwise output 'bad'"
}

Training data

Each row is a messages conversation, a user turn with the question and an assistant turn with the answer, plus a sibling context field holding the passage the answer comes from. The context sits next to messages, not inside it.

Aim for 20+ diverse rows, with passages the length your retriever actually returns.

{"messages": [{"role": "user", "content": "What does it mean when currencies are left to \"float?\""}, {"role": "assistant", "content": "to \"float\" (rise and fall according to market demand)"}], "context": "On August 15, 1971, the United States unilaterally pulled out of the Bretton Woods Accord. The US abandoned the Gold Exchange Standard whereby the value of the dollar had been pegged to the price of gold and all other currencies were pegged to the dollar, whose value was left to \"float\" (rise and fall according to market demand). Shortly thereafter, Britain followed, floating the pound sterling. Because oil was priced in dollars, oil producers' real income decreased. In September 1971, OPEC issued a joint communiqu\u00e9 stating that, from then on, they would price oil in terms of a fixed amount of gold."}
{"messages": [{"role": "user", "content": "What are those with lower incomes often unable to manage?"}, {"role": "assistant", "content": "their finances"}], "context": "Firstly, certain costs are difficult to avoid and are shared by everyone, such as the costs of housing, pensions, education and health care. If the state does not provide these services, then for those on lower incomes, the costs must be borrowed and often those on lower incomes are those who are worse equipped to manage their finances."}
{"messages": [{"role": "user", "content": "In what part of the city did residents suffer from a lack of city services?"}, {"role": "assistant", "content": "unincorporated suburbs"}], "context": "Much of the city's tax base dissipated, leading to problems with funding education, sanitation, and traffic control within the city limits. In addition, residents in unincorporated suburbs had difficulty obtaining municipal services, such as sewage and building code enforcement."}
{"messages": [{"role": "user", "content": "How many species of insects are known in the region?"}, {"role": "assistant", "content": "2.5 million"}], "context": "The region is home to about 2.5 million insect species, tens of thousands of plants, and some 2,000 birds and mammals. To date, at least 40,000 plant species, 2,200 fishes, 1,294 birds, 427 mammals, 428 amphibians, and 378 reptiles have been scientifically classified in the region."}
{"messages": [{"role": "user", "content": "In the fall quarter of 2014, how many students signed up for the university's professional schools?"}, {"role": "assistant", "content": "5,984"}], "context": "In the fall quarter of 2014, the University of Chicago enrolled 5,792 students in the College, 3,468 students in its four graduate divisions, 5,984 students in its professional schools, and 15,244 students overall."}

Test data

Same format as train.jsonl, context field and all. This is what every score you’ll see is measured against.

No row may be identical to a training row, or validation fails. Keep each answer genuinely present in its own context, since that’s the behaviour you’re training: answering from the passage rather than from what the model already knows.

{"messages": [{"role": "user", "content": "In what year did the Bretton Woods Accord collapse?"}, {"role": "assistant", "content": "1971"}], "context": "On August 15, 1971, the United States unilaterally pulled out of the Bretton Woods Accord. The US abandoned the Gold Exchange Standard whereby the value of the dollar had been pegged to the price of gold."}
{"messages": [{"role": "user", "content": "Which currency did Britain float shortly after the US?"}, {"role": "assistant", "content": "the pound sterling"}], "context": "Shortly thereafter, Britain followed, floating the pound sterling. Because oil was priced in dollars, oil producers' real income decreased."}
{"messages": [{"role": "user", "content": "How many bird species have been scientifically classified in the region?"}, {"role": "assistant", "content": "1,294"}], "context": "To date, at least 40,000 plant species, 2,200 fishes, 1,294 birds, 427 mammals, 428 amphibians, and 378 reptiles have been scientifically classified in the region."}
{"messages": [{"role": "user", "content": "How many students were enrolled in the College in the fall quarter of 2014?"}, {"role": "assistant", "content": "5,792"}], "context": "In the fall quarter of 2014, the University of Chicago enrolled 5,792 students in the College, 3,468 students in its four graduate divisions, 5,984 students in its professional schools, and 15,244 students overall."}

Unstructured data (required)

Unstructured data steers the teacher toward diverse, domain-specific examples. For open book QA, supply realistic passages of the kind your retriever returns at inference time. Rows carry a single context field.

{"context":"For months each side had been building forward rifle pits and defensive positions, which resulted in many skirmishes. Artillery fire aiming to gain superiority over the enemy guns.:450\u2013462 September saw the final assault. On 5 September, another French bombardment (the sixth) was followed by an assault by the French Army on 8 September resulting in the capture of the Malakoff by the French, and following their failure to retake it, the collapse of the Russian defences. Meanwhile, the British captured the Great Redan, just south of the city of Sevastopol. The Russians retreated to the north, blowing up their magazines and the city fell on 9 September 1855 after a 337-day-long siege.:106"}
{"context":"\"Funday Night at the Movies\" was replaced in 2008 by \"Essentials Jr.\", a youth-oriented version of its weekly series The Essentials (originally hosted by actors Abigail Breslin and Chris O'Donnell, then by John Lithgow from 2009 to 2011, and then by Bill Hader starting with the 2011 season), which included such family-themed films as National Velvet, Captains Courageous and Yours, Mine and Ours, as well as more eclectic selections as Sherlock, Jr., The Music Box, Harvey, Mutiny on the Bounty and The Man Who Knew Too Much."}
{"context":"This view of 'diplomatic drift' as the cause of the war was first popularised by A. W, Kinglake, who portrayed the British as victims of newspaper sensationalism and duplicitous French and Ottoman diplomacy. More recently, the historians Andrew Lambert and Winfried Baumgart have argued that, first, Britain was following a geopolitical strategy in aiming to destroy a fledgling Russian Navy which might challenge the Royal Navy for control of the seas, and second that the war was a joint European response to a century of Russian expansion not just southwards but also into western Europe."}
{"context":"The original city council line-up of 14 members (nine district-based and five at-large positions) was based on a U.S. Justice Department mandate which took effect in 1979. At-large council members represent the entire city. Under the city charter, once the population in the city limits exceeded 2.1 million residents, two additional districts were to be added. The city of Houston's official 2010 census count was 600 shy of the required number; however, as the city was expected to grow beyond 2.1 million shortly thereafter, the two additional districts were added for, and the positions filled during, the August 2011 elections."}
{"context":"The school newspaper of the University of Kansas is University Daily Kansan, which placed first in the Intercollegiate Writing Competition of the prestigious William Randolph Hearst Writing Foundation competition, often called \"The Pulitzers of College Journalism\" in 2007. In Winter 2008, a group of students created KUpedia, a wiki about all things KU. They have received student funding for operations in 2008\u201309. The KU Department of English publishes the Coal City Review, an annual literary journal of prose, poetry, reviews and illustrations. The Review typically features the work of many writers, but periodically spotlights one author, as in the case of 2006 Nelson Poetry Book Award-winner Voyeur Poems by Matthew Porubsky."}

Config

The task type, plus the two models:

base:
  task: question-answering-open-book
  student_model_name: Qwen3-0.6B
  teacher_model_name: openai.gpt-oss-120b

Every other field has a default. See Config file for the full table and Supported models for the values you can use.

Create the seed dataset, which validates your files at the same time:

distil seed-dataset create --data ./your-data-dir

Then run teacher evaluation.