Train an SLM for Document QA over Contracts

Contract QA is an open-book task (question-answering-open-book) and its hard part is upstream of the model. A 60-page agreement does not fit in a small student’s context window, and a clause split across two chunks yields a fluent, confident, wrong answer.

A contract does not fit in the window, and clauses do not survive naive chunking

Fixed-size chunking is the default in most RAG stacks and it is wrong for contracts specifically. Legal meaning is carried by structures that fixed windows cut through: a liability cap qualified by a carve-out two paragraphs later, a definition in section 1 that governs a clause in section 12, a termination right whose notice period lives in an annex.

Split those and you have not lost information — you have created misinformation. The retrieved chunk still reads like a complete answer, so the model answers from it, and neither the model nor the reader can tell that the qualifier is missing.

Chunking strategy	What breaks	Verdict for contracts
Fixed token window	Clauses, cross-references, tables	Avoid
Fixed window with overlap	Fewer clean splits, more duplication	Tolerable fallback
Clause / section boundary	Preserves the unit a lawyer reasons about	Preferred
Whole document	Exceeds a small student’s context	Not viable

Step 1: Chunk on clause boundaries

Convert the PDF preserving structure — headings, numbering, and tables all need to survive. pdfplumber or a comparable extractor gets you there; a plain text dump that flattens numbering does not.

Then segment on the document’s own boundaries: numbered clauses, schedules, annexes. Attach the section heading and number to each chunk as a prefix, so the passage is self-locating. Where a clause is short and depends on a definition, include the definition in the chunk rather than trusting the retriever to fetch both.

Clean before you generate. In our industrial documentation study, filtering out tables of contents, indexes, and formatting artefacts — keeping only chunks with more than 50% alphanumeric content — reduced the corpus from 4,259 chunks to 1,493. Contracts carry the same ballast: signature blocks, page furniture, repeated headers.

Step 2: Write questions the way reviewers ask them

Take the questions from the people who read these agreements. The task selection guide lists the shape: “What is the termination clause?”, “When does the agreement expire?” — specific, answerable from one place, phrased as a reviewer would phrase them.

Aim for roughly 200 seed question-answer-context triples across the contract families you handle, plus around 50 held out for testing. That is the scale the industrial study used, and the platform expands it into thousands of synthetic examples.

Step 3: Build train.jsonl with the context field

Open-book data differs from plain question answering in one structural way: each line carries a sibling context field holding the passage the answer must be grounded in.

{"messages": [{"role": "user", "content": "What notice period applies to termination for convenience?"}, {"role": "assistant", "content": "90 days' written notice"}], "context": "12.2 Termination for Convenience. Either party may terminate this Agreement for convenience upon ninety (90) days' prior written notice to the other party."}

Answers should be terse and quotable. A model trained to answer “90 days’ written notice” is auditable; a model trained to answer in three sentences of paraphrase is not. job_description.json needs task_description plus, ideally, llm_as_a_judge_instructions — the open-book data preparation guide has the spec.

Step 4: Steer generation with mutation topics and distractors

base:
  task: question-answering-open-book
  student_model_name: Qwen3-1.7B
  teacher_model_name: zai.glm-5
synthgen:
  teacher_temperature: 0.6
  num_distractor_context_blocks: 2
  mutation_topics:
    - "notice periods, deadlines, and how time is calculated"
    - "liability caps and their carve-outs"
    - "conditions that must be satisfied before a right is exercisable"
    - "definitions and which clauses they govern"
    - "what happens on breach, insolvency, or change of control"

num_distractor_context_blocks adds irrelevant passages alongside the correct one during generation, training the model to ignore them — the technique published as RAFT. It matters here because a contract retriever routinely returns three plausible clauses and only one is right.

mutation_topics prevents the teacher from producing a thousand variations of “what is the governing law”. If your contracts are long, a long-context teacher such as Qwen3-235B-A22B-Instruct-2507 is an option, though it is not a reasoning model and does not carry the 0.5–0.7 temperature constraint that zai.glm-5 does.

Step 5: Evaluate the teacher, train, and grade honestly

distil model create contract-qa
# Output: Model created with ID: <model-id>

distil model upload-data <model-id> --data ./data
distil model run-teacher-evaluation <model-id>
distil model teacher-evaluation <model-id>

distil model run-training <model-id>
distil model training <model-id>
distil model download-training-predictions <model-id>

Grade with an LLM judge rather than exact match — reviewers phrase the same answer several ways — but write the rubric so that inventing an unsupported term fails outright. The metrics guide and the RAG tutorial cover how the judge is applied.

Add one test the platform does not run for you: hold back a set of questions whose answer is not in the retrieved chunk, and check the model says so. A contract model that always produces an answer is a liability.

Step 6: Deploy and pass context at inference

distil model deploy local <model-id>
distil model invoke <model-id>

Open-book models expect the passage in a <context> tag inside the first user message, followed by a newline and the question — local deployment shows the exact client invocation. Your retriever fills that tag.

Where a contract QA model still needs a lawyer

Anywhere the answer is an opinion rather than a location. “Is this clause enforceable?” and “should we accept this?” are not extraction, and a model that answers them is guessing in a register that sounds authoritative.

Keep the model on locate-and-quote work: finding the clause, stating the term, flagging absence. Everything downstream of that is review. If you need answers traceable to a source document, stay open-book — open-book vs closed-book QA explains why closed-book cannot cite, and what is question answering as a training task covers the variants.