Skip to content

Vision models

A vision student reads images: it takes a prompt with one or more images and answers in text. One config key controls it: base.visual_task (default false). With it on, user messages carry images, the teacher and the judge see them, and the student is a vision model trained, evaluated and served with them. It works for question-answering only.

Use it when what the model must read is an image: scanned or photographed pages (OCR, handwriting), forms, receipts and invoices, screenshots, photos to describe or classify. When your application already extracts the text reliably, a text task on that text is simpler, and its train set can grow with synthetic data.

Two things set a vision project apart from a text one:

  • No synthetic data. Synthetic data generation is rejected for visual tasks, because the generators can’t produce images. Every row, train and test alike, is an image you supply with an answer the teacher writes by relabeling, or a row you label yourself.
  • No endpoint traces. A traces object made directly from an inference endpoint rejects image content, so the images come from files, in every iteration.

The images you collect therefore set the size, the coverage and the quality ceiling of the model.

Every teacher and judge role needs a vision model: base.teacher_model_name, trace_processing.teacher_model_name, evaluation.llm_as_a_judge_model_name and every model in trace_processing.relabelling_committee_models. The student must be a vision student. The lists are in Supported models. One non-vision model in any role fails the config, and the recommended large teacher, zai.glm-5.3-low-thinking, is text-only, so set all three fields.

moonshotai.kimi-k3-low-thinking as the teacher and the judge is a good start, or zai.glm-5.3-flash-low-thinking as the teacher when cost matters more. Answers with counts or computed values (word counts, coordinates, totals) can need more reasoning to be exact, so put moonshotai.kimi-k3-max-thinking on the teacher evaluation shortlist for them. Proprietary teachers can read images for teacher evaluation, which is how a production model such as Gemini is scored as the baseline.

base:
  task: question-answering
  visual_task: true
  student_model_name: Qwen3.5-4B
  teacher_model_name: moonshotai.kimi-k3-low-thinking
evaluation:
  llm_as_a_judge_model_name: moonshotai.kimi-k3-low-thinking
trace_processing:
  teacher_model_name: moonshotai.kimi-k3-low-thinking
  observation_format: openai_messages_with_images
tuning:
  max_completion_length: 4096
  • task_description in the job description is the whole system prompt: the output format and the conventions a correct answer follows. Put the same conventions into llm_as_a_judge_instructions (Job description). For a JSON answer, also set synthgen.output_is_json: true.
  • tuning.max_completion_length (default 2048 tokens) must be above the longest answer. A full-page transcription or a JSON record is long, and a cut-off answer scores near zero.
  • tuning.rlvr_dataset_size stays at 0. RLVR fails the config with RLVR is not supported for visual tasks.

The usual flow from files, without synthetic data:

  1. Collect the images. Sample at random from the real source, so every image type, layout, quality and empty case is there in roughly its production share. Split them into test and train by source document (a booklet, a user, a session), not by image: pages of one document look alike, and a test page whose document is also in train inflates the score. A few hundred test images is enough (How many rows); the train set is as many as your budget allows. To check the final model on images no iteration has seen, keep a further set aside.

  2. Write two trace files, test-traces.jsonl and train-traces.jsonl: one user message per line, with the prompt and the image, and no answer (Images in your data).

  3. Build the test set. Create a dataset from the test traces only, since relabeling picks traces at random from the whole pool. Relabel them into the test split, with trace_processing.num_test_relabelled set to the number of test traces, then review the rows next to their images and lock the set (Building the test set):

    distil dataset create --config input/config.yaml --job-description input/job_description.json \
      --traces test-traces.jsonl
    distil dataset relabel-traces-test <dataset-id>
    distil dataset download -d test-set <test-set-dataset-id>
  4. Evaluate the teachers on the test-set dataset: one run per vision candidate, and one with your production model as the baseline (Teacher evaluation).

  5. Build the train set. Create a new dataset with the chosen teacher in the config, the locked test.jsonl and the train traces. Relabel a smoke of 128 traces, read its rows next to their images, then relabel the rest, with trace_processing.num_train_relabelled set to the number of train traces (Relabeling):

    distil dataset create --config input/config.yaml --job-description input/job_description.json \
      --test test-set/test.jsonl --traces train-traces.jsonl
    distil dataset relabel-traces-train --config smoke/config.yaml <dataset-id>
    distil dataset relabel-traces-train <dataset-id>

    Relabeling drops some traces, so expect fewer rows than images.

  6. Train on the dataset the full run wrote (Model training), and serve the model. When it falls short, the main lever is more images of the cases it gets wrong, relabeled into the train split of a new dataset that holds the current train.jsonl, test.jsonl and the new traces (Model iterations).

Traces use the openai_messages_with_images observation format: the user message’s content is a list of text parts and image_url parts.

{"messages": [{"role": "user", "content": [{"type": "text", "text": "Transcribe the handwriting on this page."}, {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}]}]}

Train and test rows carry the images as an images list on the user message. Relabeling writes its rows this way, and rows you upload use the same form:

{"messages": [{"role": "user", "content": "Transcribe the handwriting on this page.", "images": ["data:image/jpeg;base64,..."]}, {"role": "assistant", "content": "..."}]}
  • Data URIs only. Every image is a data:image/...;base64, URI. A URL fails distil dataset create with base.visual_task=true requires every image to be inlined as a data:image/...;base64 URI.
  • Images on user messages only. A trace with an image on any other message fails to parse.
  • Order inside a message isn’t kept. The text parts become one text and the images a separate list, so a prompt can’t rely on text sitting between two images. Say in the text what each image is, in the order the images are attached.
  • Facts the image can’t show (an ID, the image’s pixel size, a source name) go in the user text, with an instruction in task_description to copy them. Nothing is added to the answers after the teacher writes them, and the student learns to produce the whole answer.
  • Leave answers out of the traces. A trace that ends on the user message gets an answer the teacher writes from the image. An answer already in the trace is corrected while staying as close to it as possible, so another model’s answer passes on its habits and mistakes. Answers you trust, such as human transcriptions, belong in train.jsonl and test.jsonl as rows.
  • synthgen.validation_max_total_length counts the text only, images excluded.

Traces with the same text and different images are distinct: deduplication compares the image bytes. The check that train and test share no rows doesn’t. It replaces each image with a placeholder for its position, so two rows with the same user text and the same answer count as one row whatever their images, for example two blank pages answered the same way. Relabeling drops a new row that matches a row of the other split, and an upload with such a pair in train and test fails with train and test datasets have N overlapping examples. When those rows matter, put something image-specific in the user text, such as an ID your application also sends.

Training is SFT only. Every image is stored at most 1024 pixels on its long side, while the teacher saw the full image, so small text, dense pages or fine detail can be legible to the teacher and not to the student. Compare a few failed predictions with their images before blaming the student size; when detail is lost, crop the images to the region that matters, in the traces and in production alike.

The judge metrics receive the images with the prompt. When the teacher wrote the test set, the primary metric is llm-as-a-judge-reference-free (Primary metric per task), so the judge checks each prediction against the image itself. A pass/fail verdict can’t tell a page with one wrong word from a perfect one; ask for a graded score when small errors matter.

  • Local vLLM. A vision student needs one more flag on top of the family flags: --enable-tower-connector-lora.
  • Requests. model_client.py sends the conversation as it is, so build each user message as content parts: a text part, then one image_url part per image, as data URIs. A test row’s images key isn’t understood by the server, so convert it before using a test row to try the model.
  • Hosted inference. Serving image requests through distil labs inference isn’t documented yet; contact contact@distillabs.ai before planning on it.

Everything that tunes generation: synthgen.train_generation_target and test_generation_target, mutators, synthetic_data_generation_instructions, and a top-up by generation. The train set grows with more images, relabeled into the train split. Tools aren’t available either: visual tasks are question answering.

Every other parameter: Config file. The stage that writes every row: Relabeling and synthetic data.