Jev is TypeSafe’s decision model: you send it a piece of text and a typed question (a choice between options, a score on a scale, a yes/no) and it returns the answer with a probability in one call, no prompt to maintain and nothing to train. Since it launched, a lot of teams have started designing their workflows around a decisions API like that. The question we hear from them is whether they could own that model instead: train it on the questions they already ask, run it on their own GPU, and change it when the questions change.
So we took Jev’s own public benchmarks (TypeSafe’s four WorkflowEvals datasets) and built a fine-tuned small model for each of the four on distil labs. NVIDIA gave us early access to Nemotron 3.5 Super VL, and we used it as the teacher for every label and every synthetic example. Our coding agent iterated on the model while we slept (using a validation set) and by the morning it had four Qwen3.5-2B models that match Jev on three of its four benchmarks and four 4B models that are above or within noise of Jev. The combination of high quality and price of Nemotron 3.5 Super VL let the agent iterate dozens of times over night and produce very capable mini-Jevs by morning.
How the agent built them
Each dataset is a set of states (a support conversation, a security incident, an agent trace, an invoice packet) and typed questions about them: a choice between options, a score on a described scale, or a yes/no. Every pair has a reference answer. We treat the training split as the graded traces a team would collect from its workflow and hold the test split out; the teacher never saw it.
The recipe has three parts that build on each other, and the agent moves between them.
Relabel the traces with the teacher. Nemotron 3.5 Super VL answers every training pair itself, about 10,000 across the four datasets, in a few hours. This helps clean up noisy labels and get a high-quality base for training.
Generate synthetic data for the rare cases. Production traffic is skewed. Some questions and some answers barely appear in the traces, and those are often the cases a model must not get wrong. To balance out the distribution, the teacher writes new question/answer pairs about real states aimed at the segments the traces under-represent.
Train, evaluate, go back to the data. Qwen3.5-2B or 4B with a LoRA adapter trains in under 6 hours on one A100, on distil labs’ own training pipeline built on Hugging Face TRL and PEFT. The student answers with a single label token, and the probability per option is read from that token’s distribution, which is what makes the output Jev-shaped.
None of this was hand-tuned. A coding agent with the distil labs skill ran the loop the way autoresearch describes it: start with a validation set and iterate on building data and training models until you run out of budget or reach a high enough score. Between 19:45 and 10:00 the next morning it submitted 35 jobs: 15 labelling passes, 5 synthetic-data runs and 15 trainings. The test split was scored once per trained model, and every variant is reported.
Two findings from the night carry over to any decision task:
- When the teacher consistently disagrees with the reference answers, fix the system prompt. On security-incidents Nemotron agreed with the reference answers on 77% of the examples, and a majority vote over three runs scored 78%. The system prompt was consistently pushing the teacher to the wrong answers, and updating it with more accurate guidance got it to 80%.
- The skewed distribution of the traces is the other problem, and the solution was to create synthetic data for under-represented segments. Each iteration the agent read the confusion table per question, found the segments the student got wrong, and generated synthetic examples for them. On security-incidents those rows added 1.4 points on top of the corrected labels, and they are what covers the answers that occur only a handful of times in the traces.
What made 35 jobs in one night affordable is the teacher. Nemotron 3.5 Super VL is a smaller model than the frontier teachers we usually reach for, which shows in both the clock and the bill: labelling a whole dataset took 23 minutes on three of the four sets, and the teacher tokens behind all four models came to about USD 19 (cost basis in the notes below). When a labelling variant is a half-hour experiment that costs a dollar or two, an agent can afford to run five of them per dataset before breakfast, and that is where the accuracy came from.
Results
Accuracy on the held-out test split. Jev’s column is TypeSafe’s published prediction for each test question, with the questions Jev did not reach on its workflow path counted as wrong, which is why security-incidents reads 0.429 (Jev scored 0.695 on the 62% it answered). The intervals are a bootstrap over the test questions.
| Benchmark | Test questions | Untuned 2B | Fine-tuned 2B (95% interval) | Fine-tuned 4B (95% interval) | Jev |
|---|---|---|---|---|---|
| customer-service | 652 | 0.725 | 0.905 (0.882 to 0.928) | 0.900 (0.877 to 0.922) | 0.899 |
| security-incidents | 303 | 0.403 | 0.799 (0.752 to 0.845) | 0.789 (0.743 to 0.835) | 0.429 |
| agent-trace-observability | 221 | 0.538 | 0.796 (0.742 to 0.846) | 0.819 (0.769 to 0.869) | 0.833 |
| invoice-processing | 1,317 | 0.680 | 0.917 (0.902 to 0.932) | 0.926 (0.913 to 0.940) | 0.940 |
Deep dive: the security-incidents build
We decided to follow the journey of the security-incidents agent, since it was the most involved with a nail-biting finish including a fix that partly backfired before the night was over. For some background, security-incidents records are alerts with evidence and 14 questions posed to every alert (is the activity unauthorised, would it survive a reboot, etc.). Jev’s published run stands at 0.429 if we count a refusal to answer as a wrong answer.
By 20:50 the agent had labelled the 1,387 training pairs and compared them with the reference answers. Agreement with original data was at 0.769, and one question stood out: the evidence-strength score, was right 20% of the time. Instead of training on that, the agent spent the next hours on the labels: a pass with five examples in the prompt, a pass with a different seed, a majority vote over all three: 0.779 every way. The passes were wrong together on the same pairs, so the teacher was clearly reading the criteria differently from whoever wrote the reference answers. A stronger teacher would not have helped here; a clearer task description would.
At 23:30 the agent read the confusion table, and the difference turned out to be easy to describe. On the question of how strong the evidence is, the teacher was consistently one level too generous, answering “corroborated”, which needs several independent indicators, where the reference answer said “suggestive”, which means a single one. For a fix, the agent added two sentences to the task description and at 23:40 relabelled the traces with them; the corrected labels came back at 01:05. When the two students finished within a minute of each other at 02:04, the one trained on the plain labels scored 0.726 and the one trained on the corrected labels 0.785, with the evidence question alone going from 0.10 to 0.56.
Around 02:10 the agent noticed that one of the two added sentences had overcorrected a different question, so it relabelled a third time without that sentence, but at 04:30 those labels came back no better overall and it kept the first corrected set. It then retrained the student on the corrected labels plus 4,223 synthetic examples aimed at the weak questions, and that model, finished at 09:25 at 0.799, is the one in the results table. It is 1.4 points above the 02:04 model.
By the time we sat down with coffee at 9.30am, the agent had (just) turned a teacher into a 2B student that gets the right answer more often than Jev does. The whole security build, from the first label to the last score, took one night, USD 9.4 and nobody’s attention!
Build your own
The three stages are the platform’s three stages, each a distil command or a sentence to a coding agent with the distil labs skill. The decision task is in private preview on distil labs and comes online for every account in the next week or two. The converter from TypeSafe’s datasets, the benchmark scripts, the night’s run log and the exact training inputs are in the repo, and the eight models are on Hugging Face as LoRA adapters, distil-labs/distil-qwen3.5-{2b,4b}-jev-<benchmark> (for example distil-qwen3.5-2b-jev-security-incidents); both go public with this post.
If your workflow already asks a model typed questions, you have the traces to start from. Export a few hundred, run the three stages with Nemotron 3.5 Super VL as the teacher, and compare the student with whatever answers those questions today on a held-out slice. The first two training runs are free. Sign up, read the docs, or install the agent skill and let it run the loop.
distil labs is a platform that fine-tunes task-specific small language models automatically. Most teams use it to swap the general-purpose LLM in one stage of their system for a smaller, purpose-built one: same quality on that task, around 80% lower cost and lower latency. You bring a task description and a few dozen examples, or the production traces you already collect, and the platform generates synthetic training data from them with its own generation pipeline, trains an SLM, and gives you the weights or an OpenAI-compatible endpoint.
distil labs trains task-specific small language models that are cheaper and faster per request than general-purpose LLMs, with equal-or-better accuracy on bounded tasks. Docs · GitHub · Slack
Notes
Cost basis. Nemotron 3.5 Super VL has no public price yet; teacher tokens are priced at the NVIDIA API list price of its predecessor, Nemotron 3 Super (USD 0.09 per million input, 0.45 per million output, 2026-09-01), as a stand-in. Evaluation and labelling tokens are counted from the stored prompts and reasoning with an approximate tokenizer; synthetic-data tokens are estimated from the prompt composition. GPU time is the duration of each training job on one A100 80 GB at RunPod Secure Cloud’s USD 1.59 per hour (2026-09-27). Per 2B model, teacher plus GPU: customer-service USD 1.17 + 5.69, security-incidents 4.31 + 5.04, agent-trace-observability 0.63 + 1.06, invoice-processing 13.3 + 5.30. Calculated from token counts and job durations, not read off a bill.
How Jev is scored. TypeSafe’s published WorkflowEvals runs cover only the questions a model reached on its own workflow path (98.5% / 61.7% / 95.9% / 99.8% of our test questions). Rerun like for like on every question, Jev scores 0.906 / 0.789 / 0.864 / 0.943. No Jev output was used in training. The 100 seed pairs per dataset carry TypeSafe’s reference answers, a consensus of OpenAI and Anthropic models; everything else was labelled or written by Nemotron 3.5 Super VL.