TL;DR
-
Knowunity’s revision-quiz feature moved from a frontier API model to NVIDIA Nemotron Tutor, based on NVIDIA Nemotron 3.5 Lightning open model post-trained by distil labs on Nebius GPUs. It now serves a large portion of Knowunity’s English quiz traffic.
-
On 114 production requests judged against Knowunity’s four review criteria, 65% of its quizzes pass all four, against 39% for the incumbent (+25 points, 95% CI +16 to +35) and 30% for the stock Nemotron.
-
It costs about $0.50 per 1,000 quizzes against $1.60 for the incumbent, and answers in 5 seconds at the median.
-
Training took about 68 GPU-hours on NVIDIA NeMo RL post-training library: supervised fine-tuning on data generated from Knowunity’s own traffic, then reinforcement learning with an LLM judge as the reward.
Say you run a feature that turns whatever a student is studying into a revision quiz: the right grade, the right school type, the right language, a sensible mix of question styles, no repeated items. It runs every time a student opens a lesson, which puts it at a volume where the per-request price of a frontier API is a line the finance team asks about, and teachers set the quality bar, so you cannot just reach for the cheapest model on the price list.
Knowunity, a Berlin-based company, is the daily learning companion for students from secondary school to the end of high school, and has exactly this feature. It ran on a frontier API model, which did the job well enough to ship and cost enough to matter. Together with NVIDIA and Nebius we replaced it with a post-trained NVIDIA Nemotron 3.5 Lightning, a 30B-parameter model that activates 3B parameters per token. On Knowunity’s own review criteria it writes better quizzes than the incumbent, at about a third of the cost per quiz.
| Frontier API model (incumbent) | Nemotron Tutor (post-trained Nemotron 3.5 Lightning) | |
|---|---|---|
| Quizzes passing all four review criteria | 39% | 65% |
| Cost per 1,000 quizzes | $1.60 | ~$0.50 |
| Latency in production, p50 / p90 / p99 | comparable | 5 s / 7 s / 10 s |
| Training compute | n/a | ~68 GPU-hours |
Measured on 114 English quiz requests from Knowunity’s production traffic, each quiz judged by Claude Opus 4.8 on the four criteria Knowunity’s reviewers use (coverage, diversity, learner fit, coherence). The rest of this post is how we got there and what bounds the result.
Why Nemotron 3.5 Lightning
We chose NVIDIA Nemotron 3.5 Lightning for three reasons. It has 30B parameters in total, which is where its answer quality comes from, and a mixture-of-experts design that activates only 3B of them per token, which is where its speed and cost come from: a quiz costs roughly what a 3B model would charge, with the knowledge of a model ten times that size. It is a hybrid Mamba-Transformer, which keeps inference fast on long requests, and quiz requests often carry a thousand tokens or more of pasted study notes. And the weights are open under the OpenMDW 1.1 licence, so the model Knowunity serves is a model Knowunity owns: no rate limits, no deprecation notices, no price changes from a provider.
We also had a reason from our own data. In our mid-size MoE fine-tuning benchmark, Nemotron 3.5 Lightning ranked third of the models we prompted and first after fine-tuning, clearly beaten on only one of six tasks. Base scores are a weak guide to what a model does once it is trained on your task, and this model is the clearest case of that we have measured.
English is among the model’s supported languages, which is why the rollout starts there.
No model at this price does the task untuned
Before training anything we ran the stock model on the same 114 requests, and it confirmed the benchmark’s pattern. With reasoning switched off, 30% of its quizzes pass all four criteria (95% CI 22% to 38%), against 39% for the incumbent, and 23% of its individual questions are judged broken: a wrong key, two defensible answers, or a false claim in an explanation. With reasoning switched on it does no better, 29% all four, and it gets there by thinking for a median of 16,000 tokens per quiz: 13x the cost of reasoning off, and 63 seconds per quiz at the median, more than ten times what Knowunity can serve. Thinking is not a substitute for knowing the task.
That is not a Nemotron problem. On the same requests, no off-the-shelf model under one dollar per 1,000 quizzes reaches 60% on all four criteria (the chart further down shows all of them). At Knowunity’s price point the task is not solved by picking a model; it is solved by training one.
How we trained it
The whole run went through the distil labs platform, which turns a production endpoint’s logs into a trained and served student model. The steps, in order:
-
Collect traces. Knowunity’s existing quiz traffic, the request and the quiz the production model returned, is the raw material. No labelling.
-
Build the test set. The platform relabels a slice of traces with a strong teacher, Kimi K3, and adds synthetic requests where the traffic is thin, so the test set covers the grades, subjects and request shapes production actually sees. Every model in this post is scored on this set.
-
Build the training set. The same two operations at scale: relabelling with Kimi K3, then synthetic data generation from the traces, with rule-based validators filtering for schema, length and novelty. The result is about 50,000 training rows, each a quiz request and a quiz the teacher wrote for it.
-
Supervised fine-tuning. Nemotron 3.5 Lightning is fine-tuned on 80% of those rows for 4 epochs.
-
Reinforcement learning. On the remaining 20% we run GRPO with verifiable rewards, where the reward is an LLM judge, GLM 5.3, scoring each generated quiz against the four criteria. This is the step that teaches the model which of its own quizzes a reviewer would send back. The judge that scores the results in this post is a different model, Claude Opus 4.8, so the model was not graded by the model that trained it.
-
Serve. The trained model goes behind an OpenAI-compatible endpoint, and Knowunity switches the base URL.
Both training stages ran on NVIDIA NeMo RL open library for post-training, on H100 GPUs provided by Nebius: about 20 GPU-hours for the supervised stage and about 48 GPU-hours for the reinforcement stage, roughly 68 GPU-hours in total, which came to about $250 on Nebius. That is the training itself; it excludes the teacher calls that built the datasets.
Training took the model from 30% of quizzes passing all four criteria to 65%, and from 23.1% of questions judged broken to 5.9%. Its teacher, Kimi K3, scores 75% on the same requests at $14.81 per 1,000 quizzes; the student keeps most of that quality at about 1/30 of the price.
Detailed results
Knowunity’s reviewers grade a quiz on four criteria. Coverage: does it deliver what the request asked for, in the form it asked for, with the number of questions it asked for? Diversity: do the questions test genuinely different things (definitions, calculations, applications, interpretation) rather than one template repeated? Learner fit: is it pitched at the requested grade, school type and difficulty? Coherence: is it a designed sequence, with no duplicate questions, no broken items, and no question that gives away another’s answer? A quiz that fails any one of them is a quiz a teacher would send back, so the headline metric is the share passing all four.
Per criterion, on the 114 English requests, judged by Claude Opus 4.8:
| criterion | stock Nemotron 3.5 Lightning | incumbent | Nemotron Tutor | difference vs incumbent, 95% CI |
|---|---|---|---|---|
| Coverage | 75% | 84% | 94% | +10 [+2, +18] |
| Diversity | 56% | 68% | 89% | +20 [+11, +30] |
| Learner fit | 82% | 93% | 94% | +1 [−4, +5] |
| Coherence | 34% | 48% | 70% | +22 [+11, +32] |
| All four | 30% | 39% | 65% | +25 [+16, +35] |
| Questions judged broken | 23.1% | 5.3% | 5.9% | |
| Errors per quiz | 1.32 | 0.32 | 0.35 | +0.04 [−0.11, +0.17] |
Three of the four criteria move by a margin the interval cannot explain away: coverage, diversity and coherence. Learner fit was already high for the incumbent and stays there, a one-point difference inside the noise. Correctness is level rather than better: 5.9% of the post-trained model’s questions are judged broken against 5.3% for the incumbent, 0.35 errors per quiz against 0.32, with an interval that spans zero. So the gain is in quiz design, and the biggest remaining weakness is the criterion that improved most: 30% of the post-trained model’s quizzes still fail coherence, usually a repeated question pattern or an answer leaked across items, against 52% for the incumbent.
What bounds the headline number: 114 requests is enough to put the all-four gain at 16 points or more with 95% confidence, and not enough to rank the criteria finely against each other. The requests are English only, from the UK and US slices of Knowunity’s traffic, so these numbers say nothing yet about the other languages. And a single LLM judge, calibrated against Knowunity’s reviewers, is a proxy for a teacher, which is why the production rollout was gated on Knowunity’s own evaluation rather than on these numbers alone.
Where it sits against frontier models
The same 114 requests, the same judge, 17 models. The x-axis is what each model costs per 1,000 quizzes at list API prices, from its measured token usage.
In production
Nemotron Tutor serves a large portion of Knowunity’s English quiz traffic. Latency is 5 seconds at the median, 7 seconds at the 90th percentile and 10 seconds at the 99th, comparable to the incumbent. At the serving cost above, a quiz costs about $0.0005 to generate.
The next step is the other languages. Besides English, Knowunity has a lot of traffic in Portuguese, Spanish, German, Italian, Filipino and Polish, and the same workflow applies: the traces exist, the test set is built the same way, and the training recipe does not change.
“With distil labs, we turned our own quiz traffic into a model trained around Knowunity’s standards. In our English evaluation, it produced better-designed quizzes at roughly a third of the serving cost. We now have a model we can keep building on as our students’ needs evolve, with the weights and the ability to train new versions on our own data.”
Lucas Hild, CTO of Knowunity
If you have a task like this
The pattern generalises past quizzes: a narrow, high-volume task inside a product, currently on a frontier API, with logs. If that describes a feature of yours, the starting point is those logs. Put the distil labs endpoint in front of the model you run today, let it record a day of traffic, and start a training run from the recorded traces; the platform handles relabelling, synthetic data, training and serving from there. You can start for free, read how the workflow fits together, or see why base-model rank is a poor guide to fine-tuned rank in the benchmark that led us to Nemotron.
distil labs trains task-specific small language models that are cheaper and faster per request than general-purpose LLMs, with equal-or-better accuracy on bounded tasks. Docs · GitHub · Slack