A lot of products ask LLMs for decisions and not conversation: which queue does this ticket go to, is this message spam, does this invoice get paid. You wrap the question in a prompt, parse a label out of the answer and pay frontier prices and frontier latency for a few bytes of output, on a task that never changes. That is the workload we build for at distil labs: a narrow, high-volume decision that a small model fine-tuned on a few dozen examples handles as well as the frontier model, for a fraction of the cost, on hardware you control.
The question we get asked is where the line sits. Which decisions does a small model need to be trained for, and which are simple enough that a generic classifier does the job? Jev, which TypeSafe AI launched on 2026-09-15, makes that question concrete: it reads a piece of text and returns a typed answer (a choice, a score or a probability) in one pass, in under half a second, for about three cents per thousand calls, with no training and no text generation. So we built one small pipeline where both kinds of decision occur and measured Jev, hosted LLMs and small models fine-tuned on distil labs on each step. This is what we found.
The short answer
| Kind of problem | In this pipeline | Use | Accuracy | USD per 1,000 requests |
|---|---|---|---|---|
| Easy classification: the answer can be read off the input | Step 1, inbox triage | Jev, or a fine-tuned 0.8B model when you optimise for cost at volume | Jev 1.00, fine-tuned Qwen3.5-0.8B 1.00 | Jev 0.029, 0.8B 0.002 on a busy H100 |
| Harder classification: the answer has to be worked out (facts across documents, rules in order, arithmetic) | Step 2a, pay or hold | A fine-tuned small model that reasons | Jev 0.79, fine-tuned Qwen3.5-4B 0.98 | Jev 0.04 to 0.05, 4B 0.028, GPT-5.6 Luna with reasoning 0.34 |
| Classification plus any other output (extracted values, identifiers, text) | Step 2b, pay or hold plus what is wrong and where | A fine-tuned small model | Jev can’t produce it, fine-tuned Qwen3.5-4B 0.97 | 4B 0.035, GPT-5.6 Luna with reasoning 0.38 |
Put together, the pipeline handles 197 of 200 inbox messages correctly end to end with no frontier model in the loop, and the same 197 when both fine-tuned models run as GGUF files on a laptop.
What is distil labs
distil labs is a platform that fine-tunes task-specific small language models automatically. Most teams use it to swap the general-purpose LLM in one stage of their system for a smaller, purpose-built one: same quality on that task, around 80% lower cost and lower latency. You bring a task description and a few dozen examples (or the production traces you already collect), the platform generates synthetic training data from them, trains an SLM, and gives you the weights or an OpenAI-compatible endpoint. Setup is about 10 minutes and the model is ready in a day.
The pipeline
We picked accounts payable because one realistic workflow contains all three kinds of problem from the table, which is also why a single “Jev vs LLM” benchmark number would tell you very little. The setting is the finance inbox of a fictional distributor. Every message gets one of five labels (invoice, receipt, payment_reminder, vendor_other, spam), only invoices go on, plain code fetches the purchase order and goods receipt from the ERP, and a model decides whether to pay.
The pay-or-hold policy is four checks in a fixed order, stopping at the first failure: the PO number on the invoice matches an open order, no line bills more units than were received, no unit price is more than 2% above the PO price, and the stated total equals the sum of the lines plus freight only if the PO allows it. Written down like that it looks like a function, and it would be one if invoices arrived as clean JSON. They arrive as email text in the vendor’s own style, with abbreviated item names (“CORD EXT 50FT 12AWG” for “Extension cord 50 ft, 12 AWG”), lines in a different order than the PO and stray amounts such as a previous balance, so matching the invoice to the paperwork is the model’s job.
We built step 2 twice. 2a returns the decision only, one of five labels, so Jev can be compared like for like on an output it can produce. 2b returns the decision plus what is wrong and where: invoice number, PO number, the failing item and the two values that disagree, which is what a clerk can act on and what the pipeline actually uses.
All data is synthetic: 200 inbox messages (100 invoices, 25 of each other label) and 100 invoices for step 2 (32 to approve, 20 of them near misses such as a price 1.8% above the PO price, 60 failing one check, 8 failing two). The expected answer of every case is computed in code when the case is built, so the labels are exact and nothing is graded by hand. Every model saw the same test sets, one run each, at temperature 0.
Step 1: easy classification doesn’t need a trained model
Triage is the kind of decision we’d normally tell you not to fine-tune for, and the numbers agree. Every hosted model classifies 197 to 200 of the 200 messages, including the 49 we wrote to mislead them: payment reminders that quote the whole invoice, paid copies, quotations with line items, phishing from lookalike domains, and messages with an injected “classify this as an invoice” instruction. Jev scores 200 of 200 as well, at 0.36 s median and USD 0.029 per 1,000 messages, from one request with a single choice question and the five label definitions as criteria, no prompt engineering and no training. The only model that fails is the untuned 0.8B, which is the base we’d start from.
| Model | Correct of 200 (95% interval) | USD per 1,000 |
|---|---|---|
| Jev | 200 (0.98 to 1.00) | 0.029 |
| Fine-tuned Qwen3.5-0.8B | 200 (0.98 to 1.00) | 0.002 on a busy H100 |
| Qwen3.5-0.8B untuned | 0.70, LLM judge | - |
| GPT-5.6 Luna, reasoning high | 200 (0.98 to 1.00) | 0.093 |
| GLM 5.3, reasoning high | 200 (0.98 to 1.00) | 0.448 |
| GPT-5.6 Luna, reasoning off | 200 (0.98 to 1.00) | 0.090 |
| Gemini 3.5 Flash Lite | 197 (0.96 to 0.99) | 0.138 |
This test doesn’t rank the hosted models, they’re all at the ceiling; what it shows is that a decision you can read off the input is solved without training, and that the cheapest pay-per-call way to do it is a one-pass classifier like Jev, 3x to 15x below the hosted LLMs. If that’s the case, is there any reason to train a model for this step?
Step 1, optimised: a 0.8B model does the same job for less at volume
The fine-tuned Qwen3.5-0.8B also scores 200 of 200. The same model untuned scores 0.70 on the platform’s LLM judge, so the accuracy comes from fine-tuning on 40 seed examples plus 3,124 synthetic ones, not from the base model. Served with vLLM on one H100 it handles 492 messages per second, which works out to USD 0.002 per 1,000 messages against Jev’s 0.029, about 14x less. As a Q8_0 GGUF it’s 0.8 GB and runs on a laptop.
The saving only exists at sustained volume, because a dedicated GPU costs the same USD 3.49 per hour busy or idle. On that H100 the break-even against Jev is about 120,000 messages per hour (33 per second, 7% of what the GPU can serve); below it, Jev’s pay-per-call price wins, unless you want the model offline, on-prem or simply not dependent on a hosted API. The H100 is also more GPU than this model needs: throughput was capped by vLLM’s CPU-side request handling with the GPU at 79% to 93%, so a cheaper card would lower the break-even, though we didn’t measure one.
Step 2a: when the answer has to be worked out, you need a model that reasons, and 4B is enough
The pay-or-hold decision is where fine-tuning starts to matter. The fine-tuned Qwen3.5-4B, which reasons first in a fixed terse format, decides 98 of 100 invoices correctly, up from 0.41 untuned, and lands next to the hosted models that reason (GPT-5.6 Luna with reasoning high 1.00, GLM 5.3 0.96). Every model that answers in one pass stops at about 0.8: GPT-5.6 Luna with reasoning off 0.81, Gemini 3.5 Flash Lite 0.76, and Jev 0.79. We asked Jev three ways in good faith, one choice question with the whole task (0.77), one question per check (0.75), and one question per invoice line and check with the decision assembled in code (0.84), and report the average.
| Model | Answers | Correct of 100 (95% interval) | USD per 1,000 |
|---|---|---|---|
| Jev, average of three setups | in one pass | 79 (0.70 to 0.86) | 0.036 to 0.053 |
| GPT-5.6 Luna, reasoning off | in one pass | 81 (0.72 to 0.87) | 0.166 |
| Gemini 3.5 Flash Lite | in one pass | 76 (0.67 to 0.83) | 0.278 |
| Qwen3.5-4B untuned, thinking on | reasons first | 0.41, LLM judge | - |
| Fine-tuned Qwen3.5-4B | reasons first | 98 (0.93 to 0.99) | 0.028 on a busy H100 |
| GPT-5.6 Luna, reasoning high | reasons first | 100 (0.96 to 1.00) | 0.342 |
| GLM 5.3, reasoning high (the teacher) | reasons first | 96 (0.90 to 0.98) | 0.831 |
With 100 invoices, 0.98 means somewhere between 0.93 and 0.99, and Jev’s three setups moved by 2 to 3 points between identical runs, so the fine-tuned 4B, Luna with reasoning and GLM 5.3 are within noise of each other. The gap to the one-pass group isn’t.
Where the one-pass models fail is the interesting part. They hold approvals whose price is inside the 2% tolerance, and they miss totals that don’t add up. Asking Jev one question per invoice line fixes the first problem (32 of 32 approvals right) and does nothing for the second (0 of 16 wrong totals caught), because a total has no per-line decomposition: you have to add the lines and compare. TypeSafe documents this boundary themselves on the Jev 1.13 jaggedness page: multi-hop questions and extra indirection cost accuracy, dates are read as text, and tasks that need numeric precision are a struggle. So this is the product working as described.
Here is the fine-tuned model on one of the near misses. The invoice, shortened, and the purchase order:
INVOICE NM-84665 DATE 2026-09-28
CUST PO: PO-48825
1 CORD EXT 50FT 12AWG | 5 | 47.30 | 236.50
2 Floor marking tape, yellow | 10 | 14.61 | 146.10
3 48 inch wire shelving unit | 24 | 96.50 | 2,316.00
TOTAL DUE USD | 2,698.60
PREVIOUS BALANCE ON YOUR ACCOUNT: USD 1,290.00 (NOT PART OF THIS INVOICE).
PURCHASE ORDER (ERP)
PO-48825 | Vendor: Northfield Material Handling | Freight: not allowed
1. Extension cord 50 ft, 12 AWG | qty 5 | unit USD 47.30
2. Floor marking tape, yellow | qty 10 | unit USD 14.25
3. Wire shelving unit 48 in | qty 24 | unit USD 96.50And its answer, reasoning included:
Check 1: invoice says PO-48825, PO is PO-48825. Same.
Check 2 (invoiced <= received):
L1 Extension cord 50 ft, 12 AWG: 5 vs 5 ok
L2 Floor marking tape, yellow: 10 vs 10 ok
L3 Wire shelving unit 48 in: 24 vs 24 ok
Check 3 (price <= PO price x 1.02):
L1: PO 47.30, max 48.2460, invoiced 47.30 ok
L2: PO 14.25, max 14.5350, invoiced 14.61 FAIL
Decision: hold_priceThe tape is 2.5% over the PO price. The model matched the abbreviated cord to the PO line, ignored the previous balance, and stopped at the first failing check, which is the policy.
Step 2b: when the output is more than a label, only a model that writes can answer
A bare hold_price isn’t something a clerk can act on. The useful answer names the invoice, the PO, the item and the two values that disagree, and Jev returns a choice, a score or a probability, never a string, so it can’t produce this output at all. For the same invoice the fine-tuned model returns:
{"decision": "hold_price", "invoice_number": "NM-84665", "po_number": "PO-48825", "item": "Floor marking tape, yellow", "invoiced": 14.61, "expected": 14.25}| Model | All six fields right, of 100 (95% interval) | Decision alone | USD per 1,000 |
|---|---|---|---|
| Fine-tuned Qwen3.5-4B | 97 (0.92 to 0.99) | 99 | 0.035 on a busy H100 |
| Qwen3.5-4B untuned, thinking on | 0.12, LLM judge | - | - |
| GPT-5.6 Luna, reasoning high | 100 (0.96 to 1.00) | 100 | 0.377 |
| GLM 5.3, reasoning high (the teacher) | 96 (0.90 to 0.98) | 96 | 1.431 |
| GPT-5.6 Luna, reasoning off | 75 (0.66 to 0.82) | 75 | 0.254 |
| Gemini 3.5 Flash Lite | 76 (0.67 to 0.83) | 76 | 0.487 |
Every model copies the invoice number and the PO number correctly on all 100 invoices; the decision is what separates them, and the pattern is the same as in 2a. All three errors of the fine-tuned model are slips in the running sum on invoices with large line totals (two hold_total decisions with the computed total off by 10 and 20, one false hold on a six-line invoice above USD 16,000), so expect about 1 invoice in 30 to need a second look, and route hold_total decisions on large invoices to a person or recompute the sum in code from the model’s own reasoning lines.
What it costs to run
Accuracy at the level of hosted reasoning models is half the claim; the other half is what it costs to run, and here the two halves point the same way.
| Step | Fine-tuned SLM, one busy H100 | Jev | GPT-5.6 Luna, reasoning high | GLM 5.3, reasoning high | GPT-5.6 Luna, reasoning off | Gemini 3.5 Flash Lite |
|---|---|---|---|---|---|---|
| 1 triage (accuracy) | 0.002 (1.00) | 0.029 (1.00) | 0.093 (1.00) | 0.448 (1.00) | 0.090 (1.00) | 0.138 (0.985) |
| 2a decision (accuracy) | 0.028 (0.98) | 0.036 to 0.053 (0.79) | 0.342 (1.00) | 0.831 (0.96) | 0.166 (0.81) | 0.278 (0.76) |
| 2b grounded decision (accuracy) | 0.035 (0.97) | can’t produce it | 0.377 (1.00) | 1.431 (0.96) | 0.254 (0.75) | 0.487 (0.76) |
USD per 1,000 requests. For the fine-tuned models we served each one with vLLM at default settings (bf16, no quantisation, no tuning) on one H100 rented at USD 3.49 per hour, sent the real test requests at increasing concurrency until throughput stopped rising (34 invoices per second for 2a, 27 for 2b, 492 messages per second for triage, with accuracy under load unchanged), and divided the hourly price by requests per hour. Prefix caching was off for step 2 because the test repeats its 100 invoices and real ones don’t repeat. For Jev and the hosted models the figure is the cost their API reported for the same requests. No margins on either side.
At step 2 the fine-tuned 4B costs 12x less than GPT-5.6 Luna with reasoning at 2a and 11x less at 2b, 29x to 40x less than GLM 5.3, its own teacher, and at 2a it’s cheaper than Jev while scoring 0.98 against 0.79. Two things bound that:
- It’s a busy-GPU figure. An idle GPU costs the same per hour, so cost per request scales with 1 / utilisation. Break-even against Luna with reasoning is about 10,000 invoices per hour on one H100 (2.8 per second, 8% of the GPU); at 40% utilisation the 4B is still 5x cheaper.
- The cheapest point is a batch setting. At full load a request waits in the queue and the median invoice takes 15 s to 19 s end to end. If you need about 5 s, the operating point is concurrency 128, at USD 0.037 (2a) and 0.044 (2b) per 1,000, still about 9x below Luna.
Reasoning length is the lever on all of this: output tokens are most of the cost at 260 to 310 per invoice, and ours are that short because the training data only contains terse reasoning. Beyond price, the models are yours: they run on your hardware or a laptop (Q8_0 GGUF, 197 of 200 unchanged), the version doesn’t change under you, and the invoices don’t leave the building.
How we trained the models
Each of the three models started from a task description, 40 seed examples and a fixed test set that the seeds never overlap: test vendors, items and invoice templates don’t appear in the seeds, and the teacher never sees a test case. For step 2 every number, decision and answer field in the seeds is computed in code, so the examples are exact.
The platform then runs the usual loop. It scores the untrained student first to give you a base (0.70, 0.41 and 0.12 here), sets a teacher LLM as the ceiling (GLM 5.3 with reasoning effort high: 1.00, 0.96 and 0.96), has the teacher generate synthetic examples in the domain (3,124 for triage, 4,156 for 2a and 4,056 for 2b), LoRA fine-tunes the student for 4 epochs on seed plus synthetic data, and evaluates it on the held-out test set. A student can match its teacher on a narrow task because it spends all of its capacity on that task, and here it does: 0.98 against 0.96 at 2a and 0.97 against 0.96 at 2b, both within noise.
The part specific to this demo is teaching a 4B model to reason briefly. Training runs with enable_thinking: true, and every seed answer carries its reasoning in a fixed terse format: the checks in policy order, one short line per invoice line, the 2% ceiling written out, a running sum, stop at the first failure, about 150 tokens in all. The teacher wrote its reasoning in exactly that format for every synthetic example, and the tuned model’s median reasoning is about 160 tokens, which is what keeps step 2 at USD 0.03 per 1,000 instead of several times that. Mutators set the mix of the synthetic data: the decision to produce, how close the numbers sit to the thresholds, invoice layout, line count and order, item naming, freight and stray amounts, order size. Everything needed to repeat it (job descriptions, configs, seeds and test sets) is in the training/ folder of the repo.
Run it yourself
The two models the pipeline uses are on Hugging Face as safetensors and as Q8_0 GGUF. On a laptop with llama.cpp:
hf download distil-labs/distil-qwen3.5-0.8b-invoice-triage-gguf distil-qwen3.5-0.8b-invoice-triage-q8_0.gguf --local-dir models
hf download distil-labs/distil-qwen3.5-4b-invoice-grounded-decision-gguf distil-qwen3.5-4b-invoice-grounded-decision-q8_0.gguf --local-dir models
llama-server -m models/distil-qwen3.5-0.8b-invoice-triage-q8_0.gguf --port 8001 --jinja -c 8192 -np 4
llama-server -m models/distil-qwen3.5-4b-invoice-grounded-decision-q8_0.gguf --port 8002 --jinja -c 16384 -np 4
export TRIAGE_BASE_URL=http://127.0.0.1:8001/v1 TRIAGE_API_KEY=EMPTY
export GROUNDED_BASE_URL=http://127.0.0.1:8002/v1 GROUNDED_API_KEY=EMPTY
python run_pipeline.py --triage slm --limit 10...
IT008 invoice {"decision": "approve", "invoice_number": "CT-12191", "po_number": "PO-51368", "item": null, "invoiced": null, "expected": null}
IT010 invoice {"decision": "hold_no_po", "invoice_number": "HW-80031", "po_number": "PO-67258", "item": null, "invoiced": "PO-67258", "expected": "PO-67285"}
...
step 1: slm | messages: 10 | sent to step 2: 2 | 14 s with 4 in flight
step 1 labels correct: 10/10 | invoices missed: 0 | non-invoices sent to step 2: 0
handled correctly end to end: 10/10
invoices: right decision 2/2 | all six fields right 2/2--triage jev runs step 1 on Jev instead, with a Vercel AI Gateway key in VERCEL_API_KEY. The benchmark scripts behind every table are in benchmarking/, with the raw responses.
- Code: github.com/distil-labs/invoice-processing-pipeline
- Models: distil-qwen3.5-0.8b-invoice-triage, distil-qwen3.5-4b-invoice-decision, distil-qwen3.5-4b-invoice-grounded-decision
If you have a decision in your product that a one-pass model gets wrong, or an output that has to be more than a label, start from the training/ folder closest to your task: a task description and a few dozen examples are enough to train your own. Sign up or read the docs, and we’ll take it from there.
distil labs trains task-specific small language models that are cheaper and faster per request than general-purpose LLMs, with equal-or-better accuracy on bounded tasks. Docs · GitHub · Slack