On-Device LLM Inference in 2026
The headline figure: a fine-tuned 350M-parameter model reached 96–98% tool call equivalence across three benchmarks, matching or beating the 120B teacher it was distilled from. On-device inference stopped being a compromise for narrow tasks; it is still a compromise for broad ones.
What has actually been measured on device?
These are the figures we can point at, each with the hardware and model that produced it. Nothing here is interpolated.
| Model | Hardware / runtime | Measured | Source |
|---|---|---|---|
| FunctionGemma 270M | Phone CPU | 125 tokens/sec, ~288MB footprint | FunctionGemma write-up |
| FunctionGemma 270M, fine-tuned | Single CPU core / in-browser | Latency in the 10–50ms range | Same |
| Llama 3.2 3B | Qualcomm Dragonwing module, llama.cpp + AI Engine SDK across CPU/GPU/NPU | 10.3 tokens/sec | SECO at Embedded World |
| Qwen3-0.6B, fine-tuned | Self-hosted local, voice-assistant brain stage | 40–100ms per turn without network | Voice assistant post |
| 1B model at 4-bit | Phone or tablet | ~500MB on disk | On-device RAG post |
The spread between 10.3 and 125 tokens per second is the important part of this table. Both are real on-device measurements. They differ by roughly 12x because one is a 3B model and the other is 270M, which tells you that model size dominates every other variable on constrained hardware.
How were those numbers produced?
By different teams, on different silicon, measuring different things — which is exactly why they are reported separately rather than averaged.
The SECO figure comes from a demonstration at Embedded World 2025: a chat interface plus a local RAG pipeline running entirely on a Qualcomm Dragonwing system-on-module, with llama.cpp and Qualcomm’s AI Engine SDK distributing work across CPU, GPU and NPU cores. No cloud involved. That is a purpose-built AI board, not a typical embedded controller — our own assessment was that the same prototype would probably not run on the hardware in an average CNC machine today.
The FunctionGemma and Qwen3-0.6B figures are single-stream latency and throughput for much smaller models on ordinary CPUs. The 500MB figure is file size, computed from a 1B parameter count at 4 bits per weight, not a measurement.
Three categories, three meanings. Treat them as such.
What does 350M buy you after fine-tuning?
Accuracy that no untuned model of that size gets anywhere near.
Liquid AI’s LFM2.5-350M is built for structured output and edge deployment, and its card recommends fine-tuning for your specific use case. We did, on three multi-turn tool-calling tasks, against a GPT-oss-120B teacher:
| Task | Teacher (120B) | LFM2.5-350M base | LFM2.5-350M tuned |
|---|---|---|---|
| Shell command execution (Gorilla) | 97.03% | 61.4% | 98.0% |
| Smart home control | 92.11% | 63.2% | 96.7% |
| Banking voice assistant | 96.95% | 34.5% | 95.9% |
The base column is the one to read carefully. At 34.5% on the hardest task, the untuned model is unusable, and multi-turn compounding makes it worse: a 63% per-call rate is about 10% across a five-turn conversation. Fine-tuning is not an optimisation here, it is the difference between a demo and a product. The model itself is covered in LFM2.5 350M for tool calling.
The same pattern holds for retrieval. On 144 questions from a Siemens S7-1200 manual, a Llama 1B model went from 45.1% to 61.1% after distillation, reaching parity with a Llama 3B base at 60.4%.
What changes the numbers on your hardware?
Four things, in descending order of effect.
| Factor | Effect on device throughput |
|---|---|
| Parameter count | Largest. The 12x spread in the first table is mostly this |
| Quantization | Large. Fewer bytes per weight means less memory bandwidth per token |
| Accelerator availability | Large, and binary — NPU/GPU offload or not |
| Context length | Moderate for prefill, and it sets the KV cache allocation |
Memory bandwidth, not arithmetic throughput, is what usually binds generation on a device. This is why quantization helps speed and not just footprint, and why the practical choice of bit width matters — see Q4 vs Q8 vs FP16 for your SLM and what is quantization.
What should you not conclude from this?
That a small model is a small version of a large one.
Every accuracy figure above is on a bounded task with structured output. On open-ended generation the picture reverses, and the model cards say so — Liquid’s own guidance is that LFM2.5-350M is not recommended for knowledge-intensive tasks or programming. In our own benchmark the largest gap to the best frontier model was on docstring generation, 90.1% against 97.4%, precisely because part of that output is free-form prose that needs general world knowledge.
The honest summary for 2026: if your task has a fixed output space, an on-device model fine-tuned for it is a real option with published evidence behind it. If it does not, the device is the wrong place for it. What is a small language model covers where that line sits, and running a small language model on CPU covers how to actually stand one up.