Running a Small Language Model on CPU
CPU inference is not a fallback for people who could not get a GPU. It is the deployment that lets a model ship inside hardware that already exists — a controller, a handheld, a laptop, an on-prem box with no accelerator budget — and for a task-specific SLM it is frequently fast enough.
When does CPU inference make sense?
When the model is small, the task is narrow, and adding a GPU to the bill of materials is not an option.
Our on-device RAG write-up frames the trade the way a hardware team would: the premium path is an AI-optimised board with GPU and NPU cores running a 3B base model; the cost-effective path is a commodity CPU running a smaller fine-tuned model. Both produce working RAG. Only one fits a mid-range bill of materials.
The fine-tuning is what makes the second path viable. In that study a Llama 1B model went from 45.1% to 61.1% accuracy on 144 questions from a Siemens S7-1200 manual after distillation, reaching parity with a Llama 3B base model at 60.4% — a 1B model doing a 3B model’s job, on hardware that costs a fraction as much.
What you need before you start
Enough RAM for the quantized weights plus the KV cache, and a runtime that targets CPU well.
That runtime is llama.cpp in practice. It is a plain C/C++ implementation with CPU backends including BLAS, BLIS and ZenDNN, and it treats CPU as a first-class target rather than a degraded mode. The full recipe for the server is in run a fine-tuned SLM with llama.cpp; this article is about the CPU-specific decisions.
Rough memory guide, all of it plain arithmetic on parameter count:
| Model size | 4-bit weights | 8-bit weights | Practical RAM to have free |
|---|---|---|---|
| 270M–350M | ~0.14–0.18 GB | ~0.27–0.35 GB | 1 GB |
| 1B | ~0.5 GB | ~1 GB | 2 GB |
| 3B | ~1.5 GB | ~3 GB | 4–8 GB |
| 8B | ~4 GB | ~8 GB | 12 GB+ |
These are computed as parameters times bytes per parameter, not measured; the full working, including where the extra headroom goes, is in how much VRAM does a 1B, 3B or 8B model need and which SLM fits in 4GB of VRAM. The 500MB figure for a 4-bit 1B model matches what we quote for phone and tablet deployment.
Step 1: Pick a model that fits, then fine-tune it
Choose the smallest student that clears your quality bar, because on CPU every parameter is paid for in wall-clock time on every token.
The sub-1B tier is where CPU deployment becomes comfortable. FunctionGemma 270M runs on a phone CPU at 125 tokens per second with a roughly 288MB footprint, and after fine-tuning we measured it at 96.04% on shell-command tool calling against a 120B teacher’s 97.03%. LFM2.5-350M is designed for the same envelope and reached 96–98% tool call equivalence across three benchmarks after fine-tuning.
Both numbers depend on the fine-tuning. Base models at this size are not usable for these tasks; that is the whole point of distilling a task-specific model.
Step 2: Quantize to 4-bit
On CPU, quantization buys you two things at once: the model fits, and it runs faster, because CPU generation is bound by memory bandwidth rather than arithmetic. Fewer bytes per weight means fewer bytes to stream per token.
Q4_K_M is the usual landing spot. The format and its bits-per-weight figures are in what is GGUF, and the accuracy trade is in Q4 vs Q8 vs FP16. Establish your accuracy at full precision first so you can tell what quantization cost you.
Step 3: Run it with threads matched to your cores
llama-server -m model.gguf --jinja -t 8 -c 4096 --port 8000-t sets CPU threads and -c sets the context size. Two rules of thumb, both worth testing rather than assuming: set -t to the number of physical cores, not logical threads — hyperthreads usually cost more in contention than they return. And set -c to what your prompts actually need, because the KV cache is allocated against it and on a memory-constrained board that allocation is the difference between running and not.
Keep --jinja so the chat template embedded in the GGUF is used. If you deployed through distil model deploy local, the CLI passes it for you.
Step 4: Measure tokens per second, not benchmark scores
Decide viability from generation speed on your hardware with your prompt lengths, because that is the number that varies most and the one nobody can give you in advance.
Two published reference points, each with its setup attached. At the low end of the hardware range, SECO demonstrated an on-device assistant at 10.3 tokens per second using llama.cpp and Qualcomm’s AI Engine SDK to spread work across CPU, GPU and NPU cores on a Dragonwing module. At the small-model end, FunctionGemma’s 125 tokens per second is a phone-CPU figure for a 270M model. The gap between those two numbers is roughly the whole design space, and it is driven by model size far more than by the chip.
Prefill and decode behave differently, so measure both: time to first token tells you how the prompt length hurts, tokens per second tells you how the answer length hurts.
What you give up
Concurrency, mainly. A CPU deployment serves one request at a time acceptably and several badly — if you need to fan out, that is a vLLM or managed-endpoint problem, discussed in self-hosted vs managed inference.
You also give up headroom for growth. A CPU deployment sized for a 1B model has no path to a 4B one without new hardware, which makes the sizing decision harder to reverse than it is on a GPU.