Q4 vs Q8 vs FP16 for Your SLM
Serve at FP16 if the model fits and you have the GPU. Drop to Q8 when memory is tight and you cannot afford a measurable accuracy question. Drop to Q4 when the model has to fit hardware that FP16 cannot reach — which, on a device, it usually cannot.
How do the three precisions compare?
Bits per weight is the whole comparison, and it propagates directly into file size, memory bandwidth and speed.
| FP16 / BF16 | Q8 | Q4 | |
|---|---|---|---|
| Bits per weight | 16 | ~8.5 for Q8_0 |
4.5 for Q4_K |
| Bytes per parameter | 2 | ~1 | ~0.5 |
| 1.7B model weights | 3.4 GB | 1.7 GB | 0.85 GB |
| 4B model weights | 8.0 GB | 4.0 GB | 2.0 GB |
| 8B model weights | 16.0 GB | 8.0 GB | 4.0 GB |
| Accuracy vs FP16 baseline | Reference | Usually not measurable on a narrow task | Measurable, often acceptable |
| Typical use | GPU serving | GPU serving under memory pressure | CPU, edge, on-device |
The bits-per-weight column is documented, not estimated. The Hub’s GGUF quantization table gives Q4_K as 4.5 bits per weight: super-blocks of 8 blocks of 32 weights, with a 6-bit scale and a 6-bit minimum per block. Q8_0 is documented as 8-bit round-to-nearest over blocks of 32 with a shared scale, which works out to (32 × 8 + 16) ÷ 32 = 8.5 bits per weight if the scale is stored at 16 bits. The model-size rows are parameter count times bytes per parameter — arithmetic, not measurements on hardware.
Two things the table does not show. Filename suffixes like Q4_K_M and Q4_K_S select mixes that keep some tensors at higher precision, so a real Q4_K_M file runs slightly above 4.5 bits per weight. And every figure here is weights only; the KV cache is separate and grows with context, which is covered in how much VRAM does a 1B, 3B or 8B model need.
What is FP16 better at?
Being the number you compare everything else against.
Full half precision is what the platform evaluated your model at, so it is the only precision where the reported accuracy is directly yours. Every quantized deployment is a deviation from a baseline, and if you never ran the baseline you cannot attribute a disappointing score to the quantization rather than to the model.
It is also the practical default on a GPU with room. In our inference-cost benchmark every fine-tuned model was served at BF16 through vLLM, and a 4B model occupied 7.6 GiB of GPU memory while sustaining 222 requests per second on a single H100 — memory was not the constraint, so there was no reason to quantize.
FP16 is the wrong choice the moment the model does not fit. An 8B model at FP16 needs 16 GB of weights before the cache, which rules out most consumer cards and every device.
What is Q8 better at?
Halving memory for an accuracy cost you will usually struggle to measure on a task-specific model.
This is the low-risk step. Going from 16 bits to 8 keeps enough resolution that the quantization error stays well below the noise of a typical evaluation on a narrow task, and it turns an 8B model from a 16 GB problem into an 8 GB one. If you are memory-constrained but nervous, Q8 is where to be.
The closest thing we have to a measurement is adjacent rather than exact: in brief experiments in the same benchmark, FP8 quantization delivered roughly 15% more throughput with 44% less memory and no measurable accuracy loss on the tasks tested. FP8 is a floating-point 8-bit format rather than integer Q8, so treat this as an indication of the shape of the trade at 8 bits, not as a Q8 result.
Q8 is a poor fit for genuinely small deployments. At 350M parameters the difference between Q8 and Q4 is a couple of hundred megabytes, and if you are on a device you generally want the bandwidth saving more than the precision.
What is Q4 better at?
Making deployment possible at all on hardware that has no GPU.
Below about 8 GB of usable memory, Q4 stops being an optimisation and becomes the enabling condition. A 1B model at 4-bit is roughly 500MB, small enough to sit on a phone; a 4B model at 4-bit is about 2 GB, which fits a card that could not hold it at FP16 with any room to spare. On CPU the win is doubled, because generation is bound by memory bandwidth and fewer bytes per weight means fewer bytes streamed per token. The full setup is in running a small language model on CPU.
Q4_K_M is the default an ecosystem has converged on — Ollama picks it when pulling a GGUF repo without a tag.
The cost is real and you should expect to see it. 4-bit is the point where accuracy degradation becomes measurable rather than theoretical, and calibrated methods such as GPTQ exist specifically because naive rounding at this width loses more than it needs to. Below 4-bit, degradation becomes hard to ignore for very little further saving.
Which should you pick?
Start at FP16, measure, then descend only as far as your hardware forces you.
The procedure is three steps and it is short on purpose. Evaluate at FP16 on your own held-out set and record the number — this is your baseline, and the platform’s evaluation metrics are where it comes from. Quantize to the widest format that fits your target hardware. Re-run the same test set and compare. If the drop is inside your tolerance, ship it; if not, step back up a level or accept a smaller model at a wider precision.
That last option is the one people forget. A 1.7B model at Q8 and a 4B model at Q4 occupy about the same 1.7–2 GB. Which one wins is an empirical question about your task, not a rule — and what size model do you need plus which SLM fits in 4GB of VRAM frame both halves of it. For the mechanism behind all of this, see what is quantization; for the file format that carries it, what is GGUF.