What Is Model Quantization?
Quantization stores a model’s weights using fewer bits each (8-bit or 4-bit integers in place of 16-bit floats). The parameter count is unchanged; only the precision of each number drops. It’s a trade you choose deliberately: less memory and faster loading, against some loss of fidelity.
What actually gets smaller?
Bytes per parameter, and that’s arithmetic rather than an estimate.
| Precision | Bytes per parameter | Relative footprint |
|---|---|---|
| FP32 | 4 | 4x |
| FP16 / BF16 | 2 | 2x |
| INT8 | 1 | 1x |
| 4-bit | 0.5 | 0.5x |
Multiply by the parameter count for the weight footprint. It’s not the whole memory picture (the KV cache grows separately with sequence length), but it’s the part quantization moves.
What do you give up?
Accuracy, in an amount that depends on the format and the task. Rounding every weight introduces error, and the error compounds across layers. Formats differ in how well they contain it: the QLoRA authors introduced 4-bit NormalFloat specifically because a data type matched to normally distributed weights loses less than naive 4-bit rounding, and reported preserving full 16-bit fine-tuning task performance.
Structured outputs tend to be more sensitive than free prose, so if your model emits JSON or tool calls, measure before and after rather than assuming the drop is negligible.
Is it an alternative to distillation?
No, they shrink different things. Quantization keeps every parameter and lowers precision; distillation trains a genuinely smaller model. Distillation vs quantization works through when each applies, and they compose: you can quantize a distilled student.
Quantization also shows up during training, not just after it. LoRA vs QLoRA covers the variant that quantizes the frozen base while adapters train at higher precision. For serving, GGUF is the format most local runtimes expect.