What Is Quantization?
Quantization stores each model weight using fewer bits — 8 or 4 instead of 16 — so the model occupies less memory and moves fewer bytes per token. The architecture is unchanged: same layers, same parameter count, lower precision.
How does quantization actually work?
By mapping a group of high-precision weights onto a small set of integers, plus a scale that converts them back.
The unit is the block, not the individual weight. A block of weights shares one scaling factor, so a 4-bit quantized weight is reconstructed as roughly w = q × block_scale, sometimes with a block minimum as well. The Hub’s GGUF quantization table spells this out per type — Q4_K, for instance, uses super-blocks of 8 blocks of 32 weights with a 6-bit scale and minimum, landing at a documented 4.5 bits per weight rather than a clean 4.
That overhead is why quantized file sizes never hit the naive arithmetic exactly. Blocks need scales, and scales are stored.
Two families do the compression. Post-training quantization takes finished weights and converts them, optionally calibrating on a small sample to choose scales that minimise error — GPTQ is the canonical example. Quantization-aware training simulates the rounding during training so the model adapts to it, which costs a training run. For a distilled SLM you are almost always in the first camp: the model is already trained, and you quantize the artefact.
A related but distinct idea is LLM.int8(), which keeps a small number of outlier dimensions at higher precision because they carry disproportionate signal. The general shape of every good quantization method is the same — spend bits where they matter, save them where they do not.
What does each bit width cost?
Memory falls roughly linearly with bit width, and accuracy falls non-linearly.
| Precision | Bytes per parameter | Weights for a 4B model | Typical accuracy effect |
|---|---|---|---|
| FP32 | 4 | 16 GB | Reference, rarely used for serving |
| FP16 / BF16 | 2 | 8 GB | The baseline you evaluate against |
| INT8 / Q8 | 1 | 4 GB | Small, often not measurable on a narrow task |
| 4-bit (Q4) | ~0.5 | ~2 GB | Measurable but frequently acceptable |
| Below 4-bit | <0.5 | <2 GB | Degrades noticeably, especially on reasoning |
The weight column is parameter count times bytes per parameter — arithmetic, not measurement. Real files sit slightly above these figures because of block scales.
The non-linearity is the useful part. FP16 to 8-bit is close to free on most tasks. 8-bit to 4-bit costs something you can measure, and buys the ability to run a model on hardware that could not otherwise hold it. Below 4-bit the returns invert.
When does quantization matter for a fine-tuned SLM?
Whenever memory or memory bandwidth is the binding constraint, which on a device is always.
Generation speed on CPUs and small accelerators is bound by how fast weights can be streamed from memory, not by arithmetic. Halving the bytes per weight therefore buys throughput as well as footprint. This is why CPU deployments are almost always quantized and why the effect is larger there than on a big GPU.
There is a second, less obvious reason quantization is friendlier to a distilled model than to a general-purpose one. A task-specific model only has to do one thing. Quantization damage tends to appear first in capabilities that are not being exercised, so the loss on your narrow task is often smaller than a general benchmark would predict — but you only know that by measuring on your own test set.
We have one measured data point on the GPU side: in brief experiments during our inference-cost benchmark, FP8 quantization gave roughly 15% more throughput with 44% less memory and no measurable accuracy loss on the tasks tested. That study otherwise kept every model at BF16.
Where does it go wrong?
Three predictable places.
Quantizing before you have a baseline. If you never measured the FP16 score on your test set, you cannot attribute a bad number to quantization rather than to the model. Measure first, always.
Going below 4-bit to save a gigabyte. The accuracy cost climbs sharply and the memory saved is small in absolute terms at SLM scale. A 1B model at 4-bit is already about 500MB.
Confusing quantization with distillation. They shrink different things — precision versus architecture — and they compose rather than substitute. Distillation vs quantization covers the pipeline where you do both.
Related terms
- GGUF — the file format that carries quantized weights for llama.cpp-family runtimes: what is GGUF.
- Q4, Q8, FP16 — the practical choice between the three: Q4 vs Q8 vs FP16 for your SLM.
- QLoRA — quantizing the base model during training rather than for serving: LoRA vs QLoRA.
- VRAM sizing — what the saved bytes buy you in hardware terms: how much VRAM does a 1B, 3B or 8B model need.