What Is GGUF?
GGUF is a binary file format that stores a model’s weights and everything a runtime needs to use them — tokenizer, chat template, architecture parameters — in one file. It was developed by the author of llama.cpp and is the format llama.cpp, Ollama and LM Studio load.
What does a GGUF file actually contain?
Tensors plus a standardized block of metadata, which is the whole point of the format.
The Hugging Face GGUF documentation draws the contrast directly: unlike tensor-only formats such as safetensors, “GGUF encodes both the tensors and a standardized set of metadata.” That metadata carries the tokenizer, the architecture hyperparameters, and — critically for a fine-tuned model — tokenizer.chat_template.
That last field is why GGUF matters for a distilled SLM. A task-specific model is only correct when its messages are rendered exactly as they were during training. Because the template travels inside the file, a runtime that reads it can reproduce the training-time formatting without you configuring anything. In llama.cpp this is what the --jinja flag turns on; in Ollama the template is picked automatically from the same metadata field.
The format was created by @ggerganov, who also wrote llama.cpp, and is designed for fast load and memory-mapping rather than for training.
How do the quantization type names work?
The name encodes bit width and scheme, and each maps to a documented bits-per-weight figure.
Quantized GGUF weights are stored in blocks with a shared scale, so the real cost per weight is always slightly above the nominal bit width. The Hub documents these:
| Type | Documented bits per weight | Scheme |
|---|---|---|
| F16 / BF16 | 16 | Unquantized half precision |
| Q8_0 | 8-bit, blocks of 32 with a scale | Round-to-nearest, listed as legacy |
| Q6_K | 6.5625 | Super-blocks, 8-bit block scale |
| Q5_K | 5.5 | Super-blocks, 6-bit scale and minimum |
| Q4_K | 4.5 | Super-blocks, 6-bit scale and minimum |
| Q3_K | 3.4375 | Super-blocks, 6-bit block scale |
| Q2_K | 2.625 | Super-blocks, 4-bit scale and minimum |
| IQ4_XS | 4.25 | Importance-matrix weighted |
Figures are quoted from the Hub’s quantization table. The _K suffix means k-quants, which group weights into super-blocks; the IQ prefix means the quantization used an importance matrix to decide which weights to protect. Suffixes such as _M and _S that you see on filenames — Q4_K_M, Q4_K_S — select mixes that keep some tensors at higher precision, which is why a Q4_K_M file is a little larger than 4.5 bits per weight would suggest.
Q4_K_M is the common default: when Ollama pulls a GGUF repo from the Hub without a tag, it picks Q4_K_M if present.
When does GGUF matter for a fine-tuned SLM?
Whenever the model leaves a GPU server.
Models trained on distil labs can be pushed to a private Hugging Face repository, which produces two repos — one GGUF and one safetensors. The split is the useful mental model. Safetensors is what vLLM serves on a GPU; GGUF is what llama.cpp, Ollama and CPU-only deployments load. distil model deploy local takes the GGUF path: it extracts model.gguf from the downloaded tarball and hands it to llama-server.
The size effect is the reason edge deployment works at all. A 1B model at 4-bit lands around 500MB, small enough that our on-device RAG write-up describes it as trivial for a modern phone or tablet.
What is GGUF not good for?
Training, adapters, and high-concurrency GPU serving.
GGUF is an inference format. You cannot fine-tune a GGUF file — training happens on the safetensors weights, and LoRA adapters are merged before conversion. It also is not the format a batching GPU server wants: vLLM’s strengths are continuous batching and paged KV cache handling over standard weights, and routing that through a quantized single-file format gives up most of the reason to run it.
There is also a conversion cost. Getting from Hugging Face weights to GGUF runs through llama.cpp’s convert_hf_to_gguf.py and then llama-quantize, and a model architecture that llama.cpp has not implemented yet simply cannot be converted. Newly released architectures often reach vLLM before they reach GGUF.
Related terms
- Quantization — the operation that produces the
Q*variants. See what is quantization. - Q4 / Q8 / FP16 — the practical choice between them is in Q4 vs Q8 vs FP16 for your SLM.
- safetensors — the tensor-only counterpart used for GPU serving and training.
- Distillation vs quantization — two different ways to shrink a model, compared in distillation vs quantization.