What Is GGUF?
GGUF is a binary file format that stores a model’s weights and everything a runtime needs to use them (tokenizer, chat template, architecture parameters) in one file. It was developed by the author of llama.cpp and is the format llama.cpp, Ollama and LM Studio load.
What does a GGUF file actually contain?
Tensors plus a standardized block of metadata, which is the whole point of the format.
The Hugging Face GGUF documentation draws the contrast directly: unlike tensor-only formats such as safetensors, “GGUF encodes both the tensors and a standardized set of metadata.” That metadata carries the tokenizer, the architecture hyperparameters, and tokenizer.chat_template, which is the critical one for a fine-tuned model.
That last field is why GGUF matters for a distilled SLM. A task-specific model is only correct when its messages are rendered exactly as they were during training. Because the template travels inside the file, a runtime that reads it can reproduce the training-time formatting without you configuring anything. In llama.cpp this is what the --jinja flag turns on; in Ollama the template is picked automatically from the same metadata field.
The format was created by @ggerganov, who also wrote llama.cpp, and is designed for fast load and memory-mapping rather than for training.
How do the quantization type names work?
The name encodes bit width and scheme, and each maps to a documented bits-per-weight figure.
Quantized GGUF weights are stored in blocks with a shared scale, so the real cost per weight is always slightly above the nominal bit width. The Hub documents these:
| Type | Documented bits per weight | Scheme |
|---|---|---|
| F16 / BF16 | 16 | Unquantized half precision |
| Q8_0 | 8-bit, blocks of 32 with a scale | Round-to-nearest, listed as legacy |
| Q6_K | 6.5625 | Super-blocks, 8-bit block scale |
| Q5_K | 5.5 | Super-blocks, 6-bit scale and minimum |
| Q4_K | 4.5 | Super-blocks, 6-bit scale and minimum |
| Q3_K | 3.4375 | Super-blocks, 6-bit block scale |
| Q2_K | 2.625 | Super-blocks, 4-bit scale and minimum |
| IQ4_XS | 4.25 | Importance-matrix weighted |
Figures are quoted from the Hub’s quantization table. The _K suffix means k-quants, which group weights into super-blocks; the IQ prefix means the quantization used an importance matrix to decide which weights to protect. Suffixes such as _M and _S that you see on filenames (Q4_K_M, Q4_K_S) select mixes that keep some tensors at higher precision, which is why a Q4_K_M file is a little larger than 4.5 bits per weight would suggest.
Q4_K_M is the common default: when Ollama pulls a GGUF repo from the Hub without a tag, it picks Q4_K_M if present.
When does GGUF matter for a fine-tuned SLM?
Whenever the model leaves a GPU server.
Models trained on distil labs can be pushed to a private Hugging Face repository, which produces two repos, one GGUF and one safetensors. The split maps onto the runtimes. Safetensors is what vLLM serves on a GPU; GGUF is what llama.cpp, Ollama and CPU-only deployments load. distil model deploy local takes the GGUF path: it extracts model.gguf from the downloaded tarball and hands it to llama-server.
Size is why edge deployment works at all. A 1B model at 4-bit lands around 500MB, small enough that our on-device RAG write-up describes it as trivial for a modern phone or tablet.
What is GGUF not good for?
Training, adapters, and high-concurrency GPU serving.
GGUF is an inference format. You can’t fine-tune a GGUF file. Training happens on the safetensors weights, and LoRA adapters are merged before conversion. It also isn’t the format a batching GPU server wants: vLLM’s strengths are continuous batching and paged KV cache handling over standard weights, and routing that through a quantized single-file format gives up most of the reason to run it.
There’s also a conversion cost. Getting from Hugging Face weights to GGUF runs through llama.cpp’s convert_hf_to_gguf.py and then llama-quantize, and a model architecture that llama.cpp hasn’t implemented yet simply can’t be converted. Newly released architectures often reach vLLM before they reach GGUF.
Related terms
- Quantization. The operation that produces the
Q*variants. See what is quantization. - Q4 / Q8 / FP16. The practical choice between them is in Q4 vs Q8 vs FP16 for your SLM.
- safetensors. The tensor-only counterpart used for GPU serving and training.
- Distillation vs quantization. Two different ways to shrink a model, compared in distillation vs quantization.