Which SLM Fits in 4GB of VRAM?
Leaving a quarter of the card free for KV cache and activations, a 4 GB GPU holds about 1.5B parameters at BF16, 3B at INT8, and 6B at 4-bit. Every figure below is computed from published parameter counts, not measured on hardware.
What is the arithmetic?
Weight storage is parameter count times bytes per parameter. Nothing else.
weight bytes = parameters × bytes per parameter| Precision | Bytes per parameter | Params that fit in a 3 GB weight budget |
|---|---|---|
| FP32 | 4 | 3 ÷ 4 = 0.75B |
| BF16 / FP16 | 2 | 3 ÷ 2 = 1.5B |
| INT8 | 1 | 3 ÷ 1 = 3.0B |
| 4-bit (Q4) | ~0.5 | 3 ÷ 0.5 = 6.0B |
The 3 GB budget is 75% of a 4 GB card, leaving 1 GB for everything else. Two conventions to keep straight: 1 GB here means 10⁹ bytes, so a tool that reports GiB (2³⁰ bytes) will show the same weights as roughly 7% smaller. And 4-bit formats carry per-block scales, so real GGUF files land slightly above the 0.5 bytes-per-parameter ideal.
What fits in 4 GB?
Every student in the distil labs catalog, with weight size computed at each precision. Bold marks the models whose weights stay inside the 3 GB budget at BF16.
| Model | Params | BF16 (×2) | INT8 (×1) | 4-bit (×0.5) |
|---|---|---|---|---|
| SmolLM2 135M | 0.135B | 0.27 GB | 0.14 GB | 0.07 GB |
| Gemma 3 270M / FunctionGemma 270M | 0.27B | 0.54 GB | 0.27 GB | 0.14 GB |
| LFM2.5 350M | 0.35B | 0.70 GB | 0.35 GB | 0.18 GB |
| Qwen3 0.6B | 0.6B | 1.20 GB | 0.60 GB | 0.30 GB |
| Qwen3.5 0.8B | 0.8B | 1.60 GB | 0.80 GB | 0.40 GB |
| Gemma 3 1B | 1.0B | 2.00 GB | 1.00 GB | 0.50 GB |
| LFM2.5 1.2B | 1.2B | 2.40 GB | 1.20 GB | 0.60 GB |
| Llama 3.2 1B | 1.23B | 2.46 GB | 1.23 GB | 0.62 GB |
| Qwen3 1.7B / SmolLM2 1.7B | 1.7B | 3.40 GB | 1.70 GB | 0.85 GB |
| Qwen3.5 2B | 2.0B | 4.00 GB | 2.00 GB | 1.00 GB |
| LFM2 2.6B | 2.6B | 5.20 GB | 2.60 GB | 1.30 GB |
| Llama 3.2 3B | 3.0B | 6.00 GB | 3.00 GB | 1.50 GB |
| Qwen3 4B / Qwen3.5 4B / Gemma 3 4B | 4.0B | 8.00 GB | 4.00 GB | 2.00 GB |
| Gemma 4 E2B | 5.1B total | 10.20 GB | 5.10 GB | 2.55 GB |
| Qwen3 8B / Gemma 4 E4B | 8.0B | 16.00 GB | 8.00 GB | 4.00 GB |
| Qwen3.5 9B | 9.0B | 18.00 GB | 9.00 GB | 4.50 GB |
Parameter counts come from the catalog and the vendors’ model cards; Llama-3.2-1B-Instruct reports 1.23B, which is why it is not exactly 1.0B. Gemma 4 is listed at its total count rather than its effective count, because storage follows the total — see Gemma 4 E2B and E4B explained.
Read the bottom of the table carefully: Qwen3 8B at 4-bit is 4.00 GB of weights, which exactly consumes a 4 GB card and leaves nothing for anything else. It does not fit.
What else uses VRAM besides weights?
Three things, and they are why the budget above is 3 GB and not 4 GB.
- KV cache, which grows linearly with sequence length and with the number of key-value heads and layers. This is the one that surprises people: a model that fits empty will not fit at a 32K context.
- Activations for the forward pass, which scale with batch size.
- Runtime overhead — the CUDA context, the framework, and any paging allocator headroom.
Architecture changes the KV term sharply. Grouped-query attention shrinks it: Qwen3-1.7B uses 8 key-value heads against 16 query heads across 28 layers. Liquid’s LFM2.5 shrinks it further by replacing most attention layers with convolution blocks that hold no cache at all, which is why Liquid quotes the 350M model running under 1 GB of memory.
Does fine-tuning fit in 4 GB too?
Not the way inference does, and on distil labs you do not need it to — training runs on platform infrastructure, so your 4 GB card only has to serve the finished model.
If you do train locally, two settings in the config reference exist for exactly this. use_qlora loads the base model in 4-bit NF4 and attaches LoRA adapters in higher precision, cutting base-model VRAM by roughly 3x at the cost of slower training. memory_optimized_training adds activation offloading and gradient checkpointing, which the docs describe as significantly slower — enable it only after you have actually run out of memory. Background on the adapter choice is in LoRA vs full fine-tuning, and the no-GPU path is in fine-tune an LLM without a GPU.
Which model should you pick for a 4 GB card?
Pick the largest model whose weights sit at or under 3 GB in the precision you are willing to ship, then verify with your real context length.
At BF16 that is Qwen3-1.7B’s ceiling territory — 3.40 GB, already over budget — so BF16 realistically caps you around Llama 3.2 1B or LFM2.5 1.2B. At INT8 you reach Llama 3.2 3B exactly. At 4-bit you can run Qwen3 4B at 2.00 GB with 2 GB spare, which is the best accuracy-per-gigabyte position on this table given that Qwen3-4B topped our 12-model fine-tuned ranking.
Quantizing the student is a different lever from shrinking it — distillation vs quantization covers why they compose, and what size model do you need covers picking the parameter count in the first place.