The reason to train a small model is usually that you want to run it somewhere a frontier API cannot go: inside your own network, on a device, under a latency budget that a network round trip already blows. Deployment is where that pays off, and it is a genuinely different set of concerns from training.
Pick the runtime for the constraint
The three common runtimes solve different problems.
llama.cpp runs quantized models on CPU or GPU with minimal dependencies. It is the right answer for on-device, edge, and air-gapped deployments, and for anything that has to ship as a single binary.
Ollama wraps llama.cpp with model management and a local HTTP API. Best for local development and internal tools where convenience matters more than throughput.
vLLM is built for serving throughput — continuous batching, efficient KV cache handling, high concurrency. It is the right choice for a service handling real request volume, and overkill for a laptop.
The decision is usually made for you by where the model has to run rather than by preference.
Quantization is the main lever
Quantization reduces the numerical precision of the weights. A model stored at 4 bits per weight is roughly a quarter the size of the same model at 16 bits, and correspondingly faster to load and cheaper to keep in memory.
The trade is accuracy, and the shape of that trade is not linear. Dropping from FP16 to 8-bit is close to free on most tasks. Dropping to 4-bit costs something measurable but often acceptable, and it is what makes a 4B model fit on hardware that could not otherwise hold it. Below 4-bit the degradation becomes hard to ignore.
For a task-specific fine-tuned model the calculus is friendlier than for a general-purpose one: the model only has to do one thing, and quantization damage tends to show up first in capabilities you are not using.
Sizing
The rough working figure for VRAM is the parameter count times bytes per parameter, plus overhead for the KV cache, which grows with context length and batch size. A 1B model at 4-bit needs well under a gigabyte of weights; the same model at FP16 needs a couple. This is why the sub-1B students are interesting for device deployment — they leave room for everything else the device is doing.
CPU inference is viable for small models and genuinely useful when no GPU is available, with the understanding that throughput will be a fraction of GPU inference.
Latency is a product constraint, not just a cost
For anything interactive — a voice assistant, an autocomplete, an agent loop that makes several calls per user action — latency is the feature. A model that is accurate but takes two seconds fails a voice interaction regardless of its benchmark score. This is often the real reason to move off a frontier API, ahead of cost.
Where this cluster goes
The articles below cover each runtime with working commands, GGUF and the quantization formats, VRAM requirements by model size, realistic latency figures, the choice between self-hosted and managed inference, and air-gapped deployment.