Run a Fine-Tuned SLM with llama.cpp
llama.cpp is the right runtime when the model has to run somewhere you do not control: a laptop, a handheld, an embedded board, a machine with no Python. It is a C/C++ binary with no dependencies that loads a single quantized file and serves an OpenAI-compatible API.
When is llama.cpp the right runtime?
When the deployment target is a device rather than a server.
The project’s stated goal is “LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware — locally and in the cloud”, and that is exactly the property you are buying. One binary, one .gguf file, no virtualenv, no CUDA install, and backends for CPU, CUDA, Metal, Vulkan, HIP and SYCL from the same source tree. Apple silicon is a first-class target.
What you give up is serving throughput. llama.cpp will handle a request at a time comfortably and concurrency poorly compared to a batching server — if you are fronting real traffic, vLLM is the other half of this decision. If you want model management and a pull-by-name workflow on top of the same engine, that is Ollama.
What you need before you start
Three things: a trained model, the llama-server binary on your PATH, and uv if you want to use the generated client script.
brew install llama.cpp # macOSBinaries are also published on the releases page, and the project builds from source on Linux and Windows. Confirm the install with llama-server --help before going further — the distil CLI looks up llama-server in your PATH and stops with an error if it is not there.
Step 1: Deploy the model locally
distil model deploy local <model-id>This downloads your trained model, extracts the GGUF from the tarball into the CLI’s cache, and starts a local llama-server. The server listens on port 8000 and exposes the OpenAI-compatible API at http://localhost:8000/v1.
Two options are worth knowing on the first run:
distil model deploy local --port 9000 --logs <model-id>--logs surfaces the server’s own output, which is where the useful diagnostics live if the model fails to load. Local deployment is marked experimental in the local deployment docs.
Step 2: Understand what the CLI is running for you
Under the hood the CLI spawns roughly this:
llama-server -m /path/to/model.gguf --jinja --port 8000--jinja is the flag that matters. It tells llama-server to apply the chat template embedded in the GGUF’s metadata rather than a built-in default. A fine-tuned SLM was trained against one specific template; render its messages with a different one and the model will answer, badly, with nothing in the logs to indicate why. If you ever run llama-server by hand against a distil labs GGUF, carry --jinja across.
The other flags you are most likely to add, from the server documentation:
| Flag | What it does | When you need it |
|---|---|---|
-ngl, --gpu-layers N |
Offload N layers to the GPU | Metal, CUDA or Vulkan is available |
-t, --threads N |
CPU threads | CPU-only boxes — match physical cores |
-c, --ctx-size N |
Context window in tokens | Long prompts, RAG contexts |
--host HOST |
Bind address | Serving to another machine on the LAN |
Note that llama-server’s own default port is 8080; the 8000 you see comes from the distil CLI passing --port.
Step 3: Query it
distil model invoke <model-id>That prints a ready-to-run command pointing at the cached client script:
uv run $PATH_TO_CLIENT --conversation '[{"role": "user", "content": "Your question here"}]'For an open-book question-answering model, the retrieved context goes inside a <context> tag followed by a newline, inside the first user message:
uv run $PATH_TO_CLIENT --conversation '[{"role": "user", "content": "<context>Your context here</context>\nYour question here"}]'Use the generated script rather than writing your own request. It encodes the system prompt and message formatting the model was trained on, and the docs are blunt about the consequence of deviating: a different system prompt or formatting gives you poor performance, silently.
Step 4: Verify it is serving your model
Three checks, cheapest first.
curl http://localhost:8000/health
curl http://localhost:8000/v1/models/health returns {"status": "ok"} once the model is loaded — the CLI polls this itself and gives up after 60 seconds. /v1/models confirms which model is actually loaded. Then run your held-out test set through the endpoint and compare with the score from teacher evaluation and training metrics. If accuracy is materially below what the platform reported, suspect the template before you suspect the model.
What breaks, and how it looks
The model loads but answers generically. Almost always the chat template. Check --jinja is set.
llama-server exits immediately. Re-run with --logs. Out-of-memory on a large context is the usual cause; drop -c.
It is slower than you expected on a GPU machine. No layers were offloaded. Set -ngl explicitly.
Throughput collapses under concurrent callers. Working as designed. This is the case that wants vLLM, or a managed endpoint — see self-hosted vs managed inference.
If you want to understand the file you are loading, what is GGUF covers the format and its quantization types.