Deploy an SLM with Ollama

Ollama is the runtime you pick when you want a named, versioned model you can pull and run without thinking about files. It wraps the same llama.cpp engine, adds a registry and a persistent local daemon, and trades a little control for a lot of convenience.

When is Ollama the right runtime?

When a human or a local app is calling the model and you want model management for free.

The engine underneath is llama.cpp, so raw single-request performance is broadly the same. What Ollama adds is the part llama.cpp deliberately leaves out: models addressed by name rather than by path, a daemon that stays up and loads models on demand, automatic unloading, and a registry you can pull from — including Hugging Face directly. That makes it the natural fit for local development, internal tools, and demos on a colleague’s machine.

What it costs you is explicitness. Ollama decides the chat template, the quantization tag and the default parameters unless you override them, and for a fine-tuned SLM those defaults are exactly the things that must not drift. If you need to control every flag, run llama.cpp directly; if you need concurrency, run vLLM.

What you need before you start

Ollama installed and running, and your model as a GGUF file — either downloaded locally or pushed to Hugging Face.

Models trained on distil labs can be pushed to a private Hugging Face repository through the API, which creates one GGUF repo and one safetensors repo. The GGUF repo is the one Ollama wants. Alternatively, distil model deploy local <model-id> downloads and caches a model.gguf you can point a Modelfile at.

Step 1 (fast path): Pull straight from Hugging Face

ollama run hf.co/YOUR_USERNAME/MODEL_NAME

Any GGUF repo on the Hub works with this syntax, per the Hugging Face integration docs, and huggingface.co works in place of hf.co. To choose a specific quantization rather than letting Ollama decide, add the tag:

ollama run hf.co/YOUR_USERNAME/MODEL_NAME:Q8_0

Without a tag Ollama defaults to Q4_K_M when the repo contains one. That default is a real decision, not a formality — see Q4 vs Q8 vs FP16 for your SLM before accepting it for a model you evaluated at full precision.

For a private repo, add your Ollama public key (~/.ollama/id_ed25519.pub) to your Hugging Face account first.

Step 2 (controlled path): Write a Modelfile

For a local GGUF, or when you want to pin the system prompt and sampling parameters, create a file named Modelfile:

FROM ./model.gguf

PARAMETER temperature 0

SYSTEM """The exact system prompt your model was trained with."""

The FROM path is absolute or relative to the Modelfile itself, per the Modelfile reference. TEMPLATE is available too, and this is the one to be careful about: Ollama templates are Go templates, not the Jinja template stored inside the GGUF. Ollama will select a template automatically from the file’s tokenizer.chat_template metadata, and for a distil labs GGUF that is normally correct — so only write TEMPLATE if you have confirmed the automatic choice is wrong.

PARAMETER temperature 0 matters more than it looks. Fine-tuned task models are evaluated at temperature 0; leaving a creative default in place will make your reproduction of the platform’s score fail for reasons that have nothing to do with deployment.

Step 3: Create and run the model

ollama create my-slm -f ./Modelfile
ollama run my-slm

ollama create registers the model under a local name. From then on my-slm is how you refer to it everywhere, and the daemon loads and unloads it as needed.

Step 4: Call the OpenAI-compatible endpoint

Ollama serves an OpenAI-compatible API on port 11434:

from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:11434/v1/',
    api_key='ollama',
)

/v1/chat/completions, /v1/completions, /v1/models and /v1/embeddings are all available, so an application already written against an OpenAI client needs only the base URL and model name changed. The api_key value is required by the client library and ignored by Ollama.

Runtime	Default port	Model addressed by
Ollama	11434	Registered name (my-slm)
llama.cpp via distil CLI	8000	GGUF file path
vLLM	8000	Weights directory or HF repo id

Step 5: Verify against your test set

Run the same held-out examples you evaluated on the platform and compare the score against the reported evaluation metrics.

This step is not optional with Ollama specifically, because two of its conveniences are also its failure modes. The quantization tag it picked and the chat template it inferred are both invisible until accuracy tells you about them. A distilled model expects exactly the formatting it saw in training — the local deployment docs are explicit that a different system prompt or formatting will result in poor performance.

If accuracy is short, check in this order: temperature, then the quantization tag, then the template. If it is still short, the deployment is probably fine and the earlier decision was wrong — what size model do you need is where that conversation starts.