Deploy an SLM with vLLM

vLLM is the runtime you choose when a service, not a person, is calling the model. Its reason to exist is throughput under concurrency — continuous batching and paged KV cache — which is exactly what a laptop runtime does not give you and exactly what a production endpoint needs.

When is vLLM the right runtime?

When you have a GPU and more than one caller.

The distinguishing property is what happens as request volume rises. A single-stream runtime serves each request in turn; vLLM keeps a batch in flight and schedules across it, so per-request latency degrades gracefully instead of queueing linearly. In our inference-cost benchmark a fine-tuned Qwen3-4B served by vLLM on a single H100 sustained a measured ceiling of 222 requests per second — over 19 million requests per day from one GPU.

It is also the wrong choice for most of the other cases in this cluster. vLLM wants a GPU, a Python environment, and a process supervisor. For a device, a handheld, or anything that must ship as a binary, use llama.cpp instead; for a development loop, Ollama.

What you need before you start

A CUDA-capable GPU with room for the weights plus KV cache, Python, and the downloaded model.

Sizing first, because getting this wrong wastes an hour. The same 4B model above occupied 7.6 GiB of GPU memory in our benchmark at BF16. The general arithmetic — weights plus cache, per model size — is in how much VRAM does a 1B, 3B or 8B model need.

Step 1: Download the model

distil model download <model-id>

Extract the tarball. You get a directory containing:

├── model/
├── model-adapters/
├── model_client.py
├── README.md

model/ holds the merged weights and is what you point vLLM at. model_client.py is the client script that encodes the training-time prompt format — keep it, you need it in step 4.

If you pushed the model to a private Hugging Face repository instead, you can skip the download and serve the repo id directly.

Step 2: Install vLLM in a clean environment

python -m venv serve
source serve/bin/activate
pip install vllm openai

A dedicated virtual environment is not fussiness. vLLM pins specific torch and CUDA-adjacent versions, and installing it into an environment that already has torch is the single most common way to end up with a build that imports but will not allocate. The vLLM quickstart documents uv pip install vllm --torch-backend=auto as an alternative that resolves the backend for you.

Step 3: Serve the model

vllm serve model --api-key EMPTY

model is the directory containing the weights, not a model name. The server binds http://localhost:8000 by default and exposes /v1/completions, /v1/chat/completions and /v1/models. It runs in the foreground, so put it in a separate window, a tmux session, or a service unit.

For a tool-calling model, two more flags are mandatory:

vllm serve model --enable-auto-tool-choice --tool-call-parser hermes --api-key EMPTY

The parser has to match how your model emits calls. hermes is what the local deployment docs use; vLLM’s tool calling documentation lists the alternatives, including llama3_json for Llama 3.1/3.2, functiongemma, qwen3_xml and pythonic. A mismatched parser produces text where you expected a structured call.

Serving from Hugging Face instead of a local directory:

pip install vllm
vllm serve "YOUR_USERNAME/MODEL_NAME"

Step 4: Query it

Either through the OpenAI client:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="model",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Your question here"},
    ],
)
print(response.choices[0].message.content)

Or, preferably for a first check, through the script that shipped with the model:

python model_client.py --conversation '[{"role": "user", "content": "Your question here"}]'

For open-book question answering, the context is wrapped in a <context> tag followed by a newline, inside the first user message. The docs say plainly that a different system prompt or message format gives poor performance — the specialist model expects the format it was trained on.

Step 5: Measure before you commit

Run your held-out test set through the endpoint and compare with the platform’s evaluation metrics, then load-test at your expected concurrency.

Accuracy first, because a wrong chat template or tool parser looks identical to a working server. Then throughput, because the numbers that matter are yours. For reference, these are the figures we published for a fine-tuned Qwen3-4B on Text2SQL, served at BF16 on a single H100 node at its measured saturation point:

Metric	Value
Max sustained RPS	222
p50 latency	390ms
p95 latency	640ms
p99 latency	870ms
GPU memory	7.6 GiB

Those percentiles are measured at the sustained ceiling, not in a quiet system, which is why they look high next to single-stream numbers — what latency can you expect from an SLM unpacks that distinction. In brief experiments in the same study, FP8 quantization added roughly 15% throughput with 44% less memory and no measurable accuracy loss; the precision trade-off in general is covered in Q4 vs Q8 vs FP16.