Air-Gapped and On-Premise LLM Deployment
In an air-gapped network a hosted API is not expensive, it is unreachable. The model, its weights, its runtime and its dependencies all have to be inside the perimeter, which rules out every frontier LLM and makes a small fine-tuned model the only workable option.
What breaks in an air-gapped environment?
Everything that assumes an outbound HTTPS connection, which is most of the modern LLM stack.
The clearest illustration is the industrial case. Operational technology networks are typically segmented on the Purdue model, which divides systems into levels and uses firewalls to control what data flows upward while keeping internet access out of the lower operational layers. As our on-device RAG write-up puts it, that layering is why a cloud LLM is difficult to use even through an enterprise-grade service. The restriction is not a procurement preference; it is the security architecture working as designed.
The same shape appears elsewhere with different vocabulary. Rocketgraph’s customers run graph analytics on IBM Power hardware, and their AI features depended on hosted LLMs — which, as we wrote up with them, created privacy concerns that required another solution entirely.
The second thing that breaks is subtler: package installs. A deployment that pulls model weights at first run, or resolves Python dependencies at build time, works perfectly in staging and fails inside the perimeter. Everything has to be staged in advance.
Why does a small model fit where a large one does not?
Because the constraint is hardware you already own, and a distilled model is sized to it.
Hosting an open-weight frontier model on-premise is possible in principle and expensive in practice — the industrial case describes needing a powerful server with a decent and expensive GPU, and the difficulty of getting that past procurement for a feature considered a nice-to-have. A fine-tuned small model changes the arithmetic: a 1B model at 4-bit is around 500MB and runs on commodity CPUs with 4–8 GB of RAM, which is hardware already sitting in controllers.
The accuracy objection is the one worth taking seriously, and the answer is fine-tuning rather than optimism. Two measured cases:
| Deployment | Model | Result |
|---|---|---|
| Rocketgraph OpenCypher query generation on IBM Power | IBM Granite 3.3 8B, fine-tuned on 15,000 validated synthetic examples | ~85% of the performance of open LLMs such as Llama 405B and Claude |
| Siemens S7-1200 documentation RAG, 144 questions | Llama 1B, distilled | 61.1% vs 45.1% base, matching a Llama 3B base at 60.4% |
The Rocketgraph result also shows why a specialist can beat a generalist inside a perimeter. Their platform uses an OpenCypher variant, and public models bias toward standard Cypher — inventing NONE, ALL and ANY predicates it does not support, and missing platform-specific functions such as outdegree and indegree. No amount of general capability fixes that; training on validated platform-specific queries does. The general principle is in knowledge distillation explained.
What does a working setup look like?
Weights, a runtime, a client, and an evaluation set — all staged inside the perimeter before you need them.
| Component | Air-gapped choice | Why |
|---|---|---|
| Weights | GGUF or safetensors, copied in on media | No download at first run |
| Runtime | llama.cpp for CPU/edge, vLLM for a GPU server | Both run fully offline once installed |
| Serving interface | OpenAI-compatible HTTP on localhost or the local subnet | Existing clients need only a base URL change |
| Client | The model_client.py shipped with the model |
Encodes the exact training-time prompt format |
| Evaluation | Your held-out test set, staged alongside | The only way to verify the deployment inside the perimeter |
Get the model in first. distil model download <model-id> produces a tarball containing model/, model-adapters/, model_client.py and a README; models can also be pushed to a private Hugging Face repository, which yields a GGUF repo and a safetensors repo so you can carry across whichever format your runtime wants. Both paths are in the local deployment docs.
Then pick the runtime by hardware, not by preference: llama.cpp for anything without a GPU or with a small one, vLLM for a GPU server handling concurrency, and see running a small language model on CPU for the CPU-specific settings. Size the hardware from how much VRAM does a 1B, 3B or 8B model need.
The training does not have to happen inside the perimeter. It is the deployment that is air-gapped: models are trained on platform infrastructure and the artefact is carried in, which is also why fine-tuning without a GPU is compatible with this pattern.
What do you give up?
Three things, and it is better to know them at design time.
Updates are a project. There is no silent model upgrade. Every new version is a staged artefact, a re-validation against your test set, and a change window. Plan the re-validation set at the same time as the first deployment.
Observability is local. No hosted dashboard is watching your endpoint. Whatever you want to know about drift, failure rates or latency has to be collected and stored inside the perimeter — and if you want to use those logs to improve the model later, exporting them is itself a controlled process.
The model is a specialist and stays one. A distilled SLM is trained for a task and is not a general assistant; scope creep inside a locked-down network means another training cycle, not a better prompt. What size model do you need and what is a small language model cover where that boundary sits.
In exchange, nothing leaves. For the environments described above — industrial control networks, government analytics, healthcare records — that is not a feature, it is the requirement everything else is negotiated around.