Field note · Private AI infrastructure · July 2026
Architecting a private LLM platform for on-prem and air-gapped environments
A private LLM platform is a governed operating environment for models, data, agents and tools, not simply a model server behind a firewall.
SSO, tenant context, quotas, request classification and model policy.
Qualified models, GPU scheduling, caching, autoscaling and fallback.
Retrieval, durable state, tools, sandboxes and human approvals.
Policy, safety checks, traces, evaluations, audit and recovery.
Start with the security and data boundary
“On premises” is not one architecture. The required design depends on whether the organisation permits outbound model calls, software updates, telemetry export and remote administration. Define data classifications, residency, model licences, user identities, administrative authority and break-glass procedures before choosing the serving engine.
Separate the control plane from the customer data plane. Model prompts, embeddings, documents, credentials and raw traces should remain inside the approved boundary unless an explicit export policy permits otherwise.
Use a deployment ladder
For a developer workstation or a small internal prototype, Ollama or a single container can be the right answer. For a production private platform, use Kubernetes or an equivalent scheduler with GPU-aware placement, persistent model storage, health probes, quotas and controlled upgrades.
KServe v0.18 adds multi-node inference, OpenAI Responses API routing, namespace-scoped model caching and security hardening. NVIDIA NIM supports Helm, the NIM Operator, KServe, multi-node and air-gapped deployments. vLLM and SGLang remain strong open serving engines when the team wants more direct control of the runtime.
Reference runtime path
- IngressPrivate load balancer or API gateway terminates TLS, authenticates the caller and applies tenant quotas.
- Model gatewayResolves an approved model profile using data sensitivity, capability, residency, latency and cost policy.
- GuardrailsInput, retrieval, execution and output checks run outside the primary model and fail closed for protected actions.
- ServingvLLM, SGLang, NIM or a qualified runtime executes on dedicated GPU pools with model cache and health checks.
- Agent runtimeDurable workflows own state, retries and approvals. Narrow tools operate with workload identities and external secrets.
- EvidenceOpenTelemetry traces, evaluation results, policy decisions and audit records go to customer-controlled stores.
GPU and capacity engineering
Capacity planning begins with the workload distribution: model size, input length, output length, concurrency, latency SLO and availability target. Separate interactive and batch queues. Use dedicated pools when large batch jobs would otherwise destroy interactive tail latency.
Multi-node and disaggregated prefill can improve scale or isolate prefill and decode bottlenecks, but they increase network and failure complexity. Validate the benefit on the actual cluster fabric before adopting them. Maintain model caches near the GPU nodes and design for controlled warm-up rather than assuming every replica can cold-start quickly.
Agent tools need a separate isolation boundary
Do not run generated code, browser automation or package installation on inference nodes. Coding agents should receive ephemeral workspaces with restricted network routes, scoped credentials, CPU and memory limits, timeouts and disposable filesystems.
E2B demonstrates the cloud pattern with isolated Linux sandboxes for Claude Code, Codex and other coding agents. Fly Machines demonstrates fast-launching VM lifecycle control. For an on-prem or air-gapped platform, reproduce the principle using internal microVMs, hardened containers or dedicated worker nodes that have no standing production authority.
Guardrails are layered controls
NeMo Guardrails defines input, retrieval, dialog, execution and output rails. This is the right mental model: one content filter cannot protect an agent that can retrieve untrusted documents and call privileged tools.
Keep access control, network policy, tool arguments and approval outside the model. Use model-based safety checks where judgement is necessary, then combine them with deterministic validation and policy decisions.
Self-host observability and evaluations where the data requires it
Langfuse supports self-hosting with Docker and Kubernetes, including external PostgreSQL, ClickHouse and Redis. LangSmith offers self-hosted observability and evaluation, plus private control-plane and data-plane options for agent deployment.
Whichever platform is selected, standardise trace IDs across gateway, retrieval, model, workflow and tool calls. Keep raw prompts and sensitive traces behind the same access and retention controls as the source data. Add release evaluations and production sampling so a model, prompt or runtime upgrade cannot silently change behaviour.
Air-gapped operation is a lifecycle, not a checkbox
NVIDIA’s current NIM guidance uses a connected preparation phase followed by transfer of model assets or cache into the isolated environment. The air-gapped runtime uses local assets and no registry credentials. Apply the same pattern to containers, Python packages, guardrail models, evaluation datasets and vulnerability metadata.
Every offline release needs signed manifests, compatibility evidence, rollback assets and a controlled export path for sanitised operational evidence. Plan how certificates, licences, model updates and critical security patches enter the environment before the first deployment.
A pragmatic production stack
- ExperienceReact or internal portal, private API gateway and enterprise SSO.
- RuntimeKubernetes with KServe or Helm-managed vLLM, SGLang or NIM on GPU node pools.
- DataPostgreSQL, object storage, vector or hybrid retrieval, and customer-managed keys.
- AgentsLangGraph or another bounded runtime with Temporal-style durable workflows and isolated tool workers.
- TrustOPA-style policy, NeMo or custom guardrails, secret references, approval workflows and audit.
- OperationsOpenTelemetry, self-hosted Langfuse or LangSmith, Prometheus, logs, evaluation runner and cost attribution.
Executive takeaway
A private LLM platform is not a model server behind a firewall. It is a governed operating environment for models, retrieval, agents, tools, evidence and change. Start with one qualified use case and one complete request path, then scale only after the controls and economics are measurable.
Primary reading
- KServe 2026 release and production LLM serving updates
- NVIDIA NIM deployment guide
- NVIDIA NIM air-gap deployment
- NeMo Guardrails: input, retrieval, execution and output rails
- Langfuse self-hosted Kubernetes deployment
- LangSmith self-hosted architectures
- E2B isolated sandboxes for coding agents
- Fly Machines lifecycle and VM control