Field note · Open models · July 2026
Open weights in production: choose the model, own the system
Downloading weights gives you deployment control. It also makes model qualification, serving, security, observability and lifecycle management your responsibility.
The model artifact
- Weights and tokenizer
What can be loaded and executed. - License and provenance
What may be used, changed and redistributed. - Evaluation profile
What the model can do in your environment.
The production system
- Serving engine
vLLM, SGLang, NIM or another qualified runtime. - Gateway and policy
Identity, model routing, quotas, data classification and fallback. - Operations
Observability, patching, capacity, security, evaluation and rollback.
Open weights is not automatically open source
The Open Source Initiative distinguishes access to final model parameters from access to the preferred form needed to study, modify and reproduce the full AI system. A model may be downloadable and commercially useful while still withholding training code, data information or imposing licence restrictions.
For enterprise adoption, record the exact licence, permitted uses, redistribution terms, training-data disclosures and dependency licences. “Open” is not a sufficient procurement field.
Select for the real workload
Public leaderboards are a discovery tool, not a production acceptance test. Build a model qualification matrix for the tasks you will run: tool calling, structured output, long context, multilingual behaviour, code generation, retrieval grounding, safety and domain accuracy.
Include operational dimensions such as GPU memory, supported quantisation, tokenizer behaviour, maximum useful context, batch efficiency and licence constraints. The best model is the smallest qualified model that meets the target under representative concurrency.
Separate local development from production serving
Ollama is excellent for local experimentation, offline prototypes and a consistent developer API. Production serving usually needs stronger scheduling, batching, metrics, multi-GPU support and failure handling.
vLLM provides an OpenAI-compatible server, prefix caching, structured outputs, tool calling, LoRA, data parallelism, tracing and production deployment examples. SGLang provides a high-performance serving runtime and benchmarking tools that measure time to first token, inter-token latency, throughput and end-to-end latency. Qualify both against your own prompt lengths, concurrency and hardware.
Benchmark the service, not only the model
Record time to first token, inter-token latency, tokens per second, requests per second, queue time, memory utilisation, error rate and cost per accepted task. Run a load shape that reflects real bursts, long contexts and streaming clients.
Optimisations such as prefix caching, speculative decoding, chunked prefill and disaggregated prefill can improve different bottlenecks. They also add configuration and operational complexity. Enable them only when the measured workload shows a clear constraint.
Quantisation is an architecture choice
Lower precision can make a model fit on fewer GPUs and increase throughput, but quality loss is workload-specific. Test reasoning, tool arguments, multilingual output and safety separately. Keep the original model and quantisation recipe in the registry so a rollback does not depend on rebuilding an undocumented artifact.
Treat model artifacts as a software supply chain
Pin model revisions, verify hashes, scan containers, document the tokenizer and chat template, and record the runtime version used for qualification. Promote artifacts through environments with signed manifests and evidence, just as you would promote application binaries.
Do not allow a production server to pull an unreviewed “latest” tag from a public registry at startup. Private or air-gapped environments should use an internal model store and controlled import process.
Make adapters, routing and rollback explicit
A model gateway should resolve an approved model profile rather than exposing arbitrary model names to applications. The profile binds the base model, adapter, quantisation, prompt contract, tool schema, data classification, budget and fallback policy.
When a new model or adapter is promoted, shadow representative traffic, compare evaluations and preserve rapid rollback. Model optionality is achieved through contracts and evidence, not by claiming that every OpenAI-compatible endpoint behaves the same.
Executive takeaway
Open weights can improve privacy, control and economics, but they transfer more responsibility to the operator. The durable advantage is not owning a model file. It is owning a qualified, observable and replaceable model-serving platform.