Open-Source LLM Inference Hit Production Grade. Nobody Noticed

No launch event.

No keynote. Just changelogs. Open-source LLM inference crossed into production territory during 2026. And the receipts are version numbers: vLLM 0.32.0 shipped as a stable release. A vdf.ai comparison tracked vLLM v0.30.0 on September 22, 2026. And the current infrastructure digest describes stable 0.32.0 — SGLang v0.5.21 followed on October 2, llama.cpp hit v0.5.0 on September 23. And Ollama wrapped llama.cpp into a one-command local server (v0.35.0, September 28). Hugging Face archived its own Text Generation Inference server in March 2026, steering new deployments toward vLLM and SGLang.

Model launches get the attention. Serving infrastructure decides whether anything ships. That gap is where 2026’s real story happened. Changelogs don’t trend, so here we are.

The short version, if you’re in a hurry: vLLM and SGLang serve many concurrent users from data-center GPUs, llama.cpp and Ollama run models on one machine.

And both halves get steady, unglamorous maintenance. Boring is what production looks like.

Open-Source LLM Inference Split Into Two Jobs

How many machines, and how many users? Answer that in ten seconds and you’ve sorted the whole field.

vLLM and SGLang sit on the server side. A vdf.ai comparison describes both as production servers that batch many simultaneous requests on data-center GPUs and split large models across multiple cards.

The winder.ai comparison pins the architectures down: vLLM is built around PagedAttention, SGLang around RadixAttention.

That RadixAttention row deserves more attention than it gets. SGLang’s docs, cited by vdf.ai, describe it reusing the cached work for prompt prefixes that requests share.

So if your product fires the same system prompt on every call. Agents and support bots being the obvious cases. That shared work gets computed once instead of every single request. It’s an architecture decision that quietly changes your bill.

The other side of the divide is local-first.

The same comparison frames Ollama, llama.cpp.

And LM Studio as built to run models on one machine, while winder.ai describes llama.cpp as a C++ runtime for GGUF-quantized weights that runs on CPUs, Apple silicon, and GPUs. Ollama has wrapped that in a one-command local server since June 2026. Different tools for a other job, and the maturity story applies to both halves.

Why vLLM and SGLang Changelogs Prove Production Maturity

Here’s my test for whether infrastructure is production-grade: read the changelog before the README. Feature lists are marketing. Maintenance logs are confession.

The Inference Radar 2026-W33 digest is a perfect specimen. Naming a newsletter after ISO week numbers is a commitment I respect more every year.

It reported that vLLM “spent the week on production hardening rather than a clean headline release,” with work touching ROCm CI, Gaudi fixes, NPU/XPU paths, ModelRunner pooling, compressed-tensors FP8, FlashMLA, DeepSeek MLA, and Kimi-K3 disaggregated serving.

The Gaudi plugin alone fixed INC quantization, Qwen cache restore, MoE gate handling, paged attention layout, decode bucket bounds, Mamba prefill batching, and shared-expert synchronization. SGLang’s week covered quantized checkpoints, DFlash, multimodal and audio serving, HiCache, disaggregated serving, AMD paths, and hardware-specific kernels.

None of that is launch-day material. All of it is the work of systems with users.

The sharpest evidence sits in an agents-radar infrastructure digest: two merged bugfixes, one resolving ORCA metric aggregation across data-parallel engines and another fixing incorrect argument forwarding in beam search, both characterized as “directly affecting production-grade inference reliability.” Sit with the metric aggregation fix for a second. Your monitoring dashboard was reporting numbers wrong across engines, which means someone was making capacity decisions on bad data and lived to file the bug. The same digest reports vLLM leading in PR velocity and issue density, which is the fingerprint of a foundational engine everyone depends on. When a project’s weekly output is CI plumbing and beam search fixes, it has customers with revenue on the line.

That’s the maturity signal nobody markets.

Open-Source LLM Inference Hardware Portability

The part I’d argue matters most for small operators isn’t throughput. It’s the hardware list.

vdf.ai documents vLLM supporting NVIDIA, AMD.

And Intel GPUs plus x86, ARM. And PowerPC CPUs, with plugins for Google TPU, Intel Gaudi, Huawei Ascend, and Apple silicon.

The agents-radar digest reports critical PRs advancing ROCm support for Kimi-K3 on gfx942 and gfx950, including AITER Triton kernel alignment and NVFP4 KV cache for dense MLA models. Meanwhile llama.cpp keeps running GGUF weights on CPUs, Apple silicon, consumer GPUs, and edge backends.

For a small business, that portability is bargaining power. When the same serving stack runs on three vendors’ accelerators plus plain CPUs, you rent whichever GPU hour is cheapest this quarter and keep an exit door open. Vendor lock-in dies quietly when the plugin list reads like a hardware catalog. This is the layer where open source pays rent, and it doesn’t have much to do with model quality.

FAQ: Open-Source LLM Inference

Is vLLM production-ready?

Yes, and the case doesn’t rest on anyone’s README. vLLM 0.32.0 shipped as a stable release, the weekly digests show hardening work landing nonstop, and vLLM leads comparable projects in PR velocity and issue density. Hugging Face archiving its own Text Generation Inference server in March 2026. And steering new deployments toward vLLM and SGLang on the way out. Reads like the industry casting a vote.

vLLM vs SGLang vs llama.cpp vs Ollama. Which should I run?

Start with your constraints.

One machine, one user?

Run Ollama, which has been a one-command local server over llama.cpp since June 2026. Many concurrent users on rented GPUs? Run vLLM or SGLang, and if your requests share prompt prefixes, SGLang’s RadixAttention reuses that work. CPU-only boxes or edge hardware? Run llama.cpp with GGUF-quantized weights. That’s the whole decision tree.

Does Ollama work for multi-user serving?

Not what it’s for. Ollama wraps llama.cpp for one machine. And llama.cpp itself is a C++ runtime for GGUF-quantized weights on CPUs, Apple silicon, and GPUs. The many-simultaneous-requests job belongs to vLLM and SGLang, which batch concurrent requests on data-center GPUs and split large models across multiple cards.

What happened to Hugging Face’s TGI server?

The cautionary tale is already in the record.

Text Generation Inference was archived in March 2026, with its last release v3.3.7 on 19 December 2025. And that same reference tells you to prefer vLLM or SGLang for new deployments. A production server with a corporate backer still went to maintenance mode.

Pick your serving layer by maintenance cadence, never by README promises.

One caveat from the operator’s seat before you go: self-hosting inference means you own the pager. Open-source serving infrastructure is now genuinely production-mature. But mature infrastructure still breaks at 2 a.m., and unlike a hosted API, nobody pages you but you.

So the question stopped being whether open-source LLM inference is production-ready.

It’s which boring, actively maintained server you put in front of your users.

Do this today: find out what actually serves the AI features in your business, then check that project’s last release date. If the answer is an archived repository, you just found your next migration before it found you.

What’s serving your inference right now? Tell me in the comments. And subscribe if you want the operator’s view every time this stack shifts.