vLLM’s `VLLM_BATCH_INVARIANT=1` flag delivers reproducible LLM inference by making output “deterministic and independent of the batch size or the order of requests in a batch,” per the project’s own documentation. That one environment variable is the difference between a pipeline you can debug and one that quietly rerolls your outputs every time server load shifts. It works by swapping in deterministic kernel implementations for attention and other operations, per the vLLM-Ascend feature guide. The trade is speed: the same feature “disables certain optimizations that may introduce non-determinism (such as custom all-reduce operations in tensor parallel mode),” per an implementation guide.
Know that trade before you bet a retry-heavy pipeline on it.
Temperature 0 Does Not Make You Deterministic
The failure mode has been documented by people running the checks. A Reddit post on DetLLM’s deterministic inference checks reports it in one line: “changing batch size can change outputs even under ‘deterministic’ settings.” That matches what a Reddit summary of Thinking Machines Lab’s research calls the real culprit: “the core culprit is batching variance,” where server load unpredictably alters the arithmetic behind your tokens. The same summary closes with the cure: “Batch-invariant kernels unlock true reproducibility.”
The mechanics are unglamorous.
Serving engines pack requests into batches so the GPU stays busy. And floating-point addition rounds differently depending on the order partial sums get combined. Change the batch and you change the grouping, which changes the rounding, which occasionally flips a token. Temperature 0 picks the highest-scoring token every time; it does nothing to protect the scores that decide which token sits on top.
The BlockLock paper states the actual requirement: “Reproducible LLM inference requires that a request produces the same output regardless of the other requests in the batch.” Note the wording.
Not the same output most of the time.
The same output, no matter who else is riding the GPU with you.
What vLLM Actually Promises
vLLM’s definition covers two dimensions, not one: batch size and the order of requests within the batch.
So a request shuffled to a different position still gets the same answer, which matters more than it sounds when a scheduler is constantly reshuffling everything. The project attributes the property to “deterministic kernel implementations for attention and other operations” and “consistent numerical behavior across separate batch sizes.”
The docs even show the payoff in a code comment: “Outputs will be deterministic regardless of batch size,” sitting next to a plain `llm.generate(prompts, sampling_params)` call. No special sampling parameters, no client-side gymnastics. The guarantee lives in the serving layer, which is where it belongs.
My read: your weights were always deterministic.
Your scheduler was not. Framing this as a serving-level property instead of a model property is the honest version. And it’s the version vLLM ships.
The One-Variable Fix and Its Price
Turning it on is trivial. “Batch invariance can be enabled by setting the VLLM_BATCH_INVARIANT environment variable to 1,” per the vLLM docs. One line in your launch config, restart the server, done.
The price is where people hesitate.
The feature disables optimizations “that may introduce non-determinism (such as custom all-reduce operations in tensor parallel mode),” which means determinism and raw throughput are direct competitors on the same box.
I won’t quote you a percentage slowdown.
Because the material I have asserts the trade without publishing numbers. And inventing one would be exactly the fake precision I write this blog to avoid.
Run your own before-and-after on your own hardware.
My decision rule is simple.
Flip the flag anywhere a divergent retry costs more than the tokens you lose to the slowdown. Anything feeding an eval baseline, a prompt-keyed cache, or a retry loop qualifies.
Measure after, not before, and only walk it back if the throughput hit actually hurts.
Fixed Geometry Kernels Fix It at the Root
The deeper answer comes from kernel research.
One study on arXiv describes “a set of fixed-configuration fused-upcast GEMM kernels that load 16-bit weights from memory, upcast them to FP32 in registers.
And accumulate with IEEE-754 arithmetic.” The crucial property follows: “The complete floating-point reduction order is therefore a pure function of the problem shape.” The reduction order also stays “independent of the device, its SM count, or kernel scheduling,” and small-batch decode shapes combine their split-K partial sums “in a fixed ascending order with no atomics.”
Translate that into operator language. The shape of the computation is decided entirely by your request, not by whoever else shows up on the GPU that microsecond.
The study calls the result batch invariance “within its decode bucket,” where “a given request’s outputs are bitwise independent of how many other requests share its batch,” a property the paper credits to He and Thinking Machines Lab 2025 as key to run-to-run determinism in serving systems.
That scope phrase, “within its decode bucket,” is doing honest work. This is invariance by construction with stated boundaries, not a marketing guarantee. And it is the fixed batch geometry of my title: pin the reduction order to the problem shape. And batch composition stops being an input to your output.
What This Means If You’re a Small Operator
I build and ship AI automation solo, and batching variance hits small operators exactly where they’re most exposed: retries.
Lean pipelines lean hard on retry logic, and retry logic assumes the same input produces the same output. Break that assumption silently and your “safe” retry becomes a coin flip on a slightly other answer, which then cascades into whatever downstream step trusted the first version.
Three moves worth making this week:
– If you self-host on vLLM, set `VLLM_BATCH_INVARIANT=1` on the endpoints where identical retries matter, then measure the throughput cost yourself.
– Test before you trust. Run one prompt alone, run the same prompt inside a busy batch, and diff the outputs. Divergence means you found batching variance, not a bug in your prompt.
– If you buy inference from a provider, ask directly whether outputs are batch-invariant. Expect the answer to be no, and design your retries and caches accordingly.
Most of the discourse burns energy on which model to pick.
The serving layer underneath is quietly rerolling outputs, and nobody notices until a pipeline eats an afternoon. When you’re one person, debugging time is the scarcest resource in the stack. And a slower deterministic endpoint beats a fast one that answers differently after every retry.
Close the Loop Before It Bites
Reproducible LLM inference stopped being a research curiosity the moment it became a one-line setting in a serving stack people actually run.
The phenomenon is documented, the definition is settled, the fix ships in vLLM today. And the kernel research explains why it works.
What’s left is the part nobody does: verifying your own deployment.
Pick one pipeline you own.
Set `VLLM_BATCH_INVARIANT=1` on a test endpoint, run the alone-versus-batched diff, and see whether your outputs survive the crowd. If they match, you just made your automation debuggable, and that is worth more than any benchmark. I write these posts as I ship this stuff. So run the test and see what your stack has been hiding from you.
