Your Windows llama.cpp Build May Be Leaving VNNI on the Table

There is a class of local-AI performance failure that is unusually hard to notice: the hardware supports an optimization, the application runs correctly, and yet the optimized code path never entered the binary.

A new llama.cpp issue provides a compact example. While profiling Qwen3.8-27B Q8_0 on Windows, contributor mmpataki observed that an AVX-VNNI-capable processor was not reporting AVX_VNNI in llama.cpp’s startup feature line. The associated pull request traces the problem to build-time detection under MSVC rather than to the processor itself.

The important lesson is broader than one missing flag. Hardware capability, compiler capability, build-system detection, compiled binary capability and runtime dispatch are distinct layers. A failure at any one of them can silently remove an optimization while preserving functional correctness.

The CPU Feature Is Only the First Layer

A common mental model is too simple:

CPU supports AVX2? yes
CPU supports VNNI? yes
therefore llama.cpp uses VNNI? yes

The final implication does not necessarily follow.

Native inference software frequently contains multiple kernel implementations. Whether a particular implementation is compiled can depend on preprocessor macros or CMake feature tests. Only after the implementation exists in the binary can runtime dispatch choose it.

In issue #28295, the reporter showed llama.cpp starting before the proposed fix with a feature line containing AVX and AVX2 but no AVX-VNNI. After the proposed detection change, AVX_VNNI = 1 appears in the diagnostic output.

Those observations come from the upstream contributor. They are not RAMGPT benchmark results. The pull request remains open as of this writing, and the evidence does not justify claiming a universal performance gain for all Windows builds.

Why MSVC Is the Interesting Part

The pull request identifies a specific asymmetry in toolchain behavior. GCC builds using -march=native can expose the expected AVX-VNNI capability through compiler-defined macros. MSVC does not automatically provide the __AVXVNNI__ macro that llama.cpp’s existing guarded code expects.

That creates a subtle failure mode: the CPU may support AVX-VNNI, but the preprocessor condition used to expose the optimized implementation is never satisfied.

The proposed fix follows the pattern llama.cpp already uses for other instruction sets. Instead of trusting a compiler macro, CMake explicitly checks whether the toolchain can compile and run a small program using _mm256_dpbusd_avx_epi32.

That changes the question from:

Did this compiler define the macro we expected?

to:

Can this toolchain actually build and execute the instruction we need?

The second test is much closer to the capability the runtime cares about.

Why This Matters for Quantized Inference

VNNI-class instructions accelerate integer dot-product and multiply-accumulate patterns that are directly relevant to quantized neural-network kernels.

That does not mean enabling AVX-VNNI automatically produces a predictable tokens-per-second uplift. Local inference can be compute-bound, memory-bandwidth-bound, cache-sensitive or dominated by kernels unaffected by the missing path. Model quantization, tensor shapes and CPU microarchitecture all matter.

But the absence of a usable optimized path changes the implementation’s option set before performance measurement even begins.

For benchmarking, that distinction matters. Two machines with similar processors can produce different results if their binaries were built with different compiler capabilities or feature-detection outcomes.

This Is Also a Packaging Problem

Most llama.cpp users do not compile every binary themselves. They consume release artifacts, package-manager builds, desktop applications or third-party bundles.

That means build flags and compiler detection become part of the performance characteristics of the distributed artifact.

A package can be correct and stable while still omitting an optimization available on the host CPU. Conversely, a binary compiled too aggressively for newer instruction sets may sacrifice portability.

Runtime dispatch is intended to balance those constraints, but runtime dispatch cannot select code that was never compiled.

This gives local-AI binaries a useful hierarchy:

physical CPU capability
        ↓
compiler capability
        ↓
build-system feature detection
        ↓
compiled kernel availability
        ↓
runtime dispatch
        ↓
measured inference performance

Treating the first and last lines as the whole story can make cross-platform performance comparisons misleading.

The Startup Feature Line Is a Diagnostic Artifact

llama.cpp’s startup feature dump is easy to ignore, but this issue shows why it can be valuable.

If a processor is known to support an instruction set and llama.cpp does not report that capability, the discrepancy can indicate a build or detection problem before any tuning begins.

That does not prove the missing feature is responsible for a performance difference. It does, however, provide a concrete place to investigate before changing thread counts, quants, batch sizes or affinity settings.

For reproducible CPU benchmarking, the feature line should therefore be treated as part of the environment record, alongside compiler version, build options and llama.cpp revision.

Why We Are Not Quoting a Speedup

Issue #28295 and PR #28297 establish the feature-detection discrepancy and show the diagnostic changing after the proposed patch. They do not provide a sufficiently controlled before-and-after inference benchmark to support a claim such as “AVX-VNNI makes llama.cpp X% faster.”

So there is no such number here.

The technically defensible conclusion is narrower and more useful: a Windows build can fail to expose an optimization supported by the CPU because build-time detection and runtime hardware capability are not the same thing.

That is a small implementation detail with a large methodological consequence. When comparing local-AI performance, the model and hardware are not enough; the binary itself is part of the experiment.

Sources

• llama.cpp issue #28295: MSVC compilation doesn’t detect AVX-VNNI
• llama.cpp PR #28297: detect and enable AVX-VNNI compilation on MSVC