llama.cpp v0.2.0: What Actually Changed Under the Hood

llama.cpp v0.2.0 is out, and the obvious headline is semantic versioning.

That is real, but it is not enough to explain this release.

The more interesting story is in the changelog between v0.1.2 and v0.2.0: backend-specific decode tuning, speculative decoding for LFM2, tensor splitting for LFM2 and LFM2-MoE, long-context KV-cache work, a practical improvement to the memory behavior of llama-quantize, server authentication hardening, router lifecycle changes, and more backend coverage.

In other words, v0.2.0 is not a giant feature drop in the conventional sense. It is a snapshot of where llama.cpp engineering effort is going right now.

The pattern is increasingly clear:

less "one generic runtime for every model"
more hardware-aware kernels
more model-specific execution paths
more server lifecycle machinery
more attention to memory movement
more operational hardening

That is the part worth studying.

First: what v0.2.0 actually is

The release marks the beginning of consistent semantic versioning for llama.cpp.

The official split is now:

vX.Y.Z
stable
slower release cadence
recommended for downstream distribution and casual users

b[NUM]
nightly / development
fast release cadence
recommended for developers and technical users

The v0.2.0 tag corresponds to nightly build b10566.

There is an implementation detail here that is easy to miss: the stable release page itself does not duplicate the full matrix of prebuilt binaries. It points to the corresponding nightly tag through nightly-tag.txt; the actual platform builds live under that nightly release.

So the stable version is primarily a reproducible software checkpoint and compatibility target, not a completely separate binary distribution pipeline.

That matters to packagers and downstream projects, but the technical changes below are more interesting to anyone who actually runs llama.cpp.

The most consequential CUDA change is about choosing the right decode kernel

One of the best changes in v0.2.0 is PR #26079, which changes how CUDA chooses between two quantized matrix paths during decode.

The two paths are roughly:

mul_mat_vec_q (MVQ)
vs
MMQ

The important difference is how they pay the cost of decoding quantized weights.

MVQ re-decodes weights for each output column. MMQ decodes into shared memory and reuses the result. That means MVQ can be very efficient at low batch sizes, while MMQ becomes increasingly attractive as the number of simultaneously processed vectors grows.

The old behavior used a largely fixed crossover.

The new behavior introduces crossover points that depend on both:

hardware family
+
quantization type

That is a much more sophisticated policy than saying “MMQ after batch N” for every GPU and every quant.

Why the numbers are interesting

The pull request contains unusually useful benchmark data.

On an RTX 5090, for dense K-quants at ne11 = 8, the reported MMQ advantage over MVQ was substantial:

Those are developer-reported measurements from the PR, not a RAMGPT benchmark, and they apply to a specific decode shape and test setup. They should not be read as “v0.2.0 makes Q4_K 48% faster” in general.

The more important observation is that the winning kernel depends heavily on both quantization and hardware.

The same PR shows this clearly on DGX Spark GB10. There, only Q2_K crossed over within the tested ne11 <= 8 range. For Q3_K, MMQ was still slower at ne11 = 8; Q4_K was roughly neutral.

That is exactly why the new per-hardware/per-quant switch logic matters.

The optimization is not “use MMQ more often.”

It is:

use MMQ only where measurements show that its reuse of decoded weights beats MVQ on that hardware and quant type.

This is a more mature form of inference optimization than a global kernel choice.

DSpark support for LFM2 is a real feature, not just plumbing

Another notable v0.2.0 change is DSpark speculative decoding support for LFM2 models.

The implementation adds two important pieces:

• DSpark draft-model support for LFM2;
• partial rollback for LFM2 recurrent state.

The second point matters because LFM2 is not just a conventional transformer where speculative decoding can be treated as a simple token-verification loop. If speculative tokens are rejected, the runtime must restore model state correctly.

The PR includes sample measurements for LFM2.5-1.2B-Instruct with a DSpark-5L drafter on an RTX 4070 Laptop GPU:

The reported mean was about 2.12x across those few samples.

Again, these are the developer’s examples from the PR, not a broad benchmark. The author explicitly noted that fuller evaluations were still to come.

But the table demonstrates something important about speculative decoding that is easy to hide behind one headline number:

acceptance rate dominates the payoff.

Code-like continuation had high acceptance and much larger acceleration. More open-ended prompts such as explanation and poetry had much lower acceptance and smaller gains.

That is precisely why speculative decoding should be benchmarked by workload rather than by a single average throughput number.

It also connects directly to our earlier DSpark PCTree RTX 4090 benchmark, where higher acceptance did not automatically translate into higher end-to-end throughput once verification overhead and tree behavior were considered.

LFM2 tensor splitting was validated numerically, which is worth noticing

v0.2.0 also enables --split-mode tensor for LFM2 and LFM2-MoE.

That sounds like a normal multi-GPU feature, but the validation methodology in the PR is unusually useful.

The contributor did not only report that the models loaded and generated text. They compared split and non-split execution using:

• perplexity;
• median KL divergence;
• same-top-probability rate;
• RMS probability change;
• architecture-level NMSE tests.

The architecture tests reported Meta NMSE on the order of 1e-14 for both LFM2 model types.

The small-corpus perplexity comparison also showed very small differences between --split-mode none and tensor split for the tested variants.

This is a good example of how inference changes should be validated when numerical equivalence matters.

A model can generate plausible text in both modes and still have a meaningful distribution shift. Looking at KL divergence and token-probability agreement gives a much stronger signal than comparing a few generated answers by eye.

That methodology is particularly relevant to quantization and multi-backend benchmarking.

Long-context KV-cache performance is increasingly about data movement

Two v0.2.0 changes show how much attention llama.cpp is putting into quantized KV caches and Flash Attention.

Vulkan: stop dequantizing the same KV data repeatedly

PR #25494 targets q8_0 KV cache under the Vulkan coopmat1 Flash Attention path.

The old path could repeatedly dequantize the same KV data across workgroups during prefill.

The new approach dequantizes once into an F16 scratch layout and reorganizes it so Flash Attention can read it efficiently.

That trades some temporary memory for much less redundant work.

The PR gives a concrete long-context example at 128K context:

q8_0 KV cache: roughly 6 GB
extra F16 scratch for one layer: roughly 256 MB
F16 KV cache: roughly 12 GB

So the scratch overhead is around 4% on top of the quantized cache in that example, while preserving most of the q8_0 memory advantage.

The reported prefill result in one deep-context case improved from about 45.8 t/s to about 93 t/s after the dequantize-once + transpose path was used.

The contributor described the optimization as neutral at shallow depth and as much as roughly 2x at long context, with larger benefit on lower-bandwidth unified-memory systems.

This is a very important distinction:

quantized KV cache is not only a capacity feature.

It changes the memory-access pattern of attention. Once context becomes large enough, redundant dequantization and poor layout can erase a large part of the expected benefit.

Metal: quantized KV is also being routed through F16 for Flash Attention

Metal received related work: quantized KV data can be dequantized to F16 before Flash Attention, followed by another change that limits this behavior to larger batches where it is worthwhile.

The shape of these changes is revealing.

The optimization problem is not simply:

smaller KV cache = faster

It is closer to:

compressed KV saves memory bandwidth and capacity
but
attention kernels may prefer a different layout/type
therefore
runtime must decide when dequantization cost is worth paying

This is exactly the kind of systems tradeoff that becomes dominant at long context.

llama-quantize now behaves better on very large models

PR #22877 is less glamorous than CUDA kernel tuning, but it may matter more to people who actually produce GGUFs.

The change allows processed tensor memory pages to be released after a layer is no longer needed during operations such as llama-quantize.

Previously, large quantization jobs could retain a much larger resident set as processing progressed, especially when the source model was close to or larger than available physical RAM.

A community tester reported that while quantizing MiMo V2.5 Pro, process memory stayed around 50 GB instead of continuing to climb as the quantization advanced.

That is not a formal benchmark and should not be generalized to every model or OS.

But the design improvement is straightforward: once a source layer has been processed, keeping its pages resident provides little value.

For people quantizing very large models, memory behavior is part of usability.

A quantizer that theoretically supports a model but drives the machine into severe memory pressure halfway through the job is not practically equivalent to one that releases old layers as it goes.

Server authentication fixed a real information leak

One of the most operationally important v0.2.0 changes is easy to overlook in a long changelog.

When server authentication was enabled, model-list endpoints could still expose model information to unauthenticated clients.

The /v1/models response could include filesystem paths such as:

/home/alex/Downloads/Qwen3.6-35B-A3B-Q4_K_M.gguf

That leaks both deployment structure and potentially personally identifiable information embedded in paths.

PR #26347 changes model endpoints so they become private when authentication is configured.

This is not a performance feature, but it matters for anyone who exposes llama-server beyond localhost.

The broader point is that as llama.cpp’s server becomes more capable, its security surface also becomes more conventional.

Once you have:

authentication
model routing
remote clients
multiple loaded models
metrics
sleep/wake lifecycle

you also inherit the normal responsibilities of a serving platform.

Sleep mode and metrics are becoming operational features

PR #27376 refactors server sleep handling and allows /metrics to remain accessible while the server is sleeping.

That sounds minor until you think about how monitoring systems work.

If every metrics scrape wakes a sleeping inference server, sleep mode stops being useful. If metrics disappear entirely during sleep, monitoring systems cannot distinguish an intentionally sleeping service from an unhealthy one.

The updated behavior uses cached responses during sleep and separates metrics behavior from routes that should reset the idle timer.

This is a small but very infrastructure-like change.

It is about making model lifecycle compatible with monitoring rather than treating sleep as a special local-only mode.

Router startup is also getting more deliberate

The v0.2.0 changelog includes router work that lazy-loads startup_models after the main server setup, plus a fix where --docker-repo could incorrectly trigger router mode.

These changes are not exciting in isolation.

Together with model deduplication presets, authentication behavior, sleep lifecycle, and metrics, they show that llama-server is spending more engineering effort on model orchestration, not just request handling.

That is a meaningful shift.

The server increasingly has to answer questions such as:

Which model is loaded?
When should another model load?
Can two routes reuse the same model?
What happens while a model is sleeping?
What can unauthenticated clients see?
How do monitoring systems observe the service?

Those are platform questions.

A few smaller changes are still strategically interesting

The v0.2.0 changelog is broad enough that several smaller entries deserve mention.

--mmproj-device gives users explicit device placement control for the multimodal projector. That matters on mixed-device systems where the text model and projector do not necessarily belong on the same accelerator.

ggml_rope_set_offset gained support across Metal and then OpenCL, SYCL, WebGPU, and Hexagon. That is the kind of backend convergence work needed to keep higher-level features portable instead of CUDA-only.

Granite SWA and Granite MoE SWA architecture support landed in the release window.

Windows ARM64 CUDA was added to the manual build workflow.

Release artifact attestation was added as well, which improves supply-chain verifiability for distributed binaries.

None of these individually defines v0.2.0, but together they reinforce the same trend: llama.cpp is broadening both its hardware surface and its operational surface.

What I would actually expect to notice after upgrading

The answer depends heavily on workload.

If you run a single-stream Q4_K model on an RTX 4090 with a short context, you should not expect v0.2.0 to magically transform throughput just because the version number changed.

If you run multiple decode sequences on newer NVIDIA hardware, the new CUDA crossover policy may matter much more.

If you use LFM2, the release is substantially more interesting because tensor splitting and DSpark support both landed.

If you use deep context with quantized KV cache on Vulkan or Metal, the attention-path work is relevant.

If you quantize huge models, the layer-by-layer memory eviction can make the conversion process much more practical.

If you expose llama-server, the authentication and lifecycle changes are arguably more important than another few percent of token throughput.

That is why I would summarize v0.2.0 by workload rather than by one headline:

That table is a better description of the release than “llama.cpp now uses semantic versioning.”

The release model still matters, but for a different reason

After looking at the technical changes, semantic versioning becomes more useful, not less.

This release contains changes that are highly hardware- and workload-specific.

That makes reproducibility important.

A benchmark result such as:

Q4_K_M
RTX 5090
llama.cpp v0.2.0

is now a much cleaner reference than:

Q4_K_M
RTX 5090
some master build from last week

The same applies to bug reports, downstream bindings, packaged applications, and performance regressions.

The stable tag does not make llama.cpp technically more mature by itself.

It gives us a stable coordinate system for measuring a codebase that changes extremely quickly.

That is especially valuable when individual PRs can change kernel-selection heuristics, KV-cache paths, speculative state handling, or server behavior.

My takeaway

My main takeaway from v0.2.0 is no longer “llama.cpp has semantic releases.”

It is this:

llama.cpp optimization is becoming increasingly conditional.

The best CUDA decode path depends on hardware, quant type, and batch shape.

The value of speculative decoding depends on acceptance rate and workload.

The value of quantized KV depends on how attention kernels dequantize and lay out that cache.

Multi-GPU support needs numerical-equivalence validation, not just “it runs.”

Server features now need conventional authentication and monitoring semantics.

Large-model quantization needs active memory-lifecycle management.

That is a much more interesting picture than a version-number story.

v0.2.0 is useful because it freezes a moment where all of those threads are visible at once.

For the broader architecture, see llama.cpp Is More Than a GGUF Runner: A Local Inference Stack Emerges. For quantized memory behavior, What Is KV Cache? provides the background. And for a concrete example of why speculative decoding needs workload-level measurement, see our DSpark PCTree RTX 4090 benchmark.