The One-Line Vulkan Fix Hiding an 83% Prefill Gain

Sometimes a performance bug is not a slow kernel. It is a fast kernel that the program quietly decides not to use.

That appears to be what happened in llama.cpp’s Vulkan Flash Attention path for hybrid models with quantized KV caches. Issue #28135, opened September 1, traced unexpectedly slow prefill on Qwen3.8 to a layout guard that rejected a valid per-layer KV-cache view. PR #28190 then reduced the proposed fix to a one-line change.

The fun part is not that one line can make a benchmark move. The fun part is why the line existed, why it was almost right, and why hybrid architectures made its assumption wrong.

A Fast Path That Was Effectively Invisible

The Vulkan backend has an f16-scratch path for Flash Attention with quantized K/V data. Instead of making the attention shader repeatedly work directly from quantized cache data, the path can dequantize into an f16 scratch representation suited to the operation.

But llama.cpp does not enable that path blindly. It first checks whether the KV tensor layout is dense enough for the dequantization/remap logic to be safe.

The relevant predicate included this condition:

t->nb[3] == t->nb[1] * t->ne[1]

For the tensor shape the check originally had in mind, that is sensible: the stride into dimension 3 should equal the size implied by the preceding dimensions.

The trouble is that a hybrid model’s KV cache is not necessarily presented as that kind of self-contained dense tensor.

Qwen3.8 mixes attention layers with recurrent/state-space layers. In llama.cpp it uses a non-unified KV-cache arrangement. The attention layer can therefore receive a view covering the currently used n_kv cells while retaining a stride inherited from the full cache allocation.

Those are two different quantities:

view length     = currently used KV cells
stored stride   = full KV-cache capacity

The view can be perfectly usable while failing a check that assumes those numbers describe the same extent.

The Guard Confused a View With Its Backing Buffer

Issue #28135 lays out the mismatch clearly. For the hybrid per-layer view, ne[1] reflects the active view, while nb[3] carries the full-buffer stride.

So the old equality effectively asked the cache to satisfy something close to:

used KV length == allocated KV length

That only becomes true when the cache is full.

Until then:

valid hybrid KV view
        |
        v
strict dense-layout test
        |
        v
fails nb[3] equality
        |
        v
use_dequant_kv = false
        |
        v
fast path never selected

Nothing crashes. Output is not obviously wrong. The model simply runs a slower path.

That class of bug is easy to miss because conventional correctness testing is almost useless against it. A test that asks whether the model produces the right answer can pass perfectly while a major optimization remains unreachable.

The Reported Numbers Are Large

The issue reporter tested Qwen3.8-27B on an AMD Radeon RX 9070 using Vulkan with a 21.5K-token prompt. Their reported prefill increased from 65 tokens/s to 119 tokens/s after enabling the path, an 83% increase, while their needle-test result remained exact and decode performance was unchanged.

Those are third-party measurements from the issue, not RAMGPT benchmarks, and the initial candidate fix was explicitly tested only on that setup.

PR #28190 reports a separate result: +29% llama-server prefill on a 30B MoE with Q8_0 KV, with byte-identical output. The PR is still open as of September 2, but it has received two approving reviews.

The difference between +83% and +29% is itself useful. This should not be advertised as a universal “83% Vulkan speedup.” The actual gain depends on model, cache type, workload, hardware, and how expensive the fallback path was. The stronger conclusion is architectural: a supposedly available optimization was being excluded from an important class of tensor views.

Why the PR Is More Interesting Than the Benchmark

The final PR changes only the last part of the layout check:

(t->ne[3] == 1 || t->nb[3] == t->nb[1] * t->ne[1])

That is more conservative than simply weakening the stride comparison.

The key observation is that when ne[3] == 1, there is only one element along that outer dimension. The shader never advances to a second element there, so nb[3] is not used to traverse multiple streams. Requiring the stride to describe a tightly packed fourth dimension is therefore unnecessary in that case.

For multi-stream cases, the original equality remains required.

In other words, the patch does not say:

Strides no longer matter.

It says:

Do not validate a stride that this operation will never dereference.

That is a much better kind of relaxation.

Hybrid Models Are Stress-Testing Old Assumptions

This is becoming a recurring theme in local inference engines.

A lot of optimization code was designed in a world where the mental model was straightforward:

transformer
+ uniform layers
+ conventional KV cache
+ predictable tensor lifetime

Hybrid architectures add recurrent state, attention only on selected layers, non-unified caches, views over shared allocations, MTP heads, and increasingly complicated execution paths.

The kernels may still be correct. What breaks is often the dispatch logic around them.

A predicate written to protect a kernel can encode assumptions about tensor ownership, contiguity, cache geometry, or batching that stop being universally true when a new architecture arrives.

That suggests a useful distinction when profiling modern llama.cpp:

kernel performance
!=
optimization availability

You can have an excellent optimized kernel and still get terrible performance if dispatch never reaches it.

What I Would Measure Before Blaming Vulkan

If a hybrid model is unexpectedly slow, comparing Vulkan against CUDA or another backend is useful, but it does not tell you why Vulkan lost.

I would separate the investigation into three questions:

1. Is the optimized kernel implemented?
2. Is the runtime actually selecting it for this tensor shape and cache configuration?
3. If selected, is the kernel itself slow?

Those are very different failures.

Issue #28135 was mostly a failure of question two.

That also means performance diagnostics should expose more dispatch decisions. A user should ideally be able to determine that a dequantized Flash Attention path was rejected, and which guard rejected it, without modifying the backend.

A hidden fallback is operationally convenient but diagnostically expensive.

A Better Regression Test Is About Reachability

Correctness tests remain necessary, especially because relaxing tensor-layout guards can create silent corruption if done carelessly. But this bug points to another test category: fast-path reachability.

For representative hybrid layouts, CI could verify not only that an operation produces correct output but that the intended implementation is actually eligible.

Conceptually:

representative tensor view
        |
        +--> correctness invariant
        |
        +--> dispatch invariant

The first asks whether the result is right.

The second asks whether the performance path we think we support is genuinely reachable.

That distinction matters more as backends accumulate specialized kernels and model architectures accumulate unusual memory layouts.

Bottom Line

PR #28190 is still open, so the exact patch can change before merge. Its reported performance figures should also remain attributed to the contributors who measured them.

But the underlying bug is a nice systems lesson.

The Vulkan backend did not need a dramatically faster attention algorithm. It already had a faster path. A tensor-layout predicate simply interpreted a hybrid cache view as though it had to be a tightly packed standalone buffer.

One line can unlock a large speedup when that line sits at the boundary between what the backend can do and what the runtime believes it is allowed to do.

As local inference engines absorb increasingly hybrid architectures, that boundary may be one of the most productive places to look for performance hiding in plain sight.