llama.cpp Vulkan Wrong Results: The Sliced KV Cache Regression

A crash is noisy. A wrong answer from a GPU kernel is worse.

On September 15, 2026, llama.cpp contributor ServeurpersoCom opened PR #28956 with a compact but important title:

vulkan: fix wrong results when a mul_mat reads a slice of a larger cache

The failure class matters because the program can keep running while the computation is wrong. If you are debugging local inference, that can look like a model problem, a bad quant, sampling instability, or corrupted KV state instead of a backend bug.

This article is a source analysis of the upstream change. I have not independently reproduced the regression, so I am not going to invent a before/after benchmark.

The short version

The bug is about a Vulkan matrix multiplication consuming a slice whose backing storage belongs to a larger cache tensor.

Conceptually, the dangerous assumption is:

logical tensor view == whole backing allocation

But a view can begin at an offset inside a larger buffer:

larger cache allocation
[................................................]
              [ slice used by mul_mat ]
              ^ non-zero logical start

If a backend path handles the allocation as though the slice starts at the beginning, the shader can read the wrong region while all of the high-level tensor shapes still look plausible.

That is exactly the kind of failure that deserves attention in an inference runtime: valid execution is not the same as valid computation.

Why cache tensors make this easier to trigger

LLM inference reuses large cache allocations aggressively. The KV cache is not recreated from scratch for every token. Runtime code creates views and slices into storage as sequences advance and batching decisions change.

A matrix multiplication may therefore receive something that is logically a tensor but physically a window into a larger allocation.

At the model level you might think in terms of:

K for these tokens
V for these tokens

At the backend level the memory can look more like:

base buffer
+ byte offset
+ shape
+ stride

Ignoring any one of those pieces can produce a correct-looking dispatch over incorrect bytes.

Why this is different from an out-of-memory error

An OOM usually gives you a strong signal:

allocation failed

A shape mismatch often gives another strong signal:

expected X, got Y

A sliced-buffer addressing bug can instead produce:

request succeeds
model emits tokens
output is wrong

That expands the debugging search space dramatically.

When users see nonsense output, they naturally test the model file, chat template, quantization, temperature, tokenizer, or context state. Those are reasonable suspects. But if the same GGUF behaves correctly on CPU or another GPU backend and fails only on Vulkan, the backend becomes a first-class suspect.

The control experiment I would run first

Do not start by changing ten sampling flags.

Hold the model and prompt constant and change only the backend:

same GGUF
same prompt
same context
same sampling settings

Vulkan -> suspicious output
CPU    -> expected output

If CUDA, Metal, HIP, or CPU is available, another backend is an even stronger control.

The question is not whether two stochastic generations are character-for-character identical. Use deterministic or near-deterministic settings where possible and look for a large qualitative divergence that follows the backend.

Why git pull may actually be the correct fix here

RAMGPT’s troubleshooting guide makes a distinction between errors fixed by updating and errors where updating cannot repair the real problem.

This one belongs in the first category once the upstream fix is merged into the version you install.

If your Vulkan build predates the fix and your workload hits this exact path, changing the model is treating the symptom.

The repair sequence is:

git pull
git rev-parse --short HEAD
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j
./build/bin/llama-cli --version

For packaged binaries or containers, verify that the package actually contains a commit after the relevant upstream fix. latest is not a commit identifier.

But do not diagnose every bad Vulkan output as PR #28956

The PR title is specific: mul_mat reading a slice of a larger cache.

That does not establish that every Vulkan gibberish report has the same cause.

Before claiming you hit this bug, record:

llama.cpp build number and commit
GPU and driver
Vulkan implementation
model + quantization
context size
batch / ubatch
whether KV cache quantization is enabled
whether CPU reproduces it
whether another GPU backend reproduces it

Then reduce the prompt and command until the backend difference survives in the smallest case you can produce.

Silent correctness bugs need stronger tests than crash bugs

A crash regression can often be caught by asking whether the process exits successfully.

A correctness regression requires an oracle.

For inference runtimes, useful backend tests can compare a known operation against a reference implementation within an appropriate tolerance:

reference CPU result
        vs
backend result on the same tensor/view

The important word here is view. A test that covers only standalone contiguous tensors can miss a bug that appears when a tensor is a slice of larger storage.

This is why backend test matrices need more than dtype and shape. They also need memory-layout cases:

contiguous tensor
strided tensor
view with non-zero offset
slice of larger allocation
cache-backed view

What this says about local LLM debugging

There is a useful hierarchy when output suddenly becomes wrong after a runtime update:

1. Freeze the model and prompt.
2. Record the exact runtime commit.
3. Compare backends.
4. Compare the last known-good runtime.
5. Minimize the failing command.
6. Search upstream by the failure mechanism, not only the model name.

For this regression, searching only:

<model name> Vulkan bad output

can miss the useful clue.

The upstream language is much more mechanical:

Vulkan mul_mat slice larger cache wrong results

That is often where the real search asymmetry is.

The practical takeaway

PR #28956 is interesting not because it promises a speedup. It does not need one.

It highlights a more fundamental requirement: a local inference backend must preserve tensor-view semantics all the way down to the GPU address calculation.

When a model suddenly produces bad output on one backend, do not immediately blame the quantization or the model.

Ask a simpler systems question first:

Is the backend reading the same logical tensor that the graph thinks it is reading?

That question can save hours of debugging.