Why a Small Vulkan Fusion Gave Gemma 4 About 4% in llama.cpp

A roughly four-percent speedup is easy to dismiss when local-AI discussions are full of claims measured in multiples. But a fresh llama.cpp Vulkan change is interesting precisely because it is not a new quantization format, a new model, or a giant algorithmic trick.

It is graph plumbing.

On September 7, llama.cpp release b10833 incorporated PR #28024, titled vulkan: rms_norm fusion opportunities. The author reports that the change is worth around 4% in Gemma 4 on their system. That number is a third-party measurement from the PR author, not a RAMGPT benchmark, and the hardware/configuration is not specified in the PR description. So the responsible reading is not “Gemma 4 is now 4% faster everywhere.”

The more useful question is: what kind of work disappeared to make a few percent appear?

The patch gives a good answer. It extends llama.cpp’s Vulkan graph fusion beyond a simple RMSNorm-plus-multiply path and recognizes several longer operator sequences:

RMS_NORM → MUL → ADD
RMS_NORM → MUL → ADD → MUL
RMS_NORM → VIEW → SET_ROWS
RMS_NORM → MUL → ROPE → VIEW → SET_ROWS
ROPE → VIEW → SET_ROWS

That list looks mundane. It is also a compact lesson in why inference performance is not determined only by FLOPs.

The important unit is sometimes the operator chain

At the model level, operations such as normalization, elementwise multiplication, residual addition, rotary position encoding, views, and row updates are logically distinct. A framework graph naturally represents them as distinct nodes.

A GPU does not necessarily want to execute them that way.

If each small operation becomes a separate dispatch, intermediate values may have to be written out and consumed by the next operation, and the runtime pays dispatch and synchronization overhead repeatedly. For a large matrix multiplication, that overhead may be relatively small compared with the useful arithmetic. Around lightweight elementwise operations, it can matter much more.

Kernel fusion changes the execution unit. Instead of treating:

normalize
then multiply
then add
then multiply again

as four independent pieces of GPU work, a backend can recognize the graph pattern and execute more of the chain through a fused shader path.

The mathematical model has not changed. The execution graph has.

What PR #28024 actually adds

The source diff is more revealing than the short release note.

The Vulkan backend adds explicit graph patterns for the new RMSNorm chains and introduces an rms_norm_mode enum with modes including:

RMS_NORM_MUL
RMS_NORM_MUL_ADD
RMS_NORM_MUL_ADD_MUL
RMS_NORM_MUL_ROPE
RMS_NORM_MUL_ROPE_VIEW_SET_ROWS
RMS_NORM_VIEW_SET_ROWS

It also adds Vulkan pipelines for fused RMSNorm-plus-add variants and RMSNorm-to-SET_ROWS paths. In other words, this is not merely a compiler hint saying “please fuse more.” llama.cpp is explicitly teaching its Vulkan backend which graph shapes are legal candidates and which pipeline should execute them.

That distinction matters in a portable runtime. llama.cpp supports many backends, but each backend has different kernels, dispatch costs, memory behavior, and fusion opportunities. A graph that is semantically portable does not imply that its fastest physical execution is portable.

Why VIEW is interesting even though it does almost no math

One of the stranger-looking patterns is:

RMS_NORM → VIEW → SET_ROWS

A VIEW is primarily about interpreting existing storage with a different tensor view; it is not the sort of expensive numerical operation people normally optimize first.

But graph optimization is not only about expensive nodes. A cheap node can sit between two operations that would otherwise be profitable to combine. If the backend’s pattern matcher cannot see through the structural operation, the fusion boundary stops there.

PR #28024 explicitly adds edges and handling for these longer graph shapes. The optimization therefore says something broader: structural graph nodes can obstruct physical fusion even when they contribute almost no arithmetic themselves.

That is useful when reading inference traces. Counting “heavy” operations alone can miss why a backend launches more work than expected.

The RoPE extension points to the same problem

The patch also extends the existing ROPE → VIEW → SET_ROWS handling to support IMROPE.

Again, the interesting part is not that rotary position encoding became cheaper in isolation. The backend is learning to preserve a fused execution path across a sequence that includes positional transformation plus tensor-layout and row-update mechanics.

This is the kind of optimization that tends to be architecture-sensitive. Different model graphs expose different repeated sequences. Gemma 4 apparently presents enough of the newly recognized RMSNorm-related shapes for the PR author to observe a measurable end-to-end gain.

That also explains why a backend optimization can look tiny in code-review language and still move model throughput. If the pattern sits in a repeated layer path, a small saving is paid back many times per token.

Why four percent is more interesting than it sounds

The reported number should be treated carefully. The PR says only:

Worth around 4% in gemma4 on my system.

There is no RAMGPT reproduction here, and the PR description does not establish whether the number refers to prompt processing, token generation, a combined workload, or a particular Gemma 4 size and quantization. It would be wrong to manufacture that missing context.

But the patch itself establishes the mechanism independently of the benchmark number: new graph patterns are recognized, new fused Vulkan pipelines are added, and execution paths are selected according to those patterns.

For local inference, a few percent from this class of change is valuable because it compounds with optimizations elsewhere. Quantization may reduce memory traffic. Better matrix kernels may accelerate the large linear layers. Flash attention may change attention behavior. Fusion then attacks overhead and intermediate work around the smaller operators.

These are not mutually exclusive speedups.

This is also why backend comparisons age quickly

Suppose two runtimes support the same model and both use Vulkan. It is tempting to think their performance difference mostly reflects shader quality for matrix multiplication.

PR #28024 is a reminder that the comparison can depend on something more granular: which exact graph fragments each runtime recognizes today.

Two implementations can perform mathematically equivalent work while producing different numbers of dispatches and different intermediate-memory traffic. A benchmark taken before one fusion lands can become stale without any model-format change, quantization change, or hardware change.

This is especially relevant to llama.cpp because its release cadence is extremely fast. The fusion landed as merge commit 9ac8c408... and appeared in b10833 on September 7. A Vulkan result from an older build is not automatically representative of the current backend.

A useful way to inspect future performance patches

When a backend PR claims a speedup, I now think there are three separate questions worth asking.

First: what mathematical work changed? Sometimes the answer is none.

Second: what physical work changed? Here, the answer is the dispatch/fusion structure around repeated RMSNorm-related graph sequences.

Third: how broad is the evidence? Here, the source code clearly demonstrates the new execution paths, while the performance evidence in the PR is a single author’s approximate Gemma 4 observation. Those deserve different confidence levels.

That separation keeps source analysis useful without turning one developer measurement into a universal benchmark claim.

The larger lesson

Local LLM optimization is often described at the level of model size, bits per weight, KV-cache precision, or attention algorithms. Those are important because they are easy to see.

But inference engines also win or lose time in the seams between operations.

PR #28024 is a clean example. Gemma 4 did not become smaller. Its weights did not become more accurate. The model architecture did not change. llama.cpp’s Vulkan backend simply learned that several graph nodes which look separate at the framework level can be treated as a larger execution unit.

The reported payoff was only “around 4%.”

For a mature inference runtime, finding four percent by removing work the GPU never needed to treat separately is exactly the sort of small drama worth watching.