Qwen4 in llama.cpp: Dissecting PR #27742

Qwen3.8-Flash-Next turns the Qwen4 discussion from architecture speculation into runtime engineering. The interesting artifact is not the model card by itself but the concrete llama.cpp code, because a real port exposes which parts of an architecture are genuinely new to an inference engine and which parts are only new compositions of machinery the engine already has.

The implementation examined here is llama.cpp PR #27742, model: add Qwen3.8-Flash-Next (qwen4exp), authored by Daniel Hanchen of Unsloth. This analysis follows its August 26, 2026 revision. The PR merged on August 27; the code and review observations below describe the pinned pre-merge revision. At that revision GitHub marked it as draft: false, although the PR body retained its original “Draft” wording from the period when the weights were not publicly reproducible. This article therefore analyzes an implementation under active review, not merged llama.cpp behavior.

There is also a chronology correction worth making before discussing “the first Qwen4 port.” PR #27739, model : add Qwen4-Exp, opened roughly fifteen minutes before #27742 and was later closed unmerged after the overlap became clear. So #27742 is not literally the first Qwen4Exp pull request by timestamp. It is the implementation that remained active and attracted the deeper maintainer review of cache semantics, state lifecycle, PLE handling, and upstream abstraction boundaries.

The central result is unusually clean:

git diff master --stat -- ggml/

is empty.

PR #27742 adds no new GGML operation and modifies no existing GGML operation. The difficult work is one layer above the operator algebra: state topology, memory indirection, sparse selection, converter semantics, giant-tensor staging, and backend representation invariants.

The Most Important Diff Is the Empty One

A new model architecture often creates an instinct to search for a missing kernel. In llama.cpp that instinct is expensive. A genuinely new GGML operation creates implementation and maintenance obligations across CPU, CUDA, Metal, Vulkan, SYCL, RPC, and future backends. Even if only one backend is optimized initially, the operation becomes part of GGML’s mathematical vocabulary and portability surface.

Qwen3.8-Flash-Next largely avoids that expansion.

The decomposition reported by the PR author and visible in the branch is approximately:

Gated DeltaNet
    -> existing delta-net substrate

MoE
    -> existing build_moe_ffn()

interleaved partial MRoPE
    -> existing ggml_rope_multi()

Hyper-Connections
    -> model-local graph composition in qwen4exp.cpp

PLE n-gram memory
    -> host integer addressing + ggml_get_rows()

QSA
    -> architecture-local selector + indexed cache-cell visibility

Vision
    -> existing Qwen3-VL clip path

The Gated DeltaNet path is particularly revealing. Qwen4Exp reuses the existing delta-net base. The architectural delta called out by the PR is the output gate: Qwen3.5 uses SiLU while this Qwen4Exp checkpoint uses sigmoid. That is a numerically important semantic difference, but it does not require a new recurrence primitive.

The MoE is similarly evolutionary: 512 experts, top-10 routing, and a gated shared expert are lowered through the existing build_moe_ffn() path. Interleaved MRoPE reuses ggml_rope_multi(). Hyper-Connections remain self-contained in qwen4exp.cpp and decompose into ordinary normalization, matrix multiplication, activation, elementwise multiplication, reshape, and reduction operations.

The architecture therefore looks exotic at the paper level while remaining surprisingly conventional at the GGML level.

A better abstraction is:

$$ \text{Qwen4 support} = \text{existing tensor algebra} + \text{new state topology} + \text{new addressing semantics} + \text{new placement constraints}. $$

Qwen4 Is a Memory-Routing Architecture

The most useful way to reason about Qwen3.8-Flash-Next is not as “a 125B MoE with sparse attention.” It is as a model that performs conditional access across several independent address spaces.

MoE selects a subset of parameter space. PLE selects a tiny subset of a gigantic lexical embedding address space. QSA selects a bounded subset of historical context. Gated DeltaNet maintains recurrent state instead of repeatedly materializing an attention history for three of every four layers. Hyper-Connections preserve multiple residual streams across the layer stack.

The runtime therefore contains several forms of persistence and indirection that do not share the same lifecycle:

Qwen4 runtime state
|
|-- GDN recurrent state
|-- GDN convolution state
|-- ordinary attention K/V
|-- QSA indexer/cache state
|-- PLE predecessor-token history
|-- PLE convolution state
|-- Hyper-Connection residual streams
`-- multimodal positional/token state where applicable

The classical model:

$$ \text{Model} = \text{Weights} + \text{KV cache} + \text{forward graph} $$

is no longer sufficient. A more accurate systems model is:

$$ \text{Model} = \text{Computation Graph} + \text{State Topology} + \text{Memory Placement Policy} + \text{Addressing Semantics}. $$

The computation graph is only one term. The other terms determine whether sequence cloning works, whether session restore is exact, whether an offload policy causes pathological data movement, and whether sparse execution remains sparse after it reaches real hardware.

A first-order decode-time model should therefore separate indirection costs rather than hiding them inside a single “model time” term:

$$ T_{\mathrm{token}} = T_{\mathrm{GDN}} + T_{\mathrm{MoE}} + T_{\mathrm{PLE}} + T_{\mathrm{QSA-index}} + T_{\mathrm{QSA-attn}} + T_{\mathrm{state}}. $$

Several of these terms are bounded by memory locality and representation transitions rather than raw FLOPs.

QSA Is an Address-Selection Problem, Not a New Attention Equation

Qwen Sparse Attention is the subsystem most likely to tempt an implementation toward an architecture-specific backend operation. PR #27742 takes the opposite route.

There is no ggml_qsa_attention operation. The model-specific graph produces a set of cache-cell addresses and then reuses existing attention machinery over the selected memory.

The execution shape is approximately:

indexer projections
    -> compressed-block representation
    -> indexer scoring
    -> top-k selection
    -> cache-cell IDs
    -> sparse/indexed KQ mask
    -> existing attention machinery

The learned selector is architecture-specific. The lower-level contract is much more general:

selector
    -> visible cache-cell IDs
    -> attention over those cells

That is a memory interface, not a new attention equation.

For the released configuration, the PR reports an indexer budget of 2048 and a compression ratio of 4. If n_kv is the number of cached positions, the effective visible width is bounded by:

$$ w = \min\left(n_{\mathrm{kv}}, 2048 + 4 - 1\right) = \min(n_{\mathrm{kv}}, 2051). $$

That boundary gives the port an unusually strong correctness oracle. Below 2051 cached positions, the selector cannot discard anything. Sparse execution must be equivalent to the dense reference because there is no legal memory to omit.

The PR author reports exactly that: QSA versus dense is bit-identical below the budget, with maximum logit delta 0.0 across all 2051 rows. This simultaneously exercises block expansion, causal-tail handling, cache-cell addressing, mask construction, and the main attention path.

Once the budget binds, validation changes character. At 8192 tokens the author reports divergence on roughly 3% of positions between QSA and a forced-dense comparison. The indexer comparison against the reference reports mean Jaccard overlap around 0.975, with a quoted precision floor around 0.991.

For selected sets A and B:

$$ J(A,B) = \frac{|A \cap B|}{|A \cup B|}. $$

This metric matters because top-k selection is discontinuous. Two score vectors can be numerically extremely close while a tiny perturbation near the cutoff exchanges one selected block for another. Above the sparse budget, numerical parity is therefore partly a set-selection problem.

A useful validation rule is: below the activation threshold, demand dense equality; above it, inspect both score error and discrete address-set overlap.

Maintainer Review Moved the Abstraction Boundary

The original PR body says QSA uses a new graph plus an optional third cache in llama_memory_hybrid. That description became stale during review.

Georgi Gerganov’s first substantive review focused on llama-kv-cache and llama-memory-hybrid, not on GGML operators. He said the implementation generally looked reasonable, noted positively that most changes lived in qwen4exp.cpp, and asked whether indexed behavior should be isolated into something like llama-memory-hybrid-idx. He also questioned missing state save/load for the indexer and whether checkpoint save/restore would work.

The author then refactored the branch in that direction.

The current branch contains a new llama_memory_hybrid_idx. Daniel reported that after the refactor, the ordinary llama-kv-cache, llama-memory-hybrid, and llama-graph files were again byte-identical to master. He also reported an exact pre/post-refactor graph comparison over 8191 rows, with maximum delta 0.0 for both sparse and forced-dense paths.

The architectural lesson is stronger than the line-count reduction. Indexed memory looked structurally similar to an existing hybrid cache, but its semantics were different enough to justify a separate memory class.

The main K/V cache contains the representation that attention consumes. The QSA indexer cache exists to decide where attention should look. Two tensors being “key-like” is not enough reason for them to share a lifecycle, serialization contract, or storage invariant.

The PLE discussion points in the same direction. Predecessor-token history participates in future address generation. If that history lives outside the serialized runtime memory object, a restored session can recover visible tensors while silently restoring the wrong n-gram state.

This is why state save/load is not a peripheral feature for Qwen4. Hidden address-generation state is part of model correctness.

PLE Is a 97.7 GiB Address Space With Exact Integer Semantics

PLE is the subsystem that makes the model look absurdly large while touching very little data per token.

PR #27742 describes the n-gram table as approximately 97.7 GiB. Operationally it is not a dense matrix that must be streamed through an accelerator. It is a gigantic row-addressed lexical memory.

The execution path is conceptually simple:

token history
    -> host-side n-gram hashing
    -> exact integer row indices
    -> ggml_get_rows(PLE, indices)
    -> small selected payload
    -> downstream PLE processing

There is no ANN database, no vector search index, and no Qwen-specific GGML retrieval primitive.

The critical detail is that the hash constants are not approximate neural weights. They are part of the address function. The converter must preserve values such as layer multipliers, head offsets, and head vocabulary sizes exactly as 64-bit integers.

A simplified head address is:

$$ i_j = (h \bmod V_j) + O_j. $$

Here h is the deterministic n-gram hash, V_j is the head vocabulary size, and O_j is the row offset.

The converter exposed a subtle failure mode: ordinary preparation could route non-floating data through FP32 before architecture-specific handling. That is unsafe for large roughly 45-bit integer constants. FP32 cannot represent arbitrary integers at that magnitude exactly.

For a dense weight, a one-bit representation error may be a small numerical perturbation. For a hash multiplier, one-bit corruption is semantic corruption. The result is not a slightly different row weight. It is a different row address.

That is why Qwen4Exp’s converter treats these constants as exact metadata and why GGUF metadata here should be understood as an execution contract, not descriptive annotation.

The runtime needs enough metadata to reconstruct Hyper-Connection geometry, QSA indexer dimensions and budget, attention compression, PLE layer identity, n-gram configuration, hash multipliers, offsets, and vocabulary partitions. Shapes alone are insufficient to recover execution semantics.

Giant-Tensor Quantization Breaks Whole-Table Staging

PLE also exposes an assumption that is almost invisible with ordinary transformer weights: conversion and quantization paths often assume that a temporary representation of one tensor can fit comfortably in host memory.

That assumption fails when one logical tensor contains roughly 51.2 billion elements.

The PR author describes an older work-buffer estimate based on roughly:

$$ 4N $$

bytes for a tensor with N elements. Applied to PLE geometry, the conservative bound produced a process allocation on the order of 150 GB even though the required size was already known more precisely.

Fixing that estimate was necessary but not sufficient.

A later PR-thread report found another path that failed earlier: f32_conv_buf attempted to materialize the entire dequantized PLE tensor in FP32. The arithmetic is direct:

$$ 51.2 \times 10^9 \times 4 = 204.8 \times 10^9\ \text{bytes}. $$

That is about 204.8 GB of decimal payload for one temporary buffer before allocator overhead. A 96 GB machine cannot reach the later quantizer logic because staging itself fails with std::bad_alloc.

The correct shape is row-banded processing:

read a bounded row band
    -> dequantize/transform band
    -> quantize band
    -> write band
    -> release band
    -> repeat

Peak staging changes from whole-table scale to bounded-band scale:

$$ O(N_{\mathrm{table}}) \rightarrow O(N_{\mathrm{band}}). $$

A community contributor reported that this direction reduced staging to a few gigabytes and allowed Q4_K_M conversion to complete on a 96 GB machine. That is a PR-thread report, not a statement that row-banded quantization is merged in llama.cpp master.

The broader lesson is that tensor geometry can invalidate tooling assumptions even when the model graph itself is straightforward.

PLE Residency Is a Placement Problem, Not Merely a Capacity Problem

The PLE table is semantically attached to the model but operationally behaves unlike ordinary dense weights. It is enormous, mostly cold, and accessed by sparse row lookup.

That makes blanket offload rules dangerous.

A useful placement model is:

GPU / accelerator candidates
--------------------------------
GDN compute
QSA projections
Hyper-Connection projections
selected MoE compute
ordinary attention K/V
small PLE post-processing

CPU / system-memory candidates
--------------------------------
PLE hash generation
PLE table
cold/offloaded experts
mmap backing

The important metric for PLE is not nominal parameter count. It is random-row locality, page behavior, TLB pressure, cache-line amplification, NUMA placement, and whether the selected rows can be issued with enough memory-level parallelism.

A PR-thread report on DGX Spark illustrates why “mmap works” is not the same as “mmap is optimal.” One user reported model load time around 22 seconds with mmap disabled and around 224 seconds with mmap enabled on that platform. They also reported that once loaded, mmap-backed PLE did not obviously destroy inference performance in their testing.

Those numbers are platform-specific community observations, not universal behavior, but they expose the correct engineering question: PLE needs an independent memory-placement policy because its access pattern is qualitatively different from dense model weights.

Quantized KV Exposed a Backend Storage Invariant

Community testing found another class of failure that had nothing to do with architecture-level attention semantics.

A tester reported that qwen4exp crashed with quantized KV configurations such as:

-ctk q8_0 -ctv q8_0

at an assertion equivalent to:

GGML_ASSERT(inp->self_k_rot == nullptr && inp->self_v_rot == nullptr)

The reason is a llama.cpp storage invariant. For compatible dimensions, quantized K/V can be stored in a Hadamard-rotated basis to spread channel outliers before block quantization. The normal attention path understands that representation. The QSA-specific path initially assumed those rotation tensors were absent.

If H is orthonormal and the query/key representations are rotated consistently:

$$ Q’ = HQ,\qquad K’ = HK,\qquad {Q’}^{\top}K’ = Q^{\top}H^{\top}HK = Q^{\top}K. $$

The attention scores are preserved.

The PR thread contains a community-proposed and CUDA-tested patch that mirrors the normal attention path:

rotate Q/K/V
    -> write rotated K/V into cache
    -> attention in rotated basis
    -> unrotate attention output
    -> nonlinear gate
    -> output projection

The ordering matters. The output must be returned to the original basis before nonlinear gating because the gate does not generally commute with the Hadamard transform.

For an involutive Hadamard transform:

$$ H^2 = I. $$

The same transform can be used to return the value output to the original basis.

Crucially, this patch was discussed and tested in the PR thread; it should not be described as merged llama.cpp behavior, and it should not be described as present in a branch head unless the head is rechecked after the patch lands.

The QSA indexer cache does not require the same transformation because it ultimately produces cache-cell addresses. Rotating the main K/V representation changes what attention reads, not which cell IDs the selector returns.

This is an excellent example of a broader inference-engine failure mode: architecture semantics can be correct while an alternate execution path violates a backend storage invariant.

The PR’s Numerical Evidence Is Layered Correctly

The PR author reports several independent correctness checks:

WikiText-2 perplexity, 145 chunks, ctx 2048:
llama.cpp  4.0068 +/- 0.02271
reference  4.0126

top-1 agreement over 512 prose tokens:
98.0%

QSA vs dense below sparse budget:
bit identical
max logit delta 0.0 over 2051 rows

QSA vs dense above budget at 8192:
diverges on roughly 3% of positions

indexer selection vs reference:
mean Jaccard about 0.975
quoted precision floor about 0.991

test-llama-archs -a qwen4exp:
CPU   0.00e+00
CUDA  8.00e-08
roundtrip OK

The important point is not any single number. It is the validation hierarchy.

CPU is a semantic oracle. CUDA is compared against the CPU graph. QSA has an exact dense-equivalence regime below the sparse threshold. Above the threshold, set-selection metrics supplement logit comparisons. Converter/architecture roundtrip checks cover serialization.

For a stateful sparse architecture, a single “it generated coherent text” check would be almost meaningless.

Community Performance Numbers Need Strict Provenance

One PR-thread user reported:

Qwen3.8-Flash-Next-BF16.gguf      354 GB
Qwen3.8-Flash-Next-Q4_K_M.gguf    119 GB

pp4096 ~ 750 tok/s
tg128  ~ 95 tok/s

These values are useful as evidence that the port was already being exercised in real local-inference configurations, but they are not RAMGPT benchmark numbers and should not be treated as portable performance claims.

These are community measurements reported in the PR thread.

The file-size reduction is still operationally significant. A roughly 119 GB artifact enters the range where 128 GB host-memory systems become interesting, but “fits in RAM” is not the same as “runs efficiently.” PLE residency, expert placement, KV representation, and page-cache behavior still determine whether the architecture’s sparsity produces low real traffic.

Why Short-Context Tests Can Lie

QSA has two execution regimes.

Below the effective 2051-position visibility threshold:

$$ T_{\mathrm{QSA}}(L) \approx T_{\mathrm{dense}}(L)\quad\text{for }L \lesssim 2051. $$

Once the budget binds, a better approximation is:

$$ T_{\mathrm{QSA}}(L) \approx T_{\mathrm{index}}(L) + T_{\mathrm{attn}}(2048). $$

The main attention width becomes bounded, while the indexer still scores an increasing number of compressed candidates.

At a 262,144-token context and compression ratio 4, the selector may address on the order of:

$$ \frac{262144}{4} = 65536 $$

compressed blocks per sparse-attention layer.

That means a short 1K or 2K test does not exercise the architecture’s defining sparse-selection regime. A port can be perfectly correct and apparently fast at short context while still being wrong, slow, or synchronization-heavy once QSA becomes active.

A serious test matrix needs to cross at least:

CPU / CUDA
F16 KV / quantized KV
<2051 cache / >2051 cache
4K / 8K / 32K / 128K / 262K context
single sequence / multiple sequences
save / restore
sequence clone / remove
context shift
PLE cold pages / warm pages
different GPU offload levels

For this model, short-context correctness is not long-context correctness.

The State-Lifecycle Test Matrix Is More Important Than Another Kernel

Maintainer review was right to focus on save/load and memory boundaries. A complex persistent model can pass perplexity while still failing server semantics.

The minimum lifecycle matrix should include:

prefill
decode
save session
restore session
continue decode

sequence clone
sequence remove
slot reuse
parallel sequence execution
context shift

Each operation must preserve every state component whose future outputs depend on prior tokens.

For ordinary Transformers this often means K/V plus positional bookkeeping. For Qwen4Exp it includes recurrent state, convolution state, ordinary K/V, QSA selector state, PLE lexical history, and Hyper-Connection residual topology.

This is exactly why the hard part is no longer “can GGML compute the equations?”

The hard part is “can the runtime preserve the model’s complete state topology while sequences are created, moved, serialized, restored, and partially discarded?”

What Should Be Optimized After Correctness

No new GGML operation is required for functional support, but that does not mean the current decomposition is performance-complete.

The likely optimization targets are the boundaries where conditional access becomes hardware traffic:

PLE
    random host-resident row access
    row dequantization
    page/TLB locality

MoE
    expert residency
    selected-weight transfer
    expert batching

QSA indexer
    block scoring
    top-k
    selector cache locality

QSA attention
    indexed K/V gather
    sparse K/V access
    attention over selected cells

The selector is a particularly plausible future fusion target. The absence of a custom ggml_qsa_attention op today says that no new mathematical primitive is needed for correctness. It does not say that architecture-specific or backend-specific kernels will never be worthwhile.

The right sequence is still:

correct model-local graph
    -> numerical parity
    -> state-lifecycle parity
    -> profile real long context
    -> fuse measured bottlenecks

Prematurely inventing a new backend op would move semantics into the most expensive layer of the stack before the correct abstraction boundary is known.

A Better Mental Model for llama.cpp Model Support

The Qwen4Exp port suggests a more useful decomposition for modern inference-engine support.

Traditional support questions often reduce to:

Can the runtime load the weights?
Can it build the forward graph?
Does CUDA have the kernels?

Qwen4 requires additional questions:

What state persists across decode?
Which state is serializable?
Which state is sequence-local?
Which tensors are row-addressed rather than densely consumed?
Which selectors return values and which return addresses?
Which representations change under quantized storage?
Which placement rules preserve sparsity instead of creating bulk movement?

That leads to a more general systems equation:

$$ \text{Model support} = \text{operator coverage} + \text{state lifecycle} + \text{representation invariants} + \text{placement policy}. $$

For Qwen3.8-Flash-Next, operator coverage is the least surprising term.

Final Assessment

PR #27742 substantially changes the engineering picture established in our earlier Qwen4 implementation strategy.

Before concrete code existed, it was reasonable to ask whether Hyper-Connections, PLE, QSA, or the Gated DeltaNet variant would require substantial expansion of GGML itself.

The first serious upstream candidate shows the opposite.

GDN                -> reuse
MoE                -> reuse
MRoPE              -> reuse
Hyper-Connections  -> compose
PLE                -> exact hash metadata + get_rows
QSA                -> select cache cells + reuse attention

The complexity sits at the boundaries between those components.

The most revealing review comments are therefore about llama-memory-hybrid, indexed memory isolation, state save/load, predecessor-token history, and backend storage representations. The most revealing community bug is a quantized-KV crash caused by a QSA-specific path violating a Hadamard-rotated cache invariant. The most revealing tooling failures are giant temporary allocations created by whole-tensor assumptions around the 51.2-billion-element PLE table.

All of those failures share the same pattern: the tensor algebra is valid, but the runtime’s assumptions about memory are too narrow.

The hard part is making sparsity survive contact with the hardware.

Qwen4 does not require expanding GGML’s mathematical vocabulary. It requires expanding llama.cpp’s notion of model memory.