Porting Qwen4 to llama.cpp: Qwen3.8-Flash-Next Internals

Implementation-Level Architecture Analysis

This article was originally framed as an implementation strategy before a concrete upstream llama.cpp port existed. The arrival of PR #27742 now provides a direct test of those assumptions. The strategy remains useful because it explains why the successful port can reuse so much of llama.cpp and where the true architectural risk sits.

The main thesis is simple: Qwen3.8-Flash-Next looks much more exotic at the model-description layer than it does at the low-level tensor-algebra layer. Most of the arithmetic can be decomposed into operations llama.cpp already understands. The hard work is state topology, sparse memory selection, host-resident lexical storage, and preserving those semantics under heterogeneous execution.

A disciplined implementation plan should therefore optimize for semantic isolation first and backend specialization later.

Start From the Execution Graph, Not the Model Name

The text model combines four recurring ideas:

Gated DeltaNet
MoE
Qwen Sparse Attention
Hyper-Connections

with a giant hashed PLE table and model-specific positional behavior.

A useful decomposition is:

Qwen4Exp
|
|-- existing Gated DeltaNet substrate
|-- existing MoE substrate
|-- existing RoPE/MRoPE machinery
|-- model-local Hyper-Connection graph
|-- host-side PLE address generation
|-- row lookup through existing tensor operations
`-- QSA selector feeding indexed attention memory

This decomposition avoids the common porting mistake of treating every model-level novelty as a new backend primitive.

A backend operator should be introduced only when the semantics cannot be expressed reasonably from existing operations or when profiling proves that graph-level decomposition leaves an unacceptable bottleneck.

CPU Should Be the First Correctness Oracle

The first useful milestone for a new llama.cpp architecture is not maximum GPU offload. It is a CPU graph that agrees with the reference implementation.

The target should be conceptually:

llama-cli \
  -m Qwen3.8-Flash-Next.gguf \
  -ngl 0

and a short deterministic prompt whose logits agree closely enough with the reference to establish graph correctness.

CPU-first bring-up has three advantages.

First, it proves that model support is not secretly coupled to CUDA-specific behavior. Second, it narrows later accelerator debugging: if CPU is correct and CUDA diverges, the search space is the backend representation rather than the model graph. Third, it provides a stable oracle for graph changes, quantization changes, and sparse-selection refactors.

The later PR results validate this strategy: the qwen4exp architecture test reports essentially exact CPU behavior and a tiny CUDA delta.

Gated DeltaNet Is Mostly a Reuse Problem

The Qwen4Exp linear-attention path is structurally close to the Gated DeltaNet machinery already used by earlier Qwen models.

The recurrent state can be viewed schematically as:

$$ S_t = \alpha_t S_{t-1} + \beta_t\left(v_t - S_{t-1}k_t\right)k_t^\top. $$

The exact projection and gating parameterization is model-specific, but the state update itself does not demand a fundamentally new recurrence engine.

The implementation problem is therefore not “write Qwen4 linear attention from scratch.” It is “map Qwen4 projections, convolution state, and output gating onto the existing recurrent substrate without changing semantics.”

The output gate is an example of a small difference with large correctness consequences. Qwen4Exp uses a sigmoid output gate where the related Qwen3.5 path used SiLU. A port that silently inherits the older activation can produce plausible text while failing logit-level parity.

This is why model-local configuration must remain explicit even when the recurrent kernel is reused.

Hyper-Connections Change Residual Topology

A conventional transformer propagates one residual stream:

$$ x \in \mathbb{R}^{d}. $$

Qwen4Exp maintains multiple residual streams. For the released configuration:

$$ n_{\mathrm{HC}} = 4,\qquad d = 2560. $$

A useful representation is:

$$ H \in \mathbb{R}^{4 \times 2560}. $$

The expensive block itself does not need to operate at width 10240. The Hyper-Connection module reads from the persistent streams, constructs a block input at the ordinary hidden width, evaluates GDN/QSA/MoE, and then writes the result back into the multi-stream residual state.

The read path can be built from existing operations:

grouped normalization
    -> low-rank projection
    -> SiLU
    -> projection
    -> sigmoid
    -> elementwise weighting
    -> reduction across streams

The write path similarly uses ordinary projections and elementwise gates.

The architectural novelty is therefore residual-state topology, not a new mathematical primitive.

This distinction matters for backend design. A custom GGML_OP_HYPER_CONNECTION would prematurely encode model semantics into GGML. A model-local graph keeps the architecture isolated and lets every existing backend execute the same ordinary operations.

PLE Is Giant Sparse Lexical Storage

The visually alarming part of Qwen3.8-Flash-Next is the n-gram memory. The table is enormous, but the per-token working set is tiny.

Conceptually, token history produces deterministic n-gram hashes:

$$ h_n = (t_0m_0) \oplus (t_1m_1) \oplus \cdots \oplus (t_{n-1}m_{n-1}). $$

Each hash head maps the result into a vocabulary partition:

$$ i_j = (h_n \bmod V_j) + O_j. $$

The resulting indices select a small number of embedding rows from a huge table.

This is not ANN retrieval. It is not vector search. It is not attention over a giant memory bank.

It is sparse row addressing.

That distinction determines the implementation strategy.

token history
    -> integer hash
    -> row IDs
    -> sparse embedding rows
    -> small PLE feature vector

The PLE table should therefore be treated more like a host-resident sparse address space than like an accelerator matrix.

Parameter Count Is the Wrong Runtime Metric for PLE

Suppose a token selects 16 PLE heads and each head contributes 160 BF16 values. The useful payload is:

$$ 16 \times 160 = 2560\ \text{elements}. $$

At two bytes per BF16 element:

$$ 2560 \times 2 = 5120\ \text{bytes}. $$

That is about 5 KiB of useful row payload per token.

The table can be close to one hundred GiB while the token touches only a microscopic fraction of it.

The performance question is therefore not “how do we fit the table in VRAM?” The real questions are:

DRAM latency
page locality
TLB behavior
cache-line amplification
NUMA placement
mmap behavior
row quantization layout
memory-level parallelism

Bulk GPU migration would destroy the architecture’s main advantage: the table is huge precisely because most of it can stay cold for any individual token.

GGUF Metadata Must Preserve Address Semantics

PLE introduces a converter correctness problem that ordinary dense weights rarely expose.

Hash multipliers, vocabulary sizes, and offsets are integer address constants. They are not approximate neural weights.

A large integer routed through FP32 can lose low-order bits. For a dense matrix that might produce a small numerical perturbation. For a hash multiplier it changes the row address.

That is a categorical error.

The model format must therefore preserve exact PLE metadata. GGUF becomes more than a tensor container; it becomes an execution contract describing deterministic address generation.

This same principle applies to QSA metadata. Indexer head geometry, top-k budget, compression ratio, and layer identity directly determine which historical memory is visible.

A model can have every tensor shape correct and still execute the wrong architecture if those metadata are wrong.

Quantizing PLE Requires Streaming-Oriented Tooling

Conventional quantization code often assumes that a temporary representation of one tensor fits in host memory.

PLE breaks that assumption.

For a tensor with roughly 51.2 billion elements, an FP32 staging representation is approximately:

$$ 51.2 \times 10^9 \times 4 = 204.8 \times 10^9\ \text{bytes}. $$

That is already larger than many high-end workstations.

The correct staging geometry is row-banded:

read bounded rows
    -> transform/dequantize
    -> quantize
    -> write
    -> discard temporary band

Peak temporary memory then scales as:

$$ O(N_{\mathrm{table}}) \rightarrow O(N_{\mathrm{band}}). $$

This is not merely a Qwen4-specific workaround. It is a general tooling improvement for very large row-addressed tensors.

The model is forcing the quantizer to stop assuming that “one tensor” implies “one comfortably materializable object.”

QSA Should First Be Implemented as Selection Semantics

Qwen Sparse Attention does not require a new sparse-attention equation for initial correctness.

The important abstraction is:

selector
    -> selected historical blocks
    -> cache-cell IDs
    -> attention over selected memory

The selector is model-specific. The attention calculation itself can initially reuse existing machinery.

For the released configuration:

indexer budget     = 2048
compression ratio  = 4

The effective visible width is bounded by:

$$ w = \min\left(n_{\mathrm{kv}}, 2048 + 4 - 1\right) = \min(n_{\mathrm{kv}}, 2051). $$

Below that threshold the sparse path cannot omit anything. It therefore becomes an exact dense-reference oracle.

That is an unusually useful bring-up property.

The first implementation should exploit it aggressively:

short context
    -> selector path active
    -> no actual pruning possible
    -> require dense equality

Only after that boundary is exact should validation move to long-context sparse selection.

Above the Budget, Correctness Becomes Set Selection

Once the QSA budget binds, tiny score perturbations can change the discrete top-k set.

For selected sets A and B, Jaccard overlap is:

$$ J(A,B) = \frac{|A \cap B|}{|A \cup B|}. $$

This metric captures something that maximum score error does not. Two selector score vectors can be numerically almost identical while one block crosses the top-k boundary.

Sparse-attention validation therefore needs two layers:

continuous checks
    score/logit error

discrete checks
    selected block IDs
    overlap / precision / recall

This is a general lesson for inference engines implementing learned routing. Once floating-point outputs feed a discrete selector, numerical parity must be evaluated both before and after discretization.

The Indexer Cache Deserves Its Own Memory Semantics

An early implementation instinct would be to attach the QSA indexer state to the existing hybrid memory object because it is historically indexed and key-like.

That is not necessarily the right abstraction.

The ordinary K/V cache stores the representation attention consumes. The indexer cache stores state used to decide which memory locations attention will consume.

Those are different contracts.

A dedicated indexed-memory abstraction makes it easier to define:

allocation
sequence ownership
copy/clone semantics
remove semantics
save/load
context shift
parallel decoding behavior

The subsequent upstream review of PR #27742 moved in exactly this direction with llama_memory_hybrid_idx.

This is the architectural reason the port is more about memory than math.

State Save/Restore Must Include Hidden Address State

A one-shot llama-cli generation is an incomplete correctness test for this architecture.

The runtime must preserve:

GDN recurrent state
GDN convolution state
ordinary K/V
QSA indexer state
PLE predecessor-token history
PLE convolution state
Hyper-Connection residual streams

If session restore omits a hidden state component, the restored model can look valid for the first token while diverging thereafter.

PLE is especially subtle. Future hashes depend on prior token history. If the visible tensors are restored but the lexical-history state is not, the runtime will select the wrong rows even though all tensor dimensions remain valid.

A serious test matrix must include:

save
restore
continue generation

sequence clone
sequence remove
slot reuse
parallel sequences
context shift

For stateful sparse models, lifecycle semantics are part of architecture support.

CUDA Optimization Should Follow State Correctness

Once CPU parity is established, GPU work can proceed in layers.

The first accelerator milestone should move ordinary compute without changing sparse semantics:

GDN
MoE
Hyper-Connections
QSA projections
dense projections

PLE hashing and sparse host lookup can remain on CPU initially.

The next target is the QSA selector:

compressed key state
    -> GPU scoring
    -> reduction
    -> top-k

The final attention path can evolve incrementally.

A practical intermediate design is indexed gather:

selected block IDs
    -> gather selected K/V
    -> contiguous temporary K/V
    -> existing optimized attention

Only after profiling should the backend consider a native sparse kernel that directly consumes selected block IDs.

The optimized endpoint might be:

selected block IDs
    -> direct indexed K loads
    -> direct indexed V loads
    -> online softmax
    -> output accumulation

This avoids building large dense masks or materializing unnecessary K/V.

The key rule is that performance work must preserve the selector’s conditional-access semantics.

Quantized KV Is a Representation Contract

The later upstream implementation exposed an important backend invariant: quantized K/V may use a Hadamard-rotated storage basis.

If query and key are transformed consistently:

$$ Q’ = HQ,\qquad K’ = HK. $$

For orthonormal H:

$$ {Q’}^\top K’ = Q^\top H^\top HK = Q^\top K. $$

So attention scores are unchanged.

But an architecture-specific attention path must still participate in the same storage contract. If the normal attention builder rotates before cache write and unrotates the output, a QSA-specific path cannot simply bypass that representation because its high-level attention equation is correct.

This is a useful warning for CUDA optimization: alternate fast paths inherit backend invariants even when they implement the same model mathematics.

Three Sparse Subsystems, Three Different Traffic Patterns

MoE, PLE, and QSA are all sparse, but calling them all “offload” obscures the engineering.

PLE
    enormous address space
    tiny random row working set

MoE
    enormous expert parameter set
    large structured selected matrices

QSA
    enormous temporal address space
    bounded selected K/V set

Their bottlenecks differ.

PLE
    DRAM latency
    TLB/page locality
    row dequantization

MoE
    selected-weight bandwidth
    expert residency
    host/GPU transfer

QSA
    selector scoring
    top-k
    indexed K/V locality

A single global host/GPU placement rule is therefore unlikely to be optimal.

The architecture should be treated as several independent conditional-access systems sharing one execution graph.

Why Qwen4 Fits llama.cpp Conceptually

Qwen3.8-Flash-Next is built around a principle that aligns unusually well with explicit heterogeneous inference:

Total model capacity does not need to equal per-token data movement.

MoE applies that principle to parameter space.

PLE applies it to lexical memory.

QSA applies it to temporal memory.

The three mechanisms can be summarized as:

$$ \text{MoE} \rightarrow \text{conditional parameter activation}. $$

$$ \text{PLE} \rightarrow \text{conditional lexical-row activation}. $$

$$ \text{QSA} \rightarrow \text{conditional context activation}. $$

The nominal model size is therefore a poor proxy for token-level traffic.

An inference engine such as llama.cpp is interesting precisely because it exposes RAM residency, mmap behavior, quantization, accelerator placement, and state management directly enough to preserve that conditionality.

Proposed Bring-Up Sequence

A robust implementation sequence is:

1. architecture metadata and tensor map
2. CPU graph construction
3. Hyper-Connection residual topology
4. existing Gated DeltaNet reuse
5. existing MoE reuse
6. exact PLE integer metadata
7. host PLE row lookup
8. QSA selector with dense-equivalence validation
9. dedicated indexed-memory lifecycle
10. full CPU/reference parity
11. ordinary CUDA offload
12. GPU QSA selector
13. indexed K/V gather
14. native sparse attention only if profiling justifies it

This order deliberately delays heroic kernel work.

The first meaningful definition of support is not “a fused CUDA path exists.” It is:

the GGUF conversion is semantically exact
the CPU graph matches reference
persistent state survives lifecycle operations
sparse selection is correct
heterogeneous placement preserves conditional access

Everything after that is optimization.

Final Strategy

Before concrete code existed, Qwen4Exp looked like it might require several new backend primitives.

It does not.

Most of the mathematical substrate is already present:

Gated DeltaNet      -> reusable
MoE                 -> reusable
RMSNorm / RoPE      -> reusable
dense projections   -> reusable
convolution         -> reusable

Hyper-Connections are graph-level composition over familiar operations.

PLE is enormous but computationally simple: exact deterministic hashing followed by sparse row lookup and conventional post-processing.

QSA is the subsystem that demands the most new performance engineering, but even there the clean abstraction is selection of cache addresses, not invention of a new attention equation.

The correct engineering trajectory is therefore:

semantic correctness
    -> complete state lifecycle
    -> heterogeneous placement correctness
    -> long-context profiling
    -> targeted kernel specialization

The real challenge is not teaching GGML new mathematics.

It is preserving Qwen4’s conditional memory access all the way from model semantics to RAM, mmap, cache representation, PCIe traffic, and accelerator execution.

That is the implementation problem the later upstream PR confirmed in practice.