Qwen4 in llama.cpp: A Pre-Merge Runtime Audit of PR #27742

The first implementation question for a new architecture is usually whether the forward graph is mathematically correct. For Qwen3.8-Flash-Next, that question is increasingly becoming the easy part.

The active llama.cpp implementation in PR #27742 already reports strong reference agreement on ordinary inference: WikiText-2 perplexity close to the reference implementation, 98 percent top-1 agreement over a 512-token prose run, bit-identical QSA behavior below the sparse-selection budget, and near-exact architecture-test parity on CPU and CUDA. As of August 26, 2026, however, the PR remains open and unmerged at head 035e22731a7fd70b9854b3a2d64ec68e9b1a45d3.

The remaining risk is not primarily another missing activation function or tensor transpose. It is that Qwen4Exp introduces several forms of persistent state that must survive every operation llama.cpp is allowed to perform on a context: slot allocation, sequence copy, sequence removal, cache shift, rewind, session serialization, session restore, multi-slot scheduling, unified-KV execution, quantized-KV representation changes, graph reservation, and multimodal input substitution.

That changes what “correct model support” means.

$$ \text{correct support} = \text{forward parity} + \text{state-lifecycle parity} + \text{representation parity} + \text{scheduler parity}. $$

PR #27742 is already surprisingly strong on the first term. The pre-merge audit should now focus on the other three.

The Forward Graph Is No Longer the Main Risk

The architectural surface is impressive but relatively well contained. Qwen4Exp reuses the existing delta-net substrate, existing MoE machinery, existing RoPE/MRoPE operations, ordinary GGML tensor algebra for Hyper-Connections, host-side integer addressing plus ggml_get_rows() for PLE, and existing attention machinery behind a model-specific QSA selector.

That is why one of the most important properties of the PR remains the empty GGML diff: no new core GGML operation is required.

The danger is that this cleanliness can create a false sense of completeness. A forward pass can be numerically correct while the model remains incorrect as a llama.cpp runtime citizen.

For a conventional decoder-only transformer, much of persistent execution state can be approximated as “the KV cache.” Qwen4Exp has a more complicated topology:

Qwen4Exp persistent execution state
|
|-- ordinary attention K/V
|-- QSA indexer keys
|-- Gated DeltaNet recurrent state
|-- recurrent convolution state
|-- PLE convolution state
|-- PLE predecessor-token history
`-- scheduler/cache metadata mapping sequences to physical cells

Each of these state components has to move through the same lifecycle. If one component is copied while another is not, or one is restored while another remains empty, the model can continue running while producing the wrong sparse memory addresses.

That is the class of bug that should worry reviewers most.

P0: The QSA Indexer Cache Is Not Serialized

The largest explicit state-management gap is the new QSA indexer cache.

After maintainer review, the implementation moved QSA-specific memory behavior into a separate llama_memory_hybrid_idx abstraction. This is a good architectural change. It keeps ordinary llama_memory_hybrid behavior isolated and gives Qwen4Exp a third cache dedicated to indexer keys.

Conceptually the object is:

llama_memory_hybrid_idx
|
|-- mem_attn
|-- mem_recr
`-- mem_idx

The problem is that llama_memory_hybrid_idx does not currently override the state serialization interface.

The parent llama_memory_hybrid implementation writes the attention cache and recurrent memory:

void llama_memory_hybrid::state_write(...) const {
    if ((flags & LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY) == 0) {
        mem_attn->state_write(io, seq_id, flags);
    }
    mem_recr->state_write(io, seq_id, flags);
}

and restores the same two components:

void llama_memory_hybrid::state_read(...) {
    if ((flags & LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY) == 0) {
        mem_attn->state_read(io, seq_id, flags);
    }
    mem_recr->state_read(io, seq_id, flags);
}

The new mem_idx does not participate.

Georgi Gerganov identified this directly in review: indexer state save/load appeared to be missing, which would make checkpoint save/restore unreliable.

The important failure mode is not necessarily a crash.

Consider a session with more than 2051 cached positions, where QSA is genuinely sparse:

prompt
  -> attention K/V populated
  -> indexer K populated
  -> QSA top-k selects historical cells
  -> save session
  -> destroy context
  -> restore session

After restore, the ordinary attention history can be valid while the indexer history is absent or stale.

The next token can therefore execute over a valid K/V cache using an invalid selector state. The attention kernel itself is functioning. The sparse address set is wrong.

That produces one of the worst possible inference-engine bugs: coherent output with corrupted hidden execution semantics.

This should be treated as a merge blocker, not a future optimization.

P0: PLE History Currently Has the Wrong Owner

The second design-level problem is more fundamental.

Qwen4Exp needs predecessor token IDs to construct its hashed n-gram PLE addresses. The current implementation stores that predecessor history directly on the model object:

struct llama_model_qwen4exp : public llama_model_base {
    struct ple_history {
        llama_pos next_pos = -1;
        std::vector<llama_token> toks;
    };

    mutable std::unordered_map<llama_seq_id, ple_history> ple_hist;
};

The PLE graph input then mutates pmodel.ple_hist as batches are processed.

This is convenient, but it violates the runtime ownership boundary.

A model object represents immutable model structure and weights. Sequence history belongs to the execution context or memory object. Once mutable sequence state is attached to the model, every operation that modifies memory can leave the PLE history behind.

The current design is vulnerable to at least four classes of desynchronization.

First, clearing memory does not naturally clear a map owned by the model. Reviewer ngxson has now called this out explicitly: ple_hist is not attached to KV memory and can go out of sync as soon as memory is cleared.

Second, session state serialization does not include it.

Third, memory operations such as seq_rm, seq_cp, seq_keep, seq_add, and seq_div operate through llama_memory_i. The model-level PLE map is outside that contract.

Fourth, multiple contexts can share the same model.

That last case is particularly dangerous. Two contexts can legitimately use the same llama_model while both use sequence ID zero:

context A, seq 0:
"The capital of France is ..."

context B, seq 0:
"def fibonacci(n): ..."

With a model-global map keyed only by llama_seq_id, both execution contexts address the same ple_hist[0].

Even without explicit concurrency, ownership is wrong. With concurrent execution, it also becomes a synchronization problem because the map is mutable and model-global.

This is not a cosmetic abstraction complaint. It can become cross-context state contamination.

PR #27762 Is the Right Architectural Direction for PLE

The appearance of PR #27762, llama: add token ID tracking to KV cell, is therefore important.

That PR extracts a generic mechanism motivated by both n-gram architectures and Qwen4Exp: retain token IDs alongside KV-cell metadata so that n-gram lookups can reconstruct predecessor tokens from the actual memory topology.

This changes the ownership model from:

model
`-- ad hoc mutable predecessor map

to:

memory cell
|-- position
|-- sequence membership
|-- K/V
`-- token identity

That is much closer to the correct abstraction.

If a predecessor token is part of a sequence’s persistent execution history, it should follow the same operations that already define the lifetime of that history:

clear
seq_rm
seq_cp
shift
save
restore
slot reuse

PR #27762 merged on August 26, and the model-support PR #27742 merged on August 27. This audit examines the earlier pinned revision; its risk list describes that review point. The correct long-term Qwen4 implementation probably should not carry ple_hist as model-local mutable state at all.

P0: Multi-Slot Execution Already Exposes Cache Desynchronization

The lifecycle problem is not only theoretical.

A community tester reported a reproducible llama-server failure with the current branch using the default multi-slot configuration. The first request completed successfully. A later request landed on a different LRU-selected slot and aborted at the Qwen4Exp assertion that requires the indexer cache and attention cache to have identical logical lengths.

The assertion is:

GGML_ASSERT(
    mctx_idx->get_n_kv() == inp->mctx->get_attn()->get_n_kv() &&
    "the indexer cache must track the attention cache cell for cell");

The workaround reported in the PR thread was --parallel 1.

This is exactly the kind of failure that the new llama_memory_hybrid_idx abstraction was intended to prevent. The implementation now hands the indexer cache the attention cache’s slot descriptors rather than asking it to allocate slots independently.

That is necessary, but it is not yet sufficient.

Sharing a slot layout is weaker than sharing a lifecycle.

Two buffers can be told that physical cell 137 represents the same token and still diverge later if one of them receives a stream copy, shift, restore, clear, or slot transition that the other does not.

The correct invariant is not merely:

attention slot descriptor == indexer slot descriptor

It is:

for every legal memory operation:
logical token identity(attention cell j)
==
logical token identity(indexer cell j)

That invariant must hold over time, not just at batch preparation.

P1: The Indexer Cache Does Not Participate in init_update()

There is an additional lifecycle hole visible directly in the current implementation.

llama_memory_hybrid_idx::init_update() returns a llama_memory_hybrid_idx_context, but the update-context constructor initializes only the parent hybrid context:

llama_memory_hybrid_idx_context(
        llama_memory_hybrid_idx * mem,
        llama_context * lctx,
        bool optimize) :
    llama_memory_hybrid_context(mem, lctx, optimize),
    mem(mem) {}

The header explicitly documents that ctx_idx is null for full and update contexts.

That matters because KV-cache update contexts are not ceremonial. llama_kv_cache::init_update() consumes pending state such as K-shift and queued stream-copy information.

For multi-stream caches, seq_cp() can update metadata immediately while deferring physical K/V buffer copies into sc_info for the next update cycle.

Qwen4Exp currently forwards sequence copy into the indexer cache:

attention.seq_cp(...)
recurrent.seq_cp(...)
indexer.seq_cp(...)

but the subsequent update phase does not appear to create or apply an indexer update context.

The resulting failure mode is subtle:

indexer metadata:
    destination sequence looks copied

indexer K buffer:
    destination stream data not physically copied

QSA can then read a cell that is logically valid but contains stale indexer vectors.

This deserves a targeted unit test even if the currently reported multi-slot assertion is fixed by another patch. Otherwise a crash can simply turn into silent sparse-selection corruption.

P1: Unified-KV Block Construction May Mix Sequences

There is another potential correctness issue in set_input_qsa() that deserves adversarial testing.

The host-side QSA preparation maps cache cells into compressed blocks using position:

const llama_pos p = cells.pos_get(j);
const int64_t b = p / r;

blk_of[j] = (int32_t) b;
cur_blk_cells[b*r + (p%r)] = (int32_t) j;
filled[b]++;

The block key is effectively:

position / compression_ratio

within a stream.

It is not explicitly keyed by sequence ID.

That is safe if each stream contains only one logical sequence. It becomes suspicious under unified-KV execution, where multiple sequences can share one physical cell stream.

Suppose two sequences both contain positions zero through three and the compression ratio is four:

sequence A: positions 0 1 2 3
sequence B: positions 0 1 2 3

Both map into block zero.

The cur_blk_cells entries for one sequence can overwrite the entries from the other, while filled[0] can rise above the expected block width.

The later bias construction does check cells.seq_has(j, seq_id) before exposing a token to a query, which protects final visibility. But QSA does not select individual cells directly at the beginning of the pipeline. It first pools indexer keys into block representations and scores those blocks.

If block membership has already mixed sequence identities, a later visibility mask cannot retroactively repair the pooled selector representation.

This is not yet a confirmed upstream bug. It is an implementation-level risk inferred from the current code and should be tested with a minimal unified-KV case containing overlapping position ranges across sequences.

A correct block identity may need to be defined over something closer to:

(stream, sequence, compressed position)

rather than compressed position alone.

P0/P1: Quantized KV Violates the QSA Representation Contract

The quantized-KV failure is already confirmed.

Current Qwen4Exp code begins build_attn_qsa() with:

GGML_ASSERT(inp->self_k_rot == nullptr && inp->self_v_rot == nullptr);

That condition is false when llama.cpp enables its Hadamard-rotated representation for compatible quantized K/V caches.

The standard attention path understands this representation. Qwen4Exp’s separate QSA path currently does not.

For quantized KV, llama.cpp can store K and V in a rotated basis to spread channel outliers before block quantization. Queries and keys must be transformed consistently. With an orthonormal Hadamard transform H, rotating both Q and K preserves the dot product because H^T H = I.

A community-proposed patch in the PR thread applies the same representation contract to Qwen4Exp:

before cache write:
    Q -> H Q
    K -> H K
    V -> H V

run attention in rotated representation

before nonlinear gate/output projection:
    attention output -> H output

The placement of the final transform matters. It cannot be pushed through arbitrary nonlinear gating and projection logic.

Community testing reported coherent CUDA output for q8/q8, q8/f16, and f16 configurations, including prompts long enough to activate the sparse QSA path.

But at the inspected PR head, the original assertion is still present.

This is an excellent example of why backend representation is part of model correctness. The architecture does not need a new attention equation, but an architecture-specific attention path must still honor the representation invariants established by the generic KV-cache backend.

P1: graph_max_nodes() Does Not Know Qwen4Exp Exists

A simpler but confirmed missing integration point is graph reservation.

llama_context::graph_max_nodes() assigns larger node budgets to architectures with unusually complex graphs, including Qwen3Next, Qwen3.5, DeepSeek4, MiniMax, and others.

The current head does not include LLM_ARCH_QWEN4EXP in that high-budget branch.

Qwen4Exp therefore falls back to the ordinary graph-node estimate even though Hyper-Connections and QSA add substantial graph construction.

Community reports have already shown GGML_ASSERT(obj_new) failures under --fit on, with stack traces reaching Qwen4Exp Hyper-Connection graph construction. Another reviewer independently pointed out the missing architecture switch.

The direct patch is trivial: classify Qwen4Exp with the other high-node-count architectures.

The deeper question is whether that is enough.

P1: init_full() Does Not Model the Real QSA Graph

The new indexed hybrid memory abstraction also treats ctx_idx as absent in full and update contexts.

That means the graph-reservation path can see:

mctx_hyb->get_idx() == nullptr

and therefore disable QSA while constructing a full-cache graph.

The runtime graph, however, can include:

indexer Q projection
indexer K projection
cache write
block gather
block pooling
normalization
RoPE
score computation
ReLU
head reduction
top-k
indexed KQ mask construction
sparse attention

If full-graph reservation does not instantiate the indexer path, the allocator is not reserving against the same graph topology that real long-context inference will execute.

The ordinary dense attention path may still dominate some tensor sizes, so this does not prove an allocation failure by itself. But graph-node count and intermediate-buffer topology are independent of the largest tensor.

For a new architecture with already-observed graph-reservation failures, the conservative requirement should be straightforward:

init_full() must exercise every graph branch that can appear during real execution, including QSA.

Otherwise --fit, scheduler reservation, and architecture tests can validate a graph that is structurally different from production inference.

Multimodal Support Still Needs Semantic Parity, Not Just Successful Generation

The PR reports vision support through the existing Qwen3-VL path, but multimodal correctness intersects with both PLE and QSA.

The PLE input path contains special handling for embedding batches where ubatch->token is null. The current comment says the multimodal layer has consumed the image placeholder token IDs and therefore substitutes a configured image token for PLE hashing.

Reviewer ngxson challenged that explanation and asked for confirmation because mtmd does not literally “consume” tokens in the way the comment suggests.

The important question is not the wording. It is whether the reconstructed token identity exactly matches the reference model’s input_ids semantics for every image-expanded position.

A visually plausible answer is not enough. PLE is a deterministic integer-addressing subsystem. It can be tested directly.

For multimodal input, the useful oracle is:

reference PLE row IDs
vs
llama.cpp PLE row IDs

for every expanded image position and neighboring text token.

QSA introduces a second geometric concern. The current block-position code treats compressed block positions as scalar multiples of the compression ratio and duplicates that value across MRoPE sections, with comments acknowledging that the approximation is exact for text and approximate for images.

That should trigger another direct comparison:

reference QSA selected block IDs
vs
llama.cpp selected block IDs

specifically inside and around image-token regions.

“Vision runs” establishes integration. It does not yet establish selector parity.

MTP Is Still Explicitly Disabled

The converter currently declares:

supports_mtp_export = False
no_mtp = True

So the base Qwen3.8-Flash-Next model path and the full checkpoint feature set should not be conflated.

MTP can reasonably land later. It does not have to block first-pass text and vision support.

But the limitation should be treated as explicit product surface:

base model inference: implemented
QSA: implemented, lifecycle fixes pending
PLE: implemented, state ownership refactor pending
vision: implemented, parity still deserves targeted tests
MTP: not exported

That is a more accurate statement than “Qwen4 support complete.”

Giant PLE Quantization Still Exposes a Tooling Assumption

PR #27742 includes several quantizer fixes motivated by the enormous PLE tensor, but community testing exposed another giant-tensor staging problem.

The PLE table is logically tens of billions of elements. A conversion buffer that materializes the entire tensor as FP32 can require roughly 204.8 GB for a 51.2-billion-element table.

A tester reported std::bad_alloc on a 96 GB machine before reaching the PR’s earlier work-buffer fix because f32_conv_buf attempted whole-tensor staging.

The scalable shape is not:

whole tensor
-> dequantize to FP32
-> requantize

It is:

bounded row band
-> dequantize
-> quantize
-> write
-> release
-> next row band

This is technically a quantizer/tooling issue rather than a forward-runtime bug, but it matters for upstream support. A model that can be downloaded but cannot be converted or requantized on realistic memory configurations is only partially integrated into llama.cpp’s workflow.

Qwen4Exp is useful here because it exposes an assumption that was already unsafe in the generic quantizer: tensor size can no longer be treated as a proxy for feasible staging size.

What the Pre-Merge Test Matrix Should Actually Look Like

Repeating WikiText perplexity is no longer the highest-value test. The next failures are much more likely to come from state transitions.

A serious pre-merge matrix should include at least the following execution patterns.

State Round-Trip

Run a prompt beyond the QSA dense threshold, save state, continue generation, restore into a fresh context, and generate the same continuation.

Compare logits, not just decoded text.

The test should prove that attention K/V, recurrent state, PLE predecessor identity, PLE convolution state, and QSA indexer keys all round-trip together.

Sequence Copy

Populate a sequence beyond 2051 tokens, copy it to another sequence ID, then feed an identical suffix to source and destination.

If seq_cp() is complete, both sequences should produce identical logits.

This specifically exercises deferred stream-copy behavior in the indexer cache.

Rewind and Replay

Generate a suffix, remove the suffix with the public sequence-memory API, and replay the same tokens.

The result should match a clean context that never generated the removed suffix.

This is the natural test for stale PLE predecessor history.

Multi-Slot Server Execution

Use multiple server slots and alternate requests whose contexts exceed the QSA sparse threshold.

Run the same logical workload with --parallel 1 and with the default multi-slot scheduler.

The sparse selector should be invariant to which physical slot LRU chooses.

Unified-KV Overlapping Positions

Place multiple sequences with overlapping logical position ranges into unified KV.

Verify that QSA block construction never mixes indexer representations across sequences.

This is the adversarial test for block identity.

Quantized KV

Test f16/f16, q8/f16, q8/q8, and at least one lower-bit supported cache configuration.

Prompts must exceed 2051 cached positions so the actual sparse path executes.

The target is not merely “no assert.” Compare logits against f16 KV within an expected quantization envelope.

Shared Model, Multiple Contexts

Create multiple contexts from one model, use the same sequence IDs in each, and run them independently or concurrently.

This should detect any remaining model-global mutable sequence state.

A ThreadSanitizer build would be valuable here.

Full Graph Reservation and --fit

Exercise short, medium, and very long context sizes with QSA active.

Ensure the reserved graph includes indexer operations and that node-count estimation is not relying on an architecture switch as a bandage for a structurally incomplete init_full() path.

Multimodal Address Parity

Compare PLE row IDs and QSA selected block IDs against the reference implementation, especially at image/text boundaries.

This isolates integer-addressing correctness from the much noisier signal of final decoded text.

Bounded-Memory Quantization

Convert or requantize a full model on 96 GB and 128 GB hosts while measuring peak RSS.

A giant tensor should not require a giant temporary FP32 mirror.

The Architectural Lesson Is Bigger Than Qwen4

The most interesting part of PR #27742 is no longer whether llama.cpp can express Qwen4Exp’s math. It clearly can.

The harder problem is that modern sparse and hybrid models create state that is not naturally represented by “weights plus KV cache.”

A sparse selector cache is state.

An n-gram predecessor stream is state.

A recurrent convolution tail is state.

A mapping from logical tokens to physical cache cells is state.

A backend-specific rotated K/V basis is part of the state representation contract.

Once these are acknowledged as first-class runtime objects, the implementation rules become much clearer.

Every persistent architecture component should answer the same questions:

Who owns it?

How is it allocated?

How is it copied?

How is it removed?

How is it shifted?

How is it serialized?

How is it restored?

How does it behave with multiple sequences?

How does it behave with multiple contexts?

What backend representation does it use?

What graph-reservation path accounts for it?

If any answer is “the model keeps an extra mutable map” or “this side cache happens to use the same slot descriptors,” the abstraction is probably not finished.

The best version of Qwen4 support will therefore not be the version with the most Qwen4-specific code. It will be the version that makes Qwen4’s unusual state look ordinary to llama.cpp’s memory system.

That is the real significance of the review-driven move toward llama_memory_hybrid_idx and the parallel work in PR #27762.

Final Assessment

PR #27742 is much closer to a correct architecture port than its size suggests. The forward graph has strong evidence behind it, the implementation avoids contaminating GGML with unnecessary new operations, and several of the original architecture risks have already been reduced to existing llama.cpp primitives.

But the implementation is not merge-complete simply because deterministic single-context inference works.

The remaining blockers are system-level:

indexer cache serialization
PLE state ownership
multi-slot cache lifecycle
indexer update/stream-copy lifecycle
quantized-KV representation parity
full-graph reservation
unified-KV sparse block identity
multimodal address parity

The most important engineering principle is therefore not “implement Qwen4 correctly.”

It is more specific:

A sparse model is only correct when every state transition preserves the same sparse address space.

That is where PR #27742 should be reviewed next. The tensor algebra is mostly there. The memory semantics still need to become impossible to desynchronize.