Qwen4Exp MTP in llama.cpp: The Prototype Works, but It Is Not Upstream

Qwen3.8-Flash-Next has carried an obvious unused asset in llama.cpp: its own NextN/MTP prediction head.

llama.cpp can run the qwen4exp architecture, but mainline does not currently use that extra head as a speculative drafter. On September 8, a new pull request, #28610, showed what wiring it up looks like. The implementation loaded the draft weights, built a dedicated MTP graph and context, and used the one-layer head to propose tokens for the full model to verify.

Then the PR was closed.

That makes the current status unusually easy to misunderstand: there is now a working, tested Qwen4Exp MTP implementation in a llama.cpp fork, but it is not part of upstream llama.cpp.

For people already finding RAMGPT through qwen4exp and Qwen3.8-Flash-Next searches, that distinction is the useful part.

What the model already contains

MTP, or multi-token prediction, is useful here because the model ships with an auxiliary prediction head that can act as a small drafter. Instead of running a separate small language model, the speculative path can ask the model’s own lightweight NextN head to propose the next token and then let the full target model verify it.

That is a good fit for speculative decoding because the draft is closely related to the target. It also avoids loading an unrelated draft model with a potentially different tokenizer or behavior.

The missing piece was runtime plumbing.

The existing RAMGPT Qwen4Exp implementation analysis focused on getting the unusual architecture itself into llama.cpp. The new work targets a different search intent: can llama.cpp actually use Flash-Next’s MTP head for faster decoding?

As of September 9, 2026, upstream’s answer is still no.

PR #28610 added two things

The prototype was deliberately small in scope. Its PR description identifies two main changes.

First, it teaches the Qwen4Exp loader how to find MTP draft tensors in three layouts:

• a separate sidecar draft GGUF;
• draft tensors stored in the main GGUF;
• a draft-head-only GGUF using the Unsloth-style layout.

The implementation makes nextn_predict_layers optional. If the metadata key or draft tensors are absent, the model falls back to ordinary loading. That matters because existing mainline GGUF conversions drop the MTP head; old files should not suddenly stop loading just because the runtime learns about NextN.

Second, it adds the actual draft execution path: a Qwen4Exp MTP graph, a separate MTP context, the h_nextn output, and integration with speculative verification.

The code also has to avoid a subtle architectural trap. Qwen4Exp is hybrid. The main model mixes recurrent and full-attention behavior, but the NextN block is supposed to run through the full-attention QSA path. The prototype explicitly marks the MTP layer so it is not accidentally classified as recurrent by the normal layer rule.

That is the kind of implementation detail that makes “support MTP” much more than exposing a command-line flag.

Why a sidecar GGUF is interesting

The prototype supports an MTP-only sidecar rather than requiring every existing Qwen3.8-Flash-Next GGUF to be regenerated.

That is operationally useful. A user can keep a large quantized target model and pair it with a much smaller file containing the draft head and shared pieces required by the drafter.

The loader detects that the sidecar lacks trunk tensors and treats those tensors as optional. The NextN tensors, by contrast, are loaded for the MTP context.

This resembles the practical pattern already used for some other speculative heads: keep the target artifact stable and attach the specialized drafter separately.

It also creates a clean failure mode. A normal GGUF without MTP data remains a normal target model instead of becoming an invalid half-speculative model.

The reported performance result

The PR author tested an IQ4_NL quant of Qwen3.8-Flash-Next with the MTP head split into a sidecar draft GGUF.

The headline reported result is 54.4 tokens/s with the speculative path versus the author’s non-speculative baseline on a mixed consumer-GPU machine. The PR also reports a draft-depth sweep in which depth 1 was best: roughly 54–58 tok/s at depth 1, 48.7–53.8 tok/s at depth 2, and 43.5–46.2 tok/s at depth 3.

Those are the contributor’s measurements, not RAMGPT benchmarks.

The hardware caveat is important: the test rig combines an RTX 5090, RTX 3090s and RTX 5060 Tis over PCIe Gen3 x4 links. The author explicitly describes it as interconnect-bound and expects Gen4/Gen5 systems to behave better. That expectation is plausible, but it is not a measurement, so it should not be turned into a claimed Gen4/Gen5 speedup.

The more interesting result is actually the depth sweep. The model has a one-layer prediction head, and asking it to build deeper speculative chains reduced effective throughput as acceptance fell. In this setup, “more draft tokens” was not “more speed.”

It was tested for correctness too

The author did more than report tokens per second.

According to the PR, a deterministic 1.5K-token reproducer at temperature 0 produced an exact hit under the speculative path against the non-speculative layer-split reference. A fresh 25K-token needle-recall test also produced an exact hit.

The author additionally reports a 36-run quality battery using a replicated-loss criterion with no replicated quality losses against the base model and source Q8_0 reference.

Again, these are contributor-reported validation results. They are useful because speculative decoding must preserve the target distribution when implemented correctly, but RAMGPT has not independently reproduced this branch.

Why the PR is closed

This is where a search result can become stale within hours.

PR #28610 opened on September 8. The llama.cpp automated checker flagged the PR template and detected AI-generated text in the PR description or commit-message material. llama.cpp’s contribution guidance allows AI-generated code but asks contributors to write PR descriptions and commit messages themselves.

Early on September 9, the author closed the PR and wrote:

“Closing for now — not pursuing upstream at this time.”

The author also said the flashnext-mtp fork branch would remain available for people who want the implementation.

So this is not a merged llama.cpp feature, not an upcoming release feature, and not something users should expect in current mainline binaries. The code is a fork implementation with public test evidence.

That status is more important than the benchmark number.

What mainline users should expect today

If you download a current upstream llama.cpp build and run a Qwen3.8-Flash-Next GGUF, the qwen4exp architecture support is real. The built-in MTP head is not currently wired into upstream speculative decoding by this PR.

If you see references to Qwen4Exp MTP support, check whether they refer to:

1. upstream llama.cpp architecture support;
2. the closed PR #28610;
3. the noonr48/llama.cpp flashnext-mtp fork;
4. a later, separate upstream implementation.

Those are not interchangeable states.

This also means there is no reason to regenerate a working mainline GGUF merely because this prototype exists. The PR was designed specifically so ordinary GGUFs without the MTP tensors would continue to load normally.

Why this matters for Qwen4Exp beyond one PR

Qwen4Exp has been a useful case study in how modern model support arrives in layers.

First comes architecture recognition: fixing failures such as unknown model architecture: 'qwen4exp' and teaching the runtime the graph.

Then come model-specific performance and deployment issues. RAMGPT has already looked at the Qwen4Exp n-gram embedding I/O pathology and Qwen3.8-Flash-Next memory behavior.

MTP is another layer. A runtime can correctly load and generate with a model while still leaving architecture-specific acceleration on the table.

That is why “llama.cpp supports model X” is increasingly an incomplete statement. For a model with hybrid recurrence, sparse attention, PLE, MoE, NextN heads and unusual cache behavior, support is a stack of capabilities rather than a Boolean.

The useful takeaway

The new prototype answers an important technical question: yes, Qwen3.8-Flash-Next’s own NextN/MTP head can be integrated into llama.cpp’s speculative-decoding machinery, and the contributor reports a measurable decode improvement on a difficult consumer multi-GPU setup.

But the implementation did not land upstream.

Today the clean status is:

• Qwen4Exp itself: supported upstream;
• Qwen4Exp NextN/MTP speculative decoding: demonstrated in a fork;
• PR #28610: closed, unmerged;
• reported best draft depth in the contributor’s test: 1;
• upstream availability: not yet.

If a new upstream MTP PR appears, that will be a materially different event worth retesting and documenting. Until then, users searching for “Qwen4Exp MTP llama.cpp” should not mistake a working prototype for a shipped feature.