DFlash2 Lands in llama.cpp: What PR #27342 Actually Changes

DFlash2 has now crossed an important line for local inference: it is no longer just a research-side drafter with engine patches scattered across experimental branches. The llama.cpp implementation developed in PR #27342 has been integrated upstream.

The merge path is slightly more interesting than the PR badge suggests. PR #27342 was merged on August 27, 2026 into the xsn/dflash2 integration branch. Minutes later, PR #27816 carried the finalized implementation into master, explicitly dropping an experimental CUDA top-k.cu change after end-to-end review showed inconsistent gains. Current llama.cpp master contains the DFlash2 metadata, model tensors, dynamic-convolution graph, candidate selector, speculative runtime integration, M-RoPE handling, and the CPU-side selector walk.

That matters because DFlash2 is not simply “DFlash with a better checkpoint.” It changes what the drafter computes.

Plain DFlash makes block drafting parallel by predicting the positions of a speculative block together. That removes the conventional autoregressive draft loop, but it also creates an obvious weakness: each position can be individually plausible while the block as a sequence is locally incoherent. DFlash2 attacks that failure mode without giving up parallel drafting.

Its two new mechanisms are:

• grouped dynamic depthwise convolution, which gives neighboring draft positions an explicit low-cost local communication path; and
• a candidate selector, which keeps several plausible tokens per position and chooses a coherent path through those candidates instead of committing immediately to each position’s top-1 token.

The useful mental model is not “a more accurate draft model.” It is this:

DFlash 1
parallel block prediction
        |
        v
one independent top-1 token per position
        |
        v
target verification

DFlash 2
parallel block prediction
        |
        +--> local dynamic convolution
        |
        v
top-k candidates per position
        |
        v
pairwise transition lattice
        |
        v
cheap path selection
        |
        v
target verification

The target model still verifies the speculative block, so the speedup mechanism remains compatible with lossless speculative decoding. The selector changes which draft path is proposed; it does not replace target verification.

The Central Insight: Selection Is Cheaper Than Re-Prediction

The strongest idea in DFlash2 is almost embarrassingly simple once stated correctly.

DFlash’s top-1 prediction at a draft position can be wrong even when the correct token is already present in that position’s short candidate list. The official DFlash2 analysis reports a large gap between top-1 accuracy and top-16 recall across the draft block. In other words, much of the missing acceptance length is not a representation problem. The drafter often already generated the right token. It selected the wrong member of its own candidate set.

A conventional response would be to add another sequential correction stage. DFlash2 instead asks whether the candidate list itself contains enough information to repair the path cheaply.

For each draft position, llama.cpp first keeps the selector’s top-k candidates. Then it scores transitions from candidates at the previous position into candidates at the current position. Conceptually, the transition score is:

edge(previous -> candidate)
    = unary(candidate)
    + pairwise(previous, candidate, current_hidden)

The unary term is the drafter’s own score for the candidate. The pairwise term uses a predecessor codebook, a successor codebook, and a context-dependent hidden projection. The implementation therefore does not ask only “is this token locally likely?” It asks “is this token likely after that predecessor under this block context?”

That turns a set of independent token guesses into a small layered graph.

The important implementation property is that the expensive part remains parallel. llama.cpp computes candidate sets and transition scores for all relevant positions in the graph. Only the final walk through the precomputed lattice is sequential, and that walk operates over top_k scores rather than rerunning the draft transformer or its full vocabulary projection.

This is the distinction that makes DFlash2 architecturally interesting:

DFlash2 does not restore autoregression to the draft model. It restores local sequence coherence after parallel prediction.

What llama.cpp Actually Builds for the Selector

The current src/models/dflash.cpp implementation makes the dataflow explicit.

For each token position, it first computes:

candidates = ggml_top_k(logits, top_k)

It then gathers the unary scores corresponding to those candidates and projects the current hidden state through dflash_selector_hidden.

The tensors are reshaped by block so that candidate index, block position, and sequence remain explicit. For each run of positions, llama.cpp gathers:

• successor codebook rows for the current candidates;
• predecessor codebook rows for the previous candidates; and
• the projected hidden gate for the current position.

The pairwise score is produced with ordinary GGML tensor algebra: gather, reshape, elementwise multiply, matrix multiply, broadcast, and add. There is no DFlash2-specific GGML primitive.

That is a meaningful engineering choice. The feature is new at the model level, but it does not enlarge GGML’s mathematical vocabulary.

The result is packed into t_h_nextn, using the existing next-token embedding/result channel as a transport for the selector lattice. Each row contains the candidate IDs followed by the transition scores required by the speculative runtime.

The runtime side in common/speculative.cpp then performs the cheap walk:

predecessor = anchor

for each draft position:
    read scores conditioned on predecessor
    choose best successor
    predecessor = chosen candidate index
    emit chosen token

This is why DFlash2 disables the ordinary backend draft sampler path in llama.cpp. The speculative driver does not consume raw draft logits position by position. It consumes the selector lattice already constructed by the model graph.

That difference is easy to miss if one only looks at the CLI. DFlash2 still uses the existing draft-dflash speculative type, but the checkpoint metadata changes the runtime behavior automatically.

No New Flag: The Checkpoint Selects the Execution Mode

The PR deliberately avoids introducing a separate --spec-type dflash2 mode.

At load time, llama.cpp reads DFlash2-specific metadata including:

• convolution kernel size;
• convolution group size;
• selector rank; and
• selector top-k.

The speculative implementation queries llama_model_dflash_selector_top_k(). If the value is positive, the draft is treated as DFlash2.

So the external interface stays:

--spec-type draft-dflash

while the checkpoint determines whether the engine follows DFlash1 or DFlash2 semantics.

That is the right abstraction. DFlash2 is a property of the drafter architecture, not a sampling preference that the user should have to toggle manually.

It also makes old DFlash checkpoints continue through the original path. The new code is largely gated on the presence of DFlash2 tensors such as selector_hidden.weight and the convolution projections.

Dynamic Local Convolution: Giving Parallel Positions a Side Channel

The second major addition is the grouped dynamic depthwise convolution.

This exists because parallel drafting has a structural problem beyond candidate choice. A token late in the block needs information from earlier draft positions, yet forcing that communication through full attention can be inefficient, and making the draft autoregressive would defeat the purpose.

DFlash2 inserts a local convolution around both the attention and FFN sublayers.

The PR describes the effective operation as a static per-channel kernel plus an input-dependent correction shared within channel groups. A simplified view is:

output(position, channel)
    = sum over local taps [
        static_weight(tap, channel)
        + dynamic_weight(position, tap, group(channel))
      ] * input(position - tap, channel)

The llama.cpp implementation reflects that structure directly.

build_dflash2_conv() receives:

• the hidden states;
• dynamically projected coefficients;
• the learned base convolution weights; and
• a side selecting the pre- or post-sublayer kernel.

The hidden dimension is divided into groups. Dynamic coefficients are predicted per group rather than independently for every channel, then broadcast across channels inside each group. The static base remains channel-specific.

The graph walks the small convolution kernel tap by tap, shifts each block locally with explicit zero padding, multiplies the shifted activations by the combined static-plus-dynamic weights, and sums the terms.

There is an important boundary condition here: the operation is block-local. It must not accidentally mix the end of one speculative sequence with the beginning of another sequence sharing the same batch. The implementation reshapes by n_blocks and applies the shift inside each block, which preserves that boundary.

In the DFlash decoder, the same projected dynamic coefficients are used on both sides of attention:

RMSNorm
  -> dynamic conv side 0
  -> Q/K/V + attention
  -> dynamic conv side 1

and similarly around the FFN:

RMSNorm
  -> dynamic conv side 0
  -> FFN
  -> dynamic conv side 1

This gives local neighboring positions a dedicated communication channel while allowing attention to spend more of its capacity reading the already-verified context.

The design is especially appealing for a parallel drafter because the convolution does not introduce token-by-token execution. Every block position is still evaluated in the same graph.

The Selector Is Not Really a “+2M Parameter” Object in GGUF

One review discussion exposed a useful distinction between model-design parameter accounting and runtime representation.

The DFlash2 blog describes the selector as roughly a +2.0M parameter addition in one reported Qwen3-4B setup. A reviewer noticed that the llama.cpp checkpoint representation appears much larger. With selector rank 256 and a Qwen3.8 vocabulary around 248k tokens, two materialized token codebooks alone are on the order of:

2 * 256 * 248,320 ~= 127 million values

The PR author clarified the discrepancy: the predecessor and successor codebooks can be understood as projections derived from the target embeddings, while the small quoted parameter count refers to the learned projection machinery in the research accounting. For implementation convenience, the llama.cpp-compatible checkpoint materializes the projected codebooks.

That distinction matters for local inference.

“Additional learned parameters” and “additional serialized/runtime values” are not the same quantity.

From a training-design perspective, the selector can remain lightweight. From a GGUF and memory-bandwidth perspective, materializing two vocabulary-sized rank-256 tables is real storage that has to be loaded and accessed.

This does not invalidate the selector’s low-compute argument. The codebooks avoid repeatedly projecting the entire vocabulary at runtime. But it is exactly the kind of detail that disappears if DFlash2 is summarized only as “+2M parameters.”

p_min Means Something Different Under DFlash2

DFlash2 also subtly changes the semantics of the speculative confidence threshold.

For ordinary DFlash, llama.cpp samples the candidate distribution at each noise position and can stop drafting when the top draft probability falls below p_min.

For DFlash2, the engine is not selecting directly from each position’s original draft distribution. It has transition scores conditioned on the previously chosen candidate.

The implementation therefore computes a softmax confidence over the current row of transition scores at the selected argmax. If that selector confidence is below p_min, the lattice walk stops early.

Conceptually:

DFlash1 p_min
    -> confidence of this position's standalone draft token

DFlash2 p_min
    -> confidence of this transition given the chosen predecessor

That is a better fit for what the model is actually doing. The confidence threshold now measures whether the selected path remains locally decisive, not merely whether the underlying unary draft logit is large.

M-RoPE Was a Real Integration Issue, Not Cleanup Noise

One of the more instructive parts of the PR history was the review-driven reduction of unrelated changes.

An early review correctly complained that the branch modified too much generic speculative and server logic. The author subsequently reverted and refactored a number of changes so DFlash2 could fit the existing speculative abstraction instead of forcing the server to understand selector-specific state.

But not every cross-cutting change was unnecessary.

DFlash2 needed correct position handling for M-RoPE drafts. The final speculative implementation detects whether the draft model uses M-RoPE, allocates four position rows for embedding injection, and explicitly supplies the degenerate draft positions when target features are encoded and injected into the draft cache.

This is an important distinction in mature inference engines: preserving an abstraction does not mean pretending new architectures have no integration requirements. The right outcome is to isolate those requirements in the narrowest layer that owns them.

The final PR is significantly better in that respect than its early revisions.

Why the Experimental CUDA Top-K Kernel Did Not Ship

PR #27342 originally included CUDA top-k work because DFlash2 calls top-k as a first-class part of candidate construction. A reviewer initially argued that keeping the kernel change in the same PR could make sense because selector performance is central to DFlash2.

The final review produced a more useful answer: benchmark it end to end.

A reviewer tested the proposed top-k changes on DGX Spark and B4500 systems across concurrency 1 and 4. The measured differences ranged from small positive changes to regressions, including a reported negative result at higher concurrency on B4500. The conclusion was that the performance benefit was not consistent enough to justify expanding the scope of the DFlash2 merge.

That led to a CHANGES_REQUESTED review asking for ggml/src/ggml-cuda/top-k.cu to be removed.

Because the original PR came from an organization branch that the maintainer could not directly update, the integration took a two-step path:

1. #27342 merged the complete feature work into xsn/dflash2.
2. #27816 removed the experimental top-k backend change and merged the finalized DFlash2 implementation into master.

The changed-file list for #27816 contains the model, speculative runtime, converter, GGUF metadata, architecture, context, and tests, but no ggml/src/ggml-cuda/top-k.cu.

This is a healthy upstream outcome. A new decoding algorithm should not smuggle an uncertain backend optimization into core CUDA merely because the algorithm happens to exercise that primitive heavily.

The model feature and the backend optimization have different evidence requirements.

What the PR Benchmark Actually Shows

The PR reports a deliberately small benchmark on an Apple M5 Pro with 64 GB memory, using Qwen3.8-27B Q4_K_M as the target and the first eight GSM8K problems.

The reported decode results were:

All eight runs reportedly reached normal stops.

Two things are more interesting than the headline 1.85x.

First, acceptance stays around five tokens even when the draft is quantized to Q8_0 or Q4_K_M. On this narrow workload, aggressive draft quantization did not destroy the selector’s usefulness.

Second, BF16, Q8_0, and Q4_K_M decode throughput are relatively close. That suggests the end-to-end cycle is not dominated solely by raw draft weight bandwidth. Target verification, synchronization, graph overhead, and the rest of the speculative cycle still matter.

But the sample is only eight GSM8K problems on one Apple system. It should not be promoted into a general hardware claim.

The official DFlash2 material reports broader acceptance-length and serving-throughput results across GSM8K, MATH-500, HumanEval, MBPP, and MT-Bench, including stronger average acceptance than Qwen3.8-27B’s native MTP in those reported tests. Those are useful context, but they remain vendor-reported results rather than RAMGPT benchmarks.

For llama.cpp specifically, the upstream PR establishes something narrower and more important: the implementation is functional, quantized DFlash2 drafts work, and the runtime architecture can express the selector without reintroducing sequential draft-model forward passes.

The Most Important Performance Question Is Now Different

With ordinary speculative decoding, the obvious optimization question is often:

How fast is the draft model?

With DFlash2, that is no longer sufficient.

The useful quantity is closer to:

accepted target tokens per complete draft-verify cycle
------------------------------------------------------
             cycle latency

DFlash2 intentionally spends a small amount of extra work inside the draft cycle to increase the number of useful tokens that survive verification.

That creates several competing costs:

• dynamic convolution overhead;
• selector hidden projection;
• top-k candidate extraction;
• codebook gathers;
• pairwise candidate scoring;
• CPU-side lattice walk;
• target verification; and
• rejected-token cleanup.

An optimization that makes ggml_top_k faster in isolation can still fail to move end-to-end throughput if top-k occupies only a small share of the cycle. The review’s decision to remove the CUDA kernel experiment is a concrete demonstration of that principle.

For local inference benchmarking, this means microbenchmarks are not enough. DFlash2 should be measured at the full speculative-cycle level with acceptance, cycle latency, target verification cost, context length, concurrency, and draft quantization all reported together.

Why This Is More Interesting Than Another Speculative Decoder

The deeper reason DFlash2 matters is that it changes the usual tradeoff between parallelism and coherence.

Autoregressive drafting has coherence because every new token conditions on the previously drafted tokens, but it pays for that dependency with sequential forward passes.

Parallel block drafting removes those forward-pass dependencies, but independent positions lose local consistency.

DFlash2 splits the problem into two cheaper forms of communication:

representation communication
    -> dynamic local convolution

candidate-level communication
    -> pairwise path selector

Neither requires running the whole draft transformer autoregressively across the block.

This suggests a broader design direction for speculative inference: once a parallel drafter already has high recall in a small candidate set, spending more compute on full-vocabulary re-prediction may be the wrong optimization target. A compact selector can exploit information the drafter already computed.

The slogan is simple:

If the right token is already in the beam, stop paying transformer FLOPs to rediscover it.

DFlash2 operationalizes that idea in a form that fits llama.cpp’s existing model graph and speculative runtime.

What Entered master

After the two-stage merge, current llama.cpp master includes the parts that matter for actual DFlash2 execution:

• GGUF metadata for convolution and selector dimensions;
• tensor mappings for attention/FFN convolution bases and projections;
• predecessor, successor, and hidden selector tensors;
• converter support;
• DFlash2 checkpoint validation;
• grouped dynamic convolution in the DFlash graph;
• candidate-lattice construction;
• automatic DFlash2 detection from model metadata;
• selector-aware p_min handling;
• M-RoPE position handling for injected features;
• DFlash2-specific graph-node reservation; and
• speculative runtime traversal of the selector lattice.

What did not enter with the feature is equally important: the experimental CUDA top-k kernel was deliberately excluded from the final master integration.

So the upstream state is cleaner than the original #27342 diff history alone would suggest.

What We Would Benchmark Next

The PR is merged, but there is still a large difference between “supported” and “characterized.”

A serious local-inference evaluation should compare DFlash2 against autoregressive decoding, native Qwen3.8 MTP, and plain DFlash where matching drafters exist. At minimum, it should vary:

• target quantization;
• DFlash2 draft quantization;
• --spec-draft-n-max;
• p_min;
• context length;
• concurrency;
• CUDA versus Metal versus Vulkan where practical;
• math, code, prose, structured output, and multilingual workloads; and
• acceptance length versus complete cycle latency.

The most valuable experiment is not simply finding the highest tokens-per-second number. It is identifying where the selector’s extra acceptance stops paying for its extra graph work.

On fast GPUs, the crossover may be very different from Apple unified memory. Under concurrency, it may be different again. Long context changes target verification cost. Quantized target KV changes another term. Structured decoding may interact with the candidate path in ways that ordinary prose does not.

Those are now implementation questions, not reasons to doubt the architecture.

The Bottom Line

PR #27342 is significant because it makes DFlash2 a first-class llama.cpp model/runtime behavior rather than a custom speculative-decoding fork.

The engineering lesson is larger than the feature itself.

DFlash made drafting parallel. DFlash2 accepts that parallel prediction will produce locally imperfect blocks and repairs them without putting the transformer back into an autoregressive loop. Dynamic convolution improves communication between neighboring draft positions; the selector exploits high candidate recall to choose a coherent path; target verification preserves speculative-decoding correctness.

And the final upstream review improved the implementation in exactly the right places: generic server changes were pushed back out, selector state stayed inside the speculative abstraction, M-RoPE requirements were handled explicitly, and a questionable CUDA optimization was separated from the model feature after end-to-end measurements failed to justify it.

That is the version now worth benchmarking.

The useful abstraction is no longer “draft several tokens at once.”

It is:

Draft the candidate space in parallel, then spend the minimum possible sequential work choosing a path through it.