When Lossless Speculation Stops Being Lossless on Blackwell

Speculative decoding is normally sold with a reassuring property: if the target model verifies every drafted token, the optimization can make generation faster without changing the result.

That statement hides an implementation assumption.

The target model must compute the same answer when several verification tokens are evaluated together as it would have computed by decoding those tokens one at a time.

Two new llama.cpp reports on NVIDIA Blackwell expose what happens when that assumption fails. One report isolates flash attention. The other isolates quantized matrix multiplication. In both cases, changing the batch width changes the numerical path through CUDA kernels. That is enough to make greedy speculative decoding produce different text from ordinary one-token decoding.

The important lesson is broader than either bug:

Lossless speculative decoding requires batch-invariant target evaluation, not merely mathematically correct verification logic.

That requirement deserves to become a first-class backend test.

The Hidden Contract Behind Lossless Speculation

Consider normal greedy decoding. The target model evaluates one new token position, obtains logits, selects the maximum-probability token, and repeats.

A speculative verifier instead receives a short block of drafted tokens and evaluates several positions in one target-model pass.

Conceptually:

ordinary decode

token 1 -> target(width=1)
token 2 -> target(width=1)
token 3 -> target(width=1)

speculative verification

tokens 1..3 -> target(width=3)

The speculative algorithm assumes those two routes are equivalent for the accepted prefix.

They do not need every floating-point intermediate to be mathematically identical in an abstract sense. But if different kernel geometry changes logits enough to alter the winning token, the user-visible result changes. Under greedy sampling, that violates the practical meaning of lossless speculation.

This is especially dangerous because the verifier itself is still doing exactly what the algorithm asks. The failure sits underneath it, in backend execution.

Flash Attention Changes Geometry With Batch Width

llama.cpp PR #28108 reports a concrete Blackwell case in flash attention.

The CUDA dispatcher chooses a template based partly on the query width. Ordinary single-token decoding and speculative verification therefore can select different kernel instances.

The reporter tested FLASH_ATTN_EXT on an RTX 5080 Laptop across three KV lengths and token widths from one through four. Widths one and two matched the serial reference. Widths three and four did not.

The reported matrix was unusually clean:

n_tokens = 1  -> pass
n_tokens = 2  -> pass
n_tokens = 3  -> fail
n_tokens = 4  -> fail

Those failing widths are not arbitrary. A short speculative draft naturally creates exactly this kind of verification batch.

The report attributes the divergence to different tile geometry, which changes KV traversal and reduction/combine order. The proposed fix routes widths three and four through the same four-wide vector tile family used compatibly across the narrow range.

After that change, the reporter’s batch-invariance test moved from 6/12 passing cases to 12/12.

The end-to-end result is more important than the kernel test. With Qwen3.8-27B-UD-Q3_K_XL at temperature zero and a fixed seed, the reporter observed that current master with n-gram speculative decoding diverged from the non-speculative reference. With the proposed flash-attention change, the output was byte-identical in that test.

These are contributor measurements on one RTX 5080 Laptop, not RAMGPT measurements, and the pull request is still open. The report explicitly does not establish that every architecture or every Blackwell GPU behaves identically.

Correctness Has a Performance Price

The proposed flash-attention fix is also interesting because it quantifies the cost of preserving invariance.

For the affected widths, the reporter measured roughly a 15–17% kernel-level slowdown. Width one remained essentially unchanged, which matters because width one is the normal token-generation path.

That creates a useful systems tradeoff:

optimize each batch width independently
        -> best local kernel speed
        -> potentially different numerical path

force compatible geometry across speculative widths
        -> some verification overhead
        -> stable decode equivalence

An alternative implementation reportedly preserved performance at widths three and four but made width-one flash attention about 84.5% slower. That was rejected because it would tax the ordinary decode path to repair a speculative-only problem.

This is exactly the kind of optimization boundary that local inference engines increasingly face. Kernel autotuning cannot be judged only by microseconds. Some choices participate in higher-level semantic guarantees.

Quantized Matrix Multiplication Shows the Same Class of Failure

A separate issue, #28111, reports batch-dependent output from ggml_mul_mat with IQ quantization on the same SM120-class environment.

The test compares a batched matrix multiplication against a sequence of single-column matrix multiplications using the same weights and shapes representative of Qwen3.8 dense projections.

The reporter found that widths one through four matched, while widths five and eight failed for IQ3_S and IQ4_XS geometries. Overall, 12 of 18 test cases passed.

This matters because it shows the problem is not confined to flash attention.

A speculative verifier changes tensor shapes throughout the graph. If any numerically sensitive operation changes implementation strategy based on that width, serial and batched target evaluation can cease to be equivalent.

The reporter also tested the obvious suspect: the MMVQ-to-MMQ crossover. Adjusting the runtime threshold did not remove the width-five and width-eight divergence. Lowering the threshold far enough actually caused more disagreements because the batched and serial cases were then routed to different kernel families.

The issue therefore points toward width-specific behavior inside the quantized matrix-vector path itself rather than merely one high-level kernel switch.

Again, this remains an open report on one SM120 environment. It should not be generalized into “IQ quants are broken on Blackwell.” The stronger and safer conclusion is that llama.cpp currently lacks a systematic guarantee that relevant target-model operations are batch invariant across speculative widths.

Throughput Tuning Can Quietly Become Correctness Tuning

The matrix-multiplication report references earlier work that tunes the MMVQ/MMQ crossover by hardware for performance.

That is a subtle but important connection.

A threshold that once looked like a pure throughput knob can become part of a correctness boundary when the same logical token computation is reached through different batch shapes.

The old mental model is:

kernel dispatch policy
    = performance engineering

The emerging model is:

kernel dispatch policy
    = performance engineering
    + numerical behavior
    + speculative-decoding equivalence

This does not mean every backend must be bitwise identical for every possible tensor shape. Floating-point kernels have always differed by reduction order and implementation.

But speculative decoding creates a special requirement: two execution paths that represent the same greedy decode must not cross a decision boundary often enough to change the accepted token stream.

The safest engineering test is therefore stronger than checking average numerical error.

Why Tiny Numerical Differences Can Change Entire Answers

A backend may differ only slightly in logits while producing the same token for most prompts.

That can make the bug look harmless during casual testing.

The PR author explicitly notes that a low-entropy code-editing prompt produced identical text even before the flash-attention fix. The divergence appeared when the top candidates were close enough that the numerical difference changed the winner.

Once one token changes, autoregressive generation amplifies the difference:

small kernel-level numerical difference
        |
        v
one greedy argmax flips
        |
        v
next token sees a different prefix
        |
        v
all later logits are conditioned differently
        |
        v
entire response can diverge

That is why measuring maximum tensor error alone is not enough to characterize user impact.

For a lossless speculative decoder, the meaningful integration test is simple:

With deterministic greedy settings, does speculative generation produce the same bytes as non-speculative generation?

But even that should be paired with lower-level tests so a failure can be localized to attention, matrix multiplication, or another operation.

A Better Backend Test Matrix

These reports suggest a missing class of tests for inference runtimes.

For operations used by target verification, compare a batched evaluation at realistic speculative widths against repeated width-one evaluations over the same inputs.

The matrix should cover at least:

• attention implementations;
• quantized and dense matrix multiplication;
• common speculative widths such as 2, 3, 4, 5, and 8;
• relevant GPU architectures;
• quantization formats used by real local models;
• representative long-context KV lengths;
• end-to-end greedy generation equivalence.

There are two useful levels of acceptance.

The strictest kernel test is bitwise equality where the implementation can reasonably provide it. The user-facing test is token-stream equality under deterministic greedy decoding.

The first finds backend drift early. The second proves that the optimization still satisfies its advertised semantic contract.

This Changes How Speculative Decoding Should Be Benchmarked

Local-AI benchmarks usually focus on acceptance rate and tokens per second.

Those remain necessary, but they are insufficient.

A speculative benchmark should now report at least three independent properties:

speed
  -> how much faster is generation?

acceptance
  -> how much draft work survives?

equivalence
  -> does deterministic output match non-speculative decode?

The third metric is not quality evaluation. It is implementation validation.

If an optimization advertised as lossless changes deterministic output because the backend evaluates a verification batch differently, a high tokens-per-second result is beside the point.

This is particularly relevant as local inference moves toward increasingly specialized kernels, architecture-specific autotuning, quantized operators, and hybrid speculative methods. Each optimization expands the number of execution paths that must preserve higher-level invariants.

Bottom Line

PR #28108 and issue #28111 are early Blackwell-specific reports, not proof of a universal CUDA problem. Their measurements come from a contributor’s RTX 5080 Laptop environment, and one of the two problems already has a proposed narrow fix while the other remains unresolved.

But together they expose an important architectural requirement that deserves more attention.

Speculative decoding is only lossless if the target model used for verification behaves consistently across the batch shapes introduced by speculation.

The algorithm can be correct while the implementation violates that assumption underneath it.

For local inference runtimes, batch invariance is therefore not merely a numerical curiosity. It is part of the correctness contract of speculative decoding.

The next generation of speculative-decoding benchmarks should test that contract directly.