Adaptive Speculative Decoding on RTX 4090: Fixed n=7 Was 13.5% Faster

Agention’s August 24 adaptive-speculation report introduced an interesting idea: do not use a fixed speculative draft length.

Instead, measure how many draft tokens the target model is actually accepting and dynamically adjust the number proposed on the next round.

On AMD Strix Halo, the result looked dramatic.

For structured generation with Qwen3.8-27B, the project reported:

The explanation was equally interesting.

With a fixed draft length of seven, acceptance reportedly collapsed to around 18%. Adaptive drafting kept acceptance near 96%, allowing the system to exploit longer drafts only when they were likely to survive target verification.

The project’s author explicitly asked for validation on other hardware.

So I compiled the fork and tested the same basic idea on an NVIDIA GeForce RTX 4090 using CUDA.

I did reproduce a major speculative-decoding speedup.

But I did not reproduce the most important part of the original result.

On my RTX 4090, fixed n=7 was faster than adaptive drafting.

And not by a small amount.

Across ten measured runs in an interleaved benchmark, fixed n=7 averaged 80.11 tok/s, while adaptive n=3–7 averaged 70.61 tok/s.

That makes fixed drafting approximately 13.45% faster for this workload.

The result is a useful reminder that the best speculative-decoding policy may depend not only on the model and workload, but also on the economics of the hardware running the draft and verification passes.

The experiment

I used Laurent Zuijdwijk’s experimental llama.cpp fork containing the new adaptive speculation implementation.

The build reported:

build: b10642-54ca36d4f

The runtime was compiled with CUDA support and executed on a single RTX 4090.

The model configuration was:

This is not an apples-to-apples hardware reproduction of the Strix Halo benchmark.

The original headline result used an FP4 target and FP4 DFlash2 sidecar on an AMD Strix Halo system with a heavily optimized Vulkan path.

My test instead used standard Q4_K_M GGUFs on CUDA.

That distinction turns out to be important.

First result: DFlash2 itself works extremely well on the RTX 4090

I first ran Qwen3.8-27B without speculative decoding.

The structured prompt asked the model to generate fictional server records as JSON.

Bare decoding produced:

Prompt: 388.2 tok/s
Generation: 28.7 tok/s

I then enabled DFlash2 with a fixed draft length of three.

Generation jumped to:

65.3 tok/s

That is approximately:

28.7 → 65.3 tok/s

or about a 2.28× increase over bare decoding.

So before testing adaptive drafting at all, DFlash2 had already delivered a substantial practical win.

Then fixed n=7 got even faster

The original Strix Halo benchmark showed the opposite behavior.

Increasing fixed draft length from three to seven caused throughput there to collapse.

On the RTX 4090, I measured:

Instead of collapsing, fixed n=7 became the fastest configuration tested.

It was around 23% faster than fixed n=3.

More surprisingly, it was also substantially faster than the new adaptive controller.

That immediately raised an obvious question:

Was 80.1 versus 70.8 just GPU clock, temperature, or run-order noise?

Ten-run interleaved test

To answer that, I repeated fixed n=7 and adaptive n=3–7 in five palindrome-ordered rounds:

fixed7
adaptive
adaptive
fixed7

This ordering helps cancel gradual thermal or clock drift because each configuration appears on both sides of the other.

The ten fixed-n=7 measurements were tightly clustered around 80 tok/s.

The ten adaptive measurements were tightly clustered around 70.6 tok/s.

Aggregate result

The fixed policy therefore won by:

13.45%

The difference is dramatically larger than the run-to-run variation.

This does not look like a boost-clock artifact.

It is a repeatable property of this configuration and workload.

Why?

Throughput alone tells us which configuration won.

The fork’s speculative-decoding statistics help explain how.

I reran both policies with trace logging enabled.

For fixed n=7:

#calls(b,g,a) = 1 44 44
#gen drafts = 44
#acc drafts = 42
#gen tokens = 308
#acc tokens = 254
#mean acc len = 6.77
#acc rate/pos = (0.955, 0.932, 0.864, 0.795, 0.750, 0.750, 0.727)

For adaptive n=3–7:

#calls(b,g,a) = 1 47 47
#gen drafts = 47
#acc drafts = 46
#gen tokens = 288
#acc tokens = 252
#mean acc len = 6.36
#acc rate/pos = (0.979, 0.957, 0.872, 0.745, 0.723, 0.617, 0.468)

The most useful numbers are not actually the headline acceptance percentages.

Look at the amount of accepted work:

Both configurations ultimately accepted almost the same number of speculative tokens:

254 versus 252.

Adaptive drafting did what it was designed to do: it avoided generating some unnecessary draft tokens.

It generated 288 draft tokens instead of 308.

But doing less draft work did not make the complete system faster.

The adaptive run required 47 speculative verification rounds, compared with only 44 for fixed n=7.

That difference is small in absolute terms, but speculative decoding is fundamentally a game of round economics.

Every extra round requires the large target model to participate again.

The fixed policy spends a little more work on the cheap draft path in exchange for making more forward progress per verification round.

On this RTX 4090 configuration, that trade appears to be favorable.

Adaptive starts conservatively

The implementation also explains why adaptive drafting does not immediately behave like fixed n=7.

The controller tracks an exponential moving average of accepted tokens.

Its initial acceptance estimate is approximately two tokens. With n_min=3, the effective draft therefore begins at three.

When a complete draft is accepted, the controller probes upward.

Conceptually, a highly predictable sequence can therefore move through something resembling:

3 → 4 → 5 → 6 → 7

rather than beginning at seven immediately.

That makes sense as a general-purpose controller.

If the workload suddenly becomes unpredictable, aggressively generating seven candidates can waste substantial draft and verification work.

But our JSON prompt was highly structured.

Keys, punctuation, indentation, repeated object layouts, IP-address patterns, numeric fields, and repeated vocabulary create a sequence that DFlash2 can predict unusually well.

Fixed n=7 can exploit that predictability immediately.

Adaptive drafting has to discover it.

Be careful with the per-position acceptance statistics

There is another subtle measurement detail worth mentioning.

At first glance, the trace appears to show:

fixed position 7:    0.727
adaptive position 7: 0.468

It would be tempting to conclude that adaptive drafting somehow makes the seventh token substantially less accurate.

That is not what the statistic means.

Adaptive mode does not necessarily draft seven tokens on every round.

The per-position counter is divided across all speculative calls, including calls where the adaptive controller never generated a seventh candidate.

Consequently, later positions naturally appear lower when adaptive drafting frequently chooses shorter sequences.

The more meaningful comparison here is the complete set of round counts, generated tokens, accepted tokens, and resulting end-to-end throughput.

This does not invalidate the Strix Halo result

The RTX 4090 result is almost the mirror image of the published Strix Halo result.

But the two experiments are running under very different hardware conditions.

The Strix Halo headline stack used:

• AMD Strix Halo
• Vulkan
• ROCmFP4 target weights
• an FP4 DFlash2 sidecar
• custom optimizations specifically targeting speculative batch widths

My test used:

• RTX 4090
• CUDA
• Q4_K_M target weights
• Q4_K_M DFlash2 weights

The original Strix Halo data showed fixed n=7 suffering very low useful acceptance while adaptive drafting remained effective.

My RTX 4090 run instead produced a mean accepted length of 6.77 tokens with fixed n=7.

Those are fundamentally different operating regimes.

The original adaptive result therefore remains interesting.

The RTX 4090 result adds a qualification:

The best policy cannot necessarily be selected from draft acceptance alone or transferred unchanged between hardware stacks.

Hardware changes speculative-decoding economics

It is easy to think of speculative decoding as a purely algorithmic optimization.

It is not.

Suppose a draft produces seven candidates and the target accepts six.

Whether drafting the seventh candidate was a mistake depends on how much that candidate cost compared with another target-model verification round.

On one architecture, reducing excess draft work may dominate.

On another, the draft path may be cheap enough that aggressively over-drafting is worthwhile if it eliminates even a few target passes.

This is particularly relevant when comparing something like a high-end discrete NVIDIA GPU against a bandwidth-constrained integrated GPU.

The same speculative controller is sitting on top of very different cost functions.

A useful abstraction is:

throughput ≠ acceptance rate

A more complete objective is closer to:

accepted progress per round
-------------------------------------------
draft cost + target verification cost

That is why a policy with apparently more wasted draft work can still win overall.

Workload matters too

There is an equally important limitation in this benchmark.

The workload was deliberately structured JSON.

That is precisely the sort of output where speculative models can achieve long accepted runs.

Natural prose is different.

Code is different again.

Tool-calling output may have a mixture of predictable syntax and unpredictable argument values.

A strong adaptive policy should be most valuable when predictability changes during generation.

That means this result should not be interpreted as:

Always use fixed n=7 on an RTX 4090.

The defensible conclusion is narrower:

On Qwen3.8-27B Q4_K_M with a Q4_K_M DFlash2 sidecar, CUDA, an RTX 4090, and this structured JSON workload, fixed n=7 was consistently faster than adaptive n=3–7.

That is already an important result.

The next useful experiment is to repeat the comparison across prose, coding, extraction, and tool-use workloads.

I would expect the optimal policy to move as content predictability changes.

Recorded CUDA build

The measured binary reported b10642-54ca36d4f. Its CUDA build used the commands below. A new checkout is a new runtime revision and should be recorded separately when repeating the comparison:

git clone https://github.com/LaurentZuijdwijk/llama.cpp.git
cd llama.cpp

cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON

cmake --build build -j"$(nproc)"

The adaptive options can be confirmed with:

./build/bin/llama-cli --help | grep -E "spec-draft-(adaptive|n-min|n-max|ngl)"

Fixed n=7 command

PROMPT='Generate a JSON array of 30 fictional server records. Each object must contain hostname, ip_address, operating_system, cpu_cores, memory_gb, environment, status, owner, and last_patch_date. Output valid JSON only.'

./build/bin/llama-cli \
  -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \
  -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \
  --spec-type draft-dflash \
  --spec-draft-n-max 7 \
  --spec-draft-n-min 7 \
  -ngl 99 \
  -ngld 99 \
  -fa on \
  -c 4096 \
  -n 300 \
  -st \
  --reasoning off \
  --temp 0 \
  --top-k 1 \
  --seed 42 \
  -p "$PROMPT"

Adaptive n=3–7 command

./build/bin/llama-cli \
  -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \
  -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \
  --spec-type draft-dflash \
  --spec-draft-n-max 7 \
  --spec-draft-n-min 3 \
  --spec-draft-adaptive \
  -ngl 99 \
  -ngld 99 \
  -fa on \
  -c 4096 \
  -n 300 \
  -st \
  --reasoning off \
  --temp 0 \
  --top-k 1 \
  --seed 42 \
  -p "$PROMPT"

The bigger finding

I originally ran this benchmark to see whether I could reproduce a new 65 tok/s adaptive-speculation result from Strix Halo.

The RTX 4090 easily crossed that number.

But the more interesting result was not the absolute throughput.

It was the ordering:

Bare             28.7 tok/s
Fixed n=3        65.3 tok/s
Adaptive 3–7     70.6 tok/s
Fixed n=7        80.1 tok/s

DFlash2 is clearly useful on this hardware.

Adaptive drafting is also clearly fast.

But for this particular workload, the aggressive fixed policy is faster still.

That suggests a broader direction for speculative decoding.

Instead of adapting only to acceptance behavior, future controllers may benefit from adapting to measured hardware cost as well: draft latency, target verification latency, batch-width efficiency, and the expected value of eliminating another target round.

The best speculative policy may not simply be the one that wastes the fewest tokens.

It may be the one that wastes the cheapest tokens.