Speculative Decoding's Hidden Context-Reuse Failure

Speculative decoding is usually discussed as a throughput optimization: draft more tokens, verify them cheaply, and increase generation speed without changing the target model’s output distribution.

A new llama.cpp bug report exposes a less obvious constraint. In a stateful inference engine, speculative decoding is also a state-management protocol. If the speculative path leaves even a tiny amount of state on the wrong side of an end-of-generation boundary, the cost can be vastly larger than the number of bad tokens.

Issue #28049, opened on August 30, describes exactly that failure mode on a Qwen3.8-27B hybrid attention/recurrent model. The reporter observed that MTP speculative decoding could leave one or two accepted tokens in the server slot after the turn-ending token. Those tokens were not visible in the returned completion. Token counts looked correct. The user-visible answer ended correctly.

But the internal slot no longer matched the prefix of the next conversation turn.

The result was dramatic: instead of reusing the previous answer, llama.cpp could fall back to an older recurrent-state checkpoint and re-prefill essentially the entire answer.

The interesting lesson is larger than this individual bug:

In hybrid LLM inference, a token can be semantically invisible yet computationally expensive.

The Failure Is an Amplifier

The reported sequence is roughly:

MTP proposes speculative tokens
        |
        v
target accepts a block
        |
        v
accepted block contains EOG
        |
        v
1-2 accepted tokens remain after EOG internally
        |
        v
server slot != prefix of next prompt
        |
        v
normal prefix reuse fails
        |
        v
recurrent state cannot be trimmed arbitrarily
        |
        v
restore older checkpoint
        |
        v
re-prefill previous answer

This is why measuring only decode tokens per second can miss an important performance failure.

The speculative decoder may still look fast while it is generating. In the measurements reported in #28049, draft-mtp produced roughly 21 tokens/s for the cited code workload, versus roughly 9 tokens/s without speculation. Yet follow-up turns could lose reuse of the preceding answer.

For an interactive workload with long assistant responses, the relevant objective is therefore not simply:

decode throughput

but something closer to:

conversation cost
  = generation cost
  + state repair cost
  + repeated prefill cost

A technique that wins the first term can still lose badly if it destabilizes the other two.

Why Hybrid Models Make This Worse

With a conventional transformer KV cache, removing a suffix is conceptually straightforward: discard cache entries after a position.

Hybrid models complicate that model of state.

Architectures mixing attention with recurrent or state-space components maintain state that is not necessarily representable as a simple per-token KV suffix. Once the recurrent state has advanced, reconstructing the state for an arbitrary earlier token may require rollback machinery or a previously saved checkpoint.

That changes the engineering meaning of speculative acceptance.

For a pure transformer, an unwanted speculative tail can often be treated as cache cleanup.

For a recurrent hybrid, it can become a state-consistency problem.

The #28049 report says llama.cpp’s usable checkpoint in this case was located before the end of the previous prompt. Once the internal slot diverged from the next request, restoration therefore discarded much more useful work than the one or two stray tokens that caused the mismatch.

That is the amplification effect:

2 wrong internal tokens
        !=
2 tokens of wasted compute

2 wrong internal tokens
        -> prefix mismatch
        -> checkpoint fallback
        -> thousands of tokens re-prefilled

The reporter gave three concrete examples where the lost reuse was the answer length plus four tokens: 16,791 tokens lost for a 16,787-token answer, 12,045 for 12,041, and 1,503 for 1,499.

Those are reporter measurements, not RAMGPT benchmarks, and the upstream issue remains labeled bug-unconfirmed at the time of writing. But the mechanism is specific enough to be technically interesting even before upstream resolution.

Why Normal API Metrics Can Hide It

One of the most useful details in the report is that the extra internal tokens were not exposed by the normal completion result.

The returned answer stopped correctly. Usage counts also appeared correct.

The clue was llama.cpp’s slot-prefix similarity and reuse behavior: the reported f_keep fell below the perfect-reuse case.

This creates a benchmarking trap.

Suppose an experiment records:

• output tokens per second;
• accepted speculative tokens;
• response latency;
• visible token counts.

All of those can look reasonable while the engine is carrying internal state that makes the next request much more expensive.

For long-context chat benchmarks, RAMGPT would therefore treat these as separate measurements:

1. decode throughput — how quickly the current answer is generated;
2. speculative acceptance — how much draft work survives verification;
3. prefix reuse — how much prior conversation survives into the next request;
4. re-prefill volume — how many supposedly historical tokens are processed again;
5. state restoration frequency — how often the engine falls back to a checkpoint or replay path.

The fourth metric is especially important. It converts an internal correctness detail into a directly measurable systems cost.

The Proposed Fix Is About Where the Boundary Is Enforced

The issue reporter experimented with removing the trailing tokens after the accepted block had already been processed. That ran into recurrent rollback constraints.

The proposed workaround instead truncates the accepted speculative sequence at the first EOG before rollback accounting is computed.

Conceptually:

accepted speculative block
        |
        v
find first EOG
        |
        v
truncate everything behind EOG
        |
        v
compute rollback/state transition once

That ordering matters.

Trying to repair the state after speculative verification can require a second mutation of recurrent state. Preventing the invalid tail from entering the committed accepted sequence avoids needing that second repair.

The reporter’s limited post-change measurements showed zero tokens retained past the turn in 19/19 rounds and previous-answer reuse in 12/13 reuse trials, while reported draft acceptance and decode throughput remained approximately unchanged. Again, these are third-party measurements from the issue, not independent RAMGPT validation.

A Second Issue Points to the Same Deeper Problem

A separate llama.cpp report, #28060, describes a Vulkan speculative-decoding livelock involving checkpoint restoration and replay. Its reported mechanism differs: replayed accepted tokens can be re-verified under backend-dependent numerical conditions, potentially restoring the same checkpoint repeatedly without forward progress.

The two reports should not be collapsed into one bug. Their evidence and triggers are different.

But they point toward the same architectural pressure:

speculative decoding over stateful hybrid models requires transaction-like semantics.

A speculative round has something resembling:

checkpoint
   |
   v
speculate
   |
   v
verify
   |
   +---- reject ----> restore/replay
   |
 accept
   |
   v
commit exactly one coherent state

Once recurrent memory, checkpoint restoration, backend-dependent replay, EOG boundaries, and multiple speculative methods interact, thinking of speculation as merely “predict N tokens and verify them” becomes incomplete.

The engine needs a well-defined state transaction:

• what state belongs to the speculative region;
• which accepted tokens are committed;
• what happens when EOG occurs inside a block;
• which state can be rolled back;
• whether replay is reconstruction or a new verification event;
• and what invariants must hold before the slot is reusable by the next request.

The Invariant That Matters

A useful invariant for a conversational inference server is:

After a completed turn, the engine’s committed internal token/state sequence must correspond exactly to the externally committed conversation prefix.

That sounds obvious, but speculative execution makes it nontrivial.

The user sees only committed output. The engine temporarily sees candidate futures. If part of a candidate future survives internally after the externally visible conversation has ended, the two histories diverge.

Pure transformer inference can sometimes make that divergence cheap to repair. Recurrent hybrid inference may not.

This suggests a broader design principle for next-generation local inference engines:

speculative correctness
        =
output correctness
        +
state correctness

Lossless verification guarantees the first property. It does not automatically guarantee the second property at every server lifecycle boundary.

What This Changes for Benchmarking

The most practical takeaway is methodological.

A single-turn benchmark is increasingly insufficient for evaluating advanced local inference stacks.

Modern engines combine:

• prompt caching;
• recurrent or hybrid state;
• MTP draft heads;
• n-gram speculation;
• checkpoint restoration;
• long context;
• continuous conversation slots.

The benchmark unit should therefore become a conversation transition, not merely a completion.

A minimal state-aware benchmark would run:

Turn 1: long prompt -> long answer
Turn 2: reuse Turn 1 + short new question
Turn 3: reuse Turn 1+2 + another question

and record both generation and repeated prompt processing at each boundary.

That catches an entire class of failures invisible to isolated tokens-per-second tests.

Bottom Line

Issue #28049 is currently an unconfirmed upstream bug report, not a settled llama.cpp finding. Its measurements should be treated accordingly.

But the systems lesson is strong regardless of how the final patch evolves.

Speculative decoding is moving into models whose inference state is much richer than a transformer KV cache. On those architectures, the most expensive speculative-decoding bug may not be a bad draft or a lower acceptance rate. It may be a tiny bookkeeping error that invalidates a huge amount of reusable state.

The performance question is no longer just:

How many speculative tokens did we accept?

It is also:

After accepting them, is the engine still in exactly the state that the next request thinks it is?

For long-context local inference, that second question can be worth thousands of tokens.