llama.cpp Speculative Decode Crash: When the Scheduler Hash Set Is Too Small

A speculative-decoding crash can look like a model problem even when the model is fine.

On September 16, a new llama.cpp draft pull request documented a much narrower failure: the GGML backend scheduler allocates a tensor hash set at initialization, then assumes later computation graphs will never exceed that capacity. Larger speculative-decoding graphs can violate that assumption.

The result is not graceful backoff. It is an assertion.

The proposed fix in llama.cpp PR #28972 changes the scheduler from a fixed-capacity assumption to a grow-on-demand path.

This is interesting because the failure boundary is unusually concrete.

The assertion that fails

The PR identifies this scheduler check:

GGML_ASSERT((int)sched->hash_set.size >= graph->n_nodes + graph->n_leafs);

The scheduler hash set is initially sized from graph_size. But the graph it later receives can contain more nodes and leaves than that initial estimate anticipated.

That matters for speculative decoding because draft graphs grow as the number of speculative tokens increases.

The PR specifically calls out these draft paths:

draft-mtp
draft-dspark
draft-eagle3

and the parameter:

--spec-draft-n-max

So this is not a generic statement that “speculative decoding uses more memory.” The reported failure is a capacity mismatch in a specific scheduler data structure.

Why n_max=8 can work and n_max=9 can crash

The linked diagnosis from issue #28614 produced a particularly useful boundary.

According to the maintainer reproduction quoted in PR #28972:

n_max=8
hash capacity = 2053
nodes = 1706
leafs = 294
total = 2000

That fits.

At the next setting:

n_max=9
hash capacity = 2053
nodes = 1760
leafs = 294
total = 2054

The graph needs 2054 entries while the hash set has room for 2053.

One entry is enough to trip the assertion.

These figures are upstream contributor/maintainer observations, not RAMGPT benchmark results.

That distinction matters: the useful result here is the diagnosed boundary, not a performance claim.

Why this is easy to misdiagnose

Imagine tuning speculative decoding experimentally.

You start with a conservative draft length and the server works. You increase --spec-draft-n-max one step at a time. Through eight, everything looks normal. At nine, the process suddenly terminates in GGML scheduler code.

A reasonable first suspicion might be:

• an MTP incompatibility,
• a bad draft model,
• insufficient VRAM,
• a backend-specific kernel bug,
• or a malformed speculative graph.

But none of those explanations follows from this assertion.

The reported graph at n_max=9 is simply one slot larger than the scheduler’s preallocated hash capacity.

This is a good example of why exact error strings and boundary conditions are more useful than broad labels such as “llama.cpp speculative decoding crash.”

The code already hinted at the problem

PR #28972 notes that initialization already contains a FIXME indicating that capacity needs to account for leaf tensors as well as graph size.

That is an important source-level clue.

The scheduler’s later requirement is effectively:

capacity >= nodes + leafs

but initialization can establish a capacity that does not remain sufficient for a graph whose shape grows later.

Speculative decoding exposes the mismatch because increasing the draft-token limit changes the graph.

The proposed fix: grow instead of assert

The draft PR introduces an ensure_graph_capacity helper.

Rather than maintaining several assertions that assume the initial hash set is large enough, the scheduler checks the incoming graph and grows its storage when necessary.

The proposal also resizes the two arrays associated with tensors in the hash set.

The new allocation uses a 25% margin, matching the margin used by the graph allocator path referenced by the author.

Conceptually the behavior changes from:

initialize fixed capacity
-> receive larger graph
-> assert

to:

initialize capacity
-> receive graph
-> check required size
-> grow if necessary
-> continue graph splitting/allocation

This is a more robust invariant because graph size becomes something the scheduler validates at runtime instead of something it assumes forever from initialization.

What the PR does not prove yet

The pull request was opened as a draft on September 16.

Its author states that the change builds and directly addresses the diagnosed assertion, but also explains that their AMD/Vulkan fork hits an earlier llama-graph.cpp failure in the draft-DSpark path. Because of that separate crash, they could not run the complete n_max=9 decode end to end in their own environment.

The author therefore points to the reproduction in issue #28614 as the acceptance test.

That limitation should not be erased when summarizing the patch.

At publication time, this is a proposed upstream fix with a strong source-level diagnosis, not a merged fix that RAMGPT independently validated.

What to check if you hit this assertion

If your speculative-decoding setup crashes around the scheduler hash-set assertion, capture the exact configuration before changing several variables at once.

The useful fields include:

llama.cpp commit/build
main model
speculative method / draft model
--spec-draft-n-max
backend and GPU
hash_set.size
graph->n_nodes
graph->n_leafs

Then compare:

graph->n_nodes + graph->n_leafs

against the scheduler hash capacity.

If the sum exceeds capacity, especially immediately after increasing --spec-draft-n-max, PR #28972 describes the same failure class.

Temporarily lowering the draft-token maximum may avoid crossing the reported boundary, but that is a workaround, not the structural fix.

Why this small patch matters

Speculative decoding is full of interactions between models, cache state, graph construction, schedulers, kernels, and backend memory behavior. That makes failures tempting to describe at too high a level.

This one is refreshingly mechanical:

larger speculative draft
-> larger computation graph
-> nodes + leafs exceed fixed hash capacity
-> scheduler assertion

The lesson is broader than one assert. Runtime structures sized from an early graph estimate need either a guaranteed upper bound or a way to grow when later graph shapes change.

For llama.cpp users pushing MTP, DSpark, or EAGLE-style speculative decoding, that distinction can be the difference between blaming the model and finding a one-entry scheduler overflow.