Research · Guides · Explainers
Independent AI research & technical writing
Topic hub · 21 articles · updated
When guessing ahead pays off, and when it quietly does not.
Draft models, MTP heads, EAGLE-3, DFlash and DSpark: setup, acceptance-rate diagnosis, correctness failures, and measured throughput on real GPUs.
New to Speculative Decoding & MTP?
AI Foundations #31 explains draft-and-verify generation, acceptance and rejection, why the target distribution can be preserved, and when speculative decoding actually reduces latency.
Current llama.cpp supports MTP speculative decoding through draft-mtp. Learn how to verify model support, enable it, isolate failed MTP contexts, and measure whether it actually improves your workload.
Independent RTX 4090 CUDA testing of adaptive DFlash2 in llama.cpp: fixed n=7 reached 80.11 tok/s on structured JSON, versus 70.61 tok/s for adaptive 3–7.
Search this page for your exact error text
output_hc_norm.weight not found: Current llama.cpp Fix StatusIf a Qwen3.8-Flash-Next MTP draft GGUF fails on output_hc_norm.weight, separate target-model loading from detached draft-head compatibility, inspect the sidecar tensor layout, and verify the exact llama.cpp MTP implementation.
blk.32.attn_norm.weight Not Found: Fix MTP MetadataFix the Qwen3.5-family blk.32.attn_norm.weight missing-tensor error when config metadata declares an MTP/NextN layer that the checkpoint does not contain.
If RX 7800 XT Vulkan MTP throughput drops after an update, pin the last known-good build and bisect the 146-commit regression window; issue #29410 remains open.
If EAGLE-3 acceptance collapses only with backend sampling, A/B --no-spec-draft-backend-sampling; allocator fix PR #29370 remains open.
Diagnose vLLM speculative-decoding CUDA illegal-memory-access crashes by comparing target and draft context limits; route over-limit requests without speculation or align the limits.
Issue #29168 initially bisected an MTP acceptance drop to CUDA MoE fusion, but the reporter later found it was specific to one raw prompt and closed the issue.
Diagnose llama.cpp speculative-decoding assertions when --spec-draft-n-max grows the graph past scheduler hash capacity; lower n_max as a workaround while PR #28972 remains open.
Tested on real hardware
An independent Ubuntu RTX 4090 reproduction of NInfer's Qwen3.8-27B A8 claims reaches 4,999.5 tok/s pp2048, 500.9 tok/s edit decode, and exact retrieval at 260K context.
What changed, and why it matters
A vLLM 0.30/0.31 report isolates output corruption to prefix-cache hits combined with hidden-state speculative drafters on Qwen3.8-27B NVFP4.
A new llama.cpp prototype wires Qwen3.8-Flash-Next's built-in NextN/MTP head into speculative decoding. Here is what it changes, the reported speedup, and why mainline still cannot use it.
llama.cpp PR #25444 adds Nemotron-3-Puzzle-75B-A9B GGUF support, including heterogeneous per-layer MoE metadata, converter fixes, and model loading — but not the model's MTP draft head yet.
Two new llama.cpp reports show how batch-dependent CUDA kernels can break greedy speculative-decoding equivalence on Blackwell GPUs.
A forensic analysis of llama.cpp issue #28049, where accepted tokens past EOG can invalidate reusable state and trigger large re-prefill costs.
A code-level analysis of llama.cpp's DFlash2 merge: dynamic local convolution, lattice path selection, runtime integration, benchmarks, and review tradeoffs.
A detailed look at llama.cpp v0.2.0: CUDA decode tuning, DSpark for LFM2, KV-cache work, lower-memory quantization, server hardening, and the new stable release track.
An independent llama.cpp PCTree test on RTX 4090 with Qwen3-8B Q8: DSpark doubled decode speed, while wider trees raised acceptance but still lost throughput.
Why llama.cpp is evolving from a GGUF model runner into a broader local inference stack spanning servers, scheduling, KV caches, and speculative decoding.
A hands-on guide to running Qwen3.8-27B on one RTX 4090 with patched vLLM, 64K context, MTP speculative decoding, tool calling, and OpenWebUI.