vLLM Qwen3.8 NVFP4: Prefix-Cache Hits Corrupt DFlash2/DSpark Output
A vLLM 0.30/0.31 report isolates output corruption to prefix-cache hits combined with hidden-state speculative drafters on Qwen3.8-27B NVFP4.
Platform Engineering & MLOps Contributor
lanceryan111
lanceryan111 writes about platform engineering, MLOps, production inference, CI/CD, release engineering, and developer infrastructure. His work focuses on reliability, operational simplicity, reproducibility, and the engineering patterns required to move AI systems from experimental implementations into dependable production environments.
Published work
Articles
23 published
llama.cpp ARM BF16 Slow: Check the Scalar Fallback
If BF16 inference or multimodal encoding is dramatically slower than F16 or Q8_0 on ARM, current llama.cpp may be falling back to a scalar ggml_vec_dot_bf16 path when the CPU lacks a matching vector implementation.
By lanceryan111 ·
llama.cpp 'invalid argument: --no-mmap': Use --load-mode Correctly
If llama.cpp rejects --no-mmap, verify the binary and wrapper first. Current upstream deprecated the old flag in favor of --load-mode none, while wrappers and older builds may parse it differently.
By lanceryan111 ·
llama.cpp Cuts Qwen4exp Indexer Score Memory: What Actually Changed
An October 3 llama.cpp change rewrites Qwen4exp indexer scoring around the lightning indexer, reducing live score-buffer memory and extending backend handling across CUDA, Metal, and Vulkan.
By lanceryan111 ·
llama.cpp MTP Support: Use draft-mtp and Diagnose Failures
Current llama.cpp supports MTP speculative decoding through draft-mtp. Learn how to verify model support, enable it, isolate failed MTP contexts, and measure whether it actually improves your workload.
By lanceryan111 ·
llama.cpp 'unknown model architecture: mllama': What Actually Fixes It
Fix llama.cpp's unknown model architecture mllama error by confirming the GGUF architecture, ruling out an old binary, and switching to a runtime that supports Mllama vision.
By lanceryan111 ·
llama.cpp Max Context vs GPU Memory: Find the Real Limit
Find the largest llama.cpp context that fits your GPU by separating weights, KV cache, compute buffers, concurrency, auto-fit, and backend limits.
By lanceryan111 ·
vLLM Mistral 'Duplicate tool call id': Stop Suffix Truncation
Fix Mistral tokenizer 400s and Duplicate tool call id failures by avoiding last-nine-character truncation; PR #59213 proposes stable unique 9-character mappings.
By lanceryan111 ·
llama.cpp qwen4exp 'SPLIT_MODE_TENSOR not implemented': Update the Runtime
Fix the qwen4exp SPLIT_MODE_TENSOR not implemented error by updating stale llama.cpp builds, verifying the actual binary, and using layer split as a fallback.
By lanceryan111 ·
llama.cpp 'failed to allocate compute pp buffers': Fix -ub and VRAM
Fix llama.cpp compute pp buffer failures by lowering -ub/--ubatch-size first, then checking VRAM headroom, backend allocation limits, --fit, and speculation.
By lanceryan111 ·
Qwen3.8-27B on RTX 4090: 5,000 tok/s Prefill Reproduced
An independent Ubuntu RTX 4090 reproduction of NInfer's Qwen3.8-27B A8 claims reaches 4,999.5 tok/s pp2048, 500.9 tok/s edit decode, and exact retrieval at 260K context.
By lanceryan111 ·
vLLM 0.30.0 Pixtral ImportError with Transformers 5.17: Fix It
Fix vLLM 0.30.0 PixtralRotaryEmbedding ImportError with Transformers 5.17. vLLM 0.31.0 contains the compatibility path for the renamed axial RoPE API.
By lanceryan111 ·
SGLang Duplicate Text on Streaming Abort: Disable Incremental Mode or Patch
Fix SGLang duplicate text after graceful abort: current main still replays full text in incremental mode, while proposed PR #40902 was closed unmerged.
By lanceryan111 ·
llama.cpp Can Pick the Wrong GGUF From a Model Directory: What PR #28809 Changes
A source-level look at llama.cpp PR #28809, which makes server router discovery prefer a GGUF whose filename matches its model directory instead of relying on directory iteration order.
By lanceryan111 ·
llama.cpp Vision Latency: The Hidden Scheduler Reserve Cost
PR 28751 traces multi-image llama.cpp latency to repeated scheduler reservation when causal attention changes around vision chunks.
By lanceryan111 ·
vLLM GGUF 'Unknown gguf model_type: deepseek_v4': Fix Status
Diagnose vLLM GGUF's Unknown gguf model_type: deepseek_v4 error, check whether plugin PR #126 is present, and separate architecture mapping from tensor-loading failures.
By lanceryan111 ·
Why llama.cpp's Blackwell NVFP4 Prefill Was Starving Tensor Cores
A new llama.cpp CUDA patch shows that Blackwell NVFP4 prefill was limited by tile delivery, barriers, and register pressure—not simply tensor-core arithmetic.
By lanceryan111 ·
When a 10GbE Model Server Crawls Below 1 Token/s
A llama.cpp Qwen4 experimental report exposes how tiny inference-time reads can defeat fast storage, NFS caching, and otherwise capable local-AI infrastructure.
By lanceryan111 ·
llama.cpp Learns to Borrow Idle VRAM for Vision
A production-inference analysis of PR #28320, which temporarily reclaims llama.cpp compute scratch so a vision projector can run on GPU without reducing context.
By lanceryan111 ·
Qwen3.8-27B at 200K: The Performance Cliff Is VRAM, Not IQ3
Independent RTX 4090 testing finds UD-IQ3_XXS only ~2% slower than UD-Q3_K_XL at matched context, while reducing the 200K GPU memory budget from 16 to 15 GiB cuts prompt processing by 19%.
By lanceryan111 ·
GSQ-RCO Beats Unsloth on WikiText-2 at ~8.4GB — But Not on Throughput
An RTX 4090 reproduction of Qwen3.8-27B at ~2.5 bpw finds 1.93% lower WikiText-2 perplexity for GSQ-RCO than Unsloth UD-IQ2_S, while tensor dumps reveal a far more heterogeneous precision allocation and slightly lower throughput.
By lanceryan111 ·
Qwen3.8's 38GB PLE: Only 222MB Resident at 8K Context
An RTX 4090 reproduction of Qwen3.8-Flash-Next PLE paging shows a 38.4GB table reaching only 221.7MiB resident after an 8K high-diversity prompt sweep.
By lanceryan111 ·
DSpark PCTree on an RTX 4090: Higher Acceptance, Lower Throughput
An independent llama.cpp PCTree test on RTX 4090 with Qwen3-8B Q8: DSpark doubled decode speed, while wider trees raised acceptance but still lost throughput.
By lanceryan111 ·