Running Qwen3.8-27B on an RTX 4090 with vLLM + OpenWebUI
A hands-on guide to running Qwen3.8-27B on one RTX 4090 with patched vLLM, 64K context, MTP speculative decoding, tool calling, and OpenWebUI.
Topic hub · 37 articles · updated
vLLM & SGLang.
Serving errors fixed, and the scheduler behavior behind them.
Startup and memory failures, tool calling and structured outputs, prefix caching, admission control, and model-specific regressions in vLLM and SGLang.
New to vLLM & SGLang?
Start here
vLLM KV Cache Too Small: Fix max_model_len Startup Failures
Fix vLLM startup errors caused by insufficient KV-cache capacity. Separate weight OOM from long-context sizing, batch concurrency and CUDA graph memory.
vLLM Per-Request Metrics: Separate Queue, Prefill, and Decode
A practical guide to vLLM response-level timing: distinguish queue wait, time to first token, decode speed, streaming usage and Prometheus aggregates.
Search this page for your exact error text
Error fixes
- vLLM guided_json Removed: Fix Structured Outputs Requests
Fix vLLM structured-output requests after the guided_json and guided_decoding_backend API removal, and distinguish obsolete fields from XGrammar JSON Schema compilation errors.
- vLLM Tool Calling 400 or Empty tool_calls: Fix Parser and Template
Diagnose vLLM tool-calling failures by separating HTTP 400 startup-flag errors from tool_choice behavior, parser mismatches, chat templates, and malformed arguments.
- vLLM EmbeddingGemma 2 dimensions=768 Error: Omit dimensions for Full Width
If vLLM rejects EmbeddingGemma 2 dimensions=768 but returns 768 values by default, omit dimensions for full-width output and verify the vector length.
- vLLM 404
Model Does Not Exist: Fix --served-model-nameFix vLLM 404 model-not-found errors by reading /v1/models, using the exact data[].id, and correcting --served-model-name or the client endpoint.
- vLLM LLM.generate() Hangs with msgspec Error: Fix SamplingParams Types
Fix a vLLM 0.31.0 LLM.generate() hang when msgspec reports Expected int, got float by validating integer SamplingParams fields before generation.
- vLLM DeepGEMM
Runtime Is Already Registered: Update Past the FixFix vLLM DeepGEMM's Runtime is already registered ImportError by using a build containing merged PR #48962, which moved vendored DeepGEMM away from the duplicate pybind11 registration path.
- vLLM H100 FA4 Decode Up to 48% Slower: Check the SplitKV Pin
Diagnose H100/SM90 FA4 decode regressions by checking the pinned FlashAttention commit: v0.31.0 still uses the pre-fix pin, while current vLLM main includes PR #198.
- vLLM Mistral
Duplicate tool call id: Stop Suffix TruncationFix Mistral tokenizer 400s and Duplicate tool call id failures by avoiding last-nine-character truncation; PR #59213 proposes stable unique 9-character mappings.
- vLLM Qwen3-Omni Position IDs Shift or Crash: Update the M-RoPE Fix
Fix Qwen3-Omni position-id length crashes and silent M-RoPE shifts with merged PR #58890; verify the exact build because release branches may not contain the merge.
- vLLM 0.30.0 Pixtral ImportError with Transformers 5.17: Fix It
Fix vLLM 0.30.0 PixtralRotaryEmbedding ImportError with Transformers 5.17. vLLM 0.31.0 contains the compatibility path for the renamed axial RoPE API.
- SGLang Duplicate Text on Streaming Abort: Disable Incremental Mode or Patch
Fix SGLang duplicate text after graceful abort: current main still replays full text in incremental mode, while proposed PR #40902 was closed unmerged.
- vLLM DeepSeek-V4 max_model_len Auto-Fit Crash: Update or Set It Explicitly
Fix DeepSeek-V4 CUDA-graph startup assertions after max_model_len auto-fit by using vLLM 0.31.0+ with the ModelState fix or setting an explicit context that fits.
- vLLM CUDA Illegal Memory Access: Check Draft vs Target Context
Diagnose vLLM speculative-decoding CUDA illegal-memory-access crashes by comparing target and draft context limits; route over-limit requests without speculation or align the limits.
- vLLM Mistral-Large-3 Accuracy Regression: Update the YaRN Fix
Fix the Mistral-Large-3 YaRN accuracy regression by using vLLM 0.31.0+ or a build containing PR #57563, which restores the missing no-mscale attention flag.
- vLLM GGUF
Unknown gguf model_type: deepseek_v4: Fix StatusDiagnose vLLM GGUF's Unknown gguf model_type: deepseek_v4 error, check whether plugin PR #126 is present, and separate architecture mapping from tensor-loading failures.
- How to Reduce LLM VRAM Usage Without Breaking Performance
Learn how to reduce local LLM VRAM usage with quantization, shorter context, KV-cache compression, CPU offloading, smaller models, and runtime tuning.
- Qwen3.8-27B W4A16 AutoRound: vLLM Artifact or GGUF?
Explains what dbirks/Qwen3.8-27B-W4A16-AutoRound actually contains, why vLLM is its native target, and what current llama.cpp conversion does with pack-quantized weights.
Understand it, then set it up
Guides & explainers
vLLM 0.31 Admission Control: Active Sequences vs Queue Limits
Diagnose vLLM 0.31.0 concurrency using max_num_active_seqs, max_num_seqs, and queue admission: configuration, failure modes, and a controlled test plan.
Constrained Decoding: How an LLM Can Follow a JSON Schema
AI Foundations #34 explains token masks, grammars, JSON Schema, why syntax constraints are not factual guarantees, and what engineers should test when serving structured model outputs.
Disaggregated Prefill and Decode: Two Jobs Inside One LLM Request
AI Foundations #33 explains why LLM servers may separate prompt processing from token generation, how KV state moves, and the trade-offs between latency, throughput and operational cost.
Tensor, Data, Pipeline and Expert Parallelism for LLM Inference
AI Foundations #32 explains the four major ways inference workloads are split across GPUs, what each one replicates or shards, and why communication changes the trade-off.
Speculative Decoding: Why a Draft Model Can Speed Up a Larger LLM
AI Foundations #31 explains draft-and-verify generation, acceptance and rejection, why the target distribution can be preserved, and when speculative decoding actually reduces latency.
PagedAttention: How LLM Servers Stop KV Cache From Wasting Memory
AI Foundations #30 explains why variable-length KV caches fragment GPU memory, how paging divides cache into blocks, how logical token positions map to physical memory, and why this helps serving throughput.
Prefix Caching and KV Reuse: Why Repeated Prompts Can Be Cheaper
AI Foundations #29 explains prefix caching, KV reuse, exact token matching, cache hits, eviction, prompt-template effects, and why reuse reduces prefill work without caching the answer.
Continuous Batching and Scheduling: How LLM Servers Share a GPU
AI Foundations #28 explains static batching, continuous batching, request scheduling, chunked prefill, head-of-line blocking, fairness, and why server throughput and user latency can pull in opposite directions.
GGUF vs AWQ: What's the Difference and Which Should You Use?
Compare GGUF artifacts with AWQ checkpoints by runtime, hardware, metadata, memory, and measured performance, including vLLM's current GGUF plugin requirement.
How Much VRAM Do You Need for Local LLMs? 8B to 70B Guide
See which model sizes fit 8, 12, 16, 24, 48, and 80 GB GPUs, then account for quantization, KV cache, context length, and runtime overhead.
What Is a KV Cache? How It Works and Why It Uses VRAM
Learn what an LLM KV cache stores, why it speeds up token generation, how to estimate its VRAM use, and how context length and quantization affect memory.
What changed, and why it matters
Analysis & deep dives
vLLM CUDA Wheel and PyTorch ABI: Diagnose a Broken Upgrade
A version-aware guide to vLLM import failures after PyTorch or CUDA changes, with environment checks, clean-install controls, and a recovery decision tree.
vLLM Prefix Cache Salts: Who Can Reuse Prompt State?
A guide to vLLM automatic prefix caching, the purpose of cache_salt, hash collision risks, and a controlled latency test that separates cache reuse from isolation.
vLLM on GB10: Investigating an FP8 cuDNN GEMM Hang
An October 10 vLLM report traces a long-running DGX Spark inference freeze to a cuDNN FP8 GEMM path. Symptoms, control tests, and an attributed kernel-selection workaround.
vLLM Qwen3.8 NVFP4: Prefix-Cache Hits Corrupt DFlash2/DSpark Output
A vLLM 0.30/0.31 report isolates output corruption to prefix-cache hits combined with hidden-state speculative drafters on Qwen3.8-27B NVFP4.
vLLM Hybrid Mamba Prefix Caching Can Cost Throughput Before a Cache Hit
vLLM issue #60008 reports an 11–16% output-throughput cost from Mamba align-mode prefix caching with zero cache hits, exposing host-side launch and prefill-splitting overhead.
AREX-2 Local Readiness: Qwen3.8 Base, Runtime and Quantization
A local-deployment audit of BAAI AREX-2: its Qwen3.8 base, Qwen3.5-compatible model class, memory budget, multimodal conversion and reference-runtime checks.