GGUF Quantization Levels Explained: Q4, Q5, Q6 and Q8
Learn what GGUF quantization labels such as Q4_K_M, Q5_K_M, Q6_K and Q8_0 mean, how they affect model size, quality, RAM, VRAM, and speed.
Topic hub · 33 articles · updated
Quantization & GGUF.
Which file to download, and what the quantization actually cost.
Quantization levels explained, GGUF versus other formats, FP8 and NVFP4 conversion, and QuantBench comparisons of real model files against pinned BF16 references.
New to Quantization & GGUF?
Start here
GGUF vs AWQ: What's the Difference and Which Should You Use?
Compare GGUF artifacts with AWQ checkpoints by runtime, hardware, metadata, memory, and measured performance, including vLLM's current GGUF plugin requirement.
RAMGPT QuantBench: Compare GGUF Files Against BF16
Explore three published GGUF comparison groups, inspect KL and decision-flip results, and download the QuantBench forensic summaries and comparison registry.
Search this page for your exact error text
Error fixes
- llama.cpp File Not Found: Fix GGUF Paths, Shards, and Router 404s
Diagnose llama.cpp File Not Found errors by checking the actual GGUF path, process working directory, split-model shards, router configuration, and file readability before blaming model compatibility.
- vLLM Mistral-Large-3 Accuracy Regression: Update the YaRN Fix
Fix the Mistral-Large-3 YaRN accuracy regression by using vLLM 0.31.0+ or a build containing PR #57563, which restores the missing no-mscale attention flag.
- vLLM GGUF
Unknown gguf model_type: deepseek_v4: Fix StatusDiagnose vLLM GGUF's Unknown gguf model_type: deepseek_v4 error, check whether plugin PR #126 is present, and separate architecture mapping from tensor-loading failures.
- How to Reduce LLM VRAM Usage Without Breaking Performance
Learn how to reduce local LLM VRAM usage with quantization, shorter context, KV-cache compression, CPU offloading, smaller models, and runtime tuning.
- Does llama.cpp Support NVFP4? Check the Exact Path Before You Convert
If you searched whether llama.cpp supports NVFP4, the answer depends on the checkpoint, converter, model architecture, backend, and quant-scale path. Use this checklist to identify the real blocker.
- Qwen3.8-27B W4A16 AutoRound: vLLM Artifact or GGUF?
Explains what dbirks/Qwen3.8-27B-W4A16-AutoRound actually contains, why vLLM is its native target, and what current llama.cpp conversion does with pack-quantized weights.
Understand it, then set it up
Guides & explainers
Quantization: How Fewer Bits Make Large Models Smaller
AI Foundations #18 explains post-training weight quantization, scale and rounding error, memory estimates, mixed precision, kernels, and quality tradeoffs.
Why llama.cpp's Blackwell NVFP4 Prefill Was Starving Tensor Cores
A new llama.cpp CUDA patch shows that Blackwell NVFP4 prefill was limited by tile delivery, barriers, and register pressure—not simply tensor-core arithmetic.
llama.cpp vs ik_llama.cpp: The Fork, Drama and Performance Race
A lively look at how ik_llama.cpp split from llama.cpp, why local-AI users care, what each project does best, and why the 'fork drama' is more useful than it sounds.
What Is LLM Quantization? FP16, INT8 and INT4 Explained
Learn how LLM quantization reduces model memory, what FP16, INT8 and INT4 mean, and how GGUF, AWQ, GPTQ, scales, groups, and quality trade-offs work.
How Much VRAM Do You Need for Local LLMs? 8B to 70B Guide
See which model sizes fit 8, 12, 16, 24, 48, and 80 GB GPUs, then account for quantization, KV cache, context length, and runtime overhead.
How to Run an LLM Locally: A Beginner's Guide
Learn how to run an LLM locally on Windows, Linux, or macOS, choose a model and quantization, understand hardware needs, and avoid common mistakes.
Tested on real hardware
Measured results
MiMo-V2.6 QuantBench: Public Q4 and IQ4_XS Compared
We tested public MiMo-V2.6-Distill-Qwen-9B GGUFs on one RTX 4090, comparing loadability, perplexity, speed, exact-task accuracy, and BF16 fidelity.
Qwen3.8-27B at 200K: The Performance Cliff Is VRAM, Not IQ3
Independent RTX 4090 testing finds UD-IQ3_XXS only ~2% slower than UD-Q3_K_XL at matched context, while reducing the 200K GPU memory budget from 16 to 15 GiB cuts prompt processing by 19%.
GSQ-RCO Beats Unsloth on WikiText-2 at ~8.4GB — But Not on Throughput
An RTX 4090 reproduction of Qwen3.8-27B at ~2.5 bpw finds 1.93% lower WikiText-2 perplexity for GSQ-RCO than Unsloth UD-IQ2_S, while tensor dumps reveal a far more heterogeneous precision allocation and slightly lower throughput.
RAMGPT QuantBench
Compare quantized files against BF16.
Verified GGUF comparisons against pinned BF16 references, with the method and limitations published alongside.
What changed, and why it matters
Analysis & deep dives
vLLM on GB10: Investigating an FP8 cuDNN GEMM Hang
An October 10 vLLM report traces a long-running DGX Spark inference freeze to a cuDNN FP8 GEMM path. Symptoms, control tests, and an attributed kernel-selection workaround.
vLLM Qwen3.8 NVFP4: Prefix-Cache Hits Corrupt DFlash2/DSpark Output
A vLLM 0.30/0.31 report isolates output corruption to prefix-cache hits combined with hidden-state speculative drafters on Qwen3.8-27B NVFP4.
AREX-2 Local Readiness: Qwen3.8 Base, Runtime and Quantization
A local-deployment audit of BAAI AREX-2: its Qwen3.8 base, Qwen3.5-compatible model class, memory budget, multimodal conversion and reference-runtime checks.
MiMo 9B EXL3: 145 tok/s on RTX 4090, 56 tok/s on RTX 3060
Independent EXL3 4.0 bpw measurements for Xiaomi MiMo-V2.6-Distill-Qwen-9B across RTX 4090, RTX 3060 12GB, mixed-GPU layer split, long context, TabbyAPI, vision, and tool calling.
llama.cpp FP8/NVFP4 Conversion: Why Quantization Scales Need First-Class Support
PR #28898 proposes moving FP8 and NVFP4 quantization scales into GGML so more scaled safetensors checkpoints can convert directly instead of relying on model-specific handling.
llama.cpp Can Pick the Wrong GGUF From a Model Directory: What PR #28809 Changes
A source-level look at llama.cpp PR #28809, which makes server router discovery prefer a GGUF whose filename matches its model directory instead of relying on directory iteration order.
llama.cpp Q4_K Prefill Fusion: Gate, Up, and SwiGLU in One CUDA Path
A new llama.cpp CUDA PR targets duplicate activation quantization and intermediate FP32 traffic in dense Q4_K MLP prefill, with contributor-reported gains from 4.1% to 14.7%.
We Tried to Break a 27B Q4 Model. The Benchmark Broke First.
A RAMGPT ExactBench research note on Qwen3.8-27B Q4: 94 short deterministic reasoning items, zero wrong answers, and several benchmark-design failures uncovered along the way.
The 16 KB Cliff That Can Turn GGUF Conversion Into an All-Day Job
A fixed 16-row quantization batch can cross a Windows allocator boundary, making GGUF conversion time depend dramatically on tensor shape.
The One-Line Vulkan Fix Hiding an 83% Prefill Gain
A llama.cpp Vulkan guard rejected valid hybrid-model KV-cache views, silently disabling a faster Flash Attention path for quantized caches.
When Lossless Speculation Stops Being Lossless on Blackwell
Two new llama.cpp reports show how batch-dependent CUDA kernels can break greedy speculative-decoding equivalence on Blackwell GPUs.
Qwen3-4B: 4.72% Lower KL Than UD-Q4_K_XL at Smaller Size
RAMGPT's tensor-aware Qwen3-4B recipe cut mean full-vocabulary KL 4.72% vs Unsloth UD-Q4_K_XL while producing a 0.10% smaller GGUF.
llama.cpp v0.2.0: What Actually Changed Under the Hood
A detailed look at llama.cpp v0.2.0: CUDA decode tuning, DSpark for LFM2, KV-cache work, lower-memory quantization, server hardening, and the new stable release track.
1-Bit LLMs Are Not Just Quantization: Why Ternary Models Matter
Why 1-bit and ternary LLMs may reshape local AI, and why tiny weights do not mean one-quarter the inference cost of a Q4 model.
llama.cpp Is More Than a GGUF Runner: A Local Inference Stack Emerges
Why llama.cpp is evolving from a GGUF model runner into a broader local inference stack spanning servers, scheduling, KV caches, and speculative decoding.