llama.cpp for Beginners: Build, Run and Tune Your First Local LLM
A simple step-by-step llama.cpp guide for beginners: compile from source, run a GGUF model, use GPU acceleration, and tune the main inference settings.
Topic hub · 80 articles · updated
llama.cpp.
Fix the error in front of you, then understand the runtime underneath it.
Error fixes, backend choices (CUDA, Vulkan, Metal), context and memory limits, MTP and speculative decoding, and source-level analysis of new llama.cpp changes, tested on real hardware.
New to llama.cpp?
Start here
llama.cpp Errors and Fixes: Diagnose the Failure Before Tuning
Match common llama.cpp errors to the right fix: update stale runtimes, repair GGUF paths, isolate tensor mismatches, reduce KV or compute memory, and identify regressions.
llama.cpp CUDA vs Vulkan: Which Backend Should You Use?
A practical llama.cpp CUDA-vs-Vulkan guide: verify the backend actually loaded, compare device support and kernels, benchmark the same workload, and avoid mistaking CPU fallback for a backend result.
Search this page for your exact error text
Error fixes
- llama.cpp
failed to allocate buffer for rs cache: Fix Recurrent-State OOMFix llama.cpp startup failures reporting failed to allocate buffer for rs cache by separating recurrent-state memory from KV cache, GPU weights, and server concurrency.
- llama.cpp ARM BF16 Slow: Check the Scalar Fallback
If BF16 inference or multimodal encoding is dramatically slower than F16 or Q8_0 on ARM, current llama.cpp may be falling back to a scalar ggml_vec_dot_bf16 path when the CPU lacks a matching vector implementation.
- llama.cpp SPLIT_AXIS_UNKNOWN: Fix Tensor-Split Startup Crashes
Fix llama.cpp SPLIT_AXIS_UNKNOWN assertions by testing layer split, updating old builds, isolating tensor-distribution graph failures, and separating them from VRAM errors.
- Qwen3.8 MTP
output_hc_norm.weight not found: Current llama.cpp Fix StatusIf a Qwen3.8-Flash-Next MTP draft GGUF fails on output_hc_norm.weight, separate target-model loading from detached draft-head compatibility, inspect the sidecar tensor layout, and verify the exact llama.cpp MTP implementation.
- llama.cpp File Not Found: Fix GGUF Paths, Shards, and Router 404s
Diagnose llama.cpp File Not Found errors by checking the actual GGUF path, process working directory, split-model shards, router configuration, and file readability before blaming model compatibility.
- llama.cpp
blk.32.attn_norm.weightNot Found: Fix MTP MetadataFix the Qwen3.5-family blk.32.attn_norm.weight missing-tensor error when config metadata declares an MTP/NextN layer that the checkpoint does not contain.
- llama.cpp
unknown model architecture: mllama: What Actually Fixes ItFix llama.cpp's unknown model architecture mllama error by confirming the GGUF architecture, ruling out an old binary, and switching to a runtime that supports Mllama vision.
- llama.cpp qwen4exp
SPLIT_MODE_TENSOR not implemented: Update the RuntimeFix the qwen4exp SPLIT_MODE_TENSOR not implemented error by updating stale llama.cpp builds, verifying the actual binary, and using layer split as a fallback.
- llama.cpp check_tensor_dims: Fix
tensor not foundandwrong shapeFix llama.cpp check_tensor_dims errors by separating missing tensors from wrong shapes, testing the unquantized GGUF, and matching converter, config, and runtime revisions.
- llama.cpp
failed to allocate compute pp buffers: Fix -ub and VRAMFix llama.cpp compute pp buffer failures by lowering -ub/--ubatch-size first, then checking VRAM headroom, backend allocation limits, --fit, and speculation.
- llama.cpp
failed to allocate buffer for kv cache: Fix VRAM and ContextFix llama.cpp KV-cache allocation failures by lowering context or parallelism, leaving VRAM headroom, checking cache types, and reading the real allocator error.
- llama.cpp
unknown model architecture: glm5next: Update Your RuntimeFix llama.cpp's unknown model architecture glm5next error by running a build that includes merged GLM-5.3-Flash support and verifying the actual binary or bundled backend.
- llama.cpp RX 7800 XT Vulkan MTP 54% Slower: Pin and Bisect
If RX 7800 XT Vulkan MTP throughput drops after an update, pin the last known-good build and bisect the 146-commit regression window; issue #29410 remains open.
- llama.cpp EAGLE-3 Acceptance Collapse: Backend Sampling Workaround
If EAGLE-3 acceptance collapses only with backend sampling, A/B --no-spec-draft-backend-sampling; allocator fix PR #29370 remains open.
- llama.cpp MTP Acceptance Drop After CUDA MoE Fusion: Verify the Prompt First
Issue #29168 initially bisected an MTP acceptance drop to CUDA MoE fusion, but the reporter later found it was specific to one raw prompt and closed the issue.
- llama.cpp OpenVINO 4 GiB KV Allocation Limit: Update Past #29852
Fix OpenVINO large-context allocation failures by using llama.cpp master with #29852, which reports the GPU's real single-allocation limit so GGML can split buffers.
- llama.cpp Vulkan Wrong Results from Sliced KV Cache: Update #28956
Fix llama.cpp Vulkan wrong-result regressions from sliced cache views by updating past merged PR #28956, then compare the same GGUF and prompt against CPU.
- llama.cpp Speculative Decode Crash at Higher n_max: Scheduler Hash Fix
Diagnose llama.cpp speculative-decoding assertions when --spec-draft-n-max grows the graph past scheduler hash capacity; lower n_max as a workaround while PR #28972 remains open.
- llama.cpp Metal 2 GiB Slice Bug: Update to the Merged Fix
Fix llama.cpp Metal corruption around the 2 GiB slice-offset boundary by updating to a build containing merged PR #28748, then retesting the same large workload.
- How to Reduce LLM VRAM Usage Without Breaking Performance
Learn how to reduce local LLM VRAM usage with quantization, shorter context, KV-cache compression, CPU offloading, smaller models, and runtime tuning.
- Local LLM Slow? Diagnose Prefill, Decode, GPU Offload, and Memory
Find why a local LLM is slow by separating prompt processing from token generation, checking GPU offload, memory pressure, context growth, and CPU bottlenecks.
- llama.cpp
invalid argument: --no-mmap: Use --load-mode CorrectlyIf llama.cpp rejects --no-mmap, verify the binary and wrapper first. Current upstream deprecated the old flag in favor of --load-mode none, while wrappers and older builds may parse it differently.
- Does llama.cpp Support NVFP4? Check the Exact Path Before You Convert
If you searched whether llama.cpp supports NVFP4, the answer depends on the checkpoint, converter, model architecture, backend, and quant-scale path. Use this checklist to identify the real blocker.
- llama.cpp Max Context vs GPU Memory: Find the Real Limit
Find the largest llama.cpp context that fits your GPU by separating weights, KV cache, compute buffers, concurrency, auto-fit, and backend limits.
- llama.cpp MTP Support: Use draft-mtp and Diagnose Failures
Current llama.cpp supports MTP speculative decoding through draft-mtp. Learn how to verify model support, enable it, isolate failed MTP contexts, and measure whether it actually improves your workload.
Understand it, then set it up
Guides & explainers
Constrained Decoding: How an LLM Can Follow a JSON Schema
AI Foundations #34 explains token masks, grammars, JSON Schema, why syntax constraints are not factual guarantees, and what engineers should test when serving structured model outputs.
Why llama.cpp's Blackwell NVFP4 Prefill Was Starving Tensor Cores
A new llama.cpp CUDA patch shows that Blackwell NVFP4 prefill was limited by tile delivery, barriers, and register pressure—not simply tensor-core arithmetic.
Why a Small Vulkan Fusion Gave Gemma 4 About 4% in llama.cpp
A source-level look at llama.cpp PR #28024: how fusing RMSNorm, residual math, RoPE, views, and row writes cuts Vulkan graph overhead in Gemma 4.
llama.cpp vs ik_llama.cpp: The Fork, Drama and Performance Race
A lively look at how ik_llama.cpp split from llama.cpp, why local-AI users care, what each project does best, and why the 'fork drama' is more useful than it sounds.
GGUF Quantization Levels Explained: Q4, Q5, Q6 and Q8
Learn what GGUF quantization labels such as Q4_K_M, Q5_K_M, Q6_K and Q8_0 mean, how they affect model size, quality, RAM, VRAM, and speed.
GGUF vs AWQ: What's the Difference and Which Should You Use?
Compare GGUF artifacts with AWQ checkpoints by runtime, hardware, metadata, memory, and measured performance, including vLLM's current GGUF plugin requirement.
How Much VRAM Do You Need for Local LLMs? 8B to 70B Guide
See which model sizes fit 8, 12, 16, 24, 48, and 80 GB GPUs, then account for quantization, KV cache, context length, and runtime overhead.
How to Run an LLM Locally: A Beginner's Guide
Learn how to run an LLM locally on Windows, Linux, or macOS, choose a model and quantization, understand hardware needs, and avoid common mistakes.
What Is a KV Cache? How It Works and Why It Uses VRAM
Learn what an LLM KV cache stores, why it speeds up token generation, how to estimate its VRAM use, and how context length and quantization affect memory.
Tested on real hardware
Measured results
Gemma 4 vs Qwen3.8: Parallel Tool Calling Changes Agent Scaling
A local RTX 4090 agent benchmark shows Gemma 4 winning small tasks on speed while Qwen3.8 keeps agent turns flat by batching parallel tool calls.
MiMo-V2.6 QuantBench: Public Q4 and IQ4_XS Compared
We tested public MiMo-V2.6-Distill-Qwen-9B GGUFs on one RTX 4090, comparing loadability, perplexity, speed, exact-task accuracy, and BF16 fidelity.
We Gave Three Qwen3.8-27B Variants a Cybersecurity Exam
A first-pass RAMGPT field test of Base, Heretic, and OrcaRouter Qwen3.8-27B variants across AI-agent security, secure code review, and modern glibc exploitation.
HereticBench: What Actually Changes When You Remove a Model's Refusal Behavior?
A paired benchmark of refusal behavior, calibrated reasoning, open-ended response drift, and epistemic integrity in a 27B Base vs Heretic model comparison.
Qwen3.8-27B at 200K: The Performance Cliff Is VRAM, Not IQ3
Independent RTX 4090 testing finds UD-IQ3_XXS only ~2% slower than UD-Q3_K_XL at matched context, while reducing the 200K GPU memory budget from 16 to 15 GiB cuts prompt processing by 19%.
GSQ-RCO Beats Unsloth on WikiText-2 at ~8.4GB — But Not on Throughput
An RTX 4090 reproduction of Qwen3.8-27B at ~2.5 bpw finds 1.93% lower WikiText-2 perplexity for GSQ-RCO than Unsloth UD-IQ2_S, while tensor dumps reveal a far more heterogeneous precision allocation and slightly lower throughput.
Qwen3.8's 38GB PLE: Only 222MB Resident at 8K Context
An RTX 4090 reproduction of Qwen3.8-Flash-Next PLE paging shows a 38.4GB table reaching only 221.7MiB resident after an 8K high-diversity prompt sweep.
What changed, and why it matters
Analysis & deep dives
llama.cpp v0.6.0 Extended Batch API: A Migration Checklist
llama.cpp v0.6.0 adds llama_batch_ext and llama_process for mixed token and embedding input. Here is what C API integrators need to test before upgrading.
llama.cpp Host-RAM MoE Pipeline Parallelism: What PR #29963 Changes
A source-level look at llama.cpp PR #29963, which explores pipeline parallelism when MoE expert weights remain in host RAM and must be uploaded to the GPU that owns each layer.
llama.cpp K-Pool Graph Reallocation: The GLM-5.3/Qwen4exp Fix
Merged llama.cpp PR 29958 stabilizes k-pool graph shapes for GLM-5.3 and Qwen4exp, addressing reallocation after mutable state changed the reserved graph.
llama.cpp Cuts Qwen4exp Indexer Score Memory: What Actually Changed
An October 3 llama.cpp change rewrites Qwen4exp indexer scoring around the lightning indexer, reducing live score-buffer memory and extending backend handling across CUDA, Metal, and Vulkan.
AREX-2 Local Readiness: Qwen3.8 Base, Runtime and Quantization
A local-deployment audit of BAAI AREX-2: its Qwen3.8 base, Qwen3.5-compatible model class, memory budget, multimodal conversion and reference-runtime checks.
llama.cpp Vulkan Can Reuse Corrupted Scratch Data After Flash Attention
Source analysis of a Sep 28 llama.cpp Vulkan bug where flash attention or large softmax can overwrite cached scratch data, causing a later matmul to reuse corrupted input.
Adding a Second GPU Made My 4090 Slower on Qwen3.8
An independent Qwen3.8 Flash-Next benchmark on RTX 4090 + RTX 3060 showing how auto-fit, manual MoE placement, quantization, and MTP changed decode speed.
llama.cpp FP8/NVFP4 Conversion: Why Quantization Scales Need First-Class Support
PR #28898 proposes moving FP8 and NVFP4 quantization scales into GGML so more scaled safetensors checkpoints can convert directly instead of relying on model-specific handling.
llama.cpp Can Pick the Wrong GGUF From a Model Directory: What PR #28809 Changes
A source-level look at llama.cpp PR #28809, which makes server router discovery prefer a GGUF whose filename matches its model directory instead of relying on directory iteration order.
llama.cpp Vision Latency: The Hidden Scheduler Reserve Cost
PR 28751 traces multi-image llama.cpp latency to repeated scheduler reservation when causal attention changes around vision chunks.
llama.cpp Q4_K Prefill Fusion: Gate, Up, and SwiGLU in One CUDA Path
A new llama.cpp CUDA PR targets duplicate activation quantization and intermediate FP32 traffic in dense Q4_K MLP prefill, with contributor-reported gains from 4.1% to 14.7%.
Qwen4Exp MTP in llama.cpp: The Prototype Works, but It Is Not Upstream
A new llama.cpp prototype wires Qwen3.8-Flash-Next's built-in NextN/MTP head into speculative decoding. Here is what it changes, the reported speedup, and why mainline still cannot use it.
Nemotron-3-Puzzle 75B-A9B Lands in llama.cpp — MTP Is Still Missing
llama.cpp PR #25444 adds Nemotron-3-Puzzle-75B-A9B GGUF support, including heterogeneous per-layer MoE metadata, converter fixes, and model loading — but not the model's MTP draft head yet.
We Tried to Break a 27B Q4 Model. The Benchmark Broke First.
A RAMGPT ExactBench research note on Qwen3.8-27B Q4: 94 short deterministic reasoning items, zero wrong answers, and several benchmark-design failures uncovered along the way.
Ollama and llama.cpp: It's Complicated
A source-driven history of the increasingly uncomfortable relationship between Ollama and llama.cpp: upstream engineering, attribution disputes, forks, compatibility failures, community backlash, and the cost of hiding the engine.
Your Windows llama.cpp Build May Be Leaving VNNI on the Table
A fresh llama.cpp issue shows how MSVC feature detection can silently compile out AVX-VNNI paths even when the CPU supports them.
The 16 KB Cliff That Can Turn GGUF Conversion Into an All-Day Job
A fixed 16-row quantization batch can cross a Windows allocator boundary, making GGUF conversion time depend dramatically on tensor shape.
When a 10GbE Model Server Crawls Below 1 Token/s
A llama.cpp Qwen4 experimental report exposes how tiny inference-time reads can defeat fast storage, NFS caching, and otherwise capable local-AI infrastructure.
llama.cpp Learns to Borrow Idle VRAM for Vision
A production-inference analysis of PR #28320, which temporarily reclaims llama.cpp compute scratch so a vision projector can run on GPU without reducing context.
The One-Line Vulkan Fix Hiding an 83% Prefill Gain
A llama.cpp Vulkan guard rejected valid hybrid-model KV-cache views, silently disabling a faster Flash Attention path for quantized caches.
When Lossless Speculation Stops Being Lossless on Blackwell
Two new llama.cpp reports show how batch-dependent CUDA kernels can break greedy speculative-decoding equivalence on Blackwell GPUs.
Speculative Decoding's Hidden Context-Reuse Failure
A forensic analysis of llama.cpp issue #28049, where accepted tokens past EOG can invalidate reusable state and trigger large re-prefill costs.
LongCat Sparse Lands in llama.cpp — But the Attention Path Is Still Dense
A source-path experiment on the first LongCat-Flash-Lite-Sparse llama.cpp fork: LSA token selection is real, but selected tokens are converted into a mask over full K/V rather than compact sparse attention execution.
Qwen3.8 Flash-Next Lost 67% of Its Decode Speed Under Memory Contention
Independent RTX 4090 testing shows Qwen3.8 Flash-Next decode falling from 21.00 to 6.93 tok/s under concurrent STREAM traffic, while matched CPU-only contention reduced throughput by only 5.4%.
Qwen3-4B: 4.72% Lower KL Than UD-Q4_K_XL at Smaller Size
RAMGPT's tensor-aware Qwen3-4B recipe cut mean full-vocabulary KL 4.72% vs Unsloth UD-Q4_K_XL while producing a 0.10% smaller GGUF.
DFlash2 Lands in llama.cpp: What PR #27342 Actually Changes
A code-level analysis of llama.cpp's DFlash2 merge: dynamic local convolution, lattice path selection, runtime integration, benchmarks, and review tradeoffs.
Porting Qwen4 to llama.cpp: Qwen3.8-Flash-Next Internals
Implementation strategy for Qwen3.8-Flash-Next in llama.cpp: Gated DeltaNet reuse, Hyper-Connections, PLE, QSA, state handling, CUDA, and GGUF.
Qwen4 in llama.cpp: Dissecting PR #27742
PR #27742 shows Qwen4 needs no new GGML ops; the hard engineering is indexed state, PLE residency, quantizer staging, and backend memory invariants.
Qwen4 in llama.cpp: A Pre-Merge Runtime Audit of PR #27742
A systems audit of Qwen4Exp in llama.cpp: state ownership, indexer-cache lifecycle, multi-slot correctness, quantized KV, graph reservation, and PLE.
Adaptive Speculative Decoding on RTX 4090: Fixed n=7 Was 13.5% Faster
Independent RTX 4090 CUDA testing of adaptive DFlash2 in llama.cpp: fixed n=7 reached 80.11 tok/s on structured JSON, versus 70.61 tok/s for adaptive 3–7.
RAMGPT QuantBench: Compare GGUF Files Against BF16
Explore three published GGUF comparison groups, inspect KL and decision-flip results, and download the QuantBench forensic summaries and comparison registry.
llama.cpp v0.2.0: What Actually Changed Under the Hood
A detailed look at llama.cpp v0.2.0: CUDA decode tuning, DSpark for LFM2, KV-cache work, lower-memory quantization, server hardening, and the new stable release track.
DSpark PCTree on an RTX 4090: Higher Acceptance, Lower Throughput
An independent llama.cpp PCTree test on RTX 4090 with Qwen3-8B Q8: DSpark doubled decode speed, while wider trees raised acceptance but still lost throughput.
1-Bit LLMs Are Not Just Quantization: Why Ternary Models Matter
Why 1-bit and ternary LLMs may reshape local AI, and why tiny weights do not mean one-quarter the inference cost of a Q4 model.
llama.cpp Is More Than a GGUF Runner: A Local Inference Stack Emerges
Why llama.cpp is evolving from a GGUF model runner into a broader local inference stack spanning servers, scheduling, KV caches, and speculative decoding.
Qwen3.8-27B on 24GB GPUs: The New Local LLM Sweet Spot?
Qwen3.8-27B is here. See which GGUF fits 24GB GPUs, the best llama.cpp starting point, context limits, MTP, and what makes this 27B model exciting.