How Much VRAM Do You Need for Local LLMs? 8B to 70B Guide
See which model sizes fit 8, 12, 16, 24, 48, and 80 GB GPUs, then account for quantization, KV cache, context length, and runtime overhead.
Topic hub · 36 articles · updated
GPUs, CUDA & VRAM.
What fits, what crashes, and what is really slowing your GPU down.
VRAM sizing, CUDA and PyTorch errors, out-of-memory diagnosis, multi-GPU behavior, and measured prefill and decode performance on RTX 4090-class hardware.
New to GPUs, CUDA & VRAM?
Start here
How to Reduce LLM VRAM Usage Without Breaking Performance
Learn how to reduce local LLM VRAM usage with quantization, shorter context, KV-cache compression, CPU offloading, smaller models, and runtime tuning.
Local LLM Slow? Diagnose Prefill, Decode, GPU Offload, and Memory
Find why a local LLM is slow by separating prompt processing from token generation, checking GPU offload, memory pressure, context growth, and CPU bottlenecks.
Search this page for your exact error text
Error fixes
- PyTorch CUDA OOM: Allocated vs Reserved GPU Memory
Diagnose CUDA out-of-memory errors by separating live tensors, PyTorch's caching allocator, non-PyTorch allocations and fragmentation before changing batch sizes or allocator settings.
- llama.cpp
failed to allocate buffer for rs cache: Fix Recurrent-State OOMFix llama.cpp startup failures reporting failed to allocate buffer for rs cache by separating recurrent-state memory from KV cache, GPU weights, and server concurrency.
- torch.cuda.is_available() Is False: Fix PyTorch CUDA Detection
Fix torch.cuda.is_available() returning False by checking nvidia-smi, the active PyTorch wheel, Python environment, GPU visibility, and Docker passthrough.
- vLLM KV Cache Too Small: Fix max_model_len Startup Failures
Fix vLLM startup errors caused by insufficient KV-cache capacity. Separate weight OOM from long-context sizing, batch concurrency and CUDA graph memory.
- llama.cpp SPLIT_AXIS_UNKNOWN: Fix Tensor-Split Startup Crashes
Fix llama.cpp SPLIT_AXIS_UNKNOWN assertions by testing layer split, updating old builds, isolating tensor-distribution graph failures, and separating them from VRAM errors.
- vLLM H100 FA4 Decode Up to 48% Slower: Check the SplitKV Pin
Diagnose H100/SM90 FA4 decode regressions by checking the pinned FlashAttention commit: v0.31.0 still uses the pre-fix pin, while current vLLM main includes PR #198.
- llama.cpp qwen4exp
SPLIT_MODE_TENSOR not implemented: Update the RuntimeFix the qwen4exp SPLIT_MODE_TENSOR not implemented error by updating stale llama.cpp builds, verifying the actual binary, and using layer split as a fallback.
- llama.cpp
failed to allocate compute pp buffers: Fix -ub and VRAMFix llama.cpp compute pp buffer failures by lowering -ub/--ubatch-size first, then checking VRAM headroom, backend allocation limits, --fit, and speculation.
- llama.cpp
failed to allocate buffer for kv cache: Fix VRAM and ContextFix llama.cpp KV-cache allocation failures by lowering context or parallelism, leaving VRAM headroom, checking cache types, and reading the real allocator error.
- vLLM CUDA Illegal Memory Access: Check Draft vs Target Context
Diagnose vLLM speculative-decoding CUDA illegal-memory-access crashes by comparing target and draft context limits; route over-limit requests without speculation or align the limits.
- llama.cpp MTP Acceptance Drop After CUDA MoE Fusion: Verify the Prompt First
Issue #29168 initially bisected an MTP acceptance drop to CUDA MoE fusion, but the reporter later found it was specific to one raw prompt and closed the issue.
- llama.cpp CUDA vs Vulkan: Which Backend Should You Use?
A practical llama.cpp CUDA-vs-Vulkan guide: verify the backend actually loaded, compare device support and kernels, benchmark the same workload, and avoid mistaking CPU fallback for a backend result.
- llama.cpp Max Context vs GPU Memory: Find the Real Limit
Find the largest llama.cpp context that fits your GPU by separating weights, KV cache, compute buffers, concurrency, auto-fit, and backend limits.
Understand it, then set it up
Guides & explainers
PagedAttention: How LLM Servers Stop KV Cache From Wasting Memory
AI Foundations #30 explains why variable-length KV caches fragment GPU memory, how paging divides cache into blocks, how logical token positions map to physical memory, and why this helps serving throughput.
Continuous Batching and Scheduling: How LLM Servers Share a GPU
AI Foundations #28 explains static batching, continuous batching, request scheduling, chunked prefill, head-of-line blocking, fairness, and why server throughput and user latency can pull in opposite directions.
Why llama.cpp's Blackwell NVFP4 Prefill Was Starving Tensor Cores
A new llama.cpp CUDA patch shows that Blackwell NVFP4 prefill was limited by tile delivery, barriers, and register pressure—not simply tensor-core arithmetic.
What Does Context Length Mean in an LLM?
Learn what LLM context length means, what counts toward the context window, how tokens, chat history, KV cache, VRAM, truncation, and long-context models work.
What Is a KV Cache? How It Works and Why It Uses VRAM
Learn what an LLM KV cache stores, why it speeds up token generation, how to estimate its VRAM use, and how context length and quantization affect memory.
Tested on real hardware
Measured results
Strata Linux Multi-GPU: 8 GiB Host-Registration Cap Cut Prefill 3×
A controlled RTX 4090 + RTX 3060 experiment traced a severe Strata Q2_0 multi-GPU prefill slowdown to the Linux expert-copy path. Allowing full CUDA host registration raised 32K prefill from 665 to 2,110 tok/s.
Qwen3.8-27B on RTX 4090: 5,000 tok/s Prefill Reproduced
An independent Ubuntu RTX 4090 reproduction of NInfer's Qwen3.8-27B A8 claims reaches 4,999.5 tok/s pp2048, 500.9 tok/s edit decode, and exact retrieval at 260K context.
Qwen3.8-27B at 200K: The Performance Cliff Is VRAM, Not IQ3
Independent RTX 4090 testing finds UD-IQ3_XXS only ~2% slower than UD-Q3_K_XL at matched context, while reducing the 200K GPU memory budget from 16 to 15 GiB cuts prompt processing by 19%.
What changed, and why it matters
Analysis & deep dives
vLLM CUDA Wheel and PyTorch ABI: Diagnose a Broken Upgrade
A version-aware guide to vLLM import failures after PyTorch or CUDA changes, with environment checks, clean-install controls, and a recovery decision tree.
llama.cpp Host-RAM MoE Pipeline Parallelism: What PR #29963 Changes
A source-level look at llama.cpp PR #29963, which explores pipeline parallelism when MoE expert weights remain in host RAM and must be uploaded to the GPU that owns each layer.
llama.cpp K-Pool Graph Reallocation: The GLM-5.3/Qwen4exp Fix
Merged llama.cpp PR 29958 stabilizes k-pool graph shapes for GLM-5.3 and Qwen4exp, addressing reallocation after mutable state changed the reserved graph.
MiMo 9B EXL3: 145 tok/s on RTX 4090, 56 tok/s on RTX 3060
Independent EXL3 4.0 bpw measurements for Xiaomi MiMo-V2.6-Distill-Qwen-9B across RTX 4090, RTX 3060 12GB, mixed-GPU layer split, long context, TabbyAPI, vision, and tool calling.
Adding a Second GPU Made My 4090 Slower on Qwen3.8
An independent Qwen3.8 Flash-Next benchmark on RTX 4090 + RTX 3060 showing how auto-fit, manual MoE placement, quantization, and MTP changed decode speed.
llama.cpp Q4_K Prefill Fusion: Gate, Up, and SwiGLU in One CUDA Path
A new llama.cpp CUDA PR targets duplicate activation quantization and intermediate FP32 traffic in dense Q4_K MLP prefill, with contributor-reported gains from 4.1% to 14.7%.
llama.cpp Learns to Borrow Idle VRAM for Vision
A production-inference analysis of PR #28320, which temporarily reclaims llama.cpp compute scratch so a vision projector can run on GPU without reducing context.
When Lossless Speculation Stops Being Lossless on Blackwell
Two new llama.cpp reports show how batch-dependent CUDA kernels can break greedy speculative-decoding equivalence on Blackwell GPUs.
Adaptive Speculative Decoding on RTX 4090: Fixed n=7 Was 13.5% Faster
Independent RTX 4090 CUDA testing of adaptive DFlash2 in llama.cpp: fixed n=7 reached 80.11 tok/s on structured JSON, versus 70.61 tok/s for adaptive 3–7.
DSpark PCTree on an RTX 4090: Higher Acceptance, Lower Throughput
An independent llama.cpp PCTree test on RTX 4090 with Qwen3-8B Q8: DSpark doubled decode speed, while wider trees raised acceptance but still lost throughput.
Running Qwen3.8-27B on an RTX 4090 with vLLM + OpenWebUI
A hands-on guide to running Qwen3.8-27B on one RTX 4090 with patched vLLM, 64K context, MTP speculative decoding, tool calling, and OpenWebUI.
Qwen3.8-27B on 24GB GPUs: The New Local LLM Sweet Spot?
Qwen3.8-27B is here. See which GGUF fits 24GB GPUs, the best llama.cpp starting point, context limits, MTP, and what makes this 27B model exciting.