llama.cpp CUDA vs Vulkan: Which Backend Should You Use?
A practical llama.cpp CUDA-vs-Vulkan guide: verify the backend actually loaded, compare device support and kernels, benchmark the same workload, and avoid mistaking CPU fallback for a backend result.
Approximately 3 min read
On an NVIDIA GPU, start with CUDA as the baseline. Test Vulkan when you need a cross-vendor deployment, encounter a CUDA-specific problem, or want to compare a particular workload.
Before timing either backend, confirm that both runs use the intended GPU. A backend can register successfully while significant work remains on the CPU.
llama-server --version
llama-server --list-devices
Read each run’s startup log for the detected device, offloaded layers, and memory placement. An executable’s filename does not establish which backend performed the inference.
Make the comparison about kernels
CUDA and Vulkan execute a shared high-level model graph through different backend implementations. Operation coverage, fused kernels, quantized formats, Flash Attention, and multi-GPU paths can differ. The upstream operation matrix helps identify which paths a model needs.
Use builds from the same llama.cpp revision. Otherwise, a kernel added between releases can look like a CUDA-versus-Vulkan advantage.
| Hold fixed | Why it matters |
|---|---|
| Exact GGUF | Quantization and architecture select different operations |
| Physical GPU | Device numbering can differ between backends |
| Context and KV types | These change memory demand and cache traffic |
| Batch and micro-batch | These change prompt-processing kernel shapes |
| Flash Attention setting | The attention execution path may change |
| GPU layer placement | CPU offload adds another bottleneck |
| Prompt and output length | Short and long workloads exercise different costs |
Where a setting is unsupported by one backend, record the difference explicitly. A deployment comparison can still be useful, but its result includes that configuration difference.
Read prompt processing and generation separately
Use llama-bench to measure prompt processing and token generation. Its README documents benchmark options for the build you are using.
Record both throughput figures, peak device memory, host RAM, and startup time. Repeat the same cases to see whether the ordering survives run-to-run variation.
Prompt processing supplies many positions at once. Generation repeatedly produces the next position and reads weights plus retained attention state. A backend can lead on long-prompt processing while trailing on generation.
The architecture matters too: dense Transformers, MoE models, and recurrent hybrids create different operation mixes. Keep results attached to the tested model and quantization rather than extending one result to every GGUF.
When a result looks unexpectedly slow
First compare actual placement. Check --gpu-layers and --device in the binary’s help and the resulting startup log. Auto-fit can place different amounts of work on the GPU in two runs.
High CPU use, heavy host-memory traffic, or fewer offloaded layers can explain a large gap. On a multi-GPU machine, verify physical device identity; CUDA visibility settings can change the device list.
If both runs have matched placement, narrow the workload. Compare prompt processing against generation, then short against long context. This helps distinguish an attention path, quantized matrix operation, or memory-capacity problem from a general backend preference.
Choose the configuration that meets the actual workload’s speed, memory, stability, and packaging needs. Keep its commit and command with the result so a later backend update can be tested against the same baseline.