Does llama.cpp Support NVFP4? Check the Exact Path Before You Convert
If you searched whether llama.cpp supports NVFP4, the answer depends on the checkpoint, converter, model architecture, backend, and quant-scale path. Use this checklist to identify the real blocker.
Approximately 3 min read
llama.cpp defines GGML_TYPE_NVFP4 and includes conversion handling for NVFP4 quantization metadata. A working deployment also needs the checkpoint’s scales, architecture mapping, backend operations, and hardware path to agree.
Find the first stage that fails:
| Observed failure | Inspect first |
|---|---|
| Converter rejects quantization metadata | Checkpoint config and converter recognition |
| Converter or loader cannot find a tensor | Architecture mapping, tensor names, and scale tensors |
| Model loads, then an operation fails | Backend implementation and device support |
| Model generates but runs slowly | Kernel selection, fallback, and GPU placement |
This is more useful than changing CUDA options when conversion has not produced a usable file.
Start with the checkpoint’s scale representation
NVFP4 values require scaling information. Two checkpoints labeled NVFP4 can store their quantization metadata differently and therefore take different conversion paths.
The conversion code examines metadata such as quant_method, quant_format, quant_algo, config groups, and per-layer entries. Save the checkpoint’s quantization configuration and the first conversion traceback when diagnosing a rejected model.
The existence of an NVFP4 tensor type in ggml.h establishes representation support, but a loader still needs to map each named tensor into its architecture-specific role.
Attention projections, experts, output heads, and recurrent state can require different mappings. Missing scale tensors or unexpected names therefore deserve inspection before re-quantizing the checkpoint.
What the upstream scale work changes
Discussion #22042 covers NVFP4 representation and scaling. PR #28898 proposes moving quantization-scale support into GGML to enable direct conversion of more FP8/NVFP4 safetensors checkpoints.
As checked October 16, 2026, that PR remains open and draft. Its prototype includes Vulkan validation and requires backend scale handling. It is implementation work under discussion, rather than evidence that every checkpoint already follows a supported direct-conversion path.
Record the exact revision used for conversion as well as inference. A result from a prototype branch may not apply to a packaged release.
Verify execution after loading
For a GGUF that loads successfully, capture the executable version and device list:
llama-cli --version
llama-cli --list-devices
Then run one prompt with a modest context and a single sequence. Save the first unsupported operation or tensor error, along with the backend and GPU.
Blackwell hardware capability alone does not establish which optimized kernel llama.cpp selected. A successful load, correct execution, and accelerated execution are three separate checks.
Once generation works, inspect actual GPU placement and benchmark prompt processing separately from decode. If another backend is available, a comparison using the same GGUF can help locate a backend-specific failure.
Keep the model artifact, revision, and workload fixed while changing one setting. That preserves the evidence needed to tell a converter improvement from a loader fix or a faster kernel.