llama.cpp FP8/NVFP4 Conversion: Why Quantization Scales Need First-Class Support

A quantized checkpoint is not fully described by saying its weights are FP8 or NVFP4.

The missing piece is often the scale that tells the runtime how the stored low-precision value maps back into the numerical range used by the computation.

That sounds like metadata. In practice it can become an architectural boundary.

On September 14, llama.cpp contributor 0cc4m opened draft PR #28898, “ggml: support quant scales for fp8 and nvfp4.” The proposal moves quantization-scale handling into GGML itself rather than leaving conversion paths and individual backends to reconstruct the relationship ad hoc.

The immediate goal is practical: according to the contributor, this would allow direct conversion of many FP8 and NVFP4 safetensors checkpoints from Hugging Face while keeping existing MUL_MAT and MUL_MAT_ID uses backward compatible.

The more interesting part is what this says about the next stage of local-model formats.

A quantized tensor can need another tensor to explain it

A simplified scaled representation can be written as:

real_value ≈ stored_low_precision_value × scale

The scale may apply to an entire tensor, a channel, an expert, or a finer grouping depending on the format.

That means a converter cannot always treat a weight tensor as a self-contained block of numbers. It also has to preserve the scaling structure expected by the computation.

PR #28898 adds GGML-side support for tensor scaling described in its commit history as including per-tensor, per-channel, per-expert, and per-expert-per-channel cases.

This is important for MoE checkpoints in particular: an expert dimension can make scale layout part of the tensor semantics rather than a single scalar attached to a weight matrix.

Why this belongs below the model loader

One approach is to solve every new scaled format in the converter or model implementation.

That works until multiple architectures, operations, and backends need the same concept.

The PR instead proposes making scales visible to GGML operations. The contributor describes backend support as relatively simple: the scale can be applied through an epilogue multiplication after the relevant matrix operation.

They also report having explored graph-based fallbacks, but rejected that direction for this prototype because of the added complexity.

This is a useful design distinction:

converter workaround
    -> knows how one checkpoint encodes scales

GGML scale semantics
    -> lets operations and backends understand scaled tensors directly

The second approach moves responsibility deeper into the runtime, but it can reduce repeated model-specific conversion logic.

The conversion problem is the user-facing part

The most immediately useful claim in the PR is not a speed number.

It is checkpoint compatibility.

The contributor says the design supports direct conversion of many FP8/NVFP4 safetensors models on Hugging Face. That matters because modern model releases increasingly arrive in deployment-oriented numerical formats rather than only BF16/FP16 weights that can be quantized later by the local runtime community.

If a checkpoint stores scaled FP8 weights, a converter needs to understand both the low-precision values and their scale representation. Losing or misinterpreting the latter is not merely a small precision change; it changes the represented weights.

So an error such as “the tensor dtype is recognized, but this checkpoint still cannot convert correctly” may actually be a scale-semantics problem, not a missing FP8 datatype enum.

Why MUL_MAT_ID matters

The PR explicitly mentions compatibility with both MUL_MAT and MUL_MAT_ID.

MUL_MAT_ID is relevant to routed/expert computation, where different rows or tokens can select different weight matrices. If scales can vary by expert, the runtime needs the selected expert’s scale information to travel with the selected weights.

This is where apparently small format work starts touching model architecture.

A single dense matrix can often be imagined as:

weight -> matmul -> output

A scaled expert representation is closer to:

expert selection
    -> quantized expert weight
    -> corresponding scale
    -> matmul
    -> scaling
    -> output

That relationship has to survive conversion, GGUF representation, graph construction, and backend execution.

Vulkan is being used as a validation path

The draft also includes Vulkan FP8 and scaled-matmul support. The contributor says the primary purpose is validation and that the Vulkan portion can be split later.

That is a useful signal about the maturity of the work: PR #28898 is a prototype and discussion branch, not merged production support.

At the time of this analysis it is still marked Draft and requires code-owner review.

Anyone converting a model today should therefore not read this article as “llama.cpp now supports every FP8/NVFP4 checkpoint.”

The accurate statement is narrower:

llama.cpp is actively prototyping first-class GGML scale semantics intended to make scaled FP8/NVFP4 conversion and backend execution more general.

No performance conclusion yet

The PR does not provide a RAMGPT-verifiable performance comparison, and RAMGPT has not benchmarked this branch.

That is worth stating because FP8 and NVFP4 discussions quickly become throughput discussions. This PR is primarily about representation, conversion, and execution semantics. Even if the final implementation later enables faster kernels, that is a separate measurement question.

For now the interesting engineering change is that quantization scales are being promoted from something a converter/backend may privately know into something GGML can model explicitly.

What to watch next

The draft has several questions that will determine whether this becomes a durable llama.cpp interface:

• whether maintainers accept scale semantics at the GGML tensor/op level;
• how the scale metadata is represented through GGUF conversion;
• which backends implement scaled matmul directly;
• whether fallback behavior is needed for backends without native support;
• which FP8/NVFP4 Hugging Face checkpoints become directly convertible once the design stabilizes.

If the proposal lands, the visible feature may look deceptively simple: more checkpoints convert and run.

Underneath, the important change is that the runtime has learned that a quantized tensor’s numerical meaning can depend on structured scale data that must remain attached to the computation.