Nemotron-3-Puzzle 75B-A9B Lands in llama.cpp — MTP Is Still Missing

If you tried to run NVIDIA’s Nemotron-Labs-3-Puzzle-75B-A9B through stock llama.cpp a few weeks ago, the interesting part was not the model’s benchmark chart. The interesting part was that its architecture did not fit llama.cpp’s existing Nemotron assumptions cleanly enough to convert and load as an ordinary GGUF.

That changed with llama.cpp PR #25444, merged on September 3.

The headline is simple: Nemotron-3-Puzzle-75B-A9B now has mainline llama.cpp support. The useful details are less simple. Puzzle is not just a smaller Nemotron-3-Super. Its MoE shape changes from layer to layer, its released checkpoints have appeared with more than one tensor naming convention, and its Multi-Token Prediction head still does not have an inference path in llama.cpp.

So “supported” currently means the trunk model works. It does not mean every feature NVIDIA ships with the model is available in llama.cpp yet.

The exact architecture llama.cpp had to learn

NVIDIA describes Puzzle-75B-A9B as a deployment-optimized compression of Nemotron-3-Super-120B-A12B. The total model shrinks from 120.7B parameters to 75.3B, while active parameters drop from 12.8B to 9.3B.

The base architecture is already unusual: a hybrid stack mixing Mamba, attention, and MoE blocks. Puzzle keeps 88 trunk blocks, but those blocks are not uniform.

The PR describes them as:

• 40 Mamba2 blocks;
• 8 attention blocks;
• 40 MoE blocks.

The attention blocks use GQA with 32 query heads and 2 KV heads, and the MoE blocks are heterogeneous. Their expert feed-forward width varies from 1280 to 2688, while num_experts_per_tok varies from 4 to 18 depending on the block.

That last point is the real loader problem.

A lot of model formats and runtimes are happier when architecture metadata is a scalar: one FFN width, one top-k expert count, one shape repeated across a model family. Puzzle breaks that assumption. NVIDIA’s own config has a block_configs array because individual layers need their own configuration.

PR #25444 changes llama.cpp so the existing expert-related GGUF metadata can be scalar or an array. Older uniform models still broadcast a scalar value, so the change is backward-compatible without inventing a new GGUF key just for Puzzle.

That is a small-looking format change with a large architectural implication: GGUF now has to describe a model whose routed compute budget is deliberately uneven across depth.

Why “75B-A9B” is not enough to identify it

Puzzle and Nemotron-3-Super both have 88 layers. That creates another awkward problem: layer count by itself does not tell llama.cpp which variant it is looking at.

The merged implementation distinguishes them by checking the per-layer expert top-k metadata. Puzzle is heterogeneous; Super is uniform.

This is exactly the kind of detail that gets lost in model-size shorthand.

Two models can share a broad family name, layer count, hybrid structure, and MoE machinery while requiring different loader logic because one of them changes expert activation policy from layer to layer.

For anyone debugging a future unknown model architecture, tensor-shape mismatch, or bad GGUF conversion on a new MoE, this PR is worth reading. The architectural identifier is only the first problem. The runtime also has to represent what varies inside that architecture.

The converter had to handle checkpoint drift too

There was a second source of friction: the released checkpoints did not all use the same tensor names.

The PR supports both the original Nemotron-style backbone.* naming and NVIDIA’s later BF16 upload using model.*, including model.layers.*, model.embeddings, and model.norm_f.

It also normalizes the router-bias spelling from e_score_correction_bias to the form expected by the existing Nemotron machinery.

This is mundane compared with a new kernel, but it is exactly the kind of thing that determines whether a conversion script works for the file people actually downloaded.

The converter also handles the FP8 checkpoint by absorbing the weight scales during dequantization before writing GGUF.

In other words, PR #25444 is not merely an enum entry for NemotronHPuzzleForCausalLM. It is model-family plumbing across config parsing, GGUF metadata, tensor naming, model identification, and loader validation.

The important missing piece: MTP

NVIDIA ships Puzzle with Multi-Token Prediction. The model card explicitly lists MTP as part of the architecture and deployment stack, and NVIDIA’s paper discusses it as one of the techniques used to improve serving efficiency.

But llama.cpp’s merged Puzzle support does not load or execute that draft head yet.

The PR originally included MTP/NEXTN support. Maintainers asked for the work to be split so the base model port and the draft-head path could be reviewed independently.

The merged converter therefore skips mtp.* tensors for Puzzle, and the model class sets:

supports_mtp_export = False

This is not just a converter toggle. Puzzle’s MTP head uses mtp_block_configs, which does not match the mtp.layers.* layout already supported for other NemotronH variants. There is also no Puzzle-specific MTP inference path in the current tree.

That distinction matters for search results and model cards. If you see “Nemotron-3-Puzzle now works in llama.cpp,” read it as:

GGUF conversion and normal autoregressive inference are supported. Puzzle’s native MTP acceleration is not yet part of that support.

A follow-up PR is expected to handle it separately.

What was actually tested

The PR author reports an earlier full-scope branch was tested end to end on a Strix Halo system with 128 GB unified memory and a Radeon 8060S.

Reported validation included:

• FP8 to Q8_0 conversion producing an 83.4 GB GGUF with 781 tensors;
• successful load and coherent generation with -ngl 99;
• an 18K-token long-context needle retrieval;
• WikiText-2 perplexity checks across 24 chunks;
• decode in the 17.7–19.9 tokens/s range;
• prefill in the 211–243 tokens/s range on ROCm.

The reported perplexities were 5.325 for Q8_0, 5.377 for an IQ4_XS mix, and 5.404 for Q4_K_M.

Those are the PR author’s measurements, not RAMGPT benchmarks.

There is also an important qualification in the PR: after MTP was split out, the slimmed branch was compile-gated but was not re-run through the complete Puzzle conversion/load smoke test on the author’s machine because the local model files were no longer available. The conversion path was checked during review, the tree built cleanly, and the final model code was verified to emit the expected 88 trunk blocks without nextn references.

That is a much more useful statement than pretending “merged” automatically means every combination has been exhaustively re-tested.

One more backend warning: Vulkan on Strix Halo

The PR also contains a note that is separate from the model-support change but worth preserving.

On the author’s Strix Halo system, Vulkan using RADV on gfx1151 reportedly suffered a severe decode collapse with 4-bit MoE quants in the mul_mat_id path, while Q8_0 did not show the same problem. ROCm/HIP was the faster backend for this model on that machine.

Again, that is one contributor’s hardware observation, not a general Vulkan verdict.

But it creates a clean follow-up search intent: if somebody loads Puzzle successfully on Strix Halo and then sees unexpectedly terrible 4-bit decode speed under Vulkan, the problem may not be the quant itself.

That deserves its own issue or reproduction rather than being mixed into the basic support story.

Why Puzzle is a particularly interesting local model

The model was not designed primarily to win a parameter-count beauty contest. NVIDIA’s paper frames it around deployment efficiency.

At matched user-throughput constraints on a single 8×B200 node, NVIDIA reports roughly 2× the server throughput of Nemotron-3-Super. For a 1M-token deployment on a single H100, NVIDIA reports sustainable concurrency increasing from one request to eight.

Those are NVIDIA’s server measurements, on NVIDIA hardware, under the paper’s workload definitions. They are not evidence that a Q4 GGUF on a desktop GPU will be twice as fast as Super.

What does transfer to the local-AI world is the design idea: instead of making every MoE block equally large and equally active, Puzzle spends routed capacity unevenly across the network.

That is why llama.cpp needed per-layer expert metadata in the first place.

The model’s compression strategy is visible directly in the runtime implementation.

What to use now

If you want to run Nemotron-3-Puzzle-75B-A9B as GGUF, use a llama.cpp build that includes PR #25444 or any later mainline build containing merge commit c61b98b875eaa5e654a3f5c73b34c310d2c6ab4c.

Community GGUFs built around the PR already exist, including the PR author’s repository on Hugging Face.

If a GGUF was produced before mainline support landed, check how it was converted. Some early community conversions were pinned to intermediate PR revisions and may describe support that included code later split out or changed during review.

And if your goal is specifically MTP-assisted generation, wait for the Puzzle draft-head follow-up rather than assuming the mtp tag on NVIDIA’s model card means stock llama.cpp can use it today.

That is the state of support as of September 8, 2026:

Nemotron-3-Puzzle-75B-A9B is now a mainline llama.cpp architecture. Its heterogeneous MoE trunk is supported. Its native MTP path is not — yet.