vLLM Mistral-Large-3 YaRN Regression: Why Accuracy Fell 4.58 GSM8K Points

The most expensive inference bugs do not always crash.

Sometimes the server starts, the checkpoint loads, tokens stream normally, and the model simply becomes less accurate.

On September 18, 2026, vLLM PR #57563 reported exactly that failure mode for Mistral-Large-3. The proposed fix is only a few lines in the Mistral configuration adapter, but the upstream author reports a 4.58 percentage-point GSM8K recovery after restoring one missing RoPE flag.

That makes this a useful debugging case because the failure is not in a GEMM kernel, quantizer, tokenizer, or model weight. It is in the translation between one model’s configuration format and the attention implementation that consumes it.

This article is a source analysis of the upstream PR. RAMGPT did not independently run the 675B checkpoint or reproduce the benchmark numbers below.

The short version

Mistral-Large-3 ships a Mistral-format params.json containing two position-scaling mechanisms:

"yarn": {
  "factor": 36,
  "beta": 32,
  "alpha": 1,
  "original_max_position_embeddings": 8192,
  "apply_scale": false
},
"llama_4_scaling": {
  "beta": 0.1,
  "original_max_position_embeddings": 8192
}

The important detail is:

apply_scale = false

According to PR #57563, this means the model wants YaRN frequency interpolation without YaRN’s magnitude correction, while also using Llama-4-style position-dependent attention scaling.

vLLM correctly remapped apply_scale: false into:

attention_factor = 1.0

but after an earlier YaRN cleanup, the synthesized configuration no longer carried the separate flag that DeepSeek-style attention uses to choose the correct RoPE path:

apply_yarn_scaling = false

The result was a configuration that looked reasonable but selected the wrong attention-scaling behavior.

Why Mistral-Large-3 goes through DeepSeek attention code

This is the part that makes the bug easy to miss when searching by model name.

In vLLM, Mistral-Large-3 is represented as:

MistralLarge3ForCausalLM

which is a subclass of:

DeepseekV3ForCausalLM

The Mistral checkpoint is therefore adapted into a configuration that eventually drives DeepSeek-style attention code.

The path is roughly:

Mistral params.json
-> vLLM Mistral config adapter
-> synthesized rope_parameters
-> DeepSeek-family attention implementation

If you only search for:

Mistral Large 3 bad accuracy

you can miss the real dependency: a Mistral configuration field controls a branch inside DeepSeek attention.

That is exactly the kind of runtime search asymmetry that makes source-level debugging valuable.

The regression came from a reasonable cleanup

PR #57563 points to vLLM commit c191787a68, from PR #56446, merged September 11.

That earlier change aligned vLLM’s YaRN handling with Transformers and stopped re-scaling max_model_len.

The problem was not that the cleanup was obviously reckless. The regression appeared because Mistral-Large-3 has an unusual combination:

Mistral config remapping
+ YaRN frequency interpolation
+ no YaRN magnitude scaling
+ Llama-4-style attention scaling
+ DeepSeek-derived model implementation

The PR author notes that Mistral-Large-3 is the model that combines these conditions, which explains why a broad config change could pass while this specific checkpoint moved substantially.

What the broken config looked like

The upstream author loaded the checkpoint through:

get_config(..., config_format="mistral")

and inspected the synthesized rope_parameters on current main.

It contained:

attention_factor: 1.0

but did not contain:

apply_yarn_scaling: false

Those two values may sound redundant, but in this code path they are not.

The test added by PR #57563 explains the distinction directly: Transformers-style configuration expresses the disabled magnitude correction using attention_factor = 1.0, while DeepseekV2Attention and related implementations use apply_yarn_scaling to decide between their RoPE scaling paths.

So one layer of the stack had preserved the numeric consequence, while another layer still needed the semantic switch.

The proposed fix is only three lines

The patch in vllm/transformers_utils/configs/mistral.py extends the existing apply_scale remap:

if not yarn_config.pop("apply_scale", True):
    config["rope_parameters"]["attention_factor"] = 1.0
    config["rope_parameters"]["apply_yarn_scaling"] = False
    config["ignore_keys_at_rope_validation"] = {"apply_yarn_scaling"}

The first line already existed.

The new behavior carries the disabled YaRN magnitude scaling far enough downstream for the DeepSeek attention implementation to see it.

The accompanying regression test constructs a Mistral-Large-3-like config and verifies all three important facts:

architecture == MistralLarge3ForCausalLM
attention_factor == 1.0
apply_yarn_scaling == false

That is a strong test because it exercises the configuration translation boundary rather than only checking a helper in isolation.

Why one missing boolean can change model quality

RoPE scaling is part of the attention computation, not just metadata printed during startup.

If the runtime chooses the wrong scaling formula, every affected attention layer can operate with systematically different position-dependent values.

The PR’s regression test description is more specific: without the missing flag, the DeepSeek path can apply a spurious yarn_get_mscale(factor)^2 attention scaling.

That means the failure is not random noise from one kernel launch. It changes the mathematical behavior of the model across inference.

And because the dimensions still match, nothing has to crash.

weights load        -> yes
server starts       -> yes
requests complete   -> yes
outputs look fluent -> possibly
accuracy preserved  -> no

This is why a clean startup is not sufficient validation after model-config changes.

The upstream A/B result

PR #57563 evaluates the same Mistral-Large-3 checkpoint on GSM8K with the same serving configuration for both arms.

The checkpoint was:

Mistral-Large-3-675B-Instruct-2512-NVFP4

The server setup used TP1, DP4, EP4, --max-model-len 5120, temperature 0, seed 42, and 1,319 GSM8K questions. The author ran five passes per arm.

These are upstream measurements from the PR author, not RAMGPT benchmarks:

The reported mean difference is:

92.63 - 88.05 = 4.58 percentage points

The PR author also notes that the fixed result matches measurements taken before #56446 was merged.

The important engineering conclusion is not that GSM8K is a perfect model-quality metric. It is that the before/after change is large enough to expose a real behavioral regression under a controlled runtime comparison.

How I would diagnose this class of regression

If a model becomes worse after a runtime update but still serves normally, I would avoid starting with sampling parameters.

First establish a version boundary:

last known-good vLLM commit
first known-bad vLLM commit
current commit

Then keep these fixed:

checkpoint
prompt/eval set
chat template
tokenizer
sampling parameters
parallelism
quantization

If quality moves with the runtime commit, inspect changes in:

model config adapters
rope_parameters
architecture mapping
attention backend selection
quantization config translation
chat-template/parser behavior

For Mistral-Large-3 specifically, print the synthesized configuration rather than trusting the original params.json alone.

The runtime does not execute the source configuration directly. It executes the remapped configuration.

A useful rule: test semantic adapters as public APIs

Configuration adapters can look like plumbing, but for modern inference engines they are effectively part of the numerical model implementation.

A field can travel through several naming systems:

checkpoint-native field
-> adapter field
-> Transformers-compatible field
-> runtime-specific field
-> kernel/attention branch

Every translation is an opportunity to preserve the value while losing the meaning.

The regression test in #57563 is a good pattern: construct the native config, run the real adapter, and assert the exact fields consumed downstream.

That is stronger than testing only whether JSON parsing succeeds.

Should you just upgrade vLLM?

Not yet, unless the build you install actually contains the fix.

At the time of this analysis, PR #57563 is open and unmerged.

So the practical choices for an affected deployment are:

1. pin a known-good revision from before the regression,
2. test the PR branch/patch in your own environment,
3. wait for the fix to merge and verify the release/nightly commit that contains it.

Do not assume that a package called latest contains an open PR.

And do not assume that reducing max_model_len repairs a wrong RoPE configuration. The upstream evaluation here already used a 5,120-token maximum; the problem is the scaling path, not simply an extreme context length.

The practical takeaway

This regression is a useful reminder that model support is more than “the checkpoint loads.”

For a complex model family, correctness depends on preserving architecture semantics through every compatibility layer:

checkpoint metadata
-> config adapter
-> model class
-> attention implementation
-> numerical behavior

Mistral-Large-3 did not need a new matrix kernel to recover the reported accuracy. It needed one semantic fact — do not apply YaRN magnitude scaling — to survive the trip from params.json to DeepSeek-style attention.

When a runtime update causes a large quality drop without a crash, configuration translation deserves the same suspicion as kernels and weights.