Troubleshooting

vLLM DeepSeek-V4 max_model_len Auto-Fit Crash: Update or Set It Explicitly

Fix DeepSeek-V4 CUDA-graph startup assertions after max_model_len auto-fit by using vLLM 0.31.0+ with the ModelState fix or setting an explicit context that fits.

Approximately 11 min read

Quick fix: if DeepSeek-V4 starts only when more VRAM is free, and the failing log shows max_model_len being auto-reduced before a sparse-MLA/CUDA-graph assertion, use a vLLM build containing PR #58149. The fix is present in the v0.31.0 tag. If you cannot update, set an explicit --max-model-len that fits instead of relying on auto-fit.

The bug occurs when the engine reduces max_model_len for the available KV cache but a model-state consumer keeps the old length. Two parts of startup then prepare for different maximum sequence lengths.

On the reported DeepSeek-V4 sparse-MLA setup, that disagreement reaches a hard assertion during CUDA-graph capture:

max_model_len: reduced from 1048576 to 980224 to fit in available GPU memory
...
File ".../vllm/models/deepseek_v4/sparse_mla.py", in _build_c128a_metadata
    assert active_topk_width >= cm.max_seq_len // self.compress_ratio
AssertionError

The proposed patch in vLLM PR #58149 is tiny: when GPUModelRunner.update_max_model_len() receives the auto-fitted length, update self.model_state.max_model_len too.

The interesting part is not the line count. It is the lifecycle bug behind it:

Model loading copies the original context limit into several objects. KV profiling then reduces the limit, but one cached copy survives. Attention metadata uses the fitted value while CUDA-graph capture reads the old value, causing the startup assertion.

The reported configuration

Issue #57034 uses a development vLLM build whose relevant paths were reported to match upstream main at the time of investigation.

The environment was:

hardware: 2 x NVIDIA GB10
compute capability: 12.1
PyTorch: 2.13.0+cu130
CUDA: 13.0
Python: 3.12.3
model: DeepSeek-V4-Flash-0731
tensor parallel: 2
speculative decoding: DSpark, k=5
max model length: auto
full model context: 1,048,576
KV cache dtype: fp8
gpu memory utilization: 0.85

The important setting is not speculative decoding. It is:

--max-model-len auto

When vLLM profiles available memory and discovers that the KV cache cannot support the checkpoint’s full context, it is allowed to reduce the effective maximum length.

In the failing startup, the log reports:

max_model_len: reduced from 1048576 to 980224

The reporter measured about 9.81 GiB of available KV cache in that run.

On another startup with about 12.3 GiB available, no reduction was needed and the same configuration started successfully.

That is why the bug can look nondeterministic.

The model and command do not have to change. A small difference in free GPU memory can decide whether auto-fit executes, which decides whether the stale-state path exists.

Auto-fit is supposed to make startup safer

The purpose of context auto-fitting is straightforward.

Suppose the model advertises:

1,048,576 tokens

but the KV cache that fits in the current GPU memory budget can support only:

980,224 tokens

A reasonable runtime can choose:

Hardware profiling lowers the requested maximum to the effective KV capacity: 980,224 tokens.

That should turn a hard memory failure into a smaller but usable context window.

The bug in #57034 is not that vLLM computes the reduced value incorrectly.

The reduced value is propagated correctly to some runtime state. The problem is that not every consumer switches to the same new value.

One number existed in several places

Modern inference engines are full of derived state. A configuration value such as max_model_len does not remain a single field read from one global object forever.

Different components cache it because they need fast access or because they are constructed at different phases of startup.

The issue traces three important copies.

First, the V2 GPU runner has its own maximum length:

GPUModelRunner.max_model_len

Second, request state tracks a maximum:

req_states.max_model_len

Third, the model-state object caches the value when it is constructed:

ModelState.max_model_len

Before PR #58149, the runner’s update method handled the first two:

self.max_model_len = max_model_len
self.req_states.max_model_len = max_model_len

but not the third.

So after auto-fit, the state could become:

Consumer Maximum sequence length
Model configuration 980,224
GPUModelRunner 980,224
Request state 980,224
ModelState, stale copy 1,048,576

That is the whole bug in one table.

Why ModelState has the old value

The order of construction matters.

ModelState is created during model loading, before KV-memory profiling has finished deciding whether the full context fits.

At construction, it copies the then-current value from the model config:

1,048,576

Later, vLLM profiles memory and reduces the model configuration to:

980,224

The worker broadcasts that fitted value and calls the runner’s update_max_model_len() method.

But because ModelState already copied the original integer into its own field, changing the config object later does not retroactively change the cached copy.

This is a classic snapshot problem:

Assigning A = config.max_model_len stores the current integer. A later assignment to config.max_model_len does not update A.

Nothing about Python references saves you here because the cached field is a scalar value, not a live property pointing back to the config.

Why DeepSeek-V4 sparse MLA exposes the mismatch

A stale field is only dangerous if two later computations use different copies.

DeepSeek-V4’s sparse-MLA metadata path provides exactly that situation.

The attention metadata builders are constructed after auto-fitting. They therefore see the reduced model configuration and size their C128A structures for 980,224 tokens.

But CUDA-graph capture later asks ModelState for the worst-case sequence length. That field still says 1,048,576.

So one side prepares capacity for:

980224

while the capture path asks it to represent:

1048576

The assertion is therefore not random. It is a consistency check catching two different views of the same supposed runtime limit.

The numbers line up exactly

The issue provides enough information to follow the failed inequality.

For the reduced context, the metadata builder computes its compressed capacity around a compression ratio of 128.

The reported calculation is:

c128a_max_compressed
= align_up(cdiv(980224, 128), 128)
= 7680

During capture, the stale ModelState provides:

max_seq_len = 1048576

and:

1048576 / 128 = 8192

The active width is clamped by the capacity built from the reduced value:

active_topk_width <= 7680

but the assertion requires enough width for the stale capture length:

active_topk_width >= 8192

Those conditions cannot both be true.

So the runtime reaches:

7680 >= 8192

and fails.

That exact numerical contradiction is useful because it distinguishes this problem from a vague “DeepSeek CUDA graph issue.” The assertion is doing its job: two subsystems were configured with incompatible maximum lengths.

Why the crash happens during CUDA-graph capture

CUDA graphs try to capture reusable execution shapes so later work can launch with less CPU overhead.

For capture, the runtime often constructs worst-case or representative metadata rather than processing an ordinary live request.

The model-state capture path needs a maximum sequence length to describe that worst case.

Before the patch, it reads:

ModelState.max_model_len

which is precisely the stale field.

That explains the timing:

Loading, KV profiling, context fitting, and metadata construction can all complete before warmup reads the stale length. The engine then fails before serving requests.

The failure is therefore a startup crash, not a decode-time out-of-memory condition.

Why more free memory can make the bug disappear

This is one of the best diagnostic clues.

If there is enough KV capacity for the full advertised context, auto-fit does nothing:

The original and fitted limits both remain 1,048,576.

Now even a stale copy is harmless because every copy has the same value.

If memory pressure forces a reduction:

The original limit is 1,048,576, but the fitted limit becomes 980,224.

then the stale copy becomes observable.

So a pattern like this is plausible:

A run with more free VRAM can start; a run with less free VRAM can trigger fitting and expose the stale-state assertion.

That can look like GPU instability if the operator does not notice the auto-fit log line.

The first thing to compare between successful and failed startups should therefore be the resolved context length, not only the final traceback.

The proposed fix is deliberately small

PR #58149 changes GPUModelRunner.update_max_model_len() to fan the fitted value into ModelState too:

self.max_model_len = max_model_len
self.req_states.max_model_len = max_model_len
self.model_state.max_model_len = max_model_len

The principle is more important than the syntax:

if a runtime API promises to update a derived configuration value, every cached copy that affects execution must participate in that update contract.

The PR author notes that the older V1 runner already updates another derived value, effective_drafter_max_model_len, through this method. So propagating to model_state follows an existing pattern rather than inventing a separate mechanism.

The regression test checks fan-out, not GPU execution

PR #58149 adds a CPU-only test around the update method.

It creates a stand-in runner where all cached lengths begin at:

FULL_LEN = 1048576

then calls:

update_max_model_len(980224)

The first assertion checks that model_state.max_model_len changed.

The second test checks all represented cached copies and fails if any still carries the original number.

The PR reports:

Regression-test run Result
Without the source change 2 failed
With the source change 2 passed

These are upstream PR test results, not RAMGPT measurements.

Also note what the test does not prove. It does not load DeepSeek-V4, allocate a KV cache, or reproduce the CUDA-graph assertion on a GPU. It verifies the state-propagation mechanism that the issue identified as the cause.

That is appropriate for a regression test on a simple setter, but it should not be misreported as an end-to-end hardware reproduction.

The PR author checked current main’s source path

The original issue came from a development build and explicitly said the reporter had not independently reproduced the bug on a fresh upstream-main binary.

PR #58149 closes part of that uncertainty by inspecting current upstream source at commit 362a64b21d and confirming the three required conditions still existed before the patch:

ModelState caches the limit, the update method omits that copy, and capture subsequently reads it.

That is strong source evidence that the stale-state path exists on main.

It is still different from saying the PR author performed the original two-GB10 end-to-end reproduction on main. The article should keep those claims separate.

This can affect more than one model-state capture class

DeepSeek-V4 sparse MLA produces the visible assertion, but PR #58149 points out that the stale max_model_len field is consumed by multiple model-state capture paths:

default.py, encoder_decoder.py, encoder_only.py, and mamba_hybrid.py.

That does not prove all of those paths currently crash.

It means the stale value is part of shared state used beyond one DeepSeek-specific class. DeepSeek-V4’s C128A assertion is the loud symptom because its buffer capacity calculation exposes the mismatch directly.

This is another useful debugging pattern: the first model that trips an invariant is not necessarily the only model touched by the underlying state bug.

Why rope_state was not changed

The PR also mentions another object initialized with the pre-auto-fit context: rope_state.

Why not update everything blindly?

Because reducing the maximum context leaves that particular structure over-allocated, not under-sized. An over-sized RoPE state may waste memory, but it does not create the same reduced-buffer-versus-larger-request contradiction in this path.

So the patch stays narrow rather than broadening into unrelated lifecycle changes.

That restraint matters in runtime fixes. Once a stale-state bug is found, it is tempting to rewrite configuration ownership everywhere. A minimal repair plus a test for the exact update contract is easier to review and less likely to create another regression.

The immediate workaround does not require patching vLLM

Issue #57034 reports a practical workaround:

Set an explicit --max-model-len that already fits the KV cache.

The reporter used:

--max-model-len 900000

The exact safe value is hardware- and configuration-dependent. 900000 is not a universal recommendation.

The point is to avoid this state transition:

An explicit limit avoids constructing model state at a larger value and shrinking the configuration afterward.

If the runtime begins with an explicit length that fits, the objects are constructed with one consistent maximum and no auto-fit fan-out is required.

For operators running a build without the merged change, that is a useful mitigation.

A diagnostic checklist for this exact failure

If vLLM dies during warmup or CUDA-graph capture with a DeepSeek-V4 sparse-MLA assertion, check the startup log in this order:

  1. Check whether --max-model-len was auto or -1.
  2. Find any startup message reducing max_model_len.
  3. Locate _build_c128a_metadata in the traceback.
  4. Distinguish its assertion from a direct CUDA OOM.
  5. Test an explicit context length that fits.
  6. Check whether the build includes PR #58149.

If the successful startup shows no reduction but the failed startup does, that correlation is particularly strong.

Do not reduce the diagnosis to “the context is too large.” Auto-fit already recognized that and chose a smaller context. The bug is that one consumer failed to adopt the chosen value.

This differs from the recent speculative-draft context crash

The speculative-draft context investigation covers a different vLLM length mismatch: a target model accepted positions beyond a speculative draft model’s shorter RoPE cache, eventually surfacing as a CUDA illegal-memory-access error.

This new bug has a different contract and a different failure stage:

Failure Inconsistent values Failure stage
Speculative-draft mismatch Target context exceeds draft positional capacity Long request during inference
Auto-fit mismatch Cached length exceeds the newly fitted limit Startup CUDA-graph capture

The speculative case concerns two models with different positional limits. The auto-fit case concerns one runtime retaining an outdated value after profiling.

Current status

vLLM PR #58149 merged on September 22, 2026, as commit 71a239284507e4f9477f8256e04fd28d1aa9a2db.

The merged change includes a CPU regression test, and the v0.31.0 source exposes ModelState.max_model_len as a property backed by the resolved model config. For older builds, an explicit context length that fits the KV-cache budget remains the practical workaround.

Sources and further reading